From 4521e5f1830687d043c4495bac673798e73e2e94 Mon Sep 17 00:00:00 2001 From: Philipp Moritz Date: Fri, 16 Sep 2016 18:26:57 -0700 Subject: [PATCH 01/17] Initial commit --- .gitignore | 33 +++++++++ LICENSE | 201 +++++++++++++++++++++++++++++++++++++++++++++++++++++ README.md | 2 + 3 files changed, 236 insertions(+) create mode 100644 .gitignore create mode 100644 LICENSE create mode 100644 README.md diff --git a/.gitignore b/.gitignore new file mode 100644 index 000000000..f805e810e --- /dev/null +++ b/.gitignore @@ -0,0 +1,33 @@ +# Object files +*.o +*.ko +*.obj +*.elf + +# Precompiled Headers +*.gch +*.pch + +# Libraries +*.lib +*.a +*.la +*.lo + +# Shared objects (inc. Windows DLLs) +*.dll +*.so +*.so.* +*.dylib + +# Executables +*.exe +*.out +*.app +*.i*86 +*.x86_64 +*.hex + +# Debug files +*.dSYM/ +*.su diff --git a/LICENSE b/LICENSE new file mode 100644 index 000000000..8dada3eda --- /dev/null +++ b/LICENSE @@ -0,0 +1,201 @@ + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "{}" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright {yyyy} {name of copyright owner} + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/README.md b/README.md new file mode 100644 index 000000000..fb354f8ea --- /dev/null +++ b/README.md @@ -0,0 +1,2 @@ +# yokoh +A local scheduler and node manager for Ray From 9e4203c18c4ee56f881e0165d0d4a1d07b0b4644 Mon Sep 17 00:00:00 2001 From: Robert Nishihara Date: Sat, 17 Sep 2016 00:30:31 -0700 Subject: [PATCH 02/17] Update README.md --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index fb354f8ea..b95056c2f 100644 --- a/README.md +++ b/README.md @@ -1,2 +1,2 @@ -# yokoh -A local scheduler and node manager for Ray +# Halo +A local scheduler and node manager for Ray. From 675ea40171b6ccec135b64e8d2b9166e04d20003 Mon Sep 17 00:00:00 2001 From: Philipp Moritz Date: Sun, 25 Sep 2016 22:49:55 -0700 Subject: [PATCH 03/17] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index b95056c2f..92e08a503 100644 --- a/README.md +++ b/README.md @@ -1,2 +1,2 @@ -# Halo +# Photon A local scheduler and node manager for Ray. From 184242a85c7ed85766e20dfcd117304ba22c8c5a Mon Sep 17 00:00:00 2001 From: Robert Nishihara Date: Tue, 27 Sep 2016 00:13:28 -0700 Subject: [PATCH 04/17] Add basic travis file and lint check. --- .travis.yml | 36 ++ .travis/check-git-clang-format-output.sh | 18 + .travis/git-clang-format | 476 +++++++++++++++++++++++ 3 files changed, 530 insertions(+) create mode 100644 .travis.yml create mode 100644 .travis/check-git-clang-format-output.sh create mode 100644 .travis/git-clang-format diff --git a/.travis.yml b/.travis.yml new file mode 100644 index 000000000..e77f446da --- /dev/null +++ b/.travis.yml @@ -0,0 +1,36 @@ +sudo: required + +language: generic + +matrix: + include: + - os: linux + dist: trusty + python: "2.7" + - os: linux + dist: trusty + python: "3.5" + - os: osx + osx_image: xcode7 + python: "2.7" + - os: osx + osx_image: xcode7 + python: "3.5" + - os: linux + dist: trusty + python: "2.7" + env: LINT=1 + before_install: + # In case we ever want to use a different version of clang-format: + #- wget -O - http://apt.llvm.org/llvm-snapshot.gpg.key | sudo apt-key add - + #- echo "deb http://apt.llvm.org/trusty/ llvm-toolchain-trusty main" | sudo tee -a /etc/apt/sources.list > /dev/null + - sudo apt-get update -qq + - sudo apt-get install -qq clang-format-3.8 + install: [] + script: + - .travis/check-git-clang-format-output.sh + +install: + - make + +script: diff --git a/.travis/check-git-clang-format-output.sh b/.travis/check-git-clang-format-output.sh new file mode 100644 index 000000000..d71f78357 --- /dev/null +++ b/.travis/check-git-clang-format-output.sh @@ -0,0 +1,18 @@ +#!/bin/bash +if [ "$TRAVIS_PULL_REQUEST" == "false" ] ; then + # Not in a pull request, so compare against parent commit + base_commit="HEAD^" + echo "Running clang-format against parent commit $(git rev-parse $base_commit)" +else + base_commit="$TRAVIS_BRANCH" + echo "Running clang-format against branch $base_commit, with hash $(git rev-parse $base_commit)" +fi +output="$(.travis/git-clang-format --binary clang-format-3.8 --commit $base_commit --diff --exclude ^thirdparty/)" +if [ "$output" == "no modified files to format" ] || [ "$output" == "clang-format did not modify any files" ] ; then + echo "clang-format passed." + exit 0 +else + echo "clang-format failed:" + echo "$output" + exit 1 +fi diff --git a/.travis/git-clang-format b/.travis/git-clang-format new file mode 100644 index 000000000..b0e458303 --- /dev/null +++ b/.travis/git-clang-format @@ -0,0 +1,476 @@ +#!/usr/bin/env python +# +#===- git-clang-format - ClangFormat Git Integration ---------*- python -*--===# +# +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. +# +#===------------------------------------------------------------------------===# + +r""" +clang-format git integration +============================ + +This file provides a clang-format integration for git. Put it somewhere in your +path and ensure that it is executable. Then, "git clang-format" will invoke +clang-format on the changes in current files or a specific commit. + +For further details, run: +git clang-format -h + +Requires Python 2.7 +""" + +import argparse +import collections +import contextlib +import errno +import os +import re +import subprocess +import sys + +usage = 'git clang-format [OPTIONS] [] [--] [...]' + +desc = ''' +Run clang-format on all lines that differ between the working directory +and , which defaults to HEAD. Changes are only applied to the working +directory. +The following git-config settings set the default of the corresponding option: + clangFormat.binary + clangFormat.commit + clangFormat.extension + clangFormat.style +''' + +# Name of the temporary index file in which save the output of clang-format. +# This file is created within the .git directory. +temp_index_basename = 'clang-format-index' + + +Range = collections.namedtuple('Range', 'start, count') + + +def main(): + config = load_git_config() + + # In order to keep '--' yet allow options after positionals, we need to + # check for '--' ourselves. (Setting nargs='*' throws away the '--', while + # nargs=argparse.REMAINDER disallows options after positionals.) + argv = sys.argv[1:] + try: + idx = argv.index('--') + except ValueError: + dash_dash = [] + else: + dash_dash = argv[idx:] + argv = argv[:idx] + + default_extensions = ','.join([ + # From clang/lib/Frontend/FrontendOptions.cpp, all lower case + 'c', 'h', # C + 'm', # ObjC + 'mm', # ObjC++ + 'cc', 'cp', 'cpp', 'c++', 'cxx', 'hpp', # C++ + # Other languages that clang-format supports + 'proto', 'protodevel', # Protocol Buffers + 'js', # JavaScript + 'ts', # TypeScript + ]) + + p = argparse.ArgumentParser( + usage=usage, formatter_class=argparse.RawDescriptionHelpFormatter, + description=desc) + p.add_argument('--binary', + default=config.get('clangformat.binary', 'clang-format'), + help='path to clang-format'), + p.add_argument('--commit', + default=config.get('clangformat.commit', 'HEAD'), + help='default commit to use if none is specified'), + p.add_argument('--diff', action='store_true', + help='print a diff instead of applying the changes') + p.add_argument('--extensions', + default=config.get('clangformat.extensions', + default_extensions), + help=('comma-separated list of file extensions to format, ' + 'excluding the period and case-insensitive')), + p.add_argument('--exclude', help='Exclude files matching this regex.') + p.add_argument('-f', '--force', action='store_true', + help='allow changes to unstaged files') + p.add_argument('-p', '--patch', action='store_true', + help='select hunks interactively') + p.add_argument('-q', '--quiet', action='count', default=0, + help='print less information') + p.add_argument('--style', + default=config.get('clangformat.style', None), + help='passed to clang-format'), + p.add_argument('-v', '--verbose', action='count', default=0, + help='print extra information') + # We gather all the remaining positional arguments into 'args' since we need + # to use some heuristics to determine whether or not was present. + # However, to print pretty messages, we make use of metavar and help. + p.add_argument('args', nargs='*', metavar='', + help='revision from which to compute the diff') + p.add_argument('ignored', nargs='*', metavar='...', + help='if specified, only consider differences in these files') + opts = p.parse_args(argv) + + opts.verbose -= opts.quiet + del opts.quiet + + commit, files = interpret_args(opts.args, dash_dash, opts.commit) + changed_lines = compute_diff_and_extract_lines(commit, files) + if opts.verbose >= 1: + ignored_files = set(changed_lines) + filter_by_extension(changed_lines, opts.extensions.lower().split(',')) + if opts.exclude: + for filename in changed_lines.keys(): + if re.match(opts.exclude, filename): + del changed_lines[filename] + if opts.verbose >= 1: + ignored_files.difference_update(changed_lines) + if ignored_files: + print 'Ignoring changes in the following files:' + for filename in ignored_files: + print ' ', filename + if changed_lines: + print 'Running clang-format on the following files:' + for filename in changed_lines: + print ' ', filename + if not changed_lines: + print 'no modified files to format' + return + # The computed diff outputs absolute paths, so we must cd before accessing + # those files. + cd_to_toplevel() + old_tree = create_tree_from_workdir(changed_lines) + new_tree = run_clang_format_and_save_to_tree(changed_lines, + binary=opts.binary, + style=opts.style) + if opts.verbose >= 1: + print 'old tree:', old_tree + print 'new tree:', new_tree + if old_tree == new_tree: + if opts.verbose >= 0: + print 'clang-format did not modify any files' + elif opts.diff: + print_diff(old_tree, new_tree) + else: + changed_files = apply_changes(old_tree, new_tree, force=opts.force, + patch_mode=opts.patch) + if (opts.verbose >= 0 and not opts.patch) or opts.verbose >= 1: + print 'changed files:' + for filename in changed_files: + print ' ', filename + + +def load_git_config(non_string_options=None): + """Return the git configuration as a dictionary. + All options are assumed to be strings unless in `non_string_options`, in which + is a dictionary mapping option name (in lower case) to either "--bool" or + "--int".""" + if non_string_options is None: + non_string_options = {} + out = {} + for entry in run('git', 'config', '--list', '--null').split('\0'): + if entry: + name, value = entry.split('\n', 1) + if name in non_string_options: + value = run('git', 'config', non_string_options[name], name) + out[name] = value + return out + + +def interpret_args(args, dash_dash, default_commit): + """Interpret `args` as "[commit] [--] [files...]" and return (commit, files). + It is assumed that "--" and everything that follows has been removed from + args and placed in `dash_dash`. + If "--" is present (i.e., `dash_dash` is non-empty), the argument to its + left (if present) is taken as commit. Otherwise, the first argument is + checked if it is a commit or a file. If commit is not given, + `default_commit` is used.""" + if dash_dash: + if len(args) == 0: + commit = default_commit + elif len(args) > 1: + die('at most one commit allowed; %d given' % len(args)) + else: + commit = args[0] + object_type = get_object_type(commit) + if object_type not in ('commit', 'tag'): + if object_type is None: + die("'%s' is not a commit" % commit) + else: + die("'%s' is a %s, but a commit was expected" % (commit, object_type)) + files = dash_dash[1:] + elif args: + if disambiguate_revision(args[0]): + commit = args[0] + files = args[1:] + else: + commit = default_commit + files = args + else: + commit = default_commit + files = [] + return commit, files + + +def disambiguate_revision(value): + """Returns True if `value` is a revision, False if it is a file, or dies.""" + # If `value` is ambiguous (neither a commit nor a file), the following + # command will die with an appropriate error message. + run('git', 'rev-parse', value, verbose=False) + object_type = get_object_type(value) + if object_type is None: + return False + if object_type in ('commit', 'tag'): + return True + die('`%s` is a %s, but a commit or filename was expected' % + (value, object_type)) + + +def get_object_type(value): + """Returns a string description of an object's type, or None if it is not + a valid git object.""" + cmd = ['git', 'cat-file', '-t', value] + p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE) + stdout, stderr = p.communicate() + if p.returncode != 0: + return None + return stdout.strip() + + +def compute_diff_and_extract_lines(commit, files): + """Calls compute_diff() followed by extract_lines().""" + diff_process = compute_diff(commit, files) + changed_lines = extract_lines(diff_process.stdout) + diff_process.stdout.close() + diff_process.wait() + if diff_process.returncode != 0: + # Assume error was already printed to stderr. + sys.exit(2) + return changed_lines + + +def compute_diff(commit, files): + """Return a subprocess object producing the diff from `commit`. + The return value's `stdin` file object will produce a patch with the + differences between the working directory and `commit`, filtered on `files` + (if non-empty). Zero context lines are used in the patch.""" + cmd = ['git', 'diff-index', '-p', '-U0', commit, '--'] + cmd.extend(files) + p = subprocess.Popen(cmd, stdin=subprocess.PIPE, stdout=subprocess.PIPE) + p.stdin.close() + return p + + +def extract_lines(patch_file): + """Extract the changed lines in `patch_file`. + The return value is a dictionary mapping filename to a list of (start_line, + line_count) pairs. + The input must have been produced with ``-U0``, meaning unidiff format with + zero lines of context. The return value is a dict mapping filename to a + list of line `Range`s.""" + matches = {} + for line in patch_file: + match = re.search(r'^\+\+\+\ [^/]+/(.*)', line) + if match: + filename = match.group(1).rstrip('\r\n') + match = re.search(r'^@@ -[0-9,]+ \+(\d+)(,(\d+))?', line) + if match: + start_line = int(match.group(1)) + line_count = 1 + if match.group(3): + line_count = int(match.group(3)) + if line_count > 0: + matches.setdefault(filename, []).append(Range(start_line, line_count)) + return matches + + +def filter_by_extension(dictionary, allowed_extensions): + """Delete every key in `dictionary` that doesn't have an allowed extension. + `allowed_extensions` must be a collection of lowercase file extensions, + excluding the period.""" + allowed_extensions = frozenset(allowed_extensions) + for filename in dictionary.keys(): + base_ext = filename.rsplit('.', 1) + if len(base_ext) == 1 or base_ext[1].lower() not in allowed_extensions: + del dictionary[filename] + + +def cd_to_toplevel(): + """Change to the top level of the git repository.""" + toplevel = run('git', 'rev-parse', '--show-toplevel') + os.chdir(toplevel) + + +def create_tree_from_workdir(filenames): + """Create a new git tree with the given files from the working directory. + Returns the object ID (SHA-1) of the created tree.""" + return create_tree(filenames, '--stdin') + + +def run_clang_format_and_save_to_tree(changed_lines, binary='clang-format', + style=None): + """Run clang-format on each file and save the result to a git tree. + Returns the object ID (SHA-1) of the created tree.""" + def index_info_generator(): + for filename, line_ranges in changed_lines.iteritems(): + mode = oct(os.stat(filename).st_mode) + blob_id = clang_format_to_blob(filename, line_ranges, binary=binary, + style=style) + yield '%s %s\t%s' % (mode, blob_id, filename) + return create_tree(index_info_generator(), '--index-info') + + +def create_tree(input_lines, mode): + """Create a tree object from the given input. + If mode is '--stdin', it must be a list of filenames. If mode is + '--index-info' is must be a list of values suitable for "git update-index + --index-info", such as " ". Any other mode + is invalid.""" + assert mode in ('--stdin', '--index-info') + cmd = ['git', 'update-index', '--add', '-z', mode] + with temporary_index_file(): + p = subprocess.Popen(cmd, stdin=subprocess.PIPE) + for line in input_lines: + p.stdin.write('%s\0' % line) + p.stdin.close() + if p.wait() != 0: + die('`%s` failed' % ' '.join(cmd)) + tree_id = run('git', 'write-tree') + return tree_id + + +def clang_format_to_blob(filename, line_ranges, binary='clang-format', + style=None): + """Run clang-format on the given file and save the result to a git blob. + Returns the object ID (SHA-1) of the created blob.""" + clang_format_cmd = [binary, filename] + if style: + clang_format_cmd.extend(['-style='+style]) + clang_format_cmd.extend([ + '-lines=%s:%s' % (start_line, start_line+line_count-1) + for start_line, line_count in line_ranges]) + try: + clang_format = subprocess.Popen(clang_format_cmd, stdin=subprocess.PIPE, + stdout=subprocess.PIPE) + except OSError as e: + if e.errno == errno.ENOENT: + die('cannot find executable "%s"' % binary) + else: + raise + clang_format.stdin.close() + hash_object_cmd = ['git', 'hash-object', '-w', '--path='+filename, '--stdin'] + hash_object = subprocess.Popen(hash_object_cmd, stdin=clang_format.stdout, + stdout=subprocess.PIPE) + clang_format.stdout.close() + stdout = hash_object.communicate()[0] + if hash_object.returncode != 0: + die('`%s` failed' % ' '.join(hash_object_cmd)) + if clang_format.wait() != 0: + die('`%s` failed' % ' '.join(clang_format_cmd)) + return stdout.rstrip('\r\n') + + +@contextlib.contextmanager +def temporary_index_file(tree=None): + """Context manager for setting GIT_INDEX_FILE to a temporary file and deleting + the file afterward.""" + index_path = create_temporary_index(tree) + old_index_path = os.environ.get('GIT_INDEX_FILE') + os.environ['GIT_INDEX_FILE'] = index_path + try: + yield + finally: + if old_index_path is None: + del os.environ['GIT_INDEX_FILE'] + else: + os.environ['GIT_INDEX_FILE'] = old_index_path + os.remove(index_path) + + +def create_temporary_index(tree=None): + """Create a temporary index file and return the created file's path. + If `tree` is not None, use that as the tree to read in. Otherwise, an + empty index is created.""" + gitdir = run('git', 'rev-parse', '--git-dir') + path = os.path.join(gitdir, temp_index_basename) + if tree is None: + tree = '--empty' + run('git', 'read-tree', '--index-output='+path, tree) + return path + + +def print_diff(old_tree, new_tree): + """Print the diff between the two trees to stdout.""" + # We use the porcelain 'diff' and not plumbing 'diff-tree' because the output + # is expected to be viewed by the user, and only the former does nice things + # like color and pagination. + subprocess.check_call(['git', 'diff', old_tree, new_tree, '--']) + + +def apply_changes(old_tree, new_tree, force=False, patch_mode=False): + """Apply the changes in `new_tree` to the working directory. + Bails if there are local changes in those files and not `force`. If + `patch_mode`, runs `git checkout --patch` to select hunks interactively.""" + changed_files = run('git', 'diff-tree', '-r', '-z', '--name-only', old_tree, + new_tree).rstrip('\0').split('\0') + if not force: + unstaged_files = run('git', 'diff-files', '--name-status', *changed_files) + if unstaged_files: + print >>sys.stderr, ('The following files would be modified but ' + 'have unstaged changes:') + print >>sys.stderr, unstaged_files + print >>sys.stderr, 'Please commit, stage, or stash them first.' + sys.exit(2) + if patch_mode: + # In patch mode, we could just as well create an index from the new tree + # and checkout from that, but then the user will be presented with a + # message saying "Discard ... from worktree". Instead, we use the old + # tree as the index and checkout from new_tree, which gives the slightly + # better message, "Apply ... to index and worktree". This is not quite + # right, since it won't be applied to the user's index, but oh well. + with temporary_index_file(old_tree): + subprocess.check_call(['git', 'checkout', '--patch', new_tree]) + index_tree = old_tree + else: + with temporary_index_file(new_tree): + run('git', 'checkout-index', '-a', '-f') + return changed_files + + +def run(*args, **kwargs): + stdin = kwargs.pop('stdin', '') + verbose = kwargs.pop('verbose', True) + strip = kwargs.pop('strip', True) + for name in kwargs: + raise TypeError("run() got an unexpected keyword argument '%s'" % name) + p = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE, + stdin=subprocess.PIPE) + stdout, stderr = p.communicate(input=stdin) + if p.returncode == 0: + if stderr: + if verbose: + print >>sys.stderr, '`%s` printed to stderr:' % ' '.join(args) + print >>sys.stderr, stderr.rstrip() + if strip: + stdout = stdout.rstrip('\r\n') + return stdout + if verbose: + print >>sys.stderr, '`%s` returned %s' % (' '.join(args), p.returncode) + if stderr: + print >>sys.stderr, stderr.rstrip() + sys.exit(2) + + +def die(message): + print >>sys.stderr, 'error:', message + sys.exit(2) + + +if __name__ == '__main__': + main() From 57214b3a11b43f1e62c91111fa083b7dc761440e Mon Sep 17 00:00:00 2001 From: Robert Nishihara Date: Tue, 27 Sep 2016 00:15:54 -0700 Subject: [PATCH 05/17] Fixes. --- .travis.yml | 1 - .travis/check-git-clang-format-output.sh | 0 2 files changed, 1 deletion(-) mode change 100644 => 100755 .travis/check-git-clang-format-output.sh diff --git a/.travis.yml b/.travis.yml index e77f446da..fd43573ca 100644 --- a/.travis.yml +++ b/.travis.yml @@ -31,6 +31,5 @@ matrix: - .travis/check-git-clang-format-output.sh install: - - make script: diff --git a/.travis/check-git-clang-format-output.sh b/.travis/check-git-clang-format-output.sh old mode 100644 new mode 100755 From 2776d6a37cab0be22d4d15f410fab5b03b2151c6 Mon Sep 17 00:00:00 2001 From: Robert Nishihara Date: Tue, 27 Sep 2016 00:19:21 -0700 Subject: [PATCH 06/17] Fix permissions. --- .travis/git-clang-format | 0 1 file changed, 0 insertions(+), 0 deletions(-) mode change 100644 => 100755 .travis/git-clang-format diff --git a/.travis/git-clang-format b/.travis/git-clang-format old mode 100644 new mode 100755 From ff8018db753a4a7643b9f611ad1785c68b3b2e13 Mon Sep 17 00:00:00 2001 From: Philipp Moritz Date: Tue, 27 Sep 2016 19:11:09 -0700 Subject: [PATCH 07/17] First pass at local scheduler (#2) * draft of local scheduler * API * update APIs * fix * update * Rename halo -> photon. * Add build directory. * Update common submodule. * More renaming. * Fix python ctypes. * Compile in travis. * Process generic messages and not just tasks. * Move free outside of switch. * Formatting and address comments. * Remove event loop from local scheduler state. * Use accept_client from common. * Use bind_ipc_sock from common. * Fix tests. * Update common submodule. * Fix formatting. --- .gitmodules | 3 ++ .travis.yml | 6 +++ Makefile | 21 +++++++++ build/.gitkeep | 0 common | 1 + lib/python/photon.py | 38 +++++++++++++++ photon.c | 109 +++++++++++++++++++++++++++++++++++++++++++ photon.h | 14 ++++++ photon_client.c | 27 +++++++++++ photon_client.h | 27 +++++++++++ photon_scheduler.h | 15 ++++++ setup-env.sh | 5 ++ test/test.py | 37 +++++++++++++++ 13 files changed, 303 insertions(+) create mode 100644 .gitmodules create mode 100644 Makefile create mode 100644 build/.gitkeep create mode 160000 common create mode 100644 lib/python/photon.py create mode 100644 photon.c create mode 100644 photon.h create mode 100644 photon_client.c create mode 100644 photon_client.h create mode 100644 photon_scheduler.h create mode 100755 setup-env.sh create mode 100644 test/test.py diff --git a/.gitmodules b/.gitmodules new file mode 100644 index 000000000..3f2c8add4 --- /dev/null +++ b/.gitmodules @@ -0,0 +1,3 @@ +[submodule "common"] + path = common + url = https://github.com/ray-project/common diff --git a/.travis.yml b/.travis.yml index fd43573ca..45ef9a286 100644 --- a/.travis.yml +++ b/.travis.yml @@ -31,5 +31,11 @@ matrix: - .travis/check-git-clang-format-output.sh install: + - make script: + - cd common + - make test + - cd .. + - source setup-env.sh + - python test/test.py diff --git a/Makefile b/Makefile new file mode 100644 index 000000000..7c494a74e --- /dev/null +++ b/Makefile @@ -0,0 +1,21 @@ +CC = gcc +CFLAGS = -g -Wall --std=c99 -D_XOPEN_SOURCE=500 -D_POSIX_C_SOURCE=200809L -Icommon/thirdparty +BUILD = build + +all: $(BUILD)/photon_scheduler $(BUILD)/photon_client.so + +$(BUILD)/photon_client.so: photon_client.h photon_client.c common + $(CC) $(CFLAGS) photon_client.c common/build/libcommon.a -fPIC -shared -o $(BUILD)/photon_client.so + +$(BUILD)/photon_scheduler: photon.h photon.c common + $(CC) $(CFLAGS) -o $@ photon.c common/build/libcommon.a common/thirdparty/hiredis/libhiredis.a -Icommon/thirdparty -Icommon/ + +common: FORCE + git submodule update --init --recursive + cd common; make + +clean: + cd common; make clean + rm -r $(BUILD)/* + +FORCE: diff --git a/build/.gitkeep b/build/.gitkeep new file mode 100644 index 000000000..e69de29bb diff --git a/common b/common new file mode 160000 index 000000000..084220b0e --- /dev/null +++ b/common @@ -0,0 +1 @@ +Subproject commit 084220b0e70de6bed466e97e08f4b6909133aafb diff --git a/lib/python/photon.py b/lib/python/photon.py new file mode 100644 index 000000000..b0fc263a5 --- /dev/null +++ b/lib/python/photon.py @@ -0,0 +1,38 @@ +import ctypes +import os + +photon_client_library_path = os.path.join(os.path.abspath(os.path.dirname(__file__)), "../../build/photon_client.so") +photon_client_library = ctypes.cdll.LoadLibrary(photon_client_library_path) +photon_client_library.alloc_task_spec.restype = ctypes.c_void_p +photon_client_library.photon_connect.restype = ctypes.c_void_p +photon_client_library.photon_submit.restype = None + +ID = ctypes.c_ubyte * 20 + +class UniqueID(ctypes.Structure): + _fields_ = [("unique_id", ID)] + +def make_id(string): + if len(string) != 20: + raise Exception("PlasmaIDs must be 20 characters long") + unique_id = map(ord, string) + return UniqueID(unique_id=ID(*unique_id)) + +class Task(object): + def __init__(self, function_id, args): + function_id = make_id(function_id) + self.task_spec = ctypes.c_void_p(photon_client_library.alloc_task_spec(function_id, len(args), 1, 0)) + for arg in args: + photon_client_library.task_args_add_ref(self.task_spec, arg) + + def __del__(self): + photon_client_library.free_task_spec(self.task_spec) + +class PhotonClient(object): + + def __init__(self, socket_name): + self.photon_conn = ctypes.c_void_p(photon_client_library.photon_connect(socket_name)) + + def submit(self, function_id, args): + task = Task(function_id, args) + photon_client_library.photon_submit(self.photon_conn, task.task_spec) diff --git a/photon.c b/photon.c new file mode 100644 index 000000000..8c84c2b95 --- /dev/null +++ b/photon.c @@ -0,0 +1,109 @@ +#include +#include +#include +#include +#include +#include + +#include "common.h" +#include "event_loop.h" +#include "io.h" +#include "photon.h" +#include "state/db.h" +#include "state/task_queue.h" +#include "task.h" +#include "utarray.h" + +typedef struct { + db_handle *db; + UT_array *task_queue; +} local_scheduler_state; + +event_loop *init_local_scheduler() { return event_loop_create(); }; + +void process_message(event_loop *loop, int client_sock, void *context, + int events) { + local_scheduler_state *s = context; + + uint8_t *message; + int64_t type; + int64_t length; + read_message(client_sock, &type, &length, &message); + + switch (type) { + case SUBMIT_TASK: { + task_spec *task = (task_spec *)message; + CHECK(task_size(task) == length); + unique_id id = globally_unique_id(); + task_queue_submit_task(s->db, id, task); + } break; + case TASK_DONE: { + } break; + case DISCONNECT_CLIENT: { + LOG_INFO("Disconnecting client on fd %d", client_sock); + event_loop_remove_file(loop, client_sock); + } break; + case LOG_MESSAGE: { + } break; + default: + /* This code should be unreachable. */ + CHECK(0); + } + free(message); +} + +void new_client_connection(event_loop *loop, int listener_sock, void *context, + int events) { + local_scheduler_state *s = context; + int new_socket = accept_client(listener_sock); + event_loop_add_file(loop, new_socket, EVENT_LOOP_READ, process_message, s); + LOG_INFO("new connection with fd %d", new_socket); +} + +void start_server(const char *socket_name, const char *redis_addr, + int redis_port) { + int fd = bind_ipc_sock(socket_name); + local_scheduler_state state; + event_loop *loop = init_local_scheduler(); + + state.db = db_connect(redis_addr, redis_port, "photon", "", -1); + db_attach(state.db, loop); + + /* Run event loop. */ + event_loop_add_file(loop, fd, EVENT_LOOP_READ, new_client_connection, &state); + event_loop_run(loop); +} + +int main(int argc, char *argv[]) { + /* Path of the listening socket of the local scheduler. */ + char *scheduler_socket_name = NULL; + /* IP address and port of redis. */ + char *redis_addr_port = NULL; + int c; + while ((c = getopt(argc, argv, "s:r:")) != -1) { + switch (c) { + case 's': + scheduler_socket_name = optarg; + break; + case 'r': + redis_addr_port = optarg; + break; + default: + LOG_ERR("unknown option %c", c); + exit(-1); + } + } + if (!scheduler_socket_name) { + LOG_ERR("please specify socket for incoming connections with -s switch"); + exit(-1); + } + char redis_addr[16] = {0}; + char redis_port[6] = {0}; + if (!redis_addr_port || + sscanf(redis_addr_port, "%15[0-9.]:%5[0-9]", redis_addr, redis_port) != + 2) { + LOG_ERR("need to specify redis address like 127.0.0.1:6379 with -r switch"); + exit(-1); + } + start_server(scheduler_socket_name, &redis_addr[0], atoi(redis_port)); +} diff --git a/photon.h b/photon.h new file mode 100644 index 000000000..6a213c4a5 --- /dev/null +++ b/photon.h @@ -0,0 +1,14 @@ +#ifndef PHOTON_H +#define PHOTON_H + +enum photon_message_type { + /** Notify the local scheduler that a task has finished. */ + TASK_DONE = 64, +}; + +struct photon_conn_impl { + /* File descriptor of the Unix domain socket that connects to photon. */ + int conn; +}; + +#endif diff --git a/photon_client.c b/photon_client.c new file mode 100644 index 000000000..a33b25631 --- /dev/null +++ b/photon_client.c @@ -0,0 +1,27 @@ +#include "photon_client.h" + +#include "common/io.h" +#include "common/task.h" +#include + +photon_conn *photon_connect(const char *photon_socket) { + photon_conn *result = malloc(sizeof(photon_conn)); + result->conn = connect_ipc_sock(photon_socket); + return result; +} + +void photon_submit(photon_conn *conn, task_spec *task) { + write_message(conn->conn, SUBMIT_TASK, task_size(task), (uint8_t *)task); +} + +void photon_task_done(photon_conn *conn) { + write_message(conn->conn, TASK_DONE, 0, NULL); +} + +void photon_disconnect(photon_conn *conn) { + write_message(conn->conn, DISCONNECT_CLIENT, 0, NULL); +} + +void photon_log_message(photon_conn *conn) { + write_message(conn->conn, LOG_MESSAGE, 0, NULL); +} diff --git a/photon_client.h b/photon_client.h new file mode 100644 index 000000000..3163d8b7d --- /dev/null +++ b/photon_client.h @@ -0,0 +1,27 @@ +#ifndef PHOTON_CLIENT_H +#define PHOTON_CLIENT_H + +#include "common/task.h" +#include "photon.h" + +typedef struct photon_conn_impl photon_conn; + +/* Connect to the local scheduler. */ +photon_conn *photon_connect(const char *photon_socket); + +/* Submit a task to the local scheduler. */ +void photon_submit(photon_conn *conn, task_spec *task); + +/* Get next task for this client. */ +task_spec *photon_get_task(photon_conn *conn); + +/* Tell the local scheduler that the client has finished executing a task. */ +void photon_task_done(photon_conn *conn); + +/* Disconnect from the local scheduler. */ +void photon_disconnect(photon_conn *conn); + +/* Send a log message to the local scheduler. */ +void photon_log_message(photon_conn *conn); + +#endif diff --git a/photon_scheduler.h b/photon_scheduler.h new file mode 100644 index 000000000..cce91155d --- /dev/null +++ b/photon_scheduler.h @@ -0,0 +1,15 @@ +#ifndef PHOTON_SCHEDULER +#define PHOTON_SCHEDULER + +/* Establish a connection to a new client. */ +void new_client_connection(local_scheduler_state *s, int listener_sock); + +/* schedule a task on a given worker. */ +void schedule_on_worker(local_scheduler_state *s, task_spec *task, + int client_id); + +/* Handle new incoming task that was scheduled by the globl scheduler on + * this local scheduler. */ +void schedule_task(local_scheduler_state *s, task_spec *task) + +#endif diff --git a/setup-env.sh b/setup-env.sh new file mode 100755 index 000000000..7c4350150 --- /dev/null +++ b/setup-env.sh @@ -0,0 +1,5 @@ +echo "Adding Photon to PYTHONPATH" 1>&2 + +ROOT_DIR=$(cd "$(dirname "${BASH_SOURCE:-$0}")"; pwd) + +export PYTHONPATH="$ROOT_DIR/lib/python/:$PYTHONPATH" diff --git a/test/test.py b/test/test.py new file mode 100644 index 000000000..2307f2276 --- /dev/null +++ b/test/test.py @@ -0,0 +1,37 @@ +from __future__ import print_function + +import os +import subprocess +import sys +import unittest +import random +import time + +import photon + +class TestPhotonClient(unittest.TestCase): + + def setUp(self): + # Start Redis. + redis_executable = os.path.join(os.path.abspath(os.path.dirname(__file__)), "../common/thirdparty/redis-3.2.3/src/redis-server") + self.p1 = subprocess.Popen([redis_executable, "--loglevel", "warning"]) + time.sleep(0.1) + scheduler_executable = os.path.join(os.path.abspath(os.path.dirname(__file__)), "../build/photon_scheduler") + scheduler_name = "/tmp/scheduler{}".format(random.randint(0, 10000)) + self.p2 = subprocess.Popen([scheduler_executable, "-s", scheduler_name, "-r", "127.0.0.1:6379"]) + time.sleep(0.1) + # Connect to the scheduler. + self.photon_client = photon.PhotonClient(scheduler_name) + + def tearDown(self): + # Kill the Redis server. + self.p1.kill() + # Kill the local scheduler. + self.p2.kill() + + def test_create(self): + l = [photon.make_id(20 * "a"), photon.make_id(20 * "b"), photon.make_id(20 * "c")] + self.photon_client.submit(20 * "a", l) + +if __name__ == "__main__": + unittest.main(verbosity=2) From 64c5e0880e9db4cca8f58954bc9d87c11e2f4ce2 Mon Sep 17 00:00:00 2001 From: Robert Nishihara Date: Tue, 4 Oct 2016 12:55:10 -0700 Subject: [PATCH 08/17] Let workers get tasks from local scheduler. (#5) * Restructure to have separate client and scheduler files. Shared stuff is in photon.h. * Let workers get tasks from local scheduler. --- Makefile | 4 +- lib/python/photon.py | 69 ++++++++++++++++- photon.c | 109 -------------------------- photon.h | 10 +-- photon_client.c | 14 ++++ photon_client.h | 53 +++++++++++-- photon_scheduler.c | 181 +++++++++++++++++++++++++++++++++++++++++++ test/test.py | 8 +- 8 files changed, 319 insertions(+), 129 deletions(-) delete mode 100644 photon.c create mode 100644 photon_scheduler.c diff --git a/Makefile b/Makefile index 7c494a74e..ef5de50a7 100644 --- a/Makefile +++ b/Makefile @@ -7,8 +7,8 @@ all: $(BUILD)/photon_scheduler $(BUILD)/photon_client.so $(BUILD)/photon_client.so: photon_client.h photon_client.c common $(CC) $(CFLAGS) photon_client.c common/build/libcommon.a -fPIC -shared -o $(BUILD)/photon_client.so -$(BUILD)/photon_scheduler: photon.h photon.c common - $(CC) $(CFLAGS) -o $@ photon.c common/build/libcommon.a common/thirdparty/hiredis/libhiredis.a -Icommon/thirdparty -Icommon/ +$(BUILD)/photon_scheduler: photon.h photon_scheduler.c common + $(CC) $(CFLAGS) -o $@ photon_scheduler.c common/build/libcommon.a common/thirdparty/hiredis/libhiredis.a -Icommon/thirdparty -Icommon/ common: FORCE git submodule update --init --recursive diff --git a/lib/python/photon.py b/lib/python/photon.py index b0fc263a5..36f06ff2a 100644 --- a/lib/python/photon.py +++ b/lib/python/photon.py @@ -6,9 +6,57 @@ photon_client_library = ctypes.cdll.LoadLibrary(photon_client_library_path) photon_client_library.alloc_task_spec.restype = ctypes.c_void_p photon_client_library.photon_connect.restype = ctypes.c_void_p photon_client_library.photon_submit.restype = None +photon_client_library.photon_get_task.restype = ctypes.c_void_p ID = ctypes.c_ubyte * 20 +buffer_from_read_write_memory = ctypes.pythonapi.PyBuffer_FromReadWriteMemory +buffer_from_read_write_memory.argtypes = [ctypes.c_void_p, ctypes.c_int64] +buffer_from_read_write_memory.restype = ctypes.py_object + +buffer_from_memory = ctypes.pythonapi.PyBuffer_FromMemory +buffer_from_memory.argtypes = [ctypes.c_void_p, ctypes.c_int64] +buffer_from_memory.restype = ctypes.py_object + +photon_client_library.task_function.restype = ctypes.c_void_p +photon_client_library.task_num_args.restype = ctypes.c_int64 +photon_client_library.task_num_returns.restype = ctypes.c_int64 +photon_client_library.task_arg_type.restype = ctypes.c_int8 +photon_client_library.task_arg_id.restype = ctypes.c_void_p +photon_client_library.task_arg_val.restype = ctypes.c_void_p +photon_client_library.task_arg_length.restype = ctypes.c_void_p +photon_client_library.task_return.restype = ctypes.c_void_p + + +class TaskInfo(object): + def __init__(self, function_id, args, return_ids): + self.function_id = function_id + self.args = args + self.return_ids = return_ids + +def extract_task(c_task): + function_id = buffer_from_memory(photon_client_library.task_function(c_task), 20)[:] + num_args = photon_client_library.task_num_args(c_task) + num_returns = photon_client_library.task_num_returns(c_task) + arg_vals_and_ids = [] + for i in range(num_args): + arg_type = photon_client_library.task_arg_type(c_task, i) + if arg_type == 0: + arg_id = buffer_from_memory(photon_client_library.task_arg_id(c_task, i), 20) + arg_vals_and_ids.append((arg_type, arg_id)) + elif arg_type == 1: + arg_val = photon_client_library.task_arg_val(c_task, i)[:] + arg_length = photon_client_library.task_arg_length(c_task, i) + arg_value = buffer_from_memory(arg_val, arg_length)[:] + arg_vals_and_ids.append((arg_type, arg_value)) + else: + raise Exception("arg_type must be 0 or 1") + return_ids = [] + for i in range(num_returns): + ret_id = buffer_from_memory(photon_client_library.task_return(c_task, i), 20) + return_ids.append(ret_id[:]) + return TaskInfo(function_id, arg_vals_and_ids, return_ids) + class UniqueID(ctypes.Structure): _fields_ = [("unique_id", ID)] @@ -19,11 +67,18 @@ def make_id(string): return UniqueID(unique_id=ID(*unique_id)) class Task(object): - def __init__(self, function_id, args): + def __init__(self, function_id, args, return_ids): function_id = make_id(function_id) self.task_spec = ctypes.c_void_p(photon_client_library.alloc_task_spec(function_id, len(args), 1, 0)) for arg in args: - photon_client_library.task_args_add_ref(self.task_spec, arg) + photon_client_library.task_args_add_ref(self.task_spec, make_id(arg)) + + # Add return IDs. This may not be the appropriate place for this. + num_returns = photon_client_library.task_num_returns(self.task_spec) + for i in range(num_returns): + ret_id = buffer_from_read_write_memory(photon_client_library.task_return(self.task_spec, i), 20) + for j in range(20): + ret_id[j] = return_ids[i][j] def __del__(self): photon_client_library.free_task_spec(self.task_spec) @@ -33,6 +88,12 @@ class PhotonClient(object): def __init__(self, socket_name): self.photon_conn = ctypes.c_void_p(photon_client_library.photon_connect(socket_name)) - def submit(self, function_id, args): - task = Task(function_id, args) + def submit(self, function_id, args, return_ids): + task = Task(function_id, args, return_ids) photon_client_library.photon_submit(self.photon_conn, task.task_spec) + + def get_task(self): + c_task = ctypes.c_void_p(photon_client_library.photon_get_task(self.photon_conn)) + task = c_task # TODO Extract the actual task. EXTRACT...(c_task) + # photon_client_library.free_task_spec(c_task) + return extract_task(task) diff --git a/photon.c b/photon.c deleted file mode 100644 index 8c84c2b95..000000000 --- a/photon.c +++ /dev/null @@ -1,109 +0,0 @@ -#include -#include -#include -#include -#include -#include - -#include "common.h" -#include "event_loop.h" -#include "io.h" -#include "photon.h" -#include "state/db.h" -#include "state/task_queue.h" -#include "task.h" -#include "utarray.h" - -typedef struct { - db_handle *db; - UT_array *task_queue; -} local_scheduler_state; - -event_loop *init_local_scheduler() { return event_loop_create(); }; - -void process_message(event_loop *loop, int client_sock, void *context, - int events) { - local_scheduler_state *s = context; - - uint8_t *message; - int64_t type; - int64_t length; - read_message(client_sock, &type, &length, &message); - - switch (type) { - case SUBMIT_TASK: { - task_spec *task = (task_spec *)message; - CHECK(task_size(task) == length); - unique_id id = globally_unique_id(); - task_queue_submit_task(s->db, id, task); - } break; - case TASK_DONE: { - } break; - case DISCONNECT_CLIENT: { - LOG_INFO("Disconnecting client on fd %d", client_sock); - event_loop_remove_file(loop, client_sock); - } break; - case LOG_MESSAGE: { - } break; - default: - /* This code should be unreachable. */ - CHECK(0); - } - free(message); -} - -void new_client_connection(event_loop *loop, int listener_sock, void *context, - int events) { - local_scheduler_state *s = context; - int new_socket = accept_client(listener_sock); - event_loop_add_file(loop, new_socket, EVENT_LOOP_READ, process_message, s); - LOG_INFO("new connection with fd %d", new_socket); -} - -void start_server(const char *socket_name, const char *redis_addr, - int redis_port) { - int fd = bind_ipc_sock(socket_name); - local_scheduler_state state; - event_loop *loop = init_local_scheduler(); - - state.db = db_connect(redis_addr, redis_port, "photon", "", -1); - db_attach(state.db, loop); - - /* Run event loop. */ - event_loop_add_file(loop, fd, EVENT_LOOP_READ, new_client_connection, &state); - event_loop_run(loop); -} - -int main(int argc, char *argv[]) { - /* Path of the listening socket of the local scheduler. */ - char *scheduler_socket_name = NULL; - /* IP address and port of redis. */ - char *redis_addr_port = NULL; - int c; - while ((c = getopt(argc, argv, "s:r:")) != -1) { - switch (c) { - case 's': - scheduler_socket_name = optarg; - break; - case 'r': - redis_addr_port = optarg; - break; - default: - LOG_ERR("unknown option %c", c); - exit(-1); - } - } - if (!scheduler_socket_name) { - LOG_ERR("please specify socket for incoming connections with -s switch"); - exit(-1); - } - char redis_addr[16] = {0}; - char redis_port[6] = {0}; - if (!redis_addr_port || - sscanf(redis_addr_port, "%15[0-9.]:%5[0-9]", redis_addr, redis_port) != - 2) { - LOG_ERR("need to specify redis address like 127.0.0.1:6379 with -r switch"); - exit(-1); - } - start_server(scheduler_socket_name, &redis_addr[0], atoi(redis_port)); -} diff --git a/photon.h b/photon.h index 6a213c4a5..a59e5566f 100644 --- a/photon.h +++ b/photon.h @@ -4,11 +4,11 @@ enum photon_message_type { /** Notify the local scheduler that a task has finished. */ TASK_DONE = 64, -}; - -struct photon_conn_impl { - /* File descriptor of the Unix domain socket that connects to photon. */ - int conn; + /** Get a new task from the local scheduler. */ + GET_TASK, + /** This is sent from the local scheduler to a worker to tell the worker to + * execute a task. */ + EXECUTE_TASK, }; #endif diff --git a/photon_client.c b/photon_client.c index a33b25631..1bf87f491 100644 --- a/photon_client.c +++ b/photon_client.c @@ -14,6 +14,20 @@ void photon_submit(photon_conn *conn, task_spec *task) { write_message(conn->conn, SUBMIT_TASK, task_size(task), (uint8_t *)task); } +task_spec *photon_get_task(photon_conn *conn) { + write_message(conn->conn, GET_TASK, 0, NULL); + int64_t type; + int64_t length; + uint8_t *message; + /* Receive a task from the local scheduler. This will block until the local + * scheduler gives this client a task. */ + read_message(conn->conn, &type, &length, &message); + CHECK(type == EXECUTE_TASK); + task_spec *task = (task_spec *)message; + CHECK(length == task_size(task)); + return task; +} + void photon_task_done(photon_conn *conn) { write_message(conn->conn, TASK_DONE, 0, NULL); } diff --git a/photon_client.h b/photon_client.h index 3163d8b7d..76b09455c 100644 --- a/photon_client.h +++ b/photon_client.h @@ -4,24 +4,63 @@ #include "common/task.h" #include "photon.h" -typedef struct photon_conn_impl photon_conn; +typedef struct { + /* File descriptor of the Unix domain socket that connects to photon. */ + int conn; +} photon_conn; -/* Connect to the local scheduler. */ +/** + * Connect to the local scheduler. + * + * @param photon_socket The name of the socket to use to connect to the local + scheduler. + * @return The connection information. + */ photon_conn *photon_connect(const char *photon_socket); -/* Submit a task to the local scheduler. */ +/** + * Submit a task to the local scheduler. + * + * @param conn The connection information. + * @param task The address of the task to submit. + * @return Void. + */ void photon_submit(photon_conn *conn, task_spec *task); -/* Get next task for this client. */ +/** + * Get next task for this client. This will block until the scheduler assigns + * a task to this worker. This allocates and returns a task, and so the task + * must be freed by the caller. + * + * @todo When does this actually get freed? + * + * @param conn The connection information. + * @return The address of the assigned task. + */ task_spec *photon_get_task(photon_conn *conn); -/* Tell the local scheduler that the client has finished executing a task. */ +/** + * Tell the local scheduler that the client has finished executing a task. + * + * @param conn The connection information. + * @return Void. + */ void photon_task_done(photon_conn *conn); -/* Disconnect from the local scheduler. */ +/** + * Disconnect from the local scheduler. + * + * @param conn The connection information. + * @return Void. + */ void photon_disconnect(photon_conn *conn); -/* Send a log message to the local scheduler. */ +/** + * Send a log message to the local scheduler. + * + * @param conn The connection information. + * @return Void. + */ void photon_log_message(photon_conn *conn); #endif diff --git a/photon_scheduler.c b/photon_scheduler.c new file mode 100644 index 000000000..bf5672bcd --- /dev/null +++ b/photon_scheduler.c @@ -0,0 +1,181 @@ +#include +#include +#include +#include +#include +#include + +#include "common.h" +#include "event_loop.h" +#include "io.h" +#include "photon.h" +#include "state/db.h" +#include "state/task_queue.h" +#include "task.h" +#include "utarray.h" + +typedef struct { + /** The file descriptor used to communicate with the worker. */ + int client_sock; +} available_worker; + +/* These are needed to define the UT_arrays. */ +UT_icd task_ptr_icd = {sizeof(task_spec *), NULL, NULL, NULL}; +UT_icd worker_icd = {sizeof(available_worker), NULL, NULL, NULL}; + +typedef struct { + db_handle *db; + /** This is an array of pointers to tasks that are waiting to be scheduled. */ + UT_array *task_queue; + /** This is an array of file descriptors corresponding to clients that are + * waiting for tasks. */ + UT_array *available_worker_queue; +} local_scheduler_state; + +void try_to_assign_task(task_spec *task, local_scheduler_state *s); +void try_to_assign_task_to_worker(int client_sock, local_scheduler_state *s); + +event_loop *init_local_scheduler() { return event_loop_create(); }; + +void process_message(event_loop *loop, int client_sock, void *context, + int events) { + local_scheduler_state *s = context; + + uint8_t *message; + int64_t type; + int64_t length; + read_message(client_sock, &type, &length, &message); + + switch (type) { + case SUBMIT_TASK: { + task_spec *task = (task_spec *)message; + CHECK(task_size(task) == length); + /* Create a unique task instance ID. This is different from the task ID and + * is used to distinguish between potentially multiple executions of the + * task. */ + unique_id id = globally_unique_id(); + // task_queue_submit_task(s->db, id, task); + /* Try to assign the task to a worker locally. TODO(rkn): This should + * probably go somewhere else. */ + try_to_assign_task(task, s); + } break; + case TASK_DONE: { + } break; + case GET_TASK: { + try_to_assign_task_to_worker(client_sock, s); + } break; + case DISCONNECT_CLIENT: { + LOG_INFO("Disconnecting client on fd %d", client_sock); + event_loop_remove_file(loop, client_sock); + } break; + case LOG_MESSAGE: { + } break; + default: + /* This code should be unreachable. */ + CHECK(0); + } + free(message); +} + +void try_to_assign_task(task_spec *task, local_scheduler_state *s) { + /* Assign this task to an available worker. If there are no available workers, + * then add this task to the local task queue. */ + if (utarray_len(s->available_worker_queue) > 0) { + /* Get the last available worker in the available worker queue. */ + available_worker *worker = + (available_worker *)utarray_back(s->available_worker_queue); + /* Tell the available worker to execute the task. */ + write_message(worker->client_sock, EXECUTE_TASK, task_size(task), + (uint8_t *)task); + utarray_pop_back(s->available_worker_queue); + /* TODO: Do we need to free the available_worker struct? */ + } else { + /* Add the task to the task queue. */ + task_spec *task_copy = malloc(task_size(task)); + memcpy(task_copy, task, task_size(task)); + utarray_push_back(s->task_queue, &task_copy); + } +} + +void try_to_assign_task_to_worker(int client_sock, local_scheduler_state *s) { + if (utarray_len(s->task_queue) > 0) { + /* Get the last task in the task queue. */ + task_spec **task_ptr = (task_spec **)utarray_back(s->task_queue); + task_spec *task = *task_ptr; + /* Send a task to the worker. */ + write_message(client_sock, EXECUTE_TASK, task_size(task), (uint8_t *)task); + /* Update the task queue data structure and free the task. */ + utarray_pop_back(s->task_queue); + free(task); + } else { + /* Check that client_sock is not already in the available workers. */ + for (available_worker *p = + (available_worker *)utarray_front(s->available_worker_queue); + p != NULL; + p = (available_worker *)utarray_next(s->available_worker_queue, p)) { + CHECK(p->client_sock != client_sock); + } + /* Add client_sock to a list of available workers. */ + available_worker worker_info = {.client_sock = client_sock}; + utarray_push_back(s->available_worker_queue, &worker_info); + LOG_INFO("Adding client_sock %d to available workers.\n", client_sock); + } +} + +void new_client_connection(event_loop *loop, int listener_sock, void *context, + int events) { + local_scheduler_state *s = context; + int new_socket = accept_client(listener_sock); + event_loop_add_file(loop, new_socket, EVENT_LOOP_READ, process_message, s); + LOG_INFO("new connection with fd %d", new_socket); +} + +void start_server(const char *socket_name, const char *redis_addr, + int redis_port) { + int fd = bind_ipc_sock(socket_name); + local_scheduler_state state; + event_loop *loop = init_local_scheduler(); + + state.db = db_connect(redis_addr, redis_port, "photon", "", -1); + db_attach(state.db, loop); + utarray_new(state.task_queue, &task_ptr_icd); + utarray_new(state.available_worker_queue, &worker_icd); + + /* Run event loop. */ + event_loop_add_file(loop, fd, EVENT_LOOP_READ, new_client_connection, &state); + event_loop_run(loop); +} + +int main(int argc, char *argv[]) { + /* Path of the listening socket of the local scheduler. */ + char *scheduler_socket_name = NULL; + /* IP address and port of redis. */ + char *redis_addr_port = NULL; + int c; + while ((c = getopt(argc, argv, "s:r:")) != -1) { + switch (c) { + case 's': + scheduler_socket_name = optarg; + break; + case 'r': + redis_addr_port = optarg; + break; + default: + LOG_ERR("unknown option %c", c); + exit(-1); + } + } + if (!scheduler_socket_name) { + LOG_ERR("please specify socket for incoming connections with -s switch"); + exit(-1); + } + char redis_addr[16] = {0}; + char redis_port[6] = {0}; + if (!redis_addr_port || + sscanf(redis_addr_port, "%15[0-9.]:%5[0-9]", redis_addr, redis_port) != + 2) { + LOG_ERR("need to specify redis address like 127.0.0.1:6379 with -r switch"); + exit(-1); + } + start_server(scheduler_socket_name, &redis_addr[0], atoi(redis_port)); +} diff --git a/test/test.py b/test/test.py index 2307f2276..0517c40a5 100644 --- a/test/test.py +++ b/test/test.py @@ -30,8 +30,12 @@ class TestPhotonClient(unittest.TestCase): self.p2.kill() def test_create(self): - l = [photon.make_id(20 * "a"), photon.make_id(20 * "b"), photon.make_id(20 * "c")] - self.photon_client.submit(20 * "a", l) + l = [20 * "a", 20 * "b", 20 * "c"] + r = [20 * "e", 20 * "f"] + # Submit a task. + self.photon_client.submit(20 * "d", l, r) + # Get the task. + task = self.photon_client.get_task() if __name__ == "__main__": unittest.main(verbosity=2) From 872e68b5b004cbde78aee18325cee187387fac9c Mon Sep 17 00:00:00 2001 From: Philipp Moritz Date: Tue, 4 Oct 2016 16:25:11 -0700 Subject: [PATCH 09/17] submit task to redis --- common | 2 +- photon_scheduler.c | 130 +++++++++++++++++++++++---------------------- photon_scheduler.h | 23 ++++---- 3 files changed, 81 insertions(+), 74 deletions(-) diff --git a/common b/common index 084220b0e..49ac871ef 160000 --- a/common +++ b/common @@ -1 +1 @@ -Subproject commit 084220b0e70de6bed466e97e08f4b6909133aafb +Subproject commit 49ac871ef6a6751835dcd8448f1d67ef4d6c82ad diff --git a/photon_scheduler.c b/photon_scheduler.c index bf5672bcd..e6b6cdad2 100644 --- a/photon_scheduler.c +++ b/photon_scheduler.c @@ -9,9 +9,9 @@ #include "event_loop.h" #include "io.h" #include "photon.h" +#include "photon_scheduler.h" #include "state/db.h" -#include "state/task_queue.h" -#include "task.h" +#include "state/task_log.h" #include "utarray.h" typedef struct { @@ -20,22 +20,76 @@ typedef struct { } available_worker; /* These are needed to define the UT_arrays. */ -UT_icd task_ptr_icd = {sizeof(task_spec *), NULL, NULL, NULL}; +UT_icd task_ptr_icd = {sizeof(task_instance *), NULL, NULL, NULL}; UT_icd worker_icd = {sizeof(available_worker), NULL, NULL, NULL}; -typedef struct { +struct local_scheduler_state { db_handle *db; /** This is an array of pointers to tasks that are waiting to be scheduled. */ UT_array *task_queue; /** This is an array of file descriptors corresponding to clients that are * waiting for tasks. */ UT_array *available_worker_queue; -} local_scheduler_state; +}; -void try_to_assign_task(task_spec *task, local_scheduler_state *s); -void try_to_assign_task_to_worker(int client_sock, local_scheduler_state *s); +local_scheduler_state *init_local_scheduler(event_loop *loop, + const char *redis_addr, + int redis_port) { + local_scheduler_state *state = malloc(sizeof(local_scheduler_state)); + state->db = db_connect(redis_addr, redis_port, "photon", "", -1); + db_attach(state->db, loop); + utarray_new(state->task_queue, &task_ptr_icd); + utarray_new(state->available_worker_queue, &worker_icd); + return state; +}; -event_loop *init_local_scheduler() { return event_loop_create(); }; +void handle_submit_task(local_scheduler_state *s, task_spec *task) { + /* Assign this task to an available worker. If there are no available workers, + * then add this task to the local task queue. */ + task_iid task_iid = globally_unique_id(); + task_instance *instance = make_task_instance(task_iid, task, TASK_WAITING, NIL_ID); + if (utarray_len(s->available_worker_queue) > 0) { + /* Get the last available worker in the available worker queue. */ + available_worker *worker = + (available_worker *)utarray_back(s->available_worker_queue); + /* Tell the available worker to execute the task. */ + write_message(worker->client_sock, EXECUTE_TASK, task_size(task), + (uint8_t *)task); + utarray_pop_back(s->available_worker_queue); + /* TODO: Do we need to free the available_worker struct? */ + } else { + /* Add the task to the task queue. */ + utarray_push_back(s->task_queue, &instance); + } + /* Submit task to redis. */ + task_log_add_task(s->db, instance); + // free(instance); +} + +void handle_get_task(local_scheduler_state *s, int client_sock) { + if (utarray_len(s->task_queue) > 0) { + /* Get the last task in the task queue. */ + task_instance **back = (task_instance **)utarray_back(s->task_queue); + task_spec *task = task_instance_task_spec(*back); + /* Send a task to the worker. */ + write_message(client_sock, EXECUTE_TASK, task_size(task), (uint8_t *)task); + /* Update the task queue data structure and free the task. */ + utarray_pop_back(s->task_queue); + free(*back); + } else { + /* Check that client_sock is not already in the available workers. */ + for (available_worker *p = + (available_worker *)utarray_front(s->available_worker_queue); + p != NULL; + p = (available_worker *)utarray_next(s->available_worker_queue, p)) { + CHECK(p->client_sock != client_sock); + } + /* Add client_sock to a list of available workers. */ + available_worker worker_info = {.client_sock = client_sock}; + utarray_push_back(s->available_worker_queue, &worker_info); + LOG_INFO("Adding client_sock %d to available workers.\n", client_sock); + } +} void process_message(event_loop *loop, int client_sock, void *context, int events) { @@ -57,12 +111,12 @@ void process_message(event_loop *loop, int client_sock, void *context, // task_queue_submit_task(s->db, id, task); /* Try to assign the task to a worker locally. TODO(rkn): This should * probably go somewhere else. */ - try_to_assign_task(task, s); + handle_submit_task(s, task); } break; case TASK_DONE: { } break; case GET_TASK: { - try_to_assign_task_to_worker(client_sock, s); + handle_get_task(s, client_sock); } break; case DISCONNECT_CLIENT: { LOG_INFO("Disconnecting client on fd %d", client_sock); @@ -77,51 +131,6 @@ void process_message(event_loop *loop, int client_sock, void *context, free(message); } -void try_to_assign_task(task_spec *task, local_scheduler_state *s) { - /* Assign this task to an available worker. If there are no available workers, - * then add this task to the local task queue. */ - if (utarray_len(s->available_worker_queue) > 0) { - /* Get the last available worker in the available worker queue. */ - available_worker *worker = - (available_worker *)utarray_back(s->available_worker_queue); - /* Tell the available worker to execute the task. */ - write_message(worker->client_sock, EXECUTE_TASK, task_size(task), - (uint8_t *)task); - utarray_pop_back(s->available_worker_queue); - /* TODO: Do we need to free the available_worker struct? */ - } else { - /* Add the task to the task queue. */ - task_spec *task_copy = malloc(task_size(task)); - memcpy(task_copy, task, task_size(task)); - utarray_push_back(s->task_queue, &task_copy); - } -} - -void try_to_assign_task_to_worker(int client_sock, local_scheduler_state *s) { - if (utarray_len(s->task_queue) > 0) { - /* Get the last task in the task queue. */ - task_spec **task_ptr = (task_spec **)utarray_back(s->task_queue); - task_spec *task = *task_ptr; - /* Send a task to the worker. */ - write_message(client_sock, EXECUTE_TASK, task_size(task), (uint8_t *)task); - /* Update the task queue data structure and free the task. */ - utarray_pop_back(s->task_queue); - free(task); - } else { - /* Check that client_sock is not already in the available workers. */ - for (available_worker *p = - (available_worker *)utarray_front(s->available_worker_queue); - p != NULL; - p = (available_worker *)utarray_next(s->available_worker_queue, p)) { - CHECK(p->client_sock != client_sock); - } - /* Add client_sock to a list of available workers. */ - available_worker worker_info = {.client_sock = client_sock}; - utarray_push_back(s->available_worker_queue, &worker_info); - LOG_INFO("Adding client_sock %d to available workers.\n", client_sock); - } -} - void new_client_connection(event_loop *loop, int listener_sock, void *context, int events) { local_scheduler_state *s = context; @@ -133,16 +142,11 @@ void new_client_connection(event_loop *loop, int listener_sock, void *context, void start_server(const char *socket_name, const char *redis_addr, int redis_port) { int fd = bind_ipc_sock(socket_name); - local_scheduler_state state; - event_loop *loop = init_local_scheduler(); - - state.db = db_connect(redis_addr, redis_port, "photon", "", -1); - db_attach(state.db, loop); - utarray_new(state.task_queue, &task_ptr_icd); - utarray_new(state.available_worker_queue, &worker_icd); + event_loop *loop = event_loop_create(); + local_scheduler_state *state = init_local_scheduler(loop, redis_addr, redis_port); /* Run event loop. */ - event_loop_add_file(loop, fd, EVENT_LOOP_READ, new_client_connection, &state); + event_loop_add_file(loop, fd, EVENT_LOOP_READ, new_client_connection, state); event_loop_run(loop); } diff --git a/photon_scheduler.h b/photon_scheduler.h index cce91155d..bef1f42d9 100644 --- a/photon_scheduler.h +++ b/photon_scheduler.h @@ -1,15 +1,18 @@ -#ifndef PHOTON_SCHEDULER -#define PHOTON_SCHEDULER +#ifndef PHOTON_SCHEDULER_H +#define PHOTON_SCHEDULER_H + +#include "task.h" + +typedef struct local_scheduler_state local_scheduler_state; /* Establish a connection to a new client. */ -void new_client_connection(local_scheduler_state *s, int listener_sock); +void new_client_connection(event_loop *loop, int listener_sock, void *context, + int events); -/* schedule a task on a given worker. */ -void schedule_on_worker(local_scheduler_state *s, task_spec *task, - int client_id); +/* Assign a task to a worker. */ +void handle_get_task(local_scheduler_state *s, int client_sock); -/* Handle new incoming task that was scheduled by the globl scheduler on - * this local scheduler. */ -void schedule_task(local_scheduler_state *s, task_spec *task) +/* Handle incoming submit request by a worker. */ +void handle_submit_task(local_scheduler_state *s, task_spec *task); -#endif +#endif /* PHOTON_SCHEDULER_H */ From 67677c3c923f6e3e949532b642952997a951cc4d Mon Sep 17 00:00:00 2001 From: Philipp Moritz Date: Tue, 4 Oct 2016 17:06:52 -0700 Subject: [PATCH 10/17] update documentation and common --- common | 2 +- photon_scheduler.c | 11 ++++++----- photon_scheduler.h | 24 +++++++++++++++++++++--- 3 files changed, 28 insertions(+), 9 deletions(-) diff --git a/common b/common index 49ac871ef..4329afbd5 160000 --- a/common +++ b/common @@ -1 +1 @@ -Subproject commit 49ac871ef6a6751835dcd8448f1d67ef4d6c82ad +Subproject commit 4329afbd53449412b08e64d4057a2857393c2212 diff --git a/photon_scheduler.c b/photon_scheduler.c index e6b6cdad2..7d9b5f7ef 100644 --- a/photon_scheduler.c +++ b/photon_scheduler.c @@ -32,9 +32,8 @@ struct local_scheduler_state { UT_array *available_worker_queue; }; -local_scheduler_state *init_local_scheduler(event_loop *loop, - const char *redis_addr, - int redis_port) { +local_scheduler_state * +init_local_scheduler(event_loop *loop, const char *redis_addr, int redis_port) { local_scheduler_state *state = malloc(sizeof(local_scheduler_state)); state->db = db_connect(redis_addr, redis_port, "photon", "", -1); db_attach(state->db, loop); @@ -47,7 +46,8 @@ void handle_submit_task(local_scheduler_state *s, task_spec *task) { /* Assign this task to an available worker. If there are no available workers, * then add this task to the local task queue. */ task_iid task_iid = globally_unique_id(); - task_instance *instance = make_task_instance(task_iid, task, TASK_WAITING, NIL_ID); + task_instance *instance = + make_task_instance(task_iid, task, TASK_STATUS_WAITING, NIL_ID); if (utarray_len(s->available_worker_queue) > 0) { /* Get the last available worker in the available worker queue. */ available_worker *worker = @@ -143,7 +143,8 @@ void start_server(const char *socket_name, const char *redis_addr, int redis_port) { int fd = bind_ipc_sock(socket_name); event_loop *loop = event_loop_create(); - local_scheduler_state *state = init_local_scheduler(loop, redis_addr, redis_port); + local_scheduler_state *state = + init_local_scheduler(loop, redis_addr, redis_port); /* Run event loop. */ event_loop_add_file(loop, fd, EVENT_LOOP_READ, new_client_connection, state); diff --git a/photon_scheduler.h b/photon_scheduler.h index bef1f42d9..54905bb88 100644 --- a/photon_scheduler.h +++ b/photon_scheduler.h @@ -5,14 +5,32 @@ typedef struct local_scheduler_state local_scheduler_state; -/* Establish a connection to a new client. */ +/** + * Establish a connection to a new client. + * + * @param loop Event loop of the local scheduler. + * @param listener_socket Socket the local scheduler is listening on for new + * client requests. + * @param context State of the local scheduler. + * @param events Flag for events that are available on the listener socket. + */ void new_client_connection(event_loop *loop, int listener_sock, void *context, int events); -/* Assign a task to a worker. */ +/** + * Assign a task to a worker. + * + * @param s State of the local scheduler. + * @param client_sock Socket by which the worker is connected. + */ void handle_get_task(local_scheduler_state *s, int client_sock); -/* Handle incoming submit request by a worker. */ +/** + * Handle incoming submit request by a worker. + * + * @param s State of the local scheduler. + * @param task Task specification of the task to be submitted. + */ void handle_submit_task(local_scheduler_state *s, task_spec *task); #endif /* PHOTON_SCHEDULER_H */ From a7a963445d50fa0c53ef771404a40da06abc3cc7 Mon Sep 17 00:00:00 2001 From: Philipp Moritz Date: Wed, 5 Oct 2016 13:30:10 -0700 Subject: [PATCH 11/17] fixes --- common | 2 +- photon_scheduler.h | 5 ++++- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/common b/common index 4329afbd5..4204500d2 160000 --- a/common +++ b/common @@ -1 +1 @@ -Subproject commit 4329afbd53449412b08e64d4057a2857393c2212 +Subproject commit 4204500d23be7726e27598badb691d29d08a0ad7 diff --git a/photon_scheduler.h b/photon_scheduler.h index 54905bb88..591ffe0f5 100644 --- a/photon_scheduler.h +++ b/photon_scheduler.h @@ -10,9 +10,10 @@ typedef struct local_scheduler_state local_scheduler_state; * * @param loop Event loop of the local scheduler. * @param listener_socket Socket the local scheduler is listening on for new - * client requests. + * client requests. * @param context State of the local scheduler. * @param events Flag for events that are available on the listener socket. + * @return Void. */ void new_client_connection(event_loop *loop, int listener_sock, void *context, int events); @@ -22,6 +23,7 @@ void new_client_connection(event_loop *loop, int listener_sock, void *context, * * @param s State of the local scheduler. * @param client_sock Socket by which the worker is connected. + * @return Void. */ void handle_get_task(local_scheduler_state *s, int client_sock); @@ -30,6 +32,7 @@ void handle_get_task(local_scheduler_state *s, int client_sock); * * @param s State of the local scheduler. * @param task Task specification of the task to be submitted. + * @return Void. */ void handle_submit_task(local_scheduler_state *s, task_spec *task); From 0f97855333370f99e31391442a4ef9b97969032b Mon Sep 17 00:00:00 2001 From: Robert Nishihara Date: Wed, 5 Oct 2016 14:11:02 -0700 Subject: [PATCH 12/17] More fixes. --- photon_scheduler.c | 37 ++++++++++++++++++++++--------------- 1 file changed, 22 insertions(+), 15 deletions(-) diff --git a/photon_scheduler.c b/photon_scheduler.c index 7d9b5f7ef..44022451c 100644 --- a/photon_scheduler.c +++ b/photon_scheduler.c @@ -43,30 +43,43 @@ init_local_scheduler(event_loop *loop, const char *redis_addr, int redis_port) { }; void handle_submit_task(local_scheduler_state *s, task_spec *task) { - /* Assign this task to an available worker. If there are no available workers, - * then add this task to the local task queue. */ + /* Create a unique task instance ID. This is different from the task ID and + * is used to distinguish between potentially multiple executions of the + * task. */ task_iid task_iid = globally_unique_id(); task_instance *instance = make_task_instance(task_iid, task, TASK_STATUS_WAITING, NIL_ID); - if (utarray_len(s->available_worker_queue) > 0) { + /* Assign this task to an available worker. If there are no available workers, + * then add this task to the local task queue. */ + int schedule_locally = utarray_len(s->available_worker_queue) > 0; + if (schedule_locally) { /* Get the last available worker in the available worker queue. */ available_worker *worker = (available_worker *)utarray_back(s->available_worker_queue); /* Tell the available worker to execute the task. */ write_message(worker->client_sock, EXECUTE_TASK, task_size(task), (uint8_t *)task); + /* Remove the available worker from the queue and free the struct. */ utarray_pop_back(s->available_worker_queue); - /* TODO: Do we need to free the available_worker struct? */ + free(worker); } else { - /* Add the task to the task queue. */ + /* Add the task to the task queue. This passes ownership of the task queue. + * And the task will be freed when it is assigned to a worker. */ utarray_push_back(s->task_queue, &instance); } - /* Submit task to redis. */ + /* Submit the task to redis. */ task_log_add_task(s->db, instance); - // free(instance); + if (schedule_locally) { + /* If the task was scheduled locally, we need to free it. Otherwise, + * ownership of the task is passed to the task_queue, and it will be freed + * when it is assigned to a worker. */ + free(instance); + } } void handle_get_task(local_scheduler_state *s, int client_sock) { + /* If there is an available task, assign that task to this worker. Otherwise + * add the worker to the queue of available workers. */ if (utarray_len(s->task_queue) > 0) { /* Get the last task in the task queue. */ task_instance **back = (task_instance **)utarray_back(s->task_queue); @@ -84,7 +97,8 @@ void handle_get_task(local_scheduler_state *s, int client_sock) { p = (available_worker *)utarray_next(s->available_worker_queue, p)) { CHECK(p->client_sock != client_sock); } - /* Add client_sock to a list of available workers. */ + /* Add client_sock to a list of available workers. This struct will be freed + * when a task is assigned to this worker. */ available_worker worker_info = {.client_sock = client_sock}; utarray_push_back(s->available_worker_queue, &worker_info); LOG_INFO("Adding client_sock %d to available workers.\n", client_sock); @@ -104,13 +118,6 @@ void process_message(event_loop *loop, int client_sock, void *context, case SUBMIT_TASK: { task_spec *task = (task_spec *)message; CHECK(task_size(task) == length); - /* Create a unique task instance ID. This is different from the task ID and - * is used to distinguish between potentially multiple executions of the - * task. */ - unique_id id = globally_unique_id(); - // task_queue_submit_task(s->db, id, task); - /* Try to assign the task to a worker locally. TODO(rkn): This should - * probably go somewhere else. */ handle_submit_task(s, task); } break; case TASK_DONE: { From e8e4aa6d8e79cd85cde002dc2a8fc850c3ed89fb Mon Sep 17 00:00:00 2001 From: Philipp Moritz Date: Wed, 5 Oct 2016 18:07:08 -0700 Subject: [PATCH 13/17] add valgrind check --- .travis.yml | 13 +++++++++++++ photon_scheduler.c | 33 ++++++++++++++++++++++++++++++--- test/test.py | 28 +++++++++++++++++++++++++--- 3 files changed, 68 insertions(+), 6 deletions(-) diff --git a/.travis.yml b/.travis.yml index 45ef9a286..caab2e356 100644 --- a/.travis.yml +++ b/.travis.yml @@ -29,6 +29,19 @@ matrix: install: [] script: - .travis/check-git-clang-format-output.sh + - os: linux + dist: trusty + python: "2.7" + env: VALGRIND=1 + before_install: + - sudo apt-get update -qq + - sudo apt-get install -qq valgrind + script: + - cd common + - make test + - cd .. + - source setup-env.sh + - python test/test.py valgrind install: - make diff --git a/photon_scheduler.c b/photon_scheduler.c index 44022451c..b876bc9f2 100644 --- a/photon_scheduler.c +++ b/photon_scheduler.c @@ -1,4 +1,5 @@ #include +#include #include #include #include @@ -24,6 +25,9 @@ UT_icd task_ptr_icd = {sizeof(task_instance *), NULL, NULL, NULL}; UT_icd worker_icd = {sizeof(available_worker), NULL, NULL, NULL}; struct local_scheduler_state { + /* The local scheduler event loop. */ + event_loop *loop; + /* The handle to the database. */ db_handle *db; /** This is an array of pointers to tasks that are waiting to be scheduled. */ UT_array *task_queue; @@ -35,6 +39,7 @@ struct local_scheduler_state { local_scheduler_state * init_local_scheduler(event_loop *loop, const char *redis_addr, int redis_port) { local_scheduler_state *state = malloc(sizeof(local_scheduler_state)); + state->loop = loop; state->db = db_connect(redis_addr, redis_port, "photon", "", -1); db_attach(state->db, loop); utarray_new(state->task_queue, &task_ptr_icd); @@ -42,6 +47,14 @@ init_local_scheduler(event_loop *loop, const char *redis_addr, int redis_port) { return state; }; +void free_local_scheduler(local_scheduler_state *s) { + db_disconnect(s->db); + utarray_free(s->task_queue); + utarray_free(s->available_worker_queue); + event_loop_destroy(s->loop); + free(s); +} + void handle_submit_task(local_scheduler_state *s, task_spec *task) { /* Create a unique task instance ID. This is different from the task ID and * is used to distinguish between potentially multiple executions of the @@ -146,19 +159,33 @@ void new_client_connection(event_loop *loop, int listener_sock, void *context, LOG_INFO("new connection with fd %d", new_socket); } +/* We need this code so we can clean up when we get a SIGTERM signal. */ + +local_scheduler_state *g_state; + +void signal_handler(int signal) { + if (signal == SIGTERM) { + free_local_scheduler(g_state); + exit(0); + } +} + +/* End of the cleanup code. */ + void start_server(const char *socket_name, const char *redis_addr, int redis_port) { int fd = bind_ipc_sock(socket_name); event_loop *loop = event_loop_create(); - local_scheduler_state *state = - init_local_scheduler(loop, redis_addr, redis_port); + g_state = init_local_scheduler(loop, redis_addr, redis_port); /* Run event loop. */ - event_loop_add_file(loop, fd, EVENT_LOOP_READ, new_client_connection, state); + event_loop_add_file(loop, fd, EVENT_LOOP_READ, new_client_connection, + g_state); event_loop_run(loop); } int main(int argc, char *argv[]) { + signal(SIGTERM, signal_handler); /* Path of the listening socket of the local scheduler. */ char *scheduler_socket_name = NULL; /* IP address and port of redis. */ diff --git a/test/test.py b/test/test.py index 0517c40a5..bbd164c37 100644 --- a/test/test.py +++ b/test/test.py @@ -1,6 +1,7 @@ from __future__ import print_function import os +import signal import subprocess import sys import unittest @@ -9,6 +10,8 @@ import time import photon +USE_VALGRIND = False + class TestPhotonClient(unittest.TestCase): def setUp(self): @@ -18,8 +21,15 @@ class TestPhotonClient(unittest.TestCase): time.sleep(0.1) scheduler_executable = os.path.join(os.path.abspath(os.path.dirname(__file__)), "../build/photon_scheduler") scheduler_name = "/tmp/scheduler{}".format(random.randint(0, 10000)) - self.p2 = subprocess.Popen([scheduler_executable, "-s", scheduler_name, "-r", "127.0.0.1:6379"]) - time.sleep(0.1) + command = [scheduler_executable, "-s", scheduler_name, "-r", "127.0.0.1:6379"] + if USE_VALGRIND: + self.p2 = subprocess.Popen(["valgrind", "--track-origins=yes", "--leak-check=full", "--show-leak-kinds=all"] + command) + else: + self.p2 = subprocess.Popen(command) + if USE_VALGRIND: + time.sleep(1.0) + else: + time.sleep(0.1) # Connect to the scheduler. self.photon_client = photon.PhotonClient(scheduler_name) @@ -27,7 +37,13 @@ class TestPhotonClient(unittest.TestCase): # Kill the Redis server. self.p1.kill() # Kill the local scheduler. - self.p2.kill() + if USE_VALGRIND: + self.p2.send_signal(signal.SIGTERM) + self.p2.wait() + os._exit(self.p2.returncode) + else: + self.p2.kill() + def test_create(self): l = [20 * "a", 20 * "b", 20 * "c"] @@ -38,4 +54,10 @@ class TestPhotonClient(unittest.TestCase): task = self.photon_client.get_task() if __name__ == "__main__": + if len(sys.argv) > 1: + # pop the argument so we don't mess with unittest's own argument parser + arg = sys.argv.pop() + if arg == "valgrind": + USE_VALGRIND = True + print("Using valgrind for tests") unittest.main(verbosity=2) From 18934c3a4566a201f1f5dcf7425420d699fb6ab1 Mon Sep 17 00:00:00 2001 From: Robert Nishihara Date: Fri, 7 Oct 2016 11:00:46 -0700 Subject: [PATCH 14/17] Make photon client into a C extension. (#8) * Make photon client into a C extension. * Fix formatting. * Rename extension from PhotonClient to Photon. * Update common submodule. * Fix Makefile to compile with fPIC. * Update common submodule. * Compile C extension against common. * Fix formatting. * Remove unnecessary include. * Update common submodule and rename Photon -> PhotonClient. * Drop global interpretor lock during get_task. --- .travis.yml | 12 ++- Makefile | 9 ++- common | 2 +- install-dependencies.sh | 21 +++++ lib/python/photon.py | 99 ------------------------ lib/python/photon_extension.c | 140 ++++++++++++++++++++++++++++++++++ lib/python/setup.py | 14 ++++ test/test.py | 55 +++++++++++-- 8 files changed, 238 insertions(+), 114 deletions(-) create mode 100755 install-dependencies.sh delete mode 100644 lib/python/photon.py create mode 100644 lib/python/photon_extension.c create mode 100644 lib/python/setup.py diff --git a/.travis.yml b/.travis.yml index caab2e356..360027785 100644 --- a/.travis.yml +++ b/.travis.yml @@ -44,11 +44,17 @@ matrix: - python test/test.py valgrind install: + - ./install-dependencies.sh - make - -script: + - cd common/lib/python + - python setup.py install --user + - cd ../../.. + - cd lib/python + - python setup.py install --user + - cd ../.. - cd common - make test - cd .. - - source setup-env.sh + +script: - python test/test.py diff --git a/Makefile b/Makefile index ef5de50a7..436502d6a 100644 --- a/Makefile +++ b/Makefile @@ -1,11 +1,11 @@ CC = gcc -CFLAGS = -g -Wall --std=c99 -D_XOPEN_SOURCE=500 -D_POSIX_C_SOURCE=200809L -Icommon/thirdparty +CFLAGS = -g -Wall --std=c99 -D_XOPEN_SOURCE=500 -D_POSIX_C_SOURCE=200809L -Icommon/thirdparty -fPIC BUILD = build -all: $(BUILD)/photon_scheduler $(BUILD)/photon_client.so +all: $(BUILD)/photon_scheduler $(BUILD)/photon_client.a -$(BUILD)/photon_client.so: photon_client.h photon_client.c common - $(CC) $(CFLAGS) photon_client.c common/build/libcommon.a -fPIC -shared -o $(BUILD)/photon_client.so +$(BUILD)/photon_client.a: photon_client.o + ar rcs $(BUILD)/photon_client.a photon_client.o $(BUILD)/photon_scheduler: photon.h photon_scheduler.c common $(CC) $(CFLAGS) -o $@ photon_scheduler.c common/build/libcommon.a common/thirdparty/hiredis/libhiredis.a -Icommon/thirdparty -Icommon/ @@ -17,5 +17,6 @@ common: FORCE clean: cd common; make clean rm -r $(BUILD)/* + rm *.o FORCE: diff --git a/common b/common index 4204500d2..7be1a93d6 160000 --- a/common +++ b/common @@ -1 +1 @@ -Subproject commit 4204500d23be7726e27598badb691d29d08a0ad7 +Subproject commit 7be1a93d64ca36fc639e11f81de1483e0bd17b8c diff --git a/install-dependencies.sh b/install-dependencies.sh new file mode 100755 index 000000000..f84da1684 --- /dev/null +++ b/install-dependencies.sh @@ -0,0 +1,21 @@ +#!/usr/bin/env bash + +ROOT_DIR=$(cd "$(dirname "${BASH_SOURCE:-$0}")"; pwd) + +platform="unknown" +unamestr="$(uname)" +if [[ "$unamestr" == "Linux" ]]; then + echo "Platform is linux." + platform="linux" +elif [[ "$unamestr" == "Darwin" ]]; then + echo "Platform is macosx." + platform="macosx" +else + echo "Unrecognized platform." + exit 1 +fi + +if [[ $platform == "linux" ]]; then + sudo apt-get update + sudo apt-get install -y git python-dev +fi diff --git a/lib/python/photon.py b/lib/python/photon.py deleted file mode 100644 index 36f06ff2a..000000000 --- a/lib/python/photon.py +++ /dev/null @@ -1,99 +0,0 @@ -import ctypes -import os - -photon_client_library_path = os.path.join(os.path.abspath(os.path.dirname(__file__)), "../../build/photon_client.so") -photon_client_library = ctypes.cdll.LoadLibrary(photon_client_library_path) -photon_client_library.alloc_task_spec.restype = ctypes.c_void_p -photon_client_library.photon_connect.restype = ctypes.c_void_p -photon_client_library.photon_submit.restype = None -photon_client_library.photon_get_task.restype = ctypes.c_void_p - -ID = ctypes.c_ubyte * 20 - -buffer_from_read_write_memory = ctypes.pythonapi.PyBuffer_FromReadWriteMemory -buffer_from_read_write_memory.argtypes = [ctypes.c_void_p, ctypes.c_int64] -buffer_from_read_write_memory.restype = ctypes.py_object - -buffer_from_memory = ctypes.pythonapi.PyBuffer_FromMemory -buffer_from_memory.argtypes = [ctypes.c_void_p, ctypes.c_int64] -buffer_from_memory.restype = ctypes.py_object - -photon_client_library.task_function.restype = ctypes.c_void_p -photon_client_library.task_num_args.restype = ctypes.c_int64 -photon_client_library.task_num_returns.restype = ctypes.c_int64 -photon_client_library.task_arg_type.restype = ctypes.c_int8 -photon_client_library.task_arg_id.restype = ctypes.c_void_p -photon_client_library.task_arg_val.restype = ctypes.c_void_p -photon_client_library.task_arg_length.restype = ctypes.c_void_p -photon_client_library.task_return.restype = ctypes.c_void_p - - -class TaskInfo(object): - def __init__(self, function_id, args, return_ids): - self.function_id = function_id - self.args = args - self.return_ids = return_ids - -def extract_task(c_task): - function_id = buffer_from_memory(photon_client_library.task_function(c_task), 20)[:] - num_args = photon_client_library.task_num_args(c_task) - num_returns = photon_client_library.task_num_returns(c_task) - arg_vals_and_ids = [] - for i in range(num_args): - arg_type = photon_client_library.task_arg_type(c_task, i) - if arg_type == 0: - arg_id = buffer_from_memory(photon_client_library.task_arg_id(c_task, i), 20) - arg_vals_and_ids.append((arg_type, arg_id)) - elif arg_type == 1: - arg_val = photon_client_library.task_arg_val(c_task, i)[:] - arg_length = photon_client_library.task_arg_length(c_task, i) - arg_value = buffer_from_memory(arg_val, arg_length)[:] - arg_vals_and_ids.append((arg_type, arg_value)) - else: - raise Exception("arg_type must be 0 or 1") - return_ids = [] - for i in range(num_returns): - ret_id = buffer_from_memory(photon_client_library.task_return(c_task, i), 20) - return_ids.append(ret_id[:]) - return TaskInfo(function_id, arg_vals_and_ids, return_ids) - -class UniqueID(ctypes.Structure): - _fields_ = [("unique_id", ID)] - -def make_id(string): - if len(string) != 20: - raise Exception("PlasmaIDs must be 20 characters long") - unique_id = map(ord, string) - return UniqueID(unique_id=ID(*unique_id)) - -class Task(object): - def __init__(self, function_id, args, return_ids): - function_id = make_id(function_id) - self.task_spec = ctypes.c_void_p(photon_client_library.alloc_task_spec(function_id, len(args), 1, 0)) - for arg in args: - photon_client_library.task_args_add_ref(self.task_spec, make_id(arg)) - - # Add return IDs. This may not be the appropriate place for this. - num_returns = photon_client_library.task_num_returns(self.task_spec) - for i in range(num_returns): - ret_id = buffer_from_read_write_memory(photon_client_library.task_return(self.task_spec, i), 20) - for j in range(20): - ret_id[j] = return_ids[i][j] - - def __del__(self): - photon_client_library.free_task_spec(self.task_spec) - -class PhotonClient(object): - - def __init__(self, socket_name): - self.photon_conn = ctypes.c_void_p(photon_client_library.photon_connect(socket_name)) - - def submit(self, function_id, args, return_ids): - task = Task(function_id, args, return_ids) - photon_client_library.photon_submit(self.photon_conn, task.task_spec) - - def get_task(self): - c_task = ctypes.c_void_p(photon_client_library.photon_get_task(self.photon_conn)) - task = c_task # TODO Extract the actual task. EXTRACT...(c_task) - # photon_client_library.free_task_spec(c_task) - return extract_task(task) diff --git a/lib/python/photon_extension.c b/lib/python/photon_extension.c new file mode 100644 index 000000000..4aa199887 --- /dev/null +++ b/lib/python/photon_extension.c @@ -0,0 +1,140 @@ +#include + +#include "common_extension.h" +#include "photon_client.h" +#include "task.h" + +PyObject *PhotonError; + +// clang-format off +typedef struct { + PyObject_HEAD + photon_conn *photon_connection; +} PyPhotonClient; +// clang-format on + +static int PyPhotonClient_init(PyPhotonClient *self, PyObject *args, + PyObject *kwds) { + char *socket_name; + if (!PyArg_ParseTuple(args, "s", &socket_name)) { + return -1; + } + self->photon_connection = photon_connect(socket_name); + return 0; +} + +static void PyPhotonClient_dealloc(PyPhotonClient *self) { + free(((PyPhotonClient *)self)->photon_connection); + Py_TYPE(self)->tp_free((PyObject *)self); +} + +static PyObject *PyPhotonClient_submit(PyObject *self, PyObject *args) { + PyObject *py_task; + if (!PyArg_ParseTuple(args, "O", &py_task)) { + return NULL; + } + photon_submit(((PyPhotonClient *)self)->photon_connection, + ((PyTask *)py_task)->spec); + Py_RETURN_NONE; +} + +// clang-format off +static PyObject *PyPhotonClient_get_task(PyObject *self) { + task_spec *task_spec; + /* Drop the global interpreter lock while we get a task because + * photon_get_task may block for a long time. */ + Py_BEGIN_ALLOW_THREADS + task_spec = photon_get_task(((PyPhotonClient *)self)->photon_connection); + Py_END_ALLOW_THREADS + return PyTask_make(task_spec); +} +// clang-format on + +static PyMethodDef PyPhotonClient_methods[] = { + {"submit", (PyCFunction)PyPhotonClient_submit, METH_VARARGS, + "Submit a task to the local scheduler."}, + {"get_task", (PyCFunction)PyPhotonClient_get_task, METH_NOARGS, + "Get a task from the local scheduler."}, + {NULL} /* Sentinel */ +}; + +static PyTypeObject PyPhotonClientType = { + PyObject_HEAD_INIT(NULL) 0, /* ob_size */ + "photon.PhotonClient", /* tp_name */ + sizeof(PyPhotonClient), /* tp_basicsize */ + 0, /* tp_itemsize */ + (destructor)PyPhotonClient_dealloc, /* tp_dealloc */ + 0, /* tp_print */ + 0, /* tp_getattr */ + 0, /* tp_setattr */ + 0, /* tp_compare */ + 0, /* tp_repr */ + 0, /* tp_as_number */ + 0, /* tp_as_sequence */ + 0, /* tp_as_mapping */ + 0, /* tp_hash */ + 0, /* tp_call */ + 0, /* tp_str */ + 0, /* tp_getattro */ + 0, /* tp_setattro */ + 0, /* tp_as_buffer */ + Py_TPFLAGS_DEFAULT, /* tp_flags */ + "PhotonClient object", /* tp_doc */ + 0, /* tp_traverse */ + 0, /* tp_clear */ + 0, /* tp_richcompare */ + 0, /* tp_weaklistoffset */ + 0, /* tp_iter */ + 0, /* tp_iternext */ + PyPhotonClient_methods, /* tp_methods */ + 0, /* tp_members */ + 0, /* tp_getset */ + 0, /* tp_base */ + 0, /* tp_dict */ + 0, /* tp_descr_get */ + 0, /* tp_descr_set */ + 0, /* tp_dictoffset */ + (initproc)PyPhotonClient_init, /* tp_init */ + 0, /* tp_alloc */ + PyType_GenericNew, /* tp_new */ +}; + +static PyMethodDef photon_methods[] = { + {"check_simple_value", check_simple_value, METH_VARARGS, + "Should the object be passed by value?"}, + {NULL} /* Sentinel */ +}; + +#ifndef PyMODINIT_FUNC /* declarations for DLL import/export */ +#define PyMODINIT_FUNC void +#endif + +PyMODINIT_FUNC initphoton(void) { + PyObject *m; + + if (PyType_Ready(&PyTaskType) < 0) + return; + + if (PyType_Ready(&PyObjectIDType) < 0) + return; + + if (PyType_Ready(&PyPhotonClientType) < 0) + return; + + m = Py_InitModule3("photon", photon_methods, + "A module for the local scheduler."); + + Py_INCREF(&PyTaskType); + PyModule_AddObject(m, "Task", (PyObject *)&PyTaskType); + + Py_INCREF(&PyObjectIDType); + PyModule_AddObject(m, "ObjectID", (PyObject *)&PyObjectIDType); + + Py_INCREF(&PyPhotonClientType); + PyModule_AddObject(m, "PhotonClient", (PyObject *)&PyPhotonClientType); + + char photon_error[] = "photon.error"; + PhotonError = PyErr_NewException(photon_error, NULL, NULL); + Py_INCREF(PhotonError); + PyModule_AddObject(m, "photon_error", PhotonError); +} diff --git a/lib/python/setup.py b/lib/python/setup.py new file mode 100644 index 000000000..d94fd5a83 --- /dev/null +++ b/lib/python/setup.py @@ -0,0 +1,14 @@ +from setuptools import setup, find_packages, Extension + +photon_module = Extension("photon", + sources=["photon_extension.c", "../../common/lib/python/common_extension.c"], + include_dirs=["../../", "../../common/", + "../../common/thirdparty/", + "../../common/lib/python"], + extra_objects=["../../build/photon_client.a", "../../common/build/libcommon.a"], + extra_compile_args=["--std=c99", "-Werror"]) + +setup(name="Photon", + version="0.1", + description="Photon library for Ray", + ext_modules=[photon_module]) diff --git a/test/test.py b/test/test.py index bbd164c37..1d4cbc5af 100644 --- a/test/test.py +++ b/test/test.py @@ -45,13 +45,54 @@ class TestPhotonClient(unittest.TestCase): self.p2.kill() - def test_create(self): - l = [20 * "a", 20 * "b", 20 * "c"] - r = [20 * "e", 20 * "f"] - # Submit a task. - self.photon_client.submit(20 * "d", l, r) - # Get the task. - task = self.photon_client.get_task() + def test_submit_and_get_task(self): + # TODO(rkn): This should be a FunctionID. + function_id = photon.ObjectID(20 * "a") + object_ids = [photon.ObjectID(20 * chr(i)) for i in range(256)] + args_list = [ + [], + 1 * [1], + 10 * [1], + 100 * [1], + 1000 * [1], + 1 * ["a"], + 10 * ["a"], + 100 * ["a"], + 1000 * ["a"], + [1, 1.3, 2L, 1L << 100, "hi", u"hi", [1, 2]], + object_ids[:1], + object_ids[:2], + object_ids[:3], + object_ids[:4], + object_ids[:5], + object_ids[:10], + object_ids[:100], + object_ids[:256], + [1, object_ids[0]], + [object_ids[0], "a"], + [1, object_ids[0], "a"], + [object_ids[0], 1, object_ids[1], "a"], + object_ids[:3] + [1, "hi", 2.3] + object_ids[:5], + object_ids + 100 * ["a"] + object_ids + ] + + for args in args_list: + for num_return_vals in [0, 1, 2, 3, 5, 10, 100]: + task = photon.Task(function_id, args, num_return_vals) + # Submit a task. + self.photon_client.submit(task) + # Get the task. + new_task = self.photon_client.get_task() + self.assertEqual(task.function_id().id(), new_task.function_id().id()) + retrieved_args = new_task.arguments() + returns = new_task.returns() + self.assertEqual(len(args), len(retrieved_args)) + self.assertEqual(num_return_vals, len(returns)) + for i in range(len(retrieved_args)): + if isinstance(args[i], photon.ObjectID): + self.assertEqual(args[i].id(), retrieved_args[i].id()) + else: + self.assertEqual(args[i], retrieved_args[i]) if __name__ == "__main__": if len(sys.argv) > 1: From e9a336a34473113df2186ae16a38ec7b6baddcd8 Mon Sep 17 00:00:00 2001 From: Robert Nishihara Date: Fri, 7 Oct 2016 12:31:57 -0700 Subject: [PATCH 15/17] Fix incorrect free. (#9) --- photon_scheduler.c | 1 - test/test.py | 12 +++++++++++- 2 files changed, 11 insertions(+), 2 deletions(-) diff --git a/photon_scheduler.c b/photon_scheduler.c index b876bc9f2..1ce5fb916 100644 --- a/photon_scheduler.c +++ b/photon_scheduler.c @@ -74,7 +74,6 @@ void handle_submit_task(local_scheduler_state *s, task_spec *task) { (uint8_t *)task); /* Remove the available worker from the queue and free the struct. */ utarray_pop_back(s->available_worker_queue); - free(worker); } else { /* Add the task to the task queue. This passes ownership of the task queue. * And the task will be freed when it is assigned to a worker. */ diff --git a/test/test.py b/test/test.py index 1d4cbc5af..2db6df314 100644 --- a/test/test.py +++ b/test/test.py @@ -43,7 +43,7 @@ class TestPhotonClient(unittest.TestCase): os._exit(self.p2.returncode) else: self.p2.kill() - + def test_submit_and_get_task(self): # TODO(rkn): This should be a FunctionID. @@ -94,6 +94,16 @@ class TestPhotonClient(unittest.TestCase): else: self.assertEqual(args[i], retrieved_args[i]) + # Submit all of the tasks. + for args in args_list: + for num_return_vals in [0, 1, 2, 3, 5, 10, 100]: + task = photon.Task(function_id, args, num_return_vals) + self.photon_client.submit(task) + # Get all of the tasks. + for args in args_list: + for num_return_vals in [0, 1, 2, 3, 5, 10, 100]: + new_task = self.photon_client.get_task() + if __name__ == "__main__": if len(sys.argv) > 1: # pop the argument so we don't mess with unittest's own argument parser From 63ec24478487de76a984c525c0303e3a817e885d Mon Sep 17 00:00:00 2001 From: Robert Nishihara Date: Tue, 18 Oct 2016 18:27:43 -0700 Subject: [PATCH 16/17] Connect local scheduler to Plasma. (#11) * Receive notifications about sealed objects from Plasma, and schedule tasks only when the dependencies are available locally. * Fix formatting. * Use version of Plasma with fix. * Fix. * Factor out the scheduling algorithm and use worker_index instead of the client socket to identify workers * Fixes * clang-format * fix remaining linter errors --- .clang-format | 5 ++ .travis.yml | 11 +++ Makefile | 6 +- common | 2 +- photon.h | 28 ++++++- photon_algorithm.c | 184 +++++++++++++++++++++++++++++++++++++++++++ photon_algorithm.h | 88 +++++++++++++++++++++ photon_scheduler.c | 189 ++++++++++++++++++++++++--------------------- photon_scheduler.h | 31 +++++--- test/test.py | 53 +++++++++++-- 10 files changed, 485 insertions(+), 112 deletions(-) create mode 100644 .clang-format create mode 100644 photon_algorithm.c create mode 100644 photon_algorithm.h diff --git a/.clang-format b/.clang-format new file mode 100644 index 000000000..3fcffcbd3 --- /dev/null +++ b/.clang-format @@ -0,0 +1,5 @@ +BasedOnStyle: Chromium +DerivePointerAlignment: false +IndentCaseLabels: false +PointerAlignment: Right +SpaceAfterCStyleCast: true diff --git a/.travis.yml b/.travis.yml index 360027785..ac6ca4a6a 100644 --- a/.travis.yml +++ b/.travis.yml @@ -45,6 +45,17 @@ matrix: install: - ./install-dependencies.sh + + # Install Plasma side by side. + - cd .. + - git clone https://github.com/ray-project/plasma.git + - cd plasma + - git checkout f189ca746b57f22371ef10077aa535492bbd8421 + - make + - source setup-env.sh + - cd ../photon + + # Install Photon. - make - cd common/lib/python - python setup.py install --user diff --git a/Makefile b/Makefile index 436502d6a..81f38d06c 100644 --- a/Makefile +++ b/Makefile @@ -1,5 +1,5 @@ CC = gcc -CFLAGS = -g -Wall --std=c99 -D_XOPEN_SOURCE=500 -D_POSIX_C_SOURCE=200809L -Icommon/thirdparty -fPIC +CFLAGS = -g -Wall --std=c99 -D_XOPEN_SOURCE=500 -D_POSIX_C_SOURCE=200809L -Icommon -Icommon/thirdparty -fPIC BUILD = build all: $(BUILD)/photon_scheduler $(BUILD)/photon_client.a @@ -7,8 +7,8 @@ all: $(BUILD)/photon_scheduler $(BUILD)/photon_client.a $(BUILD)/photon_client.a: photon_client.o ar rcs $(BUILD)/photon_client.a photon_client.o -$(BUILD)/photon_scheduler: photon.h photon_scheduler.c common - $(CC) $(CFLAGS) -o $@ photon_scheduler.c common/build/libcommon.a common/thirdparty/hiredis/libhiredis.a -Icommon/thirdparty -Icommon/ +$(BUILD)/photon_scheduler: photon.h photon_scheduler.c photon_algorithm.c common + $(CC) $(CFLAGS) -o $@ photon_scheduler.c photon_algorithm.c common/build/libcommon.a common/thirdparty/hiredis/libhiredis.a -Icommon/thirdparty/ -Icommon/ ../plasma/build/libplasma_client.a -I../plasma/src/ common: FORCE git submodule update --init --recursive diff --git a/common b/common index 7be1a93d6..535bc8f0b 160000 --- a/common +++ b/common @@ -1 +1 @@ -Subproject commit 7be1a93d64ca36fc639e11f81de1483e0bd17b8c +Subproject commit 535bc8f0b8dac8f3ef0b66c3fd8b265ab0e6c787 diff --git a/photon.h b/photon.h index a59e5566f..298a27383 100644 --- a/photon.h +++ b/photon.h @@ -1,6 +1,11 @@ #ifndef PHOTON_H #define PHOTON_H +#include "common/task.h" +#include "common/state/db.h" +#include "utarray.h" +#include "uthash.h" + enum photon_message_type { /** Notify the local scheduler that a task has finished. */ TASK_DONE = 64, @@ -11,4 +16,25 @@ enum photon_message_type { EXECUTE_TASK, }; -#endif +// clang-format off +/** Contains all information that is associated to a worker. */ +typedef struct { + int sock; +} worker; +// clang-format on + +/* These are needed to define the UT_arrays. */ +UT_icd task_ptr_icd; +UT_icd worker_icd; + +/** Resources that are exposed to the scheduling algorithm. */ +typedef struct { + /** List of workers available to this node. The index into this array + * is the worker_index and is used to identify workers throughout + * the program. */ + UT_array *workers; + /* The handle to the database. */ + db_handle *db; +} scheduler_info; + +#endif /* PHOTON_H */ diff --git a/photon_algorithm.c b/photon_algorithm.c new file mode 100644 index 000000000..9f3f65d78 --- /dev/null +++ b/photon_algorithm.c @@ -0,0 +1,184 @@ +#include "photon_algorithm.h" + +#include +#include "utarray.h" + +#include "state/task_log.h" +#include "photon.h" +#include "photon_scheduler.h" + +typedef struct { + /* Object id of this object. */ + object_id object_id; + /* Handle for the uthash table. */ + UT_hash_handle handle; +} available_object; + +/** Part of the photon state that is maintained by the scheduling algorithm. */ +struct scheduler_state { + /** An array of pointers to tasks that are waiting to be scheduled. */ + UT_array *task_queue; + /** An array of worker indices corresponding to clients that are + * waiting for tasks. */ + UT_array *available_workers; + /** A hash map of the objects that are available in the local Plasma store. + * This information could be a little stale. */ + available_object *local_objects; +}; + +scheduler_state *make_scheduler_state(void) { + scheduler_state *state = malloc(sizeof(scheduler_state)); + /* Initialize an empty hash map for the cache of local available objects. */ + state->local_objects = NULL; + /* Initialize the local data structures used for queuing tasks and workers. */ + utarray_new(state->task_queue, &task_ptr_icd); + utarray_new(state->available_workers, &ut_int_icd); + return state; +} + +void free_scheduler_state(scheduler_state *s) { + utarray_free(s->task_queue); + utarray_free(s->available_workers); + free(s); +} + +/** + * Check if all of the remote object arguments for a task are available in the + * local object store. + * + * @param s The scheduler state. + * @param task Task specification of the task to check. + * @return This returns 1 if all of the remote object arguments for the task are + * present in the local object store, otherwise it returns 0. + */ +bool can_run(scheduler_state *s, task_spec *task) { + int64_t num_args = task_num_args(task); + for (int i = 0; i < num_args; ++i) { + if (task_arg_type(task, i) == ARG_BY_REF) { + object_id obj_id = *task_arg_id(task, i); + available_object *entry; + HASH_FIND(handle, s->local_objects, &obj_id, sizeof(object_id), entry); + if (entry == NULL) { + /* The object is not present locally, so this task cannot be scheduled + * right now. */ + return false; + } + } + } + return true; +} + +/** + * If there is a task whose dependencies are available locally, assign it to the + * worker. This does not remove the worker from the available worker queue. + * + * @param s The scheduler state. + * @param worker_index The index of the worker. + * @return This returns 1 if it successfully assigned a task to the worker, + * otherwise it returns 0. + */ +int find_and_schedule_task_if_possible(scheduler_info *info, + scheduler_state *state, + int worker_index) { + int found_task_to_schedule = 0; + /* Find the first task whose dependencies are available locally. */ + task_spec *spec; + task_instance **task; + int i = 0; + for (; i < utarray_len(state->task_queue); ++i) { + task = (task_instance **) utarray_eltptr(state->task_queue, i); + spec = task_instance_task_spec(*task); + if (can_run(state, spec)) { + found_task_to_schedule = 1; + break; + } + } + if (found_task_to_schedule) { + /* This task's dependencies are available locally, so assign the task to the + * worker. */ + assign_task_to_worker(info, spec, worker_index); + /* Update the task queue data structure and free the task. */ + free(*task); + utarray_erase(state->task_queue, i, 1); + } + return found_task_to_schedule; +} + +void handle_task_submitted(scheduler_info *info, + scheduler_state *s, + task_spec *task) { + /* Create a unique task instance ID. This is different from the task ID and + * is used to distinguish between potentially multiple executions of the + * task. */ + task_iid task_iid = globally_unique_id(); + task_instance *instance = + make_task_instance(task_iid, task, TASK_STATUS_WAITING, NIL_ID); + /* If this task's dependencies are available locally, and if there is an + * available worker, then assign this task to an available worker. Otherwise, + * add this task to the local task queue. */ + int schedule_locally = + (utarray_len(s->available_workers) > 0) && can_run(s, task); + if (schedule_locally) { + /* Get the last available worker in the available worker queue. */ + int *worker_index = (int *) utarray_back(s->available_workers); + /* Tell the available worker to execute the task. */ + assign_task_to_worker(info, task, *worker_index); + /* Remove the available worker from the queue and free the struct. */ + utarray_pop_back(s->available_workers); + } else { + /* Add the task to the task queue. This passes ownership of the task queue. + * And the task will be freed when it is assigned to a worker. */ + utarray_push_back(s->task_queue, &instance); + } + /* Submit the task to redis. */ + task_log_add_task(info->db, instance); + if (schedule_locally) { + /* If the task was scheduled locally, we need to free it. Otherwise, + * ownership of the task is passed to the task_queue, and it will be freed + * when it is assigned to a worker. */ + free(instance); + } +} + +void handle_worker_available(scheduler_info *info, + scheduler_state *state, + int worker_index) { + int scheduled_task = + find_and_schedule_task_if_possible(info, state, worker_index); + /* If we couldn't find a task to schedule, add the worker to the queue of + * available workers. */ + if (!scheduled_task) { + for (int *p = (int *) utarray_front(state->available_workers); p != NULL; + p = (int *) utarray_next(state->available_workers, p)) { + CHECK(*p != worker_index); + } + /* Add client_sock to a list of available workers. This struct will be freed + * when a task is assigned to this worker. */ + utarray_push_back(state->available_workers, &worker_index); + LOG_INFO("Adding worker_index %d to available workers.\n", worker_index); + } +} + +void handle_object_available(scheduler_info *info, + scheduler_state *state, + object_id object_id) { + /* TODO(rkn): When does this get freed? */ + available_object *entry = + (available_object *) malloc(sizeof(available_object)); + entry->object_id = object_id; + HASH_ADD(handle, state->local_objects, object_id, sizeof(object_id), entry); + + /* Check if we can schedule any tasks. */ + int num_tasks_scheduled = 0; + for (int *p = (int *) utarray_front(state->available_workers); p != NULL; + p = (int *) utarray_next(state->available_workers, p)) { + /* Schedule a task on this worker if possible. */ + int scheduled_task = find_and_schedule_task_if_possible(info, state, *p); + if (!scheduled_task) { + /* There are no tasks we can schedule, so exit the loop. */ + break; + } + num_tasks_scheduled += 1; + } + utarray_erase(state->available_workers, 0, num_tasks_scheduled); +} diff --git a/photon_algorithm.h b/photon_algorithm.h new file mode 100644 index 000000000..714de8869 --- /dev/null +++ b/photon_algorithm.h @@ -0,0 +1,88 @@ +#ifndef PHOTON_ALGORITHM_H +#define PHOTON_ALGORITHM_H + +#include "photon.h" +#include "common/task.h" + +/* ==== The scheduling algorithm ==== + * + * This file contains declaration for all functions and data structures + * that need to be provided if you want to implement a new algorithms + * for the local scheduler. + * + */ + +/** Internal state of the scheduling algorithm. */ +typedef struct scheduler_state scheduler_state; + +/** + * Initialize the scheduler state. + * + * @return Internal state of the scheduling algorithm. + */ +scheduler_state *make_scheduler_state(void); + +/** + * Free the scheduler state. + * + * @param state Internal state of the scheduling algorithm. + * @return Void. + */ +void free_scheduler_state(scheduler_state *state); + +/** + * This function will be called when a new task is submitted by a worker for + * execution. + * + * @param info Info about resources exposed by photon to the scheduling + * algorithm. + * @param state State of the scheduling algorithm. + * @param task Task that is submitted by the worker. + * @return Void. + */ +void handle_task_submitted(scheduler_info *info, + scheduler_state *state, + task_spec *task); + +/** + * This function will be called when a task is assigned by the global scheduler + * for execution on this local scheduler. + * + * @param info Info about resources exposed by photon to the scheduling + * algorithm. + * @param state State of the scheduling algorithm. + * @param task Task that is assigned by the global scheduler. + * @return Void. + */ +void handle_task_assigned(scheduler_info *info, + scheduler_state *state, + task_spec *task); + +/** + * This function is called if a new object becomes available in the local + * plasma store. + * + * @param info Info about resources exposed by photon to the scheduling + * algorithm. + * @param state State of the scheduling algorithm. + * @param object_id ID of the object that became available. + * @return Void. + */ +void handle_object_available(scheduler_info *info, + scheduler_state *state, + object_id object_id); + +/** + * This function is called when a new worker becomes available + * + * @param info Info about resources exposed by photon to the scheduling + * algorithm. + * @param state State of the scheduling algorithm. + * @param worker_index The index of the worker that becomes available. + * @return Void. + */ +void handle_worker_available(scheduler_info *info, + scheduler_state *state, + int worker_index); + +#endif /* PHOTON_ALGORITHM_H */ diff --git a/photon_scheduler.c b/photon_scheduler.c index 1ce5fb916..bdd18857d 100644 --- a/photon_scheduler.c +++ b/photon_scheduler.c @@ -10,111 +10,92 @@ #include "event_loop.h" #include "io.h" #include "photon.h" +#include "photon_algorithm.h" #include "photon_scheduler.h" +#include "plasma_client.h" #include "state/db.h" #include "state/task_log.h" #include "utarray.h" +#include "uthash.h" -typedef struct { - /** The file descriptor used to communicate with the worker. */ - int client_sock; -} available_worker; - -/* These are needed to define the UT_arrays. */ UT_icd task_ptr_icd = {sizeof(task_instance *), NULL, NULL, NULL}; -UT_icd worker_icd = {sizeof(available_worker), NULL, NULL, NULL}; +UT_icd worker_icd = {sizeof(worker), NULL, NULL, NULL}; + +/** Association between the socket fd of a worker and its worker_index. */ +typedef struct { + /** The socket fd of a worker. */ + int sock; + /** The index of the worker in scheduler_info->workers. */ + int64_t worker_index; + /** Handle for the hash table. */ + UT_hash_handle hh; +} worker_index; struct local_scheduler_state { /* The local scheduler event loop. */ event_loop *loop; - /* The handle to the database. */ - db_handle *db; - /** This is an array of pointers to tasks that are waiting to be scheduled. */ - UT_array *task_queue; - /** This is an array of file descriptors corresponding to clients that are - * waiting for tasks. */ - UT_array *available_worker_queue; + /* The Plasma client. */ + plasma_store_conn *plasma_conn; + /* Association between client socket and worker index. */ + worker_index *worker_index; + /* Info that is exposed to the scheduling algorithm. */ + scheduler_info *scheduler_info; + /* State for the scheduling algorithm. */ + scheduler_state *scheduler_state; }; -local_scheduler_state * -init_local_scheduler(event_loop *loop, const char *redis_addr, int redis_port) { +local_scheduler_state *init_local_scheduler(event_loop *loop, + const char *redis_addr, + int redis_port, + const char *plasma_socket_name) { local_scheduler_state *state = malloc(sizeof(local_scheduler_state)); state->loop = loop; - state->db = db_connect(redis_addr, redis_port, "photon", "", -1); - db_attach(state->db, loop); - utarray_new(state->task_queue, &task_ptr_icd); - utarray_new(state->available_worker_queue, &worker_icd); + /* Connect to Plasma. This method will retry if Plasma hasn't started yet. */ + state->plasma_conn = plasma_store_connect(plasma_socket_name); + /* Subscribe to notifications about sealed objects. */ + int plasma_fd = plasma_subscribe(state->plasma_conn); + /* Add the callback that processes the notification to the event loop. */ + event_loop_add_file(loop, plasma_fd, EVENT_LOOP_READ, + process_plasma_notification, state); + state->worker_index = NULL; + /* Add scheduler info. */ + state->scheduler_info = malloc(sizeof(scheduler_info)); + utarray_new(state->scheduler_info->workers, &worker_icd); + /* Connect to Redis. */ + state->scheduler_info->db = + db_connect(redis_addr, redis_port, "photon", "", -1); + db_attach(state->scheduler_info->db, loop); + /* Add scheduler state. */ + state->scheduler_state = make_scheduler_state(); return state; }; void free_local_scheduler(local_scheduler_state *s) { - db_disconnect(s->db); - utarray_free(s->task_queue); - utarray_free(s->available_worker_queue); + db_disconnect(s->scheduler_info->db); + free(s->scheduler_info); + free_scheduler_state(s->scheduler_state); event_loop_destroy(s->loop); free(s); } -void handle_submit_task(local_scheduler_state *s, task_spec *task) { - /* Create a unique task instance ID. This is different from the task ID and - * is used to distinguish between potentially multiple executions of the - * task. */ - task_iid task_iid = globally_unique_id(); - task_instance *instance = - make_task_instance(task_iid, task, TASK_STATUS_WAITING, NIL_ID); - /* Assign this task to an available worker. If there are no available workers, - * then add this task to the local task queue. */ - int schedule_locally = utarray_len(s->available_worker_queue) > 0; - if (schedule_locally) { - /* Get the last available worker in the available worker queue. */ - available_worker *worker = - (available_worker *)utarray_back(s->available_worker_queue); - /* Tell the available worker to execute the task. */ - write_message(worker->client_sock, EXECUTE_TASK, task_size(task), - (uint8_t *)task); - /* Remove the available worker from the queue and free the struct. */ - utarray_pop_back(s->available_worker_queue); - } else { - /* Add the task to the task queue. This passes ownership of the task queue. - * And the task will be freed when it is assigned to a worker. */ - utarray_push_back(s->task_queue, &instance); - } - /* Submit the task to redis. */ - task_log_add_task(s->db, instance); - if (schedule_locally) { - /* If the task was scheduled locally, we need to free it. Otherwise, - * ownership of the task is passed to the task_queue, and it will be freed - * when it is assigned to a worker. */ - free(instance); - } +void assign_task_to_worker(scheduler_info *info, + task_spec *task, + int worker_index) { + CHECK(worker_index < utarray_len(info->workers)); + worker *w = (worker *) utarray_eltptr(info->workers, worker_index); + write_message(w->sock, EXECUTE_TASK, task_size(task), (uint8_t *) task); } -void handle_get_task(local_scheduler_state *s, int client_sock) { - /* If there is an available task, assign that task to this worker. Otherwise - * add the worker to the queue of available workers. */ - if (utarray_len(s->task_queue) > 0) { - /* Get the last task in the task queue. */ - task_instance **back = (task_instance **)utarray_back(s->task_queue); - task_spec *task = task_instance_task_spec(*back); - /* Send a task to the worker. */ - write_message(client_sock, EXECUTE_TASK, task_size(task), (uint8_t *)task); - /* Update the task queue data structure and free the task. */ - utarray_pop_back(s->task_queue); - free(*back); - } else { - /* Check that client_sock is not already in the available workers. */ - for (available_worker *p = - (available_worker *)utarray_front(s->available_worker_queue); - p != NULL; - p = (available_worker *)utarray_next(s->available_worker_queue, p)) { - CHECK(p->client_sock != client_sock); - } - /* Add client_sock to a list of available workers. This struct will be freed - * when a task is assigned to this worker. */ - available_worker worker_info = {.client_sock = client_sock}; - utarray_push_back(s->available_worker_queue, &worker_info); - LOG_INFO("Adding client_sock %d to available workers.\n", client_sock); - } +void process_plasma_notification(event_loop *loop, + int client_sock, + void *context, + int events) { + local_scheduler_state *s = context; + /* Read the notification from Plasma. */ + uint8_t *message = (uint8_t *) malloc(sizeof(object_id)); + recv(client_sock, message, sizeof(object_id), 0); + object_id *obj_id = (object_id *) message; + handle_object_available(s->scheduler_info, s->scheduler_state, *obj_id); } void process_message(event_loop *loop, int client_sock, void *context, @@ -126,16 +107,22 @@ void process_message(event_loop *loop, int client_sock, void *context, int64_t length; read_message(client_sock, &type, &length, &message); + LOG_DEBUG("New event of type %" PRId64, type); + switch (type) { case SUBMIT_TASK: { - task_spec *task = (task_spec *)message; - CHECK(task_size(task) == length); - handle_submit_task(s, task); + task_spec *spec = (task_spec *) message; + CHECK(task_size(spec) == length); + handle_task_submitted(s->scheduler_info, s->scheduler_state, spec); } break; case TASK_DONE: { } break; case GET_TASK: { - handle_get_task(s, client_sock); + worker_index *wi; + HASH_FIND_INT(s->worker_index, &client_sock, wi); + printf("worker_index is %" PRId64 "\n", wi->worker_index); + handle_worker_available(s->scheduler_info, s->scheduler_state, + wi->worker_index); } break; case DISCONNECT_CLIENT: { LOG_INFO("Disconnecting client on fd %d", client_sock); @@ -156,6 +143,14 @@ void new_client_connection(event_loop *loop, int listener_sock, void *context, int new_socket = accept_client(listener_sock); event_loop_add_file(loop, new_socket, EVENT_LOOP_READ, process_message, s); LOG_INFO("new connection with fd %d", new_socket); + /* Add worker to list of workers. */ + /* TODO(pcm): Where shall we free this? */ + worker_index *new_worker_index = malloc(sizeof(worker_index)); + new_worker_index->sock = new_socket; + new_worker_index->worker_index = utarray_len(s->scheduler_info->workers); + HASH_ADD_INT(s->worker_index, sock, new_worker_index); + worker worker = {.sock = new_socket}; + utarray_push_back(s->scheduler_info->workers, &worker); } /* We need this code so we can clean up when we get a SIGTERM signal. */ @@ -171,11 +166,14 @@ void signal_handler(int signal) { /* End of the cleanup code. */ -void start_server(const char *socket_name, const char *redis_addr, - int redis_port) { +void start_server(const char *socket_name, + const char *redis_addr, + int redis_port, + const char *plasma_socket_name) { int fd = bind_ipc_sock(socket_name); event_loop *loop = event_loop_create(); - g_state = init_local_scheduler(loop, redis_addr, redis_port); + g_state = + init_local_scheduler(loop, redis_addr, redis_port, plasma_socket_name); /* Run event loop. */ event_loop_add_file(loop, fd, EVENT_LOOP_READ, new_client_connection, @@ -189,8 +187,10 @@ int main(int argc, char *argv[]) { char *scheduler_socket_name = NULL; /* IP address and port of redis. */ char *redis_addr_port = NULL; + /* Socket name for the local Plasma store. */ + char *plasma_socket_name = NULL; int c; - while ((c = getopt(argc, argv, "s:r:")) != -1) { + while ((c = getopt(argc, argv, "s:r:p:")) != -1) { switch (c) { case 's': scheduler_socket_name = optarg; @@ -198,6 +198,9 @@ int main(int argc, char *argv[]) { case 'r': redis_addr_port = optarg; break; + case 'p': + plasma_socket_name = optarg; + break; default: LOG_ERR("unknown option %c", c); exit(-1); @@ -207,6 +210,11 @@ int main(int argc, char *argv[]) { LOG_ERR("please specify socket for incoming connections with -s switch"); exit(-1); } + if (!plasma_socket_name) { + LOG_ERR("please specify socket for connecting to Plasma with -p switch"); + exit(-1); + } + /* Parse the Redis address into an IP address and a port. */ char redis_addr[16] = {0}; char redis_port[6] = {0}; if (!redis_addr_port || @@ -215,5 +223,6 @@ int main(int argc, char *argv[]) { LOG_ERR("need to specify redis address like 127.0.0.1:6379 with -r switch"); exit(-1); } - start_server(scheduler_socket_name, &redis_addr[0], atoi(redis_port)); + start_server(scheduler_socket_name, &redis_addr[0], atoi(redis_port), + plasma_socket_name); } diff --git a/photon_scheduler.h b/photon_scheduler.h index 591ffe0f5..3ebb03432 100644 --- a/photon_scheduler.h +++ b/photon_scheduler.h @@ -2,6 +2,7 @@ #define PHOTON_SCHEDULER_H #include "task.h" +#include "event_loop.h" typedef struct local_scheduler_state local_scheduler_state; @@ -15,25 +16,37 @@ typedef struct local_scheduler_state local_scheduler_state; * @param events Flag for events that are available on the listener socket. * @return Void. */ -void new_client_connection(event_loop *loop, int listener_sock, void *context, +void new_client_connection(event_loop *loop, + int listener_sock, + void *context, int events); /** - * Assign a task to a worker. + * This function can be called by the scheduling algorithm to assign a task + * to a worker. * - * @param s State of the local scheduler. - * @param client_sock Socket by which the worker is connected. + * @param info + * @param task The task that is submitted to the worker. + * @param worker_index The index of the worker the task is submitted to. * @return Void. */ -void handle_get_task(local_scheduler_state *s, int client_sock); +void assign_task_to_worker(scheduler_info *info, + task_spec *task, + int worker_index); /** - * Handle incoming submit request by a worker. + * This is the callback that is used to process a notification from the Plasma + * store that an object has been sealed. * - * @param s State of the local scheduler. - * @param task Task specification of the task to be submitted. + * @param loop The local scheduler's event loop. + * @param client_sock The file descriptor to read the notification from. + * @param context The local scheduler state. + * @param events * @return Void. */ -void handle_submit_task(local_scheduler_state *s, task_spec *task); +void process_plasma_notification(event_loop *loop, + int client_sock, + void *context, + int events); #endif /* PHOTON_SCHEDULER_H */ diff --git a/test/test.py b/test/test.py index 2db6df314..fb35702ad 100644 --- a/test/test.py +++ b/test/test.py @@ -6,9 +6,11 @@ import subprocess import sys import unittest import random +import threading import time import photon +import plasma USE_VALGRIND = False @@ -18,14 +20,19 @@ class TestPhotonClient(unittest.TestCase): # Start Redis. redis_executable = os.path.join(os.path.abspath(os.path.dirname(__file__)), "../common/thirdparty/redis-3.2.3/src/redis-server") self.p1 = subprocess.Popen([redis_executable, "--loglevel", "warning"]) + # Start Plasma. + plasma_executable = os.path.join(os.path.abspath(os.path.dirname(__file__)), "../../plasma/build/plasma_store") + plasma_socket = "/tmp/plasma_store{}".format(random.randint(0, 10000)) + self.p2 = subprocess.Popen([plasma_executable, "-s", plasma_socket]) time.sleep(0.1) + self.plasma_client = plasma.PlasmaClient(plasma_socket) scheduler_executable = os.path.join(os.path.abspath(os.path.dirname(__file__)), "../build/photon_scheduler") scheduler_name = "/tmp/scheduler{}".format(random.randint(0, 10000)) - command = [scheduler_executable, "-s", scheduler_name, "-r", "127.0.0.1:6379"] + command = [scheduler_executable, "-s", scheduler_name, "-r", "127.0.0.1:6379", "-p", plasma_socket] if USE_VALGRIND: - self.p2 = subprocess.Popen(["valgrind", "--track-origins=yes", "--leak-check=full", "--show-leak-kinds=all"] + command) + self.p3 = subprocess.Popen(["valgrind", "--track-origins=yes", "--leak-check=full", "--show-leak-kinds=all"] + command) else: - self.p2 = subprocess.Popen(command) + self.p3 = subprocess.Popen(command) if USE_VALGRIND: time.sleep(1.0) else: @@ -36,21 +43,30 @@ class TestPhotonClient(unittest.TestCase): def tearDown(self): # Kill the Redis server. self.p1.kill() + # Kill Plasma. + self.p2.kill() # Kill the local scheduler. if USE_VALGRIND: - self.p2.send_signal(signal.SIGTERM) - self.p2.wait() - os._exit(self.p2.returncode) + self.p3.send_signal(signal.SIGTERM) + self.p3.wait() + os._exit(self.p3.returncode) else: - self.p2.kill() - + self.p3.kill() def test_submit_and_get_task(self): # TODO(rkn): This should be a FunctionID. function_id = photon.ObjectID(20 * "a") object_ids = [photon.ObjectID(20 * chr(i)) for i in range(256)] + # Create and seal the objects in the object store so that we can schedule + # all of the subsequent tasks. + for object_id in object_ids: + self.plasma_client.create(object_id.id(), 0) + self.plasma_client.seal(object_id.id()) + # Define some arguments to use for the tasks. args_list = [ [], + #{}, + #(), 1 * [1], 10 * [1], 100 * [1], @@ -104,6 +120,27 @@ class TestPhotonClient(unittest.TestCase): for num_return_vals in [0, 1, 2, 3, 5, 10, 100]: new_task = self.photon_client.get_task() + def test_scheduling_when_objects_ready(self): + # Create a task and submit it. + object_id = photon.ObjectID(20 * chr(0)) + # TODO(rkn): This should be a FunctionID. + function_id = photon.ObjectID(20 * "a") + task = photon.Task(function_id, [object_id], 0) + self.photon_client.submit(task) + # Launch a thread to get the task. + def get_task(): + self.photon_client.get_task() + t = threading.Thread(target=get_task) + t.start() + # Sleep to give the thread time to call get_task. + time.sleep(0.1) + # Create and seal the object ID in the object store. This should trigger a + # scheduling event. + self.plasma_client.create(object_id.id(), 0) + self.plasma_client.seal(object_id.id()) + # Wait until the thread finishes so that we know the task was scheduled. + t.join() + if __name__ == "__main__": if len(sys.argv) > 1: # pop the argument so we don't mess with unittest's own argument parser From ad55166472f46e85fe8aae273763df45eb5fa4a4 Mon Sep 17 00:00:00 2001 From: Robert Nishihara Date: Tue, 25 Oct 2016 14:16:23 -0700 Subject: [PATCH 17/17] Rearrange local scheduler files to prepare to merge into Ray. --- .clang-format | 5 - .gitignore | 33 -- .gitmodules | 3 - .travis.yml | 71 --- .travis/check-git-clang-format-output.sh | 18 - .travis/git-clang-format | 476 ------------------ LICENSE | 201 -------- README.md | 2 - common | 1 - install-dependencies.sh | 21 - setup-env.sh | 5 - Makefile => src/photon/Makefile | 0 {build => src/photon/build}/.gitkeep | 0 .../photon/lib}/python/photon_extension.c | 0 {lib => src/photon/lib}/python/setup.py | 0 photon.h => src/photon/photon.h | 0 .../photon/photon_algorithm.c | 0 .../photon/photon_algorithm.h | 0 photon_client.c => src/photon/photon_client.c | 0 photon_client.h => src/photon/photon_client.h | 0 .../photon/photon_scheduler.c | 0 .../photon/photon_scheduler.h | 0 {test => src/photon/test}/test.py | 0 23 files changed, 836 deletions(-) delete mode 100644 .clang-format delete mode 100644 .gitignore delete mode 100644 .gitmodules delete mode 100644 .travis.yml delete mode 100755 .travis/check-git-clang-format-output.sh delete mode 100755 .travis/git-clang-format delete mode 100644 LICENSE delete mode 100644 README.md delete mode 160000 common delete mode 100755 install-dependencies.sh delete mode 100755 setup-env.sh rename Makefile => src/photon/Makefile (100%) rename {build => src/photon/build}/.gitkeep (100%) rename {lib => src/photon/lib}/python/photon_extension.c (100%) rename {lib => src/photon/lib}/python/setup.py (100%) rename photon.h => src/photon/photon.h (100%) rename photon_algorithm.c => src/photon/photon_algorithm.c (100%) rename photon_algorithm.h => src/photon/photon_algorithm.h (100%) rename photon_client.c => src/photon/photon_client.c (100%) rename photon_client.h => src/photon/photon_client.h (100%) rename photon_scheduler.c => src/photon/photon_scheduler.c (100%) rename photon_scheduler.h => src/photon/photon_scheduler.h (100%) rename {test => src/photon/test}/test.py (100%) diff --git a/.clang-format b/.clang-format deleted file mode 100644 index 3fcffcbd3..000000000 --- a/.clang-format +++ /dev/null @@ -1,5 +0,0 @@ -BasedOnStyle: Chromium -DerivePointerAlignment: false -IndentCaseLabels: false -PointerAlignment: Right -SpaceAfterCStyleCast: true diff --git a/.gitignore b/.gitignore deleted file mode 100644 index f805e810e..000000000 --- a/.gitignore +++ /dev/null @@ -1,33 +0,0 @@ -# Object files -*.o -*.ko -*.obj -*.elf - -# Precompiled Headers -*.gch -*.pch - -# Libraries -*.lib -*.a -*.la -*.lo - -# Shared objects (inc. Windows DLLs) -*.dll -*.so -*.so.* -*.dylib - -# Executables -*.exe -*.out -*.app -*.i*86 -*.x86_64 -*.hex - -# Debug files -*.dSYM/ -*.su diff --git a/.gitmodules b/.gitmodules deleted file mode 100644 index 3f2c8add4..000000000 --- a/.gitmodules +++ /dev/null @@ -1,3 +0,0 @@ -[submodule "common"] - path = common - url = https://github.com/ray-project/common diff --git a/.travis.yml b/.travis.yml deleted file mode 100644 index ac6ca4a6a..000000000 --- a/.travis.yml +++ /dev/null @@ -1,71 +0,0 @@ -sudo: required - -language: generic - -matrix: - include: - - os: linux - dist: trusty - python: "2.7" - - os: linux - dist: trusty - python: "3.5" - - os: osx - osx_image: xcode7 - python: "2.7" - - os: osx - osx_image: xcode7 - python: "3.5" - - os: linux - dist: trusty - python: "2.7" - env: LINT=1 - before_install: - # In case we ever want to use a different version of clang-format: - #- wget -O - http://apt.llvm.org/llvm-snapshot.gpg.key | sudo apt-key add - - #- echo "deb http://apt.llvm.org/trusty/ llvm-toolchain-trusty main" | sudo tee -a /etc/apt/sources.list > /dev/null - - sudo apt-get update -qq - - sudo apt-get install -qq clang-format-3.8 - install: [] - script: - - .travis/check-git-clang-format-output.sh - - os: linux - dist: trusty - python: "2.7" - env: VALGRIND=1 - before_install: - - sudo apt-get update -qq - - sudo apt-get install -qq valgrind - script: - - cd common - - make test - - cd .. - - source setup-env.sh - - python test/test.py valgrind - -install: - - ./install-dependencies.sh - - # Install Plasma side by side. - - cd .. - - git clone https://github.com/ray-project/plasma.git - - cd plasma - - git checkout f189ca746b57f22371ef10077aa535492bbd8421 - - make - - source setup-env.sh - - cd ../photon - - # Install Photon. - - make - - cd common/lib/python - - python setup.py install --user - - cd ../../.. - - cd lib/python - - python setup.py install --user - - cd ../.. - - cd common - - make test - - cd .. - -script: - - python test/test.py diff --git a/.travis/check-git-clang-format-output.sh b/.travis/check-git-clang-format-output.sh deleted file mode 100755 index d71f78357..000000000 --- a/.travis/check-git-clang-format-output.sh +++ /dev/null @@ -1,18 +0,0 @@ -#!/bin/bash -if [ "$TRAVIS_PULL_REQUEST" == "false" ] ; then - # Not in a pull request, so compare against parent commit - base_commit="HEAD^" - echo "Running clang-format against parent commit $(git rev-parse $base_commit)" -else - base_commit="$TRAVIS_BRANCH" - echo "Running clang-format against branch $base_commit, with hash $(git rev-parse $base_commit)" -fi -output="$(.travis/git-clang-format --binary clang-format-3.8 --commit $base_commit --diff --exclude ^thirdparty/)" -if [ "$output" == "no modified files to format" ] || [ "$output" == "clang-format did not modify any files" ] ; then - echo "clang-format passed." - exit 0 -else - echo "clang-format failed:" - echo "$output" - exit 1 -fi diff --git a/.travis/git-clang-format b/.travis/git-clang-format deleted file mode 100755 index b0e458303..000000000 --- a/.travis/git-clang-format +++ /dev/null @@ -1,476 +0,0 @@ -#!/usr/bin/env python -# -#===- git-clang-format - ClangFormat Git Integration ---------*- python -*--===# -# -# The LLVM Compiler Infrastructure -# -# This file is distributed under the University of Illinois Open Source -# License. See LICENSE.TXT for details. -# -#===------------------------------------------------------------------------===# - -r""" -clang-format git integration -============================ - -This file provides a clang-format integration for git. Put it somewhere in your -path and ensure that it is executable. Then, "git clang-format" will invoke -clang-format on the changes in current files or a specific commit. - -For further details, run: -git clang-format -h - -Requires Python 2.7 -""" - -import argparse -import collections -import contextlib -import errno -import os -import re -import subprocess -import sys - -usage = 'git clang-format [OPTIONS] [] [--] [...]' - -desc = ''' -Run clang-format on all lines that differ between the working directory -and , which defaults to HEAD. Changes are only applied to the working -directory. -The following git-config settings set the default of the corresponding option: - clangFormat.binary - clangFormat.commit - clangFormat.extension - clangFormat.style -''' - -# Name of the temporary index file in which save the output of clang-format. -# This file is created within the .git directory. -temp_index_basename = 'clang-format-index' - - -Range = collections.namedtuple('Range', 'start, count') - - -def main(): - config = load_git_config() - - # In order to keep '--' yet allow options after positionals, we need to - # check for '--' ourselves. (Setting nargs='*' throws away the '--', while - # nargs=argparse.REMAINDER disallows options after positionals.) - argv = sys.argv[1:] - try: - idx = argv.index('--') - except ValueError: - dash_dash = [] - else: - dash_dash = argv[idx:] - argv = argv[:idx] - - default_extensions = ','.join([ - # From clang/lib/Frontend/FrontendOptions.cpp, all lower case - 'c', 'h', # C - 'm', # ObjC - 'mm', # ObjC++ - 'cc', 'cp', 'cpp', 'c++', 'cxx', 'hpp', # C++ - # Other languages that clang-format supports - 'proto', 'protodevel', # Protocol Buffers - 'js', # JavaScript - 'ts', # TypeScript - ]) - - p = argparse.ArgumentParser( - usage=usage, formatter_class=argparse.RawDescriptionHelpFormatter, - description=desc) - p.add_argument('--binary', - default=config.get('clangformat.binary', 'clang-format'), - help='path to clang-format'), - p.add_argument('--commit', - default=config.get('clangformat.commit', 'HEAD'), - help='default commit to use if none is specified'), - p.add_argument('--diff', action='store_true', - help='print a diff instead of applying the changes') - p.add_argument('--extensions', - default=config.get('clangformat.extensions', - default_extensions), - help=('comma-separated list of file extensions to format, ' - 'excluding the period and case-insensitive')), - p.add_argument('--exclude', help='Exclude files matching this regex.') - p.add_argument('-f', '--force', action='store_true', - help='allow changes to unstaged files') - p.add_argument('-p', '--patch', action='store_true', - help='select hunks interactively') - p.add_argument('-q', '--quiet', action='count', default=0, - help='print less information') - p.add_argument('--style', - default=config.get('clangformat.style', None), - help='passed to clang-format'), - p.add_argument('-v', '--verbose', action='count', default=0, - help='print extra information') - # We gather all the remaining positional arguments into 'args' since we need - # to use some heuristics to determine whether or not was present. - # However, to print pretty messages, we make use of metavar and help. - p.add_argument('args', nargs='*', metavar='', - help='revision from which to compute the diff') - p.add_argument('ignored', nargs='*', metavar='...', - help='if specified, only consider differences in these files') - opts = p.parse_args(argv) - - opts.verbose -= opts.quiet - del opts.quiet - - commit, files = interpret_args(opts.args, dash_dash, opts.commit) - changed_lines = compute_diff_and_extract_lines(commit, files) - if opts.verbose >= 1: - ignored_files = set(changed_lines) - filter_by_extension(changed_lines, opts.extensions.lower().split(',')) - if opts.exclude: - for filename in changed_lines.keys(): - if re.match(opts.exclude, filename): - del changed_lines[filename] - if opts.verbose >= 1: - ignored_files.difference_update(changed_lines) - if ignored_files: - print 'Ignoring changes in the following files:' - for filename in ignored_files: - print ' ', filename - if changed_lines: - print 'Running clang-format on the following files:' - for filename in changed_lines: - print ' ', filename - if not changed_lines: - print 'no modified files to format' - return - # The computed diff outputs absolute paths, so we must cd before accessing - # those files. - cd_to_toplevel() - old_tree = create_tree_from_workdir(changed_lines) - new_tree = run_clang_format_and_save_to_tree(changed_lines, - binary=opts.binary, - style=opts.style) - if opts.verbose >= 1: - print 'old tree:', old_tree - print 'new tree:', new_tree - if old_tree == new_tree: - if opts.verbose >= 0: - print 'clang-format did not modify any files' - elif opts.diff: - print_diff(old_tree, new_tree) - else: - changed_files = apply_changes(old_tree, new_tree, force=opts.force, - patch_mode=opts.patch) - if (opts.verbose >= 0 and not opts.patch) or opts.verbose >= 1: - print 'changed files:' - for filename in changed_files: - print ' ', filename - - -def load_git_config(non_string_options=None): - """Return the git configuration as a dictionary. - All options are assumed to be strings unless in `non_string_options`, in which - is a dictionary mapping option name (in lower case) to either "--bool" or - "--int".""" - if non_string_options is None: - non_string_options = {} - out = {} - for entry in run('git', 'config', '--list', '--null').split('\0'): - if entry: - name, value = entry.split('\n', 1) - if name in non_string_options: - value = run('git', 'config', non_string_options[name], name) - out[name] = value - return out - - -def interpret_args(args, dash_dash, default_commit): - """Interpret `args` as "[commit] [--] [files...]" and return (commit, files). - It is assumed that "--" and everything that follows has been removed from - args and placed in `dash_dash`. - If "--" is present (i.e., `dash_dash` is non-empty), the argument to its - left (if present) is taken as commit. Otherwise, the first argument is - checked if it is a commit or a file. If commit is not given, - `default_commit` is used.""" - if dash_dash: - if len(args) == 0: - commit = default_commit - elif len(args) > 1: - die('at most one commit allowed; %d given' % len(args)) - else: - commit = args[0] - object_type = get_object_type(commit) - if object_type not in ('commit', 'tag'): - if object_type is None: - die("'%s' is not a commit" % commit) - else: - die("'%s' is a %s, but a commit was expected" % (commit, object_type)) - files = dash_dash[1:] - elif args: - if disambiguate_revision(args[0]): - commit = args[0] - files = args[1:] - else: - commit = default_commit - files = args - else: - commit = default_commit - files = [] - return commit, files - - -def disambiguate_revision(value): - """Returns True if `value` is a revision, False if it is a file, or dies.""" - # If `value` is ambiguous (neither a commit nor a file), the following - # command will die with an appropriate error message. - run('git', 'rev-parse', value, verbose=False) - object_type = get_object_type(value) - if object_type is None: - return False - if object_type in ('commit', 'tag'): - return True - die('`%s` is a %s, but a commit or filename was expected' % - (value, object_type)) - - -def get_object_type(value): - """Returns a string description of an object's type, or None if it is not - a valid git object.""" - cmd = ['git', 'cat-file', '-t', value] - p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE) - stdout, stderr = p.communicate() - if p.returncode != 0: - return None - return stdout.strip() - - -def compute_diff_and_extract_lines(commit, files): - """Calls compute_diff() followed by extract_lines().""" - diff_process = compute_diff(commit, files) - changed_lines = extract_lines(diff_process.stdout) - diff_process.stdout.close() - diff_process.wait() - if diff_process.returncode != 0: - # Assume error was already printed to stderr. - sys.exit(2) - return changed_lines - - -def compute_diff(commit, files): - """Return a subprocess object producing the diff from `commit`. - The return value's `stdin` file object will produce a patch with the - differences between the working directory and `commit`, filtered on `files` - (if non-empty). Zero context lines are used in the patch.""" - cmd = ['git', 'diff-index', '-p', '-U0', commit, '--'] - cmd.extend(files) - p = subprocess.Popen(cmd, stdin=subprocess.PIPE, stdout=subprocess.PIPE) - p.stdin.close() - return p - - -def extract_lines(patch_file): - """Extract the changed lines in `patch_file`. - The return value is a dictionary mapping filename to a list of (start_line, - line_count) pairs. - The input must have been produced with ``-U0``, meaning unidiff format with - zero lines of context. The return value is a dict mapping filename to a - list of line `Range`s.""" - matches = {} - for line in patch_file: - match = re.search(r'^\+\+\+\ [^/]+/(.*)', line) - if match: - filename = match.group(1).rstrip('\r\n') - match = re.search(r'^@@ -[0-9,]+ \+(\d+)(,(\d+))?', line) - if match: - start_line = int(match.group(1)) - line_count = 1 - if match.group(3): - line_count = int(match.group(3)) - if line_count > 0: - matches.setdefault(filename, []).append(Range(start_line, line_count)) - return matches - - -def filter_by_extension(dictionary, allowed_extensions): - """Delete every key in `dictionary` that doesn't have an allowed extension. - `allowed_extensions` must be a collection of lowercase file extensions, - excluding the period.""" - allowed_extensions = frozenset(allowed_extensions) - for filename in dictionary.keys(): - base_ext = filename.rsplit('.', 1) - if len(base_ext) == 1 or base_ext[1].lower() not in allowed_extensions: - del dictionary[filename] - - -def cd_to_toplevel(): - """Change to the top level of the git repository.""" - toplevel = run('git', 'rev-parse', '--show-toplevel') - os.chdir(toplevel) - - -def create_tree_from_workdir(filenames): - """Create a new git tree with the given files from the working directory. - Returns the object ID (SHA-1) of the created tree.""" - return create_tree(filenames, '--stdin') - - -def run_clang_format_and_save_to_tree(changed_lines, binary='clang-format', - style=None): - """Run clang-format on each file and save the result to a git tree. - Returns the object ID (SHA-1) of the created tree.""" - def index_info_generator(): - for filename, line_ranges in changed_lines.iteritems(): - mode = oct(os.stat(filename).st_mode) - blob_id = clang_format_to_blob(filename, line_ranges, binary=binary, - style=style) - yield '%s %s\t%s' % (mode, blob_id, filename) - return create_tree(index_info_generator(), '--index-info') - - -def create_tree(input_lines, mode): - """Create a tree object from the given input. - If mode is '--stdin', it must be a list of filenames. If mode is - '--index-info' is must be a list of values suitable for "git update-index - --index-info", such as " ". Any other mode - is invalid.""" - assert mode in ('--stdin', '--index-info') - cmd = ['git', 'update-index', '--add', '-z', mode] - with temporary_index_file(): - p = subprocess.Popen(cmd, stdin=subprocess.PIPE) - for line in input_lines: - p.stdin.write('%s\0' % line) - p.stdin.close() - if p.wait() != 0: - die('`%s` failed' % ' '.join(cmd)) - tree_id = run('git', 'write-tree') - return tree_id - - -def clang_format_to_blob(filename, line_ranges, binary='clang-format', - style=None): - """Run clang-format on the given file and save the result to a git blob. - Returns the object ID (SHA-1) of the created blob.""" - clang_format_cmd = [binary, filename] - if style: - clang_format_cmd.extend(['-style='+style]) - clang_format_cmd.extend([ - '-lines=%s:%s' % (start_line, start_line+line_count-1) - for start_line, line_count in line_ranges]) - try: - clang_format = subprocess.Popen(clang_format_cmd, stdin=subprocess.PIPE, - stdout=subprocess.PIPE) - except OSError as e: - if e.errno == errno.ENOENT: - die('cannot find executable "%s"' % binary) - else: - raise - clang_format.stdin.close() - hash_object_cmd = ['git', 'hash-object', '-w', '--path='+filename, '--stdin'] - hash_object = subprocess.Popen(hash_object_cmd, stdin=clang_format.stdout, - stdout=subprocess.PIPE) - clang_format.stdout.close() - stdout = hash_object.communicate()[0] - if hash_object.returncode != 0: - die('`%s` failed' % ' '.join(hash_object_cmd)) - if clang_format.wait() != 0: - die('`%s` failed' % ' '.join(clang_format_cmd)) - return stdout.rstrip('\r\n') - - -@contextlib.contextmanager -def temporary_index_file(tree=None): - """Context manager for setting GIT_INDEX_FILE to a temporary file and deleting - the file afterward.""" - index_path = create_temporary_index(tree) - old_index_path = os.environ.get('GIT_INDEX_FILE') - os.environ['GIT_INDEX_FILE'] = index_path - try: - yield - finally: - if old_index_path is None: - del os.environ['GIT_INDEX_FILE'] - else: - os.environ['GIT_INDEX_FILE'] = old_index_path - os.remove(index_path) - - -def create_temporary_index(tree=None): - """Create a temporary index file and return the created file's path. - If `tree` is not None, use that as the tree to read in. Otherwise, an - empty index is created.""" - gitdir = run('git', 'rev-parse', '--git-dir') - path = os.path.join(gitdir, temp_index_basename) - if tree is None: - tree = '--empty' - run('git', 'read-tree', '--index-output='+path, tree) - return path - - -def print_diff(old_tree, new_tree): - """Print the diff between the two trees to stdout.""" - # We use the porcelain 'diff' and not plumbing 'diff-tree' because the output - # is expected to be viewed by the user, and only the former does nice things - # like color and pagination. - subprocess.check_call(['git', 'diff', old_tree, new_tree, '--']) - - -def apply_changes(old_tree, new_tree, force=False, patch_mode=False): - """Apply the changes in `new_tree` to the working directory. - Bails if there are local changes in those files and not `force`. If - `patch_mode`, runs `git checkout --patch` to select hunks interactively.""" - changed_files = run('git', 'diff-tree', '-r', '-z', '--name-only', old_tree, - new_tree).rstrip('\0').split('\0') - if not force: - unstaged_files = run('git', 'diff-files', '--name-status', *changed_files) - if unstaged_files: - print >>sys.stderr, ('The following files would be modified but ' - 'have unstaged changes:') - print >>sys.stderr, unstaged_files - print >>sys.stderr, 'Please commit, stage, or stash them first.' - sys.exit(2) - if patch_mode: - # In patch mode, we could just as well create an index from the new tree - # and checkout from that, but then the user will be presented with a - # message saying "Discard ... from worktree". Instead, we use the old - # tree as the index and checkout from new_tree, which gives the slightly - # better message, "Apply ... to index and worktree". This is not quite - # right, since it won't be applied to the user's index, but oh well. - with temporary_index_file(old_tree): - subprocess.check_call(['git', 'checkout', '--patch', new_tree]) - index_tree = old_tree - else: - with temporary_index_file(new_tree): - run('git', 'checkout-index', '-a', '-f') - return changed_files - - -def run(*args, **kwargs): - stdin = kwargs.pop('stdin', '') - verbose = kwargs.pop('verbose', True) - strip = kwargs.pop('strip', True) - for name in kwargs: - raise TypeError("run() got an unexpected keyword argument '%s'" % name) - p = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE, - stdin=subprocess.PIPE) - stdout, stderr = p.communicate(input=stdin) - if p.returncode == 0: - if stderr: - if verbose: - print >>sys.stderr, '`%s` printed to stderr:' % ' '.join(args) - print >>sys.stderr, stderr.rstrip() - if strip: - stdout = stdout.rstrip('\r\n') - return stdout - if verbose: - print >>sys.stderr, '`%s` returned %s' % (' '.join(args), p.returncode) - if stderr: - print >>sys.stderr, stderr.rstrip() - sys.exit(2) - - -def die(message): - print >>sys.stderr, 'error:', message - sys.exit(2) - - -if __name__ == '__main__': - main() diff --git a/LICENSE b/LICENSE deleted file mode 100644 index 8dada3eda..000000000 --- a/LICENSE +++ /dev/null @@ -1,201 +0,0 @@ - Apache License - Version 2.0, January 2004 - http://www.apache.org/licenses/ - - TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION - - 1. Definitions. - - "License" shall mean the terms and conditions for use, reproduction, - and distribution as defined by Sections 1 through 9 of this document. - - "Licensor" shall mean the copyright owner or entity authorized by - the copyright owner that is granting the License. - - "Legal Entity" shall mean the union of the acting entity and all - other entities that control, are controlled by, or are under common - control with that entity. For the purposes of this definition, - "control" means (i) the power, direct or indirect, to cause the - direction or management of such entity, whether by contract or - otherwise, or (ii) ownership of fifty percent (50%) or more of the - outstanding shares, or (iii) beneficial ownership of such entity. - - "You" (or "Your") shall mean an individual or Legal Entity - exercising permissions granted by this License. - - "Source" form shall mean the preferred form for making modifications, - including but not limited to software source code, documentation - source, and configuration files. - - "Object" form shall mean any form resulting from mechanical - transformation or translation of a Source form, including but - not limited to compiled object code, generated documentation, - and conversions to other media types. - - "Work" shall mean the work of authorship, whether in Source or - Object form, made available under the License, as indicated by a - copyright notice that is included in or attached to the work - (an example is provided in the Appendix below). - - "Derivative Works" shall mean any work, whether in Source or Object - form, that is based on (or derived from) the Work and for which the - editorial revisions, annotations, elaborations, or other modifications - represent, as a whole, an original work of authorship. For the purposes - of this License, Derivative Works shall not include works that remain - separable from, or merely link (or bind by name) to the interfaces of, - the Work and Derivative Works thereof. - - "Contribution" shall mean any work of authorship, including - the original version of the Work and any modifications or additions - to that Work or Derivative Works thereof, that is intentionally - submitted to Licensor for inclusion in the Work by the copyright owner - or by an individual or Legal Entity authorized to submit on behalf of - the copyright owner. For the purposes of this definition, "submitted" - means any form of electronic, verbal, or written communication sent - to the Licensor or its representatives, including but not limited to - communication on electronic mailing lists, source code control systems, - and issue tracking systems that are managed by, or on behalf of, the - Licensor for the purpose of discussing and improving the Work, but - excluding communication that is conspicuously marked or otherwise - designated in writing by the copyright owner as "Not a Contribution." - - "Contributor" shall mean Licensor and any individual or Legal Entity - on behalf of whom a Contribution has been received by Licensor and - subsequently incorporated within the Work. - - 2. Grant of Copyright License. Subject to the terms and conditions of - this License, each Contributor hereby grants to You a perpetual, - worldwide, non-exclusive, no-charge, royalty-free, irrevocable - copyright license to reproduce, prepare Derivative Works of, - publicly display, publicly perform, sublicense, and distribute the - Work and such Derivative Works in Source or Object form. - - 3. Grant of Patent License. Subject to the terms and conditions of - this License, each Contributor hereby grants to You a perpetual, - worldwide, non-exclusive, no-charge, royalty-free, irrevocable - (except as stated in this section) patent license to make, have made, - use, offer to sell, sell, import, and otherwise transfer the Work, - where such license applies only to those patent claims licensable - by such Contributor that are necessarily infringed by their - Contribution(s) alone or by combination of their Contribution(s) - with the Work to which such Contribution(s) was submitted. If You - institute patent litigation against any entity (including a - cross-claim or counterclaim in a lawsuit) alleging that the Work - or a Contribution incorporated within the Work constitutes direct - or contributory patent infringement, then any patent licenses - granted to You under this License for that Work shall terminate - as of the date such litigation is filed. - - 4. Redistribution. You may reproduce and distribute copies of the - Work or Derivative Works thereof in any medium, with or without - modifications, and in Source or Object form, provided that You - meet the following conditions: - - (a) You must give any other recipients of the Work or - Derivative Works a copy of this License; and - - (b) You must cause any modified files to carry prominent notices - stating that You changed the files; and - - (c) You must retain, in the Source form of any Derivative Works - that You distribute, all copyright, patent, trademark, and - attribution notices from the Source form of the Work, - excluding those notices that do not pertain to any part of - the Derivative Works; and - - (d) If the Work includes a "NOTICE" text file as part of its - distribution, then any Derivative Works that You distribute must - include a readable copy of the attribution notices contained - within such NOTICE file, excluding those notices that do not - pertain to any part of the Derivative Works, in at least one - of the following places: within a NOTICE text file distributed - as part of the Derivative Works; within the Source form or - documentation, if provided along with the Derivative Works; or, - within a display generated by the Derivative Works, if and - wherever such third-party notices normally appear. The contents - of the NOTICE file are for informational purposes only and - do not modify the License. You may add Your own attribution - notices within Derivative Works that You distribute, alongside - or as an addendum to the NOTICE text from the Work, provided - that such additional attribution notices cannot be construed - as modifying the License. - - You may add Your own copyright statement to Your modifications and - may provide additional or different license terms and conditions - for use, reproduction, or distribution of Your modifications, or - for any such Derivative Works as a whole, provided Your use, - reproduction, and distribution of the Work otherwise complies with - the conditions stated in this License. - - 5. Submission of Contributions. Unless You explicitly state otherwise, - any Contribution intentionally submitted for inclusion in the Work - by You to the Licensor shall be under the terms and conditions of - this License, without any additional terms or conditions. - Notwithstanding the above, nothing herein shall supersede or modify - the terms of any separate license agreement you may have executed - with Licensor regarding such Contributions. - - 6. Trademarks. This License does not grant permission to use the trade - names, trademarks, service marks, or product names of the Licensor, - except as required for reasonable and customary use in describing the - origin of the Work and reproducing the content of the NOTICE file. - - 7. Disclaimer of Warranty. Unless required by applicable law or - agreed to in writing, Licensor provides the Work (and each - Contributor provides its Contributions) on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or - implied, including, without limitation, any warranties or conditions - of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A - PARTICULAR PURPOSE. You are solely responsible for determining the - appropriateness of using or redistributing the Work and assume any - risks associated with Your exercise of permissions under this License. - - 8. Limitation of Liability. In no event and under no legal theory, - whether in tort (including negligence), contract, or otherwise, - unless required by applicable law (such as deliberate and grossly - negligent acts) or agreed to in writing, shall any Contributor be - liable to You for damages, including any direct, indirect, special, - incidental, or consequential damages of any character arising as a - result of this License or out of the use or inability to use the - Work (including but not limited to damages for loss of goodwill, - work stoppage, computer failure or malfunction, or any and all - other commercial damages or losses), even if such Contributor - has been advised of the possibility of such damages. - - 9. Accepting Warranty or Additional Liability. While redistributing - the Work or Derivative Works thereof, You may choose to offer, - and charge a fee for, acceptance of support, warranty, indemnity, - or other liability obligations and/or rights consistent with this - License. However, in accepting such obligations, You may act only - on Your own behalf and on Your sole responsibility, not on behalf - of any other Contributor, and only if You agree to indemnify, - defend, and hold each Contributor harmless for any liability - incurred by, or claims asserted against, such Contributor by reason - of your accepting any such warranty or additional liability. - - END OF TERMS AND CONDITIONS - - APPENDIX: How to apply the Apache License to your work. - - To apply the Apache License to your work, attach the following - boilerplate notice, with the fields enclosed by brackets "{}" - replaced with your own identifying information. (Don't include - the brackets!) The text should be enclosed in the appropriate - comment syntax for the file format. We also recommend that a - file or class name and description of purpose be included on the - same "printed page" as the copyright notice for easier - identification within third-party archives. - - Copyright {yyyy} {name of copyright owner} - - Licensed under the Apache License, Version 2.0 (the "License"); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. diff --git a/README.md b/README.md deleted file mode 100644 index 92e08a503..000000000 --- a/README.md +++ /dev/null @@ -1,2 +0,0 @@ -# Photon -A local scheduler and node manager for Ray. diff --git a/common b/common deleted file mode 160000 index 535bc8f0b..000000000 --- a/common +++ /dev/null @@ -1 +0,0 @@ -Subproject commit 535bc8f0b8dac8f3ef0b66c3fd8b265ab0e6c787 diff --git a/install-dependencies.sh b/install-dependencies.sh deleted file mode 100755 index f84da1684..000000000 --- a/install-dependencies.sh +++ /dev/null @@ -1,21 +0,0 @@ -#!/usr/bin/env bash - -ROOT_DIR=$(cd "$(dirname "${BASH_SOURCE:-$0}")"; pwd) - -platform="unknown" -unamestr="$(uname)" -if [[ "$unamestr" == "Linux" ]]; then - echo "Platform is linux." - platform="linux" -elif [[ "$unamestr" == "Darwin" ]]; then - echo "Platform is macosx." - platform="macosx" -else - echo "Unrecognized platform." - exit 1 -fi - -if [[ $platform == "linux" ]]; then - sudo apt-get update - sudo apt-get install -y git python-dev -fi diff --git a/setup-env.sh b/setup-env.sh deleted file mode 100755 index 7c4350150..000000000 --- a/setup-env.sh +++ /dev/null @@ -1,5 +0,0 @@ -echo "Adding Photon to PYTHONPATH" 1>&2 - -ROOT_DIR=$(cd "$(dirname "${BASH_SOURCE:-$0}")"; pwd) - -export PYTHONPATH="$ROOT_DIR/lib/python/:$PYTHONPATH" diff --git a/Makefile b/src/photon/Makefile similarity index 100% rename from Makefile rename to src/photon/Makefile diff --git a/build/.gitkeep b/src/photon/build/.gitkeep similarity index 100% rename from build/.gitkeep rename to src/photon/build/.gitkeep diff --git a/lib/python/photon_extension.c b/src/photon/lib/python/photon_extension.c similarity index 100% rename from lib/python/photon_extension.c rename to src/photon/lib/python/photon_extension.c diff --git a/lib/python/setup.py b/src/photon/lib/python/setup.py similarity index 100% rename from lib/python/setup.py rename to src/photon/lib/python/setup.py diff --git a/photon.h b/src/photon/photon.h similarity index 100% rename from photon.h rename to src/photon/photon.h diff --git a/photon_algorithm.c b/src/photon/photon_algorithm.c similarity index 100% rename from photon_algorithm.c rename to src/photon/photon_algorithm.c diff --git a/photon_algorithm.h b/src/photon/photon_algorithm.h similarity index 100% rename from photon_algorithm.h rename to src/photon/photon_algorithm.h diff --git a/photon_client.c b/src/photon/photon_client.c similarity index 100% rename from photon_client.c rename to src/photon/photon_client.c diff --git a/photon_client.h b/src/photon/photon_client.h similarity index 100% rename from photon_client.h rename to src/photon/photon_client.h diff --git a/photon_scheduler.c b/src/photon/photon_scheduler.c similarity index 100% rename from photon_scheduler.c rename to src/photon/photon_scheduler.c diff --git a/photon_scheduler.h b/src/photon/photon_scheduler.h similarity index 100% rename from photon_scheduler.h rename to src/photon/photon_scheduler.h diff --git a/test/test.py b/src/photon/test/test.py similarity index 100% rename from test/test.py rename to src/photon/test/test.py