From a35de3b2872deb8500e42a1ba0a2e24dff20aa87 Mon Sep 17 00:00:00 2001 From: Philipp Moritz Date: Sat, 10 Sep 2016 19:42:08 -0700 Subject: [PATCH 01/36] Initial commit --- .gitignore | 33 +++++++++ LICENSE | 201 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 234 insertions(+) create mode 100644 .gitignore create mode 100644 LICENSE diff --git a/.gitignore b/.gitignore new file mode 100644 index 000000000..f805e810e --- /dev/null +++ b/.gitignore @@ -0,0 +1,33 @@ +# Object files +*.o +*.ko +*.obj +*.elf + +# Precompiled Headers +*.gch +*.pch + +# Libraries +*.lib +*.a +*.la +*.lo + +# Shared objects (inc. Windows DLLs) +*.dll +*.so +*.so.* +*.dylib + +# Executables +*.exe +*.out +*.app +*.i*86 +*.x86_64 +*.hex + +# Debug files +*.dSYM/ +*.su diff --git a/LICENSE b/LICENSE new file mode 100644 index 000000000..8dada3eda --- /dev/null +++ b/LICENSE @@ -0,0 +1,201 @@ + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "{}" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright {yyyy} {name of copyright owner} + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. From 7d629d4e489e1ab10bd85b5fc94e861f4ae159cc Mon Sep 17 00:00:00 2001 From: Philipp Moritz Date: Tue, 13 Sep 2016 18:54:26 -0700 Subject: [PATCH 02/36] Adding object table (#1) * code for maintaining the object table * Makefile fix * Clone git submodules. * directory -> object_table * Fix Makefile and remove unnecessary files. * Fix formatting. * make code more generic --- .clang-format | 6 + .gitignore | 2 + .gitmodules | 3 + .travis.yml | 26 + .travis/check-git-clang-format-output.sh | 18 + .travis/git-clang-format | 476 ++++++++++ Makefile | 24 + build/.gitkeep | 0 common.c | 15 + common.h | 29 + event_loop.c | 92 ++ event_loop.h | 38 + state/db.h | 29 + state/object_table.h | 15 + state/redis.c | 188 ++++ state/redis.h | 26 + test/db_tests.c | 69 ++ thirdparty/build-redis.sh | 4 + thirdparty/greatest.h | 1023 ++++++++++++++++++++++ thirdparty/hiredis | 1 + thirdparty/utarray.h | 238 +++++ 21 files changed, 2322 insertions(+) create mode 100644 .clang-format create mode 100644 .gitmodules create mode 100644 .travis.yml create mode 100755 .travis/check-git-clang-format-output.sh create mode 100755 .travis/git-clang-format create mode 100644 Makefile create mode 100644 build/.gitkeep create mode 100644 common.c create mode 100644 common.h create mode 100644 event_loop.c create mode 100644 event_loop.h create mode 100644 state/db.h create mode 100644 state/object_table.h create mode 100644 state/redis.c create mode 100644 state/redis.h create mode 100644 test/db_tests.c create mode 100644 thirdparty/build-redis.sh create mode 100644 thirdparty/greatest.h create mode 160000 thirdparty/hiredis create mode 100644 thirdparty/utarray.h diff --git a/.clang-format b/.clang-format new file mode 100644 index 000000000..90d254290 --- /dev/null +++ b/.clang-format @@ -0,0 +1,6 @@ +BasedOnStyle: Chromium +DerivePointerAlignment: true +IndentCaseLabels: false +PointerAlignment: Right +SpaceAfterCStyleCast: true + diff --git a/.gitignore b/.gitignore index f805e810e..2a07abca4 100644 --- a/.gitignore +++ b/.gitignore @@ -1,3 +1,5 @@ +*~ + # Object files *.o *.ko diff --git a/.gitmodules b/.gitmodules new file mode 100644 index 000000000..4026f8268 --- /dev/null +++ b/.gitmodules @@ -0,0 +1,3 @@ +[submodule "thirdparty/hiredis"] + path = thirdparty/hiredis + url = https://github.com/redis/hiredis diff --git a/.travis.yml b/.travis.yml new file mode 100644 index 000000000..220df4b86 --- /dev/null +++ b/.travis.yml @@ -0,0 +1,26 @@ +sudo: required + +language: generic + +matrix: + include: + - os: linux + dist: trusty + - os: osx + osx_image: xcode7 + - os: linux + dist: trusty + env: LINT=1 + before_install: + # In case we ever want to use a different version of clang-format: + #- wget -O - http://apt.llvm.org/llvm-snapshot.gpg.key | sudo apt-key add - + #- echo "deb http://apt.llvm.org/trusty/ llvm-toolchain-trusty main" | sudo tee -a /etc/apt/sources.list > /dev/null + - sudo apt-get update -qq + - sudo apt-get install -qq clang-format-3.8 + install: [] + script: + - .travis/check-git-clang-format-output.sh + +install: + - make + - make test diff --git a/.travis/check-git-clang-format-output.sh b/.travis/check-git-clang-format-output.sh new file mode 100755 index 000000000..d71f78357 --- /dev/null +++ b/.travis/check-git-clang-format-output.sh @@ -0,0 +1,18 @@ +#!/bin/bash +if [ "$TRAVIS_PULL_REQUEST" == "false" ] ; then + # Not in a pull request, so compare against parent commit + base_commit="HEAD^" + echo "Running clang-format against parent commit $(git rev-parse $base_commit)" +else + base_commit="$TRAVIS_BRANCH" + echo "Running clang-format against branch $base_commit, with hash $(git rev-parse $base_commit)" +fi +output="$(.travis/git-clang-format --binary clang-format-3.8 --commit $base_commit --diff --exclude ^thirdparty/)" +if [ "$output" == "no modified files to format" ] || [ "$output" == "clang-format did not modify any files" ] ; then + echo "clang-format passed." + exit 0 +else + echo "clang-format failed:" + echo "$output" + exit 1 +fi diff --git a/.travis/git-clang-format b/.travis/git-clang-format new file mode 100755 index 000000000..37b352835 --- /dev/null +++ b/.travis/git-clang-format @@ -0,0 +1,476 @@ +#!/usr/bin/env python +# +#===- git-clang-format - ClangFormat Git Integration ---------*- python -*--===# +# +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. +# +#===------------------------------------------------------------------------===# + +r""" +clang-format git integration +============================ + +This file provides a clang-format integration for git. Put it somewhere in your +path and ensure that it is executable. Then, "git clang-format" will invoke +clang-format on the changes in current files or a specific commit. + +For further details, run: +git clang-format -h + +Requires Python 2.7 +""" + +import argparse +import collections +import contextlib +import errno +import os +import re +import subprocess +import sys + +usage = 'git clang-format [OPTIONS] [] [--] [...]' + +desc = ''' +Run clang-format on all lines that differ between the working directory +and , which defaults to HEAD. Changes are only applied to the working +directory. +The following git-config settings set the default of the corresponding option: + clangFormat.binary + clangFormat.commit + clangFormat.extension + clangFormat.style +''' + +# Name of the temporary index file in which save the output of clang-format. +# This file is created within the .git directory. +temp_index_basename = 'clang-format-index' + + +Range = collections.namedtuple('Range', 'start, count') + + +def main(): + config = load_git_config() + + # In order to keep '--' yet allow options after positionals, we need to + # check for '--' ourselves. (Setting nargs='*' throws away the '--', while + # nargs=argparse.REMAINDER disallows options after positionals.) + argv = sys.argv[1:] + try: + idx = argv.index('--') + except ValueError: + dash_dash = [] + else: + dash_dash = argv[idx:] + argv = argv[:idx] + + default_extensions = ','.join([ + # From clang/lib/Frontend/FrontendOptions.cpp, all lower case + 'c', 'h', # C + 'm', # ObjC + 'mm', # ObjC++ + 'cc', 'cp', 'cpp', 'c++', 'cxx', 'hpp', # C++ + # Other languages that clang-format supports + 'proto', 'protodevel', # Protocol Buffers + 'js', # JavaScript + 'ts', # TypeScript + ]) + + p = argparse.ArgumentParser( + usage=usage, formatter_class=argparse.RawDescriptionHelpFormatter, + description=desc) + p.add_argument('--binary', + default=config.get('clangformat.binary', 'clang-format'), + help='path to clang-format'), + p.add_argument('--commit', + default=config.get('clangformat.commit', 'HEAD'), + help='default commit to use if none is specified'), + p.add_argument('--diff', action='store_true', + help='print a diff instead of applying the changes') + p.add_argument('--extensions', + default=config.get('clangformat.extensions', + default_extensions), + help=('comma-separated list of file extensions to format, ' + 'excluding the period and case-insensitive')), + p.add_argument('--exclude', help='Exclude files matching this regex.') + p.add_argument('-f', '--force', action='store_true', + help='allow changes to unstaged files') + p.add_argument('-p', '--patch', action='store_true', + help='select hunks interactively') + p.add_argument('-q', '--quiet', action='count', default=0, + help='print less information') + p.add_argument('--style', + default=config.get('clangformat.style', None), + help='passed to clang-format'), + p.add_argument('-v', '--verbose', action='count', default=0, + help='print extra information') + # We gather all the remaining positional arguments into 'args' since we need + # to use some heuristics to determine whether or not was present. + # However, to print pretty messages, we make use of metavar and help. + p.add_argument('args', nargs='*', metavar='', + help='revision from which to compute the diff') + p.add_argument('ignored', nargs='*', metavar='...', + help='if specified, only consider differences in these files') + opts = p.parse_args(argv) + + opts.verbose -= opts.quiet + del opts.quiet + + commit, files = interpret_args(opts.args, dash_dash, opts.commit) + changed_lines = compute_diff_and_extract_lines(commit, files) + if opts.verbose >= 1: + ignored_files = set(changed_lines) + filter_by_extension(changed_lines, opts.extensions.lower().split(',')) + if opts.exclude: + for filename in changed_lines.keys(): + if re.match(opts.exclude, filename): + del changed_lines[filename] + if opts.verbose >= 1: + ignored_files.difference_update(changed_lines) + if ignored_files: + print 'Ignoring changes in the following files:' + for filename in ignored_files: + print ' ', filename + if changed_lines: + print 'Running clang-format on the following files:' + for filename in changed_lines: + print ' ', filename + if not changed_lines: + print 'no modified files to format' + return + # The computed diff outputs absolute paths, so we must cd before accessing + # those files. + cd_to_toplevel() + old_tree = create_tree_from_workdir(changed_lines) + new_tree = run_clang_format_and_save_to_tree(changed_lines, + binary=opts.binary, + style=opts.style) + if opts.verbose >= 1: + print 'old tree:', old_tree + print 'new tree:', new_tree + if old_tree == new_tree: + if opts.verbose >= 0: + print 'clang-format did not modify any files' + elif opts.diff: + print_diff(old_tree, new_tree) + else: + changed_files = apply_changes(old_tree, new_tree, force=opts.force, + patch_mode=opts.patch) + if (opts.verbose >= 0 and not opts.patch) or opts.verbose >= 1: + print 'changed files:' + for filename in changed_files: + print ' ', filename + + +def load_git_config(non_string_options=None): + """Return the git configuration as a dictionary. + All options are assumed to be strings unless in `non_string_options`, in which + is a dictionary mapping option name (in lower case) to either "--bool" or + "--int".""" + if non_string_options is None: + non_string_options = {} + out = {} + for entry in run('git', 'config', '--list', '--null').split('\0'): + if entry: + name, value = entry.split('\n', 1) + if name in non_string_options: + value = run('git', 'config', non_string_options[name], name) + out[name] = value + return out + + +def interpret_args(args, dash_dash, default_commit): + """Interpret `args` as "[commit] [--] [files...]" and return (commit, files). + It is assumed that "--" and everything that follows has been removed from + args and placed in `dash_dash`. + If "--" is present (i.e., `dash_dash` is non-empty), the argument to its + left (if present) is taken as commit. Otherwise, the first argument is + checked if it is a commit or a file. If commit is not given, + `default_commit` is used.""" + if dash_dash: + if len(args) == 0: + commit = default_commit + elif len(args) > 1: + die('at most one commit allowed; %d given' % len(args)) + else: + commit = args[0] + object_type = get_object_type(commit) + if object_type not in ('commit', 'tag'): + if object_type is None: + die("'%s' is not a commit" % commit) + else: + die("'%s' is a %s, but a commit was expected" % (commit, object_type)) + files = dash_dash[1:] + elif args: + if disambiguate_revision(args[0]): + commit = args[0] + files = args[1:] + else: + commit = default_commit + files = args + else: + commit = default_commit + files = [] + return commit, files + + +def disambiguate_revision(value): + """Returns True if `value` is a revision, False if it is a file, or dies.""" + # If `value` is ambiguous (neither a commit nor a file), the following + # command will die with an appropriate error message. + run('git', 'rev-parse', value, verbose=False) + object_type = get_object_type(value) + if object_type is None: + return False + if object_type in ('commit', 'tag'): + return True + die('`%s` is a %s, but a commit or filename was expected' % + (value, object_type)) + + +def get_object_type(value): + """Returns a string description of an object's type, or None if it is not + a valid git object.""" + cmd = ['git', 'cat-file', '-t', value] + p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE) + stdout, stderr = p.communicate() + if p.returncode != 0: + return None + return stdout.strip() + + +def compute_diff_and_extract_lines(commit, files): + """Calls compute_diff() followed by extract_lines().""" + diff_process = compute_diff(commit, files) + changed_lines = extract_lines(diff_process.stdout) + diff_process.stdout.close() + diff_process.wait() + if diff_process.returncode != 0: + # Assume error was already printed to stderr. + sys.exit(2) + return changed_lines + + +def compute_diff(commit, files): + """Return a subprocess object producing the diff from `commit`. + The return value's `stdin` file object will produce a patch with the + differences between the working directory and `commit`, filtered on `files` + (if non-empty). Zero context lines are used in the patch.""" + cmd = ['git', 'diff-index', '-p', '-U0', commit, '--'] + cmd.extend(files) + p = subprocess.Popen(cmd, stdin=subprocess.PIPE, stdout=subprocess.PIPE) + p.stdin.close() + return p + + +def extract_lines(patch_file): + """Extract the changed lines in `patch_file`. + The return value is a dictionary mapping filename to a list of (start_line, + line_count) pairs. + The input must have been produced with ``-U0``, meaning unidiff format with + zero lines of context. The return value is a dict mapping filename to a + list of line `Range`s.""" + matches = {} + for line in patch_file: + match = re.search(r'^\+\+\+\ [^/]+/(.*)', line) + if match: + filename = match.group(1).rstrip('\r\n') + match = re.search(r'^@@ -[0-9,]+ \+(\d+)(,(\d+))?', line) + if match: + start_line = int(match.group(1)) + line_count = 1 + if match.group(3): + line_count = int(match.group(3)) + if line_count > 0: + matches.setdefault(filename, []).append(Range(start_line, line_count)) + return matches + + +def filter_by_extension(dictionary, allowed_extensions): + """Delete every key in `dictionary` that doesn't have an allowed extension. + `allowed_extensions` must be a collection of lowercase file extensions, + excluding the period.""" + allowed_extensions = frozenset(allowed_extensions) + for filename in dictionary.keys(): + base_ext = filename.rsplit('.', 1) + if len(base_ext) == 1 or base_ext[1].lower() not in allowed_extensions: + del dictionary[filename] + + +def cd_to_toplevel(): + """Change to the top level of the git repository.""" + toplevel = run('git', 'rev-parse', '--show-toplevel') + os.chdir(toplevel) + + +def create_tree_from_workdir(filenames): + """Create a new git tree with the given files from the working directory. + Returns the object ID (SHA-1) of the created tree.""" + return create_tree(filenames, '--stdin') + + +def run_clang_format_and_save_to_tree(changed_lines, binary='clang-format', + style=None): + """Run clang-format on each file and save the result to a git tree. + Returns the object ID (SHA-1) of the created tree.""" + def index_info_generator(): + for filename, line_ranges in changed_lines.iteritems(): + mode = oct(os.stat(filename).st_mode) + blob_id = clang_format_to_blob(filename, line_ranges, binary=binary, + style=style) + yield '%s %s\t%s' % (mode, blob_id, filename) + return create_tree(index_info_generator(), '--index-info') + + +def create_tree(input_lines, mode): + """Create a tree object from the given input. + If mode is '--stdin', it must be a list of filenames. If mode is + '--index-info' is must be a list of values suitable for "git update-index + --index-info", such as " ". Any other mode + is invalid.""" + assert mode in ('--stdin', '--index-info') + cmd = ['git', 'update-index', '--add', '-z', mode] + with temporary_index_file(): + p = subprocess.Popen(cmd, stdin=subprocess.PIPE) + for line in input_lines: + p.stdin.write('%s\0' % line) + p.stdin.close() + if p.wait() != 0: + die('`%s` failed' % ' '.join(cmd)) + tree_id = run('git', 'write-tree') + return tree_id + + +def clang_format_to_blob(filename, line_ranges, binary='clang-format', + style=None): + """Run clang-format on the given file and save the result to a git blob. + Returns the object ID (SHA-1) of the created blob.""" + clang_format_cmd = [binary, filename] + if style: + clang_format_cmd.extend(['-style='+style]) + clang_format_cmd.extend([ + '-lines=%s:%s' % (start_line, start_line+line_count-1) + for start_line, line_count in line_ranges]) + try: + clang_format = subprocess.Popen(clang_format_cmd, stdin=subprocess.PIPE, + stdout=subprocess.PIPE) + except OSError as e: + if e.errno == errno.ENOENT: + die('cannot find executable "%s"' % binary) + else: + raise + clang_format.stdin.close() + hash_object_cmd = ['git', 'hash-object', '-w', '--path='+filename, '--stdin'] + hash_object = subprocess.Popen(hash_object_cmd, stdin=clang_format.stdout, + stdout=subprocess.PIPE) + clang_format.stdout.close() + stdout = hash_object.communicate()[0] + if hash_object.returncode != 0: + die('`%s` failed' % ' '.join(hash_object_cmd)) + if clang_format.wait() != 0: + die('`%s` failed' % ' '.join(clang_format_cmd)) + return stdout.rstrip('\r\n') + + +@contextlib.contextmanager +def temporary_index_file(tree=None): + """Context manager for setting GIT_INDEX_FILE to a temporary file and deleting + the file afterward.""" + index_path = create_temporary_index(tree) + old_index_path = os.environ.get('GIT_INDEX_FILE') + os.environ['GIT_INDEX_FILE'] = index_path + try: + yield + finally: + if old_index_path is None: + del os.environ['GIT_INDEX_FILE'] + else: + os.environ['GIT_INDEX_FILE'] = old_index_path + os.remove(index_path) + + +def create_temporary_index(tree=None): + """Create a temporary index file and return the created file's path. + If `tree` is not None, use that as the tree to read in. Otherwise, an + empty index is created.""" + gitdir = run('git', 'rev-parse', '--git-dir') + path = os.path.join(gitdir, temp_index_basename) + if tree is None: + tree = '--empty' + run('git', 'read-tree', '--index-output='+path, tree) + return path + + +def print_diff(old_tree, new_tree): + """Print the diff between the two trees to stdout.""" + # We use the porcelain 'diff' and not plumbing 'diff-tree' because the output + # is expected to be viewed by the user, and only the former does nice things + # like color and pagination. + subprocess.check_call(['git', 'diff', old_tree, new_tree, '--']) + + +def apply_changes(old_tree, new_tree, force=False, patch_mode=False): + """Apply the changes in `new_tree` to the working directory. + Bails if there are local changes in those files and not `force`. If + `patch_mode`, runs `git checkout --patch` to select hunks interactively.""" + changed_files = run('git', 'diff-tree', '-r', '-z', '--name-only', old_tree, + new_tree).rstrip('\0').split('\0') + if not force: + unstaged_files = run('git', 'diff-files', '--name-status', *changed_files) + if unstaged_files: + print >>sys.stderr, ('The following files would be modified but ' + 'have unstaged changes:') + print >>sys.stderr, unstaged_files + print >>sys.stderr, 'Please commit, stage, or stash them first.' + sys.exit(2) + if patch_mode: + # In patch mode, we could just as well create an index from the new tree + # and checkout from that, but then the user will be presented with a + # message saying "Discard ... from worktree". Instead, we use the old + # tree as the index and checkout from new_tree, which gives the slightly + # better message, "Apply ... to index and worktree". This is not quite + # right, since it won't be applied to the user's index, but oh well. + with temporary_index_file(old_tree): + subprocess.check_call(['git', 'checkout', '--patch', new_tree]) + index_tree = old_tree + else: + with temporary_index_file(new_tree): + run('git', 'checkout-index', '-a', '-f') + return changed_files + + +def run(*args, **kwargs): + stdin = kwargs.pop('stdin', '') + verbose = kwargs.pop('verbose', True) + strip = kwargs.pop('strip', True) + for name in kwargs: + raise TypeError("run() got an unexpected keyword argument '%s'" % name) + p = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE, + stdin=subprocess.PIPE) + stdout, stderr = p.communicate(input=stdin) + if p.returncode == 0: + if stderr: + if verbose: + print >>sys.stderr, '`%s` printed to stderr:' % ' '.join(args) + print >>sys.stderr, stderr.rstrip() + if strip: + stdout = stdout.rstrip('\r\n') + return stdout + if verbose: + print >>sys.stderr, '`%s` returned %s' % (' '.join(args), p.returncode) + if stderr: + print >>sys.stderr, stderr.rstrip() + sys.exit(2) + + +def die(message): + print >>sys.stderr, 'error:', message + sys.exit(2) + + +if __name__ == '__main__': + main() diff --git a/Makefile b/Makefile new file mode 100644 index 000000000..fd9da0b97 --- /dev/null +++ b/Makefile @@ -0,0 +1,24 @@ +CC = gcc +CFLAGS = -g -Wall --std=c99 -D_XOPEN_SOURCE=500 -D_POSIX_C_SOURCE=200809L +BUILD = build + +CFLAGS += -Wmissing-prototypes +CFLAGS += -Wstrict-prototypes +CFLAGS += -Wmissing-declarations + +$(BUILD)/db_tests: hiredis test/db_tests.c thirdparty/greatest.h event_loop.c state/redis.c common.c + $(CC) -o $@ test/db_tests.c event_loop.c state/redis.c common.c thirdparty/hiredis/libhiredis.a $(CFLAGS) -I. -Ithirdparty + +clean: + rm -r $(BUILD)/* + +redis: + cd thirdparty ; bash ./build-redis.sh + +hiredis: + git submodule update --init --recursive -- "thirdparty/hiredis" ; cd thirdparty/hiredis ; make + +test: hiredis redis $(BUILD)/db_tests FORCE + ./thirdparty/redis-3.2.3/src/redis-server & sleep 1s ; ./build/db_tests + +FORCE: diff --git a/build/.gitkeep b/build/.gitkeep new file mode 100644 index 000000000..e69de29bb diff --git a/common.c b/common.c new file mode 100644 index 000000000..e227eb16e --- /dev/null +++ b/common.c @@ -0,0 +1,15 @@ +#include "common.h" + +char *sha1_to_hex(const unsigned char *sha1, char *buffer) { + static const char hex[] = "0123456789abcdef"; + char *buf = buffer; + + for (int i = 0; i < UNIQUE_ID_SIZE; i++) { + unsigned int val = *sha1++; + *buf++ = hex[val >> 4]; + *buf++ = hex[val & 0xf]; + } + *buf = '\0'; + + return buffer; +} diff --git a/common.h b/common.h new file mode 100644 index 000000000..61abc7b50 --- /dev/null +++ b/common.h @@ -0,0 +1,29 @@ +#ifndef COMMON_H +#define COMMON_H + +#include + +#ifdef NDEBUG +#define LOG_DEBUG(M, ...) +#else +#define LOG_DEBUG(M, ...) \ + fprintf(stderr, "[DEBUG] (%s:%d) " M "\n", __FILE__, __LINE__, ##__VA_ARGS__) +#endif + +#define LOG_ERR(M, ...) \ + fprintf(stderr, "[ERROR] (%s:%d: errno: %s) " M "\n", __FILE__, __LINE__, \ + errno == 0 ? "None" : strerror(errno), ##__VA_ARGS__) + +#define LOG_INFO(M, ...) \ + fprintf(stderr, "[INFO] (%s:%d) " M "\n", __FILE__, __LINE__, ##__VA_ARGS__) + +#define UNIQUE_ID_SIZE 20 + +typedef struct { unsigned char id[UNIQUE_ID_SIZE]; } unique_id; + +/* Convert a 20 byte sha1 hash to a hexdecimal string. This function assumes + * that buffer points to an already allocated char array of size 2 * + * UNIQUE_ID_SIZE + 1 */ +char *sha1_to_hex(const unsigned char *sha1, char *buffer); + +#endif diff --git a/event_loop.c b/event_loop.c new file mode 100644 index 000000000..d7169f5d8 --- /dev/null +++ b/event_loop.c @@ -0,0 +1,92 @@ +#include "event_loop.h" + +#include +#include + +UT_icd item_icd = {sizeof(event_loop_item), NULL, NULL, NULL}; +UT_icd poll_icd = {sizeof(struct pollfd), NULL, NULL, NULL}; + +/* Initializes the event loop. + * This function needs to be called before any other event loop function. */ +void event_loop_init(event_loop *loop) { + utarray_new(loop->items, &item_icd); + utarray_new(loop->waiting, &poll_icd); +} + +/* Add a new file descriptor fd to the event loop. + * This function sets a user defined type and id for the file descriptor + * which can be queried using event_loop_type and event_loop_id. The parameter + * events is the same as in http://linux.die.net/man/2/poll. + * Returns the index of the item in the event loop. */ +int64_t event_loop_attach(event_loop *loop, + int type, + void *data, + int fd, + int events) { + assert(utarray_len(loop->items) == utarray_len(loop->waiting)); + int64_t index = utarray_len(loop->items); + event_loop_item item = {.type = type, .data = data}; + utarray_push_back(loop->items, &item); + struct pollfd waiting = {.fd = fd, .events = events}; + utarray_push_back(loop->waiting, &waiting); + return index; +} + +/* Detach a file descriptor from the event loop. + * This invalidates all other indices into the event loop items, but leaves + * the ids of the event loop items valid. */ +void event_loop_detach(event_loop *loop, int64_t index, int shall_close) { + struct pollfd *waiting_item = + (struct pollfd *) utarray_eltptr(loop->waiting, index); + struct pollfd *waiting_back = (struct pollfd *) utarray_back(loop->waiting); + if (shall_close) { + close(waiting_item->fd); + } + *waiting_item = *waiting_back; + utarray_pop_back(loop->waiting); + + event_loop_item *items_item = + (event_loop_item *) utarray_eltptr(loop->items, index); + event_loop_item *items_back = (event_loop_item *) utarray_back(loop->items); + *items_item = *items_back; + utarray_pop_back(loop->items); +} + +/* Poll the file descriptors associated to this event loop. + * See http://linux.die.net/man/2/poll */ +int event_loop_poll(event_loop *loop) { + return poll((struct pollfd *) utarray_front(loop->waiting), + utarray_len(loop->waiting), -1); +} + +/* Get the total number of file descriptors participating in the event loop. */ +int64_t event_loop_size(event_loop *loop) { + return utarray_len(loop->waiting); +} + +/* Get the pollfd structure associated to a file descriptor participating in the + * event loop. */ +struct pollfd *event_loop_get(event_loop *loop, int64_t index) { + return (struct pollfd *) utarray_eltptr(loop->waiting, index); +} + +/* Set the data connection information for participant in the event loop. */ +void event_loop_set_data(event_loop *loop, int64_t index, void *data) { + event_loop_item *item = + (event_loop_item *) utarray_eltptr(loop->items, index); + item->data = data; +} + +/* Get the data connection information for participant in the event loop. */ +void *event_loop_get_data(event_loop *loop, int64_t index) { + event_loop_item *item = + (event_loop_item *) utarray_eltptr(loop->items, index); + return item->data; +} + +/* Free the space associated to the event loop. + * Does not free the event_loop datastructure itself. */ +void event_loop_free(event_loop *loop) { + utarray_free(loop->items); + utarray_free(loop->waiting); +} diff --git a/event_loop.h b/event_loop.h new file mode 100644 index 000000000..0903bb9d4 --- /dev/null +++ b/event_loop.h @@ -0,0 +1,38 @@ +#ifndef EVENT_LOOP_H +#define EVENT_LOOP_H + +#include +#include + +#include "utarray.h" + +typedef struct { + /* The type of connection (e.g. redis, client, manager, data transfer). */ + int type; + /* Data associated with the connection (managed by the user) */ + void *data; +} event_loop_item; + +typedef struct { + /* Array of event_loop_items that hold information for connections. */ + UT_array *items; + /* Array of file descriptors that are waiting, corresponding to items. */ + UT_array *waiting; +} event_loop; + +/* Event loop functions. */ +void event_loop_init(event_loop *loop); +void event_loop_free(event_loop *loop); +int64_t event_loop_attach(event_loop *loop, + int type, + void *data, + int fd, + int events); +void event_loop_detach(event_loop *loop, int64_t index, int shall_close); +int event_loop_poll(event_loop *loop); +int64_t event_loop_size(event_loop *loop); +struct pollfd *event_loop_get(event_loop *loop, int64_t index); +void event_loop_set_data(event_loop *loop, int64_t index, void *data); +void *event_loop_get_data(event_loop *loop, int64_t index); + +#endif diff --git a/state/db.h b/state/db.h new file mode 100644 index 000000000..b586f9acf --- /dev/null +++ b/state/db.h @@ -0,0 +1,29 @@ +#ifndef DB_H +#define DB_H + +#include "event_loop.h" + +typedef struct db_conn_impl db_conn; + +/* Connect to the global system store at address and port. The last + * parameter is an output parameter and we assume the memory is + * allocated by the caller. */ +void db_connect(const char *db_address, + int db_port, + const char *client_type, + const char *client_addr, + int client_port, + db_conn *db); + +/* Attach global system store onnection to event loop. Returns the index of the + * connection in the loop. */ +int64_t db_attach(db_conn *db, event_loop *loop, int connection_type); + +/* This function will be called by the user if there is a new event in the + * event loop associated with the global system store connection. */ +void db_event(db_conn *db); + +/* Disconnect from the global system store. */ +void db_disconnect(db_conn *db); + +#endif diff --git a/state/object_table.h b/state/object_table.h new file mode 100644 index 000000000..6b4d62e4b --- /dev/null +++ b/state/object_table.h @@ -0,0 +1,15 @@ +#include "common.h" +#include "db.h" + +typedef void (*lookup_callback)(void *); + +/* Register a new object with the directory. */ +void object_table_add(db_conn *db, unique_id object_id); + +/* Remove object from the directory */ +void object_table_remove(db_conn *db, unique_id object_id); + +/* Look up entry from the directory */ +void object_table_lookup(db_conn *db, + unique_id object_id, + lookup_callback callback); diff --git a/state/redis.c b/state/redis.c new file mode 100644 index 000000000..a8029a063 --- /dev/null +++ b/state/redis.c @@ -0,0 +1,188 @@ +/* Redis implementation of the global state store */ + +#include + +#include "common.h" +#include "db.h" +#include "object_table.h" +#include "event_loop.h" +#include "redis.h" + +static void poll_add_read(void *privdata) { + db_conn *conn = (db_conn *) privdata; + if (!conn->reading) { + conn->reading = 1; + event_loop_get(conn->loop, 0)->events |= POLLIN; + } +} + +static void poll_del_read(void *privdata) { + db_conn *conn = (db_conn *) privdata; + if (conn->reading) { + conn->reading = 0; + event_loop_get(conn->loop, 0)->events &= ~POLLIN; + } +} + +static void poll_add_write(void *privdata) { + db_conn *conn = (db_conn *) privdata; + if (!conn->writing) { + conn->writing = 1; + event_loop_get(conn->loop, 0)->events |= POLLOUT; + } +} + +static void poll_del_write(void *privdata) { + db_conn *conn = (db_conn *) privdata; + if (conn->writing) { + conn->writing = 0; + event_loop_get(conn->loop, 0)->events &= ~POLLOUT; + } +} + +#define LOG_REDIS_ERR(context, M, ...) \ + fprintf(stderr, "[ERROR] (%s:%d: message: %s) " M "\n", __FILE__, __LINE__, \ + context->errstr, ##__VA_ARGS__) + +#define CHECK_REDIS_CONNECT(CONTEXT_TYPE, context, M, ...) \ + do { \ + CONTEXT_TYPE *_context = (context); \ + if (!_context) { \ + LOG_ERR("could not allocate redis context"); \ + exit(-1); \ + } \ + if (_context->err) { \ + LOG_REDIS_ERR(_context, M, ##__VA_ARGS__); \ + exit(-1); \ + } \ + } while (0); + +void db_connect(const char *address, + int port, + const char *client_type, + const char *client_addr, + int client_port, + db_conn *db) { + /* Sync connection for initial handshake */ + redisReply *reply; + long long num_clients; + redisContext *context = redisConnect(address, port); + CHECK_REDIS_CONNECT(redisContext, context, "could not connect to redis %s:%d", + address, port); + /* Add new client using optimistic locking. */ + while (1) { + reply = redisCommand(context, "WATCH %s", client_type); + freeReplyObject(reply); + reply = redisCommand(context, "HLEN %s", client_type); + num_clients = reply->integer; + freeReplyObject(reply); + reply = redisCommand(context, "MULTI"); + freeReplyObject(reply); + reply = redisCommand(context, "HSET %s %lld %s:%d", client_type, + num_clients, client_addr, client_port); + freeReplyObject(reply); + reply = redisCommand(context, "EXEC"); + if (reply) { + freeReplyObject(reply); + break; + } + freeReplyObject(reply); + } + redisFree(context); + + db->client_type = strdup(client_type); + db->client_id = num_clients; + db->reading = 0; + db->writing = 0; + + /* Establish async connection */ + db->context = redisAsyncConnect(address, port); + CHECK_REDIS_CONNECT(redisAsyncContext, db->context, + "could not connect to redis %s:%d", address, port); + db->context->data = (void *) db; +} + +void db_event(db_conn *db) { + if (db->reading) { + redisAsyncHandleRead(db->context); + } + if (db->writing) { + redisAsyncHandleWrite(db->context); + } +} + +int64_t db_attach(db_conn *db, event_loop *loop, int connection_type) { + db->loop = loop; + + redisAsyncContext *ac = db->context; + redisContext *c = &(ac->c); + + if (ac->ev.data != NULL) { + return REDIS_ERR; + } + + ac->ev.addRead = poll_add_read; + ac->ev.delRead = poll_del_read; + ac->ev.addWrite = poll_add_write; + ac->ev.delWrite = poll_del_write; + // TODO(pcm): Implement cleanup function + + ac->ev.data = db; + + return event_loop_attach(loop, connection_type, NULL, c->fd, + POLLIN | POLLOUT); +} + +void object_table_add(db_conn *db, unique_id object_id) { + static char hex_object_id[2 * UNIQUE_ID_SIZE + 1]; + sha1_to_hex(&object_id.id[0], &hex_object_id[0]); + redisAsyncCommand(db->context, NULL, NULL, "SADD obj:%s %d", + &hex_object_id[0], 0); + if (db->context->err) { + LOG_REDIS_ERR(db->context, "could not add object_table entry"); + } +} + +void object_table_lookup_callback(redisAsyncContext *c, + void *r, + void *privdata) { + redisReply *reply = r; + if (reply == NULL) + return; + lookup_callback callback = privdata; + char *str = malloc(reply->len); + memcpy(str, reply->str, reply->len); + callback(str); +} + +void object_table_fetch_addr_port(redisAsyncContext *c, + void *r, + void *privdata) { + redisReply *reply = r; + if (reply == NULL) + return; + long long manager_id = -1; + if (reply->type == REDIS_REPLY_STRING) { + manager_id = strtoll(reply->str, NULL, 10); + } else if (reply->type != REDIS_REPLY_INTEGER) { + manager_id = reply->integer; + } else { + LOG_ERR("expected integer or string, received type %d", reply->type); + exit(-1); + } + db_conn *db = c->data; + redisAsyncCommand(db->context, object_table_lookup_callback, privdata, + "HGET %s %lld", db->client_type, manager_id); +} + +void object_table_lookup(db_conn *db, + unique_id object_id, + lookup_callback callback) { + static char hex_object_id[2 * UNIQUE_ID_SIZE + 1]; + sha1_to_hex(&object_id.id[0], &hex_object_id[0]); + redisAsyncCommand(db->context, object_table_fetch_addr_port, callback, + "SRANDMEMBER obj:%s", &hex_object_id[0]); + if (db->context->err) { + LOG_REDIS_ERR(db->context, "error in object_table lookup"); + } +} diff --git a/state/redis.h b/state/redis.h new file mode 100644 index 000000000..471044b06 --- /dev/null +++ b/state/redis.h @@ -0,0 +1,26 @@ +#include "db.h" +#include "object_table.h" + +#include "hiredis/hiredis.h" +#include "hiredis/async.h" + +struct db_conn_impl { + /* String that identifies this client type. */ + char *client_type; + /* Unique ID for this client within the type. */ + int64_t client_id; + /* Redis context for this global state store connection. */ + redisAsyncContext *context; + /* Which events are we processing (read, write)? */ + int reading, writing; + /* The event loop this global state store connection is part of. */ + event_loop *loop; +}; + +void object_table_fetch_addr_port(redisAsyncContext *c, + void *r, + void *privdata); + +void object_table_lookup_callback(redisAsyncContext *c, + void *r, + void *privdata); diff --git a/test/db_tests.c b/test/db_tests.c new file mode 100644 index 000000000..b3a0d582e --- /dev/null +++ b/test/db_tests.c @@ -0,0 +1,69 @@ +#include "greatest.h" + +#include + +#include "event_loop.h" +#include "state/db.h" +#include "state/object_table.h" +#include "state/redis.h" + +SUITE(db_tests); + +int lookup_successful = 0; +const char *manager_addr = "127.0.0.1"; +int manager_port = 12345; +char received_addr[16] = {0}; +char received_port[6] = {0}; + +void test_callback(void *userdata); + +void test_callback(void *userdata) { + char *reply = userdata; + lookup_successful = 1; + if (!reply || + sscanf(reply, "%15[0-9.]:%5[0-9]", received_addr, received_port) != 2) { + assert(0); + } + free(reply); +} + +TEST object_table_lookup_test(void) { + event_loop loop; + event_loop_init(&loop); + db_conn conn; + db_connect("127.0.0.1", 6379, "plasma_manager", manager_addr, manager_port, + &conn); + int64_t index = db_attach(&conn, &loop, 0); + unique_id id = {{0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0}}; + object_table_add(&conn, id); + object_table_lookup(&conn, id, test_callback); + while (!lookup_successful) { + int num_ready = event_loop_poll(&loop); + if (num_ready < 0) { + exit(-1); + } + for (int i = 0; i < event_loop_size(&loop); ++i) { + struct pollfd *waiting = event_loop_get(&loop, i); + if (waiting->revents == 0) + continue; + if (i == index) { + db_event(&conn); + } + } + } + ASSERT_STR_EQ(&received_addr[0], manager_addr); + ASSERT_EQ(atoi(received_port), manager_port); + PASS(); +} + +SUITE(db_tests) { + RUN_TEST(object_table_lookup_test); +} + +GREATEST_MAIN_DEFS(); + +int main(int argc, char **argv) { + GREATEST_MAIN_BEGIN(); + RUN_SUITE(db_tests); + GREATEST_MAIN_END(); +} diff --git a/thirdparty/build-redis.sh b/thirdparty/build-redis.sh new file mode 100644 index 000000000..57c68c97b --- /dev/null +++ b/thirdparty/build-redis.sh @@ -0,0 +1,4 @@ +wget http://download.redis.io/releases/redis-3.2.3.tar.gz +tar xvfz redis-3.2.3.tar.gz +cd redis-3.2.3 +make diff --git a/thirdparty/greatest.h b/thirdparty/greatest.h new file mode 100644 index 000000000..eb34ff426 --- /dev/null +++ b/thirdparty/greatest.h @@ -0,0 +1,1023 @@ +/* + * Copyright (c) 2011-2016 Scott Vokes + * + * Permission to use, copy, modify, and/or distribute this software for any + * purpose with or without fee is hereby granted, provided that the above + * copyright notice and this permission notice appear in all copies. + * + * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES + * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF + * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR + * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES + * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN + * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF + * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. + */ + +#ifndef GREATEST_H +#define GREATEST_H + +#ifdef __cplusplus +extern "C" { +#endif + +/* 1.2.1 */ +#define GREATEST_VERSION_MAJOR 1 +#define GREATEST_VERSION_MINOR 2 +#define GREATEST_VERSION_PATCH 1 + +/* A unit testing system for C, contained in 1 file. + * It doesn't use dynamic allocation or depend on anything + * beyond ANSI C89. + * + * An up-to-date version can be found at: + * https://github.com/silentbicycle/greatest/ + */ + + +/********************************************************************* + * Minimal test runner template + *********************************************************************/ +#if 0 +#include "greatest.h" +TEST foo_should_foo(void) { + PASS(); +} +static void setup_cb(void *data) { + printf("setup callback for each test case\n"); +} +static void teardown_cb(void *data) { + printf("teardown callback for each test case\n"); +} +SUITE(suite) { + /* Optional setup/teardown callbacks which will be run before/after + * every test case. If using a test suite, they will be cleared when + * the suite finishes. */ + SET_SETUP(setup_cb, voidp_to_callback_data); + SET_TEARDOWN(teardown_cb, voidp_to_callback_data); + RUN_TEST(foo_should_foo); +} +/* Add definitions that need to be in the test runner's main file. */ +GREATEST_MAIN_DEFS(); +/* Set up, run suite(s) of tests, report pass/fail/skip stats. */ +int run_tests(void) { + GREATEST_INIT(); /* init. greatest internals */ + /* List of suites to run (if any). */ + RUN_SUITE(suite); + /* Tests can also be run directly, without using test suites. */ + RUN_TEST(foo_should_foo); + GREATEST_PRINT_REPORT(); /* display results */ + return greatest_all_passed(); +} +/* main(), for a standalone command-line test runner. + * This replaces run_tests above, and adds command line option + * handling and exiting with a pass/fail status. */ +int main(int argc, char **argv) { + GREATEST_MAIN_BEGIN(); /* init & parse command-line args */ + RUN_SUITE(suite); + GREATEST_MAIN_END(); /* display results */ +} +#endif +/*********************************************************************/ + + +#include +#include +#include +#include + +/*********** + * Options * + ***********/ + +/* Default column width for non-verbose output. */ +#ifndef GREATEST_DEFAULT_WIDTH +#define GREATEST_DEFAULT_WIDTH 72 +#endif + +/* FILE *, for test logging. */ +#ifndef GREATEST_STDOUT +#define GREATEST_STDOUT stdout +#endif + +/* Remove GREATEST_ prefix from most commonly used symbols? */ +#ifndef GREATEST_USE_ABBREVS +#define GREATEST_USE_ABBREVS 1 +#endif + +/* Set to 0 to disable all use of setjmp/longjmp. */ +#ifndef GREATEST_USE_LONGJMP +#define GREATEST_USE_LONGJMP 1 +#endif + +#if GREATEST_USE_LONGJMP +#include +#endif + +/* Set to 0 to disable all use of time.h / clock(). */ +#ifndef GREATEST_USE_TIME +#define GREATEST_USE_TIME 1 +#endif + +#if GREATEST_USE_TIME +#include +#endif + +/* Floating point type, for ASSERT_IN_RANGE. */ +#ifndef GREATEST_FLOAT +#define GREATEST_FLOAT double +#define GREATEST_FLOAT_FMT "%g" +#endif + +/********* + * Types * + *********/ + +/* Info for the current running suite. */ +typedef struct greatest_suite_info { + unsigned int tests_run; + unsigned int passed; + unsigned int failed; + unsigned int skipped; + +#if GREATEST_USE_TIME + /* timers, pre/post running suite and individual tests */ + clock_t pre_suite; + clock_t post_suite; + clock_t pre_test; + clock_t post_test; +#endif +} greatest_suite_info; + +/* Type for a suite function. */ +typedef void (greatest_suite_cb)(void); + +/* Types for setup/teardown callbacks. If non-NULL, these will be run + * and passed the pointer to their additional data. */ +typedef void (greatest_setup_cb)(void *udata); +typedef void (greatest_teardown_cb)(void *udata); + +/* Type for an equality comparison between two pointers of the same type. + * Should return non-0 if equal, otherwise 0. + * UDATA is a closure value, passed through from ASSERT_EQUAL_T[m]. */ +typedef int greatest_equal_cb(const void *exp, const void *got, void *udata); + +/* Type for a callback that prints a value pointed to by T. + * Return value has the same meaning as printf's. + * UDATA is a closure value, passed through from ASSERT_EQUAL_T[m]. */ +typedef int greatest_printf_cb(const void *t, void *udata); + +/* Callbacks for an arbitrary type; needed for type-specific + * comparisons via GREATEST_ASSERT_EQUAL_T[m].*/ +typedef struct greatest_type_info { + greatest_equal_cb *equal; + greatest_printf_cb *print; +} greatest_type_info; + +typedef struct greatest_memory_cmp_env { + const unsigned char *exp; + const unsigned char *got; + size_t size; +} greatest_memory_cmp_env; + +/* Callbacks for string and raw memory types. */ +extern greatest_type_info greatest_type_info_string; +extern greatest_type_info greatest_type_info_memory; + +typedef enum { + GREATEST_FLAG_FIRST_FAIL = 0x01, + GREATEST_FLAG_LIST_ONLY = 0x02 +} greatest_flag_t; + +/* Struct containing all test runner state. */ +typedef struct greatest_run_info { + unsigned char flags; + unsigned char verbosity; + unsigned int tests_run; /* total test count */ + + /* overall pass/fail/skip counts */ + unsigned int passed; + unsigned int failed; + unsigned int skipped; + unsigned int assertions; + + /* currently running test suite */ + greatest_suite_info suite; + + /* info to print about the most recent failure */ + const char *fail_file; + unsigned int fail_line; + const char *msg; + + /* current setup/teardown hooks and userdata */ + greatest_setup_cb *setup; + void *setup_udata; + greatest_teardown_cb *teardown; + void *teardown_udata; + + /* formatting info for ".....s...F"-style output */ + unsigned int col; + unsigned int width; + + /* only run a specific suite or test */ + const char *suite_filter; + const char *test_filter; + +#if GREATEST_USE_TIME + /* overall timers */ + clock_t begin; + clock_t end; +#endif + +#if GREATEST_USE_LONGJMP + jmp_buf jump_dest; +#endif +} greatest_run_info; + +struct greatest_report_t { + /* overall pass/fail/skip counts */ + unsigned int passed; + unsigned int failed; + unsigned int skipped; + unsigned int assertions; +}; + +/* Global var for the current testing context. + * Initialized by GREATEST_MAIN_DEFS(). */ +extern greatest_run_info greatest_info; + +/* Type for ASSERT_ENUM_EQ's ENUM_STR argument. */ +typedef const char *greatest_enum_str_fun(int value); + +/********************** + * Exported functions * + **********************/ + +/* These are used internally by greatest. */ +void greatest_do_pass(const char *name); +void greatest_do_fail(const char *name); +void greatest_do_skip(const char *name); +int greatest_pre_test(const char *name); +void greatest_post_test(const char *name, int res); +void greatest_usage(const char *name); +int greatest_do_assert_equal_t(const void *exp, const void *got, + greatest_type_info *type_info, void *udata); + +/* These are part of the public greatest API. */ +void GREATEST_SET_SETUP_CB(greatest_setup_cb *cb, void *udata); +void GREATEST_SET_TEARDOWN_CB(greatest_teardown_cb *cb, void *udata); +int greatest_all_passed(void); +void greatest_set_test_filter(const char *name); +void greatest_set_suite_filter(const char *name); +void greatest_get_report(struct greatest_report_t *report); +unsigned int greatest_get_verbosity(void); +void greatest_set_verbosity(unsigned int verbosity); +void greatest_set_flag(greatest_flag_t flag); + + +/******************** +* Language Support * +********************/ + +/* If __VA_ARGS__ (C99) is supported, allow parametric testing +* without needing to manually manage the argument struct. */ +#if __STDC_VERSION__ >= 19901L || _MSC_VER >= 1800 +#define GREATEST_VA_ARGS +#endif + + +/********** + * Macros * + **********/ + +/* Define a suite. */ +#define GREATEST_SUITE(NAME) void NAME(void); void NAME(void) + +/* Declare a suite, provided by another compilation unit. */ +#define GREATEST_SUITE_EXTERN(NAME) void NAME(void) + +/* Start defining a test function. + * The arguments are not included, to allow parametric testing. */ +#define GREATEST_TEST static enum greatest_test_res + +/* PASS/FAIL/SKIP result from a test. Used internally. */ +typedef enum greatest_test_res { + GREATEST_TEST_RES_PASS = 0, + GREATEST_TEST_RES_FAIL = -1, + GREATEST_TEST_RES_SKIP = 1 +} greatest_test_res; + +/* Run a suite. */ +#define GREATEST_RUN_SUITE(S_NAME) greatest_run_suite(S_NAME, #S_NAME) + +/* Run a test in the current suite. */ +#define GREATEST_RUN_TEST(TEST) \ + do { \ + if (greatest_pre_test(#TEST) == 1) { \ + enum greatest_test_res res = GREATEST_SAVE_CONTEXT(); \ + if (res == GREATEST_TEST_RES_PASS) { \ + res = TEST(); \ + } \ + greatest_post_test(#TEST, res); \ + } else if (GREATEST_LIST_ONLY()) { \ + fprintf(GREATEST_STDOUT, " %s\n", #TEST); \ + } \ + } while (0) + +/* Ignore a test, don't warn about it being unused. */ +#define GREATEST_IGNORE_TEST(TEST) (void)TEST + +/* Run a test in the current suite with one void * argument, + * which can be a pointer to a struct with multiple arguments. */ +#define GREATEST_RUN_TEST1(TEST, ENV) \ + do { \ + if (greatest_pre_test(#TEST) == 1) { \ + int res = TEST(ENV); \ + greatest_post_test(#TEST, res); \ + } else if (GREATEST_LIST_ONLY()) { \ + fprintf(GREATEST_STDOUT, " %s\n", #TEST); \ + } \ + } while (0) + +#ifdef GREATEST_VA_ARGS +#define GREATEST_RUN_TESTp(TEST, ...) \ + do { \ + if (greatest_pre_test(#TEST) == 1) { \ + int res = TEST(__VA_ARGS__); \ + greatest_post_test(#TEST, res); \ + } else if (GREATEST_LIST_ONLY()) { \ + fprintf(GREATEST_STDOUT, " %s\n", #TEST); \ + } \ + } while (0) +#endif + + +/* Check if the test runner is in verbose mode. */ +#define GREATEST_IS_VERBOSE() ((greatest_info.verbosity) > 0) +#define GREATEST_LIST_ONLY() \ + (greatest_info.flags & GREATEST_FLAG_LIST_ONLY) +#define GREATEST_FIRST_FAIL() \ + (greatest_info.flags & GREATEST_FLAG_FIRST_FAIL) +#define GREATEST_FAILURE_ABORT() \ + (greatest_info.suite.failed > 0 && GREATEST_FIRST_FAIL()) + +/* Message-less forms of tests defined below. */ +#define GREATEST_PASS() GREATEST_PASSm(NULL) +#define GREATEST_FAIL() GREATEST_FAILm(NULL) +#define GREATEST_SKIP() GREATEST_SKIPm(NULL) +#define GREATEST_ASSERT(COND) \ + GREATEST_ASSERTm(#COND, COND) +#define GREATEST_ASSERT_OR_LONGJMP(COND) \ + GREATEST_ASSERT_OR_LONGJMPm(#COND, COND) +#define GREATEST_ASSERT_FALSE(COND) \ + GREATEST_ASSERT_FALSEm(#COND, COND) +#define GREATEST_ASSERT_EQ(EXP, GOT) \ + GREATEST_ASSERT_EQm(#EXP " != " #GOT, EXP, GOT) +#define GREATEST_ASSERT_EQ_FMT(EXP, GOT, FMT) \ + GREATEST_ASSERT_EQ_FMTm(#EXP " != " #GOT, EXP, GOT, FMT) +#define GREATEST_ASSERT_IN_RANGE(EXP, GOT, TOL) \ + GREATEST_ASSERT_IN_RANGEm(#EXP " != " #GOT " +/- " #TOL, EXP, GOT, TOL) +#define GREATEST_ASSERT_EQUAL_T(EXP, GOT, TYPE_INFO, UDATA) \ + GREATEST_ASSERT_EQUAL_Tm(#EXP " != " #GOT, EXP, GOT, TYPE_INFO, UDATA) +#define GREATEST_ASSERT_STR_EQ(EXP, GOT) \ + GREATEST_ASSERT_STR_EQm(#EXP " != " #GOT, EXP, GOT) +#define GREATEST_ASSERT_STRN_EQ(EXP, GOT, SIZE) \ + GREATEST_ASSERT_STRN_EQm(#EXP " != " #GOT, EXP, GOT, SIZE) +#define GREATEST_ASSERT_MEM_EQ(EXP, GOT, SIZE) \ + GREATEST_ASSERT_MEM_EQm(#EXP " != " #GOT, EXP, GOT, SIZE) +#define GREATEST_ASSERT_ENUM_EQ(EXP, GOT, ENUM_STR) \ + GREATEST_ASSERT_ENUM_EQm(#EXP " != " #GOT, EXP, GOT, ENUM_STR) + +/* The following forms take an additional message argument first, + * to be displayed by the test runner. */ + +/* Fail if a condition is not true, with message. */ +#define GREATEST_ASSERTm(MSG, COND) \ + do { \ + greatest_info.assertions++; \ + if (!(COND)) { GREATEST_FAILm(MSG); } \ + } while (0) + +/* Fail if a condition is not true, longjmping out of test. */ +#define GREATEST_ASSERT_OR_LONGJMPm(MSG, COND) \ + do { \ + greatest_info.assertions++; \ + if (!(COND)) { GREATEST_FAIL_WITH_LONGJMPm(MSG); } \ + } while (0) + +/* Fail if a condition is not false, with message. */ +#define GREATEST_ASSERT_FALSEm(MSG, COND) \ + do { \ + greatest_info.assertions++; \ + if ((COND)) { GREATEST_FAILm(MSG); } \ + } while (0) + +/* Fail if EXP != GOT (equality comparison by ==). */ +#define GREATEST_ASSERT_EQm(MSG, EXP, GOT) \ + do { \ + greatest_info.assertions++; \ + if ((EXP) != (GOT)) { GREATEST_FAILm(MSG); } \ + } while (0) + +/* Fail if EXP != GOT (equality comparison by ==). + * Warning: EXP and GOT will be evaluated more than once on failure. */ +#define GREATEST_ASSERT_EQ_FMTm(MSG, EXP, GOT, FMT) \ + do { \ + const char *greatest_FMT = ( FMT ); \ + greatest_info.assertions++; \ + if ((EXP) != (GOT)) { \ + fprintf(GREATEST_STDOUT, "\nExpected: "); \ + fprintf(GREATEST_STDOUT, greatest_FMT, EXP); \ + fprintf(GREATEST_STDOUT, "\n Got: "); \ + fprintf(GREATEST_STDOUT, greatest_FMT, GOT); \ + fprintf(GREATEST_STDOUT, "\n"); \ + GREATEST_FAILm(MSG); \ + } \ + } while (0) + +/* Fail if EXP is not equal to GOT, printing enum IDs. */ +#define GREATEST_ASSERT_ENUM_EQm(MSG, EXP, GOT, ENUM_STR) \ + do { \ + int greatest_EXP = (int)(EXP); \ + int greatest_GOT = (int)(GOT); \ + greatest_enum_str_fun *greatest_ENUM_STR = ENUM_STR; \ + if (greatest_EXP != greatest_GOT) { \ + fprintf(GREATEST_STDOUT, "\nExpected: %s", \ + greatest_ENUM_STR(greatest_EXP)); \ + fprintf(GREATEST_STDOUT, "\n Got: %s\n", \ + greatest_ENUM_STR(greatest_GOT)); \ + GREATEST_FAILm(MSG); \ + } \ + } while (0) \ + +/* Fail if GOT not in range of EXP +|- TOL. */ +#define GREATEST_ASSERT_IN_RANGEm(MSG, EXP, GOT, TOL) \ + do { \ + GREATEST_FLOAT greatest_EXP = (EXP); \ + GREATEST_FLOAT greatest_GOT = (GOT); \ + GREATEST_FLOAT greatest_TOL = (TOL); \ + greatest_info.assertions++; \ + if ((greatest_EXP > greatest_GOT && \ + greatest_EXP - greatest_GOT > greatest_TOL) || \ + (greatest_EXP < greatest_GOT && \ + greatest_GOT - greatest_EXP > greatest_TOL)) { \ + fprintf(GREATEST_STDOUT, \ + "\nExpected: " GREATEST_FLOAT_FMT \ + " +/- " GREATEST_FLOAT_FMT \ + "\n Got: " GREATEST_FLOAT_FMT \ + "\n", \ + greatest_EXP, greatest_TOL, greatest_GOT); \ + GREATEST_FAILm(MSG); \ + } \ + } while (0) + +/* Fail if EXP is not equal to GOT, according to strcmp. */ +#define GREATEST_ASSERT_STR_EQm(MSG, EXP, GOT) \ + do { \ + GREATEST_ASSERT_EQUAL_Tm(MSG, EXP, GOT, \ + &greatest_type_info_string, NULL); \ + } while (0) \ + +/* Fail if EXP is not equal to GOT, according to strcmp. */ +#define GREATEST_ASSERT_STRN_EQm(MSG, EXP, GOT, SIZE) \ + do { \ + size_t size = SIZE; \ + GREATEST_ASSERT_EQUAL_Tm(MSG, EXP, GOT, \ + &greatest_type_info_string, &size); \ + } while (0) \ + +/* Fail if EXP is not equal to GOT, according to memcmp. */ +#define GREATEST_ASSERT_MEM_EQm(MSG, EXP, GOT, SIZE) \ + do { \ + greatest_memory_cmp_env env; \ + env.exp = (const unsigned char *)EXP; \ + env.got = (const unsigned char *)GOT; \ + env.size = SIZE; \ + GREATEST_ASSERT_EQUAL_Tm(MSG, env.exp, env.got, \ + &greatest_type_info_memory, &env); \ + } while (0) \ + +/* Fail if EXP is not equal to GOT, according to a comparison + * callback in TYPE_INFO. If they are not equal, optionally use a + * print callback in TYPE_INFO to print them. */ +#define GREATEST_ASSERT_EQUAL_Tm(MSG, EXP, GOT, TYPE_INFO, UDATA) \ + do { \ + greatest_type_info *type_info = (TYPE_INFO); \ + greatest_info.assertions++; \ + if (!greatest_do_assert_equal_t(EXP, GOT, \ + type_info, UDATA)) { \ + if (type_info == NULL || type_info->equal == NULL) { \ + GREATEST_FAILm("type_info->equal callback missing!"); \ + } else { \ + GREATEST_FAILm(MSG); \ + } \ + } \ + } while (0) \ + +/* Pass. */ +#define GREATEST_PASSm(MSG) \ + do { \ + greatest_info.msg = MSG; \ + return GREATEST_TEST_RES_PASS; \ + } while (0) + +/* Fail. */ +#define GREATEST_FAILm(MSG) \ + do { \ + greatest_info.fail_file = __FILE__; \ + greatest_info.fail_line = __LINE__; \ + greatest_info.msg = MSG; \ + return GREATEST_TEST_RES_FAIL; \ + } while (0) + +/* Optional GREATEST_FAILm variant that longjmps. */ +#if GREATEST_USE_LONGJMP +#define GREATEST_FAIL_WITH_LONGJMP() GREATEST_FAIL_WITH_LONGJMPm(NULL) +#define GREATEST_FAIL_WITH_LONGJMPm(MSG) \ + do { \ + greatest_info.fail_file = __FILE__; \ + greatest_info.fail_line = __LINE__; \ + greatest_info.msg = MSG; \ + longjmp(greatest_info.jump_dest, GREATEST_TEST_RES_FAIL); \ + } while (0) +#endif + +/* Skip the current test. */ +#define GREATEST_SKIPm(MSG) \ + do { \ + greatest_info.msg = MSG; \ + return GREATEST_TEST_RES_SKIP; \ + } while (0) + +/* Check the result of a subfunction using ASSERT, etc. */ +#define GREATEST_CHECK_CALL(RES) \ + do { \ + enum greatest_test_res greatest_RES = RES; \ + if (greatest_RES != GREATEST_TEST_RES_PASS) { \ + return greatest_RES; \ + } \ + } while (0) \ + +#if GREATEST_USE_TIME +#define GREATEST_SET_TIME(NAME) \ + NAME = clock(); \ + if (NAME == (clock_t) -1) { \ + fprintf(GREATEST_STDOUT, \ + "clock error: %s\n", #NAME); \ + exit(EXIT_FAILURE); \ + } + +#define GREATEST_CLOCK_DIFF(C1, C2) \ + fprintf(GREATEST_STDOUT, " (%lu ticks, %.3f sec)", \ + (long unsigned int) (C2) - (long unsigned int)(C1), \ + (double)((C2) - (C1)) / (1.0 * (double)CLOCKS_PER_SEC)) +#else +#define GREATEST_SET_TIME(UNUSED) +#define GREATEST_CLOCK_DIFF(UNUSED1, UNUSED2) +#endif + +#if GREATEST_USE_LONGJMP +#define GREATEST_SAVE_CONTEXT() \ + /* setjmp returns 0 (GREATEST_TEST_RES_PASS) on first call */ \ + /* so the test runs, then RES_FAIL from FAIL_WITH_LONGJMP. */ \ + ((enum greatest_test_res)(setjmp(greatest_info.jump_dest))) +#else +#define GREATEST_SAVE_CONTEXT() \ + /*a no-op, since setjmp/longjmp aren't being used */ \ + GREATEST_TEST_RES_PASS +#endif + +/* Include several function definitions in the main test file. */ +#define GREATEST_MAIN_DEFS() \ + \ +/* Is FILTER a subset of NAME? */ \ +static int greatest_name_match(const char *name, \ + const char *filter) { \ + size_t offset = 0; \ + size_t filter_len = strlen(filter); \ + while (name[offset] != '\0') { \ + if (name[offset] == filter[0]) { \ + if (0 == strncmp(&name[offset], filter, filter_len)) { \ + return 1; \ + } \ + } \ + offset++; \ + } \ + \ + return 0; \ +} \ + \ +int greatest_pre_test(const char *name) { \ + if (!GREATEST_LIST_ONLY() \ + && (!GREATEST_FIRST_FAIL() || greatest_info.suite.failed == 0) \ + && (greatest_info.test_filter == NULL || \ + greatest_name_match(name, greatest_info.test_filter))) { \ + GREATEST_SET_TIME(greatest_info.suite.pre_test); \ + if (greatest_info.setup) { \ + greatest_info.setup(greatest_info.setup_udata); \ + } \ + return 1; /* test should be run */ \ + } else { \ + return 0; /* skipped */ \ + } \ +} \ + \ +void greatest_post_test(const char *name, int res) { \ + GREATEST_SET_TIME(greatest_info.suite.post_test); \ + if (greatest_info.teardown) { \ + void *udata = greatest_info.teardown_udata; \ + greatest_info.teardown(udata); \ + } \ + \ + if (res <= GREATEST_TEST_RES_FAIL) { \ + greatest_do_fail(name); \ + } else if (res >= GREATEST_TEST_RES_SKIP) { \ + greatest_do_skip(name); \ + } else if (res == GREATEST_TEST_RES_PASS) { \ + greatest_do_pass(name); \ + } \ + greatest_info.suite.tests_run++; \ + greatest_info.col++; \ + if (GREATEST_IS_VERBOSE()) { \ + GREATEST_CLOCK_DIFF(greatest_info.suite.pre_test, \ + greatest_info.suite.post_test); \ + fprintf(GREATEST_STDOUT, "\n"); \ + } else if (greatest_info.col % greatest_info.width == 0) { \ + fprintf(GREATEST_STDOUT, "\n"); \ + greatest_info.col = 0; \ + } \ + if (GREATEST_STDOUT == stdout) fflush(stdout); \ +} \ + \ +static void report_suite(void) { \ + if (greatest_info.suite.tests_run > 0) { \ + fprintf(GREATEST_STDOUT, \ + "\n%u test%s - %u passed, %u failed, %u skipped", \ + greatest_info.suite.tests_run, \ + greatest_info.suite.tests_run == 1 ? "" : "s", \ + greatest_info.suite.passed, \ + greatest_info.suite.failed, \ + greatest_info.suite.skipped); \ + GREATEST_CLOCK_DIFF(greatest_info.suite.pre_suite, \ + greatest_info.suite.post_suite); \ + fprintf(GREATEST_STDOUT, "\n"); \ + } \ +} \ + \ +static void update_counts_and_reset_suite(void) { \ + greatest_info.setup = NULL; \ + greatest_info.setup_udata = NULL; \ + greatest_info.teardown = NULL; \ + greatest_info.teardown_udata = NULL; \ + greatest_info.passed += greatest_info.suite.passed; \ + greatest_info.failed += greatest_info.suite.failed; \ + greatest_info.skipped += greatest_info.suite.skipped; \ + greatest_info.tests_run += greatest_info.suite.tests_run; \ + memset(&greatest_info.suite, 0, sizeof(greatest_info.suite)); \ + greatest_info.col = 0; \ +} \ + \ +static void greatest_run_suite(greatest_suite_cb *suite_cb, \ + const char *suite_name) { \ + if (greatest_info.suite_filter && \ + !greatest_name_match(suite_name, greatest_info.suite_filter)) { \ + return; \ + } \ + update_counts_and_reset_suite(); \ + if (GREATEST_FIRST_FAIL() && greatest_info.failed > 0) { return; } \ + fprintf(GREATEST_STDOUT, "\n* Suite %s:\n", suite_name); \ + GREATEST_SET_TIME(greatest_info.suite.pre_suite); \ + suite_cb(); \ + GREATEST_SET_TIME(greatest_info.suite.post_suite); \ + report_suite(); \ +} \ + \ +void greatest_do_pass(const char *name) { \ + if (GREATEST_IS_VERBOSE()) { \ + fprintf(GREATEST_STDOUT, "PASS %s: %s", \ + name, greatest_info.msg ? greatest_info.msg : ""); \ + } else { \ + fprintf(GREATEST_STDOUT, "."); \ + } \ + greatest_info.suite.passed++; \ +} \ + \ +void greatest_do_fail(const char *name) { \ + if (GREATEST_IS_VERBOSE()) { \ + fprintf(GREATEST_STDOUT, \ + "FAIL %s: %s (%s:%u)", \ + name, greatest_info.msg ? greatest_info.msg : "", \ + greatest_info.fail_file, greatest_info.fail_line); \ + } else { \ + fprintf(GREATEST_STDOUT, "F"); \ + greatest_info.col++; \ + /* add linebreak if in line of '.'s */ \ + if (greatest_info.col != 0) { \ + fprintf(GREATEST_STDOUT, "\n"); \ + greatest_info.col = 0; \ + } \ + fprintf(GREATEST_STDOUT, "FAIL %s: %s (%s:%u)\n", \ + name, \ + greatest_info.msg ? greatest_info.msg : "", \ + greatest_info.fail_file, greatest_info.fail_line); \ + } \ + greatest_info.suite.failed++; \ +} \ + \ +void greatest_do_skip(const char *name) { \ + if (GREATEST_IS_VERBOSE()) { \ + fprintf(GREATEST_STDOUT, "SKIP %s: %s", \ + name, \ + greatest_info.msg ? \ + greatest_info.msg : "" ); \ + } else { \ + fprintf(GREATEST_STDOUT, "s"); \ + } \ + greatest_info.suite.skipped++; \ +} \ + \ +int greatest_do_assert_equal_t(const void *exp, const void *got, \ + greatest_type_info *type_info, void *udata) { \ + int eq = 0; \ + if (type_info == NULL || type_info->equal == NULL) { \ + return 0; \ + } \ + eq = type_info->equal(exp, got, udata); \ + if (!eq) { \ + if (type_info->print != NULL) { \ + fprintf(GREATEST_STDOUT, "\nExpected: "); \ + (void)type_info->print(exp, udata); \ + fprintf(GREATEST_STDOUT, "\n Got: "); \ + (void)type_info->print(got, udata); \ + fprintf(GREATEST_STDOUT, "\n"); \ + } else { \ + fprintf(GREATEST_STDOUT, \ + "GREATEST_ASSERT_EQUAL_T failure at %s:%u\n", \ + greatest_info.fail_file, \ + greatest_info.fail_line); \ + } \ + } \ + return eq; \ +} \ + \ +void greatest_usage(const char *name) { \ + fprintf(GREATEST_STDOUT, \ + "Usage: %s [-hlfv] [-s SUITE] [-t TEST]\n" \ + " -h, --help print this Help\n" \ + " -l List suites and their tests, then exit\n" \ + " -f Stop runner after first failure\n" \ + " -v Verbose output\n" \ + " -s SUITE only run suites containing string SUITE\n" \ + " -t TEST only run tests containing string TEST\n", \ + name); \ +} \ + \ +static void greatest_parse_args(int argc, char **argv) { \ + int i = 0; \ + for (i = 1; i < argc; i++) { \ + if (0 == strncmp("-t", argv[i], 2)) { \ + if (argc <= i + 1) { \ + greatest_usage(argv[0]); \ + exit(EXIT_FAILURE); \ + } \ + greatest_info.test_filter = argv[i+1]; \ + i++; \ + } else if (0 == strncmp("-s", argv[i], 2)) { \ + if (argc <= i + 1) { \ + greatest_usage(argv[0]); \ + exit(EXIT_FAILURE); \ + } \ + greatest_info.suite_filter = argv[i+1]; \ + i++; \ + } else if (0 == strncmp("-f", argv[i], 2)) { \ + greatest_info.flags |= GREATEST_FLAG_FIRST_FAIL; \ + } else if (0 == strncmp("-v", argv[i], 2)) { \ + greatest_info.verbosity++; \ + } else if (0 == strncmp("-l", argv[i], 2)) { \ + greatest_info.flags |= GREATEST_FLAG_LIST_ONLY; \ + } else if (0 == strncmp("-h", argv[i], 2) || \ + 0 == strncmp("--help", argv[i], 6)) { \ + greatest_usage(argv[0]); \ + exit(EXIT_SUCCESS); \ + } else if (0 == strncmp("--", argv[i], 2)) { \ + break; \ + } else { \ + fprintf(GREATEST_STDOUT, \ + "Unknown argument '%s'\n", argv[i]); \ + greatest_usage(argv[0]); \ + exit(EXIT_FAILURE); \ + } \ + } \ +} \ + \ +int greatest_all_passed(void) { return (greatest_info.failed == 0); } \ + \ +void greatest_set_test_filter(const char *name) { \ + greatest_info.test_filter = name; \ +} \ + \ +void greatest_set_suite_filter(const char *name) { \ + greatest_info.suite_filter = name; \ +} \ + \ +void greatest_get_report(struct greatest_report_t *report) { \ + if (report) { \ + report->passed = greatest_info.passed; \ + report->failed = greatest_info.failed; \ + report->skipped = greatest_info.skipped; \ + report->assertions = greatest_info.assertions; \ + } \ +} \ + \ +unsigned int greatest_get_verbosity(void) { \ + return greatest_info.verbosity; \ +} \ + \ +void greatest_set_verbosity(unsigned int verbosity) { \ + greatest_info.verbosity = (unsigned char)verbosity; \ +} \ + \ +void greatest_set_flag(greatest_flag_t flag) { \ + greatest_info.flags |= flag; \ +} \ + \ +void GREATEST_SET_SETUP_CB(greatest_setup_cb *cb, void *udata) { \ + greatest_info.setup = cb; \ + greatest_info.setup_udata = udata; \ +} \ + \ +void GREATEST_SET_TEARDOWN_CB(greatest_teardown_cb *cb, \ + void *udata) { \ + greatest_info.teardown = cb; \ + greatest_info.teardown_udata = udata; \ +} \ + \ +static int greatest_string_equal_cb(const void *exp, const void *got, \ + void *udata) { \ + size_t *size = (size_t *)udata; \ + return (size != NULL \ + ? (0 == strncmp((const char *)exp, (const char *)got, *size)) \ + : (0 == strcmp((const char *)exp, (const char *)got))); \ +} \ + \ +static int greatest_string_printf_cb(const void *t, void *udata) { \ + (void)udata; /* note: does not check \0 termination. */ \ + return fprintf(GREATEST_STDOUT, "%s", (const char *)t); \ +} \ + \ +greatest_type_info greatest_type_info_string = { \ + greatest_string_equal_cb, \ + greatest_string_printf_cb, \ +}; \ + \ +static int greatest_memory_equal_cb(const void *exp, const void *got, \ + void *udata) { \ + greatest_memory_cmp_env *env = (greatest_memory_cmp_env *)udata; \ + return (0 == memcmp(exp, got, env->size)); \ +} \ + \ +static int greatest_memory_printf_cb(const void *t, void *udata) { \ + greatest_memory_cmp_env *env = (greatest_memory_cmp_env *)udata; \ + unsigned char *buf = (unsigned char *)t, diff_mark = ' '; \ + FILE *out = GREATEST_STDOUT; \ + size_t i, line_i, line_len = 0; \ + int len = 0; /* format hexdump with differences highlighted */ \ + for (i = 0; i < env->size; i+= line_len) { \ + diff_mark = ' '; \ + line_len = env->size - i; \ + if (line_len > 16) { line_len = 16; } \ + for (line_i = i; line_i < i + line_len; line_i++) { \ + if (env->exp[line_i] != env->got[line_i]) diff_mark = 'X'; \ + } \ + len += fprintf(out, "\n%04x %c ", (unsigned int)i, diff_mark); \ + for (line_i = i; line_i < i + line_len; line_i++) { \ + int m = env->exp[line_i] == env->got[line_i]; /* match? */ \ + len += fprintf(out, "%02x%c", buf[line_i], m ? ' ' : '<'); \ + } \ + for (line_i = 0; line_i < 16 - line_len; line_i++) { \ + len += fprintf(out, " "); \ + } \ + fprintf(out, " "); \ + for (line_i = i; line_i < i + line_len; line_i++) { \ + unsigned char c = buf[line_i]; \ + len += fprintf(out, "%c", isprint(c) ? c : '.'); \ + } \ + } \ + len += fprintf(out, "\n"); \ + return len; \ +} \ + \ +greatest_type_info greatest_type_info_memory = { \ + greatest_memory_equal_cb, \ + greatest_memory_printf_cb, \ +}; \ + \ +greatest_run_info greatest_info + +/* Init internals. */ +#define GREATEST_INIT() \ + do { \ + /* Suppress unused function warning if features aren't used */ \ + (void)greatest_run_suite; \ + (void)greatest_parse_args; \ + \ + memset(&greatest_info, 0, sizeof(greatest_info)); \ + greatest_info.width = GREATEST_DEFAULT_WIDTH; \ + GREATEST_SET_TIME(greatest_info.begin); \ + } while (0) \ + +/* Handle command-line arguments, etc. */ +#define GREATEST_MAIN_BEGIN() \ + do { \ + GREATEST_INIT(); \ + greatest_parse_args(argc, argv); \ + } while (0) + +/* Report passes, failures, skipped tests, the number of + * assertions, and the overall run time. */ +#define GREATEST_PRINT_REPORT() \ + do { \ + if (!GREATEST_LIST_ONLY()) { \ + update_counts_and_reset_suite(); \ + GREATEST_SET_TIME(greatest_info.end); \ + fprintf(GREATEST_STDOUT, \ + "\nTotal: %u test%s", \ + greatest_info.tests_run, \ + greatest_info.tests_run == 1 ? "" : "s"); \ + GREATEST_CLOCK_DIFF(greatest_info.begin, \ + greatest_info.end); \ + fprintf(GREATEST_STDOUT, ", %u assertion%s\n", \ + greatest_info.assertions, \ + greatest_info.assertions == 1 ? "" : "s"); \ + fprintf(GREATEST_STDOUT, \ + "Pass: %u, fail: %u, skip: %u.\n", \ + greatest_info.passed, \ + greatest_info.failed, greatest_info.skipped); \ + } \ + } while (0) + +/* Report results, exit with exit status based on results. */ +#define GREATEST_MAIN_END() \ + do { \ + GREATEST_PRINT_REPORT(); \ + return (greatest_all_passed() ? EXIT_SUCCESS : EXIT_FAILURE); \ + } while (0) + +/* Make abbreviations without the GREATEST_ prefix for the + * most commonly used symbols. */ +#if GREATEST_USE_ABBREVS +#define TEST GREATEST_TEST +#define SUITE GREATEST_SUITE +#define SUITE_EXTERN GREATEST_SUITE_EXTERN +#define RUN_TEST GREATEST_RUN_TEST +#define RUN_TEST1 GREATEST_RUN_TEST1 +#define RUN_SUITE GREATEST_RUN_SUITE +#define IGNORE_TEST GREATEST_IGNORE_TEST +#define ASSERT GREATEST_ASSERT +#define ASSERTm GREATEST_ASSERTm +#define ASSERT_FALSE GREATEST_ASSERT_FALSE +#define ASSERT_EQ GREATEST_ASSERT_EQ +#define ASSERT_EQ_FMT GREATEST_ASSERT_EQ_FMT +#define ASSERT_IN_RANGE GREATEST_ASSERT_IN_RANGE +#define ASSERT_EQUAL_T GREATEST_ASSERT_EQUAL_T +#define ASSERT_STR_EQ GREATEST_ASSERT_STR_EQ +#define ASSERT_STRN_EQ GREATEST_ASSERT_STRN_EQ +#define ASSERT_MEM_EQ GREATEST_ASSERT_MEM_EQ +#define ASSERT_ENUM_EQ GREATEST_ASSERT_ENUM_EQ +#define ASSERT_FALSEm GREATEST_ASSERT_FALSEm +#define ASSERT_EQm GREATEST_ASSERT_EQm +#define ASSERT_EQ_FMTm GREATEST_ASSERT_EQ_FMTm +#define ASSERT_IN_RANGEm GREATEST_ASSERT_IN_RANGEm +#define ASSERT_EQUAL_Tm GREATEST_ASSERT_EQUAL_Tm +#define ASSERT_STR_EQm GREATEST_ASSERT_STR_EQm +#define ASSERT_STRN_EQm GREATEST_ASSERT_STRN_EQm +#define ASSERT_MEM_EQm GREATEST_ASSERT_MEM_EQm +#define ASSERT_ENUM_EQm GREATEST_ASSERT_ENUM_EQm +#define PASS GREATEST_PASS +#define FAIL GREATEST_FAIL +#define SKIP GREATEST_SKIP +#define PASSm GREATEST_PASSm +#define FAILm GREATEST_FAILm +#define SKIPm GREATEST_SKIPm +#define SET_SETUP GREATEST_SET_SETUP_CB +#define SET_TEARDOWN GREATEST_SET_TEARDOWN_CB +#define CHECK_CALL GREATEST_CHECK_CALL + +#ifdef GREATEST_VA_ARGS +#define RUN_TESTp GREATEST_RUN_TESTp +#endif + +#if GREATEST_USE_LONGJMP +#define ASSERT_OR_LONGJMP GREATEST_ASSERT_OR_LONGJMP +#define ASSERT_OR_LONGJMPm GREATEST_ASSERT_OR_LONGJMPm +#define FAIL_WITH_LONGJMP GREATEST_FAIL_WITH_LONGJMP +#define FAIL_WITH_LONGJMPm GREATEST_FAIL_WITH_LONGJMPm +#endif + +#endif /* USE_ABBREVS */ + +#ifdef __cplusplus +} +#endif + +#endif diff --git a/thirdparty/hiredis b/thirdparty/hiredis new file mode 160000 index 000000000..5f98e1d35 --- /dev/null +++ b/thirdparty/hiredis @@ -0,0 +1 @@ +Subproject commit 5f98e1d35dcf00a026793ada2662f6e1ba77eb17 diff --git a/thirdparty/utarray.h b/thirdparty/utarray.h new file mode 100644 index 000000000..979e99e98 --- /dev/null +++ b/thirdparty/utarray.h @@ -0,0 +1,238 @@ +/* +Copyright (c) 2008-2016, Troy D. Hanson http://troydhanson.github.com/uthash/ +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + + * Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS +IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED +TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A +PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER +OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, +EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, +PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR +PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF +LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING +NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +*/ + +/* a dynamic array implementation using macros + */ +#ifndef UTARRAY_H +#define UTARRAY_H + +#define UTARRAY_VERSION 2.0.1 + +#ifdef __GNUC__ +#define _UNUSED_ __attribute__ ((__unused__)) +#else +#define _UNUSED_ +#endif + +#include /* size_t */ +#include /* memset, etc */ +#include /* exit */ + +#ifndef oom +#define oom() exit(-1) +#endif + +typedef void (ctor_f)(void *dst, const void *src); +typedef void (dtor_f)(void *elt); +typedef void (init_f)(void *elt); +typedef struct { + size_t sz; + init_f *init; + ctor_f *copy; + dtor_f *dtor; +} UT_icd; + +typedef struct { + unsigned i,n;/* i: index of next available slot, n: num slots */ + UT_icd icd; /* initializer, copy and destructor functions */ + char *d; /* n slots of size icd->sz*/ +} UT_array; + +#define utarray_init(a,_icd) do { \ + memset(a,0,sizeof(UT_array)); \ + (a)->icd = *(_icd); \ +} while(0) + +#define utarray_done(a) do { \ + if ((a)->n) { \ + if ((a)->icd.dtor) { \ + unsigned _ut_i; \ + for(_ut_i=0; _ut_i < (a)->i; _ut_i++) { \ + (a)->icd.dtor(utarray_eltptr(a,_ut_i)); \ + } \ + } \ + free((a)->d); \ + } \ + (a)->n=0; \ +} while(0) + +#define utarray_new(a,_icd) do { \ + (a) = (UT_array*)malloc(sizeof(UT_array)); \ + if ((a) == NULL) oom(); \ + utarray_init(a,_icd); \ +} while(0) + +#define utarray_free(a) do { \ + utarray_done(a); \ + free(a); \ +} while(0) + +#define utarray_reserve(a,by) do { \ + if (((a)->i+(by)) > (a)->n) { \ + char *utarray_tmp; \ + while (((a)->i+(by)) > (a)->n) { (a)->n = ((a)->n ? (2*(a)->n) : 8); } \ + utarray_tmp=(char*)realloc((a)->d, (a)->n*(a)->icd.sz); \ + if (utarray_tmp == NULL) oom(); \ + (a)->d=utarray_tmp; \ + } \ +} while(0) + +#define utarray_push_back(a,p) do { \ + utarray_reserve(a,1); \ + if ((a)->icd.copy) { (a)->icd.copy( _utarray_eltptr(a,(a)->i++), p); } \ + else { memcpy(_utarray_eltptr(a,(a)->i++), p, (a)->icd.sz); }; \ +} while(0) + +#define utarray_pop_back(a) do { \ + if ((a)->icd.dtor) { (a)->icd.dtor( _utarray_eltptr(a,--((a)->i))); } \ + else { (a)->i--; } \ +} while(0) + +#define utarray_extend_back(a) do { \ + utarray_reserve(a,1); \ + if ((a)->icd.init) { (a)->icd.init(_utarray_eltptr(a,(a)->i)); } \ + else { memset(_utarray_eltptr(a,(a)->i),0,(a)->icd.sz); } \ + (a)->i++; \ +} while(0) + +#define utarray_len(a) ((a)->i) + +#define utarray_eltptr(a,j) (((j) < (a)->i) ? _utarray_eltptr(a,j) : NULL) +#define _utarray_eltptr(a,j) ((a)->d + ((a)->icd.sz * (j))) + +#define utarray_insert(a,p,j) do { \ + if ((j) > (a)->i) utarray_resize(a,j); \ + utarray_reserve(a,1); \ + if ((j) < (a)->i) { \ + memmove( _utarray_eltptr(a,(j)+1), _utarray_eltptr(a,j), \ + ((a)->i - (j))*((a)->icd.sz)); \ + } \ + if ((a)->icd.copy) { (a)->icd.copy( _utarray_eltptr(a,j), p); } \ + else { memcpy(_utarray_eltptr(a,j), p, (a)->icd.sz); }; \ + (a)->i++; \ +} while(0) + +#define utarray_inserta(a,w,j) do { \ + if (utarray_len(w) == 0) break; \ + if ((j) > (a)->i) utarray_resize(a,j); \ + utarray_reserve(a,utarray_len(w)); \ + if ((j) < (a)->i) { \ + memmove(_utarray_eltptr(a,(j)+utarray_len(w)), \ + _utarray_eltptr(a,j), \ + ((a)->i - (j))*((a)->icd.sz)); \ + } \ + if ((a)->icd.copy) { \ + unsigned _ut_i; \ + for(_ut_i=0;_ut_i<(w)->i;_ut_i++) { \ + (a)->icd.copy(_utarray_eltptr(a, (j) + _ut_i), _utarray_eltptr(w, _ut_i)); \ + } \ + } else { \ + memcpy(_utarray_eltptr(a,j), _utarray_eltptr(w,0), \ + utarray_len(w)*((a)->icd.sz)); \ + } \ + (a)->i += utarray_len(w); \ +} while(0) + +#define utarray_resize(dst,num) do { \ + unsigned _ut_i; \ + if ((dst)->i > (unsigned)(num)) { \ + if ((dst)->icd.dtor) { \ + for (_ut_i = (num); _ut_i < (dst)->i; ++_ut_i) { \ + (dst)->icd.dtor(_utarray_eltptr(dst, _ut_i)); \ + } \ + } \ + } else if ((dst)->i < (unsigned)(num)) { \ + utarray_reserve(dst, (num) - (dst)->i); \ + if ((dst)->icd.init) { \ + for (_ut_i = (dst)->i; _ut_i < (unsigned)(num); ++_ut_i) { \ + (dst)->icd.init(_utarray_eltptr(dst, _ut_i)); \ + } \ + } else { \ + memset(_utarray_eltptr(dst, (dst)->i), 0, (dst)->icd.sz*((num) - (dst)->i)); \ + } \ + } \ + (dst)->i = (num); \ +} while(0) + +#define utarray_concat(dst,src) do { \ + utarray_inserta(dst, src, utarray_len(dst)); \ +} while(0) + +#define utarray_erase(a,pos,len) do { \ + if ((a)->icd.dtor) { \ + unsigned _ut_i; \ + for (_ut_i = 0; _ut_i < (len); _ut_i++) { \ + (a)->icd.dtor(utarray_eltptr(a, (pos) + _ut_i)); \ + } \ + } \ + if ((a)->i > ((pos) + (len))) { \ + memmove(_utarray_eltptr(a, pos), _utarray_eltptr(a, (pos) + (len)), \ + ((a)->i - ((pos) + (len))) * (a)->icd.sz); \ + } \ + (a)->i -= (len); \ +} while(0) + +#define utarray_renew(a,u) do { \ + if (a) utarray_clear(a); \ + else utarray_new(a, u); \ +} while(0) + +#define utarray_clear(a) do { \ + if ((a)->i > 0) { \ + if ((a)->icd.dtor) { \ + unsigned _ut_i; \ + for(_ut_i=0; _ut_i < (a)->i; _ut_i++) { \ + (a)->icd.dtor(_utarray_eltptr(a, _ut_i)); \ + } \ + } \ + (a)->i = 0; \ + } \ +} while(0) + +#define utarray_sort(a,cmp) do { \ + qsort((a)->d, (a)->i, (a)->icd.sz, cmp); \ +} while(0) + +#define utarray_find(a,v,cmp) bsearch((v),(a)->d,(a)->i,(a)->icd.sz,cmp) + +#define utarray_front(a) (((a)->i) ? (_utarray_eltptr(a,0)) : NULL) +#define utarray_next(a,e) (((e)==NULL) ? utarray_front(a) : ((((a)->i) > (utarray_eltidx(a,e)+1)) ? _utarray_eltptr(a,utarray_eltidx(a,e)+1) : NULL)) +#define utarray_prev(a,e) (((e)==NULL) ? utarray_back(a) : ((utarray_eltidx(a,e) > 0) ? _utarray_eltptr(a,utarray_eltidx(a,e)-1) : NULL)) +#define utarray_back(a) (((a)->i) ? (_utarray_eltptr(a,(a)->i-1)) : NULL) +#define utarray_eltidx(a,e) (((char*)(e) >= (a)->d) ? (((char*)(e) - (a)->d)/(a)->icd.sz) : -1) + +/* last we pre-define a few icd for common utarrays of ints and strings */ +static void utarray_str_cpy(void *dst, const void *src) { + char **_src = (char**)src, **_dst = (char**)dst; + *_dst = (*_src == NULL) ? NULL : strdup(*_src); +} +static void utarray_str_dtor(void *elt) { + char **eltc = (char**)elt; + if (*eltc != NULL) free(*eltc); +} +static const UT_icd ut_str_icd _UNUSED_ = {sizeof(char*),NULL,utarray_str_cpy,utarray_str_dtor}; +static const UT_icd ut_int_icd _UNUSED_ = {sizeof(int),NULL,NULL,NULL}; +static const UT_icd ut_ptr_icd _UNUSED_ = {sizeof(void*),NULL,NULL,NULL}; + + +#endif /* UTARRAY_H */ From 73f4b962535bc0573f5c8c7159f4b20b0324213f Mon Sep 17 00:00:00 2001 From: Stephanie Wang Date: Thu, 15 Sep 2016 16:28:52 -0700 Subject: [PATCH 03/36] Sockets (#3) * Socket methods to be used by an event loop * Git ignore build files * File renames * Some fixes * Fixes * Fixes * Memory leakage fix --- .gitignore | 3 ++ Makefile | 8 +++- event_loop.c | 1 - sockets.c | 111 ++++++++++++++++++++++++++++++++++++++++++++ sockets.h | 10 ++++ test/socket_tests.c | 45 ++++++++++++++++++ 6 files changed, 175 insertions(+), 3 deletions(-) create mode 100644 sockets.c create mode 100644 sockets.h create mode 100644 test/socket_tests.c diff --git a/.gitignore b/.gitignore index 2a07abca4..fff8ef269 100644 --- a/.gitignore +++ b/.gitignore @@ -33,3 +33,6 @@ # Debug files *.dSYM/ *.su + +# Build files +build/* diff --git a/Makefile b/Makefile index fd9da0b97..8ca748e1d 100644 --- a/Makefile +++ b/Makefile @@ -9,6 +9,9 @@ CFLAGS += -Wmissing-declarations $(BUILD)/db_tests: hiredis test/db_tests.c thirdparty/greatest.h event_loop.c state/redis.c common.c $(CC) -o $@ test/db_tests.c event_loop.c state/redis.c common.c thirdparty/hiredis/libhiredis.a $(CFLAGS) -I. -Ithirdparty +$(BUILD)/socket_tests: test/socket_tests.c thirdparty/greatest.h sockets.c + $(CC) -o $@ test/socket_tests.c sockets.c $(CFLAGS) -I. -Ithirdparty + clean: rm -r $(BUILD)/* @@ -18,7 +21,8 @@ redis: hiredis: git submodule update --init --recursive -- "thirdparty/hiredis" ; cd thirdparty/hiredis ; make -test: hiredis redis $(BUILD)/db_tests FORCE - ./thirdparty/redis-3.2.3/src/redis-server & sleep 1s ; ./build/db_tests +test: hiredis redis $(BUILD)/db_tests $(BUILD)/socket_tests FORCE + ./thirdparty/redis-3.2.3/src/redis-server & + sleep 1s ; ./build/db_tests ; ./build/socket_tests FORCE: diff --git a/event_loop.c b/event_loop.c index d7169f5d8..ebc6ebc13 100644 --- a/event_loop.c +++ b/event_loop.c @@ -1,7 +1,6 @@ #include "event_loop.h" #include -#include UT_icd item_icd = {sizeof(event_loop_item), NULL, NULL, NULL}; UT_icd poll_icd = {sizeof(struct pollfd), NULL, NULL, NULL}; diff --git a/sockets.c b/sockets.c new file mode 100644 index 000000000..6fd41e476 --- /dev/null +++ b/sockets.c @@ -0,0 +1,111 @@ +#include "sockets.h" + +#include +#include +#include +#include +#include +#include + +#include "common.h" + +/* Binds to a Unix domain datagram socket at the given + * pathname. Removes any existing file at the pathname. Returns + * a file descriptor for the socket, or -1 if an error + * occurred. */ +int bind_ipc_sock(const char *socket_pathname) { + struct sockaddr_un socket_address; + int socket_fd; + + socket_fd = socket(AF_UNIX, SOCK_DGRAM, 0); + if (socket_fd < 0) { + LOG_ERR("socket() failed for pathname %s.", socket_pathname); + return -1; + } + + unlink(socket_pathname); + memset(&socket_address, 0, sizeof(struct sockaddr_un)); + socket_address.sun_family = AF_UNIX; + if (strlen(socket_pathname) + 1 > sizeof(socket_address.sun_path)) { + LOG_ERR("Socket pathname is too long."); + return -1; + } + strncpy(socket_address.sun_path, socket_pathname, + strlen(socket_pathname) + 1); + + if (bind(socket_fd, (struct sockaddr *) &socket_address, + sizeof(struct sockaddr_un)) != 0) { + LOG_ERR("Bind failed for pathname %s.", socket_pathname); + return -1; + } + + return socket_fd; +} + +/* Connects to a Unix domain datagram socket at the given + * pathname. Returns a file descriptor for the socket, or -1 if + * an error occurred. */ +int connect_ipc_sock(const char *socket_pathname) { + struct sockaddr_un socket_address; + int socket_fd; + + socket_fd = socket(AF_UNIX, SOCK_DGRAM, 0); + if (socket_fd < 0) { + LOG_ERR("socket() failed for pathname %s.", socket_pathname); + return -1; + } + + memset(&socket_address, 0, sizeof(struct sockaddr_un)); + socket_address.sun_family = AF_UNIX; + if (strlen(socket_pathname) + 1 > sizeof(socket_address.sun_path)) { + LOG_ERR("Socket pathname is too long."); + return -1; + } + strncpy(socket_address.sun_path, socket_pathname, + strlen(socket_pathname) + 1); + + if (connect(socket_fd, (struct sockaddr *) &socket_address, + sizeof(struct sockaddr_un)) != 0) { + LOG_ERR("Connection to socket failed for pathname %s.", socket_pathname); + return -1; + } + + return socket_fd; +} + +/* Sends a message on the given socket file descriptor. */ +void send_ipc_sock(int socket_fd, char *message) { + int length = strlen(message); + int nbytes; + nbytes = send(socket_fd, (char *) &length, sizeof(length), 0); + if (nbytes == -1) { + fprintf(stderr, "Error sending to socket.\n"); + return; + } + nbytes = send(socket_fd, (char *) message, length * sizeof(char), 0); + if (nbytes == -1) { + fprintf(stderr, "Error sending to socket.\n"); + return; + } +} + +/* Receives a message on the given socket file descriptor. Allocates and + * returns a pointer to the message. + * NOTE: Caller must free the message! */ +char *recv_ipc_sock(int socket_fd) { + int length; + int nbytes; + nbytes = recv(socket_fd, &length, sizeof(length), 0); + if (nbytes == -1) { + fprintf(stderr, "Error receiving from socket.\n"); + return NULL; + } + char *message = malloc((length + 1) * sizeof(char)); + nbytes = recv(socket_fd, message, length * sizeof(char), 0); + if (nbytes == -1) { + fprintf(stderr, "Error receiving from socket.\n"); + return NULL; + } + message[length] = '\0'; + return message; +} diff --git a/sockets.h b/sockets.h new file mode 100644 index 000000000..7ad0c7141 --- /dev/null +++ b/sockets.h @@ -0,0 +1,10 @@ +#ifndef SOCKETS_H +#define SOCKETS_H + +/* Helper functions for socket communication. */ +int bind_ipc_sock(const char* socket_pathname); +int connect_ipc_sock(const char* socket_pathname); +void send_ipc_sock(int socket_fd, char* message); +char* recv_ipc_sock(int socket_fd); + +#endif diff --git a/test/socket_tests.c b/test/socket_tests.c new file mode 100644 index 000000000..1fa352246 --- /dev/null +++ b/test/socket_tests.c @@ -0,0 +1,45 @@ +#include "greatest.h" + +#include +#include + +#include "sockets.h" + +SUITE(event_loop_tests); + +TEST ipc_socket_test(void) { + const char* socket_pathname = "test-socket"; + int socket_fd = bind_ipc_sock(socket_pathname); + ASSERT(socket_fd >= 0); + + char* test_string = "hello world"; + pid_t pid = fork(); + if (pid == 0) { + close(socket_fd); + socket_fd = connect_ipc_sock(socket_pathname); + ASSERT(socket_fd >= 0); + send_ipc_sock(socket_fd, test_string); + close(socket_fd); + } else { + char* message = recv_ipc_sock(socket_fd); + ASSERT(message != NULL); + ASSERT_STR_EQ(test_string, message); + free(message); + close(socket_fd); + unlink(socket_pathname); + } + + PASS(); +} + +SUITE(event_loop_tests) { + RUN_TEST(ipc_socket_test); +} + +GREATEST_MAIN_DEFS(); + +int main(int argc, char** argv) { + GREATEST_MAIN_BEGIN(); + RUN_SUITE(event_loop_tests); + GREATEST_MAIN_END(); +} From 0b7d81cae682b7445c4a38c8979ab8b835b8f54b Mon Sep 17 00:00:00 2001 From: Philipp Moritz Date: Sat, 17 Sep 2016 00:03:10 -0700 Subject: [PATCH 04/36] API for creating task specifications (#5) * API for creating task specifications * fixes * add more checks and improve comments --- Makefile | 7 ++- common.h | 8 +++ task.c | 133 ++++++++++++++++++++++++++++++++++++++++++++++ task.h | 53 ++++++++++++++++++ test/task_tests.c | 54 +++++++++++++++++++ 5 files changed, 253 insertions(+), 2 deletions(-) create mode 100644 task.c create mode 100644 task.h create mode 100644 test/task_tests.c diff --git a/Makefile b/Makefile index 8ca748e1d..d02a9cb53 100644 --- a/Makefile +++ b/Makefile @@ -12,6 +12,9 @@ $(BUILD)/db_tests: hiredis test/db_tests.c thirdparty/greatest.h event_loop.c st $(BUILD)/socket_tests: test/socket_tests.c thirdparty/greatest.h sockets.c $(CC) -o $@ test/socket_tests.c sockets.c $(CFLAGS) -I. -Ithirdparty +$(BUILD)/task_tests: test/task_tests.c task.c sockets.c common.h + $(CC) -o $@ test/task_tests.c task.c sockets.c $(CFLAGS) -I. -Ithirdparty + clean: rm -r $(BUILD)/* @@ -21,8 +24,8 @@ redis: hiredis: git submodule update --init --recursive -- "thirdparty/hiredis" ; cd thirdparty/hiredis ; make -test: hiredis redis $(BUILD)/db_tests $(BUILD)/socket_tests FORCE +test: hiredis redis $(BUILD)/db_tests $(BUILD)/socket_tests $(BUILD)/task_tests FORCE ./thirdparty/redis-3.2.3/src/redis-server & - sleep 1s ; ./build/db_tests ; ./build/socket_tests + sleep 1s ; ./build/db_tests ; ./build/socket_tests ; ./build/task_tests FORCE: diff --git a/common.h b/common.h index 61abc7b50..d38556dcf 100644 --- a/common.h +++ b/common.h @@ -17,6 +17,14 @@ #define LOG_INFO(M, ...) \ fprintf(stderr, "[INFO] (%s:%d) " M "\n", __FILE__, __LINE__, ##__VA_ARGS__) +#define CHECK(COND) \ + do { \ + if (!(COND)) { \ + LOG_ERR("Check failure: %s", #COND); \ + exit(-1); \ + } \ + } while (0); + #define UNIQUE_ID_SIZE 20 typedef struct { unsigned char id[UNIQUE_ID_SIZE]; } unique_id; diff --git a/task.c b/task.c new file mode 100644 index 000000000..c337cc1a3 --- /dev/null +++ b/task.c @@ -0,0 +1,133 @@ +#include +#include +#include + +#include "task.h" +#include "common.h" +#include "sockets.h" + +/* Tasks are stored in a consecutive chunk of memory, the first + * sizeof(task_spec) bytes are arranged according to the struct + * task_spec. Then there is an array of task_args of length + * (num_args + num_returns), and then follows the data of + * pass-by-value arguments of size args_value_size. The offsets in the + * task_arg.val are with respect to the end of the augmented structure, + * i.e. with respect to the address &task_spec.args_and_returns[0] + + * (task_spec->num_args + task_spec->num_returns) * sizeof(task_arg). */ + +typedef struct { + /* Either ARG_BY_REF or ARG_BY_VAL. */ + int8_t type; + union { + object_id obj_id; + struct { + /* Offset where the data associated to this arg is located relative + * to &task_spec.args_and_returns[0]. */ + ptrdiff_t offset; + int64_t length; + } value; + }; +} task_arg; + +struct task_spec_impl { + function_id func_id; + /* Total number of arguments. */ + int64_t num_args; + /* Index of the last argument that has been constructed. */ + int64_t arg_index; + /* Number of return values. */ + int64_t num_returns; + /* Number of bytes the pass-by-value arguments are occupying. */ + int64_t args_value_size; + /* The offset of the number of bytes of pass-by-value data that + * has been written so far, relative to &task_spec->args_and_returns[0] + + * (task_spec->num_args + task_spec->num_returns) * sizeof(task_arg) */ + int64_t args_value_offset; + /* Argument and return IDs as well as offsets for pass-by-value args. */ + task_arg args_and_returns[0]; +}; + +task_spec *alloc_task_spec(function_id func_id, + int64_t num_args, + int64_t num_returns, + int64_t args_value_size) { + int64_t size = sizeof(task_spec) + + (num_args + num_returns) * sizeof(task_arg) + args_value_size; + task_spec *task = malloc(size); + memset(task, 0, size); + task->func_id = func_id; + task->num_args = num_args; + task->arg_index = 0; + task->num_returns = num_returns; + task->args_value_size = args_value_size; + return task; +} + +int64_t task_num_args(task_spec *spec) { + return spec->num_args; +} + +int64_t task_num_returns(task_spec *spec) { + return spec->num_returns; +} + +int8_t task_arg_type(task_spec *spec, int64_t arg_index) { + CHECK(0 <= arg_index && arg_index < spec->num_args); + return spec->args_and_returns[arg_index].type; +} + +object_id *task_arg_id(task_spec *spec, int64_t arg_index) { + CHECK(0 <= arg_index && arg_index < spec->num_args); + task_arg *arg = &spec->args_and_returns[arg_index]; + CHECK(arg->type == ARG_BY_REF) + return &arg->obj_id; +} + +uint8_t *task_arg_val(task_spec *spec, int64_t arg_index) { + CHECK(0 <= arg_index && arg_index < spec->num_args); + task_arg *arg = &spec->args_and_returns[arg_index]; + CHECK(arg->type == ARG_BY_VAL); + uint8_t *data = (uint8_t *) &spec->args_and_returns[0]; + data += (spec->num_args + spec->num_returns) * sizeof(task_arg); + return data + arg->value.offset; +} + +int64_t task_arg_length(task_spec *spec, int64_t arg_index) { + CHECK(0 <= arg_index && arg_index < spec->num_args); + task_arg *arg = &spec->args_and_returns[arg_index]; + CHECK(arg->type == ARG_BY_VAL); + return arg->value.length; +} + +int64_t task_args_add_ref(task_spec *spec, object_id obj_id) { + task_arg *arg = &spec->args_and_returns[spec->arg_index]; + arg->type = ARG_BY_REF; + arg->obj_id = obj_id; + return spec->arg_index++; +} + +int64_t task_args_add_val(task_spec *spec, uint8_t *data, int64_t length) { + task_arg *arg = &spec->args_and_returns[spec->arg_index]; + arg->type = ARG_BY_VAL; + arg->value.offset = spec->args_value_offset; + arg->value.length = length; + uint8_t *addr = task_arg_val(spec, spec->arg_index); + CHECK(spec->args_value_offset + length <= spec->args_value_size); + CHECK(spec->arg_index != spec->num_args - 1 || + spec->args_value_offset + length == spec->args_value_size); + memcpy(addr, data, length); + spec->args_value_offset += length; + return spec->arg_index++; +} + +object_id *task_return(task_spec *spec, int64_t ret_index) { + CHECK(0 <= ret_index && ret_index < spec->num_returns); + task_arg *ret = &spec->args_and_returns[spec->num_args + ret_index]; + CHECK(ret->type == ARG_BY_REF); /* No memory corruption. */ + return &ret->obj_id; +} + +void free_task_spec(task_spec *spec) { + CHECK(spec->arg_index == spec->num_args); /* Task was fully constructed */ + free(spec); +} diff --git a/task.h b/task.h new file mode 100644 index 000000000..fbf557fe0 --- /dev/null +++ b/task.h @@ -0,0 +1,53 @@ +/* This API specifies the task data structure. It is in C so we can + * easily construct tasks from other languages like Python. The datastructures + * are also defined in such a way that memory is contiguous and all pointers + * are relative, so that we can memcpy the datastructure and ship it over the + * network without serialization and deserialization. */ + +#include +#include +#include "common.h" + +typedef unique_id function_id; +typedef unique_id object_id; + +typedef struct task_spec_impl task_spec; + +/* If argument is passed by value or reference. */ +enum arg_type { ARG_BY_REF, ARG_BY_VAL }; + +/* Construct and modify task specifications. */ + +/* Allocating and initializing a task. */ +task_spec *alloc_task_spec(function_id func_id, + int64_t num_args, + int64_t num_returns, + int64_t args_value_size); + +/* Getting the number of arguments and returns. */ +int64_t task_num_args(task_spec *spec); +int64_t task_num_returns(task_spec *spec); + +/* Getting task arguments. */ +int8_t task_arg_type(task_spec *spec, int64_t arg_index); +unique_id *task_arg_id(task_spec *spec, int64_t arg_index); +uint8_t *task_arg_val(task_spec *spec, int64_t arg_index); +int64_t task_arg_length(task_spec *spec, int64_t arg_index); + +/* Setting task arguments. Note that this API only allows you to set the + * arguments in their order of appearance. */ +int64_t task_args_add_ref(task_spec *spec, object_id obj_id); +int64_t task_args_add_val(task_spec *spec, uint8_t *data, int64_t length); + +/* Getting and setting return arguments. Tasks return by reference for now. */ +unique_id *task_return(task_spec *spec, int64_t ret_index); + +/* Freeing the task datastructure. */ +void free_task_spec(task_spec *spec); + +/* Write the task specification to a file or socket. */ +int send_task(int fd, task_spec *spec); + +/* Read the task specification from a file or socket. It is the user's + * responsibility to free the task after it has been used. */ +task_spec *recv_task(int fd); diff --git a/test/task_tests.c b/test/task_tests.c new file mode 100644 index 000000000..d8443e68c --- /dev/null +++ b/test/task_tests.c @@ -0,0 +1,54 @@ +#include "greatest.h" + +#include "task.h" + +SUITE(task_tests); + +TEST task_test(void) { + function_id func_id = { + {1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1}}; + task_spec* task = alloc_task_spec(func_id, 4, 2, 10); + ASSERT(task_num_args(task) == 4); + ASSERT(task_num_returns(task) == 2); + + unique_id arg1 = { + {2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2}}; + ASSERT(task_args_add_ref(task, arg1) == 0); + ASSERT(task_args_add_val(task, (uint8_t*) "hello", 5) == 1); + unique_id arg2 = { + {3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3}}; + ASSERT(task_args_add_ref(task, arg2) == 2); + ASSERT(task_args_add_val(task, (uint8_t*) "world", 5) == 3); + + unique_id ret0 = { + {4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4}}; + unique_id ret1 = { + {5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5}}; + memcpy(task_return(task, 0), &ret0, sizeof(ret0)); + memcpy(task_return(task, 1), &ret1, sizeof(ret1)); + + ASSERT(memcmp(task_arg_id(task, 0), &arg1, sizeof(arg1)) == 0); + ASSERT(memcmp(task_arg_val(task, 1), (uint8_t*) "hello", + task_arg_length(task, 1)) == 0); + ASSERT(memcmp(task_arg_id(task, 2), &arg2, sizeof(arg2)) == 0); + ASSERT(memcmp(task_arg_val(task, 3), (uint8_t*) "world", + task_arg_length(task, 3)) == 0); + + ASSERT(memcmp(task_return(task, 0), &ret0, sizeof(unique_id)) == 0); + ASSERT(memcmp(task_return(task, 1), &ret1, sizeof(unique_id)) == 0); + + free_task_spec(task); + PASS(); +} + +SUITE(task_tests) { + RUN_TEST(task_test); +} + +GREATEST_MAIN_DEFS(); + +int main(int argc, char** argv) { + GREATEST_MAIN_BEGIN(); + RUN_SUITE(task_tests); + GREATEST_MAIN_END(); +} From ff11ee21efc11b341e3300d5140509e2088a992d Mon Sep 17 00:00:00 2001 From: Stephanie Wang Date: Sat, 17 Sep 2016 15:15:18 -0700 Subject: [PATCH 05/36] Convert to streaming sockets (#7) * Convert to streaming sockets * Formatting --- sockets.c | 42 ++++++++++++++++++++++++++++++------------ sockets.h | 1 + test/socket_tests.c | 12 ++++++++---- 3 files changed, 39 insertions(+), 16 deletions(-) diff --git a/sockets.c b/sockets.c index 6fd41e476..82ca490f7 100644 --- a/sockets.c +++ b/sockets.c @@ -17,7 +17,7 @@ int bind_ipc_sock(const char *socket_pathname) { struct sockaddr_un socket_address; int socket_fd; - socket_fd = socket(AF_UNIX, SOCK_DGRAM, 0); + socket_fd = socket(AF_UNIX, SOCK_STREAM, 0); if (socket_fd < 0) { LOG_ERR("socket() failed for pathname %s.", socket_pathname); return -1; @@ -38,6 +38,7 @@ int bind_ipc_sock(const char *socket_pathname) { LOG_ERR("Bind failed for pathname %s.", socket_pathname); return -1; } + listen(socket_fd, 5); return socket_fd; } @@ -49,7 +50,7 @@ int connect_ipc_sock(const char *socket_pathname) { struct sockaddr_un socket_address; int socket_fd; - socket_fd = socket(AF_UNIX, SOCK_DGRAM, 0); + socket_fd = socket(AF_UNIX, SOCK_STREAM, 0); if (socket_fd < 0) { LOG_ERR("socket() failed for pathname %s.", socket_pathname); return -1; @@ -77,33 +78,50 @@ int connect_ipc_sock(const char *socket_pathname) { void send_ipc_sock(int socket_fd, char *message) { int length = strlen(message); int nbytes; - nbytes = send(socket_fd, (char *) &length, sizeof(length), 0); + nbytes = write(socket_fd, (char *) &length, sizeof(length)); if (nbytes == -1) { - fprintf(stderr, "Error sending to socket.\n"); + LOG_ERR("Error sending to socket.\n"); return; } - nbytes = send(socket_fd, (char *) message, length * sizeof(char), 0); + nbytes = write(socket_fd, (char *) message, length * sizeof(char)); if (nbytes == -1) { - fprintf(stderr, "Error sending to socket.\n"); + LOG_ERR("Error sending to socket.\n"); return; } } +/* Accept a new client connection on the given socket + * descriptor. Returns a descriptor for the new socket. */ +int accept_client(int socket_fd) { + struct sockaddr_un client_addr; + int client_fd, client_len; + client_len = sizeof(client_addr); + client_fd = accept(socket_fd, (struct sockaddr *) &client_addr, + (socklen_t *) &client_len); + if (client_fd < 0) { + LOG_ERR("Error reading from socket."); + return -1; + } + return client_fd; +} + /* Receives a message on the given socket file descriptor. Allocates and * returns a pointer to the message. * NOTE: Caller must free the message! */ char *recv_ipc_sock(int socket_fd) { int length; int nbytes; - nbytes = recv(socket_fd, &length, sizeof(length), 0); - if (nbytes == -1) { - fprintf(stderr, "Error receiving from socket.\n"); + nbytes = read(socket_fd, &length, sizeof(length)); + if (nbytes < 0) { + LOG_ERR("Error reading length of message from socket."); return NULL; } + char *message = malloc((length + 1) * sizeof(char)); - nbytes = recv(socket_fd, message, length * sizeof(char), 0); - if (nbytes == -1) { - fprintf(stderr, "Error receiving from socket.\n"); + nbytes = read(socket_fd, message, length); + if (nbytes < 0) { + LOG_ERR("Error reading message from socket."); + free(message); return NULL; } message[length] = '\0'; diff --git a/sockets.h b/sockets.h index 7ad0c7141..a563470c9 100644 --- a/sockets.h +++ b/sockets.h @@ -5,6 +5,7 @@ int bind_ipc_sock(const char* socket_pathname); int connect_ipc_sock(const char* socket_pathname); void send_ipc_sock(int socket_fd, char* message); +int accept_client(int socket_fd); char* recv_ipc_sock(int socket_fd); #endif diff --git a/test/socket_tests.c b/test/socket_tests.c index 1fa352246..b57ab9000 100644 --- a/test/socket_tests.c +++ b/test/socket_tests.c @@ -5,7 +5,7 @@ #include "sockets.h" -SUITE(event_loop_tests); +SUITE(socket_tests); TEST ipc_socket_test(void) { const char* socket_pathname = "test-socket"; @@ -20,11 +20,15 @@ TEST ipc_socket_test(void) { ASSERT(socket_fd >= 0); send_ipc_sock(socket_fd, test_string); close(socket_fd); + exit(0); } else { - char* message = recv_ipc_sock(socket_fd); + int client_fd = accept_client(socket_fd); + ASSERT(client_fd >= 0); + char* message = recv_ipc_sock(client_fd); ASSERT(message != NULL); ASSERT_STR_EQ(test_string, message); free(message); + close(client_fd); close(socket_fd); unlink(socket_pathname); } @@ -32,7 +36,7 @@ TEST ipc_socket_test(void) { PASS(); } -SUITE(event_loop_tests) { +SUITE(socket_tests) { RUN_TEST(ipc_socket_test); } @@ -40,6 +44,6 @@ GREATEST_MAIN_DEFS(); int main(int argc, char** argv) { GREATEST_MAIN_BEGIN(); - RUN_SUITE(event_loop_tests); + RUN_SUITE(socket_tests); GREATEST_MAIN_END(); } From b18f214d555d353daee304b69718e0acc2e8440d Mon Sep 17 00:00:00 2001 From: Philipp Moritz Date: Sun, 18 Sep 2016 13:35:43 -0700 Subject: [PATCH 06/36] Make it possible to read and write data that is not null-terminated (#9) * Make it possible to read and write data that is not null-terminated * formating --- Makefile | 12 ++--- sockets.c => io.c | 74 +++++++++++++++++------------ io.h | 21 ++++++++ sockets.h | 11 ----- task.c | 2 +- test/{socket_tests.c => io_tests.c} | 25 ++++++---- 6 files changed, 87 insertions(+), 58 deletions(-) rename sockets.c => io.c (70%) create mode 100644 io.h delete mode 100644 sockets.h rename test/{socket_tests.c => io_tests.c} (57%) diff --git a/Makefile b/Makefile index d02a9cb53..e1c894737 100644 --- a/Makefile +++ b/Makefile @@ -9,11 +9,11 @@ CFLAGS += -Wmissing-declarations $(BUILD)/db_tests: hiredis test/db_tests.c thirdparty/greatest.h event_loop.c state/redis.c common.c $(CC) -o $@ test/db_tests.c event_loop.c state/redis.c common.c thirdparty/hiredis/libhiredis.a $(CFLAGS) -I. -Ithirdparty -$(BUILD)/socket_tests: test/socket_tests.c thirdparty/greatest.h sockets.c - $(CC) -o $@ test/socket_tests.c sockets.c $(CFLAGS) -I. -Ithirdparty +$(BUILD)/io_tests: test/io_tests.c thirdparty/greatest.h io.c + $(CC) -o $@ test/io_tests.c io.c $(CFLAGS) -I. -Ithirdparty -$(BUILD)/task_tests: test/task_tests.c task.c sockets.c common.h - $(CC) -o $@ test/task_tests.c task.c sockets.c $(CFLAGS) -I. -Ithirdparty +$(BUILD)/task_tests: test/task_tests.c task.c io.c common.h + $(CC) -o $@ test/task_tests.c task.c io.c $(CFLAGS) -I. -Ithirdparty clean: rm -r $(BUILD)/* @@ -24,8 +24,8 @@ redis: hiredis: git submodule update --init --recursive -- "thirdparty/hiredis" ; cd thirdparty/hiredis ; make -test: hiredis redis $(BUILD)/db_tests $(BUILD)/socket_tests $(BUILD)/task_tests FORCE +test: hiredis redis $(BUILD)/db_tests $(BUILD)/io_tests $(BUILD)/task_tests FORCE ./thirdparty/redis-3.2.3/src/redis-server & - sleep 1s ; ./build/db_tests ; ./build/socket_tests ; ./build/task_tests + sleep 1s ; ./build/db_tests ; ./build/io_tests ; ./build/task_tests FORCE: diff --git a/sockets.c b/io.c similarity index 70% rename from sockets.c rename to io.c index 82ca490f7..1d16a78e0 100644 --- a/sockets.c +++ b/io.c @@ -1,4 +1,4 @@ -#include "sockets.h" +#include "io.h" #include #include @@ -6,6 +6,7 @@ #include #include #include +#include #include "common.h" @@ -74,22 +75,6 @@ int connect_ipc_sock(const char *socket_pathname) { return socket_fd; } -/* Sends a message on the given socket file descriptor. */ -void send_ipc_sock(int socket_fd, char *message) { - int length = strlen(message); - int nbytes; - nbytes = write(socket_fd, (char *) &length, sizeof(length)); - if (nbytes == -1) { - LOG_ERR("Error sending to socket.\n"); - return; - } - nbytes = write(socket_fd, (char *) message, length * sizeof(char)); - if (nbytes == -1) { - LOG_ERR("Error sending to socket.\n"); - return; - } -} - /* Accept a new client connection on the given socket * descriptor. Returns a descriptor for the new socket. */ int accept_client(int socket_fd) { @@ -105,25 +90,52 @@ int accept_client(int socket_fd) { return client_fd; } -/* Receives a message on the given socket file descriptor. Allocates and - * returns a pointer to the message. - * NOTE: Caller must free the message! */ -char *recv_ipc_sock(int socket_fd) { - int length; - int nbytes; - nbytes = read(socket_fd, &length, sizeof(length)); +/* Write a sequence of bytes on a file descriptor. */ +void write_bytes(int fd, uint8_t *bytes, int64_t length) { + ssize_t nbytes = write(fd, (char *) &length, sizeof(length)); + if (nbytes == -1) { + LOG_ERR("Error sending to socket.\n"); + return; + } + nbytes = write(fd, (char *) bytes, length * sizeof(char)); + if (nbytes == -1) { + LOG_ERR("Error sending to socket.\n"); + return; + } +} + +/* Read a sequence of bytes written by write_bytes from a file descriptor. + * Allocates and returns a pointer to the bytes. + * NOTE: Caller must free the memory! */ +void read_bytes(int fd, uint8_t **bytes, int64_t *length) { + ssize_t nbytes = read(fd, length, sizeof(int64_t)); if (nbytes < 0) { LOG_ERR("Error reading length of message from socket."); - return NULL; + *bytes = NULL; + return; } - char *message = malloc((length + 1) * sizeof(char)); - nbytes = read(socket_fd, message, length); + *bytes = malloc(*length * sizeof(uint8_t)); + nbytes = read(fd, *bytes, *length); if (nbytes < 0) { LOG_ERR("Error reading message from socket."); - free(message); - return NULL; + free(*bytes); + *bytes = NULL; } - message[length] = '\0'; - return message; +} + +/* Write a null-terminated string to a file descriptor. */ +void write_string(int fd, char *message) { + /* Account for the \0 at the end of the string. */ + write_bytes(fd, (uint8_t *) message, strlen(message) + 1); +} + +/* Reads a null-terminated string from the file descriptor that has been + * written by write_string. Allocates and returns a pointer to the string. + * NOTE: Caller must free the memory! */ +char *read_string(int fd) { + uint8_t *bytes; + int64_t length; + read_bytes(fd, &bytes, &length); + return (char *) bytes; } diff --git a/io.h b/io.h new file mode 100644 index 000000000..c6dd3bb30 --- /dev/null +++ b/io.h @@ -0,0 +1,21 @@ +#ifndef IO_H +#define IO_H + +#include + +/* Helper functions for socket communication. */ + +int bind_ipc_sock(const char *socket_pathname); +int connect_ipc_sock(const char *socket_pathname); + +int accept_client(int socket_fd); + +/* Reading and writing data */ + +void write_bytes(int fd, uint8_t *bytes, int64_t length); +void read_bytes(int fd, uint8_t **bytes, int64_t *length); + +void write_string(int fd, char *message); +char *read_string(int fd); + +#endif diff --git a/sockets.h b/sockets.h deleted file mode 100644 index a563470c9..000000000 --- a/sockets.h +++ /dev/null @@ -1,11 +0,0 @@ -#ifndef SOCKETS_H -#define SOCKETS_H - -/* Helper functions for socket communication. */ -int bind_ipc_sock(const char* socket_pathname); -int connect_ipc_sock(const char* socket_pathname); -void send_ipc_sock(int socket_fd, char* message); -int accept_client(int socket_fd); -char* recv_ipc_sock(int socket_fd); - -#endif diff --git a/task.c b/task.c index c337cc1a3..3f3c01661 100644 --- a/task.c +++ b/task.c @@ -4,7 +4,7 @@ #include "task.h" #include "common.h" -#include "sockets.h" +#include "io.h" /* Tasks are stored in a consecutive chunk of memory, the first * sizeof(task_spec) bytes are arranged according to the struct diff --git a/test/socket_tests.c b/test/io_tests.c similarity index 57% rename from test/socket_tests.c rename to test/io_tests.c index b57ab9000..9216aa56e 100644 --- a/test/socket_tests.c +++ b/test/io_tests.c @@ -2,32 +2,39 @@ #include #include +#include -#include "sockets.h" +#include "io.h" -SUITE(socket_tests); +SUITE(io_tests); TEST ipc_socket_test(void) { - const char* socket_pathname = "test-socket"; + const char *socket_pathname = "test-socket"; int socket_fd = bind_ipc_sock(socket_pathname); ASSERT(socket_fd >= 0); - char* test_string = "hello world"; + char *test_string = "hello world"; + char *test_bytes = "another string"; pid_t pid = fork(); if (pid == 0) { close(socket_fd); socket_fd = connect_ipc_sock(socket_pathname); ASSERT(socket_fd >= 0); - send_ipc_sock(socket_fd, test_string); + write_string(socket_fd, test_string); + write_bytes(socket_fd, (uint8_t *) test_bytes, strlen(test_bytes)); close(socket_fd); exit(0); } else { int client_fd = accept_client(socket_fd); ASSERT(client_fd >= 0); - char* message = recv_ipc_sock(client_fd); + char *message = read_string(client_fd); ASSERT(message != NULL); ASSERT_STR_EQ(test_string, message); free(message); + int64_t len; + uint8_t *bytes; + read_bytes(client_fd, &bytes, &len); + ASSERT(memcmp(test_bytes, bytes, len) == 0); close(client_fd); close(socket_fd); unlink(socket_pathname); @@ -36,14 +43,14 @@ TEST ipc_socket_test(void) { PASS(); } -SUITE(socket_tests) { +SUITE(io_tests) { RUN_TEST(ipc_socket_test); } GREATEST_MAIN_DEFS(); -int main(int argc, char** argv) { +int main(int argc, char **argv) { GREATEST_MAIN_BEGIN(); - RUN_SUITE(socket_tests); + RUN_SUITE(io_tests); GREATEST_MAIN_END(); } From c238ae4aa01025761a6c709c812f1e35caabefcd Mon Sep 17 00:00:00 2001 From: Philipp Moritz Date: Sun, 18 Sep 2016 13:57:27 -0700 Subject: [PATCH 07/36] do not re-download and rebuild redis if it already exists (#10) --- thirdparty/build-redis.sh | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/thirdparty/build-redis.sh b/thirdparty/build-redis.sh index 57c68c97b..230e9ae29 100644 --- a/thirdparty/build-redis.sh +++ b/thirdparty/build-redis.sh @@ -1,4 +1,6 @@ -wget http://download.redis.io/releases/redis-3.2.3.tar.gz -tar xvfz redis-3.2.3.tar.gz -cd redis-3.2.3 -make +if [ ! -f redis-3.2.3/src/redis-server ]; then + wget http://download.redis.io/releases/redis-3.2.3.tar.gz + tar xvfz redis-3.2.3.tar.gz + cd redis-3.2.3 + make +fi From 37f035dbd07e080b64934722d917e3a8bb4c46f6 Mon Sep 17 00:00:00 2001 From: Philipp Moritz Date: Sun, 18 Sep 2016 18:06:42 -0700 Subject: [PATCH 08/36] implementing reading and writing tasks (#11) --- Makefile | 4 ++-- common.c | 18 ++++++++++++++++++ common.h | 4 ++++ task.c | 26 ++++++++++++++++++++++++-- task.h | 12 ++++++++++-- test/task_tests.c | 40 ++++++++++++++++++++++++++++------------ 6 files changed, 86 insertions(+), 18 deletions(-) diff --git a/Makefile b/Makefile index e1c894737..29a7ea1ba 100644 --- a/Makefile +++ b/Makefile @@ -12,8 +12,8 @@ $(BUILD)/db_tests: hiredis test/db_tests.c thirdparty/greatest.h event_loop.c st $(BUILD)/io_tests: test/io_tests.c thirdparty/greatest.h io.c $(CC) -o $@ test/io_tests.c io.c $(CFLAGS) -I. -Ithirdparty -$(BUILD)/task_tests: test/task_tests.c task.c io.c common.h - $(CC) -o $@ test/task_tests.c task.c io.c $(CFLAGS) -I. -Ithirdparty +$(BUILD)/task_tests: test/task_tests.c task.h task.c io.h io.c common.h common.h common.c + $(CC) -o $@ test/task_tests.c task.c io.c common.c $(CFLAGS) -I. -Ithirdparty clean: rm -r $(BUILD)/* diff --git a/common.c b/common.c index e227eb16e..9e0a86310 100644 --- a/common.c +++ b/common.c @@ -1,5 +1,23 @@ #include "common.h" +#include +#include +#include +#include +#include + +unique_id globally_unique_id(void) { + /* Use /dev/urandom for "real" randomness. */ + int fd; + if ((fd = open("/dev/urandom", O_RDONLY)) == -1) { + LOG_ERR("Could not generate random number"); + } + unique_id result; + read(fd, &result.id[0], UNIQUE_ID_SIZE); + close(fd); + return result; +} + char *sha1_to_hex(const unsigned char *sha1, char *buffer) { static const char hex[] = "0123456789abcdef"; char *buf = buffer; diff --git a/common.h b/common.h index d38556dcf..6dc360119 100644 --- a/common.h +++ b/common.h @@ -1,6 +1,7 @@ #ifndef COMMON_H #define COMMON_H +#include #include #ifdef NDEBUG @@ -29,6 +30,9 @@ typedef struct { unsigned char id[UNIQUE_ID_SIZE]; } unique_id; +/* Generate a globally unique ID. */ +unique_id globally_unique_id(void); + /* Convert a 20 byte sha1 hash to a hexdecimal string. This function assumes * that buffer points to an already allocated char array of size 2 * * UNIQUE_ID_SIZE + 1 */ diff --git a/task.c b/task.c index 3f3c01661..5d5271d91 100644 --- a/task.c +++ b/task.c @@ -47,12 +47,16 @@ struct task_spec_impl { task_arg args_and_returns[0]; }; +/* The size of a task specification is given by the following expression. */ +#define TASK_SPEC_SIZE(NUM_ARGS, NUM_RETURNS, ARGS_VALUE_SIZE) \ + (sizeof(task_spec) + ((NUM_ARGS) + (NUM_RETURNS)) * sizeof(task_arg) + \ + (ARGS_VALUE_SIZE)) + task_spec *alloc_task_spec(function_id func_id, int64_t num_args, int64_t num_returns, int64_t args_value_size) { - int64_t size = sizeof(task_spec) + - (num_args + num_returns) * sizeof(task_arg) + args_value_size; + int64_t size = TASK_SPEC_SIZE(num_args, num_returns, args_value_size); task_spec *task = malloc(size); memset(task, 0, size); task->func_id = func_id; @@ -63,6 +67,11 @@ task_spec *alloc_task_spec(function_id func_id, return task; } +int64_t task_size(task_spec *spec) { + return TASK_SPEC_SIZE(spec->num_args, spec->num_returns, + spec->args_value_size); +} + int64_t task_num_args(task_spec *spec) { return spec->num_args; } @@ -131,3 +140,16 @@ void free_task_spec(task_spec *spec) { CHECK(spec->arg_index == spec->num_args); /* Task was fully constructed */ free(spec); } + +void write_task(int fd, task_spec *spec) { + write_bytes(fd, (uint8_t *) spec, task_size(spec)); +} + +task_spec *read_task(int fd) { + uint8_t *bytes; + int64_t length; + read_bytes(fd, &bytes, &length); + task_spec *spec = (task_spec *) bytes; + CHECK(task_size(spec) == length); + return spec; +} diff --git a/task.h b/task.h index fbf557fe0..ad85540ee 100644 --- a/task.h +++ b/task.h @@ -1,3 +1,6 @@ +#ifndef TASK_H +#define TASK_H + /* This API specifies the task data structure. It is in C so we can * easily construct tasks from other languages like Python. The datastructures * are also defined in such a way that memory is contiguous and all pointers @@ -24,6 +27,9 @@ task_spec *alloc_task_spec(function_id func_id, int64_t num_returns, int64_t args_value_size); +/* Size of the task in bytes. */ +int64_t task_size(task_spec *spec); + /* Getting the number of arguments and returns. */ int64_t task_num_args(task_spec *spec); int64_t task_num_returns(task_spec *spec); @@ -46,8 +52,10 @@ unique_id *task_return(task_spec *spec, int64_t ret_index); void free_task_spec(task_spec *spec); /* Write the task specification to a file or socket. */ -int send_task(int fd, task_spec *spec); +void write_task(int fd, task_spec *spec); /* Read the task specification from a file or socket. It is the user's * responsibility to free the task after it has been used. */ -task_spec *recv_task(int fd); +task_spec *read_task(int fd); + +#endif diff --git a/test/task_tests.c b/test/task_tests.c index d8443e68c..68a6a6537 100644 --- a/test/task_tests.c +++ b/test/task_tests.c @@ -1,29 +1,29 @@ #include "greatest.h" +#include +#include +#include + +#include "common.h" #include "task.h" SUITE(task_tests); TEST task_test(void) { - function_id func_id = { - {1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1}}; - task_spec* task = alloc_task_spec(func_id, 4, 2, 10); + function_id func_id = globally_unique_id(); + task_spec *task = alloc_task_spec(func_id, 4, 2, 10); ASSERT(task_num_args(task) == 4); ASSERT(task_num_returns(task) == 2); - unique_id arg1 = { - {2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2}}; + unique_id arg1 = globally_unique_id(); ASSERT(task_args_add_ref(task, arg1) == 0); ASSERT(task_args_add_val(task, (uint8_t*) "hello", 5) == 1); - unique_id arg2 = { - {3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3}}; + unique_id arg2 = globally_unique_id(); ASSERT(task_args_add_ref(task, arg2) == 2); ASSERT(task_args_add_val(task, (uint8_t*) "world", 5) == 3); - unique_id ret0 = { - {4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4}}; - unique_id ret1 = { - {5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5}}; + unique_id ret0 = globally_unique_id(); + unique_id ret1 = globally_unique_id(); memcpy(task_return(task, 0), &ret0, sizeof(ret0)); memcpy(task_return(task, 1), &ret1, sizeof(ret1)); @@ -41,13 +41,29 @@ TEST task_test(void) { PASS(); } +TEST send_task(void) { + function_id func_id = globally_unique_id(); + task_spec *task = alloc_task_spec(func_id, 4, 2, 10); + *task_return(task, 1) = globally_unique_id(); + int fd[2]; + socketpair(AF_UNIX, SOCK_STREAM, 0, fd); + write_task(fd[0], task); + task_spec *result = read_task(fd[1]); + ASSERT(memcmp(task, result, task_size(task)) == 0); + ASSERT(memcmp(task, result, task_size(result)) == 0); + free(task); + free(result); + PASS(); +} + SUITE(task_tests) { RUN_TEST(task_test); + RUN_TEST(send_task); } GREATEST_MAIN_DEFS(); -int main(int argc, char** argv) { +int main(int argc, char **argv) { GREATEST_MAIN_BEGIN(); RUN_SUITE(task_tests); GREATEST_MAIN_END(); From d11161bb01a9d191888c966366a7a3185cb238c4 Mon Sep 17 00:00:00 2001 From: Philipp Moritz Date: Sun, 18 Sep 2016 20:47:53 -0700 Subject: [PATCH 09/36] make static libraries (#13) --- Makefile | 22 ++++++++++++++-------- 1 file changed, 14 insertions(+), 8 deletions(-) diff --git a/Makefile b/Makefile index 29a7ea1ba..1f3deebe2 100644 --- a/Makefile +++ b/Makefile @@ -1,22 +1,28 @@ CC = gcc -CFLAGS = -g -Wall --std=c99 -D_XOPEN_SOURCE=500 -D_POSIX_C_SOURCE=200809L +CFLAGS = -g -Wall --std=c99 -D_XOPEN_SOURCE=500 -D_POSIX_C_SOURCE=200809L -fPIC -I. -Ithirdparty BUILD = build CFLAGS += -Wmissing-prototypes CFLAGS += -Wstrict-prototypes CFLAGS += -Wmissing-declarations -$(BUILD)/db_tests: hiredis test/db_tests.c thirdparty/greatest.h event_loop.c state/redis.c common.c - $(CC) -o $@ test/db_tests.c event_loop.c state/redis.c common.c thirdparty/hiredis/libhiredis.a $(CFLAGS) -I. -Ithirdparty +all: $(BUILD)/libcommon.a -$(BUILD)/io_tests: test/io_tests.c thirdparty/greatest.h io.c - $(CC) -o $@ test/io_tests.c io.c $(CFLAGS) -I. -Ithirdparty +$(BUILD)/libcommon.a: event_loop.o common.o task.o io.o state/redis.o + ar rcs $@ $^ -$(BUILD)/task_tests: test/task_tests.c task.h task.c io.h io.c common.h common.h common.c - $(CC) -o $@ test/task_tests.c task.c io.c common.c $(CFLAGS) -I. -Ithirdparty +$(BUILD)/db_tests: hiredis test/db_tests.c $(BUILD)/libcommon.a + $(CC) -o $@ test/db_tests.c $(BUILD)/libcommon.a thirdparty/hiredis/libhiredis.a $(CFLAGS) + +$(BUILD)/io_tests: test/io_tests.c $(BUILD)/libcommon.a + $(CC) -o $@ $^ $(CFLAGS) + +$(BUILD)/task_tests: test/task_tests.c $(BUILD)/libcommon.a + $(CC) -o $@ $^ $(CFLAGS) clean: - rm -r $(BUILD)/* + rm -f *.o state/*.o test/*.o + rm -rf $(BUILD)/* redis: cd thirdparty ; bash ./build-redis.sh From 6c6f2d047309a16f75dd73906ada943a9edc67e2 Mon Sep 17 00:00:00 2001 From: Philipp Moritz Date: Tue, 20 Sep 2016 17:02:56 -0700 Subject: [PATCH 10/36] Implement object table API (#16) --- Makefile | 4 - common.h | 2 + event_loop.c | 17 +- event_loop.h | 1 + state/object_table.h | 16 +- state/redis.c | 81 ++-- state/redis.h | 26 +- state/task_queue.h | 33 ++ state/task_table.h | 13 + test/db_tests.c | 100 +++- thirdparty/uthash.h | 1074 ++++++++++++++++++++++++++++++++++++++++++ 11 files changed, 1300 insertions(+), 67 deletions(-) create mode 100644 state/task_queue.h create mode 100644 state/task_table.h create mode 100644 thirdparty/uthash.h diff --git a/Makefile b/Makefile index 1f3deebe2..29b7befd9 100644 --- a/Makefile +++ b/Makefile @@ -2,10 +2,6 @@ CC = gcc CFLAGS = -g -Wall --std=c99 -D_XOPEN_SOURCE=500 -D_POSIX_C_SOURCE=200809L -fPIC -I. -Ithirdparty BUILD = build -CFLAGS += -Wmissing-prototypes -CFLAGS += -Wstrict-prototypes -CFLAGS += -Wmissing-declarations - all: $(BUILD)/libcommon.a $(BUILD)/libcommon.a: event_loop.o common.o task.o io.o state/redis.o diff --git a/common.h b/common.h index 6dc360119..96e6402f0 100644 --- a/common.h +++ b/common.h @@ -38,4 +38,6 @@ unique_id globally_unique_id(void); * UNIQUE_ID_SIZE + 1 */ char *sha1_to_hex(const unsigned char *sha1, char *buffer); +typedef unique_id object_id; + #endif diff --git a/event_loop.c b/event_loop.c index ebc6ebc13..0fd79e6af 100644 --- a/event_loop.c +++ b/event_loop.c @@ -12,6 +12,13 @@ void event_loop_init(event_loop *loop) { utarray_new(loop->waiting, &poll_icd); } +/* Free the space associated to the event loop. + * Does not free the event_loop datastructure itself. */ +void event_loop_free(event_loop *loop) { + utarray_free(loop->items); + utarray_free(loop->waiting); +} + /* Add a new file descriptor fd to the event loop. * This function sets a user defined type and id for the file descriptor * which can be queried using event_loop_type and event_loop_id. The parameter @@ -83,9 +90,9 @@ void *event_loop_get_data(event_loop *loop, int64_t index) { return item->data; } -/* Free the space associated to the event loop. - * Does not free the event_loop datastructure itself. */ -void event_loop_free(event_loop *loop) { - utarray_free(loop->items); - utarray_free(loop->waiting); +/* Return the type of connection. */ +int event_loop_type(event_loop *loop, int64_t index) { + event_loop_item *item = + (event_loop_item *) utarray_eltptr(loop->items, index); + return item->type; } diff --git a/event_loop.h b/event_loop.h index 0903bb9d4..a96ec4643 100644 --- a/event_loop.h +++ b/event_loop.h @@ -34,5 +34,6 @@ int64_t event_loop_size(event_loop *loop); struct pollfd *event_loop_get(event_loop *loop, int64_t index); void event_loop_set_data(event_loop *loop, int64_t index, void *data); void *event_loop_get_data(event_loop *loop, int64_t index); +int event_loop_type(event_loop *loop, int64_t index); #endif diff --git a/state/object_table.h b/state/object_table.h index 6b4d62e4b..7c00ab2ba 100644 --- a/state/object_table.h +++ b/state/object_table.h @@ -1,15 +1,21 @@ #include "common.h" #include "db.h" -typedef void (*lookup_callback)(void *); +/* The callback that is called when the result of a lookup + * in the object table comes back. The callback should free + * the manager_vector array, but NOT the strings they are pointing to. */ +typedef void (*lookup_callback)(object_id object_id, + int manager_count, + const char *manager_vector[]); /* Register a new object with the directory. */ -void object_table_add(db_conn *db, unique_id object_id); +/* TODO(pcm): Retry, print for each attempt. */ +void object_table_add(db_conn *db, object_id object_id); -/* Remove object from the directory */ -void object_table_remove(db_conn *db, unique_id object_id); +/* Remove object from the directory. */ +void object_table_remove(db_conn *db, object_id object_id, const char *manager); /* Look up entry from the directory */ void object_table_lookup(db_conn *db, - unique_id object_id, + object_id object_id, lookup_callback callback); diff --git a/state/redis.c b/state/redis.c index a8029a063..c781b81ef 100644 --- a/state/redis.c +++ b/state/redis.c @@ -5,6 +5,7 @@ #include "common.h" #include "db.h" #include "object_table.h" +#include "task_queue.h" #include "event_loop.h" #include "redis.h" @@ -88,12 +89,13 @@ void db_connect(const char *address, } freeReplyObject(reply); } - redisFree(context); db->client_type = strdup(client_type); db->client_id = num_clients; db->reading = 0; db->writing = 0; + db->service_cache = NULL; + db->sync_context = context; /* Establish async connection */ db->context = redisAsyncConnect(address, port); @@ -102,6 +104,18 @@ void db_connect(const char *address, db->context->data = (void *) db; } +void db_disconnect(db_conn *db) { + redisFree(db->sync_context); + redisAsyncFree(db->context); + service_cache_entry *e, *tmp; + HASH_ITER(hh, db->service_cache, e, tmp) { + free(e->addr); + HASH_DEL(db->service_cache, e); + free(e); + } + free(db->client_type); +} + void db_event(db_conn *db) { if (db->reading) { redisAsyncHandleRead(db->context); @@ -137,51 +151,62 @@ void object_table_add(db_conn *db, unique_id object_id) { static char hex_object_id[2 * UNIQUE_ID_SIZE + 1]; sha1_to_hex(&object_id.id[0], &hex_object_id[0]); redisAsyncCommand(db->context, NULL, NULL, "SADD obj:%s %d", - &hex_object_id[0], 0); + &hex_object_id[0], db->client_id); if (db->context->err) { LOG_REDIS_ERR(db->context, "could not add object_table entry"); } } -void object_table_lookup_callback(redisAsyncContext *c, - void *r, - void *privdata) { +void object_table_get_entry(redisAsyncContext *c, void *r, void *privdata) { + db_conn *db = c->data; + lookup_callback_data *cb_data = privdata; redisReply *reply = r; if (reply == NULL) return; - lookup_callback callback = privdata; - char *str = malloc(reply->len); - memcpy(str, reply->str, reply->len); - callback(str); -} - -void object_table_fetch_addr_port(redisAsyncContext *c, - void *r, - void *privdata) { - redisReply *reply = r; - if (reply == NULL) - return; - long long manager_id = -1; - if (reply->type == REDIS_REPLY_STRING) { - manager_id = strtoll(reply->str, NULL, 10); - } else if (reply->type != REDIS_REPLY_INTEGER) { - manager_id = reply->integer; + int *result = malloc(reply->elements * sizeof(int)); + int64_t manager_count = reply->elements; + if (reply->type == REDIS_REPLY_ARRAY) { + for (int j = 0; j < reply->elements; j++) { + CHECK(reply->element[j]->type == REDIS_REPLY_STRING); + result[j] = atoi(reply->element[j]->str); + service_cache_entry *entry; + HASH_FIND_INT(db->service_cache, &result[j], entry); + if (!entry) { + redisReply *reply = redisCommand(db->sync_context, "HGET %s %lld", + db->client_type, result[j]); + CHECK(reply->type == REDIS_REPLY_STRING); + entry = malloc(sizeof(service_cache_entry)); + entry->service_id = result[j]; + entry->addr = strdup(reply->str); + HASH_ADD_INT(db->service_cache, service_id, entry); + freeReplyObject(reply); + } + } } else { LOG_ERR("expected integer or string, received type %d", reply->type); exit(-1); } - db_conn *db = c->data; - redisAsyncCommand(db->context, object_table_lookup_callback, privdata, - "HGET %s %lld", db->client_type, manager_id); + const char **manager_vector = malloc(manager_count * sizeof(char *)); + for (int j = 0; j < manager_count; ++j) { + service_cache_entry *entry; + HASH_FIND_INT(db->service_cache, &result[j], entry); + manager_vector[j] = entry->addr; + } + cb_data->callback(cb_data->object_id, manager_count, manager_vector); + free(privdata); + free(result); } void object_table_lookup(db_conn *db, - unique_id object_id, + object_id object_id, lookup_callback callback) { static char hex_object_id[2 * UNIQUE_ID_SIZE + 1]; sha1_to_hex(&object_id.id[0], &hex_object_id[0]); - redisAsyncCommand(db->context, object_table_fetch_addr_port, callback, - "SRANDMEMBER obj:%s", &hex_object_id[0]); + lookup_callback_data *cb_data = malloc(sizeof(lookup_callback_data)); + cb_data->callback = callback; + cb_data->object_id = object_id; + redisAsyncCommand(db->context, object_table_get_entry, cb_data, + "SMEMBERS obj:%s", &hex_object_id[0]); if (db->context->err) { LOG_REDIS_ERR(db->context, "error in object_table lookup"); } diff --git a/state/redis.h b/state/redis.h index 471044b06..ad8d5cbcf 100644 --- a/state/redis.h +++ b/state/redis.h @@ -3,6 +3,16 @@ #include "hiredis/hiredis.h" #include "hiredis/async.h" +#include "uthash.h" + +typedef struct { + /* Unique ID for this service. */ + int service_id; + /* IP address and port of this service. */ + const char *addr; + /* Handle for the uthash table. */ + UT_hash_handle hh; +} service_cache_entry; struct db_conn_impl { /* String that identifies this client type. */ @@ -15,11 +25,21 @@ struct db_conn_impl { int reading, writing; /* The event loop this global state store connection is part of. */ event_loop *loop; + /* Cache for the IP addresses of services. */ + service_cache_entry *service_cache; + /* Redis context for synchronous connections. + * Should only be used very rarely, it is not asynchronous. */ + redisContext *sync_context; }; -void object_table_fetch_addr_port(redisAsyncContext *c, - void *r, - void *privdata); +typedef struct { + /* The callback that will be called. */ + lookup_callback callback; + /* Object ID that is looked up. */ + object_id object_id; +} lookup_callback_data; + +void object_table_get_entry(redisAsyncContext *c, void *r, void *privdata); void object_table_lookup_callback(redisAsyncContext *c, void *r, diff --git a/state/task_queue.h b/state/task_queue.h new file mode 100644 index 000000000..0226c501b --- /dev/null +++ b/state/task_queue.h @@ -0,0 +1,33 @@ +#ifndef TASK_QUEUE_H +#define TASK_QUEUE_H + +#include "db.h" +#include "task.h" + +/* The task ID is a deterministic hash of the function ID that + * the task executes and the argument IDs or argument values */ +typedef unique_id task_id; + +/* The task instance ID is a globally unique ID generated which + * identifies this particular execution of the task */ +typedef unique_id task_iid; + +/* The node id is an identifier for the node the task is + * scheduled on */ +typedef unique_id node_id; + +/* Callback for subscribing to the task queue. The only argument this + * callback gets is the task_id of the. */ +typedef void (*task_queue_callback)(task_iid *task_iid, task_spec *task); + +/* Submit task to the global scheduler. */ +void task_queue_submit_task(db_conn *db, task_iid task_iid, task_spec *task); + +/* Submit task to a local scheduler based on the decision made by the global + * scheduler. */ +void task_queue_schedule_task(db_conn *db, task_iid task_iid, node_id node); + +/* Subscribe to task queue. */ +void task_queue_register_callback(db_conn *db, task_queue_callback callback); + +#endif diff --git a/state/task_table.h b/state/task_table.h new file mode 100644 index 000000000..64285da67 --- /dev/null +++ b/state/task_table.h @@ -0,0 +1,13 @@ +#ifndef TASK_TABLE_H +#define TASK_TABLE_H + +#include "db.h" +#include "task.h" + +/* Add task to the task table, handle errors here. */ +status task_table_add_task(db_conn *db, task_iid task_iid, task_spec *task); + +/* Get specific task from the task table. */ +status task_table_get_task(db_conn *db, task_iid task_iid, task_spec *task); + +#endif /* TASK_TABLE_H */ diff --git a/test/db_tests.c b/test/db_tests.c index b3a0d582e..0788345f2 100644 --- a/test/db_tests.c +++ b/test/db_tests.c @@ -6,37 +6,61 @@ #include "state/db.h" #include "state/object_table.h" #include "state/redis.h" +#include "task.h" SUITE(db_tests); int lookup_successful = 0; const char *manager_addr = "127.0.0.1"; -int manager_port = 12345; -char received_addr[16] = {0}; -char received_port[6] = {0}; +int manager_port1 = 12345; +int manager_port2 = 12346; +char received_addr1[16] = {0}; +char received_port1[6] = {0}; +char received_addr2[16] = {0}; +char received_port2[6] = {0}; -void test_callback(void *userdata); - -void test_callback(void *userdata) { - char *reply = userdata; +/* This is for synchronizing to make sure both entries have been written. */ +void sync_test_callback(object_id object_id, + int manager_count, + const char *manager_vector[]) { lookup_successful = 1; - if (!reply || - sscanf(reply, "%15[0-9.]:%5[0-9]", received_addr, received_port) != 2) { - assert(0); + free(manager_vector); +} + +/* This performs the actual test. */ +void test_callback(object_id object_id, + int manager_count, + const char *manager_vector[]) { + CHECK(manager_count == 2); + lookup_successful = 1; + if (!manager_vector[0] || + sscanf(manager_vector[0], "%15[0-9.]:%5[0-9]", received_addr1, + received_port1) != 2) { + CHECK(0); } - free(reply); + if (!manager_vector[1] || + sscanf(manager_vector[1], "%15[0-9.]:%5[0-9]", received_addr2, + received_port2) != 2) { + CHECK(0); + } + free(manager_vector); } TEST object_table_lookup_test(void) { event_loop loop; event_loop_init(&loop); - db_conn conn; - db_connect("127.0.0.1", 6379, "plasma_manager", manager_addr, manager_port, - &conn); - int64_t index = db_attach(&conn, &loop, 0); - unique_id id = {{0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0}}; - object_table_add(&conn, id); - object_table_lookup(&conn, id, test_callback); + db_conn conn1; + db_connect("127.0.0.1", 6379, "plasma_manager", manager_addr, manager_port1, + &conn1); + db_conn conn2; + db_connect("127.0.0.1", 6379, "plasma_manager", manager_addr, manager_port2, + &conn2); + int64_t index1 = db_attach(&conn1, &loop, 0); + int64_t index2 = db_attach(&conn2, &loop, 1); + unique_id id = globally_unique_id(); + object_table_add(&conn1, id); + object_table_add(&conn2, id); + object_table_lookup(&conn1, id, sync_test_callback); while (!lookup_successful) { int num_ready = event_loop_poll(&loop); if (num_ready < 0) { @@ -46,18 +70,50 @@ TEST object_table_lookup_test(void) { struct pollfd *waiting = event_loop_get(&loop, i); if (waiting->revents == 0) continue; - if (i == index) { - db_event(&conn); + if (i == index1) { + db_event(&conn1); + } + if (i == index2) { + db_event(&conn2); } } } - ASSERT_STR_EQ(&received_addr[0], manager_addr); - ASSERT_EQ(atoi(received_port), manager_port); + lookup_successful = 0; + object_table_lookup(&conn1, id, test_callback); + while (!lookup_successful) { + int num_ready = event_loop_poll(&loop); + if (num_ready < 0) { + exit(-1); + } + for (int i = 0; i < event_loop_size(&loop); ++i) { + struct pollfd *waiting = event_loop_get(&loop, i); + if (waiting->revents == 0) + continue; + if (i == index1) { + db_event(&conn1); + } + if (i == index2) { + db_event(&conn2); + } + } + } + int port1 = atoi(received_port1); + int port2 = atoi(received_port2); + ASSERT_STR_EQ(&received_addr1[0], manager_addr); + ASSERT((port1 == manager_port1 && port2 == manager_port2) || + (port2 == manager_port1 && port1 == manager_port2)); + + db_disconnect(&conn1); + db_disconnect(&conn2); + + event_loop_free(&loop); + PASS(); } SUITE(db_tests) { RUN_TEST(object_table_lookup_test); + /* RUN_TEST(task_queue_test); */ } GREATEST_MAIN_DEFS(); diff --git a/thirdparty/uthash.h b/thirdparty/uthash.h new file mode 100644 index 000000000..45d1f9fc1 --- /dev/null +++ b/thirdparty/uthash.h @@ -0,0 +1,1074 @@ +/* +Copyright (c) 2003-2016, Troy D. Hanson http://troydhanson.github.com/uthash/ +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + + * Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS +IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED +TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A +PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER +OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, +EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, +PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR +PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF +LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING +NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +*/ + +#ifndef UTHASH_H +#define UTHASH_H + +#define UTHASH_VERSION 2.0.1 + +#include /* memcmp,strlen */ +#include /* ptrdiff_t */ +#include /* exit() */ + +/* These macros use decltype or the earlier __typeof GNU extension. + As decltype is only available in newer compilers (VS2010 or gcc 4.3+ + when compiling c++ source) this code uses whatever method is needed + or, for VS2008 where neither is available, uses casting workarounds. */ +#if defined(_MSC_VER) /* MS compiler */ +#if _MSC_VER >= 1600 && defined(__cplusplus) /* VS2010 or newer in C++ mode */ +#define DECLTYPE(x) (decltype(x)) +#else /* VS2008 or older (or VS2010 in C mode) */ +#define NO_DECLTYPE +#define DECLTYPE(x) +#endif +#elif defined(__BORLANDC__) || defined(__LCC__) || defined(__WATCOMC__) +#define NO_DECLTYPE +#define DECLTYPE(x) +#else /* GNU, Sun and other compilers */ +#define DECLTYPE(x) (__typeof(x)) +#endif + +#ifdef NO_DECLTYPE +#define DECLTYPE_ASSIGN(dst,src) \ +do { \ + char **_da_dst = (char**)(&(dst)); \ + *_da_dst = (char*)(src); \ +} while (0) +#else +#define DECLTYPE_ASSIGN(dst,src) \ +do { \ + (dst) = DECLTYPE(dst)(src); \ +} while (0) +#endif + +/* a number of the hash function use uint32_t which isn't defined on Pre VS2010 */ +#if defined(_WIN32) +#if defined(_MSC_VER) && _MSC_VER >= 1600 +#include +#elif defined(__WATCOMC__) || defined(__MINGW32__) || defined(__CYGWIN__) +#include +#else +typedef unsigned int uint32_t; +typedef unsigned char uint8_t; +#endif +#elif defined(__GNUC__) && !defined(__VXWORKS__) +#include +#else +typedef unsigned int uint32_t; +typedef unsigned char uint8_t; +#endif + +#ifndef uthash_fatal +#define uthash_fatal(msg) exit(-1) /* fatal error (out of memory,etc) */ +#endif +#ifndef uthash_malloc +#define uthash_malloc(sz) malloc(sz) /* malloc fcn */ +#endif +#ifndef uthash_free +#define uthash_free(ptr,sz) free(ptr) /* free fcn */ +#endif +#ifndef uthash_strlen +#define uthash_strlen(s) strlen(s) +#endif +#ifndef uthash_memcmp +#define uthash_memcmp(a,b,n) memcmp(a,b,n) +#endif + +#ifndef uthash_noexpand_fyi +#define uthash_noexpand_fyi(tbl) /* can be defined to log noexpand */ +#endif +#ifndef uthash_expand_fyi +#define uthash_expand_fyi(tbl) /* can be defined to log expands */ +#endif + +/* initial number of buckets */ +#define HASH_INITIAL_NUM_BUCKETS 32U /* initial number of buckets */ +#define HASH_INITIAL_NUM_BUCKETS_LOG2 5U /* lg2 of initial number of buckets */ +#define HASH_BKT_CAPACITY_THRESH 10U /* expand when bucket count reaches */ + +/* calculate the element whose hash handle address is hhp */ +#define ELMT_FROM_HH(tbl,hhp) ((void*)(((char*)(hhp)) - ((tbl)->hho))) +/* calculate the hash handle from element address elp */ +#define HH_FROM_ELMT(tbl,elp) ((UT_hash_handle *)(((char*)(elp)) + ((tbl)->hho))) + +#define HASH_VALUE(keyptr,keylen,hashv) \ +do { \ + HASH_FCN(keyptr, keylen, hashv); \ +} while (0) + +#define HASH_FIND_BYHASHVALUE(hh,head,keyptr,keylen,hashval,out) \ +do { \ + (out) = NULL; \ + if (head) { \ + unsigned _hf_bkt; \ + HASH_TO_BKT(hashval, (head)->hh.tbl->num_buckets, _hf_bkt); \ + if (HASH_BLOOM_TEST((head)->hh.tbl, hashval) != 0) { \ + HASH_FIND_IN_BKT((head)->hh.tbl, hh, (head)->hh.tbl->buckets[ _hf_bkt ], keyptr, keylen, hashval, out); \ + } \ + } \ +} while (0) + +#define HASH_FIND(hh,head,keyptr,keylen,out) \ +do { \ + unsigned _hf_hashv; \ + HASH_VALUE(keyptr, keylen, _hf_hashv); \ + HASH_FIND_BYHASHVALUE(hh, head, keyptr, keylen, _hf_hashv, out); \ +} while (0) + +#ifdef HASH_BLOOM +#define HASH_BLOOM_BITLEN (1UL << HASH_BLOOM) +#define HASH_BLOOM_BYTELEN (HASH_BLOOM_BITLEN/8UL) + (((HASH_BLOOM_BITLEN%8UL)!=0UL) ? 1UL : 0UL) +#define HASH_BLOOM_MAKE(tbl) \ +do { \ + (tbl)->bloom_nbits = HASH_BLOOM; \ + (tbl)->bloom_bv = (uint8_t*)uthash_malloc(HASH_BLOOM_BYTELEN); \ + if (!((tbl)->bloom_bv)) { uthash_fatal( "out of memory"); } \ + memset((tbl)->bloom_bv, 0, HASH_BLOOM_BYTELEN); \ + (tbl)->bloom_sig = HASH_BLOOM_SIGNATURE; \ +} while (0) + +#define HASH_BLOOM_FREE(tbl) \ +do { \ + uthash_free((tbl)->bloom_bv, HASH_BLOOM_BYTELEN); \ +} while (0) + +#define HASH_BLOOM_BITSET(bv,idx) (bv[(idx)/8U] |= (1U << ((idx)%8U))) +#define HASH_BLOOM_BITTEST(bv,idx) (bv[(idx)/8U] & (1U << ((idx)%8U))) + +#define HASH_BLOOM_ADD(tbl,hashv) \ + HASH_BLOOM_BITSET((tbl)->bloom_bv, (hashv & (uint32_t)((1ULL << (tbl)->bloom_nbits) - 1U))) + +#define HASH_BLOOM_TEST(tbl,hashv) \ + HASH_BLOOM_BITTEST((tbl)->bloom_bv, (hashv & (uint32_t)((1ULL << (tbl)->bloom_nbits) - 1U))) + +#else +#define HASH_BLOOM_MAKE(tbl) +#define HASH_BLOOM_FREE(tbl) +#define HASH_BLOOM_ADD(tbl,hashv) +#define HASH_BLOOM_TEST(tbl,hashv) (1) +#define HASH_BLOOM_BYTELEN 0U +#endif + +#define HASH_MAKE_TABLE(hh,head) \ +do { \ + (head)->hh.tbl = (UT_hash_table*)uthash_malloc( \ + sizeof(UT_hash_table)); \ + if (!((head)->hh.tbl)) { uthash_fatal( "out of memory"); } \ + memset((head)->hh.tbl, 0, sizeof(UT_hash_table)); \ + (head)->hh.tbl->tail = &((head)->hh); \ + (head)->hh.tbl->num_buckets = HASH_INITIAL_NUM_BUCKETS; \ + (head)->hh.tbl->log2_num_buckets = HASH_INITIAL_NUM_BUCKETS_LOG2; \ + (head)->hh.tbl->hho = (char*)(&(head)->hh) - (char*)(head); \ + (head)->hh.tbl->buckets = (UT_hash_bucket*)uthash_malloc( \ + HASH_INITIAL_NUM_BUCKETS*sizeof(struct UT_hash_bucket)); \ + if (! (head)->hh.tbl->buckets) { uthash_fatal( "out of memory"); } \ + memset((head)->hh.tbl->buckets, 0, \ + HASH_INITIAL_NUM_BUCKETS*sizeof(struct UT_hash_bucket)); \ + HASH_BLOOM_MAKE((head)->hh.tbl); \ + (head)->hh.tbl->signature = HASH_SIGNATURE; \ +} while (0) + +#define HASH_REPLACE_BYHASHVALUE_INORDER(hh,head,fieldname,keylen_in,hashval,add,replaced,cmpfcn) \ +do { \ + (replaced) = NULL; \ + HASH_FIND_BYHASHVALUE(hh, head, &((add)->fieldname), keylen_in, hashval, replaced); \ + if (replaced) { \ + HASH_DELETE(hh, head, replaced); \ + } \ + HASH_ADD_KEYPTR_BYHASHVALUE_INORDER(hh, head, &((add)->fieldname), keylen_in, hashval, add, cmpfcn); \ +} while (0) + +#define HASH_REPLACE_BYHASHVALUE(hh,head,fieldname,keylen_in,hashval,add,replaced) \ +do { \ + (replaced) = NULL; \ + HASH_FIND_BYHASHVALUE(hh, head, &((add)->fieldname), keylen_in, hashval, replaced); \ + if (replaced) { \ + HASH_DELETE(hh, head, replaced); \ + } \ + HASH_ADD_KEYPTR_BYHASHVALUE(hh, head, &((add)->fieldname), keylen_in, hashval, add); \ +} while (0) + +#define HASH_REPLACE(hh,head,fieldname,keylen_in,add,replaced) \ +do { \ + unsigned _hr_hashv; \ + HASH_VALUE(&((add)->fieldname), keylen_in, _hr_hashv); \ + HASH_REPLACE_BYHASHVALUE(hh, head, fieldname, keylen_in, _hr_hashv, add, replaced); \ +} while (0) + +#define HASH_REPLACE_INORDER(hh,head,fieldname,keylen_in,add,replaced,cmpfcn) \ +do { \ + unsigned _hr_hashv; \ + HASH_VALUE(&((add)->fieldname), keylen_in, _hr_hashv); \ + HASH_REPLACE_BYHASHVALUE_INORDER(hh, head, fieldname, keylen_in, _hr_hashv, add, replaced, cmpfcn); \ +} while (0) + +#define HASH_APPEND_LIST(hh, head, add) \ +do { \ + (add)->hh.next = NULL; \ + (add)->hh.prev = ELMT_FROM_HH((head)->hh.tbl, (head)->hh.tbl->tail); \ + (head)->hh.tbl->tail->next = (add); \ + (head)->hh.tbl->tail = &((add)->hh); \ +} while (0) + +#define HASH_ADD_KEYPTR_BYHASHVALUE_INORDER(hh,head,keyptr,keylen_in,hashval,add,cmpfcn) \ +do { \ + unsigned _ha_bkt; \ + (add)->hh.hashv = (hashval); \ + (add)->hh.key = (char*) (keyptr); \ + (add)->hh.keylen = (unsigned) (keylen_in); \ + if (!(head)) { \ + (add)->hh.next = NULL; \ + (add)->hh.prev = NULL; \ + (head) = (add); \ + HASH_MAKE_TABLE(hh, head); \ + } else { \ + struct UT_hash_handle *_hs_iter = &(head)->hh; \ + (add)->hh.tbl = (head)->hh.tbl; \ + do { \ + if (cmpfcn(DECLTYPE(head) ELMT_FROM_HH((head)->hh.tbl, _hs_iter), add) > 0) \ + break; \ + } while ((_hs_iter = _hs_iter->next)); \ + if (_hs_iter) { \ + (add)->hh.next = _hs_iter; \ + if (((add)->hh.prev = _hs_iter->prev)) { \ + HH_FROM_ELMT((head)->hh.tbl, _hs_iter->prev)->next = (add); \ + } else { \ + (head) = (add); \ + } \ + _hs_iter->prev = (add); \ + } else { \ + HASH_APPEND_LIST(hh, head, add); \ + } \ + } \ + (head)->hh.tbl->num_items++; \ + HASH_TO_BKT(hashval, (head)->hh.tbl->num_buckets, _ha_bkt); \ + HASH_ADD_TO_BKT((head)->hh.tbl->buckets[_ha_bkt], &(add)->hh); \ + HASH_BLOOM_ADD((head)->hh.tbl, hashval); \ + HASH_EMIT_KEY(hh, head, keyptr, keylen_in); \ + HASH_FSCK(hh, head); \ +} while (0) + +#define HASH_ADD_KEYPTR_INORDER(hh,head,keyptr,keylen_in,add,cmpfcn) \ +do { \ + unsigned _hs_hashv; \ + HASH_VALUE(keyptr, keylen_in, _hs_hashv); \ + HASH_ADD_KEYPTR_BYHASHVALUE_INORDER(hh, head, keyptr, keylen_in, _hs_hashv, add, cmpfcn); \ +} while (0) + +#define HASH_ADD_BYHASHVALUE_INORDER(hh,head,fieldname,keylen_in,hashval,add,cmpfcn) \ + HASH_ADD_KEYPTR_BYHASHVALUE_INORDER(hh, head, &((add)->fieldname), keylen_in, hashval, add, cmpfcn) + +#define HASH_ADD_INORDER(hh,head,fieldname,keylen_in,add,cmpfcn) \ + HASH_ADD_KEYPTR_INORDER(hh, head, &((add)->fieldname), keylen_in, add, cmpfcn) + +#define HASH_ADD_KEYPTR_BYHASHVALUE(hh,head,keyptr,keylen_in,hashval,add) \ +do { \ + unsigned _ha_bkt; \ + (add)->hh.hashv = (hashval); \ + (add)->hh.key = (char*) (keyptr); \ + (add)->hh.keylen = (unsigned) (keylen_in); \ + if (!(head)) { \ + (add)->hh.next = NULL; \ + (add)->hh.prev = NULL; \ + (head) = (add); \ + HASH_MAKE_TABLE(hh, head); \ + } else { \ + (add)->hh.tbl = (head)->hh.tbl; \ + HASH_APPEND_LIST(hh, head, add); \ + } \ + (head)->hh.tbl->num_items++; \ + HASH_TO_BKT(hashval, (head)->hh.tbl->num_buckets, _ha_bkt); \ + HASH_ADD_TO_BKT((head)->hh.tbl->buckets[_ha_bkt], &(add)->hh); \ + HASH_BLOOM_ADD((head)->hh.tbl, hashval); \ + HASH_EMIT_KEY(hh, head, keyptr, keylen_in); \ + HASH_FSCK(hh, head); \ +} while (0) + +#define HASH_ADD_KEYPTR(hh,head,keyptr,keylen_in,add) \ +do { \ + unsigned _ha_hashv; \ + HASH_VALUE(keyptr, keylen_in, _ha_hashv); \ + HASH_ADD_KEYPTR_BYHASHVALUE(hh, head, keyptr, keylen_in, _ha_hashv, add); \ +} while (0) + +#define HASH_ADD_BYHASHVALUE(hh,head,fieldname,keylen_in,hashval,add) \ + HASH_ADD_KEYPTR_BYHASHVALUE(hh, head, &((add)->fieldname), keylen_in, hashval, add) + +#define HASH_ADD(hh,head,fieldname,keylen_in,add) \ + HASH_ADD_KEYPTR(hh, head, &((add)->fieldname), keylen_in, add) + +#define HASH_TO_BKT(hashv,num_bkts,bkt) \ +do { \ + bkt = ((hashv) & ((num_bkts) - 1U)); \ +} while (0) + +/* delete "delptr" from the hash table. + * "the usual" patch-up process for the app-order doubly-linked-list. + * The use of _hd_hh_del below deserves special explanation. + * These used to be expressed using (delptr) but that led to a bug + * if someone used the same symbol for the head and deletee, like + * HASH_DELETE(hh,users,users); + * We want that to work, but by changing the head (users) below + * we were forfeiting our ability to further refer to the deletee (users) + * in the patch-up process. Solution: use scratch space to + * copy the deletee pointer, then the latter references are via that + * scratch pointer rather than through the repointed (users) symbol. + */ +#define HASH_DELETE(hh,head,delptr) \ +do { \ + struct UT_hash_handle *_hd_hh_del; \ + if ( ((delptr)->hh.prev == NULL) && ((delptr)->hh.next == NULL) ) { \ + uthash_free((head)->hh.tbl->buckets, \ + (head)->hh.tbl->num_buckets*sizeof(struct UT_hash_bucket) ); \ + HASH_BLOOM_FREE((head)->hh.tbl); \ + uthash_free((head)->hh.tbl, sizeof(UT_hash_table)); \ + head = NULL; \ + } else { \ + unsigned _hd_bkt; \ + _hd_hh_del = &((delptr)->hh); \ + if ((delptr) == ELMT_FROM_HH((head)->hh.tbl,(head)->hh.tbl->tail)) { \ + (head)->hh.tbl->tail = \ + (UT_hash_handle*)((ptrdiff_t)((delptr)->hh.prev) + \ + (head)->hh.tbl->hho); \ + } \ + if ((delptr)->hh.prev != NULL) { \ + ((UT_hash_handle*)((ptrdiff_t)((delptr)->hh.prev) + \ + (head)->hh.tbl->hho))->next = (delptr)->hh.next; \ + } else { \ + DECLTYPE_ASSIGN(head,(delptr)->hh.next); \ + } \ + if (_hd_hh_del->next != NULL) { \ + ((UT_hash_handle*)((ptrdiff_t)_hd_hh_del->next + \ + (head)->hh.tbl->hho))->prev = \ + _hd_hh_del->prev; \ + } \ + HASH_TO_BKT( _hd_hh_del->hashv, (head)->hh.tbl->num_buckets, _hd_bkt); \ + HASH_DEL_IN_BKT(hh,(head)->hh.tbl->buckets[_hd_bkt], _hd_hh_del); \ + (head)->hh.tbl->num_items--; \ + } \ + HASH_FSCK(hh,head); \ +} while (0) + + +/* convenience forms of HASH_FIND/HASH_ADD/HASH_DEL */ +#define HASH_FIND_STR(head,findstr,out) \ + HASH_FIND(hh,head,findstr,(unsigned)uthash_strlen(findstr),out) +#define HASH_ADD_STR(head,strfield,add) \ + HASH_ADD(hh,head,strfield[0],(unsigned)uthash_strlen(add->strfield),add) +#define HASH_REPLACE_STR(head,strfield,add,replaced) \ + HASH_REPLACE(hh,head,strfield[0],(unsigned)uthash_strlen(add->strfield),add,replaced) +#define HASH_FIND_INT(head,findint,out) \ + HASH_FIND(hh,head,findint,sizeof(int),out) +#define HASH_ADD_INT(head,intfield,add) \ + HASH_ADD(hh,head,intfield,sizeof(int),add) +#define HASH_REPLACE_INT(head,intfield,add,replaced) \ + HASH_REPLACE(hh,head,intfield,sizeof(int),add,replaced) +#define HASH_FIND_PTR(head,findptr,out) \ + HASH_FIND(hh,head,findptr,sizeof(void *),out) +#define HASH_ADD_PTR(head,ptrfield,add) \ + HASH_ADD(hh,head,ptrfield,sizeof(void *),add) +#define HASH_REPLACE_PTR(head,ptrfield,add,replaced) \ + HASH_REPLACE(hh,head,ptrfield,sizeof(void *),add,replaced) +#define HASH_DEL(head,delptr) \ + HASH_DELETE(hh,head,delptr) + +/* HASH_FSCK checks hash integrity on every add/delete when HASH_DEBUG is defined. + * This is for uthash developer only; it compiles away if HASH_DEBUG isn't defined. + */ +#ifdef HASH_DEBUG +#define HASH_OOPS(...) do { fprintf(stderr,__VA_ARGS__); exit(-1); } while (0) +#define HASH_FSCK(hh,head) \ +do { \ + struct UT_hash_handle *_thh; \ + if (head) { \ + unsigned _bkt_i; \ + unsigned _count; \ + char *_prev; \ + _count = 0; \ + for( _bkt_i = 0; _bkt_i < (head)->hh.tbl->num_buckets; _bkt_i++) { \ + unsigned _bkt_count = 0; \ + _thh = (head)->hh.tbl->buckets[_bkt_i].hh_head; \ + _prev = NULL; \ + while (_thh) { \ + if (_prev != (char*)(_thh->hh_prev)) { \ + HASH_OOPS("invalid hh_prev %p, actual %p\n", \ + _thh->hh_prev, _prev ); \ + } \ + _bkt_count++; \ + _prev = (char*)(_thh); \ + _thh = _thh->hh_next; \ + } \ + _count += _bkt_count; \ + if ((head)->hh.tbl->buckets[_bkt_i].count != _bkt_count) { \ + HASH_OOPS("invalid bucket count %u, actual %u\n", \ + (head)->hh.tbl->buckets[_bkt_i].count, _bkt_count); \ + } \ + } \ + if (_count != (head)->hh.tbl->num_items) { \ + HASH_OOPS("invalid hh item count %u, actual %u\n", \ + (head)->hh.tbl->num_items, _count ); \ + } \ + /* traverse hh in app order; check next/prev integrity, count */ \ + _count = 0; \ + _prev = NULL; \ + _thh = &(head)->hh; \ + while (_thh) { \ + _count++; \ + if (_prev !=(char*)(_thh->prev)) { \ + HASH_OOPS("invalid prev %p, actual %p\n", \ + _thh->prev, _prev ); \ + } \ + _prev = (char*)ELMT_FROM_HH((head)->hh.tbl, _thh); \ + _thh = ( _thh->next ? (UT_hash_handle*)((char*)(_thh->next) + \ + (head)->hh.tbl->hho) : NULL ); \ + } \ + if (_count != (head)->hh.tbl->num_items) { \ + HASH_OOPS("invalid app item count %u, actual %u\n", \ + (head)->hh.tbl->num_items, _count ); \ + } \ + } \ +} while (0) +#else +#define HASH_FSCK(hh,head) +#endif + +/* When compiled with -DHASH_EMIT_KEYS, length-prefixed keys are emitted to + * the descriptor to which this macro is defined for tuning the hash function. + * The app can #include to get the prototype for write(2). */ +#ifdef HASH_EMIT_KEYS +#define HASH_EMIT_KEY(hh,head,keyptr,fieldlen) \ +do { \ + unsigned _klen = fieldlen; \ + write(HASH_EMIT_KEYS, &_klen, sizeof(_klen)); \ + write(HASH_EMIT_KEYS, keyptr, (unsigned long)fieldlen); \ +} while (0) +#else +#define HASH_EMIT_KEY(hh,head,keyptr,fieldlen) +#endif + +/* default to Jenkin's hash unless overridden e.g. DHASH_FUNCTION=HASH_SAX */ +#ifdef HASH_FUNCTION +#define HASH_FCN HASH_FUNCTION +#else +#define HASH_FCN HASH_JEN +#endif + +/* The Bernstein hash function, used in Perl prior to v5.6. Note (x<<5+x)=x*33. */ +#define HASH_BER(key,keylen,hashv) \ +do { \ + unsigned _hb_keylen=(unsigned)keylen; \ + const unsigned char *_hb_key=(const unsigned char*)(key); \ + (hashv) = 0; \ + while (_hb_keylen-- != 0U) { \ + (hashv) = (((hashv) << 5) + (hashv)) + *_hb_key++; \ + } \ +} while (0) + + +/* SAX/FNV/OAT/JEN hash functions are macro variants of those listed at + * http://eternallyconfuzzled.com/tuts/algorithms/jsw_tut_hashing.aspx */ +#define HASH_SAX(key,keylen,hashv) \ +do { \ + unsigned _sx_i; \ + const unsigned char *_hs_key=(const unsigned char*)(key); \ + hashv = 0; \ + for(_sx_i=0; _sx_i < keylen; _sx_i++) { \ + hashv ^= (hashv << 5) + (hashv >> 2) + _hs_key[_sx_i]; \ + } \ +} while (0) +/* FNV-1a variation */ +#define HASH_FNV(key,keylen,hashv) \ +do { \ + unsigned _fn_i; \ + const unsigned char *_hf_key=(const unsigned char*)(key); \ + hashv = 2166136261U; \ + for(_fn_i=0; _fn_i < keylen; _fn_i++) { \ + hashv = hashv ^ _hf_key[_fn_i]; \ + hashv = hashv * 16777619U; \ + } \ +} while (0) + +#define HASH_OAT(key,keylen,hashv) \ +do { \ + unsigned _ho_i; \ + const unsigned char *_ho_key=(const unsigned char*)(key); \ + hashv = 0; \ + for(_ho_i=0; _ho_i < keylen; _ho_i++) { \ + hashv += _ho_key[_ho_i]; \ + hashv += (hashv << 10); \ + hashv ^= (hashv >> 6); \ + } \ + hashv += (hashv << 3); \ + hashv ^= (hashv >> 11); \ + hashv += (hashv << 15); \ +} while (0) + +#define HASH_JEN_MIX(a,b,c) \ +do { \ + a -= b; a -= c; a ^= ( c >> 13 ); \ + b -= c; b -= a; b ^= ( a << 8 ); \ + c -= a; c -= b; c ^= ( b >> 13 ); \ + a -= b; a -= c; a ^= ( c >> 12 ); \ + b -= c; b -= a; b ^= ( a << 16 ); \ + c -= a; c -= b; c ^= ( b >> 5 ); \ + a -= b; a -= c; a ^= ( c >> 3 ); \ + b -= c; b -= a; b ^= ( a << 10 ); \ + c -= a; c -= b; c ^= ( b >> 15 ); \ +} while (0) + +#define HASH_JEN(key,keylen,hashv) \ +do { \ + unsigned _hj_i,_hj_j,_hj_k; \ + unsigned const char *_hj_key=(unsigned const char*)(key); \ + hashv = 0xfeedbeefu; \ + _hj_i = _hj_j = 0x9e3779b9u; \ + _hj_k = (unsigned)(keylen); \ + while (_hj_k >= 12U) { \ + _hj_i += (_hj_key[0] + ( (unsigned)_hj_key[1] << 8 ) \ + + ( (unsigned)_hj_key[2] << 16 ) \ + + ( (unsigned)_hj_key[3] << 24 ) ); \ + _hj_j += (_hj_key[4] + ( (unsigned)_hj_key[5] << 8 ) \ + + ( (unsigned)_hj_key[6] << 16 ) \ + + ( (unsigned)_hj_key[7] << 24 ) ); \ + hashv += (_hj_key[8] + ( (unsigned)_hj_key[9] << 8 ) \ + + ( (unsigned)_hj_key[10] << 16 ) \ + + ( (unsigned)_hj_key[11] << 24 ) ); \ + \ + HASH_JEN_MIX(_hj_i, _hj_j, hashv); \ + \ + _hj_key += 12; \ + _hj_k -= 12U; \ + } \ + hashv += (unsigned)(keylen); \ + switch ( _hj_k ) { \ + case 11: hashv += ( (unsigned)_hj_key[10] << 24 ); /* FALLTHROUGH */ \ + case 10: hashv += ( (unsigned)_hj_key[9] << 16 ); /* FALLTHROUGH */ \ + case 9: hashv += ( (unsigned)_hj_key[8] << 8 ); /* FALLTHROUGH */ \ + case 8: _hj_j += ( (unsigned)_hj_key[7] << 24 ); /* FALLTHROUGH */ \ + case 7: _hj_j += ( (unsigned)_hj_key[6] << 16 ); /* FALLTHROUGH */ \ + case 6: _hj_j += ( (unsigned)_hj_key[5] << 8 ); /* FALLTHROUGH */ \ + case 5: _hj_j += _hj_key[4]; /* FALLTHROUGH */ \ + case 4: _hj_i += ( (unsigned)_hj_key[3] << 24 ); /* FALLTHROUGH */ \ + case 3: _hj_i += ( (unsigned)_hj_key[2] << 16 ); /* FALLTHROUGH */ \ + case 2: _hj_i += ( (unsigned)_hj_key[1] << 8 ); /* FALLTHROUGH */ \ + case 1: _hj_i += _hj_key[0]; \ + } \ + HASH_JEN_MIX(_hj_i, _hj_j, hashv); \ +} while (0) + +/* The Paul Hsieh hash function */ +#undef get16bits +#if (defined(__GNUC__) && defined(__i386__)) || defined(__WATCOMC__) \ + || defined(_MSC_VER) || defined (__BORLANDC__) || defined (__TURBOC__) +#define get16bits(d) (*((const uint16_t *) (d))) +#endif + +#if !defined (get16bits) +#define get16bits(d) ((((uint32_t)(((const uint8_t *)(d))[1])) << 8) \ + +(uint32_t)(((const uint8_t *)(d))[0]) ) +#endif +#define HASH_SFH(key,keylen,hashv) \ +do { \ + unsigned const char *_sfh_key=(unsigned const char*)(key); \ + uint32_t _sfh_tmp, _sfh_len = (uint32_t)keylen; \ + \ + unsigned _sfh_rem = _sfh_len & 3U; \ + _sfh_len >>= 2; \ + hashv = 0xcafebabeu; \ + \ + /* Main loop */ \ + for (;_sfh_len > 0U; _sfh_len--) { \ + hashv += get16bits (_sfh_key); \ + _sfh_tmp = ((uint32_t)(get16bits (_sfh_key+2)) << 11) ^ hashv; \ + hashv = (hashv << 16) ^ _sfh_tmp; \ + _sfh_key += 2U*sizeof (uint16_t); \ + hashv += hashv >> 11; \ + } \ + \ + /* Handle end cases */ \ + switch (_sfh_rem) { \ + case 3: hashv += get16bits (_sfh_key); \ + hashv ^= hashv << 16; \ + hashv ^= (uint32_t)(_sfh_key[sizeof (uint16_t)]) << 18; \ + hashv += hashv >> 11; \ + break; \ + case 2: hashv += get16bits (_sfh_key); \ + hashv ^= hashv << 11; \ + hashv += hashv >> 17; \ + break; \ + case 1: hashv += *_sfh_key; \ + hashv ^= hashv << 10; \ + hashv += hashv >> 1; \ + } \ + \ + /* Force "avalanching" of final 127 bits */ \ + hashv ^= hashv << 3; \ + hashv += hashv >> 5; \ + hashv ^= hashv << 4; \ + hashv += hashv >> 17; \ + hashv ^= hashv << 25; \ + hashv += hashv >> 6; \ +} while (0) + +#ifdef HASH_USING_NO_STRICT_ALIASING +/* The MurmurHash exploits some CPU's (x86,x86_64) tolerance for unaligned reads. + * For other types of CPU's (e.g. Sparc) an unaligned read causes a bus error. + * MurmurHash uses the faster approach only on CPU's where we know it's safe. + * + * Note the preprocessor built-in defines can be emitted using: + * + * gcc -m64 -dM -E - < /dev/null (on gcc) + * cc -## a.c (where a.c is a simple test file) (Sun Studio) + */ +#if (defined(__i386__) || defined(__x86_64__) || defined(_M_IX86)) +#define MUR_GETBLOCK(p,i) p[i] +#else /* non intel */ +#define MUR_PLUS0_ALIGNED(p) (((unsigned long)p & 3UL) == 0UL) +#define MUR_PLUS1_ALIGNED(p) (((unsigned long)p & 3UL) == 1UL) +#define MUR_PLUS2_ALIGNED(p) (((unsigned long)p & 3UL) == 2UL) +#define MUR_PLUS3_ALIGNED(p) (((unsigned long)p & 3UL) == 3UL) +#define WP(p) ((uint32_t*)((unsigned long)(p) & ~3UL)) +#if (defined(__BIG_ENDIAN__) || defined(SPARC) || defined(__ppc__) || defined(__ppc64__)) +#define MUR_THREE_ONE(p) ((((*WP(p))&0x00ffffff) << 8) | (((*(WP(p)+1))&0xff000000) >> 24)) +#define MUR_TWO_TWO(p) ((((*WP(p))&0x0000ffff) <<16) | (((*(WP(p)+1))&0xffff0000) >> 16)) +#define MUR_ONE_THREE(p) ((((*WP(p))&0x000000ff) <<24) | (((*(WP(p)+1))&0xffffff00) >> 8)) +#else /* assume little endian non-intel */ +#define MUR_THREE_ONE(p) ((((*WP(p))&0xffffff00) >> 8) | (((*(WP(p)+1))&0x000000ff) << 24)) +#define MUR_TWO_TWO(p) ((((*WP(p))&0xffff0000) >>16) | (((*(WP(p)+1))&0x0000ffff) << 16)) +#define MUR_ONE_THREE(p) ((((*WP(p))&0xff000000) >>24) | (((*(WP(p)+1))&0x00ffffff) << 8)) +#endif +#define MUR_GETBLOCK(p,i) (MUR_PLUS0_ALIGNED(p) ? ((p)[i]) : \ + (MUR_PLUS1_ALIGNED(p) ? MUR_THREE_ONE(p) : \ + (MUR_PLUS2_ALIGNED(p) ? MUR_TWO_TWO(p) : \ + MUR_ONE_THREE(p)))) +#endif +#define MUR_ROTL32(x,r) (((x) << (r)) | ((x) >> (32 - (r)))) +#define MUR_FMIX(_h) \ +do { \ + _h ^= _h >> 16; \ + _h *= 0x85ebca6bu; \ + _h ^= _h >> 13; \ + _h *= 0xc2b2ae35u; \ + _h ^= _h >> 16; \ +} while (0) + +#define HASH_MUR(key,keylen,hashv) \ +do { \ + const uint8_t *_mur_data = (const uint8_t*)(key); \ + const int _mur_nblocks = (int)(keylen) / 4; \ + uint32_t _mur_h1 = 0xf88D5353u; \ + uint32_t _mur_c1 = 0xcc9e2d51u; \ + uint32_t _mur_c2 = 0x1b873593u; \ + uint32_t _mur_k1 = 0; \ + const uint8_t *_mur_tail; \ + const uint32_t *_mur_blocks = (const uint32_t*)(_mur_data+(_mur_nblocks*4)); \ + int _mur_i; \ + for(_mur_i = -_mur_nblocks; _mur_i!=0; _mur_i++) { \ + _mur_k1 = MUR_GETBLOCK(_mur_blocks,_mur_i); \ + _mur_k1 *= _mur_c1; \ + _mur_k1 = MUR_ROTL32(_mur_k1,15); \ + _mur_k1 *= _mur_c2; \ + \ + _mur_h1 ^= _mur_k1; \ + _mur_h1 = MUR_ROTL32(_mur_h1,13); \ + _mur_h1 = (_mur_h1*5U) + 0xe6546b64u; \ + } \ + _mur_tail = (const uint8_t*)(_mur_data + (_mur_nblocks*4)); \ + _mur_k1=0; \ + switch((keylen) & 3U) { \ + case 3: _mur_k1 ^= (uint32_t)_mur_tail[2] << 16; /* FALLTHROUGH */ \ + case 2: _mur_k1 ^= (uint32_t)_mur_tail[1] << 8; /* FALLTHROUGH */ \ + case 1: _mur_k1 ^= (uint32_t)_mur_tail[0]; \ + _mur_k1 *= _mur_c1; \ + _mur_k1 = MUR_ROTL32(_mur_k1,15); \ + _mur_k1 *= _mur_c2; \ + _mur_h1 ^= _mur_k1; \ + } \ + _mur_h1 ^= (uint32_t)(keylen); \ + MUR_FMIX(_mur_h1); \ + hashv = _mur_h1; \ +} while (0) +#endif /* HASH_USING_NO_STRICT_ALIASING */ + +/* iterate over items in a known bucket to find desired item */ +#define HASH_FIND_IN_BKT(tbl,hh,head,keyptr,keylen_in,hashval,out) \ +do { \ + if ((head).hh_head != NULL) { \ + DECLTYPE_ASSIGN(out, ELMT_FROM_HH(tbl, (head).hh_head)); \ + } else { \ + (out) = NULL; \ + } \ + while ((out) != NULL) { \ + if ((out)->hh.hashv == (hashval) && (out)->hh.keylen == (keylen_in)) { \ + if (uthash_memcmp((out)->hh.key, keyptr, keylen_in) == 0) { \ + break; \ + } \ + } \ + if ((out)->hh.hh_next != NULL) { \ + DECLTYPE_ASSIGN(out, ELMT_FROM_HH(tbl, (out)->hh.hh_next)); \ + } else { \ + (out) = NULL; \ + } \ + } \ +} while (0) + +/* add an item to a bucket */ +#define HASH_ADD_TO_BKT(head,addhh) \ +do { \ + head.count++; \ + (addhh)->hh_next = head.hh_head; \ + (addhh)->hh_prev = NULL; \ + if (head.hh_head != NULL) { (head).hh_head->hh_prev = (addhh); } \ + (head).hh_head=addhh; \ + if ((head.count >= ((head.expand_mult+1U) * HASH_BKT_CAPACITY_THRESH)) \ + && ((addhh)->tbl->noexpand != 1U)) { \ + HASH_EXPAND_BUCKETS((addhh)->tbl); \ + } \ +} while (0) + +/* remove an item from a given bucket */ +#define HASH_DEL_IN_BKT(hh,head,hh_del) \ + (head).count--; \ + if ((head).hh_head == hh_del) { \ + (head).hh_head = hh_del->hh_next; \ + } \ + if (hh_del->hh_prev) { \ + hh_del->hh_prev->hh_next = hh_del->hh_next; \ + } \ + if (hh_del->hh_next) { \ + hh_del->hh_next->hh_prev = hh_del->hh_prev; \ + } + +/* Bucket expansion has the effect of doubling the number of buckets + * and redistributing the items into the new buckets. Ideally the + * items will distribute more or less evenly into the new buckets + * (the extent to which this is true is a measure of the quality of + * the hash function as it applies to the key domain). + * + * With the items distributed into more buckets, the chain length + * (item count) in each bucket is reduced. Thus by expanding buckets + * the hash keeps a bound on the chain length. This bounded chain + * length is the essence of how a hash provides constant time lookup. + * + * The calculation of tbl->ideal_chain_maxlen below deserves some + * explanation. First, keep in mind that we're calculating the ideal + * maximum chain length based on the *new* (doubled) bucket count. + * In fractions this is just n/b (n=number of items,b=new num buckets). + * Since the ideal chain length is an integer, we want to calculate + * ceil(n/b). We don't depend on floating point arithmetic in this + * hash, so to calculate ceil(n/b) with integers we could write + * + * ceil(n/b) = (n/b) + ((n%b)?1:0) + * + * and in fact a previous version of this hash did just that. + * But now we have improved things a bit by recognizing that b is + * always a power of two. We keep its base 2 log handy (call it lb), + * so now we can write this with a bit shift and logical AND: + * + * ceil(n/b) = (n>>lb) + ( (n & (b-1)) ? 1:0) + * + */ +#define HASH_EXPAND_BUCKETS(tbl) \ +do { \ + unsigned _he_bkt; \ + unsigned _he_bkt_i; \ + struct UT_hash_handle *_he_thh, *_he_hh_nxt; \ + UT_hash_bucket *_he_new_buckets, *_he_newbkt; \ + _he_new_buckets = (UT_hash_bucket*)uthash_malloc( \ + 2UL * tbl->num_buckets * sizeof(struct UT_hash_bucket)); \ + if (!_he_new_buckets) { uthash_fatal( "out of memory"); } \ + memset(_he_new_buckets, 0, \ + 2UL * tbl->num_buckets * sizeof(struct UT_hash_bucket)); \ + tbl->ideal_chain_maxlen = \ + (tbl->num_items >> (tbl->log2_num_buckets+1U)) + \ + (((tbl->num_items & ((tbl->num_buckets*2U)-1U)) != 0U) ? 1U : 0U); \ + tbl->nonideal_items = 0; \ + for(_he_bkt_i = 0; _he_bkt_i < tbl->num_buckets; _he_bkt_i++) \ + { \ + _he_thh = tbl->buckets[ _he_bkt_i ].hh_head; \ + while (_he_thh != NULL) { \ + _he_hh_nxt = _he_thh->hh_next; \ + HASH_TO_BKT( _he_thh->hashv, tbl->num_buckets*2U, _he_bkt); \ + _he_newbkt = &(_he_new_buckets[ _he_bkt ]); \ + if (++(_he_newbkt->count) > tbl->ideal_chain_maxlen) { \ + tbl->nonideal_items++; \ + _he_newbkt->expand_mult = _he_newbkt->count / \ + tbl->ideal_chain_maxlen; \ + } \ + _he_thh->hh_prev = NULL; \ + _he_thh->hh_next = _he_newbkt->hh_head; \ + if (_he_newbkt->hh_head != NULL) { _he_newbkt->hh_head->hh_prev = \ + _he_thh; } \ + _he_newbkt->hh_head = _he_thh; \ + _he_thh = _he_hh_nxt; \ + } \ + } \ + uthash_free( tbl->buckets, tbl->num_buckets*sizeof(struct UT_hash_bucket) ); \ + tbl->num_buckets *= 2U; \ + tbl->log2_num_buckets++; \ + tbl->buckets = _he_new_buckets; \ + tbl->ineff_expands = (tbl->nonideal_items > (tbl->num_items >> 1)) ? \ + (tbl->ineff_expands+1U) : 0U; \ + if (tbl->ineff_expands > 1U) { \ + tbl->noexpand=1; \ + uthash_noexpand_fyi(tbl); \ + } \ + uthash_expand_fyi(tbl); \ +} while (0) + + +/* This is an adaptation of Simon Tatham's O(n log(n)) mergesort */ +/* Note that HASH_SORT assumes the hash handle name to be hh. + * HASH_SRT was added to allow the hash handle name to be passed in. */ +#define HASH_SORT(head,cmpfcn) HASH_SRT(hh,head,cmpfcn) +#define HASH_SRT(hh,head,cmpfcn) \ +do { \ + unsigned _hs_i; \ + unsigned _hs_looping,_hs_nmerges,_hs_insize,_hs_psize,_hs_qsize; \ + struct UT_hash_handle *_hs_p, *_hs_q, *_hs_e, *_hs_list, *_hs_tail; \ + if (head != NULL) { \ + _hs_insize = 1; \ + _hs_looping = 1; \ + _hs_list = &((head)->hh); \ + while (_hs_looping != 0U) { \ + _hs_p = _hs_list; \ + _hs_list = NULL; \ + _hs_tail = NULL; \ + _hs_nmerges = 0; \ + while (_hs_p != NULL) { \ + _hs_nmerges++; \ + _hs_q = _hs_p; \ + _hs_psize = 0; \ + for ( _hs_i = 0; _hs_i < _hs_insize; _hs_i++ ) { \ + _hs_psize++; \ + _hs_q = (UT_hash_handle*)((_hs_q->next != NULL) ? \ + ((void*)((char*)(_hs_q->next) + \ + (head)->hh.tbl->hho)) : NULL); \ + if (! (_hs_q) ) { break; } \ + } \ + _hs_qsize = _hs_insize; \ + while ((_hs_psize > 0U) || ((_hs_qsize > 0U) && (_hs_q != NULL))) {\ + if (_hs_psize == 0U) { \ + _hs_e = _hs_q; \ + _hs_q = (UT_hash_handle*)((_hs_q->next != NULL) ? \ + ((void*)((char*)(_hs_q->next) + \ + (head)->hh.tbl->hho)) : NULL); \ + _hs_qsize--; \ + } else if ( (_hs_qsize == 0U) || (_hs_q == NULL) ) { \ + _hs_e = _hs_p; \ + if (_hs_p != NULL){ \ + _hs_p = (UT_hash_handle*)((_hs_p->next != NULL) ? \ + ((void*)((char*)(_hs_p->next) + \ + (head)->hh.tbl->hho)) : NULL); \ + } \ + _hs_psize--; \ + } else if (( \ + cmpfcn(DECLTYPE(head)(ELMT_FROM_HH((head)->hh.tbl,_hs_p)), \ + DECLTYPE(head)(ELMT_FROM_HH((head)->hh.tbl,_hs_q))) \ + ) <= 0) { \ + _hs_e = _hs_p; \ + if (_hs_p != NULL){ \ + _hs_p = (UT_hash_handle*)((_hs_p->next != NULL) ? \ + ((void*)((char*)(_hs_p->next) + \ + (head)->hh.tbl->hho)) : NULL); \ + } \ + _hs_psize--; \ + } else { \ + _hs_e = _hs_q; \ + _hs_q = (UT_hash_handle*)((_hs_q->next != NULL) ? \ + ((void*)((char*)(_hs_q->next) + \ + (head)->hh.tbl->hho)) : NULL); \ + _hs_qsize--; \ + } \ + if ( _hs_tail != NULL ) { \ + _hs_tail->next = ((_hs_e != NULL) ? \ + ELMT_FROM_HH((head)->hh.tbl,_hs_e) : NULL); \ + } else { \ + _hs_list = _hs_e; \ + } \ + if (_hs_e != NULL) { \ + _hs_e->prev = ((_hs_tail != NULL) ? \ + ELMT_FROM_HH((head)->hh.tbl,_hs_tail) : NULL); \ + } \ + _hs_tail = _hs_e; \ + } \ + _hs_p = _hs_q; \ + } \ + if (_hs_tail != NULL){ \ + _hs_tail->next = NULL; \ + } \ + if ( _hs_nmerges <= 1U ) { \ + _hs_looping=0; \ + (head)->hh.tbl->tail = _hs_tail; \ + DECLTYPE_ASSIGN(head,ELMT_FROM_HH((head)->hh.tbl, _hs_list)); \ + } \ + _hs_insize *= 2U; \ + } \ + HASH_FSCK(hh,head); \ + } \ +} while (0) + +/* This function selects items from one hash into another hash. + * The end result is that the selected items have dual presence + * in both hashes. There is no copy of the items made; rather + * they are added into the new hash through a secondary hash + * hash handle that must be present in the structure. */ +#define HASH_SELECT(hh_dst, dst, hh_src, src, cond) \ +do { \ + unsigned _src_bkt, _dst_bkt; \ + void *_last_elt=NULL, *_elt; \ + UT_hash_handle *_src_hh, *_dst_hh, *_last_elt_hh=NULL; \ + ptrdiff_t _dst_hho = ((char*)(&(dst)->hh_dst) - (char*)(dst)); \ + if (src != NULL) { \ + for(_src_bkt=0; _src_bkt < (src)->hh_src.tbl->num_buckets; _src_bkt++) { \ + for(_src_hh = (src)->hh_src.tbl->buckets[_src_bkt].hh_head; \ + _src_hh != NULL; \ + _src_hh = _src_hh->hh_next) { \ + _elt = ELMT_FROM_HH((src)->hh_src.tbl, _src_hh); \ + if (cond(_elt)) { \ + _dst_hh = (UT_hash_handle*)(((char*)_elt) + _dst_hho); \ + _dst_hh->key = _src_hh->key; \ + _dst_hh->keylen = _src_hh->keylen; \ + _dst_hh->hashv = _src_hh->hashv; \ + _dst_hh->prev = _last_elt; \ + _dst_hh->next = NULL; \ + if (_last_elt_hh != NULL) { _last_elt_hh->next = _elt; } \ + if (dst == NULL) { \ + DECLTYPE_ASSIGN(dst,_elt); \ + HASH_MAKE_TABLE(hh_dst,dst); \ + } else { \ + _dst_hh->tbl = (dst)->hh_dst.tbl; \ + } \ + HASH_TO_BKT(_dst_hh->hashv, _dst_hh->tbl->num_buckets, _dst_bkt); \ + HASH_ADD_TO_BKT(_dst_hh->tbl->buckets[_dst_bkt],_dst_hh); \ + (dst)->hh_dst.tbl->num_items++; \ + _last_elt = _elt; \ + _last_elt_hh = _dst_hh; \ + } \ + } \ + } \ + } \ + HASH_FSCK(hh_dst,dst); \ +} while (0) + +#define HASH_CLEAR(hh,head) \ +do { \ + if (head != NULL) { \ + uthash_free((head)->hh.tbl->buckets, \ + (head)->hh.tbl->num_buckets*sizeof(struct UT_hash_bucket)); \ + HASH_BLOOM_FREE((head)->hh.tbl); \ + uthash_free((head)->hh.tbl, sizeof(UT_hash_table)); \ + (head)=NULL; \ + } \ +} while (0) + +#define HASH_OVERHEAD(hh,head) \ + ((head != NULL) ? ( \ + (size_t)(((head)->hh.tbl->num_items * sizeof(UT_hash_handle)) + \ + ((head)->hh.tbl->num_buckets * sizeof(UT_hash_bucket)) + \ + sizeof(UT_hash_table) + \ + (HASH_BLOOM_BYTELEN))) : 0U) + +#ifdef NO_DECLTYPE +#define HASH_ITER(hh,head,el,tmp) \ +for(((el)=(head)), ((*(char**)(&(tmp)))=(char*)((head!=NULL)?(head)->hh.next:NULL)); \ + (el) != NULL; ((el)=(tmp)), ((*(char**)(&(tmp)))=(char*)((tmp!=NULL)?(tmp)->hh.next:NULL))) +#else +#define HASH_ITER(hh,head,el,tmp) \ +for(((el)=(head)), ((tmp)=DECLTYPE(el)((head!=NULL)?(head)->hh.next:NULL)); \ + (el) != NULL; ((el)=(tmp)), ((tmp)=DECLTYPE(el)((tmp!=NULL)?(tmp)->hh.next:NULL))) +#endif + +/* obtain a count of items in the hash */ +#define HASH_COUNT(head) HASH_CNT(hh,head) +#define HASH_CNT(hh,head) ((head != NULL)?((head)->hh.tbl->num_items):0U) + +typedef struct UT_hash_bucket { + struct UT_hash_handle *hh_head; + unsigned count; + + /* expand_mult is normally set to 0. In this situation, the max chain length + * threshold is enforced at its default value, HASH_BKT_CAPACITY_THRESH. (If + * the bucket's chain exceeds this length, bucket expansion is triggered). + * However, setting expand_mult to a non-zero value delays bucket expansion + * (that would be triggered by additions to this particular bucket) + * until its chain length reaches a *multiple* of HASH_BKT_CAPACITY_THRESH. + * (The multiplier is simply expand_mult+1). The whole idea of this + * multiplier is to reduce bucket expansions, since they are expensive, in + * situations where we know that a particular bucket tends to be overused. + * It is better to let its chain length grow to a longer yet-still-bounded + * value, than to do an O(n) bucket expansion too often. + */ + unsigned expand_mult; + +} UT_hash_bucket; + +/* random signature used only to find hash tables in external analysis */ +#define HASH_SIGNATURE 0xa0111fe1u +#define HASH_BLOOM_SIGNATURE 0xb12220f2u + +typedef struct UT_hash_table { + UT_hash_bucket *buckets; + unsigned num_buckets, log2_num_buckets; + unsigned num_items; + struct UT_hash_handle *tail; /* tail hh in app order, for fast append */ + ptrdiff_t hho; /* hash handle offset (byte pos of hash handle in element */ + + /* in an ideal situation (all buckets used equally), no bucket would have + * more than ceil(#items/#buckets) items. that's the ideal chain length. */ + unsigned ideal_chain_maxlen; + + /* nonideal_items is the number of items in the hash whose chain position + * exceeds the ideal chain maxlen. these items pay the penalty for an uneven + * hash distribution; reaching them in a chain traversal takes >ideal steps */ + unsigned nonideal_items; + + /* ineffective expands occur when a bucket doubling was performed, but + * afterward, more than half the items in the hash had nonideal chain + * positions. If this happens on two consecutive expansions we inhibit any + * further expansion, as it's not helping; this happens when the hash + * function isn't a good fit for the key domain. When expansion is inhibited + * the hash will still work, albeit no longer in constant time. */ + unsigned ineff_expands, noexpand; + + uint32_t signature; /* used only to find hash tables in external analysis */ +#ifdef HASH_BLOOM + uint32_t bloom_sig; /* used only to test bloom exists in external analysis */ + uint8_t *bloom_bv; + uint8_t bloom_nbits; +#endif + +} UT_hash_table; + +typedef struct UT_hash_handle { + struct UT_hash_table *tbl; + void *prev; /* prev element in app order */ + void *next; /* next element in app order */ + struct UT_hash_handle *hh_prev; /* previous hh in bucket order */ + struct UT_hash_handle *hh_next; /* next hh in bucket order */ + void *key; /* ptr to enclosing struct's key */ + unsigned keylen; /* enclosing struct's key len */ + unsigned hashv; /* result of hash-fcn(key) */ +} UT_hash_handle; + +#endif /* UTHASH_H */ From 313241e30321a5499fef51df461233c38cbe8261 Mon Sep 17 00:00:00 2001 From: Stephanie Wang Date: Tue, 20 Sep 2016 22:40:35 -0700 Subject: [PATCH 11/36] Asynchronous Redis IPC (#14) * Asynchronous Redis IPC * make valgrind happy * cleanup --- Makefile | 7 ++- common.h | 6 ++ io.c | 4 +- state/redis.c | 35 ++++++++++-- state/redis.h | 4 ++ test/db_tests.c | 7 ++- test/redis_tests.c | 138 +++++++++++++++++++++++++++++++++++++++++++++ 7 files changed, 189 insertions(+), 12 deletions(-) create mode 100644 test/redis_tests.c diff --git a/Makefile b/Makefile index 29b7befd9..f1659c882 100644 --- a/Makefile +++ b/Makefile @@ -16,6 +16,9 @@ $(BUILD)/io_tests: test/io_tests.c $(BUILD)/libcommon.a $(BUILD)/task_tests: test/task_tests.c $(BUILD)/libcommon.a $(CC) -o $@ $^ $(CFLAGS) +$(BUILD)/redis_tests: hiredis test/redis_tests.c $(BUILD)/libcommon.a + $(CC) -o $@ test/redis_tests.c $(BUILD)/libcommon.a thirdparty/hiredis/libhiredis.a $(CFLAGS) + clean: rm -f *.o state/*.o test/*.o rm -rf $(BUILD)/* @@ -26,8 +29,8 @@ redis: hiredis: git submodule update --init --recursive -- "thirdparty/hiredis" ; cd thirdparty/hiredis ; make -test: hiredis redis $(BUILD)/db_tests $(BUILD)/io_tests $(BUILD)/task_tests FORCE +test: hiredis redis $(BUILD)/db_tests $(BUILD)/io_tests $(BUILD)/task_tests $(BUILD)/redis_tests FORCE ./thirdparty/redis-3.2.3/src/redis-server & - sleep 1s ; ./build/db_tests ; ./build/io_tests ; ./build/task_tests + sleep 1s ; ./build/db_tests ; ./build/io_tests ; ./build/task_tests ; ./build/redis_tests FORCE: diff --git a/common.h b/common.h index 96e6402f0..3f30b5661 100644 --- a/common.h +++ b/common.h @@ -28,6 +28,12 @@ #define UNIQUE_ID_SIZE 20 +// Cleanup method for running tests with the greatest library. +// Runs the test, then clears the Redis database. +#define RUN_REDIS_TEST(context, test) \ + RUN_TEST(test); \ + freeReplyObject(redisCommand(context, "FLUSHALL")); + typedef struct { unsigned char id[UNIQUE_ID_SIZE]; } unique_id; /* Generate a globally unique ID. */ diff --git a/io.c b/io.c index 1d16a78e0..99295512b 100644 --- a/io.c +++ b/io.c @@ -10,7 +10,7 @@ #include "common.h" -/* Binds to a Unix domain datagram socket at the given +/* Binds to a Unix domain streaming socket at the given * pathname. Removes any existing file at the pathname. Returns * a file descriptor for the socket, or -1 if an error * occurred. */ @@ -44,7 +44,7 @@ int bind_ipc_sock(const char *socket_pathname) { return socket_fd; } -/* Connects to a Unix domain datagram socket at the given +/* Connects to a Unix domain streaming socket at the given * pathname. Returns a file descriptor for the socket, or -1 if * an error occurred. */ int connect_ipc_sock(const char *socket_pathname) { diff --git a/state/redis.c b/state/redis.c index c781b81ef..ae3fb1a6f 100644 --- a/state/redis.c +++ b/state/redis.c @@ -8,12 +8,13 @@ #include "task_queue.h" #include "event_loop.h" #include "redis.h" +#include "io.h" static void poll_add_read(void *privdata) { db_conn *conn = (db_conn *) privdata; if (!conn->reading) { conn->reading = 1; - event_loop_get(conn->loop, 0)->events |= POLLIN; + event_loop_get(conn->loop, conn->db_index)->events |= POLLIN; } } @@ -21,7 +22,7 @@ static void poll_del_read(void *privdata) { db_conn *conn = (db_conn *) privdata; if (conn->reading) { conn->reading = 0; - event_loop_get(conn->loop, 0)->events &= ~POLLIN; + event_loop_get(conn->loop, conn->db_index)->events &= ~POLLIN; } } @@ -29,7 +30,7 @@ static void poll_add_write(void *privdata) { db_conn *conn = (db_conn *) privdata; if (!conn->writing) { conn->writing = 1; - event_loop_get(conn->loop, 0)->events |= POLLOUT; + event_loop_get(conn->loop, conn->db_index)->events |= POLLOUT; } } @@ -37,7 +38,7 @@ static void poll_del_write(void *privdata) { db_conn *conn = (db_conn *) privdata; if (conn->writing) { conn->writing = 0; - event_loop_get(conn->loop, 0)->events &= ~POLLOUT; + event_loop_get(conn->loop, conn->db_index)->events &= ~POLLOUT; } } @@ -143,8 +144,10 @@ int64_t db_attach(db_conn *db, event_loop *loop, int connection_type) { ac->ev.data = db; - return event_loop_attach(loop, connection_type, NULL, c->fd, - POLLIN | POLLOUT); + int64_t index = + event_loop_attach(loop, connection_type, NULL, c->fd, POLLIN | POLLOUT); + db->db_index = index; + return index; } void object_table_add(db_conn *db, unique_id object_id) { @@ -211,3 +214,23 @@ void object_table_lookup(db_conn *db, LOG_REDIS_ERR(db->context, "error in object_table lookup"); } } + +void send_redis_command(int socket_fd, const char *format, ...) { + char *cmd; + va_list ap; + int len; + + va_start(ap, format); + len = redisvFormatCommand(&cmd, format, ap); + va_end(ap); + if (len == -1) { + LOG_ERR("Out of memory while formatting Redis command."); + return; + } else if (len == -2) { + LOG_ERR("Invalid Redis format string."); + return; + } + + write_string(socket_fd, cmd); + free(cmd); +} diff --git a/state/redis.h b/state/redis.h index ad8d5cbcf..132724e67 100644 --- a/state/redis.h +++ b/state/redis.h @@ -25,6 +25,8 @@ struct db_conn_impl { int reading, writing; /* The event loop this global state store connection is part of. */ event_loop *loop; + /* Index of the database connection in the event loop */ + int64_t db_index; /* Cache for the IP addresses of services. */ service_cache_entry *service_cache; /* Redis context for synchronous connections. @@ -44,3 +46,5 @@ void object_table_get_entry(redisAsyncContext *c, void *r, void *privdata); void object_table_lookup_callback(redisAsyncContext *c, void *r, void *privdata); + +void send_redis_command(int socket_fd, const char *format, ...); diff --git a/test/db_tests.c b/test/db_tests.c index 0788345f2..568eeaeb1 100644 --- a/test/db_tests.c +++ b/test/db_tests.c @@ -108,12 +108,15 @@ TEST object_table_lookup_test(void) { event_loop_free(&loop); + lookup_successful = 0; PASS(); } SUITE(db_tests) { - RUN_TEST(object_table_lookup_test); - /* RUN_TEST(task_queue_test); */ + redisContext *context = redisConnect("127.0.0.1", 6379); + redisCommand(context, "FLUSHALL"); + RUN_REDIS_TEST(context, object_table_lookup_test); + redisFree(context); } GREATEST_MAIN_DEFS(); diff --git a/test/redis_tests.c b/test/redis_tests.c new file mode 100644 index 000000000..9efd3bace --- /dev/null +++ b/test/redis_tests.c @@ -0,0 +1,138 @@ +#include "greatest.h" + +#include +#include + +#include "event_loop.h" +#include "state/db.h" +#include "state/redis.h" +#include "io.h" + +SUITE(redis_tests); + +int lookup_successful = 0; +const char *test_set_format = "SET %s %s"; +const char *test_get_format = "GET %s"; +const char *test_key = "foo"; +const char *test_value = "bar"; + +void async_redis_socket_test_callback(redisAsyncContext *ac, + void *r, + void *privdata) { + redisContext *context = redisConnect("127.0.0.1", 6379); + redisReply *reply = redisCommand(context, test_get_format, test_key); + redisFree(context); + assert(reply != NULL); + if (strcmp(reply->str, test_value)) { + freeReplyObject(reply); + assert(0); + } + freeReplyObject(reply); + lookup_successful = 1; +} + +TEST redis_socket_test(void) { + const char *socket_pathname = "redis-test-socket"; + redisContext *context = redisConnect("127.0.0.1", 6379); + ASSERT(context != NULL); + int socket_fd = bind_ipc_sock(socket_pathname); + ASSERT(socket_fd >= 0); + + int client_fd = connect_ipc_sock(socket_pathname); + ASSERT(client_fd >= 0); + + send_redis_command(client_fd, test_set_format, test_key, test_value); + + int server_fd = accept_client(socket_fd); + char *cmd = read_string(server_fd); + close(client_fd); + close(server_fd); + close(socket_fd); + unlink(socket_pathname); + + redisAppendFormattedCommand(context, cmd, strlen(cmd)); + redisReply *tmp; + redisGetReply(context, &tmp); + freeReplyObject(tmp); + redisReply *reply = redisCommand(context, "GET %s", test_key); + ASSERT(reply != NULL); + ASSERT_STR_EQ(reply->str, test_value); + freeReplyObject(reply); + + free(cmd); + redisFree(context); + PASS(); +} + +TEST async_redis_socket_test(void) { + int socket_fd, server_fd, client_fd; + event_loop loop; + event_loop_init(&loop); + /* Start IPC channel. */ + const char *socket_pathname = "async-redis-test-socket"; + socket_fd = bind_ipc_sock(socket_pathname); + ASSERT(socket_fd >= 0); + int64_t ipc_index = event_loop_attach(&loop, 1, NULL, socket_fd, POLLIN); + + /* Start connection to Redis. */ + db_conn conn; + db_connect("127.0.0.1", 6379, "", "", 0, &conn); + int64_t db_index = db_attach(&conn, &loop, 0); + + /* Send a command to the Redis process. */ + client_fd = connect_ipc_sock(socket_pathname); + ASSERT(client_fd >= 0); + send_redis_command(client_fd, test_set_format, test_key, test_value); + + while (!lookup_successful) { + int num_ready = event_loop_poll(&loop); + if (num_ready < 0) { + exit(-1); + } + for (int i = 0; i < event_loop_size(&loop); ++i) { + struct pollfd *waiting = event_loop_get(&loop, i); + if (waiting->revents == 0) + continue; + if (i == db_index) { + db_event(&conn); + } else if (i == ipc_index) { + /* For some reason, this check is necessary for Travis + * to pass these tests. */ + ASSERT(waiting->revents & POLLIN); + server_fd = accept_client(socket_fd); + ASSERT(server_fd >= 0); + event_loop_attach(&loop, 1, NULL, server_fd, POLLIN); + } else { + char *cmd = read_string(waiting->fd); + redisAsyncFormattedCommand(conn.context, + async_redis_socket_test_callback, NULL, cmd, + strlen(cmd)); + free(cmd); + } + } + } + db_disconnect(&conn); + event_loop_free(&loop); + close(server_fd); + close(client_fd); + close(socket_fd); + unlink(socket_pathname); + lookup_successful = 0; + PASS(); +} + +SUITE(redis_tests) { + redisContext *context = redisConnect("127.0.0.1", 6379); + freeReplyObject(redisCommand(context, "FLUSHALL")); + RUN_REDIS_TEST(context, redis_socket_test); + RUN_REDIS_TEST(context, async_redis_socket_test); + redisFree(context); +} + +GREATEST_MAIN_DEFS(); + +int main(int argc, char **argv) { + GREATEST_MAIN_BEGIN(); + RUN_SUITE(redis_tests); + GREATEST_MAIN_END(); +} From 7a079547b009ae2737bf5cb3a0cf87f46299681a Mon Sep 17 00:00:00 2001 From: Philipp Moritz Date: Thu, 22 Sep 2016 23:15:45 -0700 Subject: [PATCH 12/36] task queue tests and extensions (#18) * task queue tests and extensions * clean up test --- Makefile | 7 +- common.c | 60 +++++++ common.h | 9 +- event_loop.c | 6 +- event_loop.h | 2 +- state/redis.c | 18 ++ task.c | 96 +++++++++- task.h | 12 +- test/common_tests.c | 27 +++ test/db_tests.c | 39 ++++- test/example_task.h | 14 ++ test/redis_tests.c | 2 +- test/task_tests.c | 20 +++ thirdparty/utstring.h | 398 ++++++++++++++++++++++++++++++++++++++++++ 14 files changed, 695 insertions(+), 15 deletions(-) create mode 100644 test/common_tests.c create mode 100644 test/example_task.h create mode 100644 thirdparty/utstring.h diff --git a/Makefile b/Makefile index f1659c882..36793877f 100644 --- a/Makefile +++ b/Makefile @@ -7,6 +7,9 @@ all: $(BUILD)/libcommon.a $(BUILD)/libcommon.a: event_loop.o common.o task.o io.o state/redis.o ar rcs $@ $^ +$(BUILD)/common_tests: test/common_tests.c $(BUILD)/libcommon.a + $(CC) -o $@ test/common_tests.c $(BUILD)/libcommon.a $(CFLAGS) + $(BUILD)/db_tests: hiredis test/db_tests.c $(BUILD)/libcommon.a $(CC) -o $@ test/db_tests.c $(BUILD)/libcommon.a thirdparty/hiredis/libhiredis.a $(CFLAGS) @@ -29,8 +32,8 @@ redis: hiredis: git submodule update --init --recursive -- "thirdparty/hiredis" ; cd thirdparty/hiredis ; make -test: hiredis redis $(BUILD)/db_tests $(BUILD)/io_tests $(BUILD)/task_tests $(BUILD)/redis_tests FORCE +test: hiredis redis $(BUILD)/common_tests $(BUILD)/db_tests $(BUILD)/io_tests $(BUILD)/task_tests $(BUILD)/redis_tests FORCE ./thirdparty/redis-3.2.3/src/redis-server & - sleep 1s ; ./build/db_tests ; ./build/io_tests ; ./build/task_tests ; ./build/redis_tests + sleep 1s ; ./build/common_tests ; ./build/db_tests ; ./build/io_tests ; ./build/task_tests ; ./build/redis_tests FORCE: diff --git a/common.c b/common.c index 9e0a86310..53e32fe13 100644 --- a/common.c +++ b/common.c @@ -31,3 +31,63 @@ char *sha1_to_hex(const unsigned char *sha1, char *buffer) { return buffer; } + +const signed char hexval_table[256] = { + -1, -1, -1, -1, -1, -1, -1, -1, /* 00-07 */ + -1, -1, -1, -1, -1, -1, -1, -1, /* 08-0f */ + -1, -1, -1, -1, -1, -1, -1, -1, /* 10-17 */ + -1, -1, -1, -1, -1, -1, -1, -1, /* 18-1f */ + -1, -1, -1, -1, -1, -1, -1, -1, /* 20-27 */ + -1, -1, -1, -1, -1, -1, -1, -1, /* 28-2f */ + +0, +1, +2, +3, +4, +5, +6, +7, /* 30-37 */ + +8, +9, -1, -1, -1, -1, -1, -1, /* 38-3f */ + -1, 10, 11, 12, 13, 14, 15, -1, /* 40-47 */ + -1, -1, -1, -1, -1, -1, -1, -1, /* 48-4f */ + -1, -1, -1, -1, -1, -1, -1, -1, /* 50-57 */ + -1, -1, -1, -1, -1, -1, -1, -1, /* 58-5f */ + -1, 10, 11, 12, 13, 14, 15, -1, /* 60-67 */ + -1, -1, -1, -1, -1, -1, -1, -1, /* 68-67 */ + -1, -1, -1, -1, -1, -1, -1, -1, /* 70-77 */ + -1, -1, -1, -1, -1, -1, -1, -1, /* 78-7f */ + -1, -1, -1, -1, -1, -1, -1, -1, /* 80-87 */ + -1, -1, -1, -1, -1, -1, -1, -1, /* 88-8f */ + -1, -1, -1, -1, -1, -1, -1, -1, /* 90-97 */ + -1, -1, -1, -1, -1, -1, -1, -1, /* 98-9f */ + -1, -1, -1, -1, -1, -1, -1, -1, /* a0-a7 */ + -1, -1, -1, -1, -1, -1, -1, -1, /* a8-af */ + -1, -1, -1, -1, -1, -1, -1, -1, /* b0-b7 */ + -1, -1, -1, -1, -1, -1, -1, -1, /* b8-bf */ + -1, -1, -1, -1, -1, -1, -1, -1, /* c0-c7 */ + -1, -1, -1, -1, -1, -1, -1, -1, /* c8-cf */ + -1, -1, -1, -1, -1, -1, -1, -1, /* d0-d7 */ + -1, -1, -1, -1, -1, -1, -1, -1, /* d8-df */ + -1, -1, -1, -1, -1, -1, -1, -1, /* e0-e7 */ + -1, -1, -1, -1, -1, -1, -1, -1, /* e8-ef */ + -1, -1, -1, -1, -1, -1, -1, -1, /* f0-f7 */ + -1, -1, -1, -1, -1, -1, -1, -1, /* f8-ff */ +}; + +static inline unsigned int hexval(unsigned char c) { + return hexval_table[c]; +} + +/* + * Convert two consecutive hexadecimal digits into a char. Return a + * negative value on error. Don't run over the end of short strings. + */ +static inline int hex2chr(const char *s) { + int val = hexval(s[0]); + return (val < 0) ? val : (val << 4) | hexval(s[1]); +} + +int hex_to_sha1(const char *hex, unsigned char *sha1) { + int i; + for (i = 0; i < UNIQUE_ID_SIZE; i++) { + int val = hex2chr(hex); + if (val < 0) + return -1; + *sha1++ = val; + hex += 2; + } + return 0; +} diff --git a/common.h b/common.h index 3f30b5661..7e4b73ebe 100644 --- a/common.h +++ b/common.h @@ -28,8 +28,8 @@ #define UNIQUE_ID_SIZE 20 -// Cleanup method for running tests with the greatest library. -// Runs the test, then clears the Redis database. +/* Cleanup method for running tests with the greatest library. + * Runs the test, then clears the Redis database. */ #define RUN_REDIS_TEST(context, test) \ RUN_TEST(test); \ freeReplyObject(redisCommand(context, "FLUSHALL")); @@ -44,6 +44,11 @@ unique_id globally_unique_id(void); * UNIQUE_ID_SIZE + 1 */ char *sha1_to_hex(const unsigned char *sha1, char *buffer); +/* Convert a hexdecimal string of length 40 to a 20 byte sha1 hash. This + * function assumes that sha1 points to an already allocated char array of size + * UNIQUE_ID_SIZE. */ +int hex_to_sha1(const char *hex, unsigned char *sha1); + typedef unique_id object_id; #endif diff --git a/event_loop.c b/event_loop.c index 0fd79e6af..d89710ba5 100644 --- a/event_loop.c +++ b/event_loop.c @@ -59,10 +59,10 @@ void event_loop_detach(event_loop *loop, int64_t index, int shall_close) { } /* Poll the file descriptors associated to this event loop. - * See http://linux.die.net/man/2/poll */ -int event_loop_poll(event_loop *loop) { + * See http://linux.die.net/man/2/poll. The timeout is in milliseconds. */ +int event_loop_poll(event_loop *loop, int timeout) { return poll((struct pollfd *) utarray_front(loop->waiting), - utarray_len(loop->waiting), -1); + utarray_len(loop->waiting), timeout); } /* Get the total number of file descriptors participating in the event loop. */ diff --git a/event_loop.h b/event_loop.h index a96ec4643..840abfd8c 100644 --- a/event_loop.h +++ b/event_loop.h @@ -29,7 +29,7 @@ int64_t event_loop_attach(event_loop *loop, int fd, int events); void event_loop_detach(event_loop *loop, int64_t index, int shall_close); -int event_loop_poll(event_loop *loop); +int event_loop_poll(event_loop *loop, int timeout); int64_t event_loop_size(event_loop *loop); struct pollfd *event_loop_get(event_loop *loop, int64_t index); void event_loop_set_data(event_loop *loop, int64_t index, void *data); diff --git a/state/redis.c b/state/redis.c index ae3fb1a6f..c2e01df5e 100644 --- a/state/redis.c +++ b/state/redis.c @@ -2,6 +2,8 @@ #include +#include "utstring.h" + #include "common.h" #include "db.h" #include "object_table.h" @@ -215,6 +217,22 @@ void object_table_lookup(db_conn *db, } } +void task_queue_submit_task(db_conn *db, task_iid task_iid, task_spec *task) { + /* For converting an id to hex, which has double the number + * of bytes compared to the id (+ 1 byte for '\0'). */ + static char hex[2 * UNIQUE_ID_SIZE + 1]; + UT_string *command; + utstring_new(command); + sha1_to_hex(&task_iid.id[0], &hex[0]); + utstring_printf(command, "HMSET queue:%s", &hex[0]); + print_task(task, command); + redisAsyncCommand(db->context, NULL, NULL, utstring_body(command)); + if (db->context->err) { + LOG_REDIS_ERR(db->context, "error in task_queue submit_task"); + } + utstring_free(command); +} + void send_redis_command(int socket_fd, const char *format, ...) { char *cmd; va_list ap; diff --git a/task.c b/task.c index 5d5271d91..fd4708d78 100644 --- a/task.c +++ b/task.c @@ -2,6 +2,8 @@ #include #include +#include "utarray.h" + #include "task.h" #include "common.h" #include "io.h" @@ -30,7 +32,8 @@ typedef struct { } task_arg; struct task_spec_impl { - function_id func_id; + /* Function ID of the task. */ + function_id function_id; /* Total number of arguments. */ int64_t num_args; /* Index of the last argument that has been constructed. */ @@ -52,14 +55,14 @@ struct task_spec_impl { (sizeof(task_spec) + ((NUM_ARGS) + (NUM_RETURNS)) * sizeof(task_arg) + \ (ARGS_VALUE_SIZE)) -task_spec *alloc_task_spec(function_id func_id, +task_spec *alloc_task_spec(function_id function_id, int64_t num_args, int64_t num_returns, int64_t args_value_size) { int64_t size = TASK_SPEC_SIZE(num_args, num_returns, args_value_size); task_spec *task = malloc(size); memset(task, 0, size); - task->func_id = func_id; + task->function_id = function_id; task->num_args = num_args; task->arg_index = 0; task->num_returns = num_returns; @@ -72,6 +75,10 @@ int64_t task_size(task_spec *spec) { spec->args_value_size); } +unique_id *task_function(task_spec *spec) { + return &spec->function_id; +} + int64_t task_num_args(task_spec *spec) { return spec->num_args; } @@ -153,3 +160,86 @@ task_spec *read_task(int fd) { CHECK(task_size(spec) == length); return spec; } + +void print_task(task_spec *spec, UT_string *output) { + /* For converting an id to hex, which has double the number + * of bytes compared to the id (+ 1 byte for '\0'). */ + static char hex[2 * UNIQUE_ID_SIZE + 1]; + /* Print function id. */ + sha1_to_hex(&task_function(spec)->id[0], &hex[0]); + utstring_printf(output, "fun %s ", &hex[0]); + /* Print arguments. */ + for (int i = 0; i < task_num_args(spec); ++i) { + sha1_to_hex(&task_arg_id(spec, i)->id[0], &hex[0]); + utstring_printf(output, " id:%d %s", i, &hex[0]); + } + /* Print return ids. */ + for (int i = 0; i < task_num_returns(spec); ++i) { + object_id *object_id = task_return(spec, i); + sha1_to_hex(&object_id->id[0], &hex[0]); + utstring_printf(output, " ret:%d %s", i, &hex[0]); + } +} + +UT_icd unique_id_icd = {sizeof(unique_id), NULL, NULL, NULL}; + +task_spec *parse_task(char *task_string, int64_t task_length) { + /* We make one pass through task_string to store all the argument ids + * in "args" and all the return ids in "returns". */ + UT_array *args; + utarray_new(args, &unique_id_icd); + UT_array *returns; + utarray_new(returns, &unique_id_icd); + function_id function_id; + char *cursor = strtok(task_string, " "); + int index = 0; + while (cursor != NULL) { + /* This will be equal to "args" or "returns" depending on whether we + * are processing an argument id or a return id. */ + UT_array *target = NULL; + if (strncmp("fun", cursor, 3) == 0) { + /* Parse function id. */ + CHECK(cursor + 2 * UNIQUE_ID_SIZE + 1 <= task_string + task_length); + cursor = strtok(NULL, " "); + hex_to_sha1(cursor, &function_id.id[0]); + cursor = strtok(NULL, " "); + CHECK(cursor); + continue; + } else if (strncmp("id:", cursor, 3) == 0) { + /* Parse pass by reference argument. */ + sscanf(cursor, "id:%d", &index); + target = args; + } else if (strncmp("val:", cursor, 4) == 0) { + /* Parse pass by value argument. */ + sscanf(cursor, "val:%d", &index); + CHECK(0); /* Not implemented yet */ + } else if (strncmp("ret:", cursor, 4) == 0) { + /* Parse return object reference. */ + sscanf(cursor, "ret:%d", &index); + target = returns; + } + cursor = strtok(NULL, " "); + CHECK(cursor); + if (index >= utarray_len(target)) { + utarray_resize(target, index + 1); + } + object_id *id = (object_id *) utarray_eltptr(target, index); + hex_to_sha1(cursor, &id->id[0]); + cursor = strtok(NULL, " "); + } + /* TODO(pcm): Implement pass by value. */ + /* Now assemble the task specification. */ + task_spec *spec = + alloc_task_spec(function_id, utarray_len(args), utarray_len(returns), 0); + for (int i = 0; i < utarray_len(args); ++i) { + object_id *id = (object_id *) utarray_eltptr(args, i); + task_args_add_ref(spec, *id); + } + for (int i = 0; i < utarray_len(returns); ++i) { + object_id *id = (object_id *) utarray_eltptr(returns, i); + *task_return(spec, i) = *id; + } + utarray_free(args); + utarray_free(returns); + return spec; +} diff --git a/task.h b/task.h index ad85540ee..96c97b80d 100644 --- a/task.h +++ b/task.h @@ -10,6 +10,7 @@ #include #include #include "common.h" +#include "utstring.h" typedef unique_id function_id; typedef unique_id object_id; @@ -22,7 +23,7 @@ enum arg_type { ARG_BY_REF, ARG_BY_VAL }; /* Construct and modify task specifications. */ /* Allocating and initializing a task. */ -task_spec *alloc_task_spec(function_id func_id, +task_spec *alloc_task_spec(function_id function_id, int64_t num_args, int64_t num_returns, int64_t args_value_size); @@ -30,6 +31,9 @@ task_spec *alloc_task_spec(function_id func_id, /* Size of the task in bytes. */ int64_t task_size(task_spec *spec); +/* Return the function ID of the task. */ +unique_id *task_function(task_spec *spec); + /* Getting the number of arguments and returns. */ int64_t task_num_args(task_spec *spec); int64_t task_num_returns(task_spec *spec); @@ -58,4 +62,10 @@ void write_task(int fd, task_spec *spec); * responsibility to free the task after it has been used. */ task_spec *read_task(int fd); +/* Print task as a humanly readable string. */ +void print_task(task_spec *spec, UT_string *output); + +/* Parse task as printed by print_task. */ +task_spec *parse_task(char *task_string, int64_t task_length); + #endif diff --git a/test/common_tests.c b/test/common_tests.c new file mode 100644 index 000000000..3673c335d --- /dev/null +++ b/test/common_tests.c @@ -0,0 +1,27 @@ +#include "greatest.h" + +#include "common.h" + +SUITE(common_tests); + +TEST sha1_test(void) { + static char hex[2 * UNIQUE_ID_SIZE + 1]; + static unsigned char id[UNIQUE_ID_SIZE]; + unique_id uid = globally_unique_id(); + sha1_to_hex(&uid.id[0], &hex[0]); + hex_to_sha1(&hex[0], &id[0]); + ASSERT(memcmp(&uid.id[0], &id[0], 20) == 0); + PASS(); +} + +SUITE(common_tests) { + RUN_TEST(sha1_test); +} + +GREATEST_MAIN_DEFS(); + +int main(int argc, char **argv) { + GREATEST_MAIN_BEGIN(); + RUN_SUITE(common_tests); + GREATEST_MAIN_END(); +} diff --git a/test/db_tests.c b/test/db_tests.c index 568eeaeb1..5bdbf25df 100644 --- a/test/db_tests.c +++ b/test/db_tests.c @@ -3,6 +3,7 @@ #include #include "event_loop.h" +#include "test/example_task.h" #include "state/db.h" #include "state/object_table.h" #include "state/redis.h" @@ -62,7 +63,7 @@ TEST object_table_lookup_test(void) { object_table_add(&conn2, id); object_table_lookup(&conn1, id, sync_test_callback); while (!lookup_successful) { - int num_ready = event_loop_poll(&loop); + int num_ready = event_loop_poll(&loop, -1); if (num_ready < 0) { exit(-1); } @@ -81,7 +82,7 @@ TEST object_table_lookup_test(void) { lookup_successful = 0; object_table_lookup(&conn1, id, test_callback); while (!lookup_successful) { - int num_ready = event_loop_poll(&loop); + int num_ready = event_loop_poll(&loop, -1); if (num_ready < 0) { exit(-1); } @@ -112,10 +113,44 @@ TEST object_table_lookup_test(void) { PASS(); } +TEST task_queue_test(void) { + event_loop loop; + event_loop_init(&loop); + db_conn conn; + db_connect("127.0.0.1", 6379, "local_scheduler", "", -1, &conn); + int64_t index = db_attach(&conn, &loop, 0); + + task_spec *task = example_task(); + task_queue_submit_task(&conn, globally_unique_id(), task); + while (1) { + int num_ready = event_loop_poll(&loop, 100); + if (num_ready < 0) { + exit(-1); + } + if (num_ready == 0) { + break; + } + for (int i = 0; i < event_loop_size(&loop); ++i) { + struct pollfd *waiting = event_loop_get(&loop, i); + if (waiting->revents == 0) + continue; + if (i == index) { + db_event(&conn); + } + } + } + + free_task_spec(task); + db_disconnect(&conn); + event_loop_free(&loop); + PASS(); +} + SUITE(db_tests) { redisContext *context = redisConnect("127.0.0.1", 6379); redisCommand(context, "FLUSHALL"); RUN_REDIS_TEST(context, object_table_lookup_test); + RUN_TEST(task_queue_test); redisFree(context); } diff --git a/test/example_task.h b/test/example_task.h new file mode 100644 index 000000000..0dddc4dc1 --- /dev/null +++ b/test/example_task.h @@ -0,0 +1,14 @@ +#ifndef EXAMPLE_TASK_H +#define EXAMPLE_TASK_H + +#include "task.h" + +task_spec *example_task(void) { + function_id func_id = globally_unique_id(); + task_spec *task = alloc_task_spec(func_id, 2, 1, 0); + task_args_add_ref(task, globally_unique_id()); + task_args_add_ref(task, globally_unique_id()); + return task; +} + +#endif diff --git a/test/redis_tests.c b/test/redis_tests.c index 9efd3bace..2f0f09110 100644 --- a/test/redis_tests.c +++ b/test/redis_tests.c @@ -85,7 +85,7 @@ TEST async_redis_socket_test(void) { send_redis_command(client_fd, test_set_format, test_key, test_value); while (!lookup_successful) { - int num_ready = event_loop_poll(&loop); + int num_ready = event_loop_poll(&loop, -1); if (num_ready < 0) { exit(-1); } diff --git a/test/task_tests.c b/test/task_tests.c index 68a6a6537..4293eec04 100644 --- a/test/task_tests.c +++ b/test/task_tests.c @@ -5,6 +5,7 @@ #include #include "common.h" +#include "test/example_task.h" #include "task.h" SUITE(task_tests); @@ -56,9 +57,28 @@ TEST send_task(void) { PASS(); } +TEST print_and_parse_task(void) { + task_spec *task = example_task(); + + UT_string *output; + utstring_new(output); + print_task(task, output); + task_spec *result = parse_task(utstring_body(output), utstring_len(output)); + utstring_free(output); + + ASSERT_EQ(task_size(task), task_size(result)); + ASSERT(memcmp(task, result, task_size(task)) == 0); + + free_task_spec(task); + free_task_spec(result); + + PASS(); +} + SUITE(task_tests) { RUN_TEST(task_test); RUN_TEST(send_task); + RUN_TEST(print_and_parse_task); } GREATEST_MAIN_DEFS(); diff --git a/thirdparty/utstring.h b/thirdparty/utstring.h new file mode 100644 index 000000000..debe5f3df --- /dev/null +++ b/thirdparty/utstring.h @@ -0,0 +1,398 @@ +/* +Copyright (c) 2008-2016, Troy D. Hanson http://troydhanson.github.com/uthash/ +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + + * Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS +IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED +TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A +PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER +OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, +EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, +PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR +PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF +LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING +NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +*/ + +/* a dynamic string implementation using macros + */ +#ifndef UTSTRING_H +#define UTSTRING_H + +#define UTSTRING_VERSION 2.0.1 + +#ifdef __GNUC__ +#define _UNUSED_ __attribute__ ((__unused__)) +#else +#define _UNUSED_ +#endif + +#include +#include +#include +#include + +#ifndef oom +#define oom() exit(-1) +#endif + +typedef struct { + char *d; + size_t n; /* allocd size */ + size_t i; /* index of first unused byte */ +} UT_string; + +#define utstring_reserve(s,amt) \ +do { \ + if (((s)->n - (s)->i) < (size_t)(amt)) { \ + char *utstring_tmp = (char*)realloc( \ + (s)->d, (s)->n + (amt)); \ + if (utstring_tmp == NULL) oom(); \ + (s)->d = utstring_tmp; \ + (s)->n += (amt); \ + } \ +} while(0) + +#define utstring_init(s) \ +do { \ + (s)->n = 0; (s)->i = 0; (s)->d = NULL; \ + utstring_reserve(s,100); \ + (s)->d[0] = '\0'; \ +} while(0) + +#define utstring_done(s) \ +do { \ + if ((s)->d != NULL) free((s)->d); \ + (s)->n = 0; \ +} while(0) + +#define utstring_free(s) \ +do { \ + utstring_done(s); \ + free(s); \ +} while(0) + +#define utstring_new(s) \ +do { \ + s = (UT_string*)calloc(sizeof(UT_string),1); \ + if (!s) oom(); \ + utstring_init(s); \ +} while(0) + +#define utstring_renew(s) \ +do { \ + if (s) { \ + utstring_clear(s); \ + } else { \ + utstring_new(s); \ + } \ +} while(0) + +#define utstring_clear(s) \ +do { \ + (s)->i = 0; \ + (s)->d[0] = '\0'; \ +} while(0) + +#define utstring_bincpy(s,b,l) \ +do { \ + utstring_reserve((s),(l)+1); \ + if (l) memcpy(&(s)->d[(s)->i], b, l); \ + (s)->i += (l); \ + (s)->d[(s)->i]='\0'; \ +} while(0) + +#define utstring_concat(dst,src) \ +do { \ + utstring_reserve((dst),((src)->i)+1); \ + if ((src)->i) memcpy(&(dst)->d[(dst)->i], (src)->d, (src)->i); \ + (dst)->i += (src)->i; \ + (dst)->d[(dst)->i]='\0'; \ +} while(0) + +#define utstring_len(s) ((unsigned)((s)->i)) + +#define utstring_body(s) ((s)->d) + +_UNUSED_ static void utstring_printf_va(UT_string *s, const char *fmt, va_list ap) { + int n; + va_list cp; + for (;;) { +#ifdef _WIN32 + cp = ap; +#else + va_copy(cp, ap); +#endif + n = vsnprintf (&s->d[s->i], s->n-s->i, fmt, cp); + va_end(cp); + + if ((n > -1) && ((size_t) n < (s->n-s->i))) { + s->i += n; + return; + } + + /* Else try again with more space. */ + if (n > -1) utstring_reserve(s,n+1); /* exact */ + else utstring_reserve(s,(s->n)*2); /* 2x */ + } +} +#ifdef __GNUC__ +/* support printf format checking (2=the format string, 3=start of varargs) */ +static void utstring_printf(UT_string *s, const char *fmt, ...) + __attribute__ (( format( printf, 2, 3) )); +#endif +_UNUSED_ static void utstring_printf(UT_string *s, const char *fmt, ...) { + va_list ap; + va_start(ap,fmt); + utstring_printf_va(s,fmt,ap); + va_end(ap); +} + +/******************************************************************************* + * begin substring search functions * + ******************************************************************************/ +/* Build KMP table from left to right. */ +_UNUSED_ static void _utstring_BuildTable( + const char *P_Needle, + size_t P_NeedleLen, + long *P_KMP_Table) +{ + long i, j; + + i = 0; + j = i - 1; + P_KMP_Table[i] = j; + while (i < (long) P_NeedleLen) + { + while ( (j > -1) && (P_Needle[i] != P_Needle[j]) ) + { + j = P_KMP_Table[j]; + } + i++; + j++; + if (i < (long) P_NeedleLen) + { + if (P_Needle[i] == P_Needle[j]) + { + P_KMP_Table[i] = P_KMP_Table[j]; + } + else + { + P_KMP_Table[i] = j; + } + } + else + { + P_KMP_Table[i] = j; + } + } + + return; +} + + +/* Build KMP table from right to left. */ +_UNUSED_ static void _utstring_BuildTableR( + const char *P_Needle, + size_t P_NeedleLen, + long *P_KMP_Table) +{ + long i, j; + + i = P_NeedleLen - 1; + j = i + 1; + P_KMP_Table[i + 1] = j; + while (i >= 0) + { + while ( (j < (long) P_NeedleLen) && (P_Needle[i] != P_Needle[j]) ) + { + j = P_KMP_Table[j + 1]; + } + i--; + j--; + if (i >= 0) + { + if (P_Needle[i] == P_Needle[j]) + { + P_KMP_Table[i + 1] = P_KMP_Table[j + 1]; + } + else + { + P_KMP_Table[i + 1] = j; + } + } + else + { + P_KMP_Table[i + 1] = j; + } + } + + return; +} + + +/* Search data from left to right. ( Multiple search mode. ) */ +_UNUSED_ static long _utstring_find( + const char *P_Haystack, + size_t P_HaystackLen, + const char *P_Needle, + size_t P_NeedleLen, + long *P_KMP_Table) +{ + long i, j; + long V_FindPosition = -1; + + /* Search from left to right. */ + i = j = 0; + while ( (j < (int)P_HaystackLen) && (((P_HaystackLen - j) + i) >= P_NeedleLen) ) + { + while ( (i > -1) && (P_Needle[i] != P_Haystack[j]) ) + { + i = P_KMP_Table[i]; + } + i++; + j++; + if (i >= (int)P_NeedleLen) + { + /* Found. */ + V_FindPosition = j - i; + break; + } + } + + return V_FindPosition; +} + + +/* Search data from right to left. ( Multiple search mode. ) */ +_UNUSED_ static long _utstring_findR( + const char *P_Haystack, + size_t P_HaystackLen, + const char *P_Needle, + size_t P_NeedleLen, + long *P_KMP_Table) +{ + long i, j; + long V_FindPosition = -1; + + /* Search from right to left. */ + j = (P_HaystackLen - 1); + i = (P_NeedleLen - 1); + while ( (j >= 0) && (j >= i) ) + { + while ( (i < (int)P_NeedleLen) && (P_Needle[i] != P_Haystack[j]) ) + { + i = P_KMP_Table[i + 1]; + } + i--; + j--; + if (i < 0) + { + /* Found. */ + V_FindPosition = j + 1; + break; + } + } + + return V_FindPosition; +} + + +/* Search data from left to right. ( One time search mode. ) */ +_UNUSED_ static long utstring_find( + UT_string *s, + long P_StartPosition, /* Start from 0. -1 means last position. */ + const char *P_Needle, + size_t P_NeedleLen) +{ + long V_StartPosition; + long V_HaystackLen; + long *V_KMP_Table; + long V_FindPosition = -1; + + if (P_StartPosition < 0) + { + V_StartPosition = s->i + P_StartPosition; + } + else + { + V_StartPosition = P_StartPosition; + } + V_HaystackLen = s->i - V_StartPosition; + if ( (V_HaystackLen >= (long) P_NeedleLen) && (P_NeedleLen > 0) ) + { + V_KMP_Table = (long *)malloc(sizeof(long) * (P_NeedleLen + 1)); + if (V_KMP_Table != NULL) + { + _utstring_BuildTable(P_Needle, P_NeedleLen, V_KMP_Table); + + V_FindPosition = _utstring_find(s->d + V_StartPosition, + V_HaystackLen, + P_Needle, + P_NeedleLen, + V_KMP_Table); + if (V_FindPosition >= 0) + { + V_FindPosition += V_StartPosition; + } + + free(V_KMP_Table); + } + } + + return V_FindPosition; +} + + +/* Search data from right to left. ( One time search mode. ) */ +_UNUSED_ static long utstring_findR( + UT_string *s, + long P_StartPosition, /* Start from 0. -1 means last position. */ + const char *P_Needle, + size_t P_NeedleLen) +{ + long V_StartPosition; + long V_HaystackLen; + long *V_KMP_Table; + long V_FindPosition = -1; + + if (P_StartPosition < 0) + { + V_StartPosition = s->i + P_StartPosition; + } + else + { + V_StartPosition = P_StartPosition; + } + V_HaystackLen = V_StartPosition + 1; + if ( (V_HaystackLen >= (long) P_NeedleLen) && (P_NeedleLen > 0) ) + { + V_KMP_Table = (long *)malloc(sizeof(long) * (P_NeedleLen + 1)); + if (V_KMP_Table != NULL) + { + _utstring_BuildTableR(P_Needle, P_NeedleLen, V_KMP_Table); + + V_FindPosition = _utstring_findR(s->d, + V_HaystackLen, + P_Needle, + P_NeedleLen, + V_KMP_Table); + + free(V_KMP_Table); + } + } + + return V_FindPosition; +} +/******************************************************************************* + * end substring search functions * + ******************************************************************************/ + +#endif /* UTSTRING_H */ From e1b8711a017d5823375f7c973150446abcc92d19 Mon Sep 17 00:00:00 2001 From: Philipp Moritz Date: Fri, 23 Sep 2016 17:10:15 -0700 Subject: [PATCH 13/36] Redis logging (#17) * Redis logging * Rearrange logging interfaces * Fix test case * Changes to logging interface and test case for logging * Fixes * Fix memory leaks * Add interface method to destroy logger * is_local -> is_direct * Merge fix --- Makefile | 4 +-- io.c | 15 ++++++++ io.h | 1 + logging.c | 78 ++++++++++++++++++++++++++++++++++++++++ logging.h | 39 ++++++++++++++++++++ state/redis.c | 20 ----------- state/redis.h | 5 ++- test/redis_tests.c | 89 ++++++++++++++++++++++++++++++++++++++++------ 8 files changed, 217 insertions(+), 34 deletions(-) create mode 100644 logging.c create mode 100644 logging.h diff --git a/Makefile b/Makefile index 36793877f..aa5ee292e 100644 --- a/Makefile +++ b/Makefile @@ -19,8 +19,8 @@ $(BUILD)/io_tests: test/io_tests.c $(BUILD)/libcommon.a $(BUILD)/task_tests: test/task_tests.c $(BUILD)/libcommon.a $(CC) -o $@ $^ $(CFLAGS) -$(BUILD)/redis_tests: hiredis test/redis_tests.c $(BUILD)/libcommon.a - $(CC) -o $@ test/redis_tests.c $(BUILD)/libcommon.a thirdparty/hiredis/libhiredis.a $(CFLAGS) +$(BUILD)/redis_tests: hiredis test/redis_tests.c $(BUILD)/libcommon.a logging.h + $(CC) -o $@ test/redis_tests.c logging.c $(BUILD)/libcommon.a thirdparty/hiredis/libhiredis.a $(CFLAGS) clean: rm -f *.o state/*.o test/*.o diff --git a/io.c b/io.c index 99295512b..d4a89a4a4 100644 --- a/io.c +++ b/io.c @@ -7,6 +7,8 @@ #include #include #include +#include +#include #include "common.h" @@ -139,3 +141,16 @@ char *read_string(int fd) { read_bytes(fd, &bytes, &length); return (char *) bytes; } + +void write_formatted_string(int socket_fd, const char *format, ...) { + UT_string *cmd; + va_list ap; + + utstring_new(cmd); + va_start(ap, format); + utstring_printf_va(cmd, format, ap); + va_end(ap); + + write_string(socket_fd, utstring_body(cmd)); + utstring_free(cmd); +} diff --git a/io.h b/io.h index c6dd3bb30..e6f227c98 100644 --- a/io.h +++ b/io.h @@ -16,6 +16,7 @@ void write_bytes(int fd, uint8_t *bytes, int64_t length); void read_bytes(int fd, uint8_t **bytes, int64_t *length); void write_string(int fd, char *message); +void write_formatted_string(int fd, const char *format, ...); char *read_string(int fd); #endif diff --git a/logging.c b/logging.c new file mode 100644 index 000000000..38abc7bf9 --- /dev/null +++ b/logging.c @@ -0,0 +1,78 @@ +#include "logging.h" + +#include +#include + +#include "state/redis.h" +#include "io.h" + +static const char *log_levels[5] = {"DEBUG", "INFO", "WARN", "ERROR", "FATAL"}; +static const char *log_fmt = + "HMSET log:%s:%s:%s log_level %s event_type %s message %s timestamp %s"; + +struct ray_logger_impl { + /* String that identifies this client type. */ + char *client_type; + /* Suppress all log messages below this level. */ + int log_level; + /* Whether or not we have a direct connection to Redis. */ + int is_direct; + /* Either a db_conn or a socket to a process with a db_conn, + * depending on the is_direct flag. */ + void *conn; +}; + +ray_logger *init_ray_logger(const char *client_type, + int log_level, + int is_direct, + void *conn) { + ray_logger *logger = malloc(sizeof(ray_logger)); + logger->client_type = client_type; + logger->log_level = log_level; + logger->is_direct = is_direct; + logger->conn = conn; + return logger; +} + +void free_ray_logger(ray_logger *logger) { + free(logger); +} + +void ray_log(ray_logger *logger, + int log_level, + const char *event_type, + const char *message) { + if (log_level < logger->log_level) { + return; + } + if (log_level < RAY_DEBUG || log_level > RAY_FATAL) { + return; + } + struct timeval tv; + UT_string *timestamp; + utstring_new(timestamp); + gettimeofday(&tv, NULL); + utstring_printf(timestamp, "%ld.%ld", tv.tv_sec, tv.tv_usec); + + UT_string *origin_id; + utstring_new(origin_id); + if (logger->is_direct) { + db_conn *db = (db_conn *) logger->conn; + utstring_printf(origin_id, "%ld:%s", db->client_id, ""); + redisAsyncCommand(db->context, NULL, NULL, log_fmt, + utstring_body(timestamp), logger->client_type, + utstring_body(origin_id), log_levels[log_level], + event_type, message, utstring_body(timestamp)); + } else { + /* If we don't own a Redis connection, we leave our client + * ID to be filled in by someone else. */ + utstring_printf(origin_id, "%s:%s", "%ld", "%ld"); + int *socket_fd = (int *) logger->conn; + write_formatted_string(*socket_fd, log_fmt, utstring_body(timestamp), + logger->client_type, utstring_body(origin_id), + log_levels[log_level], event_type, message, + utstring_body(timestamp)); + } + utstring_free(origin_id); + utstring_free(timestamp); +} diff --git a/logging.h b/logging.h new file mode 100644 index 000000000..4ef7c8fcb --- /dev/null +++ b/logging.h @@ -0,0 +1,39 @@ +#ifndef LOGGING_H +#define LOGGING_H + +#define RAY_VERBOSE -1 +#define RAY_DEBUG 0 +#define RAY_INFO 1 +#define RAY_WARNING 2 +#define RAY_ERROR 3 +#define RAY_FATAL 4 + +/* Entity types. */ +#define RAY_FUNCTION "FUNCTION" +#define RAY_OBJECT "OBJECT" +#define RAY_TASK "TASK" + +typedef struct ray_logger_impl ray_logger; + +/* Initialize a Ray logger for the given client type and logging level. If the + * is_direct flag is set, the logger will treat the given connection as a + * direct connection to the log. Otherwise, it will treat it as a socket to + * another process with a connection to the log. + * NOTE: User is responsible for freeing the returned logger. */ +ray_logger *init_ray_logger(const char *client_type, + int log_level, + int is_direct, + void *conn); + +/* Free the logger. This does not free the connection to the log. */ +void free_ray_logger(ray_logger *logger); + +/* Log an event at the given log level with the given event_type. + * NOTE: message cannot contain spaces! JSON format is recommended. + * TODO: Support spaces in messages. */ +void ray_log(ray_logger *logger, + int log_level, + const char *event_type, + const char *message); + +#endif diff --git a/state/redis.c b/state/redis.c index c2e01df5e..ac56206e8 100644 --- a/state/redis.c +++ b/state/redis.c @@ -232,23 +232,3 @@ void task_queue_submit_task(db_conn *db, task_iid task_iid, task_spec *task) { } utstring_free(command); } - -void send_redis_command(int socket_fd, const char *format, ...) { - char *cmd; - va_list ap; - int len; - - va_start(ap, format); - len = redisvFormatCommand(&cmd, format, ap); - va_end(ap); - if (len == -1) { - LOG_ERR("Out of memory while formatting Redis command."); - return; - } else if (len == -2) { - LOG_ERR("Invalid Redis format string."); - return; - } - - write_string(socket_fd, cmd); - free(cmd); -} diff --git a/state/redis.h b/state/redis.h index 132724e67..09d66bea6 100644 --- a/state/redis.h +++ b/state/redis.h @@ -1,3 +1,6 @@ +#ifndef REDIS_H +#define REDIS_H + #include "db.h" #include "object_table.h" @@ -47,4 +50,4 @@ void object_table_lookup_callback(redisAsyncContext *c, void *r, void *privdata); -void send_redis_command(int socket_fd, const char *format, ...); +#endif diff --git a/test/redis_tests.c b/test/redis_tests.c index 2f0f09110..d8df538e8 100644 --- a/test/redis_tests.c +++ b/test/redis_tests.c @@ -7,6 +7,7 @@ #include "state/db.h" #include "state/redis.h" #include "io.h" +#include "logging.h" SUITE(redis_tests); @@ -31,6 +32,16 @@ void async_redis_socket_test_callback(redisAsyncContext *ac, lookup_successful = 1; } +void logging_test_callback(redisAsyncContext *ac, void *r, void *privdata) { + redisContext *context = redisConnect("127.0.0.1", 6379); + redisReply *reply = redisCommand(context, "KEYS %s", "log:*"); + redisFree(context); + assert(reply != NULL); + assert(reply->elements > 0); + freeReplyObject(reply); + lookup_successful = 1; +} + TEST redis_socket_test(void) { const char *socket_pathname = "redis-test-socket"; redisContext *context = redisConnect("127.0.0.1", 6379); @@ -40,8 +51,7 @@ TEST redis_socket_test(void) { int client_fd = connect_ipc_sock(socket_pathname); ASSERT(client_fd >= 0); - - send_redis_command(client_fd, test_set_format, test_key, test_value); + write_formatted_string(client_fd, test_set_format, test_key, test_value); int server_fd = accept_client(socket_fd); char *cmd = read_string(server_fd); @@ -50,11 +60,10 @@ TEST redis_socket_test(void) { close(socket_fd); unlink(socket_pathname); - redisAppendFormattedCommand(context, cmd, strlen(cmd)); - redisReply *tmp; - redisGetReply(context, &tmp); - freeReplyObject(tmp); - redisReply *reply = redisCommand(context, "GET %s", test_key); + redisReply *reply; + reply = redisCommand(context, cmd, 0, 0); + freeReplyObject(reply); + reply = redisCommand(context, "GET %s", test_key); ASSERT(reply != NULL); ASSERT_STR_EQ(reply->str, test_value); freeReplyObject(reply); @@ -82,7 +91,7 @@ TEST async_redis_socket_test(void) { /* Send a command to the Redis process. */ client_fd = connect_ipc_sock(socket_pathname); ASSERT(client_fd >= 0); - send_redis_command(client_fd, test_set_format, test_key, test_value); + write_formatted_string(client_fd, test_set_format, test_key, test_value); while (!lookup_successful) { int num_ready = event_loop_poll(&loop, -1); @@ -104,9 +113,8 @@ TEST async_redis_socket_test(void) { event_loop_attach(&loop, 1, NULL, server_fd, POLLIN); } else { char *cmd = read_string(waiting->fd); - redisAsyncFormattedCommand(conn.context, - async_redis_socket_test_callback, NULL, cmd, - strlen(cmd)); + redisAsyncCommand(conn.context, async_redis_socket_test_callback, NULL, + cmd, conn.client_id, 0); free(cmd); } } @@ -121,11 +129,70 @@ TEST async_redis_socket_test(void) { PASS(); } +TEST logging_test(void) { + int socket_fd, server_fd, client_fd; + event_loop loop; + event_loop_init(&loop); + /* Start IPC channel. */ + const char *socket_pathname = "logging-test-socket"; + socket_fd = bind_ipc_sock(socket_pathname); + ASSERT(socket_fd >= 0); + int64_t ipc_index = event_loop_attach(&loop, 1, NULL, socket_fd, POLLIN); + + /* Start connection to Redis. */ + db_conn conn; + db_connect("127.0.0.1", 6379, "", "", 0, &conn); + int64_t db_index = db_attach(&conn, &loop, 0); + + /* Send a command to the Redis process. */ + client_fd = connect_ipc_sock(socket_pathname); + ASSERT(client_fd >= 0); + ray_logger *logger = init_ray_logger("worker", RAY_INFO, 0, &client_fd); + ray_log(logger, RAY_INFO, "TEST", "Message"); + + while (!lookup_successful) { + int num_ready = event_loop_poll(&loop, -1); + if (num_ready < 0) { + exit(-1); + } + for (int i = 0; i < event_loop_size(&loop); ++i) { + struct pollfd *waiting = event_loop_get(&loop, i); + if (waiting->revents == 0) + continue; + if (i == db_index) { + db_event(&conn); + } else if (i == ipc_index) { + /* For some reason, this check is necessary for Travis + * to pass these tests. */ + ASSERT(waiting->revents & POLLIN); + server_fd = accept_client(socket_fd); + ASSERT(server_fd >= 0); + event_loop_attach(&loop, 1, NULL, server_fd, POLLIN); + } else { + char *cmd = read_string(waiting->fd); + redisAsyncCommand(conn.context, logging_test_callback, NULL, cmd, + conn.client_id, 0); + free(cmd); + } + } + } + free_ray_logger(logger); + db_disconnect(&conn); + event_loop_free(&loop); + close(server_fd); + close(client_fd); + close(socket_fd); + unlink(socket_pathname); + lookup_successful = 0; + PASS(); +} + SUITE(redis_tests) { redisContext *context = redisConnect("127.0.0.1", 6379); freeReplyObject(redisCommand(context, "FLUSHALL")); RUN_REDIS_TEST(context, redis_socket_test); RUN_REDIS_TEST(context, async_redis_socket_test); + RUN_REDIS_TEST(context, logging_test); redisFree(context); } From 79079926096df869efb9e4d4fe782d0d46fea1f1 Mon Sep 17 00:00:00 2001 From: Philipp Moritz Date: Fri, 23 Sep 2016 22:53:58 -0700 Subject: [PATCH 14/36] [WIP] Event loop refactoring (#19) * task queue tests and extensions * event loop refactor * fix formating --- Makefile | 6 +- common.h | 2 + event_loop.c | 120 ++++------ event_loop.h | 93 +++++--- state/db.h | 9 +- state/redis.c | 67 +----- state/redis.h | 2 +- test/db_tests.c | 99 ++------ test/io_tests.c | 1 + test/redis_tests.c | 202 +++++++++-------- thirdparty/ae/ae.c | 465 ++++++++++++++++++++++++++++++++++++++ thirdparty/ae/ae.h | 123 ++++++++++ thirdparty/ae/ae_epoll.c | 135 +++++++++++ thirdparty/ae/ae_evport.c | 320 ++++++++++++++++++++++++++ thirdparty/ae/ae_kqueue.c | 138 +++++++++++ thirdparty/ae/ae_select.c | 106 +++++++++ thirdparty/ae/config.h | 54 +++++ thirdparty/ae/zmalloc.h | 16 ++ 18 files changed, 1610 insertions(+), 348 deletions(-) create mode 100644 thirdparty/ae/ae.c create mode 100644 thirdparty/ae/ae.h create mode 100644 thirdparty/ae/ae_epoll.c create mode 100644 thirdparty/ae/ae_evport.c create mode 100644 thirdparty/ae/ae_kqueue.c create mode 100644 thirdparty/ae/ae_select.c create mode 100644 thirdparty/ae/config.h create mode 100644 thirdparty/ae/zmalloc.h diff --git a/Makefile b/Makefile index aa5ee292e..f71273119 100644 --- a/Makefile +++ b/Makefile @@ -1,10 +1,10 @@ CC = gcc -CFLAGS = -g -Wall --std=c99 -D_XOPEN_SOURCE=500 -D_POSIX_C_SOURCE=200809L -fPIC -I. -Ithirdparty +CFLAGS = -g -Wall --std=c99 -D_XOPEN_SOURCE=500 -D_POSIX_C_SOURCE=200809L -fPIC -I. -Ithirdparty -Ithirdparty/ae BUILD = build all: $(BUILD)/libcommon.a -$(BUILD)/libcommon.a: event_loop.o common.o task.o io.o state/redis.o +$(BUILD)/libcommon.a: event_loop.o common.o task.o io.o state/redis.o thirdparty/ae/ae.o ar rcs $@ $^ $(BUILD)/common_tests: test/common_tests.c $(BUILD)/libcommon.a @@ -23,7 +23,7 @@ $(BUILD)/redis_tests: hiredis test/redis_tests.c $(BUILD)/libcommon.a logging.h $(CC) -o $@ test/redis_tests.c logging.c $(BUILD)/libcommon.a thirdparty/hiredis/libhiredis.a $(CFLAGS) clean: - rm -f *.o state/*.o test/*.o + rm -f *.o state/*.o test/*.o thirdparty/ae/*.o rm -rf $(BUILD)/* redis: diff --git a/common.h b/common.h index 7e4b73ebe..5444739bc 100644 --- a/common.h +++ b/common.h @@ -1,6 +1,8 @@ #ifndef COMMON_H #define COMMON_H +#include +#include #include #include diff --git a/event_loop.c b/event_loop.c index d89710ba5..6928705e7 100644 --- a/event_loop.c +++ b/event_loop.c @@ -1,98 +1,62 @@ #include "event_loop.h" -#include +#include "common.h" +#include -UT_icd item_icd = {sizeof(event_loop_item), NULL, NULL, NULL}; -UT_icd poll_icd = {sizeof(struct pollfd), NULL, NULL, NULL}; +#define INITIAL_EVENT_LOOP_SIZE 1024 -/* Initializes the event loop. - * This function needs to be called before any other event loop function. */ -void event_loop_init(event_loop *loop) { - utarray_new(loop->items, &item_icd); - utarray_new(loop->waiting, &poll_icd); +event_loop *event_loop_create() { + return aeCreateEventLoop(INITIAL_EVENT_LOOP_SIZE); } -/* Free the space associated to the event loop. - * Does not free the event_loop datastructure itself. */ -void event_loop_free(event_loop *loop) { - utarray_free(loop->items); - utarray_free(loop->waiting); -} - -/* Add a new file descriptor fd to the event loop. - * This function sets a user defined type and id for the file descriptor - * which can be queried using event_loop_type and event_loop_id. The parameter - * events is the same as in http://linux.die.net/man/2/poll. - * Returns the index of the item in the event loop. */ -int64_t event_loop_attach(event_loop *loop, - int type, - void *data, - int fd, - int events) { - assert(utarray_len(loop->items) == utarray_len(loop->waiting)); - int64_t index = utarray_len(loop->items); - event_loop_item item = {.type = type, .data = data}; - utarray_push_back(loop->items, &item); - struct pollfd waiting = {.fd = fd, .events = events}; - utarray_push_back(loop->waiting, &waiting); - return index; -} - -/* Detach a file descriptor from the event loop. - * This invalidates all other indices into the event loop items, but leaves - * the ids of the event loop items valid. */ -void event_loop_detach(event_loop *loop, int64_t index, int shall_close) { - struct pollfd *waiting_item = - (struct pollfd *) utarray_eltptr(loop->waiting, index); - struct pollfd *waiting_back = (struct pollfd *) utarray_back(loop->waiting); - if (shall_close) { - close(waiting_item->fd); +void event_loop_destroy(event_loop *loop) { + /* Clean up timer events. This is to make valgrind happy. */ + aeTimeEvent *te = loop->timeEventHead; + while (te) { + aeTimeEvent *next = te->next; + free(te); + te = next; } - *waiting_item = *waiting_back; - utarray_pop_back(loop->waiting); - - event_loop_item *items_item = - (event_loop_item *) utarray_eltptr(loop->items, index); - event_loop_item *items_back = (event_loop_item *) utarray_back(loop->items); - *items_item = *items_back; - utarray_pop_back(loop->items); + aeDeleteEventLoop(loop); } -/* Poll the file descriptors associated to this event loop. - * See http://linux.die.net/man/2/poll. The timeout is in milliseconds. */ -int event_loop_poll(event_loop *loop, int timeout) { - return poll((struct pollfd *) utarray_front(loop->waiting), - utarray_len(loop->waiting), timeout); +void event_loop_add_file(event_loop *loop, + int fd, + int events, + event_loop_file_handler handler, + void *context) { + /* Try to add the file descriptor. */ + int err = aeCreateFileEvent(loop, fd, events, handler, context); + /* If it cannot be added, increase the size of the event loop. */ + if (err == AE_ERR && errno == ERANGE) { + err = aeResizeSetSize(loop, 3 * aeGetSetSize(loop) / 2); + CHECK(err == AE_OK); + err = aeCreateFileEvent(loop, fd, events, handler, context); + } + /* In any case, test if there were errors. */ + CHECK(err == AE_OK); } -/* Get the total number of file descriptors participating in the event loop. */ -int64_t event_loop_size(event_loop *loop) { - return utarray_len(loop->waiting); +void event_loop_remove_file(event_loop *loop, int fd) { + aeDeleteFileEvent(loop, fd, EVENT_LOOP_READ | EVENT_LOOP_WRITE); } -/* Get the pollfd structure associated to a file descriptor participating in the - * event loop. */ -struct pollfd *event_loop_get(event_loop *loop, int64_t index) { - return (struct pollfd *) utarray_eltptr(loop->waiting, index); +int64_t event_loop_add_timer(event_loop *loop, + int64_t milliseconds, + event_loop_timer_handler handler, + void *context) { + return aeCreateTimeEvent(loop, milliseconds, handler, context, NULL); } -/* Set the data connection information for participant in the event loop. */ -void event_loop_set_data(event_loop *loop, int64_t index, void *data) { - event_loop_item *item = - (event_loop_item *) utarray_eltptr(loop->items, index); - item->data = data; +void event_loop_remove_timer(event_loop *loop, int64_t id) { + int err = aeDeleteTimeEvent(loop, id); + CHECK(err == AE_OK); /* timer id found? */ } -/* Get the data connection information for participant in the event loop. */ -void *event_loop_get_data(event_loop *loop, int64_t index) { - event_loop_item *item = - (event_loop_item *) utarray_eltptr(loop->items, index); - return item->data; +void event_loop_run(event_loop *loop) { + aeMain(loop); } -/* Return the type of connection. */ -int event_loop_type(event_loop *loop, int64_t index) { - event_loop_item *item = - (event_loop_item *) utarray_eltptr(loop->items, index); - return item->type; +void event_loop_stop(event_loop *loop) { + aeStop(loop); } diff --git a/event_loop.h b/event_loop.h index 840abfd8c..bb6afdb93 100644 --- a/event_loop.h +++ b/event_loop.h @@ -1,39 +1,74 @@ #ifndef EVENT_LOOP_H #define EVENT_LOOP_H -#include #include +#include "ae/ae.h" -#include "utarray.h" +typedef aeEventLoop event_loop; -typedef struct { - /* The type of connection (e.g. redis, client, manager, data transfer). */ - int type; - /* Data associated with the connection (managed by the user) */ - void *data; -} event_loop_item; +/* File descriptor is readable. */ +#define EVENT_LOOP_READ AE_READABLE -typedef struct { - /* Array of event_loop_items that hold information for connections. */ - UT_array *items; - /* Array of file descriptors that are waiting, corresponding to items. */ - UT_array *waiting; -} event_loop; +/* File descriptor is writable. */ +#define EVENT_LOOP_WRITE AE_WRITABLE -/* Event loop functions. */ -void event_loop_init(event_loop *loop); -void event_loop_free(event_loop *loop); -int64_t event_loop_attach(event_loop *loop, - int type, - void *data, - int fd, - int events); -void event_loop_detach(event_loop *loop, int64_t index, int shall_close); -int event_loop_poll(event_loop *loop, int timeout); -int64_t event_loop_size(event_loop *loop); -struct pollfd *event_loop_get(event_loop *loop, int64_t index); -void event_loop_set_data(event_loop *loop, int64_t index, void *data); -void *event_loop_get_data(event_loop *loop, int64_t index); -int event_loop_type(event_loop *loop, int64_t index); +/* Signature of the handler that will be called when there is a new event + * on the file descriptor that this handler has been registered for. The + * context is the one that was passed into add_file by the user. The + * events parameter indicates which event is available on the file, + * it can be EVENT_LOOP_READ or EVENT_LOOP_WRITE. */ +typedef void (*event_loop_file_handler)(event_loop *loop, + int fd, + void *context, + int events); + +/* This handler will be called when a timer times out. The id of the timer + * as well as the context that was specified when registering this handler + * are passed as arguments. */ +typedef int64_t (*event_loop_timer_handler)(event_loop *loop, + int64_t id, + void *context); + +/* Create and return a new event loop. */ +event_loop *event_loop_create(); + +/* Deallocate space associated with the event loop that was created + * with the "create" function. */ +void event_loop_destroy(event_loop *loop); + +/* Register a handler that will be called any time a new event happens on + * a file descriptor. Can specify a context that will be passed as an + * argument to the handler. Currently there can only be one handler per file. + * The events parameter specifies which events we listen to: EVENT_LOOP_READ + * or EVENT_LOOP_WRITE. */ +void event_loop_add_file(event_loop *loop, + int fd, + int events, + event_loop_file_handler handler, + void *context); + +/* Remove a registered file event handler from the event loop. */ +void event_loop_remove_file(event_loop *loop, int fd); + +/* Register a handler that will be called after a time slice of + * "milliseconds" milliseconds. Can specify a context that will be passed + * as an argument to the handler. Return the id of the time event. */ +int64_t event_loop_add_timer(event_loop *loop, + int64_t milliseconds, + event_loop_timer_handler handler, + void *context); + +/* Reset the timer timeout to a given number of milliseconds. + * NOTE: This is not implemented yet. */ +void event_loop_reset_timer(event_loop *loop, int64_t id, int64_t milliseconds); + +/* Remove a registered time event handler from the event loop. */ +void event_loop_remove_timer(event_loop *loop, int64_t id); + +/* Run the event loop. */ +void event_loop_run(event_loop *loop); + +/* Stop the event loop. */ +void event_loop_stop(event_loop *loop); #endif diff --git a/state/db.h b/state/db.h index b586f9acf..3fcf658fd 100644 --- a/state/db.h +++ b/state/db.h @@ -15,13 +15,8 @@ void db_connect(const char *db_address, int client_port, db_conn *db); -/* Attach global system store onnection to event loop. Returns the index of the - * connection in the loop. */ -int64_t db_attach(db_conn *db, event_loop *loop, int connection_type); - -/* This function will be called by the user if there is a new event in the - * event loop associated with the global system store connection. */ -void db_event(db_conn *db); +/* Attach global system store connection to event loop. */ +void db_attach(db_conn *db, event_loop *loop); /* Disconnect from the global system store. */ void db_disconnect(db_conn *db); diff --git a/state/redis.c b/state/redis.c index ac56206e8..9c4b81f03 100644 --- a/state/redis.c +++ b/state/redis.c @@ -2,6 +2,8 @@ #include +#include +#include "hiredis/adapters/ae.h" #include "utstring.h" #include "common.h" @@ -12,38 +14,6 @@ #include "redis.h" #include "io.h" -static void poll_add_read(void *privdata) { - db_conn *conn = (db_conn *) privdata; - if (!conn->reading) { - conn->reading = 1; - event_loop_get(conn->loop, conn->db_index)->events |= POLLIN; - } -} - -static void poll_del_read(void *privdata) { - db_conn *conn = (db_conn *) privdata; - if (conn->reading) { - conn->reading = 0; - event_loop_get(conn->loop, conn->db_index)->events &= ~POLLIN; - } -} - -static void poll_add_write(void *privdata) { - db_conn *conn = (db_conn *) privdata; - if (!conn->writing) { - conn->writing = 1; - event_loop_get(conn->loop, conn->db_index)->events |= POLLOUT; - } -} - -static void poll_del_write(void *privdata) { - db_conn *conn = (db_conn *) privdata; - if (conn->writing) { - conn->writing = 0; - event_loop_get(conn->loop, conn->db_index)->events &= ~POLLOUT; - } -} - #define LOG_REDIS_ERR(context, M, ...) \ fprintf(stderr, "[ERROR] (%s:%d: message: %s) " M "\n", __FILE__, __LINE__, \ context->errstr, ##__VA_ARGS__) @@ -119,37 +89,8 @@ void db_disconnect(db_conn *db) { free(db->client_type); } -void db_event(db_conn *db) { - if (db->reading) { - redisAsyncHandleRead(db->context); - } - if (db->writing) { - redisAsyncHandleWrite(db->context); - } -} - -int64_t db_attach(db_conn *db, event_loop *loop, int connection_type) { - db->loop = loop; - - redisAsyncContext *ac = db->context; - redisContext *c = &(ac->c); - - if (ac->ev.data != NULL) { - return REDIS_ERR; - } - - ac->ev.addRead = poll_add_read; - ac->ev.delRead = poll_del_read; - ac->ev.addWrite = poll_add_write; - ac->ev.delWrite = poll_del_write; - // TODO(pcm): Implement cleanup function - - ac->ev.data = db; - - int64_t index = - event_loop_attach(loop, connection_type, NULL, c->fd, POLLIN | POLLOUT); - db->db_index = index; - return index; +void db_attach(db_conn *db, event_loop *loop) { + redisAeAttach(loop, db->context); } void object_table_add(db_conn *db, unique_id object_id) { diff --git a/state/redis.h b/state/redis.h index 09d66bea6..c579e7065 100644 --- a/state/redis.h +++ b/state/redis.h @@ -12,7 +12,7 @@ typedef struct { /* Unique ID for this service. */ int service_id; /* IP address and port of this service. */ - const char *addr; + char *addr; /* Handle for the uthash table. */ UT_hash_handle hh; } service_cache_entry; diff --git a/test/db_tests.c b/test/db_tests.c index 5bdbf25df..d9dfcb563 100644 --- a/test/db_tests.c +++ b/test/db_tests.c @@ -6,12 +6,12 @@ #include "test/example_task.h" #include "state/db.h" #include "state/object_table.h" +#include "state/task_queue.h" #include "state/redis.h" #include "task.h" SUITE(db_tests); -int lookup_successful = 0; const char *manager_addr = "127.0.0.1"; int manager_port1 = 12345; int manager_port2 = 12346; @@ -20,20 +20,11 @@ char received_port1[6] = {0}; char received_addr2[16] = {0}; char received_port2[6] = {0}; -/* This is for synchronizing to make sure both entries have been written. */ -void sync_test_callback(object_id object_id, - int manager_count, - const char *manager_vector[]) { - lookup_successful = 1; - free(manager_vector); -} - -/* This performs the actual test. */ +/* Test if entries have been written to the database. */ void test_callback(object_id object_id, int manager_count, const char *manager_vector[]) { CHECK(manager_count == 2); - lookup_successful = 1; if (!manager_vector[0] || sscanf(manager_vector[0], "%15[0-9.]:%5[0-9]", received_addr1, received_port1) != 2) { @@ -47,57 +38,29 @@ void test_callback(object_id object_id, free(manager_vector); } +int64_t timeout_handler(event_loop *loop, int64_t id, void *context) { + event_loop_stop(loop); + return -1; +} + TEST object_table_lookup_test(void) { - event_loop loop; - event_loop_init(&loop); + event_loop *loop = event_loop_create(); db_conn conn1; db_connect("127.0.0.1", 6379, "plasma_manager", manager_addr, manager_port1, &conn1); db_conn conn2; db_connect("127.0.0.1", 6379, "plasma_manager", manager_addr, manager_port2, &conn2); - int64_t index1 = db_attach(&conn1, &loop, 0); - int64_t index2 = db_attach(&conn2, &loop, 1); + db_attach(&conn1, loop); + db_attach(&conn2, loop); unique_id id = globally_unique_id(); object_table_add(&conn1, id); object_table_add(&conn2, id); - object_table_lookup(&conn1, id, sync_test_callback); - while (!lookup_successful) { - int num_ready = event_loop_poll(&loop, -1); - if (num_ready < 0) { - exit(-1); - } - for (int i = 0; i < event_loop_size(&loop); ++i) { - struct pollfd *waiting = event_loop_get(&loop, i); - if (waiting->revents == 0) - continue; - if (i == index1) { - db_event(&conn1); - } - if (i == index2) { - db_event(&conn2); - } - } - } - lookup_successful = 0; + event_loop_add_timer(loop, 100, timeout_handler, NULL); + event_loop_run(loop); object_table_lookup(&conn1, id, test_callback); - while (!lookup_successful) { - int num_ready = event_loop_poll(&loop, -1); - if (num_ready < 0) { - exit(-1); - } - for (int i = 0; i < event_loop_size(&loop); ++i) { - struct pollfd *waiting = event_loop_get(&loop, i); - if (waiting->revents == 0) - continue; - if (i == index1) { - db_event(&conn1); - } - if (i == index2) { - db_event(&conn2); - } - } - } + event_loop_add_timer(loop, 100, timeout_handler, NULL); + event_loop_run(loop); int port1 = atoi(received_port1); int port2 = atoi(received_port2); ASSERT_STR_EQ(&received_addr1[0], manager_addr); @@ -107,50 +70,32 @@ TEST object_table_lookup_test(void) { db_disconnect(&conn1); db_disconnect(&conn2); - event_loop_free(&loop); - - lookup_successful = 0; + event_loop_destroy(loop); PASS(); } TEST task_queue_test(void) { - event_loop loop; - event_loop_init(&loop); + event_loop *loop = event_loop_create(); db_conn conn; db_connect("127.0.0.1", 6379, "local_scheduler", "", -1, &conn); - int64_t index = db_attach(&conn, &loop, 0); + db_attach(&conn, loop); task_spec *task = example_task(); task_queue_submit_task(&conn, globally_unique_id(), task); - while (1) { - int num_ready = event_loop_poll(&loop, 100); - if (num_ready < 0) { - exit(-1); - } - if (num_ready == 0) { - break; - } - for (int i = 0; i < event_loop_size(&loop); ++i) { - struct pollfd *waiting = event_loop_get(&loop, i); - if (waiting->revents == 0) - continue; - if (i == index) { - db_event(&conn); - } - } - } + event_loop_add_timer(loop, 100, timeout_handler, NULL); + event_loop_run(loop); free_task_spec(task); db_disconnect(&conn); - event_loop_free(&loop); + event_loop_destroy(loop); PASS(); } SUITE(db_tests) { redisContext *context = redisConnect("127.0.0.1", 6379); - redisCommand(context, "FLUSHALL"); + freeReplyObject(redisCommand(context, "FLUSHALL")); RUN_REDIS_TEST(context, object_table_lookup_test); - RUN_TEST(task_queue_test); + RUN_REDIS_TEST(context, task_queue_test); redisFree(context); } diff --git a/test/io_tests.c b/test/io_tests.c index 9216aa56e..1e8d20832 100644 --- a/test/io_tests.c +++ b/test/io_tests.c @@ -35,6 +35,7 @@ TEST ipc_socket_test(void) { uint8_t *bytes; read_bytes(client_fd, &bytes, &len); ASSERT(memcmp(test_bytes, bytes, len) == 0); + free(bytes); close(client_fd); close(socket_fd); unlink(socket_pathname); diff --git a/test/redis_tests.c b/test/redis_tests.c index d8df538e8..d527d0476 100644 --- a/test/redis_tests.c +++ b/test/redis_tests.c @@ -3,6 +3,8 @@ #include #include +#include "utarray.h" + #include "event_loop.h" #include "state/db.h" #include "state/redis.h" @@ -11,35 +13,27 @@ SUITE(redis_tests); -int lookup_successful = 0; const char *test_set_format = "SET %s %s"; const char *test_get_format = "GET %s"; const char *test_key = "foo"; const char *test_value = "bar"; +UT_array *connections = NULL; + +int async_redis_socket_test_callback_called = 0; void async_redis_socket_test_callback(redisAsyncContext *ac, void *r, void *privdata) { + async_redis_socket_test_callback_called = 1; redisContext *context = redisConnect("127.0.0.1", 6379); redisReply *reply = redisCommand(context, test_get_format, test_key); redisFree(context); - assert(reply != NULL); + CHECK(reply != NULL); if (strcmp(reply->str, test_value)) { freeReplyObject(reply); - assert(0); + CHECK(0); } freeReplyObject(reply); - lookup_successful = 1; -} - -void logging_test_callback(redisAsyncContext *ac, void *r, void *privdata) { - redisContext *context = redisConnect("127.0.0.1", 6379); - redisReply *reply = redisCommand(context, "KEYS %s", "log:*"); - redisFree(context); - assert(reply != NULL); - assert(reply->elements > 0); - freeReplyObject(reply); - lookup_successful = 1; } TEST redis_socket_test(void) { @@ -73,117 +67,145 @@ TEST redis_socket_test(void) { PASS(); } +void redis_read_callback(event_loop *loop, int fd, void *context, int events) { + db_conn *conn = context; + char *cmd = read_string(fd); + redisAsyncCommand(conn->context, async_redis_socket_test_callback, NULL, cmd, + conn->client_id, 0); + free(cmd); +} + +void redis_accept_callback(event_loop *loop, + int socket_fd, + void *context, + int events) { + int accept_fd = accept_client(socket_fd); + CHECK(accept_fd >= 0); + utarray_push_back(connections, &accept_fd); + event_loop_add_file(loop, accept_fd, EVENT_LOOP_READ, redis_read_callback, + context); +} + +int64_t timeout_handler(event_loop *loop, int64_t id, void *context) { + event_loop_stop(loop); + return -1; +} + TEST async_redis_socket_test(void) { - int socket_fd, server_fd, client_fd; - event_loop loop; - event_loop_init(&loop); + utarray_new(connections, &ut_int_icd); + event_loop *loop = event_loop_create(); + /* Start IPC channel. */ const char *socket_pathname = "async-redis-test-socket"; - socket_fd = bind_ipc_sock(socket_pathname); + int socket_fd = bind_ipc_sock(socket_pathname); ASSERT(socket_fd >= 0); - int64_t ipc_index = event_loop_attach(&loop, 1, NULL, socket_fd, POLLIN); + utarray_push_back(connections, &socket_fd); /* Start connection to Redis. */ db_conn conn; db_connect("127.0.0.1", 6379, "", "", 0, &conn); - int64_t db_index = db_attach(&conn, &loop, 0); + db_attach(&conn, loop); /* Send a command to the Redis process. */ - client_fd = connect_ipc_sock(socket_pathname); + int client_fd = connect_ipc_sock(socket_pathname); ASSERT(client_fd >= 0); + utarray_push_back(connections, &client_fd); write_formatted_string(client_fd, test_set_format, test_key, test_value); - while (!lookup_successful) { - int num_ready = event_loop_poll(&loop, -1); - if (num_ready < 0) { - exit(-1); - } - for (int i = 0; i < event_loop_size(&loop); ++i) { - struct pollfd *waiting = event_loop_get(&loop, i); - if (waiting->revents == 0) - continue; - if (i == db_index) { - db_event(&conn); - } else if (i == ipc_index) { - /* For some reason, this check is necessary for Travis - * to pass these tests. */ - ASSERT(waiting->revents & POLLIN); - server_fd = accept_client(socket_fd); - ASSERT(server_fd >= 0); - event_loop_attach(&loop, 1, NULL, server_fd, POLLIN); - } else { - char *cmd = read_string(waiting->fd); - redisAsyncCommand(conn.context, async_redis_socket_test_callback, NULL, - cmd, conn.client_id, 0); - free(cmd); - } - } - } + event_loop_add_file(loop, client_fd, EVENT_LOOP_READ, redis_read_callback, + &conn); + event_loop_add_file(loop, socket_fd, EVENT_LOOP_READ, redis_accept_callback, + &conn); + event_loop_add_timer(loop, 100, timeout_handler, NULL); + event_loop_run(loop); + + CHECK(async_redis_socket_test_callback_called); + db_disconnect(&conn); - event_loop_free(&loop); - close(server_fd); - close(client_fd); - close(socket_fd); + event_loop_destroy(loop); + for (int *p = (int *) utarray_front(connections); p != NULL; + p = (int *) utarray_next(connections, p)) { + close(*p); + } unlink(socket_pathname); - lookup_successful = 0; + utarray_free(connections); PASS(); } +int logging_test_callback_called = 0; + +void logging_test_callback(redisAsyncContext *ac, void *r, void *privdata) { + logging_test_callback_called = 1; + redisContext *context = redisConnect("127.0.0.1", 6379); + redisReply *reply = redisCommand(context, "KEYS %s", "log:*"); + redisFree(context); + CHECK(reply != NULL); + CHECK(reply->elements > 0); + freeReplyObject(reply); +} + +void logging_read_callback(event_loop *loop, + int fd, + void *context, + int events) { + db_conn *conn = context; + char *cmd = read_string(fd); + redisAsyncCommand(conn->context, logging_test_callback, NULL, cmd, + conn->client_id, 0); + free(cmd); +} + +void logging_accept_callback(event_loop *loop, + int socket_fd, + void *context, + int events) { + int accept_fd = accept_client(socket_fd); + CHECK(accept_fd >= 0); + utarray_push_back(connections, &accept_fd); + event_loop_add_file(loop, accept_fd, EVENT_LOOP_READ, logging_read_callback, + context); +} + TEST logging_test(void) { - int socket_fd, server_fd, client_fd; - event_loop loop; - event_loop_init(&loop); + utarray_new(connections, &ut_int_icd); + event_loop *loop = event_loop_create(); + /* Start IPC channel. */ const char *socket_pathname = "logging-test-socket"; - socket_fd = bind_ipc_sock(socket_pathname); + int socket_fd = bind_ipc_sock(socket_pathname); ASSERT(socket_fd >= 0); - int64_t ipc_index = event_loop_attach(&loop, 1, NULL, socket_fd, POLLIN); + utarray_push_back(connections, &socket_fd); /* Start connection to Redis. */ db_conn conn; db_connect("127.0.0.1", 6379, "", "", 0, &conn); - int64_t db_index = db_attach(&conn, &loop, 0); + db_attach(&conn, loop); /* Send a command to the Redis process. */ - client_fd = connect_ipc_sock(socket_pathname); + int client_fd = connect_ipc_sock(socket_pathname); ASSERT(client_fd >= 0); + utarray_push_back(connections, &client_fd); ray_logger *logger = init_ray_logger("worker", RAY_INFO, 0, &client_fd); ray_log(logger, RAY_INFO, "TEST", "Message"); - while (!lookup_successful) { - int num_ready = event_loop_poll(&loop, -1); - if (num_ready < 0) { - exit(-1); - } - for (int i = 0; i < event_loop_size(&loop); ++i) { - struct pollfd *waiting = event_loop_get(&loop, i); - if (waiting->revents == 0) - continue; - if (i == db_index) { - db_event(&conn); - } else if (i == ipc_index) { - /* For some reason, this check is necessary for Travis - * to pass these tests. */ - ASSERT(waiting->revents & POLLIN); - server_fd = accept_client(socket_fd); - ASSERT(server_fd >= 0); - event_loop_attach(&loop, 1, NULL, server_fd, POLLIN); - } else { - char *cmd = read_string(waiting->fd); - redisAsyncCommand(conn.context, logging_test_callback, NULL, cmd, - conn.client_id, 0); - free(cmd); - } - } - } + event_loop_add_file(loop, socket_fd, EVENT_LOOP_READ, logging_accept_callback, + &conn); + event_loop_add_file(loop, client_fd, EVENT_LOOP_READ, logging_read_callback, + &conn); + event_loop_add_timer(loop, 100, timeout_handler, NULL); + event_loop_run(loop); + + CHECK(logging_test_callback_called); + free_ray_logger(logger); db_disconnect(&conn); - event_loop_free(&loop); - close(server_fd); - close(client_fd); - close(socket_fd); + event_loop_destroy(loop); + for (int *p = (int *) utarray_front(connections); p != NULL; + p = (int *) utarray_next(connections, p)) { + close(*p); + } unlink(socket_pathname); - lookup_successful = 0; + utarray_free(connections); PASS(); } diff --git a/thirdparty/ae/ae.c b/thirdparty/ae/ae.c new file mode 100644 index 000000000..e66808a81 --- /dev/null +++ b/thirdparty/ae/ae.c @@ -0,0 +1,465 @@ +/* A simple event-driven programming library. Originally I wrote this code + * for the Jim's event-loop (Jim is a Tcl interpreter) but later translated + * it in form of a library for easy reuse. + * + * Copyright (c) 2006-2010, Salvatore Sanfilippo + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are met: + * + * * Redistributions of source code must retain the above copyright notice, + * this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * * Neither the name of Redis nor the names of its contributors may be used + * to endorse or promote products derived from this software without + * specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" + * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE + * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR + * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS + * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN + * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE + * POSSIBILITY OF SUCH DAMAGE. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "ae.h" +#include "zmalloc.h" +#include "config.h" + +/* Include the best multiplexing layer supported by this system. + * The following should be ordered by performances, descending. */ +#ifdef HAVE_EVPORT +#include "ae_evport.c" +#else + #ifdef HAVE_EPOLL + #include "ae_epoll.c" + #else + #ifdef HAVE_KQUEUE + #include "ae_kqueue.c" + #else + #include "ae_select.c" + #endif + #endif +#endif + +aeEventLoop *aeCreateEventLoop(int setsize) { + aeEventLoop *eventLoop; + int i; + + if ((eventLoop = zmalloc(sizeof(*eventLoop))) == NULL) goto err; + eventLoop->events = zmalloc(sizeof(aeFileEvent)*setsize); + eventLoop->fired = zmalloc(sizeof(aeFiredEvent)*setsize); + if (eventLoop->events == NULL || eventLoop->fired == NULL) goto err; + eventLoop->setsize = setsize; + eventLoop->lastTime = time(NULL); + eventLoop->timeEventHead = NULL; + eventLoop->timeEventNextId = 0; + eventLoop->stop = 0; + eventLoop->maxfd = -1; + eventLoop->beforesleep = NULL; + if (aeApiCreate(eventLoop) == -1) goto err; + /* Events with mask == AE_NONE are not set. So let's initialize the + * vector with it. */ + for (i = 0; i < setsize; i++) + eventLoop->events[i].mask = AE_NONE; + return eventLoop; + +err: + if (eventLoop) { + zfree(eventLoop->events); + zfree(eventLoop->fired); + zfree(eventLoop); + } + return NULL; +} + +/* Return the current set size. */ +int aeGetSetSize(aeEventLoop *eventLoop) { + return eventLoop->setsize; +} + +/* Resize the maximum set size of the event loop. + * If the requested set size is smaller than the current set size, but + * there is already a file descriptor in use that is >= the requested + * set size minus one, AE_ERR is returned and the operation is not + * performed at all. + * + * Otherwise AE_OK is returned and the operation is successful. */ +int aeResizeSetSize(aeEventLoop *eventLoop, int setsize) { + int i; + + if (setsize == eventLoop->setsize) return AE_OK; + if (eventLoop->maxfd >= setsize) return AE_ERR; + if (aeApiResize(eventLoop,setsize) == -1) return AE_ERR; + + eventLoop->events = zrealloc(eventLoop->events,sizeof(aeFileEvent)*setsize); + eventLoop->fired = zrealloc(eventLoop->fired,sizeof(aeFiredEvent)*setsize); + eventLoop->setsize = setsize; + + /* Make sure that if we created new slots, they are initialized with + * an AE_NONE mask. */ + for (i = eventLoop->maxfd+1; i < setsize; i++) + eventLoop->events[i].mask = AE_NONE; + return AE_OK; +} + +void aeDeleteEventLoop(aeEventLoop *eventLoop) { + aeApiFree(eventLoop); + zfree(eventLoop->events); + zfree(eventLoop->fired); + zfree(eventLoop); +} + +void aeStop(aeEventLoop *eventLoop) { + eventLoop->stop = 1; +} + +int aeCreateFileEvent(aeEventLoop *eventLoop, int fd, int mask, + aeFileProc *proc, void *clientData) +{ + if (fd >= eventLoop->setsize) { + errno = ERANGE; + return AE_ERR; + } + aeFileEvent *fe = &eventLoop->events[fd]; + + if (aeApiAddEvent(eventLoop, fd, mask) == -1) + return AE_ERR; + fe->mask |= mask; + if (mask & AE_READABLE) fe->rfileProc = proc; + if (mask & AE_WRITABLE) fe->wfileProc = proc; + fe->clientData = clientData; + if (fd > eventLoop->maxfd) + eventLoop->maxfd = fd; + return AE_OK; +} + +void aeDeleteFileEvent(aeEventLoop *eventLoop, int fd, int mask) +{ + if (fd >= eventLoop->setsize) return; + aeFileEvent *fe = &eventLoop->events[fd]; + if (fe->mask == AE_NONE) return; + + aeApiDelEvent(eventLoop, fd, mask); + fe->mask = fe->mask & (~mask); + if (fd == eventLoop->maxfd && fe->mask == AE_NONE) { + /* Update the max fd */ + int j; + + for (j = eventLoop->maxfd-1; j >= 0; j--) + if (eventLoop->events[j].mask != AE_NONE) break; + eventLoop->maxfd = j; + } +} + +int aeGetFileEvents(aeEventLoop *eventLoop, int fd) { + if (fd >= eventLoop->setsize) return 0; + aeFileEvent *fe = &eventLoop->events[fd]; + + return fe->mask; +} + +static void aeGetTime(long *seconds, long *milliseconds) +{ + struct timeval tv; + + gettimeofday(&tv, NULL); + *seconds = tv.tv_sec; + *milliseconds = tv.tv_usec/1000; +} + +static void aeAddMillisecondsToNow(long long milliseconds, long *sec, long *ms) { + long cur_sec, cur_ms, when_sec, when_ms; + + aeGetTime(&cur_sec, &cur_ms); + when_sec = cur_sec + milliseconds/1000; + when_ms = cur_ms + milliseconds%1000; + if (when_ms >= 1000) { + when_sec ++; + when_ms -= 1000; + } + *sec = when_sec; + *ms = when_ms; +} + +long long aeCreateTimeEvent(aeEventLoop *eventLoop, long long milliseconds, + aeTimeProc *proc, void *clientData, + aeEventFinalizerProc *finalizerProc) +{ + long long id = eventLoop->timeEventNextId++; + aeTimeEvent *te; + + te = zmalloc(sizeof(*te)); + if (te == NULL) return AE_ERR; + te->id = id; + aeAddMillisecondsToNow(milliseconds,&te->when_sec,&te->when_ms); + te->timeProc = proc; + te->finalizerProc = finalizerProc; + te->clientData = clientData; + te->next = eventLoop->timeEventHead; + eventLoop->timeEventHead = te; + return id; +} + +int aeDeleteTimeEvent(aeEventLoop *eventLoop, long long id) +{ + aeTimeEvent *te = eventLoop->timeEventHead; + while(te) { + if (te->id == id) { + te->id = AE_DELETED_EVENT_ID; + return AE_OK; + } + te = te->next; + } + return AE_ERR; /* NO event with the specified ID found */ +} + +/* Search the first timer to fire. + * This operation is useful to know how many time the select can be + * put in sleep without to delay any event. + * If there are no timers NULL is returned. + * + * Note that's O(N) since time events are unsorted. + * Possible optimizations (not needed by Redis so far, but...): + * 1) Insert the event in order, so that the nearest is just the head. + * Much better but still insertion or deletion of timers is O(N). + * 2) Use a skiplist to have this operation as O(1) and insertion as O(log(N)). + */ +static aeTimeEvent *aeSearchNearestTimer(aeEventLoop *eventLoop) +{ + aeTimeEvent *te = eventLoop->timeEventHead; + aeTimeEvent *nearest = NULL; + + while(te) { + if (!nearest || te->when_sec < nearest->when_sec || + (te->when_sec == nearest->when_sec && + te->when_ms < nearest->when_ms)) + nearest = te; + te = te->next; + } + return nearest; +} + +/* Process time events */ +static int processTimeEvents(aeEventLoop *eventLoop) { + int processed = 0; + aeTimeEvent *te, *prev; + long long maxId; + time_t now = time(NULL); + + /* If the system clock is moved to the future, and then set back to the + * right value, time events may be delayed in a random way. Often this + * means that scheduled operations will not be performed soon enough. + * + * Here we try to detect system clock skews, and force all the time + * events to be processed ASAP when this happens: the idea is that + * processing events earlier is less dangerous than delaying them + * indefinitely, and practice suggests it is. */ + if (now < eventLoop->lastTime) { + te = eventLoop->timeEventHead; + while(te) { + te->when_sec = 0; + te = te->next; + } + } + eventLoop->lastTime = now; + + prev = NULL; + te = eventLoop->timeEventHead; + maxId = eventLoop->timeEventNextId-1; + while(te) { + long now_sec, now_ms; + long long id; + + /* Remove events scheduled for deletion. */ + if (te->id == AE_DELETED_EVENT_ID) { + aeTimeEvent *next = te->next; + if (prev == NULL) + eventLoop->timeEventHead = te->next; + else + prev->next = te->next; + if (te->finalizerProc) + te->finalizerProc(eventLoop, te->clientData); + zfree(te); + te = next; + continue; + } + + /* Make sure we don't process time events created by time events in + * this iteration. Note that this check is currently useless: we always + * add new timers on the head, however if we change the implementation + * detail, this check may be useful again: we keep it here for future + * defense. */ + if (te->id > maxId) { + te = te->next; + continue; + } + aeGetTime(&now_sec, &now_ms); + if (now_sec > te->when_sec || + (now_sec == te->when_sec && now_ms >= te->when_ms)) + { + int retval; + + id = te->id; + retval = te->timeProc(eventLoop, id, te->clientData); + processed++; + if (retval != AE_NOMORE) { + aeAddMillisecondsToNow(retval,&te->when_sec,&te->when_ms); + } else { + te->id = AE_DELETED_EVENT_ID; + } + } + prev = te; + te = te->next; + } + return processed; +} + +/* Process every pending time event, then every pending file event + * (that may be registered by time event callbacks just processed). + * Without special flags the function sleeps until some file event + * fires, or when the next time event occurs (if any). + * + * If flags is 0, the function does nothing and returns. + * if flags has AE_ALL_EVENTS set, all the kind of events are processed. + * if flags has AE_FILE_EVENTS set, file events are processed. + * if flags has AE_TIME_EVENTS set, time events are processed. + * if flags has AE_DONT_WAIT set the function returns ASAP until all + * the events that's possible to process without to wait are processed. + * + * The function returns the number of events processed. */ +int aeProcessEvents(aeEventLoop *eventLoop, int flags) +{ + int processed = 0, numevents; + + /* Nothing to do? return ASAP */ + if (!(flags & AE_TIME_EVENTS) && !(flags & AE_FILE_EVENTS)) return 0; + + /* Note that we want call select() even if there are no + * file events to process as long as we want to process time + * events, in order to sleep until the next time event is ready + * to fire. */ + if (eventLoop->maxfd != -1 || + ((flags & AE_TIME_EVENTS) && !(flags & AE_DONT_WAIT))) { + int j; + aeTimeEvent *shortest = NULL; + struct timeval tv, *tvp; + + if (flags & AE_TIME_EVENTS && !(flags & AE_DONT_WAIT)) + shortest = aeSearchNearestTimer(eventLoop); + if (shortest) { + long now_sec, now_ms; + + aeGetTime(&now_sec, &now_ms); + tvp = &tv; + + /* How many milliseconds we need to wait for the next + * time event to fire? */ + long long ms = + (shortest->when_sec - now_sec)*1000 + + shortest->when_ms - now_ms; + + if (ms > 0) { + tvp->tv_sec = ms/1000; + tvp->tv_usec = (ms % 1000)*1000; + } else { + tvp->tv_sec = 0; + tvp->tv_usec = 0; + } + } else { + /* If we have to check for events but need to return + * ASAP because of AE_DONT_WAIT we need to set the timeout + * to zero */ + if (flags & AE_DONT_WAIT) { + tv.tv_sec = tv.tv_usec = 0; + tvp = &tv; + } else { + /* Otherwise we can block */ + tvp = NULL; /* wait forever */ + } + } + + numevents = aeApiPoll(eventLoop, tvp); + for (j = 0; j < numevents; j++) { + aeFileEvent *fe = &eventLoop->events[eventLoop->fired[j].fd]; + int mask = eventLoop->fired[j].mask; + int fd = eventLoop->fired[j].fd; + int rfired = 0; + + /* note the fe->mask & mask & ... code: maybe an already processed + * event removed an element that fired and we still didn't + * processed, so we check if the event is still valid. */ + if (fe->mask & mask & AE_READABLE) { + rfired = 1; + fe->rfileProc(eventLoop,fd,fe->clientData,mask); + } + if (fe->mask & mask & AE_WRITABLE) { + if (!rfired || fe->wfileProc != fe->rfileProc) + fe->wfileProc(eventLoop,fd,fe->clientData,mask); + } + processed++; + } + } + /* Check time events */ + if (flags & AE_TIME_EVENTS) + processed += processTimeEvents(eventLoop); + + return processed; /* return the number of processed file/time events */ +} + +/* Wait for milliseconds until the given file descriptor becomes + * writable/readable/exception */ +int aeWait(int fd, int mask, long long milliseconds) { + struct pollfd pfd; + int retmask = 0, retval; + + memset(&pfd, 0, sizeof(pfd)); + pfd.fd = fd; + if (mask & AE_READABLE) pfd.events |= POLLIN; + if (mask & AE_WRITABLE) pfd.events |= POLLOUT; + + if ((retval = poll(&pfd, 1, milliseconds))== 1) { + if (pfd.revents & POLLIN) retmask |= AE_READABLE; + if (pfd.revents & POLLOUT) retmask |= AE_WRITABLE; + if (pfd.revents & POLLERR) retmask |= AE_WRITABLE; + if (pfd.revents & POLLHUP) retmask |= AE_WRITABLE; + return retmask; + } else { + return retval; + } +} + +void aeMain(aeEventLoop *eventLoop) { + eventLoop->stop = 0; + while (!eventLoop->stop) { + if (eventLoop->beforesleep != NULL) + eventLoop->beforesleep(eventLoop); + aeProcessEvents(eventLoop, AE_ALL_EVENTS); + } +} + +char *aeGetApiName(void) { + return aeApiName(); +} + +void aeSetBeforeSleepProc(aeEventLoop *eventLoop, aeBeforeSleepProc *beforesleep) { + eventLoop->beforesleep = beforesleep; +} diff --git a/thirdparty/ae/ae.h b/thirdparty/ae/ae.h new file mode 100644 index 000000000..827c4c9e4 --- /dev/null +++ b/thirdparty/ae/ae.h @@ -0,0 +1,123 @@ +/* A simple event-driven programming library. Originally I wrote this code + * for the Jim's event-loop (Jim is a Tcl interpreter) but later translated + * it in form of a library for easy reuse. + * + * Copyright (c) 2006-2012, Salvatore Sanfilippo + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are met: + * + * * Redistributions of source code must retain the above copyright notice, + * this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * * Neither the name of Redis nor the names of its contributors may be used + * to endorse or promote products derived from this software without + * specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" + * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE + * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR + * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS + * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN + * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE + * POSSIBILITY OF SUCH DAMAGE. + */ + +#ifndef __AE_H__ +#define __AE_H__ + +#include + +#define AE_OK 0 +#define AE_ERR -1 + +#define AE_NONE 0 +#define AE_READABLE 1 +#define AE_WRITABLE 2 + +#define AE_FILE_EVENTS 1 +#define AE_TIME_EVENTS 2 +#define AE_ALL_EVENTS (AE_FILE_EVENTS|AE_TIME_EVENTS) +#define AE_DONT_WAIT 4 + +#define AE_NOMORE -1 +#define AE_DELETED_EVENT_ID -1 + +/* Macros */ +#define AE_NOTUSED(V) ((void) V) + +struct aeEventLoop; + +/* Types and data structures */ +typedef void aeFileProc(struct aeEventLoop *eventLoop, int fd, void *clientData, int mask); +typedef int aeTimeProc(struct aeEventLoop *eventLoop, long long id, void *clientData); +typedef void aeEventFinalizerProc(struct aeEventLoop *eventLoop, void *clientData); +typedef void aeBeforeSleepProc(struct aeEventLoop *eventLoop); + +/* File event structure */ +typedef struct aeFileEvent { + int mask; /* one of AE_(READABLE|WRITABLE) */ + aeFileProc *rfileProc; + aeFileProc *wfileProc; + void *clientData; +} aeFileEvent; + +/* Time event structure */ +typedef struct aeTimeEvent { + long long id; /* time event identifier. */ + long when_sec; /* seconds */ + long when_ms; /* milliseconds */ + aeTimeProc *timeProc; + aeEventFinalizerProc *finalizerProc; + void *clientData; + struct aeTimeEvent *next; +} aeTimeEvent; + +/* A fired event */ +typedef struct aeFiredEvent { + int fd; + int mask; +} aeFiredEvent; + +/* State of an event based program */ +typedef struct aeEventLoop { + int maxfd; /* highest file descriptor currently registered */ + int setsize; /* max number of file descriptors tracked */ + long long timeEventNextId; + time_t lastTime; /* Used to detect system clock skew */ + aeFileEvent *events; /* Registered events */ + aeFiredEvent *fired; /* Fired events */ + aeTimeEvent *timeEventHead; + int stop; + void *apidata; /* This is used for polling API specific data */ + aeBeforeSleepProc *beforesleep; +} aeEventLoop; + +/* Prototypes */ +aeEventLoop *aeCreateEventLoop(int setsize); +void aeDeleteEventLoop(aeEventLoop *eventLoop); +void aeStop(aeEventLoop *eventLoop); +int aeCreateFileEvent(aeEventLoop *eventLoop, int fd, int mask, + aeFileProc *proc, void *clientData); +void aeDeleteFileEvent(aeEventLoop *eventLoop, int fd, int mask); +int aeGetFileEvents(aeEventLoop *eventLoop, int fd); +long long aeCreateTimeEvent(aeEventLoop *eventLoop, long long milliseconds, + aeTimeProc *proc, void *clientData, + aeEventFinalizerProc *finalizerProc); +int aeDeleteTimeEvent(aeEventLoop *eventLoop, long long id); +int aeProcessEvents(aeEventLoop *eventLoop, int flags); +int aeWait(int fd, int mask, long long milliseconds); +void aeMain(aeEventLoop *eventLoop); +char *aeGetApiName(void); +void aeSetBeforeSleepProc(aeEventLoop *eventLoop, aeBeforeSleepProc *beforesleep); +int aeGetSetSize(aeEventLoop *eventLoop); +int aeResizeSetSize(aeEventLoop *eventLoop, int setsize); + +#endif diff --git a/thirdparty/ae/ae_epoll.c b/thirdparty/ae/ae_epoll.c new file mode 100644 index 000000000..410aac70d --- /dev/null +++ b/thirdparty/ae/ae_epoll.c @@ -0,0 +1,135 @@ +/* Linux epoll(2) based ae.c module + * + * Copyright (c) 2009-2012, Salvatore Sanfilippo + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are met: + * + * * Redistributions of source code must retain the above copyright notice, + * this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * * Neither the name of Redis nor the names of its contributors may be used + * to endorse or promote products derived from this software without + * specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" + * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE + * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR + * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS + * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN + * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE + * POSSIBILITY OF SUCH DAMAGE. + */ + + +#include + +typedef struct aeApiState { + int epfd; + struct epoll_event *events; +} aeApiState; + +static int aeApiCreate(aeEventLoop *eventLoop) { + aeApiState *state = zmalloc(sizeof(aeApiState)); + + if (!state) return -1; + state->events = zmalloc(sizeof(struct epoll_event)*eventLoop->setsize); + if (!state->events) { + zfree(state); + return -1; + } + state->epfd = epoll_create(1024); /* 1024 is just a hint for the kernel */ + if (state->epfd == -1) { + zfree(state->events); + zfree(state); + return -1; + } + eventLoop->apidata = state; + return 0; +} + +static int aeApiResize(aeEventLoop *eventLoop, int setsize) { + aeApiState *state = eventLoop->apidata; + + state->events = zrealloc(state->events, sizeof(struct epoll_event)*setsize); + return 0; +} + +static void aeApiFree(aeEventLoop *eventLoop) { + aeApiState *state = eventLoop->apidata; + + close(state->epfd); + zfree(state->events); + zfree(state); +} + +static int aeApiAddEvent(aeEventLoop *eventLoop, int fd, int mask) { + aeApiState *state = eventLoop->apidata; + struct epoll_event ee = {0}; /* avoid valgrind warning */ + /* If the fd was already monitored for some event, we need a MOD + * operation. Otherwise we need an ADD operation. */ + int op = eventLoop->events[fd].mask == AE_NONE ? + EPOLL_CTL_ADD : EPOLL_CTL_MOD; + + ee.events = 0; + mask |= eventLoop->events[fd].mask; /* Merge old events */ + if (mask & AE_READABLE) ee.events |= EPOLLIN; + if (mask & AE_WRITABLE) ee.events |= EPOLLOUT; + ee.data.fd = fd; + if (epoll_ctl(state->epfd,op,fd,&ee) == -1) return -1; + return 0; +} + +static void aeApiDelEvent(aeEventLoop *eventLoop, int fd, int delmask) { + aeApiState *state = eventLoop->apidata; + struct epoll_event ee = {0}; /* avoid valgrind warning */ + int mask = eventLoop->events[fd].mask & (~delmask); + + ee.events = 0; + if (mask & AE_READABLE) ee.events |= EPOLLIN; + if (mask & AE_WRITABLE) ee.events |= EPOLLOUT; + ee.data.fd = fd; + if (mask != AE_NONE) { + epoll_ctl(state->epfd,EPOLL_CTL_MOD,fd,&ee); + } else { + /* Note, Kernel < 2.6.9 requires a non null event pointer even for + * EPOLL_CTL_DEL. */ + epoll_ctl(state->epfd,EPOLL_CTL_DEL,fd,&ee); + } +} + +static int aeApiPoll(aeEventLoop *eventLoop, struct timeval *tvp) { + aeApiState *state = eventLoop->apidata; + int retval, numevents = 0; + + retval = epoll_wait(state->epfd,state->events,eventLoop->setsize, + tvp ? (tvp->tv_sec*1000 + tvp->tv_usec/1000) : -1); + if (retval > 0) { + int j; + + numevents = retval; + for (j = 0; j < numevents; j++) { + int mask = 0; + struct epoll_event *e = state->events+j; + + if (e->events & EPOLLIN) mask |= AE_READABLE; + if (e->events & EPOLLOUT) mask |= AE_WRITABLE; + if (e->events & EPOLLERR) mask |= AE_WRITABLE; + if (e->events & EPOLLHUP) mask |= AE_WRITABLE; + eventLoop->fired[j].fd = e->data.fd; + eventLoop->fired[j].mask = mask; + } + } + return numevents; +} + +static char *aeApiName(void) { + return "epoll"; +} diff --git a/thirdparty/ae/ae_evport.c b/thirdparty/ae/ae_evport.c new file mode 100644 index 000000000..5c317becb --- /dev/null +++ b/thirdparty/ae/ae_evport.c @@ -0,0 +1,320 @@ +/* ae.c module for illumos event ports. + * + * Copyright (c) 2012, Joyent, Inc. All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are met: + * + * * Redistributions of source code must retain the above copyright notice, + * this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * * Neither the name of Redis nor the names of its contributors may be used + * to endorse or promote products derived from this software without + * specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" + * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE + * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR + * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS + * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN + * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE + * POSSIBILITY OF SUCH DAMAGE. + */ + + +#include +#include +#include +#include + +#include +#include + +#include + +static int evport_debug = 0; + +/* + * This file implements the ae API using event ports, present on Solaris-based + * systems since Solaris 10. Using the event port interface, we associate file + * descriptors with the port. Each association also includes the set of poll(2) + * events that the consumer is interested in (e.g., POLLIN and POLLOUT). + * + * There's one tricky piece to this implementation: when we return events via + * aeApiPoll, the corresponding file descriptors become dissociated from the + * port. This is necessary because poll events are level-triggered, so if the + * fd didn't become dissociated, it would immediately fire another event since + * the underlying state hasn't changed yet. We must re-associate the file + * descriptor, but only after we know that our caller has actually read from it. + * The ae API does not tell us exactly when that happens, but we do know that + * it must happen by the time aeApiPoll is called again. Our solution is to + * keep track of the last fds returned by aeApiPoll and re-associate them next + * time aeApiPoll is invoked. + * + * To summarize, in this module, each fd association is EITHER (a) represented + * only via the in-kernel association OR (b) represented by pending_fds and + * pending_masks. (b) is only true for the last fds we returned from aeApiPoll, + * and only until we enter aeApiPoll again (at which point we restore the + * in-kernel association). + */ +#define MAX_EVENT_BATCHSZ 512 + +typedef struct aeApiState { + int portfd; /* event port */ + int npending; /* # of pending fds */ + int pending_fds[MAX_EVENT_BATCHSZ]; /* pending fds */ + int pending_masks[MAX_EVENT_BATCHSZ]; /* pending fds' masks */ +} aeApiState; + +static int aeApiCreate(aeEventLoop *eventLoop) { + int i; + aeApiState *state = zmalloc(sizeof(aeApiState)); + if (!state) return -1; + + state->portfd = port_create(); + if (state->portfd == -1) { + zfree(state); + return -1; + } + + state->npending = 0; + + for (i = 0; i < MAX_EVENT_BATCHSZ; i++) { + state->pending_fds[i] = -1; + state->pending_masks[i] = AE_NONE; + } + + eventLoop->apidata = state; + return 0; +} + +static int aeApiResize(aeEventLoop *eventLoop, int setsize) { + /* Nothing to resize here. */ + return 0; +} + +static void aeApiFree(aeEventLoop *eventLoop) { + aeApiState *state = eventLoop->apidata; + + close(state->portfd); + zfree(state); +} + +static int aeApiLookupPending(aeApiState *state, int fd) { + int i; + + for (i = 0; i < state->npending; i++) { + if (state->pending_fds[i] == fd) + return (i); + } + + return (-1); +} + +/* + * Helper function to invoke port_associate for the given fd and mask. + */ +static int aeApiAssociate(const char *where, int portfd, int fd, int mask) { + int events = 0; + int rv, err; + + if (mask & AE_READABLE) + events |= POLLIN; + if (mask & AE_WRITABLE) + events |= POLLOUT; + + if (evport_debug) + fprintf(stderr, "%s: port_associate(%d, 0x%x) = ", where, fd, events); + + rv = port_associate(portfd, PORT_SOURCE_FD, fd, events, + (void *)(uintptr_t)mask); + err = errno; + + if (evport_debug) + fprintf(stderr, "%d (%s)\n", rv, rv == 0 ? "no error" : strerror(err)); + + if (rv == -1) { + fprintf(stderr, "%s: port_associate: %s\n", where, strerror(err)); + + if (err == EAGAIN) + fprintf(stderr, "aeApiAssociate: event port limit exceeded."); + } + + return rv; +} + +static int aeApiAddEvent(aeEventLoop *eventLoop, int fd, int mask) { + aeApiState *state = eventLoop->apidata; + int fullmask, pfd; + + if (evport_debug) + fprintf(stderr, "aeApiAddEvent: fd %d mask 0x%x\n", fd, mask); + + /* + * Since port_associate's "events" argument replaces any existing events, we + * must be sure to include whatever events are already associated when + * we call port_associate() again. + */ + fullmask = mask | eventLoop->events[fd].mask; + pfd = aeApiLookupPending(state, fd); + + if (pfd != -1) { + /* + * This fd was recently returned from aeApiPoll. It should be safe to + * assume that the consumer has processed that poll event, but we play + * it safer by simply updating pending_mask. The fd will be + * re-associated as usual when aeApiPoll is called again. + */ + if (evport_debug) + fprintf(stderr, "aeApiAddEvent: adding to pending fd %d\n", fd); + state->pending_masks[pfd] |= fullmask; + return 0; + } + + return (aeApiAssociate("aeApiAddEvent", state->portfd, fd, fullmask)); +} + +static void aeApiDelEvent(aeEventLoop *eventLoop, int fd, int mask) { + aeApiState *state = eventLoop->apidata; + int fullmask, pfd; + + if (evport_debug) + fprintf(stderr, "del fd %d mask 0x%x\n", fd, mask); + + pfd = aeApiLookupPending(state, fd); + + if (pfd != -1) { + if (evport_debug) + fprintf(stderr, "deleting event from pending fd %d\n", fd); + + /* + * This fd was just returned from aeApiPoll, so it's not currently + * associated with the port. All we need to do is update + * pending_mask appropriately. + */ + state->pending_masks[pfd] &= ~mask; + + if (state->pending_masks[pfd] == AE_NONE) + state->pending_fds[pfd] = -1; + + return; + } + + /* + * The fd is currently associated with the port. Like with the add case + * above, we must look at the full mask for the file descriptor before + * updating that association. We don't have a good way of knowing what the + * events are without looking into the eventLoop state directly. We rely on + * the fact that our caller has already updated the mask in the eventLoop. + */ + + fullmask = eventLoop->events[fd].mask; + if (fullmask == AE_NONE) { + /* + * We're removing *all* events, so use port_dissociate to remove the + * association completely. Failure here indicates a bug. + */ + if (evport_debug) + fprintf(stderr, "aeApiDelEvent: port_dissociate(%d)\n", fd); + + if (port_dissociate(state->portfd, PORT_SOURCE_FD, fd) != 0) { + perror("aeApiDelEvent: port_dissociate"); + abort(); /* will not return */ + } + } else if (aeApiAssociate("aeApiDelEvent", state->portfd, fd, + fullmask) != 0) { + /* + * ENOMEM is a potentially transient condition, but the kernel won't + * generally return it unless things are really bad. EAGAIN indicates + * we've reached an resource limit, for which it doesn't make sense to + * retry (counter-intuitively). All other errors indicate a bug. In any + * of these cases, the best we can do is to abort. + */ + abort(); /* will not return */ + } +} + +static int aeApiPoll(aeEventLoop *eventLoop, struct timeval *tvp) { + aeApiState *state = eventLoop->apidata; + struct timespec timeout, *tsp; + int mask, i; + uint_t nevents; + port_event_t event[MAX_EVENT_BATCHSZ]; + + /* + * If we've returned fd events before, we must re-associate them with the + * port now, before calling port_get(). See the block comment at the top of + * this file for an explanation of why. + */ + for (i = 0; i < state->npending; i++) { + if (state->pending_fds[i] == -1) + /* This fd has since been deleted. */ + continue; + + if (aeApiAssociate("aeApiPoll", state->portfd, + state->pending_fds[i], state->pending_masks[i]) != 0) { + /* See aeApiDelEvent for why this case is fatal. */ + abort(); + } + + state->pending_masks[i] = AE_NONE; + state->pending_fds[i] = -1; + } + + state->npending = 0; + + if (tvp != NULL) { + timeout.tv_sec = tvp->tv_sec; + timeout.tv_nsec = tvp->tv_usec * 1000; + tsp = &timeout; + } else { + tsp = NULL; + } + + /* + * port_getn can return with errno == ETIME having returned some events (!). + * So if we get ETIME, we check nevents, too. + */ + nevents = 1; + if (port_getn(state->portfd, event, MAX_EVENT_BATCHSZ, &nevents, + tsp) == -1 && (errno != ETIME || nevents == 0)) { + if (errno == ETIME || errno == EINTR) + return 0; + + /* Any other error indicates a bug. */ + perror("aeApiPoll: port_get"); + abort(); + } + + state->npending = nevents; + + for (i = 0; i < nevents; i++) { + mask = 0; + if (event[i].portev_events & POLLIN) + mask |= AE_READABLE; + if (event[i].portev_events & POLLOUT) + mask |= AE_WRITABLE; + + eventLoop->fired[i].fd = event[i].portev_object; + eventLoop->fired[i].mask = mask; + + if (evport_debug) + fprintf(stderr, "aeApiPoll: fd %d mask 0x%x\n", + (int)event[i].portev_object, mask); + + state->pending_fds[i] = event[i].portev_object; + state->pending_masks[i] = (uintptr_t)event[i].portev_user; + } + + return nevents; +} + +static char *aeApiName(void) { + return "evport"; +} diff --git a/thirdparty/ae/ae_kqueue.c b/thirdparty/ae/ae_kqueue.c new file mode 100644 index 000000000..6796f4ceb --- /dev/null +++ b/thirdparty/ae/ae_kqueue.c @@ -0,0 +1,138 @@ +/* Kqueue(2)-based ae.c module + * + * Copyright (C) 2009 Harish Mallipeddi - harish.mallipeddi@gmail.com + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are met: + * + * * Redistributions of source code must retain the above copyright notice, + * this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * * Neither the name of Redis nor the names of its contributors may be used + * to endorse or promote products derived from this software without + * specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" + * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE + * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR + * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS + * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN + * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE + * POSSIBILITY OF SUCH DAMAGE. + */ + + +#include +#include +#include + +typedef struct aeApiState { + int kqfd; + struct kevent *events; +} aeApiState; + +static int aeApiCreate(aeEventLoop *eventLoop) { + aeApiState *state = zmalloc(sizeof(aeApiState)); + + if (!state) return -1; + state->events = zmalloc(sizeof(struct kevent)*eventLoop->setsize); + if (!state->events) { + zfree(state); + return -1; + } + state->kqfd = kqueue(); + if (state->kqfd == -1) { + zfree(state->events); + zfree(state); + return -1; + } + eventLoop->apidata = state; + return 0; +} + +static int aeApiResize(aeEventLoop *eventLoop, int setsize) { + aeApiState *state = eventLoop->apidata; + + state->events = zrealloc(state->events, sizeof(struct kevent)*setsize); + return 0; +} + +static void aeApiFree(aeEventLoop *eventLoop) { + aeApiState *state = eventLoop->apidata; + + close(state->kqfd); + zfree(state->events); + zfree(state); +} + +static int aeApiAddEvent(aeEventLoop *eventLoop, int fd, int mask) { + aeApiState *state = eventLoop->apidata; + struct kevent ke; + + if (mask & AE_READABLE) { + EV_SET(&ke, fd, EVFILT_READ, EV_ADD, 0, 0, NULL); + if (kevent(state->kqfd, &ke, 1, NULL, 0, NULL) == -1) return -1; + } + if (mask & AE_WRITABLE) { + EV_SET(&ke, fd, EVFILT_WRITE, EV_ADD, 0, 0, NULL); + if (kevent(state->kqfd, &ke, 1, NULL, 0, NULL) == -1) return -1; + } + return 0; +} + +static void aeApiDelEvent(aeEventLoop *eventLoop, int fd, int mask) { + aeApiState *state = eventLoop->apidata; + struct kevent ke; + + if (mask & AE_READABLE) { + EV_SET(&ke, fd, EVFILT_READ, EV_DELETE, 0, 0, NULL); + kevent(state->kqfd, &ke, 1, NULL, 0, NULL); + } + if (mask & AE_WRITABLE) { + EV_SET(&ke, fd, EVFILT_WRITE, EV_DELETE, 0, 0, NULL); + kevent(state->kqfd, &ke, 1, NULL, 0, NULL); + } +} + +static int aeApiPoll(aeEventLoop *eventLoop, struct timeval *tvp) { + aeApiState *state = eventLoop->apidata; + int retval, numevents = 0; + + if (tvp != NULL) { + struct timespec timeout; + timeout.tv_sec = tvp->tv_sec; + timeout.tv_nsec = tvp->tv_usec * 1000; + retval = kevent(state->kqfd, NULL, 0, state->events, eventLoop->setsize, + &timeout); + } else { + retval = kevent(state->kqfd, NULL, 0, state->events, eventLoop->setsize, + NULL); + } + + if (retval > 0) { + int j; + + numevents = retval; + for(j = 0; j < numevents; j++) { + int mask = 0; + struct kevent *e = state->events+j; + + if (e->filter == EVFILT_READ) mask |= AE_READABLE; + if (e->filter == EVFILT_WRITE) mask |= AE_WRITABLE; + eventLoop->fired[j].fd = e->ident; + eventLoop->fired[j].mask = mask; + } + } + return numevents; +} + +static char *aeApiName(void) { + return "kqueue"; +} diff --git a/thirdparty/ae/ae_select.c b/thirdparty/ae/ae_select.c new file mode 100644 index 000000000..c039a8ea3 --- /dev/null +++ b/thirdparty/ae/ae_select.c @@ -0,0 +1,106 @@ +/* Select()-based ae.c module. + * + * Copyright (c) 2009-2012, Salvatore Sanfilippo + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are met: + * + * * Redistributions of source code must retain the above copyright notice, + * this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * * Neither the name of Redis nor the names of its contributors may be used + * to endorse or promote products derived from this software without + * specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" + * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE + * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR + * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS + * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN + * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE + * POSSIBILITY OF SUCH DAMAGE. + */ + + +#include +#include + +typedef struct aeApiState { + fd_set rfds, wfds; + /* We need to have a copy of the fd sets as it's not safe to reuse + * FD sets after select(). */ + fd_set _rfds, _wfds; +} aeApiState; + +static int aeApiCreate(aeEventLoop *eventLoop) { + aeApiState *state = zmalloc(sizeof(aeApiState)); + + if (!state) return -1; + FD_ZERO(&state->rfds); + FD_ZERO(&state->wfds); + eventLoop->apidata = state; + return 0; +} + +static int aeApiResize(aeEventLoop *eventLoop, int setsize) { + /* Just ensure we have enough room in the fd_set type. */ + if (setsize >= FD_SETSIZE) return -1; + return 0; +} + +static void aeApiFree(aeEventLoop *eventLoop) { + zfree(eventLoop->apidata); +} + +static int aeApiAddEvent(aeEventLoop *eventLoop, int fd, int mask) { + aeApiState *state = eventLoop->apidata; + + if (mask & AE_READABLE) FD_SET(fd,&state->rfds); + if (mask & AE_WRITABLE) FD_SET(fd,&state->wfds); + return 0; +} + +static void aeApiDelEvent(aeEventLoop *eventLoop, int fd, int mask) { + aeApiState *state = eventLoop->apidata; + + if (mask & AE_READABLE) FD_CLR(fd,&state->rfds); + if (mask & AE_WRITABLE) FD_CLR(fd,&state->wfds); +} + +static int aeApiPoll(aeEventLoop *eventLoop, struct timeval *tvp) { + aeApiState *state = eventLoop->apidata; + int retval, j, numevents = 0; + + memcpy(&state->_rfds,&state->rfds,sizeof(fd_set)); + memcpy(&state->_wfds,&state->wfds,sizeof(fd_set)); + + retval = select(eventLoop->maxfd+1, + &state->_rfds,&state->_wfds,NULL,tvp); + if (retval > 0) { + for (j = 0; j <= eventLoop->maxfd; j++) { + int mask = 0; + aeFileEvent *fe = &eventLoop->events[j]; + + if (fe->mask == AE_NONE) continue; + if (fe->mask & AE_READABLE && FD_ISSET(j,&state->_rfds)) + mask |= AE_READABLE; + if (fe->mask & AE_WRITABLE && FD_ISSET(j,&state->_wfds)) + mask |= AE_WRITABLE; + eventLoop->fired[numevents].fd = j; + eventLoop->fired[numevents].mask = mask; + numevents++; + } + } + return numevents; +} + +static char *aeApiName(void) { + return "select"; +} diff --git a/thirdparty/ae/config.h b/thirdparty/ae/config.h new file mode 100644 index 000000000..4f8e1ea1b --- /dev/null +++ b/thirdparty/ae/config.h @@ -0,0 +1,54 @@ +/* + * Copyright (c) 2009-2012, Salvatore Sanfilippo + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are met: + * + * * Redistributions of source code must retain the above copyright notice, + * this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * * Neither the name of Redis nor the names of its contributors may be used + * to endorse or promote products derived from this software without + * specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" + * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE + * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR + * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS + * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN + * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE + * POSSIBILITY OF SUCH DAMAGE. + */ + +#ifndef __CONFIG_H +#define __CONFIG_H + +#ifdef __APPLE__ +#include +#endif + +/* Test for polling API */ +#ifdef __linux__ +#define HAVE_EPOLL 1 +#endif + +#if (defined(__APPLE__) && defined(MAC_OS_X_VERSION_10_6)) || defined(__FreeBSD__) || defined(__OpenBSD__) || defined (__NetBSD__) +#define HAVE_KQUEUE 1 +#endif + +#ifdef __sun +#include +#ifdef _DTRACE_VERSION +#define HAVE_EVPORT 1 +#endif +#endif + + +#endif diff --git a/thirdparty/ae/zmalloc.h b/thirdparty/ae/zmalloc.h new file mode 100644 index 000000000..54c8a69cb --- /dev/null +++ b/thirdparty/ae/zmalloc.h @@ -0,0 +1,16 @@ +#ifndef _ZMALLOC_H +#define _ZMALLOC_H + +#ifndef zmalloc +#define zmalloc malloc +#endif + +#ifndef zfree +#define zfree free +#endif + +#ifndef zrealloc +#define zrealloc realloc +#endif + +#endif /* _ZMALLOC_H */ From 1e08629013b6ef4c9fa737c89fae8f134b01d982 Mon Sep 17 00:00:00 2001 From: Philipp Moritz Date: Sun, 25 Sep 2016 16:51:24 -0700 Subject: [PATCH 15/36] fix submit_task for task queue (#21) --- state/redis.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/state/redis.c b/state/redis.c index 9c4b81f03..d10b06b75 100644 --- a/state/redis.c +++ b/state/redis.c @@ -165,7 +165,7 @@ void task_queue_submit_task(db_conn *db, task_iid task_iid, task_spec *task) { UT_string *command; utstring_new(command); sha1_to_hex(&task_iid.id[0], &hex[0]); - utstring_printf(command, "HMSET queue:%s", &hex[0]); + utstring_printf(command, "HMSET queue:%s ", &hex[0]); print_task(task, command); redisAsyncCommand(db->context, NULL, NULL, utstring_body(command)); if (db->context->err) { From db8c0acc711ffa643585bcec0ee659bcf2887675 Mon Sep 17 00:00:00 2001 From: Philipp Moritz Date: Sun, 25 Sep 2016 21:52:06 -0700 Subject: [PATCH 16/36] Refactor state database (#22) * make db_connect return the connection * rename db_conn -> db_handle * more renaming * clang-format * free the db_handle --- logging.c | 4 ++-- state/db.h | 23 +++++++++++------------ state/object_table.h | 8 +++++--- state/redis.c | 26 ++++++++++++++------------ state/redis.h | 2 +- state/task_queue.h | 6 +++--- state/task_table.h | 4 ++-- test/db_tests.c | 33 +++++++++++++++------------------ test/redis_tests.c | 30 ++++++++++++++---------------- 9 files changed, 67 insertions(+), 69 deletions(-) diff --git a/logging.c b/logging.c index 38abc7bf9..bf37f1ee5 100644 --- a/logging.c +++ b/logging.c @@ -17,7 +17,7 @@ struct ray_logger_impl { int log_level; /* Whether or not we have a direct connection to Redis. */ int is_direct; - /* Either a db_conn or a socket to a process with a db_conn, + /* Either a db_handle or a socket to a process with a db_handle, * depending on the is_direct flag. */ void *conn; }; @@ -57,7 +57,7 @@ void ray_log(ray_logger *logger, UT_string *origin_id; utstring_new(origin_id); if (logger->is_direct) { - db_conn *db = (db_conn *) logger->conn; + db_handle *db = (db_handle *) logger->conn; utstring_printf(origin_id, "%ld:%s", db->client_id, ""); redisAsyncCommand(db->context, NULL, NULL, log_fmt, utstring_body(timestamp), logger->client_type, diff --git a/state/db.h b/state/db.h index 3fcf658fd..e6ca089a0 100644 --- a/state/db.h +++ b/state/db.h @@ -3,22 +3,21 @@ #include "event_loop.h" -typedef struct db_conn_impl db_conn; +typedef struct db_handle_impl db_handle; -/* Connect to the global system store at address and port. The last - * parameter is an output parameter and we assume the memory is - * allocated by the caller. */ -void db_connect(const char *db_address, - int db_port, - const char *client_type, - const char *client_addr, - int client_port, - db_conn *db); +/* Connect to the global system store at address and port. Returns + * a handle to the database, which must be freed with db_disconnect + * after use. */ +db_handle *db_connect(const char *db_address, + int db_port, + const char *client_type, + const char *client_addr, + int client_port); /* Attach global system store connection to event loop. */ -void db_attach(db_conn *db, event_loop *loop); +void db_attach(db_handle *db, event_loop *loop); /* Disconnect from the global system store. */ -void db_disconnect(db_conn *db); +void db_disconnect(db_handle *db); #endif diff --git a/state/object_table.h b/state/object_table.h index 7c00ab2ba..e2eb89433 100644 --- a/state/object_table.h +++ b/state/object_table.h @@ -10,12 +10,14 @@ typedef void (*lookup_callback)(object_id object_id, /* Register a new object with the directory. */ /* TODO(pcm): Retry, print for each attempt. */ -void object_table_add(db_conn *db, object_id object_id); +void object_table_add(db_handle *db, object_id object_id); /* Remove object from the directory. */ -void object_table_remove(db_conn *db, object_id object_id, const char *manager); +void object_table_remove(db_handle *db, + object_id object_id, + const char *manager); /* Look up entry from the directory */ -void object_table_lookup(db_conn *db, +void object_table_lookup(db_handle *db, object_id object_id, lookup_callback callback); diff --git a/state/redis.c b/state/redis.c index d10b06b75..aa2011d9a 100644 --- a/state/redis.c +++ b/state/redis.c @@ -31,12 +31,12 @@ } \ } while (0); -void db_connect(const char *address, - int port, - const char *client_type, - const char *client_addr, - int client_port, - db_conn *db) { +db_handle *db_connect(const char *address, + int port, + const char *client_type, + const char *client_addr, + int client_port) { + db_handle *db = malloc(sizeof(db_handle)); /* Sync connection for initial handshake */ redisReply *reply; long long num_clients; @@ -75,9 +75,10 @@ void db_connect(const char *address, CHECK_REDIS_CONNECT(redisAsyncContext, db->context, "could not connect to redis %s:%d", address, port); db->context->data = (void *) db; + return db; } -void db_disconnect(db_conn *db) { +void db_disconnect(db_handle *db) { redisFree(db->sync_context); redisAsyncFree(db->context); service_cache_entry *e, *tmp; @@ -87,13 +88,14 @@ void db_disconnect(db_conn *db) { free(e); } free(db->client_type); + free(db); } -void db_attach(db_conn *db, event_loop *loop) { +void db_attach(db_handle *db, event_loop *loop) { redisAeAttach(loop, db->context); } -void object_table_add(db_conn *db, unique_id object_id) { +void object_table_add(db_handle *db, unique_id object_id) { static char hex_object_id[2 * UNIQUE_ID_SIZE + 1]; sha1_to_hex(&object_id.id[0], &hex_object_id[0]); redisAsyncCommand(db->context, NULL, NULL, "SADD obj:%s %d", @@ -104,7 +106,7 @@ void object_table_add(db_conn *db, unique_id object_id) { } void object_table_get_entry(redisAsyncContext *c, void *r, void *privdata) { - db_conn *db = c->data; + db_handle *db = c->data; lookup_callback_data *cb_data = privdata; redisReply *reply = r; if (reply == NULL) @@ -143,7 +145,7 @@ void object_table_get_entry(redisAsyncContext *c, void *r, void *privdata) { free(result); } -void object_table_lookup(db_conn *db, +void object_table_lookup(db_handle *db, object_id object_id, lookup_callback callback) { static char hex_object_id[2 * UNIQUE_ID_SIZE + 1]; @@ -158,7 +160,7 @@ void object_table_lookup(db_conn *db, } } -void task_queue_submit_task(db_conn *db, task_iid task_iid, task_spec *task) { +void task_queue_submit_task(db_handle *db, task_iid task_iid, task_spec *task) { /* For converting an id to hex, which has double the number * of bytes compared to the id (+ 1 byte for '\0'). */ static char hex[2 * UNIQUE_ID_SIZE + 1]; diff --git a/state/redis.h b/state/redis.h index c579e7065..23ebb2ba6 100644 --- a/state/redis.h +++ b/state/redis.h @@ -17,7 +17,7 @@ typedef struct { UT_hash_handle hh; } service_cache_entry; -struct db_conn_impl { +struct db_handle_impl { /* String that identifies this client type. */ char *client_type; /* Unique ID for this client within the type. */ diff --git a/state/task_queue.h b/state/task_queue.h index 0226c501b..92968707e 100644 --- a/state/task_queue.h +++ b/state/task_queue.h @@ -21,13 +21,13 @@ typedef unique_id node_id; typedef void (*task_queue_callback)(task_iid *task_iid, task_spec *task); /* Submit task to the global scheduler. */ -void task_queue_submit_task(db_conn *db, task_iid task_iid, task_spec *task); +void task_queue_submit_task(db_handle *db, task_iid task_iid, task_spec *task); /* Submit task to a local scheduler based on the decision made by the global * scheduler. */ -void task_queue_schedule_task(db_conn *db, task_iid task_iid, node_id node); +void task_queue_schedule_task(db_handle *db, task_iid task_iid, node_id node); /* Subscribe to task queue. */ -void task_queue_register_callback(db_conn *db, task_queue_callback callback); +void task_queue_register_callback(db_handle *db, task_queue_callback callback); #endif diff --git a/state/task_table.h b/state/task_table.h index 64285da67..71e879c2c 100644 --- a/state/task_table.h +++ b/state/task_table.h @@ -5,9 +5,9 @@ #include "task.h" /* Add task to the task table, handle errors here. */ -status task_table_add_task(db_conn *db, task_iid task_iid, task_spec *task); +status task_table_add_task(db_handle *db, task_iid task_iid, task_spec *task); /* Get specific task from the task table. */ -status task_table_get_task(db_conn *db, task_iid task_iid, task_spec *task); +status task_table_get_task(db_handle *db, task_iid task_iid, task_spec *task); #endif /* TASK_TABLE_H */ diff --git a/test/db_tests.c b/test/db_tests.c index d9dfcb563..99fad7e18 100644 --- a/test/db_tests.c +++ b/test/db_tests.c @@ -45,20 +45,18 @@ int64_t timeout_handler(event_loop *loop, int64_t id, void *context) { TEST object_table_lookup_test(void) { event_loop *loop = event_loop_create(); - db_conn conn1; - db_connect("127.0.0.1", 6379, "plasma_manager", manager_addr, manager_port1, - &conn1); - db_conn conn2; - db_connect("127.0.0.1", 6379, "plasma_manager", manager_addr, manager_port2, - &conn2); - db_attach(&conn1, loop); - db_attach(&conn2, loop); + db_handle *db1 = db_connect("127.0.0.1", 6379, "plasma_manager", manager_addr, + manager_port1); + db_handle *db2 = db_connect("127.0.0.1", 6379, "plasma_manager", manager_addr, + manager_port2); + db_attach(db1, loop); + db_attach(db2, loop); unique_id id = globally_unique_id(); - object_table_add(&conn1, id); - object_table_add(&conn2, id); + object_table_add(db1, id); + object_table_add(db2, id); event_loop_add_timer(loop, 100, timeout_handler, NULL); event_loop_run(loop); - object_table_lookup(&conn1, id, test_callback); + object_table_lookup(db1, id, test_callback); event_loop_add_timer(loop, 100, timeout_handler, NULL); event_loop_run(loop); int port1 = atoi(received_port1); @@ -67,8 +65,8 @@ TEST object_table_lookup_test(void) { ASSERT((port1 == manager_port1 && port2 == manager_port2) || (port2 == manager_port1 && port1 == manager_port2)); - db_disconnect(&conn1); - db_disconnect(&conn2); + db_disconnect(db1); + db_disconnect(db2); event_loop_destroy(loop); PASS(); @@ -76,17 +74,16 @@ TEST object_table_lookup_test(void) { TEST task_queue_test(void) { event_loop *loop = event_loop_create(); - db_conn conn; - db_connect("127.0.0.1", 6379, "local_scheduler", "", -1, &conn); - db_attach(&conn, loop); + db_handle *db = db_connect("127.0.0.1", 6379, "local_scheduler", "", -1); + db_attach(db, loop); task_spec *task = example_task(); - task_queue_submit_task(&conn, globally_unique_id(), task); + task_queue_submit_task(db, globally_unique_id(), task); event_loop_add_timer(loop, 100, timeout_handler, NULL); event_loop_run(loop); free_task_spec(task); - db_disconnect(&conn); + db_disconnect(db); event_loop_destroy(loop); PASS(); } diff --git a/test/redis_tests.c b/test/redis_tests.c index d527d0476..b1cde98b6 100644 --- a/test/redis_tests.c +++ b/test/redis_tests.c @@ -68,10 +68,10 @@ TEST redis_socket_test(void) { } void redis_read_callback(event_loop *loop, int fd, void *context, int events) { - db_conn *conn = context; + db_handle *db = context; char *cmd = read_string(fd); - redisAsyncCommand(conn->context, async_redis_socket_test_callback, NULL, cmd, - conn->client_id, 0); + redisAsyncCommand(db->context, async_redis_socket_test_callback, NULL, cmd, + db->client_id, 0); free(cmd); } @@ -102,9 +102,8 @@ TEST async_redis_socket_test(void) { utarray_push_back(connections, &socket_fd); /* Start connection to Redis. */ - db_conn conn; - db_connect("127.0.0.1", 6379, "", "", 0, &conn); - db_attach(&conn, loop); + db_handle *db = db_connect("127.0.0.1", 6379, "", "", 0); + db_attach(db, loop); /* Send a command to the Redis process. */ int client_fd = connect_ipc_sock(socket_pathname); @@ -113,15 +112,15 @@ TEST async_redis_socket_test(void) { write_formatted_string(client_fd, test_set_format, test_key, test_value); event_loop_add_file(loop, client_fd, EVENT_LOOP_READ, redis_read_callback, - &conn); + db); event_loop_add_file(loop, socket_fd, EVENT_LOOP_READ, redis_accept_callback, - &conn); + db); event_loop_add_timer(loop, 100, timeout_handler, NULL); event_loop_run(loop); CHECK(async_redis_socket_test_callback_called); - db_disconnect(&conn); + db_disconnect(db); event_loop_destroy(loop); for (int *p = (int *) utarray_front(connections); p != NULL; p = (int *) utarray_next(connections, p)) { @@ -148,7 +147,7 @@ void logging_read_callback(event_loop *loop, int fd, void *context, int events) { - db_conn *conn = context; + db_handle *conn = context; char *cmd = read_string(fd); redisAsyncCommand(conn->context, logging_test_callback, NULL, cmd, conn->client_id, 0); @@ -177,9 +176,8 @@ TEST logging_test(void) { utarray_push_back(connections, &socket_fd); /* Start connection to Redis. */ - db_conn conn; - db_connect("127.0.0.1", 6379, "", "", 0, &conn); - db_attach(&conn, loop); + db_handle *conn = db_connect("127.0.0.1", 6379, "", "", 0); + db_attach(conn, loop); /* Send a command to the Redis process. */ int client_fd = connect_ipc_sock(socket_pathname); @@ -189,16 +187,16 @@ TEST logging_test(void) { ray_log(logger, RAY_INFO, "TEST", "Message"); event_loop_add_file(loop, socket_fd, EVENT_LOOP_READ, logging_accept_callback, - &conn); + conn); event_loop_add_file(loop, client_fd, EVENT_LOOP_READ, logging_read_callback, - &conn); + conn); event_loop_add_timer(loop, 100, timeout_handler, NULL); event_loop_run(loop); CHECK(logging_test_callback_called); free_ray_logger(logger); - db_disconnect(&conn); + db_disconnect(conn); event_loop_destroy(loop); for (int *p = (int *) utarray_front(connections); p != NULL; p = (int *) utarray_next(connections, p)) { From 631de921703a750ad8223089bb61a099a04cf248 Mon Sep 17 00:00:00 2001 From: Robert Nishihara Date: Mon, 26 Sep 2016 00:12:11 -0700 Subject: [PATCH 17/36] Build redis before libcommon. (#20) --- Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Makefile b/Makefile index f71273119..ad955de98 100644 --- a/Makefile +++ b/Makefile @@ -2,7 +2,7 @@ CC = gcc CFLAGS = -g -Wall --std=c99 -D_XOPEN_SOURCE=500 -D_POSIX_C_SOURCE=200809L -fPIC -I. -Ithirdparty -Ithirdparty/ae BUILD = build -all: $(BUILD)/libcommon.a +all: hiredis $(BUILD)/libcommon.a $(BUILD)/libcommon.a: event_loop.o common.o task.o io.o state/redis.o thirdparty/ae/ae.o ar rcs $@ $^ From 084220b0e70de6bed466e97e08f4b6909133aafb Mon Sep 17 00:00:00 2001 From: Robert Nishihara Date: Tue, 27 Sep 2016 18:51:35 -0700 Subject: [PATCH 18/36] Allow reading/writing generic message types, not just tasks. (#24) * Allow reading/writing generic message types, not just tasks. * Allow messages of length 0 to be read/written, and handle closed sockets. * Address comments. * Simplify accept_client. * Allow ports to be reused in bind_ipc_sock. --- io.c | 100 +++++++++++++++++++++++++++------------------ io.h | 19 ++++++--- logging.c | 8 ++-- task.c | 13 ------ test/io_tests.c | 11 +++-- test/redis_tests.c | 10 ++--- test/task_tests.c | 10 ++++- 7 files changed, 99 insertions(+), 72 deletions(-) diff --git a/io.c b/io.c index d4a89a4a4..16073d32a 100644 --- a/io.c +++ b/io.c @@ -18,13 +18,19 @@ * occurred. */ int bind_ipc_sock(const char *socket_pathname) { struct sockaddr_un socket_address; - int socket_fd; - - socket_fd = socket(AF_UNIX, SOCK_STREAM, 0); + int socket_fd = socket(AF_UNIX, SOCK_STREAM, 0); if (socket_fd < 0) { LOG_ERR("socket() failed for pathname %s.", socket_pathname); return -1; } + /* Tell the system to allow the port to be reused. */ + int on = 1; + if (setsockopt(socket_fd, SOL_SOCKET, SO_REUSEADDR, (char *) &on, + sizeof(on)) < 0) { + LOG_ERR("setsockopt failed"); + close(socket_fd); + exit(-1); + } unlink(socket_pathname); memset(&socket_address, 0, sizeof(struct sockaddr_un)); @@ -80,11 +86,7 @@ int connect_ipc_sock(const char *socket_pathname) { /* Accept a new client connection on the given socket * descriptor. Returns a descriptor for the new socket. */ int accept_client(int socket_fd) { - struct sockaddr_un client_addr; - int client_fd, client_len; - client_len = sizeof(client_addr); - client_fd = accept(socket_fd, (struct sockaddr *) &client_addr, - (socklen_t *) &client_len); + int client_fd = accept(socket_fd, NULL, NULL); if (client_fd < 0) { LOG_ERR("Error reading from socket."); return -1; @@ -92,57 +94,77 @@ int accept_client(int socket_fd) { return client_fd; } -/* Write a sequence of bytes on a file descriptor. */ -void write_bytes(int fd, uint8_t *bytes, int64_t length) { - ssize_t nbytes = write(fd, (char *) &length, sizeof(length)); - if (nbytes == -1) { - LOG_ERR("Error sending to socket.\n"); - return; - } +/** + * Write a sequence of bytes on a file descriptor. The bytes should then be read + * by read_message. + * + * @param fd The file descriptor to write to. + * @param type The type of the message to send. + * @param length The size in bytes of the bytes parameter. + * @param bytes The address of the message to send. + * @return Void. + */ +void write_message(int fd, int64_t type, int64_t length, uint8_t *bytes) { + ssize_t nbytes = write(fd, (char *) &type, sizeof(type)); + CHECK(nbytes == sizeof(int64_t)); + nbytes = write(fd, (char *) &length, sizeof(length)); + CHECK(nbytes == sizeof(int64_t)); nbytes = write(fd, (char *) bytes, length * sizeof(char)); - if (nbytes == -1) { - LOG_ERR("Error sending to socket.\n"); - return; - } + CHECK(nbytes >= 0); } -/* Read a sequence of bytes written by write_bytes from a file descriptor. - * Allocates and returns a pointer to the bytes. - * NOTE: Caller must free the memory! */ -void read_bytes(int fd, uint8_t **bytes, int64_t *length) { - ssize_t nbytes = read(fd, length, sizeof(int64_t)); - if (nbytes < 0) { - LOG_ERR("Error reading length of message from socket."); +/** + * Read a sequence of bytes written by write_bytes from a file descriptor. This + * allocates space for the message. + * + * @note The caller must free the memory. + * + * @param fd The file descriptor to read from. + * @param type The type of the message that is read will be written at this + address. + * @param length The size in bytes of the message that is read will be written + at this address. This size does not include the bytes used to encode + the type and length. + * @param bytes The address at which to write the pointer to the bytes that are + read and allocated by this function. + * @return Void. + */ +void read_message(int fd, int64_t *type, int64_t *length, uint8_t **bytes) { + ssize_t nbytes = read(fd, type, sizeof(int64_t)); + CHECK(nbytes >= 0); + /* Handle the case in which the socket is closed. */ + if (nbytes == 0) { + *type = DISCONNECT_CLIENT; + *length = 0; *bytes = NULL; return; } - + nbytes = read(fd, length, sizeof(int64_t)); + CHECK(nbytes == sizeof(int64_t)); *bytes = malloc(*length * sizeof(uint8_t)); nbytes = read(fd, *bytes, *length); - if (nbytes < 0) { - LOG_ERR("Error reading message from socket."); - free(*bytes); - *bytes = NULL; - } + CHECK(nbytes >= 0); } /* Write a null-terminated string to a file descriptor. */ -void write_string(int fd, char *message) { +void write_log_message(int fd, char *message) { /* Account for the \0 at the end of the string. */ - write_bytes(fd, (uint8_t *) message, strlen(message) + 1); + write_message(fd, LOG_MESSAGE, strlen(message) + 1, (uint8_t *) message); } /* Reads a null-terminated string from the file descriptor that has been - * written by write_string. Allocates and returns a pointer to the string. + * written by write_log_message. Allocates and returns a pointer to the string. * NOTE: Caller must free the memory! */ -char *read_string(int fd) { +char *read_log_message(int fd) { uint8_t *bytes; + int64_t type; int64_t length; - read_bytes(fd, &bytes, &length); + read_message(fd, &type, &length, &bytes); + CHECK(type == LOG_MESSAGE); return (char *) bytes; } -void write_formatted_string(int socket_fd, const char *format, ...) { +void write_formatted_log_message(int socket_fd, const char *format, ...) { UT_string *cmd; va_list ap; @@ -151,6 +173,6 @@ void write_formatted_string(int socket_fd, const char *format, ...) { utstring_printf_va(cmd, format, ap); va_end(ap); - write_string(socket_fd, utstring_body(cmd)); + write_log_message(socket_fd, utstring_body(cmd)); utstring_free(cmd); } diff --git a/io.h b/io.h index e6f227c98..2299806f7 100644 --- a/io.h +++ b/io.h @@ -3,6 +3,15 @@ #include +enum common_message_type { + /** Disconnect a client. */ + DISCONNECT_CLIENT, + /** Log a message from a client. */ + LOG_MESSAGE, + /** Submit a task to the local scheduler. */ + SUBMIT_TASK, +}; + /* Helper functions for socket communication. */ int bind_ipc_sock(const char *socket_pathname); @@ -12,11 +21,11 @@ int accept_client(int socket_fd); /* Reading and writing data */ -void write_bytes(int fd, uint8_t *bytes, int64_t length); -void read_bytes(int fd, uint8_t **bytes, int64_t *length); +void write_message(int fd, int64_t type, int64_t length, uint8_t *bytes); +void read_message(int fd, int64_t *type, int64_t *length, uint8_t **bytes); -void write_string(int fd, char *message); -void write_formatted_string(int fd, const char *format, ...); -char *read_string(int fd); +void write_log_message(int fd, char *message); +void write_formatted_log_message(int fd, const char *format, ...); +char *read_log_message(int fd); #endif diff --git a/logging.c b/logging.c index bf37f1ee5..1a8e96820 100644 --- a/logging.c +++ b/logging.c @@ -68,10 +68,10 @@ void ray_log(ray_logger *logger, * ID to be filled in by someone else. */ utstring_printf(origin_id, "%s:%s", "%ld", "%ld"); int *socket_fd = (int *) logger->conn; - write_formatted_string(*socket_fd, log_fmt, utstring_body(timestamp), - logger->client_type, utstring_body(origin_id), - log_levels[log_level], event_type, message, - utstring_body(timestamp)); + write_formatted_log_message(*socket_fd, log_fmt, utstring_body(timestamp), + logger->client_type, utstring_body(origin_id), + log_levels[log_level], event_type, message, + utstring_body(timestamp)); } utstring_free(origin_id); utstring_free(timestamp); diff --git a/task.c b/task.c index fd4708d78..30fde8d31 100644 --- a/task.c +++ b/task.c @@ -148,19 +148,6 @@ void free_task_spec(task_spec *spec) { free(spec); } -void write_task(int fd, task_spec *spec) { - write_bytes(fd, (uint8_t *) spec, task_size(spec)); -} - -task_spec *read_task(int fd) { - uint8_t *bytes; - int64_t length; - read_bytes(fd, &bytes, &length); - task_spec *spec = (task_spec *) bytes; - CHECK(task_size(spec) == length); - return spec; -} - void print_task(task_spec *spec, UT_string *output) { /* For converting an id to hex, which has double the number * of bytes compared to the id (+ 1 byte for '\0'). */ diff --git a/test/io_tests.c b/test/io_tests.c index 1e8d20832..b73207326 100644 --- a/test/io_tests.c +++ b/test/io_tests.c @@ -20,20 +20,23 @@ TEST ipc_socket_test(void) { close(socket_fd); socket_fd = connect_ipc_sock(socket_pathname); ASSERT(socket_fd >= 0); - write_string(socket_fd, test_string); - write_bytes(socket_fd, (uint8_t *) test_bytes, strlen(test_bytes)); + write_log_message(socket_fd, test_string); + write_message(socket_fd, LOG_MESSAGE, strlen(test_bytes), + (uint8_t *) test_bytes); close(socket_fd); exit(0); } else { int client_fd = accept_client(socket_fd); ASSERT(client_fd >= 0); - char *message = read_string(client_fd); + char *message = read_log_message(client_fd); ASSERT(message != NULL); ASSERT_STR_EQ(test_string, message); free(message); + int64_t type; int64_t len; uint8_t *bytes; - read_bytes(client_fd, &bytes, &len); + read_message(client_fd, &type, &len, &bytes); + ASSERT(type == LOG_MESSAGE); ASSERT(memcmp(test_bytes, bytes, len) == 0); free(bytes); close(client_fd); diff --git a/test/redis_tests.c b/test/redis_tests.c index b1cde98b6..2277174c0 100644 --- a/test/redis_tests.c +++ b/test/redis_tests.c @@ -45,10 +45,10 @@ TEST redis_socket_test(void) { int client_fd = connect_ipc_sock(socket_pathname); ASSERT(client_fd >= 0); - write_formatted_string(client_fd, test_set_format, test_key, test_value); + write_formatted_log_message(client_fd, test_set_format, test_key, test_value); int server_fd = accept_client(socket_fd); - char *cmd = read_string(server_fd); + char *cmd = read_log_message(server_fd); close(client_fd); close(server_fd); close(socket_fd); @@ -69,7 +69,7 @@ TEST redis_socket_test(void) { void redis_read_callback(event_loop *loop, int fd, void *context, int events) { db_handle *db = context; - char *cmd = read_string(fd); + char *cmd = read_log_message(fd); redisAsyncCommand(db->context, async_redis_socket_test_callback, NULL, cmd, db->client_id, 0); free(cmd); @@ -109,7 +109,7 @@ TEST async_redis_socket_test(void) { int client_fd = connect_ipc_sock(socket_pathname); ASSERT(client_fd >= 0); utarray_push_back(connections, &client_fd); - write_formatted_string(client_fd, test_set_format, test_key, test_value); + write_formatted_log_message(client_fd, test_set_format, test_key, test_value); event_loop_add_file(loop, client_fd, EVENT_LOOP_READ, redis_read_callback, db); @@ -148,7 +148,7 @@ void logging_read_callback(event_loop *loop, void *context, int events) { db_handle *conn = context; - char *cmd = read_string(fd); + char *cmd = read_log_message(fd); redisAsyncCommand(conn->context, logging_test_callback, NULL, cmd, conn->client_id, 0); free(cmd); diff --git a/test/task_tests.c b/test/task_tests.c index 4293eec04..fcb714737 100644 --- a/test/task_tests.c +++ b/test/task_tests.c @@ -7,6 +7,7 @@ #include "common.h" #include "test/example_task.h" #include "task.h" +#include "io.h" SUITE(task_tests); @@ -48,8 +49,13 @@ TEST send_task(void) { *task_return(task, 1) = globally_unique_id(); int fd[2]; socketpair(AF_UNIX, SOCK_STREAM, 0, fd); - write_task(fd[0], task); - task_spec *result = read_task(fd[1]); + write_message(fd[0], SUBMIT_TASK, task_size(task), task); + int64_t type; + int64_t length; + uint8_t *message; + read_message(fd[1], &type, &length, &message); + task_spec *result = (task_spec *) message; + ASSERT(type == SUBMIT_TASK); ASSERT(memcmp(task, result, task_size(task)) == 0); ASSERT(memcmp(task, result, task_size(result)) == 0); free(task); From e21e9f68df4d4ff28a9fbf52b2abb801afe0ea2c Mon Sep 17 00:00:00 2001 From: Philipp Moritz Date: Thu, 29 Sep 2016 21:12:06 -0700 Subject: [PATCH 19/36] API for task log and scheduled task (#25) * API revision * update * make status a bitmap * update api * tests working * new task log APIs * update APIs * write binary structures to redis * update tests * fix clang-format * Fix formatting. --- common.c | 63 ++------------------------ common.h | 7 +-- state/redis.c | 99 ++++++++++++++++++++++++++++++++--------- state/redis.h | 17 ++++++- state/task_log.h | 41 +++++++++++++++++ state/task_queue.h | 33 -------------- state/task_table.h | 11 ++++- task.c | 106 +++++++++++++++++++------------------------- task.h | 68 ++++++++++++++++++++++++++-- test/common_tests.c | 3 -- test/db_tests.c | 55 ++++++++++++++++++++--- test/task_tests.c | 19 -------- 12 files changed, 307 insertions(+), 215 deletions(-) create mode 100644 state/task_log.h delete mode 100644 state/task_queue.h diff --git a/common.c b/common.c index 53e32fe13..d9fe1e951 100644 --- a/common.c +++ b/common.c @@ -6,6 +6,9 @@ #include #include +const unique_id NIL_ID = {{255, 255, 255, 255, 255, 255, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255, 255, 255}}; + unique_id globally_unique_id(void) { /* Use /dev/urandom for "real" randomness. */ int fd; @@ -31,63 +34,3 @@ char *sha1_to_hex(const unsigned char *sha1, char *buffer) { return buffer; } - -const signed char hexval_table[256] = { - -1, -1, -1, -1, -1, -1, -1, -1, /* 00-07 */ - -1, -1, -1, -1, -1, -1, -1, -1, /* 08-0f */ - -1, -1, -1, -1, -1, -1, -1, -1, /* 10-17 */ - -1, -1, -1, -1, -1, -1, -1, -1, /* 18-1f */ - -1, -1, -1, -1, -1, -1, -1, -1, /* 20-27 */ - -1, -1, -1, -1, -1, -1, -1, -1, /* 28-2f */ - +0, +1, +2, +3, +4, +5, +6, +7, /* 30-37 */ - +8, +9, -1, -1, -1, -1, -1, -1, /* 38-3f */ - -1, 10, 11, 12, 13, 14, 15, -1, /* 40-47 */ - -1, -1, -1, -1, -1, -1, -1, -1, /* 48-4f */ - -1, -1, -1, -1, -1, -1, -1, -1, /* 50-57 */ - -1, -1, -1, -1, -1, -1, -1, -1, /* 58-5f */ - -1, 10, 11, 12, 13, 14, 15, -1, /* 60-67 */ - -1, -1, -1, -1, -1, -1, -1, -1, /* 68-67 */ - -1, -1, -1, -1, -1, -1, -1, -1, /* 70-77 */ - -1, -1, -1, -1, -1, -1, -1, -1, /* 78-7f */ - -1, -1, -1, -1, -1, -1, -1, -1, /* 80-87 */ - -1, -1, -1, -1, -1, -1, -1, -1, /* 88-8f */ - -1, -1, -1, -1, -1, -1, -1, -1, /* 90-97 */ - -1, -1, -1, -1, -1, -1, -1, -1, /* 98-9f */ - -1, -1, -1, -1, -1, -1, -1, -1, /* a0-a7 */ - -1, -1, -1, -1, -1, -1, -1, -1, /* a8-af */ - -1, -1, -1, -1, -1, -1, -1, -1, /* b0-b7 */ - -1, -1, -1, -1, -1, -1, -1, -1, /* b8-bf */ - -1, -1, -1, -1, -1, -1, -1, -1, /* c0-c7 */ - -1, -1, -1, -1, -1, -1, -1, -1, /* c8-cf */ - -1, -1, -1, -1, -1, -1, -1, -1, /* d0-d7 */ - -1, -1, -1, -1, -1, -1, -1, -1, /* d8-df */ - -1, -1, -1, -1, -1, -1, -1, -1, /* e0-e7 */ - -1, -1, -1, -1, -1, -1, -1, -1, /* e8-ef */ - -1, -1, -1, -1, -1, -1, -1, -1, /* f0-f7 */ - -1, -1, -1, -1, -1, -1, -1, -1, /* f8-ff */ -}; - -static inline unsigned int hexval(unsigned char c) { - return hexval_table[c]; -} - -/* - * Convert two consecutive hexadecimal digits into a char. Return a - * negative value on error. Don't run over the end of short strings. - */ -static inline int hex2chr(const char *s) { - int val = hexval(s[0]); - return (val < 0) ? val : (val << 4) | hexval(s[1]); -} - -int hex_to_sha1(const char *hex, unsigned char *sha1) { - int i; - for (i = 0; i < UNIQUE_ID_SIZE; i++) { - int val = hex2chr(hex); - if (val < 0) - return -1; - *sha1++ = val; - hex += 2; - } - return 0; -} diff --git a/common.h b/common.h index 5444739bc..047644cc8 100644 --- a/common.h +++ b/common.h @@ -38,6 +38,8 @@ typedef struct { unsigned char id[UNIQUE_ID_SIZE]; } unique_id; +extern const unique_id NIL_ID; + /* Generate a globally unique ID. */ unique_id globally_unique_id(void); @@ -46,11 +48,6 @@ unique_id globally_unique_id(void); * UNIQUE_ID_SIZE + 1 */ char *sha1_to_hex(const unsigned char *sha1, char *buffer); -/* Convert a hexdecimal string of length 40 to a 20 byte sha1 hash. This - * function assumes that sha1 points to an already allocated char array of size - * UNIQUE_ID_SIZE. */ -int hex_to_sha1(const char *hex, unsigned char *sha1); - typedef unique_id object_id; #endif diff --git a/state/redis.c b/state/redis.c index aa2011d9a..db008cb5d 100644 --- a/state/redis.c +++ b/state/redis.c @@ -9,7 +9,7 @@ #include "common.h" #include "db.h" #include "object_table.h" -#include "task_queue.h" +#include "task_log.h" #include "event_loop.h" #include "redis.h" #include "io.h" @@ -65,22 +65,28 @@ db_handle *db_connect(const char *address, db->client_type = strdup(client_type); db->client_id = num_clients; - db->reading = 0; - db->writing = 0; db->service_cache = NULL; db->sync_context = context; + utarray_new(db->callback_freelist, &ut_ptr_icd); /* Establish async connection */ db->context = redisAsyncConnect(address, port); CHECK_REDIS_CONNECT(redisAsyncContext, db->context, "could not connect to redis %s:%d", address, port); db->context->data = (void *) db; + /* Establish async connection for subscription */ + db->sub_context = redisAsyncConnect(address, port); + CHECK_REDIS_CONNECT(redisAsyncContext, db->sub_context, + "could not connect to redis %s:%d", address, port); + db->sub_context->data = (void *) db; + return db; } void db_disconnect(db_handle *db) { redisFree(db->sync_context); redisAsyncFree(db->context); + redisAsyncFree(db->sub_context); service_cache_entry *e, *tmp; HASH_ITER(hh, db->service_cache, e, tmp) { free(e->addr); @@ -88,18 +94,22 @@ void db_disconnect(db_handle *db) { free(e); } free(db->client_type); + void **p = NULL; + while ((p = (void **) utarray_next(db->callback_freelist, p))) { + free(*p); + } + utarray_free(db->callback_freelist); free(db); } void db_attach(db_handle *db, event_loop *loop) { redisAeAttach(loop, db->context); + redisAeAttach(loop, db->sub_context); } void object_table_add(db_handle *db, unique_id object_id) { - static char hex_object_id[2 * UNIQUE_ID_SIZE + 1]; - sha1_to_hex(&object_id.id[0], &hex_object_id[0]); - redisAsyncCommand(db->context, NULL, NULL, "SADD obj:%s %d", - &hex_object_id[0], db->client_id); + redisAsyncCommand(db->context, NULL, NULL, "SADD obj:%b %d", &object_id.id[0], + UNIQUE_ID_SIZE, db->client_id); if (db->context->err) { LOG_REDIS_ERR(db->context, "could not add object_table entry"); } @@ -148,30 +158,75 @@ void object_table_get_entry(redisAsyncContext *c, void *r, void *privdata) { void object_table_lookup(db_handle *db, object_id object_id, lookup_callback callback) { - static char hex_object_id[2 * UNIQUE_ID_SIZE + 1]; - sha1_to_hex(&object_id.id[0], &hex_object_id[0]); lookup_callback_data *cb_data = malloc(sizeof(lookup_callback_data)); cb_data->callback = callback; cb_data->object_id = object_id; redisAsyncCommand(db->context, object_table_get_entry, cb_data, - "SMEMBERS obj:%s", &hex_object_id[0]); + "SMEMBERS obj:%b", &object_id.id[0], UNIQUE_ID_SIZE); if (db->context->err) { LOG_REDIS_ERR(db->context, "error in object_table lookup"); } } -void task_queue_submit_task(db_handle *db, task_iid task_iid, task_spec *task) { - /* For converting an id to hex, which has double the number - * of bytes compared to the id (+ 1 byte for '\0'). */ - static char hex[2 * UNIQUE_ID_SIZE + 1]; - UT_string *command; - utstring_new(command); - sha1_to_hex(&task_iid.id[0], &hex[0]); - utstring_printf(command, "HMSET queue:%s ", &hex[0]); - print_task(task, command); - redisAsyncCommand(db->context, NULL, NULL, utstring_body(command)); +void task_log_add_task(db_handle *db, task_instance *task_instance) { + task_iid task_iid = *task_instance_id(task_instance); + redisAsyncCommand(db->context, NULL, NULL, "HMSET tasklog:%b 0 %b", + (char *) &task_iid.id[0], UNIQUE_ID_SIZE, + (char *) task_instance, task_instance_size(task_instance)); if (db->context->err) { - LOG_REDIS_ERR(db->context, "error in task_queue submit_task"); + LOG_REDIS_ERR(db->context, "error setting task in task_log_add_task"); + } + node_id node = *task_instance_node(task_instance); + int32_t state = *task_instance_state(task_instance); + redisAsyncCommand(db->context, NULL, NULL, "PUBLISH task_log:%b:%d %b", + (char *) &node.id[0], UNIQUE_ID_SIZE, state, + (char *) task_instance, task_instance_size(task_instance)); + if (db->context->err) { + LOG_REDIS_ERR(db->context, "error publishing task in task_log_add_task"); + } +} + +void task_log_redis_callback(redisAsyncContext *c, + void *reply, + void *privdata) { + redisReply *r = reply; + if (reply == NULL) + return; + CHECK(r->type == REDIS_REPLY_ARRAY); + /* First entry is message type, second is topic, third is payload. */ + CHECK(r->elements > 2); + /* If this condition is true, we got the initial message that acknowledged the + * subscription. */ + if (r->element[2]->str == NULL) { + return; + } + /* Otherwise, parse the task and call the callback. */ + CHECK(privdata); + task_log_callback_data *callback_data = privdata; + task_instance *instance = malloc(r->element[2]->len); + memcpy(instance, r->element[2]->str, r->element[2]->len); + callback_data->callback(instance, callback_data->userdata); + task_instance_free(instance); +} +void task_log_register_callback(db_handle *db, + task_log_callback callback, + node_id node, + int32_t state, + void *userdata) { + task_log_callback_data *callback_data = + malloc(sizeof(task_log_callback_data)); + utarray_push_back(db->callback_freelist, &callback_data); + callback_data->callback = callback; + callback_data->userdata = userdata; + if (memcmp(&node.id[0], &NIL_ID.id[0], UNIQUE_ID_SIZE) == 0) { + redisAsyncCommand(db->sub_context, task_log_redis_callback, callback_data, + "PSUBSCRIBE task_log:*:%d", state); + } else { + redisAsyncCommand(db->sub_context, task_log_redis_callback, callback_data, + "SUBSCRIBE task_log:%b:%d", (char *) &node.id[0], + UNIQUE_ID_SIZE, state); + } + if (db->sub_context->err) { + LOG_REDIS_ERR(db->sub_context, "error in task_log_register_callback"); } - utstring_free(command); } diff --git a/state/redis.h b/state/redis.h index 23ebb2ba6..51479f8f0 100644 --- a/state/redis.h +++ b/state/redis.h @@ -3,10 +3,12 @@ #include "db.h" #include "object_table.h" +#include "task_log.h" #include "hiredis/hiredis.h" #include "hiredis/async.h" #include "uthash.h" +#include "utarray.h" typedef struct { /* Unique ID for this service. */ @@ -17,6 +19,13 @@ typedef struct { UT_hash_handle hh; } service_cache_entry; +typedef struct { + /* The callback that will be called. */ + task_log_callback callback; + /* Userdata associated with the callback. */ + void *userdata; +} task_log_callback_data; + struct db_handle_impl { /* String that identifies this client type. */ char *client_type; @@ -24,8 +33,10 @@ struct db_handle_impl { int64_t client_id; /* Redis context for this global state store connection. */ redisAsyncContext *context; - /* Which events are we processing (read, write)? */ - int reading, writing; + /* Redis context for "subscribe" communication. + * Yes, we need a separate one for that, see + * https://github.com/redis/hiredis/issues/55 */ + redisAsyncContext *sub_context; /* The event loop this global state store connection is part of. */ event_loop *loop; /* Index of the database connection in the event loop */ @@ -35,6 +46,8 @@ struct db_handle_impl { /* Redis context for synchronous connections. * Should only be used very rarely, it is not asynchronous. */ redisContext *sync_context; + /* Data structure for callbacks that needs to be freed. */ + UT_array *callback_freelist; }; typedef struct { diff --git a/state/task_log.h b/state/task_log.h new file mode 100644 index 000000000..acf5dbcd0 --- /dev/null +++ b/state/task_log.h @@ -0,0 +1,41 @@ +#ifndef TASK_LOG_H +#define TASK_LOG_H + +#include "db.h" +#include "task.h" + +/* The task log is a message bus that is used for all communication between + * local and global schedulers (and also persisted to the state database). + * Here are examples of events that are recorded by the task log: + * + * 1) local scheduler writes it when submits a task to the global scheduler; + * 2) global scheduler reads it to get the task submitted by local schedulers; + * 3) global scheduler writes it when assigning the task to a local scheduler; + * 4) local scheduler reads it to get its tasks assigned by global scheduler; + * 5) local scheduler writes it when a task finishes execution; + * 6) global scheduler reads it to get the tasks that have finished; */ + +/* Callback for subscribing to the task log. */ +typedef void (*task_log_callback)(task_instance *task_instance, void *userdata); + +/* Initially add a task instance to the task log. */ +void task_log_add_task(db_handle *db, task_instance *task_instance); + +/* Update task instance in the task log. */ +void task_log_update_task(db_handle *db, + task_iid task_iid, + int32_t state, + node_id node); + +/* Register callback for a certain event. The node specifies the node whose + * events we want to listen to. If you want to listen to all events for this + * node, use state_filter = + * TASK_WAITING | TASK_SCHEDULED | TASK_RUNNING | TASK_DONE. + * If you want to register to updates from all nodes, set node = NIL_ID. */ +void task_log_register_callback(db_handle *db, + task_log_callback callback, + node_id node, + int32_t state_filter, + void *userdata); + +#endif /* TASK_LOG_H */ diff --git a/state/task_queue.h b/state/task_queue.h deleted file mode 100644 index 92968707e..000000000 --- a/state/task_queue.h +++ /dev/null @@ -1,33 +0,0 @@ -#ifndef TASK_QUEUE_H -#define TASK_QUEUE_H - -#include "db.h" -#include "task.h" - -/* The task ID is a deterministic hash of the function ID that - * the task executes and the argument IDs or argument values */ -typedef unique_id task_id; - -/* The task instance ID is a globally unique ID generated which - * identifies this particular execution of the task */ -typedef unique_id task_iid; - -/* The node id is an identifier for the node the task is - * scheduled on */ -typedef unique_id node_id; - -/* Callback for subscribing to the task queue. The only argument this - * callback gets is the task_id of the. */ -typedef void (*task_queue_callback)(task_iid *task_iid, task_spec *task); - -/* Submit task to the global scheduler. */ -void task_queue_submit_task(db_handle *db, task_iid task_iid, task_spec *task); - -/* Submit task to a local scheduler based on the decision made by the global - * scheduler. */ -void task_queue_schedule_task(db_handle *db, task_iid task_iid, node_id node); - -/* Subscribe to task queue. */ -void task_queue_register_callback(db_handle *db, task_queue_callback callback); - -#endif diff --git a/state/task_table.h b/state/task_table.h index 71e879c2c..3a1852522 100644 --- a/state/task_table.h +++ b/state/task_table.h @@ -5,9 +5,16 @@ #include "task.h" /* Add task to the task table, handle errors here. */ -status task_table_add_task(db_handle *db, task_iid task_iid, task_spec *task); +status task_table_add_task(db_handle *db, task_spec *task); + +/* Callback for getting an entry from the task table. Task spec will be freed + * by the system after the callback */ +typedef void (*task_table_callback)(task_spec *task, void *context); /* Get specific task from the task table. */ -status task_table_get_task(db_handle *db, task_iid task_iid, task_spec *task); +status task_table_get_task(db_handle *db, + task_id task_id, + task_table_callback callback, + void *context); #endif /* TASK_TABLE_H */ diff --git a/task.c b/task.c index 30fde8d31..0086eb337 100644 --- a/task.c +++ b/task.c @@ -8,6 +8,8 @@ #include "common.h" #include "io.h" +/* TASK SPECIFICATIONS */ + /* Tasks are stored in a consecutive chunk of memory, the first * sizeof(task_spec) bytes are arranged according to the struct * task_spec. Then there is an array of task_args of length @@ -168,65 +170,49 @@ void print_task(task_spec *spec, UT_string *output) { } } -UT_icd unique_id_icd = {sizeof(unique_id), NULL, NULL, NULL}; +/* TASK INSTANCES */ -task_spec *parse_task(char *task_string, int64_t task_length) { - /* We make one pass through task_string to store all the argument ids - * in "args" and all the return ids in "returns". */ - UT_array *args; - utarray_new(args, &unique_id_icd); - UT_array *returns; - utarray_new(returns, &unique_id_icd); - function_id function_id; - char *cursor = strtok(task_string, " "); - int index = 0; - while (cursor != NULL) { - /* This will be equal to "args" or "returns" depending on whether we - * are processing an argument id or a return id. */ - UT_array *target = NULL; - if (strncmp("fun", cursor, 3) == 0) { - /* Parse function id. */ - CHECK(cursor + 2 * UNIQUE_ID_SIZE + 1 <= task_string + task_length); - cursor = strtok(NULL, " "); - hex_to_sha1(cursor, &function_id.id[0]); - cursor = strtok(NULL, " "); - CHECK(cursor); - continue; - } else if (strncmp("id:", cursor, 3) == 0) { - /* Parse pass by reference argument. */ - sscanf(cursor, "id:%d", &index); - target = args; - } else if (strncmp("val:", cursor, 4) == 0) { - /* Parse pass by value argument. */ - sscanf(cursor, "val:%d", &index); - CHECK(0); /* Not implemented yet */ - } else if (strncmp("ret:", cursor, 4) == 0) { - /* Parse return object reference. */ - sscanf(cursor, "ret:%d", &index); - target = returns; - } - cursor = strtok(NULL, " "); - CHECK(cursor); - if (index >= utarray_len(target)) { - utarray_resize(target, index + 1); - } - object_id *id = (object_id *) utarray_eltptr(target, index); - hex_to_sha1(cursor, &id->id[0]); - cursor = strtok(NULL, " "); - } - /* TODO(pcm): Implement pass by value. */ - /* Now assemble the task specification. */ - task_spec *spec = - alloc_task_spec(function_id, utarray_len(args), utarray_len(returns), 0); - for (int i = 0; i < utarray_len(args); ++i) { - object_id *id = (object_id *) utarray_eltptr(args, i); - task_args_add_ref(spec, *id); - } - for (int i = 0; i < utarray_len(returns); ++i) { - object_id *id = (object_id *) utarray_eltptr(returns, i); - *task_return(spec, i) = *id; - } - utarray_free(args); - utarray_free(returns); - return spec; +struct task_instance_impl { + task_iid iid; + int32_t state; + node_id node; + task_spec spec; +}; + +task_instance *make_task_instance(task_iid task_iid, + task_spec *spec, + int32_t state, + node_id node) { + int64_t size = sizeof(task_instance) - sizeof(task_spec) + task_size(spec); + task_instance *result = malloc(size); + memset(result, 0, size); + result->iid = task_iid; + result->state = state; + result->node = node; + memcpy(&result->spec, spec, task_size(spec)); + return result; +} + +int64_t task_instance_size(task_instance *instance) { + return sizeof(task_instance) - sizeof(task_spec) + task_size(&instance->spec); +} + +task_iid *task_instance_id(task_instance *instance) { + return &instance->iid; +} + +int32_t *task_instance_state(task_instance *instance) { + return &instance->state; +} + +node_id *task_instance_node(task_instance *instance) { + return &instance->node; +} + +task_spec *task_instance_task_spec(task_instance *instance) { + return &instance->spec; +} + +void task_instance_free(task_instance *instance) { + free(instance); } diff --git a/task.h b/task.h index 96c97b80d..9267edb65 100644 --- a/task.h +++ b/task.h @@ -1,7 +1,7 @@ #ifndef TASK_H #define TASK_H -/* This API specifies the task data structure. It is in C so we can +/* This API specifies the task data structures. It is in C so we can * easily construct tasks from other languages like Python. The datastructures * are also defined in such a way that memory is contiguous and all pointers * are relative, so that we can memcpy the datastructure and ship it over the @@ -15,6 +15,24 @@ typedef unique_id function_id; typedef unique_id object_id; +/* The task ID is a deterministic hash of the function ID that + * the task executes and the argument IDs or argument values */ +typedef unique_id task_id; + +/* The task instance ID is a globally unique ID generated which + * identifies this particular execution of the task */ +typedef unique_id task_iid; + +/* The node id is an identifier for the node the task is + * scheduled on */ +typedef unique_id node_id; + +/* + * TASK SPECIFICATIONS: Contain all the information neccessary + * to execute the task (function id, arguments, return object ids). + * + */ + typedef struct task_spec_impl task_spec; /* If argument is passed by value or reference. */ @@ -65,7 +83,51 @@ task_spec *read_task(int fd); /* Print task as a humanly readable string. */ void print_task(task_spec *spec, UT_string *output); -/* Parse task as printed by print_task. */ -task_spec *parse_task(char *task_string, int64_t task_length); +/* + * SCHEDULED TASK: Contains information about a scheduled task: + * the task iid, the task specification and the task status + * (WAITING, SCHEDULED, RUNNING, DONE) and which node the + * task is scheduled on. + * + */ + +/* The scheduling_state can be used as a flag when we are listening + * for an event, for example TASK_WAITING | TASK_SCHEDULED. */ +enum scheduling_state { + TASK_WAITING = 1, + TASK_SCHEDULED = 2, + TASK_RUNNING = 4, + TASK_DONE = 8 +}; + +/* A task instance is one execution of a task specification. + * It has a unique instance id, a state of execution (see scheduling_state) + * and a node it is scheduled on or running on. */ +typedef struct task_instance_impl task_instance; + +/* Allocate and initialize a new task instance. Must be freed with + * scheduled_task_free after use. */ +task_instance *make_task_instance(task_iid task_iid, + task_spec *task, + int32_t state, + node_id node); + +/* Size of task instance structure in bytes. */ +int64_t task_instance_size(task_instance *instance); + +/* Instance ID of the task instance. */ +task_iid *task_instance_id(task_instance *instance); + +/* The scheduling state of the task instance. */ +int32_t *task_instance_state(task_instance *instance); + +/* Node this task instance has been assigned to or is running on. */ +node_id *task_instance_node(task_instance *instance); + +/* Task specification of this task instance. */ +task_spec *task_instance_task_spec(task_instance *instance); + +/* Free this task instance datastructure. */ +void task_instance_free(task_instance *instance); #endif diff --git a/test/common_tests.c b/test/common_tests.c index 3673c335d..47b643039 100644 --- a/test/common_tests.c +++ b/test/common_tests.c @@ -6,11 +6,8 @@ SUITE(common_tests); TEST sha1_test(void) { static char hex[2 * UNIQUE_ID_SIZE + 1]; - static unsigned char id[UNIQUE_ID_SIZE]; unique_id uid = globally_unique_id(); sha1_to_hex(&uid.id[0], &hex[0]); - hex_to_sha1(&hex[0], &id[0]); - ASSERT(memcmp(&uid.id[0], &id[0], 20) == 0); PASS(); } diff --git a/test/db_tests.c b/test/db_tests.c index 99fad7e18..96f16b528 100644 --- a/test/db_tests.c +++ b/test/db_tests.c @@ -6,7 +6,7 @@ #include "test/example_task.h" #include "state/db.h" #include "state/object_table.h" -#include "state/task_queue.h" +#include "state/task_log.h" #include "state/redis.h" #include "task.h" @@ -72,27 +72,70 @@ TEST object_table_lookup_test(void) { PASS(); } -TEST task_queue_test(void) { +void task_log_test_callback(task_instance *instance, void *userdata) { + task_instance *other = userdata; + CHECK(*task_instance_state(instance) == TASK_SCHEDULED); + CHECK(task_instance_size(instance) == task_instance_size(other)); + CHECK(memcmp(instance, other, task_instance_size(instance)) == 0); +} + +TEST task_log_test(void) { event_loop *loop = event_loop_create(); db_handle *db = db_connect("127.0.0.1", 6379, "local_scheduler", "", -1); db_attach(db, loop); - + node_id node = globally_unique_id(); task_spec *task = example_task(); - task_queue_submit_task(db, globally_unique_id(), task); + task_instance *instance = + make_task_instance(globally_unique_id(), task, TASK_SCHEDULED, node); + task_log_register_callback(db, task_log_test_callback, node, TASK_SCHEDULED, + instance); + task_log_add_task(db, instance); event_loop_add_timer(loop, 100, timeout_handler, NULL); event_loop_run(loop); - + task_instance_free(instance); free_task_spec(task); db_disconnect(db); event_loop_destroy(loop); PASS(); } +int num_test_callback_called = 0; + +void task_log_all_test_callback(task_instance *instance, void *userdata) { + num_test_callback_called += 1; +} + +TEST task_log_all_test(void) { + event_loop *loop = event_loop_create(); + db_handle *db = db_connect("127.0.0.1", 6379, "local_scheduler", "", -1); + db_attach(db, loop); + task_spec *task = example_task(); + /* Schedule two tasks on different nodes. */ + task_instance *instance1 = make_task_instance( + globally_unique_id(), task, TASK_SCHEDULED, globally_unique_id()); + task_instance *instance2 = make_task_instance( + globally_unique_id(), task, TASK_SCHEDULED, globally_unique_id()); + task_log_register_callback(db, task_log_all_test_callback, NIL_ID, + TASK_SCHEDULED, NULL); + task_log_add_task(db, instance1); + task_log_add_task(db, instance2); + event_loop_add_timer(loop, 100, timeout_handler, NULL); + event_loop_run(loop); + task_instance_free(instance2); + task_instance_free(instance1); + free_task_spec(task); + db_disconnect(db); + event_loop_destroy(loop); + ASSERT(num_test_callback_called == 2); + PASS(); +} + SUITE(db_tests) { redisContext *context = redisConnect("127.0.0.1", 6379); freeReplyObject(redisCommand(context, "FLUSHALL")); RUN_REDIS_TEST(context, object_table_lookup_test); - RUN_REDIS_TEST(context, task_queue_test); + RUN_REDIS_TEST(context, task_log_test); + RUN_REDIS_TEST(context, task_log_all_test); redisFree(context); } diff --git a/test/task_tests.c b/test/task_tests.c index fcb714737..f72a0e2c2 100644 --- a/test/task_tests.c +++ b/test/task_tests.c @@ -63,28 +63,9 @@ TEST send_task(void) { PASS(); } -TEST print_and_parse_task(void) { - task_spec *task = example_task(); - - UT_string *output; - utstring_new(output); - print_task(task, output); - task_spec *result = parse_task(utstring_body(output), utstring_len(output)); - utstring_free(output); - - ASSERT_EQ(task_size(task), task_size(result)); - ASSERT(memcmp(task, result, task_size(task)) == 0); - - free_task_spec(task); - free_task_spec(result); - - PASS(); -} - SUITE(task_tests) { RUN_TEST(task_test); RUN_TEST(send_task); - RUN_TEST(print_and_parse_task); } GREATEST_MAIN_DEFS(); From f4037ad19f38dc68b186c9338d3f67c9058c556c Mon Sep 17 00:00:00 2001 From: Stephanie Wang Date: Mon, 3 Oct 2016 17:55:57 -0700 Subject: [PATCH 20/36] Plasma fixes (#29) * Allow CHECK to take in a message * Turn off debug statements by default * format arguments to CHECK message * UT list * Socket methods that block until full message is read/written * Address Robert and Philipp's comments * Formatting --- common.h | 10 +- io.c | 102 ++++- test/io_tests.c | 46 +++ thirdparty/utlist.h | 895 ++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 1032 insertions(+), 21 deletions(-) create mode 100644 thirdparty/utlist.h diff --git a/common.h b/common.h index 047644cc8..f09e91cd5 100644 --- a/common.h +++ b/common.h @@ -6,7 +6,7 @@ #include #include -#ifdef NDEBUG +#ifndef RAY_COMMON_DEBUG #define LOG_DEBUG(M, ...) #else #define LOG_DEBUG(M, ...) \ @@ -28,6 +28,14 @@ } \ } while (0); +#define CHECKM(COND, M, ...) \ + do { \ + if (!(COND)) { \ + LOG_ERR("Check failure: %s \n" M, #COND, ##__VA_ARGS__); \ + exit(-1); \ + } \ + } while (0); + #define UNIQUE_ID_SIZE 20 /* Cleanup method for running tests with the greatest library. diff --git a/io.c b/io.c index 16073d32a..32f3ed4b9 100644 --- a/io.c +++ b/io.c @@ -94,6 +94,28 @@ int accept_client(int socket_fd) { return client_fd; } +/** + * Reliably write a sequence of bytes into a file descriptor. This will block + * until one of the following happens: (1) there is an error (2) end of file, + * or (3) all length bytes have been written. + * + * @param fd The file descriptor to write to. + * @param cursor The cursor pointing to the beginning of the bytes to send. + * @param length The size of the bytes sequence to write. + * @return Void. + */ +void write_bytes(int fd, uint8_t *cursor, size_t length) { + ssize_t nbytes = 0; + while (length > 0) { + /* While we haven't written the whole message, write to the file + * descriptor, advance the cursor, and decrease the amount left to write. */ + nbytes = write(fd, cursor, length); + CHECK(nbytes > 0); + cursor += nbytes; + length -= nbytes; + } +} + /** * Write a sequence of bytes on a file descriptor. The bytes should then be read * by read_message. @@ -105,17 +127,49 @@ int accept_client(int socket_fd) { * @return Void. */ void write_message(int fd, int64_t type, int64_t length, uint8_t *bytes) { - ssize_t nbytes = write(fd, (char *) &type, sizeof(type)); - CHECK(nbytes == sizeof(int64_t)); - nbytes = write(fd, (char *) &length, sizeof(length)); - CHECK(nbytes == sizeof(int64_t)); - nbytes = write(fd, (char *) bytes, length * sizeof(char)); - CHECK(nbytes >= 0); + write_bytes(fd, (uint8_t *) &type, sizeof(type)); + write_bytes(fd, (uint8_t *) &length, sizeof(length)); + write_bytes(fd, bytes, length * sizeof(char)); } /** - * Read a sequence of bytes written by write_bytes from a file descriptor. This - * allocates space for the message. + * Reliably read a sequence of bytes from a file descriptor into a buffer. This + * will block until one of the following happens: (1) there is an error (2) end + * of file, or (3) all length bytes have been written. + * + * @note The buffer pointed to by cursor must already have length number of + * bytes allocated before calling this method. + * + * @param fd The file descriptor to read from. + * @param cursor The cursor pointing to the beginning of the buffer. + * @param length The size of the byte sequence to read. + * @return Void. + */ +int read_bytes(int fd, uint8_t *cursor, size_t length) { + ssize_t nbytes = 0; + while (length > 0) { + /* While we haven't read the whole message, read from the file descriptor, + * advance the cursor, and decrease the amount left to read. */ + nbytes = read(fd, cursor, length); + if (nbytes < 0) { + if (errno == EAGAIN || errno == EWOULDBLOCK) { + continue; + } + /* Force an exit if there was any other type of error. */ + CHECK(nbytes < 0); + } + if (nbytes == 0) { + return -1; + } + cursor += nbytes; + length -= nbytes; + } + return 0; +} + +/** + * Read a sequence of bytes written by write_message from a file descriptor. + * This allocates space for the message. * * @note The caller must free the memory. * @@ -130,20 +184,28 @@ void write_message(int fd, int64_t type, int64_t length, uint8_t *bytes) { * @return Void. */ void read_message(int fd, int64_t *type, int64_t *length, uint8_t **bytes) { - ssize_t nbytes = read(fd, type, sizeof(int64_t)); - CHECK(nbytes >= 0); - /* Handle the case in which the socket is closed. */ - if (nbytes == 0) { - *type = DISCONNECT_CLIENT; - *length = 0; - *bytes = NULL; - return; + int closed = read_bytes(fd, (uint8_t *) type, sizeof(int64_t)); + if (closed) { + goto disconnected; + } + closed = read_bytes(fd, (uint8_t *) length, sizeof(int64_t)); + if (closed) { + goto disconnected; } - nbytes = read(fd, length, sizeof(int64_t)); - CHECK(nbytes == sizeof(int64_t)); *bytes = malloc(*length * sizeof(uint8_t)); - nbytes = read(fd, *bytes, *length); - CHECK(nbytes >= 0); + closed = read_bytes(fd, *bytes, *length); + if (closed) { + free(*bytes); + goto disconnected; + } + return; + +disconnected: + /* Handle the case in which the socket is closed. */ + *type = DISCONNECT_CLIENT; + *length = 0; + *bytes = NULL; + return; } /* Write a null-terminated string to a file descriptor. */ diff --git a/test/io_tests.c b/test/io_tests.c index b73207326..56ebf0607 100644 --- a/test/io_tests.c +++ b/test/io_tests.c @@ -5,6 +5,7 @@ #include #include "io.h" +#include "utstring.h" SUITE(io_tests); @@ -47,8 +48,53 @@ TEST ipc_socket_test(void) { PASS(); } +TEST long_ipc_socket_test(void) { + const char *socket_pathname = "long-test-socket"; + int socket_fd = bind_ipc_sock(socket_pathname); + ASSERT(socket_fd >= 0); + + UT_string *test_string; + utstring_new(test_string); + for (int i = 0; i < 10000; i++) { + utstring_printf(test_string, "hello world "); + } + char *test_bytes = "another string"; + pid_t pid = fork(); + if (pid == 0) { + close(socket_fd); + socket_fd = connect_ipc_sock(socket_pathname); + ASSERT(socket_fd >= 0); + write_log_message(socket_fd, utstring_body(test_string)); + write_message(socket_fd, LOG_MESSAGE, strlen(test_bytes), + (uint8_t *) test_bytes); + close(socket_fd); + exit(0); + } else { + int client_fd = accept_client(socket_fd); + ASSERT(client_fd >= 0); + char *message = read_log_message(client_fd); + ASSERT(message != NULL); + ASSERT_STR_EQ(utstring_body(test_string), message); + free(message); + int64_t type; + int64_t len; + uint8_t *bytes; + read_message(client_fd, &type, &len, &bytes); + ASSERT(type == LOG_MESSAGE); + ASSERT(memcmp(test_bytes, bytes, len) == 0); + free(bytes); + close(client_fd); + close(socket_fd); + unlink(socket_pathname); + } + + utstring_free(test_string); + PASS(); +} + SUITE(io_tests) { RUN_TEST(ipc_socket_test); + RUN_TEST(long_ipc_socket_test); } GREATEST_MAIN_DEFS(); diff --git a/thirdparty/utlist.h b/thirdparty/utlist.h new file mode 100644 index 000000000..9b5534ffb --- /dev/null +++ b/thirdparty/utlist.h @@ -0,0 +1,895 @@ +/* +Copyright (c) 2007-2016, Troy D. Hanson http://troydhanson.github.com/uthash/ +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + + * Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS +IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED +TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A +PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER +OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, +EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, +PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR +PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF +LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING +NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +*/ + +#ifndef UTLIST_H +#define UTLIST_H + +#define UTLIST_VERSION 2.0.1 + +#include + +/* + * This file contains macros to manipulate singly and doubly-linked lists. + * + * 1. LL_ macros: singly-linked lists. + * 2. DL_ macros: doubly-linked lists. + * 3. CDL_ macros: circular doubly-linked lists. + * + * To use singly-linked lists, your structure must have a "next" pointer. + * To use doubly-linked lists, your structure must "prev" and "next" pointers. + * Either way, the pointer to the head of the list must be initialized to NULL. + * + * ----------------.EXAMPLE ------------------------- + * struct item { + * int id; + * struct item *prev, *next; + * } + * + * struct item *list = NULL: + * + * int main() { + * struct item *item; + * ... allocate and populate item ... + * DL_APPEND(list, item); + * } + * -------------------------------------------------- + * + * For doubly-linked lists, the append and delete macros are O(1) + * For singly-linked lists, append and delete are O(n) but prepend is O(1) + * The sort macro is O(n log(n)) for all types of single/double/circular lists. + */ + +/* These macros use decltype or the earlier __typeof GNU extension. + As decltype is only available in newer compilers (VS2010 or gcc 4.3+ + when compiling c++ code), this code uses whatever method is needed + or, for VS2008 where neither is available, uses casting workarounds. */ +#ifdef _MSC_VER /* MS compiler */ +#if _MSC_VER >= 1600 && defined(__cplusplus) /* VS2010 or newer in C++ mode */ +#define LDECLTYPE(x) decltype(x) +#else /* VS2008 or older (or VS2010 in C mode) */ +#define NO_DECLTYPE +#endif +#elif defined(__ICCARM__) +#define NO_DECLTYPE +#else /* GNU, Sun and other compilers */ +#define LDECLTYPE(x) __typeof(x) +#endif + +/* for VS2008 we use some workarounds to get around the lack of decltype, + * namely, we always reassign our tmp variable to the list head if we need + * to dereference its prev/next pointers, and save/restore the real head.*/ +#ifdef NO_DECLTYPE +#define IF_NO_DECLTYPE(x) x +#define LDECLTYPE(x) char* +#define _SV(elt,list) _tmp = (char*)(list); {char **_alias = (char**)&(list); *_alias = (elt); } +#define _NEXT(elt,list,next) ((char*)((list)->next)) +#define _NEXTASGN(elt,list,to,next) { char **_alias = (char**)&((list)->next); *_alias=(char*)(to); } +/* #define _PREV(elt,list,prev) ((char*)((list)->prev)) */ +#define _PREVASGN(elt,list,to,prev) { char **_alias = (char**)&((list)->prev); *_alias=(char*)(to); } +#define _RS(list) { char **_alias = (char**)&(list); *_alias=_tmp; } +#define _CASTASGN(a,b) { char **_alias = (char**)&(a); *_alias=(char*)(b); } +#else +#define IF_NO_DECLTYPE(x) +#define _SV(elt,list) +#define _NEXT(elt,list,next) ((elt)->next) +#define _NEXTASGN(elt,list,to,next) ((elt)->next)=(to) +/* #define _PREV(elt,list,prev) ((elt)->prev) */ +#define _PREVASGN(elt,list,to,prev) ((elt)->prev)=(to) +#define _RS(list) +#define _CASTASGN(a,b) (a)=(b) +#endif + +/****************************************************************************** + * The sort macro is an adaptation of Simon Tatham's O(n log(n)) mergesort * + * Unwieldy variable names used here to avoid shadowing passed-in variables. * + *****************************************************************************/ +#define LL_SORT(list, cmp) \ + LL_SORT2(list, cmp, next) + +#define LL_SORT2(list, cmp, next) \ +do { \ + LDECLTYPE(list) _ls_p; \ + LDECLTYPE(list) _ls_q; \ + LDECLTYPE(list) _ls_e; \ + LDECLTYPE(list) _ls_tail; \ + IF_NO_DECLTYPE(LDECLTYPE(list) _tmp;) \ + int _ls_insize, _ls_nmerges, _ls_psize, _ls_qsize, _ls_i, _ls_looping; \ + if (list) { \ + _ls_insize = 1; \ + _ls_looping = 1; \ + while (_ls_looping) { \ + _CASTASGN(_ls_p,list); \ + (list) = NULL; \ + _ls_tail = NULL; \ + _ls_nmerges = 0; \ + while (_ls_p) { \ + _ls_nmerges++; \ + _ls_q = _ls_p; \ + _ls_psize = 0; \ + for (_ls_i = 0; _ls_i < _ls_insize; _ls_i++) { \ + _ls_psize++; \ + _SV(_ls_q,list); _ls_q = _NEXT(_ls_q,list,next); _RS(list); \ + if (!_ls_q) break; \ + } \ + _ls_qsize = _ls_insize; \ + while (_ls_psize > 0 || (_ls_qsize > 0 && _ls_q)) { \ + if (_ls_psize == 0) { \ + _ls_e = _ls_q; _SV(_ls_q,list); _ls_q = \ + _NEXT(_ls_q,list,next); _RS(list); _ls_qsize--; \ + } else if (_ls_qsize == 0 || !_ls_q) { \ + _ls_e = _ls_p; _SV(_ls_p,list); _ls_p = \ + _NEXT(_ls_p,list,next); _RS(list); _ls_psize--; \ + } else if (cmp(_ls_p,_ls_q) <= 0) { \ + _ls_e = _ls_p; _SV(_ls_p,list); _ls_p = \ + _NEXT(_ls_p,list,next); _RS(list); _ls_psize--; \ + } else { \ + _ls_e = _ls_q; _SV(_ls_q,list); _ls_q = \ + _NEXT(_ls_q,list,next); _RS(list); _ls_qsize--; \ + } \ + if (_ls_tail) { \ + _SV(_ls_tail,list); _NEXTASGN(_ls_tail,list,_ls_e,next); _RS(list); \ + } else { \ + _CASTASGN(list,_ls_e); \ + } \ + _ls_tail = _ls_e; \ + } \ + _ls_p = _ls_q; \ + } \ + if (_ls_tail) { \ + _SV(_ls_tail,list); _NEXTASGN(_ls_tail,list,NULL,next); _RS(list); \ + } \ + if (_ls_nmerges <= 1) { \ + _ls_looping=0; \ + } \ + _ls_insize *= 2; \ + } \ + } \ +} while (0) + + +#define DL_SORT(list, cmp) \ + DL_SORT2(list, cmp, prev, next) + +#define DL_SORT2(list, cmp, prev, next) \ +do { \ + LDECLTYPE(list) _ls_p; \ + LDECLTYPE(list) _ls_q; \ + LDECLTYPE(list) _ls_e; \ + LDECLTYPE(list) _ls_tail; \ + IF_NO_DECLTYPE(LDECLTYPE(list) _tmp;) \ + int _ls_insize, _ls_nmerges, _ls_psize, _ls_qsize, _ls_i, _ls_looping; \ + if (list) { \ + _ls_insize = 1; \ + _ls_looping = 1; \ + while (_ls_looping) { \ + _CASTASGN(_ls_p,list); \ + (list) = NULL; \ + _ls_tail = NULL; \ + _ls_nmerges = 0; \ + while (_ls_p) { \ + _ls_nmerges++; \ + _ls_q = _ls_p; \ + _ls_psize = 0; \ + for (_ls_i = 0; _ls_i < _ls_insize; _ls_i++) { \ + _ls_psize++; \ + _SV(_ls_q,list); _ls_q = _NEXT(_ls_q,list,next); _RS(list); \ + if (!_ls_q) break; \ + } \ + _ls_qsize = _ls_insize; \ + while ((_ls_psize > 0) || ((_ls_qsize > 0) && _ls_q)) { \ + if (_ls_psize == 0) { \ + _ls_e = _ls_q; _SV(_ls_q,list); _ls_q = \ + _NEXT(_ls_q,list,next); _RS(list); _ls_qsize--; \ + } else if ((_ls_qsize == 0) || (!_ls_q)) { \ + _ls_e = _ls_p; _SV(_ls_p,list); _ls_p = \ + _NEXT(_ls_p,list,next); _RS(list); _ls_psize--; \ + } else if (cmp(_ls_p,_ls_q) <= 0) { \ + _ls_e = _ls_p; _SV(_ls_p,list); _ls_p = \ + _NEXT(_ls_p,list,next); _RS(list); _ls_psize--; \ + } else { \ + _ls_e = _ls_q; _SV(_ls_q,list); _ls_q = \ + _NEXT(_ls_q,list,next); _RS(list); _ls_qsize--; \ + } \ + if (_ls_tail) { \ + _SV(_ls_tail,list); _NEXTASGN(_ls_tail,list,_ls_e,next); _RS(list); \ + } else { \ + _CASTASGN(list,_ls_e); \ + } \ + _SV(_ls_e,list); _PREVASGN(_ls_e,list,_ls_tail,prev); _RS(list); \ + _ls_tail = _ls_e; \ + } \ + _ls_p = _ls_q; \ + } \ + _CASTASGN((list)->prev, _ls_tail); \ + _SV(_ls_tail,list); _NEXTASGN(_ls_tail,list,NULL,next); _RS(list); \ + if (_ls_nmerges <= 1) { \ + _ls_looping=0; \ + } \ + _ls_insize *= 2; \ + } \ + } \ +} while (0) + +#define CDL_SORT(list, cmp) \ + CDL_SORT2(list, cmp, prev, next) + +#define CDL_SORT2(list, cmp, prev, next) \ +do { \ + LDECLTYPE(list) _ls_p; \ + LDECLTYPE(list) _ls_q; \ + LDECLTYPE(list) _ls_e; \ + LDECLTYPE(list) _ls_tail; \ + LDECLTYPE(list) _ls_oldhead; \ + LDECLTYPE(list) _tmp; \ + int _ls_insize, _ls_nmerges, _ls_psize, _ls_qsize, _ls_i, _ls_looping; \ + if (list) { \ + _ls_insize = 1; \ + _ls_looping = 1; \ + while (_ls_looping) { \ + _CASTASGN(_ls_p,list); \ + _CASTASGN(_ls_oldhead,list); \ + (list) = NULL; \ + _ls_tail = NULL; \ + _ls_nmerges = 0; \ + while (_ls_p) { \ + _ls_nmerges++; \ + _ls_q = _ls_p; \ + _ls_psize = 0; \ + for (_ls_i = 0; _ls_i < _ls_insize; _ls_i++) { \ + _ls_psize++; \ + _SV(_ls_q,list); \ + if (_NEXT(_ls_q,list,next) == _ls_oldhead) { \ + _ls_q = NULL; \ + } else { \ + _ls_q = _NEXT(_ls_q,list,next); \ + } \ + _RS(list); \ + if (!_ls_q) break; \ + } \ + _ls_qsize = _ls_insize; \ + while (_ls_psize > 0 || (_ls_qsize > 0 && _ls_q)) { \ + if (_ls_psize == 0) { \ + _ls_e = _ls_q; _SV(_ls_q,list); _ls_q = \ + _NEXT(_ls_q,list,next); _RS(list); _ls_qsize--; \ + if (_ls_q == _ls_oldhead) { _ls_q = NULL; } \ + } else if (_ls_qsize == 0 || !_ls_q) { \ + _ls_e = _ls_p; _SV(_ls_p,list); _ls_p = \ + _NEXT(_ls_p,list,next); _RS(list); _ls_psize--; \ + if (_ls_p == _ls_oldhead) { _ls_p = NULL; } \ + } else if (cmp(_ls_p,_ls_q) <= 0) { \ + _ls_e = _ls_p; _SV(_ls_p,list); _ls_p = \ + _NEXT(_ls_p,list,next); _RS(list); _ls_psize--; \ + if (_ls_p == _ls_oldhead) { _ls_p = NULL; } \ + } else { \ + _ls_e = _ls_q; _SV(_ls_q,list); _ls_q = \ + _NEXT(_ls_q,list,next); _RS(list); _ls_qsize--; \ + if (_ls_q == _ls_oldhead) { _ls_q = NULL; } \ + } \ + if (_ls_tail) { \ + _SV(_ls_tail,list); _NEXTASGN(_ls_tail,list,_ls_e,next); _RS(list); \ + } else { \ + _CASTASGN(list,_ls_e); \ + } \ + _SV(_ls_e,list); _PREVASGN(_ls_e,list,_ls_tail,prev); _RS(list); \ + _ls_tail = _ls_e; \ + } \ + _ls_p = _ls_q; \ + } \ + _CASTASGN((list)->prev,_ls_tail); \ + _CASTASGN(_tmp,list); \ + _SV(_ls_tail,list); _NEXTASGN(_ls_tail,list,_tmp,next); _RS(list); \ + if (_ls_nmerges <= 1) { \ + _ls_looping=0; \ + } \ + _ls_insize *= 2; \ + } \ + } \ +} while (0) + +/****************************************************************************** + * singly linked list macros (non-circular) * + *****************************************************************************/ +#define LL_PREPEND(head,add) \ + LL_PREPEND2(head,add,next) + +#define LL_PREPEND2(head,add,next) \ +do { \ + (add)->next = (head); \ + (head) = (add); \ +} while (0) + +#define LL_CONCAT(head1,head2) \ + LL_CONCAT2(head1,head2,next) + +#define LL_CONCAT2(head1,head2,next) \ +do { \ + LDECLTYPE(head1) _tmp; \ + if (head1) { \ + _tmp = (head1); \ + while (_tmp->next) { _tmp = _tmp->next; } \ + _tmp->next=(head2); \ + } else { \ + (head1)=(head2); \ + } \ +} while (0) + +#define LL_APPEND(head,add) \ + LL_APPEND2(head,add,next) + +#define LL_APPEND2(head,add,next) \ +do { \ + LDECLTYPE(head) _tmp; \ + (add)->next=NULL; \ + if (head) { \ + _tmp = (head); \ + while (_tmp->next) { _tmp = _tmp->next; } \ + _tmp->next=(add); \ + } else { \ + (head)=(add); \ + } \ +} while (0) + +#define LL_DELETE(head,del) \ + LL_DELETE2(head,del,next) + +#define LL_DELETE2(head,del,next) \ +do { \ + LDECLTYPE(head) _tmp; \ + if ((head) == (del)) { \ + (head)=(head)->next; \ + } else { \ + _tmp = (head); \ + while (_tmp->next && (_tmp->next != (del))) { \ + _tmp = _tmp->next; \ + } \ + if (_tmp->next) { \ + _tmp->next = (del)->next; \ + } \ + } \ +} while (0) + +#define LL_COUNT(head,el,counter) \ + LL_COUNT2(head,el,counter,next) \ + +#define LL_COUNT2(head,el,counter,next) \ +do { \ + (counter) = 0; \ + LL_FOREACH2(head,el,next) { ++(counter); } \ +} while (0) + +#define LL_FOREACH(head,el) \ + LL_FOREACH2(head,el,next) + +#define LL_FOREACH2(head,el,next) \ + for ((el) = (head); el; (el) = (el)->next) + +#define LL_FOREACH_SAFE(head,el,tmp) \ + LL_FOREACH_SAFE2(head,el,tmp,next) + +#define LL_FOREACH_SAFE2(head,el,tmp,next) \ + for ((el) = (head); (el) && ((tmp) = (el)->next, 1); (el) = (tmp)) + +#define LL_SEARCH_SCALAR(head,out,field,val) \ + LL_SEARCH_SCALAR2(head,out,field,val,next) + +#define LL_SEARCH_SCALAR2(head,out,field,val,next) \ +do { \ + LL_FOREACH2(head,out,next) { \ + if ((out)->field == (val)) break; \ + } \ +} while (0) + +#define LL_SEARCH(head,out,elt,cmp) \ + LL_SEARCH2(head,out,elt,cmp,next) + +#define LL_SEARCH2(head,out,elt,cmp,next) \ +do { \ + LL_FOREACH2(head,out,next) { \ + if ((cmp(out,elt))==0) break; \ + } \ +} while (0) + +#define LL_REPLACE_ELEM2(head, el, add, next) \ +do { \ + LDECLTYPE(head) _tmp; \ + assert((head) != NULL); \ + assert((el) != NULL); \ + assert((add) != NULL); \ + (add)->next = (el)->next; \ + if ((head) == (el)) { \ + (head) = (add); \ + } else { \ + _tmp = (head); \ + while (_tmp->next && (_tmp->next != (el))) { \ + _tmp = _tmp->next; \ + } \ + if (_tmp->next) { \ + _tmp->next = (add); \ + } \ + } \ +} while (0) + +#define LL_REPLACE_ELEM(head, el, add) \ + LL_REPLACE_ELEM2(head, el, add, next) + +#define LL_PREPEND_ELEM2(head, el, add, next) \ +do { \ + if (el) { \ + LDECLTYPE(head) _tmp; \ + assert((head) != NULL); \ + assert((add) != NULL); \ + (add)->next = (el); \ + if ((head) == (el)) { \ + (head) = (add); \ + } else { \ + _tmp = (head); \ + while (_tmp->next && (_tmp->next != (el))) { \ + _tmp = _tmp->next; \ + } \ + if (_tmp->next) { \ + _tmp->next = (add); \ + } \ + } \ + } else { \ + LL_APPEND2(head, add, next); \ + } \ +} while (0) \ + +#define LL_PREPEND_ELEM(head, el, add) \ + LL_PREPEND_ELEM2(head, el, add, next) + +#define LL_APPEND_ELEM2(head, el, add, next) \ +do { \ + if (el) { \ + assert((head) != NULL); \ + assert((add) != NULL); \ + (add)->next = (el)->next; \ + (el)->next = (add); \ + } else { \ + LL_PREPEND2(head, add, next); \ + } \ +} while (0) \ + +#define LL_APPEND_ELEM(head, el, add) \ + LL_APPEND_ELEM2(head, el, add, next) + +#ifdef NO_DECLTYPE +/* Here are VS2008 / NO_DECLTYPE replacements for a few functions */ + +#undef LL_CONCAT2 +#define LL_CONCAT2(head1,head2,next) \ +do { \ + char *_tmp; \ + if (head1) { \ + _tmp = (char*)(head1); \ + while ((head1)->next) { (head1) = (head1)->next; } \ + (head1)->next = (head2); \ + _RS(head1); \ + } else { \ + (head1)=(head2); \ + } \ +} while (0) + +#undef LL_APPEND2 +#define LL_APPEND2(head,add,next) \ +do { \ + if (head) { \ + (add)->next = head; /* use add->next as a temp variable */ \ + while ((add)->next->next) { (add)->next = (add)->next->next; } \ + (add)->next->next=(add); \ + } else { \ + (head)=(add); \ + } \ + (add)->next=NULL; \ +} while (0) + +#undef LL_DELETE2 +#define LL_DELETE2(head,del,next) \ +do { \ + if ((head) == (del)) { \ + (head)=(head)->next; \ + } else { \ + char *_tmp = (char*)(head); \ + while ((head)->next && ((head)->next != (del))) { \ + (head) = (head)->next; \ + } \ + if ((head)->next) { \ + (head)->next = ((del)->next); \ + } \ + _RS(head); \ + } \ +} while (0) + +#undef LL_REPLACE_ELEM2 +#define LL_REPLACE_ELEM2(head, el, add, next) \ +do { \ + assert((head) != NULL); \ + assert((el) != NULL); \ + assert((add) != NULL); \ + if ((head) == (el)) { \ + (head) = (add); \ + } else { \ + (add)->next = head; \ + while ((add)->next->next && ((add)->next->next != (el))) { \ + (add)->next = (add)->next->next; \ + } \ + if ((add)->next->next) { \ + (add)->next->next = (add); \ + } \ + } \ + (add)->next = (el)->next; \ +} while (0) + +#undef LL_PREPEND_ELEM2 +#define LL_PREPEND_ELEM2(head, el, add, next) \ +do { \ + if (el) { \ + assert((head) != NULL); \ + assert((add) != NULL); \ + if ((head) == (el)) { \ + (head) = (add); \ + } else { \ + (add)->next = (head); \ + while ((add)->next->next && ((add)->next->next != (el))) { \ + (add)->next = (add)->next->next; \ + } \ + if ((add)->next->next) { \ + (add)->next->next = (add); \ + } \ + } \ + (add)->next = (el); \ + } else { \ + LL_APPEND2(head, add, next); \ + } \ +} while (0) \ + +#endif /* NO_DECLTYPE */ + +/****************************************************************************** + * doubly linked list macros (non-circular) * + *****************************************************************************/ +#define DL_PREPEND(head,add) \ + DL_PREPEND2(head,add,prev,next) + +#define DL_PREPEND2(head,add,prev,next) \ +do { \ + (add)->next = (head); \ + if (head) { \ + (add)->prev = (head)->prev; \ + (head)->prev = (add); \ + } else { \ + (add)->prev = (add); \ + } \ + (head) = (add); \ +} while (0) + +#define DL_APPEND(head,add) \ + DL_APPEND2(head,add,prev,next) + +#define DL_APPEND2(head,add,prev,next) \ +do { \ + if (head) { \ + (add)->prev = (head)->prev; \ + (head)->prev->next = (add); \ + (head)->prev = (add); \ + (add)->next = NULL; \ + } else { \ + (head)=(add); \ + (head)->prev = (head); \ + (head)->next = NULL; \ + } \ +} while (0) + +#define DL_CONCAT(head1,head2) \ + DL_CONCAT2(head1,head2,prev,next) + +#define DL_CONCAT2(head1,head2,prev,next) \ +do { \ + LDECLTYPE(head1) _tmp; \ + if (head2) { \ + if (head1) { \ + _CASTASGN(_tmp, (head2)->prev); \ + (head2)->prev = (head1)->prev; \ + (head1)->prev->next = (head2); \ + _CASTASGN((head1)->prev, _tmp); \ + } else { \ + (head1)=(head2); \ + } \ + } \ +} while (0) + +#define DL_DELETE(head,del) \ + DL_DELETE2(head,del,prev,next) + +#define DL_DELETE2(head,del,prev,next) \ +do { \ + assert((del)->prev != NULL); \ + if ((del)->prev == (del)) { \ + (head)=NULL; \ + } else if ((del)==(head)) { \ + (del)->next->prev = (del)->prev; \ + (head) = (del)->next; \ + } else { \ + (del)->prev->next = (del)->next; \ + if ((del)->next) { \ + (del)->next->prev = (del)->prev; \ + } else { \ + (head)->prev = (del)->prev; \ + } \ + } \ +} while (0) + +#define DL_COUNT(head,el,counter) \ + DL_COUNT2(head,el,counter,next) \ + +#define DL_COUNT2(head,el,counter,next) \ +do { \ + (counter) = 0; \ + DL_FOREACH2(head,el,next) { ++(counter); } \ +} while (0) + +#define DL_FOREACH(head,el) \ + DL_FOREACH2(head,el,next) + +#define DL_FOREACH2(head,el,next) \ + for ((el) = (head); el; (el) = (el)->next) + +/* this version is safe for deleting the elements during iteration */ +#define DL_FOREACH_SAFE(head,el,tmp) \ + DL_FOREACH_SAFE2(head,el,tmp,next) + +#define DL_FOREACH_SAFE2(head,el,tmp,next) \ + for ((el) = (head); (el) && ((tmp) = (el)->next, 1); (el) = (tmp)) + +/* these are identical to their singly-linked list counterparts */ +#define DL_SEARCH_SCALAR LL_SEARCH_SCALAR +#define DL_SEARCH LL_SEARCH +#define DL_SEARCH_SCALAR2 LL_SEARCH_SCALAR2 +#define DL_SEARCH2 LL_SEARCH2 + +#define DL_REPLACE_ELEM2(head, el, add, prev, next) \ +do { \ + assert((head) != NULL); \ + assert((el) != NULL); \ + assert((add) != NULL); \ + if ((head) == (el)) { \ + (head) = (add); \ + (add)->next = (el)->next; \ + if ((el)->next == NULL) { \ + (add)->prev = (add); \ + } else { \ + (add)->prev = (el)->prev; \ + (add)->next->prev = (add); \ + } \ + } else { \ + (add)->next = (el)->next; \ + (add)->prev = (el)->prev; \ + (add)->prev->next = (add); \ + if ((el)->next == NULL) { \ + (head)->prev = (add); \ + } else { \ + (add)->next->prev = (add); \ + } \ + } \ +} while (0) + +#define DL_REPLACE_ELEM(head, el, add) \ + DL_REPLACE_ELEM2(head, el, add, prev, next) + +#define DL_PREPEND_ELEM2(head, el, add, prev, next) \ +do { \ + if (el) { \ + assert((head) != NULL); \ + assert((add) != NULL); \ + (add)->next = (el); \ + (add)->prev = (el)->prev; \ + (el)->prev = (add); \ + if ((head) == (el)) { \ + (head) = (add); \ + } else { \ + (add)->prev->next = (add); \ + } \ + } else { \ + DL_APPEND2(head, add, prev, next); \ + } \ +} while (0) \ + +#define DL_PREPEND_ELEM(head, el, add) \ + DL_PREPEND_ELEM2(head, el, add, prev, next) + +#define DL_APPEND_ELEM2(head, el, add, prev, next) \ +do { \ + if (el) { \ + assert((head) != NULL); \ + assert((add) != NULL); \ + (add)->next = (el)->next; \ + (add)->prev = (el); \ + (el)->next = (add); \ + if ((add)->next) { \ + (add)->next->prev = (add); \ + } else { \ + (head)->prev = (add); \ + } \ + } else { \ + DL_PREPEND2(head, add, prev, next); \ + } \ +} while (0) \ + +#define DL_APPEND_ELEM(head, el, add) \ + DL_APPEND_ELEM2(head, el, add, prev, next) + +/****************************************************************************** + * circular doubly linked list macros * + *****************************************************************************/ +#define CDL_APPEND(head,add) \ + CDL_APPEND2(head,add,prev,next) + +#define CDL_APPEND2(head,add,prev,next) \ +do { \ + if (head) { \ + (add)->prev = (head)->prev; \ + (add)->next = (head); \ + (head)->prev = (add); \ + (add)->prev->next = (add); \ + } else { \ + (add)->prev = (add); \ + (add)->next = (add); \ + (head) = (add); \ + } \ +} while (0) + +#define CDL_PREPEND(head,add) \ + CDL_PREPEND2(head,add,prev,next) + +#define CDL_PREPEND2(head,add,prev,next) \ +do { \ + if (head) { \ + (add)->prev = (head)->prev; \ + (add)->next = (head); \ + (head)->prev = (add); \ + (add)->prev->next = (add); \ + } else { \ + (add)->prev = (add); \ + (add)->next = (add); \ + } \ + (head) = (add); \ +} while (0) + +#define CDL_DELETE(head,del) \ + CDL_DELETE2(head,del,prev,next) + +#define CDL_DELETE2(head,del,prev,next) \ +do { \ + if (((head)==(del)) && ((head)->next == (head))) { \ + (head) = NULL; \ + } else { \ + (del)->next->prev = (del)->prev; \ + (del)->prev->next = (del)->next; \ + if ((del) == (head)) (head)=(del)->next; \ + } \ +} while (0) + +#define CDL_COUNT(head,el,counter) \ + CDL_COUNT2(head,el,counter,next) \ + +#define CDL_COUNT2(head, el, counter,next) \ +do { \ + (counter) = 0; \ + CDL_FOREACH2(head,el,next) { ++(counter); } \ +} while (0) + +#define CDL_FOREACH(head,el) \ + CDL_FOREACH2(head,el,next) + +#define CDL_FOREACH2(head,el,next) \ + for ((el)=(head);el;(el)=(((el)->next==(head)) ? NULL : (el)->next)) + +#define CDL_FOREACH_SAFE(head,el,tmp1,tmp2) \ + CDL_FOREACH_SAFE2(head,el,tmp1,tmp2,prev,next) + +#define CDL_FOREACH_SAFE2(head,el,tmp1,tmp2,prev,next) \ + for ((el) = (head), (tmp1) = (head) ? (head)->prev : NULL; \ + (el) && ((tmp2) = (el)->next, 1); \ + (el) = ((el) == (tmp1) ? NULL : (tmp2))) + +#define CDL_SEARCH_SCALAR(head,out,field,val) \ + CDL_SEARCH_SCALAR2(head,out,field,val,next) + +#define CDL_SEARCH_SCALAR2(head,out,field,val,next) \ +do { \ + CDL_FOREACH2(head,out,next) { \ + if ((out)->field == (val)) break; \ + } \ +} while (0) + +#define CDL_SEARCH(head,out,elt,cmp) \ + CDL_SEARCH2(head,out,elt,cmp,next) + +#define CDL_SEARCH2(head,out,elt,cmp,next) \ +do { \ + CDL_FOREACH2(head,out,next) { \ + if ((cmp(out,elt))==0) break; \ + } \ +} while (0) + +#define CDL_REPLACE_ELEM2(head, el, add, prev, next) \ +do { \ + assert((head) != NULL); \ + assert((el) != NULL); \ + assert((add) != NULL); \ + if ((el)->next == (el)) { \ + (add)->next = (add); \ + (add)->prev = (add); \ + (head) = (add); \ + } else { \ + (add)->next = (el)->next; \ + (add)->prev = (el)->prev; \ + (add)->next->prev = (add); \ + (add)->prev->next = (add); \ + if ((head) == (el)) { \ + (head) = (add); \ + } \ + } \ +} while (0) + +#define CDL_REPLACE_ELEM(head, el, add) \ + CDL_REPLACE_ELEM2(head, el, add, prev, next) + +#define CDL_PREPEND_ELEM2(head, el, add, prev, next) \ +do { \ + if (el) { \ + assert((head) != NULL); \ + assert((add) != NULL); \ + (add)->next = (el); \ + (add)->prev = (el)->prev; \ + (el)->prev = (add); \ + (add)->prev->next = (add); \ + if ((head) == (el)) { \ + (head) = (add); \ + } \ + } else { \ + CDL_APPEND2(head, add, prev, next); \ + } \ +} while (0) + +#define CDL_PREPEND_ELEM(head, el, add) \ + CDL_PREPEND_ELEM2(head, el, add, prev, next) + +#define CDL_APPEND_ELEM2(head, el, add, prev, next) \ +do { \ + if (el) { \ + assert((head) != NULL); \ + assert((add) != NULL); \ + (add)->next = (el)->next; \ + (add)->prev = (el); \ + (el)->next = (add); \ + (add)->next->prev = (add); \ + } else { \ + CDL_PREPEND2(head, add, prev, next); \ + } \ +} while (0) + +#define CDL_APPEND_ELEM(head, el, add) \ + CDL_APPEND_ELEM2(head, el, add, prev, next) + +#endif /* UTLIST_H */ From da5ec3b5e0244d29eb63ea5ac1d464b1c89504ed Mon Sep 17 00:00:00 2001 From: Philipp Moritz Date: Tue, 4 Oct 2016 12:11:52 -0700 Subject: [PATCH 21/36] add documentation for tasks (#30) --- doc/tasks.md | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) create mode 100644 doc/tasks.md diff --git a/doc/tasks.md b/doc/tasks.md new file mode 100644 index 000000000..b56f89783 --- /dev/null +++ b/doc/tasks.md @@ -0,0 +1,33 @@ +# Task specifications, task instances and task logs + +A *task specification* contains all information that is needed for computing +the results of a task: + +- The function ID of the function that executes the task +- The arguments (either object IDs for pass by reference +or values for pass by value) +- The IDs of the result objects + +From these, a task ID can be computed which is also stored in the task +specification. + +A *task instance* represents one execution of a task specification. +It consists of: + +- A scheduling state (WAITING, SCHEDULED, RUNNING, DONE) +- The target node where the task is scheduled or executed +- A unique task instance ID that identifies the particular execution + of the task. + +The task data structures are defined in `common/task.h`. + +The *task log* is a mapping from the task instance ID to a sequence of +updates to the status of the task instance. It is updated by various parts +of the system: + +1. The local scheduler writes it with status WAITING when submits a task to the global scheduler +2. The global scheduler appends an update WAITING -> SCHEDULED together with the node ID when assigning the task to a local scheduler +3. The local scheduler appends an update SCHEDULED -> RUNNING when it assigns a task to a worker +4. The local scheduler appends an update RUNNING -> DONE when the task finishes execution + +The task log is defined in `common/state/task_log.h`. From 8e044535e2624c53a2d4fca974fdeceaf2b9be1c Mon Sep 17 00:00:00 2001 From: Philipp Moritz Date: Tue, 4 Oct 2016 15:56:24 -0700 Subject: [PATCH 22/36] Python API for constructing tasks (#28) * Python API for constructing tasks * Fixes. --- .clang-format | 5 +- .travis.yml | 15 ++++ install-dependencies.sh | 21 +++++ lib/python/object_id.c | 91 +++++++++++++++++++ lib/python/serialization.c | 80 +++++++++++++++++ lib/python/setup.py | 12 +++ lib/python/task.c | 177 +++++++++++++++++++++++++++++++++++++ lib/python/types.h | 33 +++++++ task.h | 1 - test/test.py | 51 +++++++++++ 10 files changed, 484 insertions(+), 2 deletions(-) create mode 100755 install-dependencies.sh create mode 100644 lib/python/object_id.c create mode 100644 lib/python/serialization.c create mode 100644 lib/python/setup.py create mode 100644 lib/python/task.c create mode 100644 lib/python/types.h create mode 100644 test/test.py diff --git a/.clang-format b/.clang-format index 90d254290..89b87e25d 100644 --- a/.clang-format +++ b/.clang-format @@ -3,4 +3,7 @@ DerivePointerAlignment: true IndentCaseLabels: false PointerAlignment: Right SpaceAfterCStyleCast: true - +AllowShortBlocksOnASingleLine: false +AllowShortCaseLabelsOnASingleLine: false +AllowShortFunctionsOnASingleLine: false +AllowShortIfStatementsOnASingleLine: false diff --git a/.travis.yml b/.travis.yml index 220df4b86..174fee43f 100644 --- a/.travis.yml +++ b/.travis.yml @@ -6,8 +6,16 @@ matrix: include: - os: linux dist: trusty + python: "2.7" + - os: linux + dist: trusty + python: "3.5" - os: osx osx_image: xcode7 + python: "2.7" + - os: osx + osx_image: xcode7 + python: "3.5" - os: linux dist: trusty env: LINT=1 @@ -22,5 +30,12 @@ matrix: - .travis/check-git-clang-format-output.sh install: + - ./install-dependencies.sh - make - make test + - cd lib/python + - python setup.py install --user + - cd ../.. + +script: + - python test/test.py diff --git a/install-dependencies.sh b/install-dependencies.sh new file mode 100755 index 000000000..f84da1684 --- /dev/null +++ b/install-dependencies.sh @@ -0,0 +1,21 @@ +#!/usr/bin/env bash + +ROOT_DIR=$(cd "$(dirname "${BASH_SOURCE:-$0}")"; pwd) + +platform="unknown" +unamestr="$(uname)" +if [[ "$unamestr" == "Linux" ]]; then + echo "Platform is linux." + platform="linux" +elif [[ "$unamestr" == "Darwin" ]]; then + echo "Platform is macosx." + platform="macosx" +else + echo "Unrecognized platform." + exit 1 +fi + +if [[ $platform == "linux" ]]; then + sudo apt-get update + sudo apt-get install -y git python-dev +fi diff --git a/lib/python/object_id.c b/lib/python/object_id.c new file mode 100644 index 000000000..bd7db9bc7 --- /dev/null +++ b/lib/python/object_id.c @@ -0,0 +1,91 @@ +#include "types.h" + +int PyObjectToUniqueID(PyObject *object, object_id *objectid) { + if (PyObject_IsInstance(object, (PyObject *) &PyObjectIDType)) { + *objectid = ((PyObjectID *) object)->object_id; + return 1; + } else { + PyErr_SetString(PyExc_TypeError, "must be an ObjectID"); + return 0; + } +} + +static int PyObjectID_init(PyObjectID *self, PyObject *args, PyObject *kwds) { + const char *data; + int size; + if (!PyArg_ParseTuple(args, "s#", &data, &size)) { + return -1; + } + if (size != UNIQUE_ID_SIZE) { + PyErr_SetString(CommonError, + "ObjectID: object id string needs to have length 20"); + return -1; + } + memcpy(&self->object_id.id[0], data, UNIQUE_ID_SIZE); + return 0; +} + +/* create PyObjectID from C */ +PyObject *PyObjectID_make(object_id object_id) { + PyObjectID *result = PyObject_New(PyObjectID, &PyObjectIDType); + result = (PyObjectID *) PyObject_Init((PyObject *) result, &PyObjectIDType); + result->object_id = object_id; + return (PyObject *) result; +} + +static PyObject *PyObjectID_id(PyObject *self) { + PyObjectID *s = (PyObjectID *) self; + return PyString_FromStringAndSize((char *) &s->object_id.id[0], + UNIQUE_ID_SIZE); +} + +static PyMethodDef PyObjectID_methods[] = { + {"id", (PyCFunction) PyObjectID_id, METH_NOARGS, + "Return the hash associated with this ObjectID"}, + {NULL} /* Sentinel */ +}; + +static PyMemberDef PyObjectID_members[] = { + {NULL} /* Sentinel */ +}; + +PyTypeObject PyObjectIDType = { + PyObject_HEAD_INIT(NULL) 0, /* ob_size */ + "common.ObjectID", /* tp_name */ + sizeof(PyObjectID), /* tp_basicsize */ + 0, /* tp_itemsize */ + 0, /* tp_dealloc */ + 0, /* tp_print */ + 0, /* tp_getattr */ + 0, /* tp_setattr */ + 0, /* tp_compare */ + 0, /* tp_repr */ + 0, /* tp_as_number */ + 0, /* tp_as_sequence */ + 0, /* tp_as_mapping */ + 0, /* tp_hash */ + 0, /* tp_call */ + 0, /* tp_str */ + 0, /* tp_getattro */ + 0, /* tp_setattro */ + 0, /* tp_as_buffer */ + Py_TPFLAGS_DEFAULT, /* tp_flags */ + "ObjectID object", /* tp_doc */ + 0, /* tp_traverse */ + 0, /* tp_clear */ + 0, /* tp_richcompare */ + 0, /* tp_weaklistoffset */ + 0, /* tp_iter */ + 0, /* tp_iternext */ + PyObjectID_methods, /* tp_methods */ + PyObjectID_members, /* tp_members */ + 0, /* tp_getset */ + 0, /* tp_base */ + 0, /* tp_dict */ + 0, /* tp_descr_get */ + 0, /* tp_descr_set */ + 0, /* tp_dictoffset */ + (initproc) PyObjectID_init, /* tp_init */ + 0, /* tp_alloc */ + PyType_GenericNew, /* tp_new */ +}; diff --git a/lib/python/serialization.c b/lib/python/serialization.c new file mode 100644 index 000000000..82cf6417c --- /dev/null +++ b/lib/python/serialization.c @@ -0,0 +1,80 @@ +#include "types.h" + +/* TODO(pcm): Add limit on total number of elements. */ + +#define SIZE_LIMIT 100 +#define NUM_ELEMENTS_LIMIT 1000 + +/** + * This method checks if a Python object is sufficiently simple that it can be + * serialized and passed by value as an argument to a task (without being put in + * the object store). The details of which objects are sufficiently simple are + * defined by this method and are not particularly important. But for + * performance reasons, it is better to place "small" objects in the task itself + * and "large" objects in the object store. + * + * @param value The Python object in question. + * @param num_elements_contained If this method returns 1, then the number of + * objects recursively contained within this object will be added to the + * value at this address. This is used to make sure that we do not + * serialize objects that are too large. + * @return 0 if the object cannot be serialized in the task and 1 if it can. + */ +int is_simple_value(PyObject *value, int *num_elements_contained) { + *num_elements_contained += 1; + if (*num_elements_contained >= NUM_ELEMENTS_LIMIT) { + return 0; + } + if (PyInt_Check(value) || PyLong_Check(value) || value == Py_False || + value == Py_True || PyFloat_Check(value) || value == Py_None) { + return 1; + } + if (PyString_CheckExact(value)) { + *num_elements_contained += PyString_Size(value); + return (*num_elements_contained < NUM_ELEMENTS_LIMIT); + } + if (PyUnicode_CheckExact(value)) { + *num_elements_contained += PyUnicode_GET_SIZE(value); + return (*num_elements_contained < NUM_ELEMENTS_LIMIT); + } + if (PyList_CheckExact(value) && PyList_Size(value) < SIZE_LIMIT) { + for (size_t i = 0; i < PyList_Size(value); ++i) { + if (!is_simple_value(PyList_GetItem(value, i), num_elements_contained)) { + return 0; + } + } + return (*num_elements_contained < NUM_ELEMENTS_LIMIT); + } + if (PyDict_CheckExact(value) && PyDict_Size(value) < SIZE_LIMIT) { + PyObject *key, *val; + Py_ssize_t pos = 0; + while (PyDict_Next(value, &pos, &key, &val)) { + if (!is_simple_value(key, num_elements_contained) || + !is_simple_value(val, num_elements_contained)) { + return 0; + } + } + return (*num_elements_contained < NUM_ELEMENTS_LIMIT); + } + if (PyTuple_CheckExact(value) && PyTuple_Size(value) < SIZE_LIMIT) { + for (size_t i = 0; i < PyTuple_Size(value); ++i) { + if (!is_simple_value(PyTuple_GetItem(value, i), num_elements_contained)) { + return 0; + } + } + return (*num_elements_contained < NUM_ELEMENTS_LIMIT); + } + return 0; +} + +PyObject *check_simple_value(PyObject *self, PyObject *args) { + PyObject *value; + if (!PyArg_ParseTuple(args, "O", &value)) { + return NULL; + } + int num_elements_contained = 0; + if (is_simple_value(value, &num_elements_contained)) { + Py_RETURN_TRUE; + } + Py_RETURN_FALSE; +} diff --git a/lib/python/setup.py b/lib/python/setup.py new file mode 100644 index 000000000..38af43f34 --- /dev/null +++ b/lib/python/setup.py @@ -0,0 +1,12 @@ +from setuptools import setup, find_packages, Extension + +common_module = Extension("common", + sources=["object_id.c", "serialization.c", "task.c"], + include_dirs=["../../", "../../thirdparty"], + extra_objects=["../../build/libcommon.a"], + extra_compile_args=["--std=c99", "-Werror"]) + +setup(name="Common", + version="0.1", + description="Common library for Ray", + ext_modules=[common_module]) diff --git a/lib/python/task.c b/lib/python/task.c new file mode 100644 index 000000000..ef685b493 --- /dev/null +++ b/lib/python/task.c @@ -0,0 +1,177 @@ +#include +#include "node.h" + +#include "types.h" +#include "task.h" +#include "utarray.h" +#include "utstring.h" + +PyObject *CommonError; + +#define MARSHAL_VERSION 2 + +static int PyTask_init(PyTask *self, PyObject *args, PyObject *kwds) { + function_id function_id; + /* Arguments of the task (can be PyObjectIDs or Python values). */ + PyObject *arguments; + /* Array of pointers to string representations of pass-by-value args. */ + UT_array *val_repr_ptrs; + utarray_new(val_repr_ptrs, &ut_ptr_icd); + int num_returns; + if (!PyArg_ParseTuple(args, "O&Oi", &PyObjectToUniqueID, &function_id, + &arguments, &num_returns)) { + return -1; + } + size_t size = PyList_Size(arguments); + /* Determine the size of pass by value data in bytes. */ + size_t value_data_bytes = 0; + for (size_t i = 0; i < size; ++i) { + PyObject *arg = PyList_GetItem(arguments, i); + if (!PyObject_IsInstance(arg, (PyObject *) &PyObjectIDType)) { + PyObject *data = PyMarshal_WriteObjectToString(arg, MARSHAL_VERSION); + value_data_bytes += PyString_Size(data); + utarray_push_back(val_repr_ptrs, &data); + } + } + /* Construct the task specification. */ + int val_repr_index = 0; + self->spec = + alloc_task_spec(function_id, size, num_returns, value_data_bytes); + for (size_t i = 0; i < size; ++i) { + PyObject *arg = PyList_GetItem(arguments, i); + if (PyObject_IsInstance(arg, (PyObject *) &PyObjectIDType)) { + task_args_add_ref(self->spec, ((PyObjectID *) arg)->object_id); + } else { + PyObject *data = + *((PyObject **) utarray_eltptr(val_repr_ptrs, val_repr_index)); + task_args_add_val(self->spec, (uint8_t *) PyString_AS_STRING(data), + PyString_GET_SIZE(data)); + Py_DECREF(data); + val_repr_index += 1; + } + } + utarray_free(val_repr_ptrs); + return 0; +} + +static void PyTask_dealloc(PyTask *self) { + free_task_spec(self->spec); + Py_TYPE(self)->tp_free((PyObject *) self); +} + +static PyObject *PyTask_function_id(PyObject *self) { + function_id function_id = *task_function(((PyTask *) self)->spec); + return PyObjectID_make(function_id); +} + +static PyObject *PyTask_arguments(PyObject *self, PyObject *args) { + int arg_index; + task_spec *spec = ((PyTask *) self)->spec; + if (!PyArg_ParseTuple(args, "i", &arg_index)) { + return NULL; + } + if (task_arg_type(spec, arg_index) == ARG_BY_REF) { + object_id object_id = *task_arg_id(spec, arg_index); + return PyObjectID_make(object_id); + } else { + PyObject *s = PyMarshal_ReadObjectFromString( + (char *) task_arg_val(spec, arg_index), + (Py_ssize_t) task_arg_length(spec, arg_index)); + Py_DECREF(s); + Py_RETURN_NONE; + } +} + +static PyObject *PyTask_returns(PyObject *self, PyObject *args) { + int ret_index; + if (!PyArg_ParseTuple(args, "i", &ret_index)) { + return NULL; + } + object_id object_id = *task_return(((PyTask *) self)->spec, ret_index); + return PyObjectID_make(object_id); +} + +static PyMethodDef PyTask_methods[] = { + {"function_id", (PyCFunction) PyTask_function_id, METH_NOARGS, + "Return the function id associated with this task."}, + {"arguments", (PyCFunction) PyTask_arguments, METH_VARARGS, + "Return the i-th argument of the task."}, + {"returns", (PyCFunction) PyTask_returns, METH_VARARGS, + "Return the i-th object reference of the task."}, + {NULL} /* Sentinel */ +}; + +static PyTypeObject PyTaskType = { + PyObject_HEAD_INIT(NULL) 0, /* ob_size */ + "task.Task", /* tp_name */ + sizeof(PyTask), /* tp_basicsize */ + 0, /* tp_itemsize */ + (destructor) PyTask_dealloc, /* tp_dealloc */ + 0, /* tp_print */ + 0, /* tp_getattr */ + 0, /* tp_setattr */ + 0, /* tp_compare */ + 0, /* tp_repr */ + 0, /* tp_as_number */ + 0, /* tp_as_sequence */ + 0, /* tp_as_mapping */ + 0, /* tp_hash */ + 0, /* tp_call */ + 0, /* tp_str */ + 0, /* tp_getattro */ + 0, /* tp_setattro */ + 0, /* tp_as_buffer */ + Py_TPFLAGS_DEFAULT, /* tp_flags */ + "Task object", /* tp_doc */ + 0, /* tp_traverse */ + 0, /* tp_clear */ + 0, /* tp_richcompare */ + 0, /* tp_weaklistoffset */ + 0, /* tp_iter */ + 0, /* tp_iternext */ + PyTask_methods, /* tp_methods */ + 0, /* tp_members */ + 0, /* tp_getset */ + 0, /* tp_base */ + 0, /* tp_dict */ + 0, /* tp_descr_get */ + 0, /* tp_descr_set */ + 0, /* tp_dictoffset */ + (initproc) PyTask_init, /* tp_init */ + 0, /* tp_alloc */ + PyType_GenericNew, /* tp_new */ +}; + +static PyMethodDef common_methods[] = { + {"check_simple_value", check_simple_value, METH_VARARGS, + "Should the object be passed by value?"}, + {NULL} /* Sentinel */ +}; + +#ifndef PyMODINIT_FUNC /* declarations for DLL import/export */ +#define PyMODINIT_FUNC void +#endif + +PyMODINIT_FUNC initcommon(void) { + PyObject *m; + + if (PyType_Ready(&PyTaskType) < 0) + return; + + if (PyType_Ready(&PyObjectIDType) < 0) + return; + + m = Py_InitModule3("common", common_methods, + "Example module that creates an extension type."); + + Py_INCREF(&PyTaskType); + PyModule_AddObject(m, "Task", (PyObject *) &PyTaskType); + + Py_INCREF(&PyObjectIDType); + PyModule_AddObject(m, "ObjectID", (PyObject *) &PyObjectIDType); + + char common_error[] = "common.error"; + CommonError = PyErr_NewException(common_error, NULL, NULL); + Py_INCREF(CommonError); + PyModule_AddObject(m, "common_error", CommonError); +} diff --git a/lib/python/types.h b/lib/python/types.h new file mode 100644 index 000000000..9e30581b5 --- /dev/null +++ b/lib/python/types.h @@ -0,0 +1,33 @@ +#ifndef TYPES_H +#define TYPES_H + +#include +#include "marshal.h" +#include "structmember.h" + +#include "common.h" +#include "task.h" + +extern PyObject *CommonError; + +// clang-format off +typedef struct { + PyObject_HEAD + object_id object_id; +} PyObjectID; + +typedef struct { + PyObject_HEAD + task_spec *spec; +} PyTask; +// clang-format on + +extern PyTypeObject PyObjectIDType; + +int PyObjectToUniqueID(PyObject *object, object_id *objectid); + +PyObject *PyObjectID_make(object_id object_id); + +PyObject *check_simple_value(PyObject *self, PyObject *args); + +#endif /* TYPES_H */ diff --git a/task.h b/task.h index 9267edb65..28886bf14 100644 --- a/task.h +++ b/task.h @@ -13,7 +13,6 @@ #include "utstring.h" typedef unique_id function_id; -typedef unique_id object_id; /* The task ID is a deterministic hash of the function ID that * the task executes and the argument IDs or argument values */ diff --git a/test/test.py b/test/test.py new file mode 100644 index 000000000..bbed21376 --- /dev/null +++ b/test/test.py @@ -0,0 +1,51 @@ +from __future__ import print_function + +import unittest + +import common + +BASE_SIMPLE_OBJECTS = [ + 0, 1, 100000, 0L, 1L, 100000L, 1L << 100, 0.0, 0.5, 0.9, 100000.1, (), [], {}, + "", 990 * "h", u"", 990 * u"h" +] + +LIST_SIMPLE_OBJECTS = [[obj] for obj in BASE_SIMPLE_OBJECTS] +TUPLE_SIMPLE_OBJECTS = [(obj,) for obj in BASE_SIMPLE_OBJECTS] +DICT_SIMPLE_OBJECTS = [{(): obj} for obj in BASE_SIMPLE_OBJECTS] + +SIMPLE_OBJECTS = (BASE_SIMPLE_OBJECTS + + LIST_SIMPLE_OBJECTS + + TUPLE_SIMPLE_OBJECTS + + DICT_SIMPLE_OBJECTS) + +# Create some complex objects that cannot be serialized by value in tasks. + +l = [] +l.append(l) + +class Foo(object): + def __init__(self): + pass + +BASE_COMPLEX_OBJECTS = [999 * "h", 999 * u"h", l, Foo(), 10 * [10 * [10 * [1]]]] + +LIST_COMPLEX_OBJECTS = [[obj] for obj in BASE_COMPLEX_OBJECTS] +TUPLE_COMPLEX_OBJECTS = [(obj,) for obj in BASE_COMPLEX_OBJECTS] +DICT_COMPLEX_OBJECTS = [{(): obj} for obj in BASE_COMPLEX_OBJECTS] + +COMPLEX_OBJECTS = (BASE_COMPLEX_OBJECTS + + LIST_COMPLEX_OBJECTS + + TUPLE_COMPLEX_OBJECTS + + DICT_COMPLEX_OBJECTS) + +class TestPlasmaClient(unittest.TestCase): + + def test_serialize_by_value(self): + + for val in SIMPLE_OBJECTS: + self.assertTrue(common.check_simple_value(val)) + for val in COMPLEX_OBJECTS: + self.assertFalse(common.check_simple_value(val)) + +if __name__ == "__main__": + unittest.main(verbosity=2) From 4329afbd53449412b08e64d4057a2857393c2212 Mon Sep 17 00:00:00 2001 From: Philipp Moritz Date: Tue, 4 Oct 2016 16:59:44 -0700 Subject: [PATCH 23/36] rename TASK_* -> TASK_STATUS_* (#31) --- task.h | 8 ++++---- test/db_tests.c | 14 +++++++------- 2 files changed, 11 insertions(+), 11 deletions(-) diff --git a/task.h b/task.h index 28886bf14..782bf685b 100644 --- a/task.h +++ b/task.h @@ -93,10 +93,10 @@ void print_task(task_spec *spec, UT_string *output); /* The scheduling_state can be used as a flag when we are listening * for an event, for example TASK_WAITING | TASK_SCHEDULED. */ enum scheduling_state { - TASK_WAITING = 1, - TASK_SCHEDULED = 2, - TASK_RUNNING = 4, - TASK_DONE = 8 + TASK_STATUS_WAITING = 1, + TASK_STATUS_SCHEDULED = 2, + TASK_STATUS_RUNNING = 4, + TASK_STATUS_DONE = 8 }; /* A task instance is one execution of a task specification. diff --git a/test/db_tests.c b/test/db_tests.c index 96f16b528..be09ad28e 100644 --- a/test/db_tests.c +++ b/test/db_tests.c @@ -74,7 +74,7 @@ TEST object_table_lookup_test(void) { void task_log_test_callback(task_instance *instance, void *userdata) { task_instance *other = userdata; - CHECK(*task_instance_state(instance) == TASK_SCHEDULED); + CHECK(*task_instance_state(instance) == TASK_STATUS_SCHEDULED); CHECK(task_instance_size(instance) == task_instance_size(other)); CHECK(memcmp(instance, other, task_instance_size(instance)) == 0); } @@ -86,9 +86,9 @@ TEST task_log_test(void) { node_id node = globally_unique_id(); task_spec *task = example_task(); task_instance *instance = - make_task_instance(globally_unique_id(), task, TASK_SCHEDULED, node); - task_log_register_callback(db, task_log_test_callback, node, TASK_SCHEDULED, - instance); + make_task_instance(globally_unique_id(), task, TASK_STATUS_SCHEDULED, node); + task_log_register_callback(db, task_log_test_callback, node, + TASK_STATUS_SCHEDULED, instance); task_log_add_task(db, instance); event_loop_add_timer(loop, 100, timeout_handler, NULL); event_loop_run(loop); @@ -112,11 +112,11 @@ TEST task_log_all_test(void) { task_spec *task = example_task(); /* Schedule two tasks on different nodes. */ task_instance *instance1 = make_task_instance( - globally_unique_id(), task, TASK_SCHEDULED, globally_unique_id()); + globally_unique_id(), task, TASK_STATUS_SCHEDULED, globally_unique_id()); task_instance *instance2 = make_task_instance( - globally_unique_id(), task, TASK_SCHEDULED, globally_unique_id()); + globally_unique_id(), task, TASK_STATUS_SCHEDULED, globally_unique_id()); task_log_register_callback(db, task_log_all_test_callback, NIL_ID, - TASK_SCHEDULED, NULL); + TASK_STATUS_SCHEDULED, NULL); task_log_add_task(db, instance1); task_log_add_task(db, instance2); event_loop_add_timer(loop, 100, timeout_handler, NULL); From 4204500d23be7726e27598badb691d29d08a0ad7 Mon Sep 17 00:00:00 2001 From: Robert Nishihara Date: Wed, 5 Oct 2016 09:17:08 -0700 Subject: [PATCH 24/36] Fix formatting. (#32) --- test/db_tests.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/test/db_tests.c b/test/db_tests.c index be09ad28e..6eb592e45 100644 --- a/test/db_tests.c +++ b/test/db_tests.c @@ -85,8 +85,8 @@ TEST task_log_test(void) { db_attach(db, loop); node_id node = globally_unique_id(); task_spec *task = example_task(); - task_instance *instance = - make_task_instance(globally_unique_id(), task, TASK_STATUS_SCHEDULED, node); + task_instance *instance = make_task_instance(globally_unique_id(), task, + TASK_STATUS_SCHEDULED, node); task_log_register_callback(db, task_log_test_callback, node, TASK_STATUS_SCHEDULED, instance); task_log_add_task(db, instance); From 75441a180d4935917f0072059de5ddf8d3a298fd Mon Sep 17 00:00:00 2001 From: Philipp Moritz Date: Wed, 5 Oct 2016 16:09:40 -0700 Subject: [PATCH 25/36] add valgrind tests (#33) * add valgrind * install valgrind --- .travis.yml | 9 +++++++++ Makefile | 7 +++++++ 2 files changed, 16 insertions(+) diff --git a/.travis.yml b/.travis.yml index 174fee43f..d0e14edf5 100644 --- a/.travis.yml +++ b/.travis.yml @@ -28,6 +28,15 @@ matrix: install: [] script: - .travis/check-git-clang-format-output.sh + - os: linux + dist: trusty + python: "2.7" + env: VALGRIND=1 + before_install: + - sudo apt-get update -qq + - sudo apt-get install -qq valgrind + script: + - make valgrind install: - ./install-dependencies.sh diff --git a/Makefile b/Makefile index ad955de98..dae76e87b 100644 --- a/Makefile +++ b/Makefile @@ -36,4 +36,11 @@ test: hiredis redis $(BUILD)/common_tests $(BUILD)/db_tests $(BUILD)/io_tests $( ./thirdparty/redis-3.2.3/src/redis-server & sleep 1s ; ./build/common_tests ; ./build/db_tests ; ./build/io_tests ; ./build/task_tests ; ./build/redis_tests +valgrind: test + valgrind --leak-check=full --error-exitcode=1 ./build/common_tests + valgrind --leak-check=full --error-exitcode=1 ./build/db_tests + valgrind --leak-check=full --error-exitcode=1 ./build/io_tests + valgrind --leak-check=full --error-exitcode=1 ./build/task_tests + valgrind --leak-check=full --error-exitcode=1 ./build/redis_tests + FORCE: From 90a6a99b0380a7b5243b18c1f2360a75209b0fa2 Mon Sep 17 00:00:00 2001 From: Robert Nishihara Date: Thu, 6 Oct 2016 16:25:04 -0700 Subject: [PATCH 26/36] Clean up task interface and add testing. (#34) * Update c extensions and python tests. * Updates. * Documentation fix. * Move c extensions into common_extension.c --- lib/python/common_extension.c | 361 +++++++++++++++++++++ lib/python/{types.h => common_extension.h} | 8 +- lib/python/object_id.c | 91 ------ lib/python/serialization.c | 80 ----- lib/python/setup.py | 2 +- lib/python/task.c | 177 ---------- test/test.py | 52 ++- 7 files changed, 418 insertions(+), 353 deletions(-) create mode 100644 lib/python/common_extension.c rename lib/python/{types.h => common_extension.h} (79%) delete mode 100644 lib/python/object_id.c delete mode 100644 lib/python/serialization.c delete mode 100644 lib/python/task.c diff --git a/lib/python/common_extension.c b/lib/python/common_extension.c new file mode 100644 index 000000000..556fea443 --- /dev/null +++ b/lib/python/common_extension.c @@ -0,0 +1,361 @@ +#include +#include "node.h" + +#include "common_extension.h" +#include "task.h" +#include "utarray.h" +#include "utstring.h" + +PyObject *CommonError; + +#define MARSHAL_VERSION 2 + +/* Define the PyObjectID class. */ + +int PyObjectToUniqueID(PyObject *object, object_id *objectid) { + if (PyObject_IsInstance(object, (PyObject *) &PyObjectIDType)) { + *objectid = ((PyObjectID *) object)->object_id; + return 1; + } else { + PyErr_SetString(PyExc_TypeError, "must be an ObjectID"); + return 0; + } +} + +static int PyObjectID_init(PyObjectID *self, PyObject *args, PyObject *kwds) { + const char *data; + int size; + if (!PyArg_ParseTuple(args, "s#", &data, &size)) { + return -1; + } + if (size != UNIQUE_ID_SIZE) { + PyErr_SetString(CommonError, + "ObjectID: object id string needs to have length 20"); + return -1; + } + memcpy(&self->object_id.id[0], data, UNIQUE_ID_SIZE); + return 0; +} + +/* Create a PyObjectID from C. */ +PyObject *PyObjectID_make(object_id object_id) { + PyObjectID *result = PyObject_New(PyObjectID, &PyObjectIDType); + result = (PyObjectID *) PyObject_Init((PyObject *) result, &PyObjectIDType); + result->object_id = object_id; + return (PyObject *) result; +} + +static PyObject *PyObjectID_id(PyObject *self) { + PyObjectID *s = (PyObjectID *) self; + return PyString_FromStringAndSize((char *) &s->object_id.id[0], + UNIQUE_ID_SIZE); +} + +static PyMethodDef PyObjectID_methods[] = { + {"id", (PyCFunction) PyObjectID_id, METH_NOARGS, + "Return the hash associated with this ObjectID"}, + {NULL} /* Sentinel */ +}; + +static PyMemberDef PyObjectID_members[] = { + {NULL} /* Sentinel */ +}; + +PyTypeObject PyObjectIDType = { + PyObject_HEAD_INIT(NULL) 0, /* ob_size */ + "common.ObjectID", /* tp_name */ + sizeof(PyObjectID), /* tp_basicsize */ + 0, /* tp_itemsize */ + 0, /* tp_dealloc */ + 0, /* tp_print */ + 0, /* tp_getattr */ + 0, /* tp_setattr */ + 0, /* tp_compare */ + 0, /* tp_repr */ + 0, /* tp_as_number */ + 0, /* tp_as_sequence */ + 0, /* tp_as_mapping */ + 0, /* tp_hash */ + 0, /* tp_call */ + 0, /* tp_str */ + 0, /* tp_getattro */ + 0, /* tp_setattro */ + 0, /* tp_as_buffer */ + Py_TPFLAGS_DEFAULT, /* tp_flags */ + "ObjectID object", /* tp_doc */ + 0, /* tp_traverse */ + 0, /* tp_clear */ + 0, /* tp_richcompare */ + 0, /* tp_weaklistoffset */ + 0, /* tp_iter */ + 0, /* tp_iternext */ + PyObjectID_methods, /* tp_methods */ + PyObjectID_members, /* tp_members */ + 0, /* tp_getset */ + 0, /* tp_base */ + 0, /* tp_dict */ + 0, /* tp_descr_get */ + 0, /* tp_descr_set */ + 0, /* tp_dictoffset */ + (initproc) PyObjectID_init, /* tp_init */ + 0, /* tp_alloc */ + PyType_GenericNew, /* tp_new */ +}; + +/* Define the PyTask class. */ + +static int PyTask_init(PyTask *self, PyObject *args, PyObject *kwds) { + function_id function_id; + /* Arguments of the task (can be PyObjectIDs or Python values). */ + PyObject *arguments; + /* Array of pointers to string representations of pass-by-value args. */ + UT_array *val_repr_ptrs; + utarray_new(val_repr_ptrs, &ut_ptr_icd); + int num_returns; + if (!PyArg_ParseTuple(args, "O&Oi", &PyObjectToUniqueID, &function_id, + &arguments, &num_returns)) { + return -1; + } + size_t size = PyList_Size(arguments); + /* Determine the size of pass by value data in bytes. */ + size_t value_data_bytes = 0; + for (size_t i = 0; i < size; ++i) { + PyObject *arg = PyList_GetItem(arguments, i); + if (!PyObject_IsInstance(arg, (PyObject *) &PyObjectIDType)) { + PyObject *data = PyMarshal_WriteObjectToString(arg, MARSHAL_VERSION); + value_data_bytes += PyString_Size(data); + utarray_push_back(val_repr_ptrs, &data); + } + } + /* Construct the task specification. */ + int val_repr_index = 0; + self->spec = + alloc_task_spec(function_id, size, num_returns, value_data_bytes); + for (size_t i = 0; i < size; ++i) { + PyObject *arg = PyList_GetItem(arguments, i); + if (PyObject_IsInstance(arg, (PyObject *) &PyObjectIDType)) { + task_args_add_ref(self->spec, ((PyObjectID *) arg)->object_id); + } else { + PyObject *data = + *((PyObject **) utarray_eltptr(val_repr_ptrs, val_repr_index)); + task_args_add_val(self->spec, (uint8_t *) PyString_AS_STRING(data), + PyString_GET_SIZE(data)); + Py_DECREF(data); + val_repr_index += 1; + } + } + utarray_free(val_repr_ptrs); + return 0; +} + +static void PyTask_dealloc(PyTask *self) { + free_task_spec(self->spec); + Py_TYPE(self)->tp_free((PyObject *) self); +} + +static PyObject *PyTask_function_id(PyObject *self) { + function_id function_id = *task_function(((PyTask *) self)->spec); + return PyObjectID_make(function_id); +} + +static PyObject *PyTask_arguments(PyObject *self) { + int64_t num_args = task_num_args(((PyTask *) self)->spec); + PyObject *arg_list = PyList_New((Py_ssize_t) num_args); + task_spec *task = ((PyTask *) self)->spec; + for (int i = 0; i < num_args; ++i) { + if (task_arg_type(task, i) == ARG_BY_REF) { + object_id object_id = *task_arg_id(task, i); + PyList_SetItem(arg_list, i, PyObjectID_make(object_id)); + } else { + PyObject *s = + PyMarshal_ReadObjectFromString((char *) task_arg_val(task, i), + (Py_ssize_t) task_arg_length(task, i)); + PyList_SetItem(arg_list, i, s); + } + } + return arg_list; +} + +static PyObject *PyTask_returns(PyObject *self) { + int64_t num_returns = task_num_returns(((PyTask *) self)->spec); + PyObject *return_id_list = PyList_New((Py_ssize_t) num_returns); + task_spec *task = ((PyTask *) self)->spec; + for (int i = 0; i < num_returns; ++i) { + object_id object_id = *task_return(task, i); + PyList_SetItem(return_id_list, i, PyObjectID_make(object_id)); + } + return return_id_list; +} + +static PyMethodDef PyTask_methods[] = { + {"function_id", (PyCFunction) PyTask_function_id, METH_NOARGS, + "Return the function ID for this task."}, + {"arguments", (PyCFunction) PyTask_arguments, METH_NOARGS, + "Return the arguments for the task."}, + {"returns", (PyCFunction) PyTask_returns, METH_NOARGS, + "Return the object IDs for the return values of the task."}, + {NULL} /* Sentinel */ +}; + +static PyTypeObject PyTaskType = { + PyObject_HEAD_INIT(NULL) 0, /* ob_size */ + "task.Task", /* tp_name */ + sizeof(PyTask), /* tp_basicsize */ + 0, /* tp_itemsize */ + (destructor) PyTask_dealloc, /* tp_dealloc */ + 0, /* tp_print */ + 0, /* tp_getattr */ + 0, /* tp_setattr */ + 0, /* tp_compare */ + 0, /* tp_repr */ + 0, /* tp_as_number */ + 0, /* tp_as_sequence */ + 0, /* tp_as_mapping */ + 0, /* tp_hash */ + 0, /* tp_call */ + 0, /* tp_str */ + 0, /* tp_getattro */ + 0, /* tp_setattro */ + 0, /* tp_as_buffer */ + Py_TPFLAGS_DEFAULT, /* tp_flags */ + "Task object", /* tp_doc */ + 0, /* tp_traverse */ + 0, /* tp_clear */ + 0, /* tp_richcompare */ + 0, /* tp_weaklistoffset */ + 0, /* tp_iter */ + 0, /* tp_iternext */ + PyTask_methods, /* tp_methods */ + 0, /* tp_members */ + 0, /* tp_getset */ + 0, /* tp_base */ + 0, /* tp_dict */ + 0, /* tp_descr_get */ + 0, /* tp_descr_set */ + 0, /* tp_dictoffset */ + (initproc) PyTask_init, /* tp_init */ + 0, /* tp_alloc */ + PyType_GenericNew, /* tp_new */ +}; + +/* Create a PyTask from a C struct. The resulting PyTask takes ownership of the + * task_spec and will deallocate the task_spec in the PyTask destructor. */ +PyObject *PyTask_make(task_spec *task_spec) { + PyTask *result = PyObject_New(PyTask, &PyTaskType); + result = (PyTask *) PyObject_Init((PyObject *) result, &PyTaskType); + result->spec = task_spec; + return (PyObject *) result; +} + +/* Define the methods for the module. */ + +#define SIZE_LIMIT 100 +#define NUM_ELEMENTS_LIMIT 1000 + +/** + * This method checks if a Python object is sufficiently simple that it can be + * serialized and passed by value as an argument to a task (without being put in + * the object store). The details of which objects are sufficiently simple are + * defined by this method and are not particularly important. But for + * performance reasons, it is better to place "small" objects in the task itself + * and "large" objects in the object store. + * + * @param value The Python object in question. + * @param num_elements_contained If this method returns 1, then the number of + * objects recursively contained within this object will be added to the + * value at this address. This is used to make sure that we do not + * serialize objects that are too large. + * @return 0 if the object cannot be serialized in the task and 1 if it can. + */ +int is_simple_value(PyObject *value, int *num_elements_contained) { + *num_elements_contained += 1; + if (*num_elements_contained >= NUM_ELEMENTS_LIMIT) { + return 0; + } + if (PyInt_Check(value) || PyLong_Check(value) || value == Py_False || + value == Py_True || PyFloat_Check(value) || value == Py_None) { + return 1; + } + if (PyString_CheckExact(value)) { + *num_elements_contained += PyString_Size(value); + return (*num_elements_contained < NUM_ELEMENTS_LIMIT); + } + if (PyUnicode_CheckExact(value)) { + *num_elements_contained += PyUnicode_GET_SIZE(value); + return (*num_elements_contained < NUM_ELEMENTS_LIMIT); + } + if (PyList_CheckExact(value) && PyList_Size(value) < SIZE_LIMIT) { + for (size_t i = 0; i < PyList_Size(value); ++i) { + if (!is_simple_value(PyList_GetItem(value, i), num_elements_contained)) { + return 0; + } + } + return (*num_elements_contained < NUM_ELEMENTS_LIMIT); + } + if (PyDict_CheckExact(value) && PyDict_Size(value) < SIZE_LIMIT) { + PyObject *key, *val; + Py_ssize_t pos = 0; + while (PyDict_Next(value, &pos, &key, &val)) { + if (!is_simple_value(key, num_elements_contained) || + !is_simple_value(val, num_elements_contained)) { + return 0; + } + } + return (*num_elements_contained < NUM_ELEMENTS_LIMIT); + } + if (PyTuple_CheckExact(value) && PyTuple_Size(value) < SIZE_LIMIT) { + for (size_t i = 0; i < PyTuple_Size(value); ++i) { + if (!is_simple_value(PyTuple_GetItem(value, i), num_elements_contained)) { + return 0; + } + } + return (*num_elements_contained < NUM_ELEMENTS_LIMIT); + } + return 0; +} + +PyObject *check_simple_value(PyObject *self, PyObject *args) { + PyObject *value; + if (!PyArg_ParseTuple(args, "O", &value)) { + return NULL; + } + int num_elements_contained = 0; + if (is_simple_value(value, &num_elements_contained)) { + Py_RETURN_TRUE; + } + Py_RETURN_FALSE; +} + +static PyMethodDef common_methods[] = { + {"check_simple_value", check_simple_value, METH_VARARGS, + "Should the object be passed by value?"}, + {NULL} /* Sentinel */ +}; + +#ifndef PyMODINIT_FUNC /* declarations for DLL import/export */ +#define PyMODINIT_FUNC void +#endif + +PyMODINIT_FUNC initcommon(void) { + PyObject *m; + + if (PyType_Ready(&PyTaskType) < 0) + return; + + if (PyType_Ready(&PyObjectIDType) < 0) + return; + + m = Py_InitModule3("common", common_methods, + "Example module that creates an extension type."); + + Py_INCREF(&PyTaskType); + PyModule_AddObject(m, "Task", (PyObject *) &PyTaskType); + + Py_INCREF(&PyObjectIDType); + PyModule_AddObject(m, "ObjectID", (PyObject *) &PyObjectIDType); + + char common_error[] = "common.error"; + CommonError = PyErr_NewException(common_error, NULL, NULL); + Py_INCREF(CommonError); + PyModule_AddObject(m, "common_error", CommonError); +} diff --git a/lib/python/types.h b/lib/python/common_extension.h similarity index 79% rename from lib/python/types.h rename to lib/python/common_extension.h index 9e30581b5..578c548d0 100644 --- a/lib/python/types.h +++ b/lib/python/common_extension.h @@ -1,5 +1,5 @@ -#ifndef TYPES_H -#define TYPES_H +#ifndef COMMON_EXTENSION_H +#define COMMON_EXTENSION_H #include #include "marshal.h" @@ -30,4 +30,6 @@ PyObject *PyObjectID_make(object_id object_id); PyObject *check_simple_value(PyObject *self, PyObject *args); -#endif /* TYPES_H */ +PyObject *PyTask_make(task_spec *task_spec); + +#endif /* COMMON_EXTENSION_H */ diff --git a/lib/python/object_id.c b/lib/python/object_id.c deleted file mode 100644 index bd7db9bc7..000000000 --- a/lib/python/object_id.c +++ /dev/null @@ -1,91 +0,0 @@ -#include "types.h" - -int PyObjectToUniqueID(PyObject *object, object_id *objectid) { - if (PyObject_IsInstance(object, (PyObject *) &PyObjectIDType)) { - *objectid = ((PyObjectID *) object)->object_id; - return 1; - } else { - PyErr_SetString(PyExc_TypeError, "must be an ObjectID"); - return 0; - } -} - -static int PyObjectID_init(PyObjectID *self, PyObject *args, PyObject *kwds) { - const char *data; - int size; - if (!PyArg_ParseTuple(args, "s#", &data, &size)) { - return -1; - } - if (size != UNIQUE_ID_SIZE) { - PyErr_SetString(CommonError, - "ObjectID: object id string needs to have length 20"); - return -1; - } - memcpy(&self->object_id.id[0], data, UNIQUE_ID_SIZE); - return 0; -} - -/* create PyObjectID from C */ -PyObject *PyObjectID_make(object_id object_id) { - PyObjectID *result = PyObject_New(PyObjectID, &PyObjectIDType); - result = (PyObjectID *) PyObject_Init((PyObject *) result, &PyObjectIDType); - result->object_id = object_id; - return (PyObject *) result; -} - -static PyObject *PyObjectID_id(PyObject *self) { - PyObjectID *s = (PyObjectID *) self; - return PyString_FromStringAndSize((char *) &s->object_id.id[0], - UNIQUE_ID_SIZE); -} - -static PyMethodDef PyObjectID_methods[] = { - {"id", (PyCFunction) PyObjectID_id, METH_NOARGS, - "Return the hash associated with this ObjectID"}, - {NULL} /* Sentinel */ -}; - -static PyMemberDef PyObjectID_members[] = { - {NULL} /* Sentinel */ -}; - -PyTypeObject PyObjectIDType = { - PyObject_HEAD_INIT(NULL) 0, /* ob_size */ - "common.ObjectID", /* tp_name */ - sizeof(PyObjectID), /* tp_basicsize */ - 0, /* tp_itemsize */ - 0, /* tp_dealloc */ - 0, /* tp_print */ - 0, /* tp_getattr */ - 0, /* tp_setattr */ - 0, /* tp_compare */ - 0, /* tp_repr */ - 0, /* tp_as_number */ - 0, /* tp_as_sequence */ - 0, /* tp_as_mapping */ - 0, /* tp_hash */ - 0, /* tp_call */ - 0, /* tp_str */ - 0, /* tp_getattro */ - 0, /* tp_setattro */ - 0, /* tp_as_buffer */ - Py_TPFLAGS_DEFAULT, /* tp_flags */ - "ObjectID object", /* tp_doc */ - 0, /* tp_traverse */ - 0, /* tp_clear */ - 0, /* tp_richcompare */ - 0, /* tp_weaklistoffset */ - 0, /* tp_iter */ - 0, /* tp_iternext */ - PyObjectID_methods, /* tp_methods */ - PyObjectID_members, /* tp_members */ - 0, /* tp_getset */ - 0, /* tp_base */ - 0, /* tp_dict */ - 0, /* tp_descr_get */ - 0, /* tp_descr_set */ - 0, /* tp_dictoffset */ - (initproc) PyObjectID_init, /* tp_init */ - 0, /* tp_alloc */ - PyType_GenericNew, /* tp_new */ -}; diff --git a/lib/python/serialization.c b/lib/python/serialization.c deleted file mode 100644 index 82cf6417c..000000000 --- a/lib/python/serialization.c +++ /dev/null @@ -1,80 +0,0 @@ -#include "types.h" - -/* TODO(pcm): Add limit on total number of elements. */ - -#define SIZE_LIMIT 100 -#define NUM_ELEMENTS_LIMIT 1000 - -/** - * This method checks if a Python object is sufficiently simple that it can be - * serialized and passed by value as an argument to a task (without being put in - * the object store). The details of which objects are sufficiently simple are - * defined by this method and are not particularly important. But for - * performance reasons, it is better to place "small" objects in the task itself - * and "large" objects in the object store. - * - * @param value The Python object in question. - * @param num_elements_contained If this method returns 1, then the number of - * objects recursively contained within this object will be added to the - * value at this address. This is used to make sure that we do not - * serialize objects that are too large. - * @return 0 if the object cannot be serialized in the task and 1 if it can. - */ -int is_simple_value(PyObject *value, int *num_elements_contained) { - *num_elements_contained += 1; - if (*num_elements_contained >= NUM_ELEMENTS_LIMIT) { - return 0; - } - if (PyInt_Check(value) || PyLong_Check(value) || value == Py_False || - value == Py_True || PyFloat_Check(value) || value == Py_None) { - return 1; - } - if (PyString_CheckExact(value)) { - *num_elements_contained += PyString_Size(value); - return (*num_elements_contained < NUM_ELEMENTS_LIMIT); - } - if (PyUnicode_CheckExact(value)) { - *num_elements_contained += PyUnicode_GET_SIZE(value); - return (*num_elements_contained < NUM_ELEMENTS_LIMIT); - } - if (PyList_CheckExact(value) && PyList_Size(value) < SIZE_LIMIT) { - for (size_t i = 0; i < PyList_Size(value); ++i) { - if (!is_simple_value(PyList_GetItem(value, i), num_elements_contained)) { - return 0; - } - } - return (*num_elements_contained < NUM_ELEMENTS_LIMIT); - } - if (PyDict_CheckExact(value) && PyDict_Size(value) < SIZE_LIMIT) { - PyObject *key, *val; - Py_ssize_t pos = 0; - while (PyDict_Next(value, &pos, &key, &val)) { - if (!is_simple_value(key, num_elements_contained) || - !is_simple_value(val, num_elements_contained)) { - return 0; - } - } - return (*num_elements_contained < NUM_ELEMENTS_LIMIT); - } - if (PyTuple_CheckExact(value) && PyTuple_Size(value) < SIZE_LIMIT) { - for (size_t i = 0; i < PyTuple_Size(value); ++i) { - if (!is_simple_value(PyTuple_GetItem(value, i), num_elements_contained)) { - return 0; - } - } - return (*num_elements_contained < NUM_ELEMENTS_LIMIT); - } - return 0; -} - -PyObject *check_simple_value(PyObject *self, PyObject *args) { - PyObject *value; - if (!PyArg_ParseTuple(args, "O", &value)) { - return NULL; - } - int num_elements_contained = 0; - if (is_simple_value(value, &num_elements_contained)) { - Py_RETURN_TRUE; - } - Py_RETURN_FALSE; -} diff --git a/lib/python/setup.py b/lib/python/setup.py index 38af43f34..db915b06b 100644 --- a/lib/python/setup.py +++ b/lib/python/setup.py @@ -1,7 +1,7 @@ from setuptools import setup, find_packages, Extension common_module = Extension("common", - sources=["object_id.c", "serialization.c", "task.c"], + sources=["common_extension.c"], include_dirs=["../../", "../../thirdparty"], extra_objects=["../../build/libcommon.a"], extra_compile_args=["--std=c99", "-Werror"]) diff --git a/lib/python/task.c b/lib/python/task.c deleted file mode 100644 index ef685b493..000000000 --- a/lib/python/task.c +++ /dev/null @@ -1,177 +0,0 @@ -#include -#include "node.h" - -#include "types.h" -#include "task.h" -#include "utarray.h" -#include "utstring.h" - -PyObject *CommonError; - -#define MARSHAL_VERSION 2 - -static int PyTask_init(PyTask *self, PyObject *args, PyObject *kwds) { - function_id function_id; - /* Arguments of the task (can be PyObjectIDs or Python values). */ - PyObject *arguments; - /* Array of pointers to string representations of pass-by-value args. */ - UT_array *val_repr_ptrs; - utarray_new(val_repr_ptrs, &ut_ptr_icd); - int num_returns; - if (!PyArg_ParseTuple(args, "O&Oi", &PyObjectToUniqueID, &function_id, - &arguments, &num_returns)) { - return -1; - } - size_t size = PyList_Size(arguments); - /* Determine the size of pass by value data in bytes. */ - size_t value_data_bytes = 0; - for (size_t i = 0; i < size; ++i) { - PyObject *arg = PyList_GetItem(arguments, i); - if (!PyObject_IsInstance(arg, (PyObject *) &PyObjectIDType)) { - PyObject *data = PyMarshal_WriteObjectToString(arg, MARSHAL_VERSION); - value_data_bytes += PyString_Size(data); - utarray_push_back(val_repr_ptrs, &data); - } - } - /* Construct the task specification. */ - int val_repr_index = 0; - self->spec = - alloc_task_spec(function_id, size, num_returns, value_data_bytes); - for (size_t i = 0; i < size; ++i) { - PyObject *arg = PyList_GetItem(arguments, i); - if (PyObject_IsInstance(arg, (PyObject *) &PyObjectIDType)) { - task_args_add_ref(self->spec, ((PyObjectID *) arg)->object_id); - } else { - PyObject *data = - *((PyObject **) utarray_eltptr(val_repr_ptrs, val_repr_index)); - task_args_add_val(self->spec, (uint8_t *) PyString_AS_STRING(data), - PyString_GET_SIZE(data)); - Py_DECREF(data); - val_repr_index += 1; - } - } - utarray_free(val_repr_ptrs); - return 0; -} - -static void PyTask_dealloc(PyTask *self) { - free_task_spec(self->spec); - Py_TYPE(self)->tp_free((PyObject *) self); -} - -static PyObject *PyTask_function_id(PyObject *self) { - function_id function_id = *task_function(((PyTask *) self)->spec); - return PyObjectID_make(function_id); -} - -static PyObject *PyTask_arguments(PyObject *self, PyObject *args) { - int arg_index; - task_spec *spec = ((PyTask *) self)->spec; - if (!PyArg_ParseTuple(args, "i", &arg_index)) { - return NULL; - } - if (task_arg_type(spec, arg_index) == ARG_BY_REF) { - object_id object_id = *task_arg_id(spec, arg_index); - return PyObjectID_make(object_id); - } else { - PyObject *s = PyMarshal_ReadObjectFromString( - (char *) task_arg_val(spec, arg_index), - (Py_ssize_t) task_arg_length(spec, arg_index)); - Py_DECREF(s); - Py_RETURN_NONE; - } -} - -static PyObject *PyTask_returns(PyObject *self, PyObject *args) { - int ret_index; - if (!PyArg_ParseTuple(args, "i", &ret_index)) { - return NULL; - } - object_id object_id = *task_return(((PyTask *) self)->spec, ret_index); - return PyObjectID_make(object_id); -} - -static PyMethodDef PyTask_methods[] = { - {"function_id", (PyCFunction) PyTask_function_id, METH_NOARGS, - "Return the function id associated with this task."}, - {"arguments", (PyCFunction) PyTask_arguments, METH_VARARGS, - "Return the i-th argument of the task."}, - {"returns", (PyCFunction) PyTask_returns, METH_VARARGS, - "Return the i-th object reference of the task."}, - {NULL} /* Sentinel */ -}; - -static PyTypeObject PyTaskType = { - PyObject_HEAD_INIT(NULL) 0, /* ob_size */ - "task.Task", /* tp_name */ - sizeof(PyTask), /* tp_basicsize */ - 0, /* tp_itemsize */ - (destructor) PyTask_dealloc, /* tp_dealloc */ - 0, /* tp_print */ - 0, /* tp_getattr */ - 0, /* tp_setattr */ - 0, /* tp_compare */ - 0, /* tp_repr */ - 0, /* tp_as_number */ - 0, /* tp_as_sequence */ - 0, /* tp_as_mapping */ - 0, /* tp_hash */ - 0, /* tp_call */ - 0, /* tp_str */ - 0, /* tp_getattro */ - 0, /* tp_setattro */ - 0, /* tp_as_buffer */ - Py_TPFLAGS_DEFAULT, /* tp_flags */ - "Task object", /* tp_doc */ - 0, /* tp_traverse */ - 0, /* tp_clear */ - 0, /* tp_richcompare */ - 0, /* tp_weaklistoffset */ - 0, /* tp_iter */ - 0, /* tp_iternext */ - PyTask_methods, /* tp_methods */ - 0, /* tp_members */ - 0, /* tp_getset */ - 0, /* tp_base */ - 0, /* tp_dict */ - 0, /* tp_descr_get */ - 0, /* tp_descr_set */ - 0, /* tp_dictoffset */ - (initproc) PyTask_init, /* tp_init */ - 0, /* tp_alloc */ - PyType_GenericNew, /* tp_new */ -}; - -static PyMethodDef common_methods[] = { - {"check_simple_value", check_simple_value, METH_VARARGS, - "Should the object be passed by value?"}, - {NULL} /* Sentinel */ -}; - -#ifndef PyMODINIT_FUNC /* declarations for DLL import/export */ -#define PyMODINIT_FUNC void -#endif - -PyMODINIT_FUNC initcommon(void) { - PyObject *m; - - if (PyType_Ready(&PyTaskType) < 0) - return; - - if (PyType_Ready(&PyObjectIDType) < 0) - return; - - m = Py_InitModule3("common", common_methods, - "Example module that creates an extension type."); - - Py_INCREF(&PyTaskType); - PyModule_AddObject(m, "Task", (PyObject *) &PyTaskType); - - Py_INCREF(&PyObjectIDType); - PyModule_AddObject(m, "ObjectID", (PyObject *) &PyObjectIDType); - - char common_error[] = "common.error"; - CommonError = PyErr_NewException(common_error, NULL, NULL); - Py_INCREF(CommonError); - PyModule_AddObject(m, "common_error", CommonError); -} diff --git a/test/test.py b/test/test.py index bbed21376..a40d2045c 100644 --- a/test/test.py +++ b/test/test.py @@ -38,7 +38,7 @@ COMPLEX_OBJECTS = (BASE_COMPLEX_OBJECTS + TUPLE_COMPLEX_OBJECTS + DICT_COMPLEX_OBJECTS) -class TestPlasmaClient(unittest.TestCase): +class TestSerialization(unittest.TestCase): def test_serialize_by_value(self): @@ -47,5 +47,55 @@ class TestPlasmaClient(unittest.TestCase): for val in COMPLEX_OBJECTS: self.assertFalse(common.check_simple_value(val)) +class TestObjectID(unittest.TestCase): + + def test_create_object_id(self): + object_id = common.ObjectID(20 * "a") + +class TestTask(unittest.TestCase): + + def test_create_task(self): + # TODO(rkn): The function ID should be a FunctionID object, not an ObjectID. + function_id = common.ObjectID(20 * "a") + object_ids = [common.ObjectID(20 * chr(i)) for i in range(256)] + args_list = [ + [], + 1 * [1], + 10 * [1], + 100 * [1], + 1000 * [1], + 1 * ["a"], + 10 * ["a"], + 100 * ["a"], + 1000 * ["a"], + [1, 1.3, 2L, 1L << 100, "hi", u"hi", [1, 2]], + object_ids[:1], + object_ids[:2], + object_ids[:3], + object_ids[:4], + object_ids[:5], + object_ids[:10], + object_ids[:100], + object_ids[:256], + [1, object_ids[0]], + [object_ids[0], "a"], + [1, object_ids[0], "a"], + [object_ids[0], 1, object_ids[1], "a"], + object_ids[:3] + [1, "hi", 2.3] + object_ids[:5], + object_ids + 100 * ["a"] + object_ids + ] + for args in args_list: + for num_return_vals in [0, 1, 2, 3, 5, 10, 100]: + task = common.Task(function_id, args, num_return_vals) + self.assertEqual(function_id.id(), task.function_id().id()) + retrieved_args = task.arguments() + self.assertEqual(num_return_vals, len(task.returns())) + self.assertEqual(len(args), len(retrieved_args)) + for i in range(len(retrieved_args)): + if isinstance(retrieved_args[i], common.ObjectID): + self.assertEqual(retrieved_args[i].id(), args[i].id()) + else: + self.assertEqual(retrieved_args[i], args[i]) + if __name__ == "__main__": unittest.main(verbosity=2) From 7be1a93d64ca36fc639e11f81de1483e0bd17b8c Mon Sep 17 00:00:00 2001 From: Robert Nishihara Date: Thu, 6 Oct 2016 19:16:09 -0700 Subject: [PATCH 27/36] Move common C extension module into a different C file. (#35) * Move common C extension module into a different C file so that the actual definitions can be more easily included in other C extensions. * Rename common_extension_module -> common_module. --- lib/python/common_extension.c | 36 +-------------------------------- lib/python/common_extension.h | 2 ++ lib/python/common_module.c | 38 +++++++++++++++++++++++++++++++++++ lib/python/setup.py | 2 +- 4 files changed, 42 insertions(+), 36 deletions(-) create mode 100644 lib/python/common_module.c diff --git a/lib/python/common_extension.c b/lib/python/common_extension.c index 556fea443..46ae9e560 100644 --- a/lib/python/common_extension.c +++ b/lib/python/common_extension.c @@ -197,7 +197,7 @@ static PyMethodDef PyTask_methods[] = { {NULL} /* Sentinel */ }; -static PyTypeObject PyTaskType = { +PyTypeObject PyTaskType = { PyObject_HEAD_INIT(NULL) 0, /* ob_size */ "task.Task", /* tp_name */ sizeof(PyTask), /* tp_basicsize */ @@ -325,37 +325,3 @@ PyObject *check_simple_value(PyObject *self, PyObject *args) { } Py_RETURN_FALSE; } - -static PyMethodDef common_methods[] = { - {"check_simple_value", check_simple_value, METH_VARARGS, - "Should the object be passed by value?"}, - {NULL} /* Sentinel */ -}; - -#ifndef PyMODINIT_FUNC /* declarations for DLL import/export */ -#define PyMODINIT_FUNC void -#endif - -PyMODINIT_FUNC initcommon(void) { - PyObject *m; - - if (PyType_Ready(&PyTaskType) < 0) - return; - - if (PyType_Ready(&PyObjectIDType) < 0) - return; - - m = Py_InitModule3("common", common_methods, - "Example module that creates an extension type."); - - Py_INCREF(&PyTaskType); - PyModule_AddObject(m, "Task", (PyObject *) &PyTaskType); - - Py_INCREF(&PyObjectIDType); - PyModule_AddObject(m, "ObjectID", (PyObject *) &PyObjectIDType); - - char common_error[] = "common.error"; - CommonError = PyErr_NewException(common_error, NULL, NULL); - Py_INCREF(CommonError); - PyModule_AddObject(m, "common_error", CommonError); -} diff --git a/lib/python/common_extension.h b/lib/python/common_extension.h index 578c548d0..1fce38e42 100644 --- a/lib/python/common_extension.h +++ b/lib/python/common_extension.h @@ -24,6 +24,8 @@ typedef struct { extern PyTypeObject PyObjectIDType; +extern PyTypeObject PyTaskType; + int PyObjectToUniqueID(PyObject *object, object_id *objectid); PyObject *PyObjectID_make(object_id object_id); diff --git a/lib/python/common_module.c b/lib/python/common_module.c new file mode 100644 index 000000000..d5222cd87 --- /dev/null +++ b/lib/python/common_module.c @@ -0,0 +1,38 @@ +#include +#include "node.h" + +#include "common_extension.h" + +static PyMethodDef common_methods[] = { + {"check_simple_value", check_simple_value, METH_VARARGS, + "Should the object be passed by value?"}, + {NULL} /* Sentinel */ +}; + +#ifndef PyMODINIT_FUNC /* declarations for DLL import/export */ +#define PyMODINIT_FUNC void +#endif + +PyMODINIT_FUNC initcommon(void) { + PyObject *m; + + if (PyType_Ready(&PyTaskType) < 0) + return; + + if (PyType_Ready(&PyObjectIDType) < 0) + return; + + m = Py_InitModule3("common", common_methods, + "A module for common types. This is used for testing."); + + Py_INCREF(&PyTaskType); + PyModule_AddObject(m, "Task", (PyObject *) &PyTaskType); + + Py_INCREF(&PyObjectIDType); + PyModule_AddObject(m, "ObjectID", (PyObject *) &PyObjectIDType); + + char common_error[] = "common.error"; + CommonError = PyErr_NewException(common_error, NULL, NULL); + Py_INCREF(CommonError); + PyModule_AddObject(m, "common_error", CommonError); +} diff --git a/lib/python/setup.py b/lib/python/setup.py index db915b06b..8ef8a1abd 100644 --- a/lib/python/setup.py +++ b/lib/python/setup.py @@ -1,7 +1,7 @@ from setuptools import setup, find_packages, Extension common_module = Extension("common", - sources=["common_extension.c"], + sources=["common_module.c", "common_extension.c"], include_dirs=["../../", "../../thirdparty"], extra_objects=["../../build/libcommon.a"], extra_compile_args=["--std=c99", "-Werror"]) From 7f515113fa3a465a5af6ed5e87fab84788683f0c Mon Sep 17 00:00:00 2001 From: Robert Nishihara Date: Fri, 7 Oct 2016 15:20:56 -0700 Subject: [PATCH 28/36] Generate return object IDs in the task constructor. (#36) --- lib/python/common_extension.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/lib/python/common_extension.c b/lib/python/common_extension.c index 46ae9e560..efc4b9e01 100644 --- a/lib/python/common_extension.c +++ b/lib/python/common_extension.c @@ -131,6 +131,7 @@ static int PyTask_init(PyTask *self, PyObject *args, PyObject *kwds) { int val_repr_index = 0; self->spec = alloc_task_spec(function_id, size, num_returns, value_data_bytes); + /* Add the task arguments. */ for (size_t i = 0; i < size; ++i) { PyObject *arg = PyList_GetItem(arguments, i); if (PyObject_IsInstance(arg, (PyObject *) &PyObjectIDType)) { @@ -145,6 +146,14 @@ static int PyTask_init(PyTask *self, PyObject *args, PyObject *kwds) { } } utarray_free(val_repr_ptrs); + /* Generate and add the object IDs for the return values. */ + for (size_t i = 0; i < num_returns; ++i) { + /* TODO(rkn): Later, this should be computed as a deterministic hash of (1) + * the contents of the task, (2) the index i, and (3) a counter of the + * number of tasks launched so far by the parent task. For now, we generate + * it randomly. */ + *task_return(self->spec, i) = globally_unique_id(); + } return 0; } From 6290cab750566bbb65449122e767484649aee2d7 Mon Sep 17 00:00:00 2001 From: Philipp Moritz Date: Wed, 12 Oct 2016 00:23:40 -0700 Subject: [PATCH 29/36] fix compiler warning for linux --- event_loop.c | 4 ++-- event_loop.h | 21 ++++++++++++--------- logging.c | 2 +- test/db_tests.c | 4 ++-- test/redis_tests.c | 4 ++-- test/task_tests.c | 2 +- 6 files changed, 20 insertions(+), 17 deletions(-) diff --git a/event_loop.c b/event_loop.c index 6928705e7..7e7835445 100644 --- a/event_loop.c +++ b/event_loop.c @@ -48,8 +48,8 @@ int64_t event_loop_add_timer(event_loop *loop, return aeCreateTimeEvent(loop, milliseconds, handler, context, NULL); } -void event_loop_remove_timer(event_loop *loop, int64_t id) { - int err = aeDeleteTimeEvent(loop, id); +void event_loop_remove_timer(event_loop *loop, timer_id timer_id) { + int err = aeDeleteTimeEvent(loop, timer_id); CHECK(err == AE_OK); /* timer id found? */ } diff --git a/event_loop.h b/event_loop.h index bb6afdb93..10762ee89 100644 --- a/event_loop.h +++ b/event_loop.h @@ -4,6 +4,8 @@ #include #include "ae/ae.h" +typedef long long timer_id; + typedef aeEventLoop event_loop; /* File descriptor is readable. */ @@ -12,6 +14,9 @@ typedef aeEventLoop event_loop; /* File descriptor is writable. */ #define EVENT_LOOP_WRITE AE_WRITABLE +/* Constant specifying that the timer is done and it will be removed. */ +#define EVENT_LOOP_TIMER_DONE AE_NOMORE + /* Signature of the handler that will be called when there is a new event * on the file descriptor that this handler has been registered for. The * context is the one that was passed into add_file by the user. The @@ -24,10 +29,12 @@ typedef void (*event_loop_file_handler)(event_loop *loop, /* This handler will be called when a timer times out. The id of the timer * as well as the context that was specified when registering this handler - * are passed as arguments. */ -typedef int64_t (*event_loop_timer_handler)(event_loop *loop, - int64_t id, - void *context); + * are passed as arguments. The return is the number of milliseconds the + * timer shall be reset to or EVENT_LOOP_TIMER_DONE if the timer shall + * not triggered again. */ +typedef int (*event_loop_timer_handler)(event_loop *loop, + timer_id timer_id, + void *context); /* Create and return a new event loop. */ event_loop *event_loop_create(); @@ -58,12 +65,8 @@ int64_t event_loop_add_timer(event_loop *loop, event_loop_timer_handler handler, void *context); -/* Reset the timer timeout to a given number of milliseconds. - * NOTE: This is not implemented yet. */ -void event_loop_reset_timer(event_loop *loop, int64_t id, int64_t milliseconds); - /* Remove a registered time event handler from the event loop. */ -void event_loop_remove_timer(event_loop *loop, int64_t id); +void event_loop_remove_timer(event_loop *loop, timer_id timer_id); /* Run the event loop. */ void event_loop_run(event_loop *loop); diff --git a/logging.c b/logging.c index 1a8e96820..301ed4b11 100644 --- a/logging.c +++ b/logging.c @@ -12,7 +12,7 @@ static const char *log_fmt = struct ray_logger_impl { /* String that identifies this client type. */ - char *client_type; + const char *client_type; /* Suppress all log messages below this level. */ int log_level; /* Whether or not we have a direct connection to Redis. */ diff --git a/test/db_tests.c b/test/db_tests.c index 6eb592e45..95e986e33 100644 --- a/test/db_tests.c +++ b/test/db_tests.c @@ -38,9 +38,9 @@ void test_callback(object_id object_id, free(manager_vector); } -int64_t timeout_handler(event_loop *loop, int64_t id, void *context) { +int timeout_handler(event_loop *loop, timer_id timer_id, void *context) { event_loop_stop(loop); - return -1; + return EVENT_LOOP_TIMER_DONE; } TEST object_table_lookup_test(void) { diff --git a/test/redis_tests.c b/test/redis_tests.c index 2277174c0..0a50c7d1b 100644 --- a/test/redis_tests.c +++ b/test/redis_tests.c @@ -86,9 +86,9 @@ void redis_accept_callback(event_loop *loop, context); } -int64_t timeout_handler(event_loop *loop, int64_t id, void *context) { +int timeout_handler(event_loop *loop, timer_id timer_id, void *context) { event_loop_stop(loop); - return -1; + return EVENT_LOOP_TIMER_DONE; } TEST async_redis_socket_test(void) { diff --git a/test/task_tests.c b/test/task_tests.c index f72a0e2c2..759e9c8b7 100644 --- a/test/task_tests.c +++ b/test/task_tests.c @@ -49,7 +49,7 @@ TEST send_task(void) { *task_return(task, 1) = globally_unique_id(); int fd[2]; socketpair(AF_UNIX, SOCK_STREAM, 0, fd); - write_message(fd[0], SUBMIT_TASK, task_size(task), task); + write_message(fd[0], SUBMIT_TASK, task_size(task), (uint8_t*) task); int64_t type; int64_t length; uint8_t *message; From 50fb53fd915618070795de3a1e47757a74178dfb Mon Sep 17 00:00:00 2001 From: Philipp Moritz Date: Wed, 12 Oct 2016 00:56:04 -0700 Subject: [PATCH 30/36] fix compiler warnings on macOS --- Makefile | 2 +- logging.c | 6 ++++-- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/Makefile b/Makefile index dae76e87b..9a0f75e06 100644 --- a/Makefile +++ b/Makefile @@ -1,5 +1,5 @@ CC = gcc -CFLAGS = -g -Wall --std=c99 -D_XOPEN_SOURCE=500 -D_POSIX_C_SOURCE=200809L -fPIC -I. -Ithirdparty -Ithirdparty/ae +CFLAGS = -g -Wall --std=c99 -D_XOPEN_SOURCE=500 -D_POSIX_C_SOURCE=200809L -fPIC -I. -Ithirdparty -Ithirdparty/ae -Wno-typedef-redefinition BUILD = build all: hiredis $(BUILD)/libcommon.a diff --git a/logging.c b/logging.c index 301ed4b11..09cad7856 100644 --- a/logging.c +++ b/logging.c @@ -1,5 +1,7 @@ #include "logging.h" +#include +#include #include #include @@ -52,13 +54,13 @@ void ray_log(ray_logger *logger, UT_string *timestamp; utstring_new(timestamp); gettimeofday(&tv, NULL); - utstring_printf(timestamp, "%ld.%ld", tv.tv_sec, tv.tv_usec); + utstring_printf(timestamp, "%ld.%ld", tv.tv_sec, (long) tv.tv_usec); UT_string *origin_id; utstring_new(origin_id); if (logger->is_direct) { db_handle *db = (db_handle *) logger->conn; - utstring_printf(origin_id, "%ld:%s", db->client_id, ""); + utstring_printf(origin_id, "%" PRId64 ":%s", db->client_id, ""); redisAsyncCommand(db->context, NULL, NULL, log_fmt, utstring_body(timestamp), logger->client_type, utstring_body(origin_id), log_levels[log_level], From 832888d4736da0133d91b6e1cacb1ebfef77c6e9 Mon Sep 17 00:00:00 2001 From: Philipp Moritz Date: Wed, 12 Oct 2016 00:59:22 -0700 Subject: [PATCH 31/36] fix formating --- event_loop.h | 2 +- test/task_tests.c | 10 +++++----- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/event_loop.h b/event_loop.h index 10762ee89..7f0659ba7 100644 --- a/event_loop.h +++ b/event_loop.h @@ -31,7 +31,7 @@ typedef void (*event_loop_file_handler)(event_loop *loop, * as well as the context that was specified when registering this handler * are passed as arguments. The return is the number of milliseconds the * timer shall be reset to or EVENT_LOOP_TIMER_DONE if the timer shall - * not triggered again. */ + * not be triggered again. */ typedef int (*event_loop_timer_handler)(event_loop *loop, timer_id timer_id, void *context); diff --git a/test/task_tests.c b/test/task_tests.c index 759e9c8b7..77f43bbf0 100644 --- a/test/task_tests.c +++ b/test/task_tests.c @@ -19,10 +19,10 @@ TEST task_test(void) { unique_id arg1 = globally_unique_id(); ASSERT(task_args_add_ref(task, arg1) == 0); - ASSERT(task_args_add_val(task, (uint8_t*) "hello", 5) == 1); + ASSERT(task_args_add_val(task, (uint8_t *) "hello", 5) == 1); unique_id arg2 = globally_unique_id(); ASSERT(task_args_add_ref(task, arg2) == 2); - ASSERT(task_args_add_val(task, (uint8_t*) "world", 5) == 3); + ASSERT(task_args_add_val(task, (uint8_t *) "world", 5) == 3); unique_id ret0 = globally_unique_id(); unique_id ret1 = globally_unique_id(); @@ -30,10 +30,10 @@ TEST task_test(void) { memcpy(task_return(task, 1), &ret1, sizeof(ret1)); ASSERT(memcmp(task_arg_id(task, 0), &arg1, sizeof(arg1)) == 0); - ASSERT(memcmp(task_arg_val(task, 1), (uint8_t*) "hello", + ASSERT(memcmp(task_arg_val(task, 1), (uint8_t *) "hello", task_arg_length(task, 1)) == 0); ASSERT(memcmp(task_arg_id(task, 2), &arg2, sizeof(arg2)) == 0); - ASSERT(memcmp(task_arg_val(task, 3), (uint8_t*) "world", + ASSERT(memcmp(task_arg_val(task, 3), (uint8_t *) "world", task_arg_length(task, 3)) == 0); ASSERT(memcmp(task_return(task, 0), &ret0, sizeof(unique_id)) == 0); @@ -49,7 +49,7 @@ TEST send_task(void) { *task_return(task, 1) = globally_unique_id(); int fd[2]; socketpair(AF_UNIX, SOCK_STREAM, 0, fd); - write_message(fd[0], SUBMIT_TASK, task_size(task), (uint8_t*) task); + write_message(fd[0], SUBMIT_TASK, task_size(task), (uint8_t *) task); int64_t type; int64_t length; uint8_t *message; From 182985015cb751fde6cb61e236afaf496d5939aa Mon Sep 17 00:00:00 2001 From: Philipp Moritz Date: Wed, 12 Oct 2016 13:20:57 -0700 Subject: [PATCH 32/36] Make warnings errors --- Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Makefile b/Makefile index 9a0f75e06..6982b7945 100644 --- a/Makefile +++ b/Makefile @@ -1,5 +1,5 @@ CC = gcc -CFLAGS = -g -Wall --std=c99 -D_XOPEN_SOURCE=500 -D_POSIX_C_SOURCE=200809L -fPIC -I. -Ithirdparty -Ithirdparty/ae -Wno-typedef-redefinition +CFLAGS = -g -Wall --std=c99 -D_XOPEN_SOURCE=500 -D_POSIX_C_SOURCE=200809L -fPIC -I. -Ithirdparty -Ithirdparty/ae -Wno-typedef-redefinition -Werror BUILD = build all: hiredis $(BUILD)/libcommon.a From e57b87928c7566137030136861a7a652e877d115 Mon Sep 17 00:00:00 2001 From: Stephanie Wang Date: Tue, 18 Oct 2016 12:38:30 -0700 Subject: [PATCH 33/36] Fixes for implementing Plasma fetch (#39) * Add ability to pass callback context to object table lookup * Propagate errors during socket writes up to caller. * Use recv and MSG_WAITALL flag instead of looping read * Error checking in write_bytes * Method to listen on a network port * Revert "Use recv and MSG_WAITALL flag instead of looping read" This reverts commit 32d9333bc6a185729aadb4b41b70b3d7f150a9c2. * Some documentation * Clearer documentation * Fix bug where database clients were getting assigned the same ID * Regression test for unique client IDs --- io.c | 161 ++++++++++++++++++++++++++++++++++--------- io.h | 3 +- state/object_table.h | 6 +- state/redis.c | 18 ++++- state/redis.h | 11 +++ test/db_tests.c | 31 ++++++++- 6 files changed, 188 insertions(+), 42 deletions(-) diff --git a/io.c b/io.c index 32f3ed4b9..1f1125bc3 100644 --- a/io.c +++ b/io.c @@ -8,14 +8,69 @@ #include #include #include +#include +#include #include #include "common.h" -/* Binds to a Unix domain streaming socket at the given - * pathname. Removes any existing file at the pathname. Returns - * a file descriptor for the socket, or -1 if an error - * occurred. */ +/** + * Binds to an Internet socket at the given port. Removes any existing file at + * the pathname. Returns a non-blocking file descriptor for the socket, or -1 + * if an error occurred. + * + * @note Since the returned file descriptor is non-blocking, it is not + * recommended to use the Linux read and write calls directly, since these + * might read or write a partial message. Instead, use the provided + * write_message and read_message methods. + * + * @param port The port to bind to. + * @return A non-blocking file descriptor for the socket, or -1 if an error + * occurs. + */ +int bind_inet_sock(const int port) { + struct sockaddr_in name; + int socket_fd = socket(PF_INET, SOCK_STREAM, 0); + if (socket_fd < 0) { + LOG_ERR("socket() failed for port %d.", port); + return -1; + } + name.sin_family = AF_INET; + name.sin_port = htons(port); + name.sin_addr.s_addr = htonl(INADDR_ANY); + int on = 1; + /* TODO(pcm): http://stackoverflow.com/q/1150635 */ + if (ioctl(socket_fd, FIONBIO, (char *) &on) < 0) { + LOG_ERR("ioctl failed"); + close(socket_fd); + return -1; + } + if (setsockopt(socket_fd, SOL_SOCKET, SO_REUSEADDR, &on, sizeof(on)) < 0) { + LOG_ERR("setsockopt failed for port %d", port); + close(socket_fd); + return -1; + } + if (bind(socket_fd, (struct sockaddr *) &name, sizeof(name)) < 0) { + LOG_ERR("Bind failed for port %d", port); + close(socket_fd); + return -1; + } + if (listen(socket_fd, 5) == -1) { + LOG_ERR("Could not listen to socket %d", port); + close(socket_fd); + return -1; + } + return socket_fd; +} + +/** + * Binds to a Unix domain streaming socket at the given + * pathname. Removes any existing file at the pathname. + * + * @param socket_pathname The pathname for the socket. + * @return A blocking file descriptor for the socket, or -1 if an error + * occurs. + */ int bind_ipc_sock(const char *socket_pathname) { struct sockaddr_un socket_address; int socket_fd = socket(AF_UNIX, SOCK_STREAM, 0); @@ -27,9 +82,9 @@ int bind_ipc_sock(const char *socket_pathname) { int on = 1; if (setsockopt(socket_fd, SOL_SOCKET, SO_REUSEADDR, (char *) &on, sizeof(on)) < 0) { - LOG_ERR("setsockopt failed"); + LOG_ERR("setsockopt failed for pathname %s", socket_pathname); close(socket_fd); - exit(-1); + return -1; } unlink(socket_pathname); @@ -37,6 +92,7 @@ int bind_ipc_sock(const char *socket_pathname) { socket_address.sun_family = AF_UNIX; if (strlen(socket_pathname) + 1 > sizeof(socket_address.sun_path)) { LOG_ERR("Socket pathname is too long."); + close(socket_fd); return -1; } strncpy(socket_address.sun_path, socket_pathname, @@ -45,16 +101,22 @@ int bind_ipc_sock(const char *socket_pathname) { if (bind(socket_fd, (struct sockaddr *) &socket_address, sizeof(struct sockaddr_un)) != 0) { LOG_ERR("Bind failed for pathname %s.", socket_pathname); + close(socket_fd); + return -1; + } + if (listen(socket_fd, 5) == -1) { + LOG_ERR("Could not listen to socket %s", socket_pathname); + close(socket_fd); return -1; } - listen(socket_fd, 5); - return socket_fd; } -/* Connects to a Unix domain streaming socket at the given +/** + * Connects to a Unix domain streaming socket at the given * pathname. Returns a file descriptor for the socket, or -1 if - * an error occurred. */ + * an error occurred. + */ int connect_ipc_sock(const char *socket_pathname) { struct sockaddr_un socket_address; int socket_fd; @@ -83,8 +145,10 @@ int connect_ipc_sock(const char *socket_pathname) { return socket_fd; } -/* Accept a new client connection on the given socket - * descriptor. Returns a descriptor for the new socket. */ +/** + * Accept a new client connection on the given socket + * descriptor. Returns a descriptor for the new socket. + */ int accept_client(int socket_fd) { int client_fd = accept(socket_fd, NULL, NULL); if (client_fd < 0) { @@ -95,55 +159,80 @@ int accept_client(int socket_fd) { } /** - * Reliably write a sequence of bytes into a file descriptor. This will block - * until one of the following happens: (1) there is an error (2) end of file, - * or (3) all length bytes have been written. + * Write a sequence of bytes into a file descriptor. This will block until one + * of the following happens: (1) there is an error (2) end of file, or (3) all + * length bytes have been written. * - * @param fd The file descriptor to write to. + * @param fd The file descriptor to write to. It can be non-blocking. * @param cursor The cursor pointing to the beginning of the bytes to send. * @param length The size of the bytes sequence to write. - * @return Void. + * @return int Whether there was an error while writing. 0 corresponds to + * success and -1 corresponds to an error (errno will be set). */ -void write_bytes(int fd, uint8_t *cursor, size_t length) { +int write_bytes(int fd, uint8_t *cursor, size_t length) { ssize_t nbytes = 0; while (length > 0) { /* While we haven't written the whole message, write to the file * descriptor, advance the cursor, and decrease the amount left to write. */ nbytes = write(fd, cursor, length); - CHECK(nbytes > 0); + if (nbytes < 0) { + if (errno == EAGAIN || errno == EWOULDBLOCK) { + continue; + } + /* TODO(swang): Return the error instead of exiting. */ + /* Force an exit if there was any other type of error. */ + CHECK(nbytes < 0); + } + if (nbytes == 0) { + return -1; + } cursor += nbytes; length -= nbytes; } + return 0; } /** * Write a sequence of bytes on a file descriptor. The bytes should then be read * by read_message. * - * @param fd The file descriptor to write to. + * @param fd The file descriptor to write to. It can be non-blocking. * @param type The type of the message to send. * @param length The size in bytes of the bytes parameter. * @param bytes The address of the message to send. - * @return Void. + * @return int Whether there was an error while writing. 0 corresponds to + * success and -1 corresponds to an error (errno will be set). */ -void write_message(int fd, int64_t type, int64_t length, uint8_t *bytes) { - write_bytes(fd, (uint8_t *) &type, sizeof(type)); - write_bytes(fd, (uint8_t *) &length, sizeof(length)); - write_bytes(fd, bytes, length * sizeof(char)); +int write_message(int fd, int64_t type, int64_t length, uint8_t *bytes) { + int closed; + closed = write_bytes(fd, (uint8_t *) &type, sizeof(type)); + if (closed) { + return closed; + } + closed = write_bytes(fd, (uint8_t *) &length, sizeof(length)); + if (closed) { + return closed; + } + closed = write_bytes(fd, bytes, length * sizeof(char)); + if (closed) { + return closed; + } + return 0; } /** - * Reliably read a sequence of bytes from a file descriptor into a buffer. This - * will block until one of the following happens: (1) there is an error (2) end - * of file, or (3) all length bytes have been written. + * Read a sequence of bytes from a file descriptor into a buffer. This will + * block until one of the following happens: (1) there is an error (2) end of + * file, or (3) all length bytes have been written. * * @note The buffer pointed to by cursor must already have length number of * bytes allocated before calling this method. * - * @param fd The file descriptor to read from. + * @param fd The file descriptor to read from. It can be non-blocking. * @param cursor The cursor pointing to the beginning of the buffer. * @param length The size of the byte sequence to read. - * @return Void. + * @return int Whether there was an error while writing. 0 corresponds to + * success and -1 corresponds to an error (errno will be set). */ int read_bytes(int fd, uint8_t *cursor, size_t length) { ssize_t nbytes = 0; @@ -173,14 +262,18 @@ int read_bytes(int fd, uint8_t *cursor, size_t length) { * * @note The caller must free the memory. * - * @param fd The file descriptor to read from. + * @param fd The file descriptor to read from. It can be non-blocking. * @param type The type of the message that is read will be written at this - address. + address. If there was an error while reading, this will be + DISCONNECT_CLIENT. * @param length The size in bytes of the message that is read will be written at this address. This size does not include the bytes used to encode - the type and length. + the type and length. If there was an error while reading, this will + be 0. * @param bytes The address at which to write the pointer to the bytes that are - read and allocated by this function. + read and allocated by this function. If there was an error while + reading, this will be NULL. + * @return Void. */ void read_message(int fd, int64_t *type, int64_t *length, uint8_t **bytes) { diff --git a/io.h b/io.h index 2299806f7..362b85fc1 100644 --- a/io.h +++ b/io.h @@ -14,6 +14,7 @@ enum common_message_type { /* Helper functions for socket communication. */ +int bind_inet_sock(const int port); int bind_ipc_sock(const char *socket_pathname); int connect_ipc_sock(const char *socket_pathname); @@ -21,7 +22,7 @@ int accept_client(int socket_fd); /* Reading and writing data */ -void write_message(int fd, int64_t type, int64_t length, uint8_t *bytes); +int write_message(int fd, int64_t type, int64_t length, uint8_t *bytes); void read_message(int fd, int64_t *type, int64_t *length, uint8_t **bytes); void write_log_message(int fd, char *message); diff --git a/state/object_table.h b/state/object_table.h index e2eb89433..bab54bc7b 100644 --- a/state/object_table.h +++ b/state/object_table.h @@ -6,7 +6,8 @@ * the manager_vector array, but NOT the strings they are pointing to. */ typedef void (*lookup_callback)(object_id object_id, int manager_count, - const char *manager_vector[]); + const char *manager_vector[], + void *context); /* Register a new object with the directory. */ /* TODO(pcm): Retry, print for each attempt. */ @@ -20,4 +21,5 @@ void object_table_remove(db_handle *db, /* Look up entry from the directory */ void object_table_lookup(db_handle *db, object_id object_id, - lookup_callback callback); + lookup_callback callback, + void *context); diff --git a/state/redis.c b/state/redis.c index db008cb5d..d53b0bf15 100644 --- a/state/redis.c +++ b/state/redis.c @@ -56,7 +56,8 @@ db_handle *db_connect(const char *address, num_clients, client_addr, client_port); freeReplyObject(reply); reply = redisCommand(context, "EXEC"); - if (reply) { + CHECK(reply); + if (reply->type != REDIS_REPLY_NIL) { freeReplyObject(reply); break; } @@ -150,17 +151,20 @@ void object_table_get_entry(redisAsyncContext *c, void *r, void *privdata) { HASH_FIND_INT(db->service_cache, &result[j], entry); manager_vector[j] = entry->addr; } - cb_data->callback(cb_data->object_id, manager_count, manager_vector); + cb_data->callback(cb_data->object_id, manager_count, manager_vector, + cb_data->context); free(privdata); free(result); } void object_table_lookup(db_handle *db, object_id object_id, - lookup_callback callback) { + lookup_callback callback, + void *context) { lookup_callback_data *cb_data = malloc(sizeof(lookup_callback_data)); cb_data->callback = callback; cb_data->object_id = object_id; + cb_data->context = context; redisAsyncCommand(db->context, object_table_get_entry, cb_data, "SMEMBERS obj:%b", &object_id.id[0], UNIQUE_ID_SIZE); if (db->context->err) { @@ -230,3 +234,11 @@ void task_log_register_callback(db_handle *db, LOG_REDIS_ERR(db->sub_context, "error in task_log_register_callback"); } } + +int get_client_id(db_handle *db) { + if (db) { + return db->client_id; + } else { + return -1; + } +} diff --git a/state/redis.h b/state/redis.h index 51479f8f0..a3e0555e7 100644 --- a/state/redis.h +++ b/state/redis.h @@ -55,6 +55,8 @@ typedef struct { lookup_callback callback; /* Object ID that is looked up. */ object_id object_id; + /* Data context for the callback. */ + void *context; } lookup_callback_data; void object_table_get_entry(redisAsyncContext *c, void *r, void *privdata); @@ -63,4 +65,13 @@ void object_table_lookup_callback(redisAsyncContext *c, void *r, void *privdata); +/** + * Returns the client ID, according to Redis. + * + * @param db The handle to the Redis database. + * @returns int The client ID for this connection to Redis. If + * this client has no connection to Redis, returns -1. + */ +int get_client_id(db_handle *db); + #endif diff --git a/test/db_tests.c b/test/db_tests.c index 95e986e33..74dd9a850 100644 --- a/test/db_tests.c +++ b/test/db_tests.c @@ -1,6 +1,8 @@ #include "greatest.h" #include +#include +#include #include "event_loop.h" #include "test/example_task.h" @@ -23,7 +25,8 @@ char received_port2[6] = {0}; /* Test if entries have been written to the database. */ void test_callback(object_id object_id, int manager_count, - const char *manager_vector[]) { + const char *manager_vector[], + void *context) { CHECK(manager_count == 2); if (!manager_vector[0] || sscanf(manager_vector[0], "%15[0-9.]:%5[0-9]", received_addr1, @@ -56,7 +59,7 @@ TEST object_table_lookup_test(void) { object_table_add(db2, id); event_loop_add_timer(loop, 100, timeout_handler, NULL); event_loop_run(loop); - object_table_lookup(db1, id, test_callback); + object_table_lookup(db1, id, test_callback, NULL); event_loop_add_timer(loop, 100, timeout_handler, NULL); event_loop_run(loop); int port1 = atoi(received_port1); @@ -130,12 +133,36 @@ TEST task_log_all_test(void) { PASS(); } +TEST unique_client_id_test(void) { + const int num_conns = 50; + + db_handle *db; + pid_t pid = fork(); + for (int i = 0; i < num_conns; ++i) { + db = db_connect("127.0.0.1", 6379, "plasma_manager", manager_addr, + manager_port1); + db_disconnect(db); + } + if (pid == 0) { + exit(0); + } else { + wait(NULL); + } + + db = db_connect("127.0.0.1", 6379, "plasma_manager", manager_addr, + manager_port1); + ASSERT_EQ(get_client_id(db), num_conns * 2); + db_disconnect(db); + PASS(); +} + SUITE(db_tests) { redisContext *context = redisConnect("127.0.0.1", 6379); freeReplyObject(redisCommand(context, "FLUSHALL")); RUN_REDIS_TEST(context, object_table_lookup_test); RUN_REDIS_TEST(context, task_log_test); RUN_REDIS_TEST(context, task_log_all_test); + RUN_REDIS_TEST(context, unique_client_id_test); redisFree(context); } From da3a3127e095f679651119f0debfafcade1b0b94 Mon Sep 17 00:00:00 2001 From: Stephanie Wang Date: Tue, 18 Oct 2016 15:12:41 -0700 Subject: [PATCH 34/36] Move get_client_id to db.h (#40) --- state/db.h | 9 +++++++++ state/redis.h | 9 --------- 2 files changed, 9 insertions(+), 9 deletions(-) diff --git a/state/db.h b/state/db.h index e6ca089a0..50536e271 100644 --- a/state/db.h +++ b/state/db.h @@ -20,4 +20,13 @@ void db_attach(db_handle *db, event_loop *loop); /* Disconnect from the global system store. */ void db_disconnect(db_handle *db); +/** + * Returns the client ID, according to the database. + * + * @param db The handle to the database. + * @returns int The client ID for this connection to the database. If + * this client has no connection to the database, returns -1. + */ +int get_client_id(db_handle *db); + #endif diff --git a/state/redis.h b/state/redis.h index a3e0555e7..cf368b992 100644 --- a/state/redis.h +++ b/state/redis.h @@ -65,13 +65,4 @@ void object_table_lookup_callback(redisAsyncContext *c, void *r, void *privdata); -/** - * Returns the client ID, according to Redis. - * - * @param db The handle to the Redis database. - * @returns int The client ID for this connection to Redis. If - * this client has no connection to Redis, returns -1. - */ -int get_client_id(db_handle *db); - #endif From db6375701efe239ba8124741a8dacaff6344dc89 Mon Sep 17 00:00:00 2001 From: Robert Nishihara Date: Fri, 21 Oct 2016 15:42:29 -0700 Subject: [PATCH 35/36] Prevent ObjectIDs from being pickled. (#42) --- lib/python/common_extension.c | 8 ++++++++ test/test.py | 18 ++++++++++++++++++ 2 files changed, 26 insertions(+) diff --git a/lib/python/common_extension.c b/lib/python/common_extension.c index efc4b9e01..14f3292d9 100644 --- a/lib/python/common_extension.c +++ b/lib/python/common_extension.c @@ -51,9 +51,17 @@ static PyObject *PyObjectID_id(PyObject *self) { UNIQUE_ID_SIZE); } +static PyObject *PyObjectID___reduce__(PyObjectID *self) { + PyErr_SetString(CommonError, "ObjectID objects cannot be serialized."); + return NULL; +} + static PyMethodDef PyObjectID_methods[] = { {"id", (PyCFunction) PyObjectID_id, METH_NOARGS, "Return the hash associated with this ObjectID"}, + {"__reduce__", (PyCFunction) PyObjectID___reduce__, METH_NOARGS, + "Say how to pickle this ObjectID. This raises an exception to prevent" + "object IDs from being serialized."}, {NULL} /* Sentinel */ }; diff --git a/test/test.py b/test/test.py index a40d2045c..359e8c030 100644 --- a/test/test.py +++ b/test/test.py @@ -1,5 +1,6 @@ from __future__ import print_function +import pickle import unittest import common @@ -52,6 +53,23 @@ class TestObjectID(unittest.TestCase): def test_create_object_id(self): object_id = common.ObjectID(20 * "a") + def test_cannot_pickle_object_ids(self): + object_ids = [common.ObjectID(20 * chr(i)) for i in range(256)] + def f(): + return object_ids + def g(val=object_ids): + return 1 + def h(): + x = object_ids[0] + return 1 + # Make sure that object IDs cannot be pickled (including functions that + # close over object IDs). + self.assertRaises(Exception, lambda : pickling.dumps(object_ids[0])) + self.assertRaises(Exception, lambda : pickling.dumps(object_ids)) + self.assertRaises(Exception, lambda : pickling.dumps(f)) + self.assertRaises(Exception, lambda : pickling.dumps(g)) + self.assertRaises(Exception, lambda : pickling.dumps(h)) + class TestTask(unittest.TestCase): def test_create_task(self): From 1915539c5f1da95c975256dbb3ef38137acec2db Mon Sep 17 00:00:00 2001 From: Robert Nishihara Date: Tue, 25 Oct 2016 13:57:23 -0700 Subject: [PATCH 36/36] Rearrange files to prepare to merge into Ray. --- .clang-format | 9 - .gitignore | 38 -- .gitmodules | 3 - .travis.yml | 50 -- .travis/check-git-clang-format-output.sh | 18 - .travis/git-clang-format | 476 ------------------ LICENSE | 201 -------- install-dependencies.sh | 21 - Makefile => src/common/Makefile | 0 {build => src/common/build}/.gitkeep | 0 common.c => src/common/common.c | 0 common.h => src/common/common.h | 0 {doc => src/common/doc}/tasks.md | 0 event_loop.c => src/common/event_loop.c | 0 event_loop.h => src/common/event_loop.h | 0 io.c => src/common/io.c | 0 io.h => src/common/io.h | 0 .../common/lib}/python/common_extension.c | 0 .../common/lib}/python/common_extension.h | 0 .../common/lib}/python/common_module.c | 0 {lib => src/common/lib}/python/setup.py | 0 logging.c => src/common/logging.c | 0 logging.h => src/common/logging.h | 0 {state => src/common/state}/db.h | 0 {state => src/common/state}/object_table.h | 0 {state => src/common/state}/redis.c | 0 {state => src/common/state}/redis.h | 0 {state => src/common/state}/task_log.h | 0 {state => src/common/state}/task_table.h | 0 task.c => src/common/task.c | 0 task.h => src/common/task.h | 0 {test => src/common/test}/common_tests.c | 0 {test => src/common/test}/db_tests.c | 0 {test => src/common/test}/example_task.h | 0 {test => src/common/test}/io_tests.c | 0 {test => src/common/test}/redis_tests.c | 0 {test => src/common/test}/task_tests.c | 0 {test => src/common/test}/test.py | 0 {thirdparty => src/common/thirdparty}/ae/ae.c | 0 {thirdparty => src/common/thirdparty}/ae/ae.h | 0 .../common/thirdparty}/ae/ae_epoll.c | 0 .../common/thirdparty}/ae/ae_evport.c | 0 .../common/thirdparty}/ae/ae_kqueue.c | 0 .../common/thirdparty}/ae/ae_select.c | 0 .../common/thirdparty}/ae/config.h | 0 .../common/thirdparty}/ae/zmalloc.h | 0 .../common/thirdparty}/build-redis.sh | 0 .../common/thirdparty}/greatest.h | 0 {thirdparty => src/common/thirdparty}/hiredis | 0 .../common/thirdparty}/utarray.h | 0 .../common/thirdparty}/uthash.h | 0 .../common/thirdparty}/utlist.h | 0 .../common/thirdparty}/utstring.h | 0 53 files changed, 816 deletions(-) delete mode 100644 .clang-format delete mode 100644 .gitignore delete mode 100644 .gitmodules delete mode 100644 .travis.yml delete mode 100755 .travis/check-git-clang-format-output.sh delete mode 100755 .travis/git-clang-format delete mode 100644 LICENSE delete mode 100755 install-dependencies.sh rename Makefile => src/common/Makefile (100%) rename {build => src/common/build}/.gitkeep (100%) rename common.c => src/common/common.c (100%) rename common.h => src/common/common.h (100%) rename {doc => src/common/doc}/tasks.md (100%) rename event_loop.c => src/common/event_loop.c (100%) rename event_loop.h => src/common/event_loop.h (100%) rename io.c => src/common/io.c (100%) rename io.h => src/common/io.h (100%) rename {lib => src/common/lib}/python/common_extension.c (100%) rename {lib => src/common/lib}/python/common_extension.h (100%) rename {lib => src/common/lib}/python/common_module.c (100%) rename {lib => src/common/lib}/python/setup.py (100%) rename logging.c => src/common/logging.c (100%) rename logging.h => src/common/logging.h (100%) rename {state => src/common/state}/db.h (100%) rename {state => src/common/state}/object_table.h (100%) rename {state => src/common/state}/redis.c (100%) rename {state => src/common/state}/redis.h (100%) rename {state => src/common/state}/task_log.h (100%) rename {state => src/common/state}/task_table.h (100%) rename task.c => src/common/task.c (100%) rename task.h => src/common/task.h (100%) rename {test => src/common/test}/common_tests.c (100%) rename {test => src/common/test}/db_tests.c (100%) rename {test => src/common/test}/example_task.h (100%) rename {test => src/common/test}/io_tests.c (100%) rename {test => src/common/test}/redis_tests.c (100%) rename {test => src/common/test}/task_tests.c (100%) rename {test => src/common/test}/test.py (100%) rename {thirdparty => src/common/thirdparty}/ae/ae.c (100%) rename {thirdparty => src/common/thirdparty}/ae/ae.h (100%) rename {thirdparty => src/common/thirdparty}/ae/ae_epoll.c (100%) rename {thirdparty => src/common/thirdparty}/ae/ae_evport.c (100%) rename {thirdparty => src/common/thirdparty}/ae/ae_kqueue.c (100%) rename {thirdparty => src/common/thirdparty}/ae/ae_select.c (100%) rename {thirdparty => src/common/thirdparty}/ae/config.h (100%) rename {thirdparty => src/common/thirdparty}/ae/zmalloc.h (100%) rename {thirdparty => src/common/thirdparty}/build-redis.sh (100%) rename {thirdparty => src/common/thirdparty}/greatest.h (100%) rename {thirdparty => src/common/thirdparty}/hiredis (100%) rename {thirdparty => src/common/thirdparty}/utarray.h (100%) rename {thirdparty => src/common/thirdparty}/uthash.h (100%) rename {thirdparty => src/common/thirdparty}/utlist.h (100%) rename {thirdparty => src/common/thirdparty}/utstring.h (100%) diff --git a/.clang-format b/.clang-format deleted file mode 100644 index 89b87e25d..000000000 --- a/.clang-format +++ /dev/null @@ -1,9 +0,0 @@ -BasedOnStyle: Chromium -DerivePointerAlignment: true -IndentCaseLabels: false -PointerAlignment: Right -SpaceAfterCStyleCast: true -AllowShortBlocksOnASingleLine: false -AllowShortCaseLabelsOnASingleLine: false -AllowShortFunctionsOnASingleLine: false -AllowShortIfStatementsOnASingleLine: false diff --git a/.gitignore b/.gitignore deleted file mode 100644 index fff8ef269..000000000 --- a/.gitignore +++ /dev/null @@ -1,38 +0,0 @@ -*~ - -# Object files -*.o -*.ko -*.obj -*.elf - -# Precompiled Headers -*.gch -*.pch - -# Libraries -*.lib -*.a -*.la -*.lo - -# Shared objects (inc. Windows DLLs) -*.dll -*.so -*.so.* -*.dylib - -# Executables -*.exe -*.out -*.app -*.i*86 -*.x86_64 -*.hex - -# Debug files -*.dSYM/ -*.su - -# Build files -build/* diff --git a/.gitmodules b/.gitmodules deleted file mode 100644 index 4026f8268..000000000 --- a/.gitmodules +++ /dev/null @@ -1,3 +0,0 @@ -[submodule "thirdparty/hiredis"] - path = thirdparty/hiredis - url = https://github.com/redis/hiredis diff --git a/.travis.yml b/.travis.yml deleted file mode 100644 index d0e14edf5..000000000 --- a/.travis.yml +++ /dev/null @@ -1,50 +0,0 @@ -sudo: required - -language: generic - -matrix: - include: - - os: linux - dist: trusty - python: "2.7" - - os: linux - dist: trusty - python: "3.5" - - os: osx - osx_image: xcode7 - python: "2.7" - - os: osx - osx_image: xcode7 - python: "3.5" - - os: linux - dist: trusty - env: LINT=1 - before_install: - # In case we ever want to use a different version of clang-format: - #- wget -O - http://apt.llvm.org/llvm-snapshot.gpg.key | sudo apt-key add - - #- echo "deb http://apt.llvm.org/trusty/ llvm-toolchain-trusty main" | sudo tee -a /etc/apt/sources.list > /dev/null - - sudo apt-get update -qq - - sudo apt-get install -qq clang-format-3.8 - install: [] - script: - - .travis/check-git-clang-format-output.sh - - os: linux - dist: trusty - python: "2.7" - env: VALGRIND=1 - before_install: - - sudo apt-get update -qq - - sudo apt-get install -qq valgrind - script: - - make valgrind - -install: - - ./install-dependencies.sh - - make - - make test - - cd lib/python - - python setup.py install --user - - cd ../.. - -script: - - python test/test.py diff --git a/.travis/check-git-clang-format-output.sh b/.travis/check-git-clang-format-output.sh deleted file mode 100755 index d71f78357..000000000 --- a/.travis/check-git-clang-format-output.sh +++ /dev/null @@ -1,18 +0,0 @@ -#!/bin/bash -if [ "$TRAVIS_PULL_REQUEST" == "false" ] ; then - # Not in a pull request, so compare against parent commit - base_commit="HEAD^" - echo "Running clang-format against parent commit $(git rev-parse $base_commit)" -else - base_commit="$TRAVIS_BRANCH" - echo "Running clang-format against branch $base_commit, with hash $(git rev-parse $base_commit)" -fi -output="$(.travis/git-clang-format --binary clang-format-3.8 --commit $base_commit --diff --exclude ^thirdparty/)" -if [ "$output" == "no modified files to format" ] || [ "$output" == "clang-format did not modify any files" ] ; then - echo "clang-format passed." - exit 0 -else - echo "clang-format failed:" - echo "$output" - exit 1 -fi diff --git a/.travis/git-clang-format b/.travis/git-clang-format deleted file mode 100755 index 37b352835..000000000 --- a/.travis/git-clang-format +++ /dev/null @@ -1,476 +0,0 @@ -#!/usr/bin/env python -# -#===- git-clang-format - ClangFormat Git Integration ---------*- python -*--===# -# -# The LLVM Compiler Infrastructure -# -# This file is distributed under the University of Illinois Open Source -# License. See LICENSE.TXT for details. -# -#===------------------------------------------------------------------------===# - -r""" -clang-format git integration -============================ - -This file provides a clang-format integration for git. Put it somewhere in your -path and ensure that it is executable. Then, "git clang-format" will invoke -clang-format on the changes in current files or a specific commit. - -For further details, run: -git clang-format -h - -Requires Python 2.7 -""" - -import argparse -import collections -import contextlib -import errno -import os -import re -import subprocess -import sys - -usage = 'git clang-format [OPTIONS] [] [--] [...]' - -desc = ''' -Run clang-format on all lines that differ between the working directory -and , which defaults to HEAD. Changes are only applied to the working -directory. -The following git-config settings set the default of the corresponding option: - clangFormat.binary - clangFormat.commit - clangFormat.extension - clangFormat.style -''' - -# Name of the temporary index file in which save the output of clang-format. -# This file is created within the .git directory. -temp_index_basename = 'clang-format-index' - - -Range = collections.namedtuple('Range', 'start, count') - - -def main(): - config = load_git_config() - - # In order to keep '--' yet allow options after positionals, we need to - # check for '--' ourselves. (Setting nargs='*' throws away the '--', while - # nargs=argparse.REMAINDER disallows options after positionals.) - argv = sys.argv[1:] - try: - idx = argv.index('--') - except ValueError: - dash_dash = [] - else: - dash_dash = argv[idx:] - argv = argv[:idx] - - default_extensions = ','.join([ - # From clang/lib/Frontend/FrontendOptions.cpp, all lower case - 'c', 'h', # C - 'm', # ObjC - 'mm', # ObjC++ - 'cc', 'cp', 'cpp', 'c++', 'cxx', 'hpp', # C++ - # Other languages that clang-format supports - 'proto', 'protodevel', # Protocol Buffers - 'js', # JavaScript - 'ts', # TypeScript - ]) - - p = argparse.ArgumentParser( - usage=usage, formatter_class=argparse.RawDescriptionHelpFormatter, - description=desc) - p.add_argument('--binary', - default=config.get('clangformat.binary', 'clang-format'), - help='path to clang-format'), - p.add_argument('--commit', - default=config.get('clangformat.commit', 'HEAD'), - help='default commit to use if none is specified'), - p.add_argument('--diff', action='store_true', - help='print a diff instead of applying the changes') - p.add_argument('--extensions', - default=config.get('clangformat.extensions', - default_extensions), - help=('comma-separated list of file extensions to format, ' - 'excluding the period and case-insensitive')), - p.add_argument('--exclude', help='Exclude files matching this regex.') - p.add_argument('-f', '--force', action='store_true', - help='allow changes to unstaged files') - p.add_argument('-p', '--patch', action='store_true', - help='select hunks interactively') - p.add_argument('-q', '--quiet', action='count', default=0, - help='print less information') - p.add_argument('--style', - default=config.get('clangformat.style', None), - help='passed to clang-format'), - p.add_argument('-v', '--verbose', action='count', default=0, - help='print extra information') - # We gather all the remaining positional arguments into 'args' since we need - # to use some heuristics to determine whether or not was present. - # However, to print pretty messages, we make use of metavar and help. - p.add_argument('args', nargs='*', metavar='', - help='revision from which to compute the diff') - p.add_argument('ignored', nargs='*', metavar='...', - help='if specified, only consider differences in these files') - opts = p.parse_args(argv) - - opts.verbose -= opts.quiet - del opts.quiet - - commit, files = interpret_args(opts.args, dash_dash, opts.commit) - changed_lines = compute_diff_and_extract_lines(commit, files) - if opts.verbose >= 1: - ignored_files = set(changed_lines) - filter_by_extension(changed_lines, opts.extensions.lower().split(',')) - if opts.exclude: - for filename in changed_lines.keys(): - if re.match(opts.exclude, filename): - del changed_lines[filename] - if opts.verbose >= 1: - ignored_files.difference_update(changed_lines) - if ignored_files: - print 'Ignoring changes in the following files:' - for filename in ignored_files: - print ' ', filename - if changed_lines: - print 'Running clang-format on the following files:' - for filename in changed_lines: - print ' ', filename - if not changed_lines: - print 'no modified files to format' - return - # The computed diff outputs absolute paths, so we must cd before accessing - # those files. - cd_to_toplevel() - old_tree = create_tree_from_workdir(changed_lines) - new_tree = run_clang_format_and_save_to_tree(changed_lines, - binary=opts.binary, - style=opts.style) - if opts.verbose >= 1: - print 'old tree:', old_tree - print 'new tree:', new_tree - if old_tree == new_tree: - if opts.verbose >= 0: - print 'clang-format did not modify any files' - elif opts.diff: - print_diff(old_tree, new_tree) - else: - changed_files = apply_changes(old_tree, new_tree, force=opts.force, - patch_mode=opts.patch) - if (opts.verbose >= 0 and not opts.patch) or opts.verbose >= 1: - print 'changed files:' - for filename in changed_files: - print ' ', filename - - -def load_git_config(non_string_options=None): - """Return the git configuration as a dictionary. - All options are assumed to be strings unless in `non_string_options`, in which - is a dictionary mapping option name (in lower case) to either "--bool" or - "--int".""" - if non_string_options is None: - non_string_options = {} - out = {} - for entry in run('git', 'config', '--list', '--null').split('\0'): - if entry: - name, value = entry.split('\n', 1) - if name in non_string_options: - value = run('git', 'config', non_string_options[name], name) - out[name] = value - return out - - -def interpret_args(args, dash_dash, default_commit): - """Interpret `args` as "[commit] [--] [files...]" and return (commit, files). - It is assumed that "--" and everything that follows has been removed from - args and placed in `dash_dash`. - If "--" is present (i.e., `dash_dash` is non-empty), the argument to its - left (if present) is taken as commit. Otherwise, the first argument is - checked if it is a commit or a file. If commit is not given, - `default_commit` is used.""" - if dash_dash: - if len(args) == 0: - commit = default_commit - elif len(args) > 1: - die('at most one commit allowed; %d given' % len(args)) - else: - commit = args[0] - object_type = get_object_type(commit) - if object_type not in ('commit', 'tag'): - if object_type is None: - die("'%s' is not a commit" % commit) - else: - die("'%s' is a %s, but a commit was expected" % (commit, object_type)) - files = dash_dash[1:] - elif args: - if disambiguate_revision(args[0]): - commit = args[0] - files = args[1:] - else: - commit = default_commit - files = args - else: - commit = default_commit - files = [] - return commit, files - - -def disambiguate_revision(value): - """Returns True if `value` is a revision, False if it is a file, or dies.""" - # If `value` is ambiguous (neither a commit nor a file), the following - # command will die with an appropriate error message. - run('git', 'rev-parse', value, verbose=False) - object_type = get_object_type(value) - if object_type is None: - return False - if object_type in ('commit', 'tag'): - return True - die('`%s` is a %s, but a commit or filename was expected' % - (value, object_type)) - - -def get_object_type(value): - """Returns a string description of an object's type, or None if it is not - a valid git object.""" - cmd = ['git', 'cat-file', '-t', value] - p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE) - stdout, stderr = p.communicate() - if p.returncode != 0: - return None - return stdout.strip() - - -def compute_diff_and_extract_lines(commit, files): - """Calls compute_diff() followed by extract_lines().""" - diff_process = compute_diff(commit, files) - changed_lines = extract_lines(diff_process.stdout) - diff_process.stdout.close() - diff_process.wait() - if diff_process.returncode != 0: - # Assume error was already printed to stderr. - sys.exit(2) - return changed_lines - - -def compute_diff(commit, files): - """Return a subprocess object producing the diff from `commit`. - The return value's `stdin` file object will produce a patch with the - differences between the working directory and `commit`, filtered on `files` - (if non-empty). Zero context lines are used in the patch.""" - cmd = ['git', 'diff-index', '-p', '-U0', commit, '--'] - cmd.extend(files) - p = subprocess.Popen(cmd, stdin=subprocess.PIPE, stdout=subprocess.PIPE) - p.stdin.close() - return p - - -def extract_lines(patch_file): - """Extract the changed lines in `patch_file`. - The return value is a dictionary mapping filename to a list of (start_line, - line_count) pairs. - The input must have been produced with ``-U0``, meaning unidiff format with - zero lines of context. The return value is a dict mapping filename to a - list of line `Range`s.""" - matches = {} - for line in patch_file: - match = re.search(r'^\+\+\+\ [^/]+/(.*)', line) - if match: - filename = match.group(1).rstrip('\r\n') - match = re.search(r'^@@ -[0-9,]+ \+(\d+)(,(\d+))?', line) - if match: - start_line = int(match.group(1)) - line_count = 1 - if match.group(3): - line_count = int(match.group(3)) - if line_count > 0: - matches.setdefault(filename, []).append(Range(start_line, line_count)) - return matches - - -def filter_by_extension(dictionary, allowed_extensions): - """Delete every key in `dictionary` that doesn't have an allowed extension. - `allowed_extensions` must be a collection of lowercase file extensions, - excluding the period.""" - allowed_extensions = frozenset(allowed_extensions) - for filename in dictionary.keys(): - base_ext = filename.rsplit('.', 1) - if len(base_ext) == 1 or base_ext[1].lower() not in allowed_extensions: - del dictionary[filename] - - -def cd_to_toplevel(): - """Change to the top level of the git repository.""" - toplevel = run('git', 'rev-parse', '--show-toplevel') - os.chdir(toplevel) - - -def create_tree_from_workdir(filenames): - """Create a new git tree with the given files from the working directory. - Returns the object ID (SHA-1) of the created tree.""" - return create_tree(filenames, '--stdin') - - -def run_clang_format_and_save_to_tree(changed_lines, binary='clang-format', - style=None): - """Run clang-format on each file and save the result to a git tree. - Returns the object ID (SHA-1) of the created tree.""" - def index_info_generator(): - for filename, line_ranges in changed_lines.iteritems(): - mode = oct(os.stat(filename).st_mode) - blob_id = clang_format_to_blob(filename, line_ranges, binary=binary, - style=style) - yield '%s %s\t%s' % (mode, blob_id, filename) - return create_tree(index_info_generator(), '--index-info') - - -def create_tree(input_lines, mode): - """Create a tree object from the given input. - If mode is '--stdin', it must be a list of filenames. If mode is - '--index-info' is must be a list of values suitable for "git update-index - --index-info", such as " ". Any other mode - is invalid.""" - assert mode in ('--stdin', '--index-info') - cmd = ['git', 'update-index', '--add', '-z', mode] - with temporary_index_file(): - p = subprocess.Popen(cmd, stdin=subprocess.PIPE) - for line in input_lines: - p.stdin.write('%s\0' % line) - p.stdin.close() - if p.wait() != 0: - die('`%s` failed' % ' '.join(cmd)) - tree_id = run('git', 'write-tree') - return tree_id - - -def clang_format_to_blob(filename, line_ranges, binary='clang-format', - style=None): - """Run clang-format on the given file and save the result to a git blob. - Returns the object ID (SHA-1) of the created blob.""" - clang_format_cmd = [binary, filename] - if style: - clang_format_cmd.extend(['-style='+style]) - clang_format_cmd.extend([ - '-lines=%s:%s' % (start_line, start_line+line_count-1) - for start_line, line_count in line_ranges]) - try: - clang_format = subprocess.Popen(clang_format_cmd, stdin=subprocess.PIPE, - stdout=subprocess.PIPE) - except OSError as e: - if e.errno == errno.ENOENT: - die('cannot find executable "%s"' % binary) - else: - raise - clang_format.stdin.close() - hash_object_cmd = ['git', 'hash-object', '-w', '--path='+filename, '--stdin'] - hash_object = subprocess.Popen(hash_object_cmd, stdin=clang_format.stdout, - stdout=subprocess.PIPE) - clang_format.stdout.close() - stdout = hash_object.communicate()[0] - if hash_object.returncode != 0: - die('`%s` failed' % ' '.join(hash_object_cmd)) - if clang_format.wait() != 0: - die('`%s` failed' % ' '.join(clang_format_cmd)) - return stdout.rstrip('\r\n') - - -@contextlib.contextmanager -def temporary_index_file(tree=None): - """Context manager for setting GIT_INDEX_FILE to a temporary file and deleting - the file afterward.""" - index_path = create_temporary_index(tree) - old_index_path = os.environ.get('GIT_INDEX_FILE') - os.environ['GIT_INDEX_FILE'] = index_path - try: - yield - finally: - if old_index_path is None: - del os.environ['GIT_INDEX_FILE'] - else: - os.environ['GIT_INDEX_FILE'] = old_index_path - os.remove(index_path) - - -def create_temporary_index(tree=None): - """Create a temporary index file and return the created file's path. - If `tree` is not None, use that as the tree to read in. Otherwise, an - empty index is created.""" - gitdir = run('git', 'rev-parse', '--git-dir') - path = os.path.join(gitdir, temp_index_basename) - if tree is None: - tree = '--empty' - run('git', 'read-tree', '--index-output='+path, tree) - return path - - -def print_diff(old_tree, new_tree): - """Print the diff between the two trees to stdout.""" - # We use the porcelain 'diff' and not plumbing 'diff-tree' because the output - # is expected to be viewed by the user, and only the former does nice things - # like color and pagination. - subprocess.check_call(['git', 'diff', old_tree, new_tree, '--']) - - -def apply_changes(old_tree, new_tree, force=False, patch_mode=False): - """Apply the changes in `new_tree` to the working directory. - Bails if there are local changes in those files and not `force`. If - `patch_mode`, runs `git checkout --patch` to select hunks interactively.""" - changed_files = run('git', 'diff-tree', '-r', '-z', '--name-only', old_tree, - new_tree).rstrip('\0').split('\0') - if not force: - unstaged_files = run('git', 'diff-files', '--name-status', *changed_files) - if unstaged_files: - print >>sys.stderr, ('The following files would be modified but ' - 'have unstaged changes:') - print >>sys.stderr, unstaged_files - print >>sys.stderr, 'Please commit, stage, or stash them first.' - sys.exit(2) - if patch_mode: - # In patch mode, we could just as well create an index from the new tree - # and checkout from that, but then the user will be presented with a - # message saying "Discard ... from worktree". Instead, we use the old - # tree as the index and checkout from new_tree, which gives the slightly - # better message, "Apply ... to index and worktree". This is not quite - # right, since it won't be applied to the user's index, but oh well. - with temporary_index_file(old_tree): - subprocess.check_call(['git', 'checkout', '--patch', new_tree]) - index_tree = old_tree - else: - with temporary_index_file(new_tree): - run('git', 'checkout-index', '-a', '-f') - return changed_files - - -def run(*args, **kwargs): - stdin = kwargs.pop('stdin', '') - verbose = kwargs.pop('verbose', True) - strip = kwargs.pop('strip', True) - for name in kwargs: - raise TypeError("run() got an unexpected keyword argument '%s'" % name) - p = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE, - stdin=subprocess.PIPE) - stdout, stderr = p.communicate(input=stdin) - if p.returncode == 0: - if stderr: - if verbose: - print >>sys.stderr, '`%s` printed to stderr:' % ' '.join(args) - print >>sys.stderr, stderr.rstrip() - if strip: - stdout = stdout.rstrip('\r\n') - return stdout - if verbose: - print >>sys.stderr, '`%s` returned %s' % (' '.join(args), p.returncode) - if stderr: - print >>sys.stderr, stderr.rstrip() - sys.exit(2) - - -def die(message): - print >>sys.stderr, 'error:', message - sys.exit(2) - - -if __name__ == '__main__': - main() diff --git a/LICENSE b/LICENSE deleted file mode 100644 index 8dada3eda..000000000 --- a/LICENSE +++ /dev/null @@ -1,201 +0,0 @@ - Apache License - Version 2.0, January 2004 - http://www.apache.org/licenses/ - - TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION - - 1. Definitions. - - "License" shall mean the terms and conditions for use, reproduction, - and distribution as defined by Sections 1 through 9 of this document. - - "Licensor" shall mean the copyright owner or entity authorized by - the copyright owner that is granting the License. - - "Legal Entity" shall mean the union of the acting entity and all - other entities that control, are controlled by, or are under common - control with that entity. For the purposes of this definition, - "control" means (i) the power, direct or indirect, to cause the - direction or management of such entity, whether by contract or - otherwise, or (ii) ownership of fifty percent (50%) or more of the - outstanding shares, or (iii) beneficial ownership of such entity. - - "You" (or "Your") shall mean an individual or Legal Entity - exercising permissions granted by this License. - - "Source" form shall mean the preferred form for making modifications, - including but not limited to software source code, documentation - source, and configuration files. - - "Object" form shall mean any form resulting from mechanical - transformation or translation of a Source form, including but - not limited to compiled object code, generated documentation, - and conversions to other media types. - - "Work" shall mean the work of authorship, whether in Source or - Object form, made available under the License, as indicated by a - copyright notice that is included in or attached to the work - (an example is provided in the Appendix below). - - "Derivative Works" shall mean any work, whether in Source or Object - form, that is based on (or derived from) the Work and for which the - editorial revisions, annotations, elaborations, or other modifications - represent, as a whole, an original work of authorship. For the purposes - of this License, Derivative Works shall not include works that remain - separable from, or merely link (or bind by name) to the interfaces of, - the Work and Derivative Works thereof. - - "Contribution" shall mean any work of authorship, including - the original version of the Work and any modifications or additions - to that Work or Derivative Works thereof, that is intentionally - submitted to Licensor for inclusion in the Work by the copyright owner - or by an individual or Legal Entity authorized to submit on behalf of - the copyright owner. For the purposes of this definition, "submitted" - means any form of electronic, verbal, or written communication sent - to the Licensor or its representatives, including but not limited to - communication on electronic mailing lists, source code control systems, - and issue tracking systems that are managed by, or on behalf of, the - Licensor for the purpose of discussing and improving the Work, but - excluding communication that is conspicuously marked or otherwise - designated in writing by the copyright owner as "Not a Contribution." - - "Contributor" shall mean Licensor and any individual or Legal Entity - on behalf of whom a Contribution has been received by Licensor and - subsequently incorporated within the Work. - - 2. Grant of Copyright License. Subject to the terms and conditions of - this License, each Contributor hereby grants to You a perpetual, - worldwide, non-exclusive, no-charge, royalty-free, irrevocable - copyright license to reproduce, prepare Derivative Works of, - publicly display, publicly perform, sublicense, and distribute the - Work and such Derivative Works in Source or Object form. - - 3. Grant of Patent License. Subject to the terms and conditions of - this License, each Contributor hereby grants to You a perpetual, - worldwide, non-exclusive, no-charge, royalty-free, irrevocable - (except as stated in this section) patent license to make, have made, - use, offer to sell, sell, import, and otherwise transfer the Work, - where such license applies only to those patent claims licensable - by such Contributor that are necessarily infringed by their - Contribution(s) alone or by combination of their Contribution(s) - with the Work to which such Contribution(s) was submitted. If You - institute patent litigation against any entity (including a - cross-claim or counterclaim in a lawsuit) alleging that the Work - or a Contribution incorporated within the Work constitutes direct - or contributory patent infringement, then any patent licenses - granted to You under this License for that Work shall terminate - as of the date such litigation is filed. - - 4. Redistribution. You may reproduce and distribute copies of the - Work or Derivative Works thereof in any medium, with or without - modifications, and in Source or Object form, provided that You - meet the following conditions: - - (a) You must give any other recipients of the Work or - Derivative Works a copy of this License; and - - (b) You must cause any modified files to carry prominent notices - stating that You changed the files; and - - (c) You must retain, in the Source form of any Derivative Works - that You distribute, all copyright, patent, trademark, and - attribution notices from the Source form of the Work, - excluding those notices that do not pertain to any part of - the Derivative Works; and - - (d) If the Work includes a "NOTICE" text file as part of its - distribution, then any Derivative Works that You distribute must - include a readable copy of the attribution notices contained - within such NOTICE file, excluding those notices that do not - pertain to any part of the Derivative Works, in at least one - of the following places: within a NOTICE text file distributed - as part of the Derivative Works; within the Source form or - documentation, if provided along with the Derivative Works; or, - within a display generated by the Derivative Works, if and - wherever such third-party notices normally appear. The contents - of the NOTICE file are for informational purposes only and - do not modify the License. You may add Your own attribution - notices within Derivative Works that You distribute, alongside - or as an addendum to the NOTICE text from the Work, provided - that such additional attribution notices cannot be construed - as modifying the License. - - You may add Your own copyright statement to Your modifications and - may provide additional or different license terms and conditions - for use, reproduction, or distribution of Your modifications, or - for any such Derivative Works as a whole, provided Your use, - reproduction, and distribution of the Work otherwise complies with - the conditions stated in this License. - - 5. Submission of Contributions. Unless You explicitly state otherwise, - any Contribution intentionally submitted for inclusion in the Work - by You to the Licensor shall be under the terms and conditions of - this License, without any additional terms or conditions. - Notwithstanding the above, nothing herein shall supersede or modify - the terms of any separate license agreement you may have executed - with Licensor regarding such Contributions. - - 6. Trademarks. This License does not grant permission to use the trade - names, trademarks, service marks, or product names of the Licensor, - except as required for reasonable and customary use in describing the - origin of the Work and reproducing the content of the NOTICE file. - - 7. Disclaimer of Warranty. Unless required by applicable law or - agreed to in writing, Licensor provides the Work (and each - Contributor provides its Contributions) on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or - implied, including, without limitation, any warranties or conditions - of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A - PARTICULAR PURPOSE. You are solely responsible for determining the - appropriateness of using or redistributing the Work and assume any - risks associated with Your exercise of permissions under this License. - - 8. Limitation of Liability. In no event and under no legal theory, - whether in tort (including negligence), contract, or otherwise, - unless required by applicable law (such as deliberate and grossly - negligent acts) or agreed to in writing, shall any Contributor be - liable to You for damages, including any direct, indirect, special, - incidental, or consequential damages of any character arising as a - result of this License or out of the use or inability to use the - Work (including but not limited to damages for loss of goodwill, - work stoppage, computer failure or malfunction, or any and all - other commercial damages or losses), even if such Contributor - has been advised of the possibility of such damages. - - 9. Accepting Warranty or Additional Liability. While redistributing - the Work or Derivative Works thereof, You may choose to offer, - and charge a fee for, acceptance of support, warranty, indemnity, - or other liability obligations and/or rights consistent with this - License. However, in accepting such obligations, You may act only - on Your own behalf and on Your sole responsibility, not on behalf - of any other Contributor, and only if You agree to indemnify, - defend, and hold each Contributor harmless for any liability - incurred by, or claims asserted against, such Contributor by reason - of your accepting any such warranty or additional liability. - - END OF TERMS AND CONDITIONS - - APPENDIX: How to apply the Apache License to your work. - - To apply the Apache License to your work, attach the following - boilerplate notice, with the fields enclosed by brackets "{}" - replaced with your own identifying information. (Don't include - the brackets!) The text should be enclosed in the appropriate - comment syntax for the file format. We also recommend that a - file or class name and description of purpose be included on the - same "printed page" as the copyright notice for easier - identification within third-party archives. - - Copyright {yyyy} {name of copyright owner} - - Licensed under the Apache License, Version 2.0 (the "License"); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. diff --git a/install-dependencies.sh b/install-dependencies.sh deleted file mode 100755 index f84da1684..000000000 --- a/install-dependencies.sh +++ /dev/null @@ -1,21 +0,0 @@ -#!/usr/bin/env bash - -ROOT_DIR=$(cd "$(dirname "${BASH_SOURCE:-$0}")"; pwd) - -platform="unknown" -unamestr="$(uname)" -if [[ "$unamestr" == "Linux" ]]; then - echo "Platform is linux." - platform="linux" -elif [[ "$unamestr" == "Darwin" ]]; then - echo "Platform is macosx." - platform="macosx" -else - echo "Unrecognized platform." - exit 1 -fi - -if [[ $platform == "linux" ]]; then - sudo apt-get update - sudo apt-get install -y git python-dev -fi diff --git a/Makefile b/src/common/Makefile similarity index 100% rename from Makefile rename to src/common/Makefile diff --git a/build/.gitkeep b/src/common/build/.gitkeep similarity index 100% rename from build/.gitkeep rename to src/common/build/.gitkeep diff --git a/common.c b/src/common/common.c similarity index 100% rename from common.c rename to src/common/common.c diff --git a/common.h b/src/common/common.h similarity index 100% rename from common.h rename to src/common/common.h diff --git a/doc/tasks.md b/src/common/doc/tasks.md similarity index 100% rename from doc/tasks.md rename to src/common/doc/tasks.md diff --git a/event_loop.c b/src/common/event_loop.c similarity index 100% rename from event_loop.c rename to src/common/event_loop.c diff --git a/event_loop.h b/src/common/event_loop.h similarity index 100% rename from event_loop.h rename to src/common/event_loop.h diff --git a/io.c b/src/common/io.c similarity index 100% rename from io.c rename to src/common/io.c diff --git a/io.h b/src/common/io.h similarity index 100% rename from io.h rename to src/common/io.h diff --git a/lib/python/common_extension.c b/src/common/lib/python/common_extension.c similarity index 100% rename from lib/python/common_extension.c rename to src/common/lib/python/common_extension.c diff --git a/lib/python/common_extension.h b/src/common/lib/python/common_extension.h similarity index 100% rename from lib/python/common_extension.h rename to src/common/lib/python/common_extension.h diff --git a/lib/python/common_module.c b/src/common/lib/python/common_module.c similarity index 100% rename from lib/python/common_module.c rename to src/common/lib/python/common_module.c diff --git a/lib/python/setup.py b/src/common/lib/python/setup.py similarity index 100% rename from lib/python/setup.py rename to src/common/lib/python/setup.py diff --git a/logging.c b/src/common/logging.c similarity index 100% rename from logging.c rename to src/common/logging.c diff --git a/logging.h b/src/common/logging.h similarity index 100% rename from logging.h rename to src/common/logging.h diff --git a/state/db.h b/src/common/state/db.h similarity index 100% rename from state/db.h rename to src/common/state/db.h diff --git a/state/object_table.h b/src/common/state/object_table.h similarity index 100% rename from state/object_table.h rename to src/common/state/object_table.h diff --git a/state/redis.c b/src/common/state/redis.c similarity index 100% rename from state/redis.c rename to src/common/state/redis.c diff --git a/state/redis.h b/src/common/state/redis.h similarity index 100% rename from state/redis.h rename to src/common/state/redis.h diff --git a/state/task_log.h b/src/common/state/task_log.h similarity index 100% rename from state/task_log.h rename to src/common/state/task_log.h diff --git a/state/task_table.h b/src/common/state/task_table.h similarity index 100% rename from state/task_table.h rename to src/common/state/task_table.h diff --git a/task.c b/src/common/task.c similarity index 100% rename from task.c rename to src/common/task.c diff --git a/task.h b/src/common/task.h similarity index 100% rename from task.h rename to src/common/task.h diff --git a/test/common_tests.c b/src/common/test/common_tests.c similarity index 100% rename from test/common_tests.c rename to src/common/test/common_tests.c diff --git a/test/db_tests.c b/src/common/test/db_tests.c similarity index 100% rename from test/db_tests.c rename to src/common/test/db_tests.c diff --git a/test/example_task.h b/src/common/test/example_task.h similarity index 100% rename from test/example_task.h rename to src/common/test/example_task.h diff --git a/test/io_tests.c b/src/common/test/io_tests.c similarity index 100% rename from test/io_tests.c rename to src/common/test/io_tests.c diff --git a/test/redis_tests.c b/src/common/test/redis_tests.c similarity index 100% rename from test/redis_tests.c rename to src/common/test/redis_tests.c diff --git a/test/task_tests.c b/src/common/test/task_tests.c similarity index 100% rename from test/task_tests.c rename to src/common/test/task_tests.c diff --git a/test/test.py b/src/common/test/test.py similarity index 100% rename from test/test.py rename to src/common/test/test.py diff --git a/thirdparty/ae/ae.c b/src/common/thirdparty/ae/ae.c similarity index 100% rename from thirdparty/ae/ae.c rename to src/common/thirdparty/ae/ae.c diff --git a/thirdparty/ae/ae.h b/src/common/thirdparty/ae/ae.h similarity index 100% rename from thirdparty/ae/ae.h rename to src/common/thirdparty/ae/ae.h diff --git a/thirdparty/ae/ae_epoll.c b/src/common/thirdparty/ae/ae_epoll.c similarity index 100% rename from thirdparty/ae/ae_epoll.c rename to src/common/thirdparty/ae/ae_epoll.c diff --git a/thirdparty/ae/ae_evport.c b/src/common/thirdparty/ae/ae_evport.c similarity index 100% rename from thirdparty/ae/ae_evport.c rename to src/common/thirdparty/ae/ae_evport.c diff --git a/thirdparty/ae/ae_kqueue.c b/src/common/thirdparty/ae/ae_kqueue.c similarity index 100% rename from thirdparty/ae/ae_kqueue.c rename to src/common/thirdparty/ae/ae_kqueue.c diff --git a/thirdparty/ae/ae_select.c b/src/common/thirdparty/ae/ae_select.c similarity index 100% rename from thirdparty/ae/ae_select.c rename to src/common/thirdparty/ae/ae_select.c diff --git a/thirdparty/ae/config.h b/src/common/thirdparty/ae/config.h similarity index 100% rename from thirdparty/ae/config.h rename to src/common/thirdparty/ae/config.h diff --git a/thirdparty/ae/zmalloc.h b/src/common/thirdparty/ae/zmalloc.h similarity index 100% rename from thirdparty/ae/zmalloc.h rename to src/common/thirdparty/ae/zmalloc.h diff --git a/thirdparty/build-redis.sh b/src/common/thirdparty/build-redis.sh similarity index 100% rename from thirdparty/build-redis.sh rename to src/common/thirdparty/build-redis.sh diff --git a/thirdparty/greatest.h b/src/common/thirdparty/greatest.h similarity index 100% rename from thirdparty/greatest.h rename to src/common/thirdparty/greatest.h diff --git a/thirdparty/hiredis b/src/common/thirdparty/hiredis similarity index 100% rename from thirdparty/hiredis rename to src/common/thirdparty/hiredis diff --git a/thirdparty/utarray.h b/src/common/thirdparty/utarray.h similarity index 100% rename from thirdparty/utarray.h rename to src/common/thirdparty/utarray.h diff --git a/thirdparty/uthash.h b/src/common/thirdparty/uthash.h similarity index 100% rename from thirdparty/uthash.h rename to src/common/thirdparty/uthash.h diff --git a/thirdparty/utlist.h b/src/common/thirdparty/utlist.h similarity index 100% rename from thirdparty/utlist.h rename to src/common/thirdparty/utlist.h diff --git a/thirdparty/utstring.h b/src/common/thirdparty/utstring.h similarity index 100% rename from thirdparty/utstring.h rename to src/common/thirdparty/utstring.h