* Start and clean up workers from the local scheduler
Ability to kill workers in photon scheduler
Test for old method of starting workers
Common codepath for killing workers
Common codepath for killing workers
Photon test case for starting and killing workers
fix build
Fix component failure test
Register a worker's pid as part of initial connection
Address comments and revert photon_connect
Set PATH during travis install
Fix
* Fix photon test case to accept clients on plasma manager fd
* attribute-based heterogeneity-awareness in global scheduler and photon
* minor post-rebase fix
* photon: enforce dynamic capacity constraint on task dispatch
* globalsched: cap the number of times we try to schedule a task in round robin
* propagating ability to specify resource capacity to ray.init
* adding resources to remote function export and fetch/register
* globalsched: remove unused functions; update cached photon resource capacity (until next photon heartbeat)
* Add some integration tests.
* globalsched: cleanup + factor out constraint checking
* lots of style
* task_spec_required_resource: global refactor
* clang format
* clang format + comment update in photon
* clang format photon comment
* valgrind
* reduce verbosity for Travis
* Add test for scheduler load balancing.
* addressing comments
* refactoring global scheduler algorithm
* Minor cleanups.
* Linting.
* Fix array_test.py and linting.
* valgrind fix for photon tests
* Attempt to fix stress tests.
* fix hashmap free
* fix hashmap free comment
* memset photon resource vectors to 0 in case they get used before the first heartbeat
* More whitespace changes.
* Undo whitespace error I introduced.
* Refactor local scheduler to remove worker indices.
* Change scheduling state enum to int in all function signatures.
* Bug fix, don't use pointers into a resizable array.
* Remove total_num_workers.
* Fix tests.
* First pass at reconstruction in the worker
Modify reconstruction stress testing to start Plasma service before rest of Ray cluster
TODO about reconstructing ray.puts
Fix ray.put error for double creates
Distinguish between empty entry and no entry in object table
Fix test case
Fix Python test
Fix tests
* Only call reconstruct on objects we have not yet received
* Address review comments
* Fix reconstruction for Python3
* remove unused code
* Address Robert's comments, stress tests are crashing
* Test and update the task's scheduling state to suppress duplicate
reconstruction requests.
* Split result table into two lookups, one for task ID and the other as a
test-and-set for the task state
* Fix object table tests
* Fix redis module result_table_lookup test case
* Multinode reconstruction tests
* Fix python3 test case
* rename
* Use new start_redis
* Remove unused code
* lint
* indent
* Address Robert's comments
* Use start_redis from ray.services in state table tests
* Remove unnecessary memset
* Provide functionality for local scheduler to start new workers.
* Pass full command for starting new worker in to local scheduler.
* Separate out configuration state of local scheduler.
* Optimizations:
- Track mapping of missing object to dependent tasks to avoid iterating over task queue
- Perform all fetch requests for missing objects using the same timer
* Fix bug and add regression test
* Record task dependencies and active fetch requests in the same hash table
* fix typo
* Fix memory leak and add test cases for scheduling when dependencies are evicted
* Fix python3 test case
* Minor details.
* Split local scheduler task queue into waiting and dispatch queue
* Fix memory leak
* Add a new task scheduling status for when a task has been queued locally
* Fix global scheduler test case and add task status doc
* Documentation
* Address Philipp's comments
* Move tasks back to the waiting queue if their dependencies become unavailable
* Update existing task table entries instead of overwriting
* Prevent plasma store and manager from dying when a worker dies.
* Check errno inside of warn_if_sigpipe. Passing in errno doesn't work because the arguments to warn_if_sigpipe can be evaluated out of order.
* Object notification test for Photon, and turn on valgrind for Photon C tests
* Test object notification handler in the plasma manager
* Fix hanging test case
* Send message from plasma client to get plasma store capacity.
* Release objects from plasma client if they are too large.
* Use doubly-linked list instead of ring buffer for plasma client release history.
* Address comments.
* Fix problem with slicing PlasmaBuffer objects.
* Fix crash in plasma manager during transfer.
* Formatting.
* Make plasma client cache larger and make caching test not throw exceptions on Travis.
* Switch to using redis modules for task table.
* Switch to using redis modules for the task table.
* Fix some tests.
* Fix naming and remove code duplication.
* Remove duplication in redis modules and add more cleanups.
* Address comments.
* global scheduler with object transfer cost awareness -- upstream rebase
* debugging global scheduler: multiple subscriptions
* global scheduler: utarray push bug fix; tasks change state to SCHEDULED
* change global scheduler test to be an integraton test
* unit and integration tests are passing for global scheduler
* improve global scheduler test: break up into several
* global scheduler checkpoint: fix photon object id bug in test
* test with timesync between object and task notifications; TODO: handle OoO object+task notifications in GS
* fallback to base policy if no object dependencies are cached (may happen due to OoO object+task notification arrivals
* clean up printfs; handle a missing LS in LS cache
* Minor changes to Python test and factor out some common code.
* refactoring handle task waiting
* addressing comments
* log_info -> log_debug
* Change object ID printing.
* PRId64 merge
* Python 3 fix.
* PRId64.
* Python 3 fix.
* resurrect differentiation between no args and missing object info; spacing
* Valgrind fix.
* Run all global scheduler tests in valgrind.
* clang format
* Comments and documentation changes.
* Minor cleanups.
* fix whitespace
* Fix.
* Documentation fix.
* Add function for driver to get address info from Redis.
* Use Redis address instead of Redis port.
* Configure Redis to run in unprotected mode.
* Add method for starting Ray processes on non-head node.
* Pass in correct node ip address to start_plasma_manager.
* Script for starting Ray processes.
* Handle the case where an object already exists in the store. Maybe this should also compare the object hashes.
* Have driver get info from Redis when start_ray_local=False.
* Fix.
* Script for killing ray processes.
* Catch some errors when the main_loop in a worker throws an exception.
* Allow redirecting stdout and stderr to /dev/null.
* Wrap start_ray.py in a shell script.
* More helpful error messages.
* Fixes.
* Wait for redis server to start up before configuring it.
* Allow seeding of deterministic object ID generation.
* Small change.
* Add RAY.CONNECT Redis command.
* Add RAY.GET_CLIENT_ADDRESS command.
* Build and clean Redis in common Makefile.
* Use custom Redis module in Ray and use custom CONNECT and GET_CLIENT_ADDRESS commands.
* Fixes.
* Remove mapping from redis client ID to ray db client ID.
* Fix.
* passing plasma ip:port association with photon through redis to global scheduler
* Fix test.
* sanity-checking aux_address inside db_connect_extended
* clang format
* fix photon tests
* clang format photon tests
* Object reconstruction in Photon and C test cases for Photon
* Fix hanging test case on mac
* Remove unnecessary event from photon tests
* make photon_disconnect not leak file descriptors
* fix some of the memory errors
* Fix valgrind
* lint
* Address Robert's comments and add test case for object reconstruction suppression
* Remove OWNER
* Consolidate wait implementations.
* Consolidate fetch implementations.
* Share callback between wait and fetch to address issue in which only one callback can be run for a given subscribe channel.
* Reactivate manager tests.
* Remove more code.
* Add some documentation.
* change plasma object notifications to carry a struct of information
* factoring out object_info for general use by several Ray components
* fixing a bug in python test
* addressing comments
* handling Robert's comments
* clang format
* Fix valgrind.
* Fetch missing dependencies from local scheduler.
* Factor out global scheduler policy state.
* Use object_table_subscribe instead of object_table_lookup.
* Fix bug in which timer was being created twice for a single fetch request.
* Free old manager vector.
* Give more informative error message when we do not know how to serialize a class.
* Check that passing arguments to remote functions and getting them does not change their values.
* fix serialization bug
* fix tests for common module
* Formatting.
* Bug fix in init_pickle_module signature.
* Use pickle with HIGHEST_PROTOCOL.
* Add Python and Redis submodules, and remove old third-party modules
* Update VS projects (WARNING: references files that do not exist yet)
* Update code & add shims for APIs except AF_UNIX/{send,recv}msg()
* Minor style changes.
* On Mac OS X, if cmake fails to find Python with custom search path, default to using find_package.
* Fix formatting in CMakeLists.txt.
* Set CUSTOM_PYTHON_EXECUTABLE variable on mac if the custom search path fails.
* Require exact Python version match.
* Use cmake mode keyword for logging messages.
* Fix.
* Find python libraries in cmake by searching near the include directories.
* Update cmakelists for numbuf.
* Use sizeof(field) instead of sizeof(type) and other fixes.
* Fix formatting.
* Bug fix.
* Zero-initialize structs. There are many more instances of these that I haven't changed yet.
* Bug fix.
* Revert from atexit to signaling to fix valgrind tests.
* Address Philipp's comments.
* Initial scheduler commit
* global scheduler
* add global scheduler
* Implement global scheduler skeleton.
* Formatting.
* Allow local scheduler to be started without a connection to redis so that we can test it without a global scheduler.
* Fail if there are no local schedulers when the global scheduler receives a task.
* Initialize uninitialized value and formatting fix.
* Generalize local scheduler table to db client table.
* Remove code duplication in local scheduler and add flag for whether a task came from the global scheduler or not.
* Queue task specs in the local scheduler instead of tasks.
* Simple global scheduler tests, including valgrind.
* Factor out functions for starting processes.
* Fixes.
* Set logging levels in Makefile using -DRAY_COMMON_LOG_LEVEL=level
* Lower level of some LOG_ERROR messages, log the name of the table operation on failure
* Address rest of Robert's comments
* Fix spurious log message
* Merge task table and task log
* Fix test in db tests
* Address Robert's comments and some better error checking
* Add a LOG_FATAL that exits the program
* Put infrastructure in place to compute task IDs and object IDs.
* Fix version number for common library.
* Compute task IDs and object IDs deterministically.
* Address Stephanie's comments.
* Update task documentation.
* Fix formatting.
* Add more tests and checks.
* Fix formatting.
* Enable DCHECKs and change CHECKs to DCHECKs.
* Basic functionality for LRU eviction.
* Test eviction.
* Factor out eviction policy.
* Move delete_object into eviction policy.
* Replace array of released objects with an LRU cache (hash table + doubly linked list).
* Finish rebase on master.
* Move actual object deletion away from eviction policy and into plasma store.
* Small fixes.
* Fixes.
* Make remove_object_from_lru_cache always remove the object.
* Minor formatting and comments.
* Pass in allowed memory as argument to Plasma store.
* Small fix.
