Summary:
Able to run 1000 tasks with object dependencies on a set of distributed Raylets.
Raylet Changes:
Finalized ClientConnection class.
Task forwarding.
NM-to-NM heartbeats.
NM resource accounting for tasks.
Simple scheduling policy with task forwarding.
Creating and maintaining NM 2 NM long-lived connections and reusing them for task forwarding.
LineageCache Changes:
LineageCache without cleanup of tasks committed by remote nodes.
Lineage cache writeback and cleanup implementation.
ObjectManager Changes:
Object manager event loop/ClientConnection refactor.
Multithreaded object manager (disabled in this PR).
Testing Changes:
Integration tests for task submission on multiple Raylets.
Stress tests for object manager (with GCS and object store integration).
Co-authored-by: Stephanie Wang <swang@cs.berkeley.edu>
Co-authored-by: Alexey Tumanov <atumanov@gmail.com>
* TABLE_APPEND call
* Convert callbacks back to taking in a string...
* GCS returns flatbuffers, define Log class
* Cleanups
* Modify client table to use the Log interface
* Fix bug where we replied twice from redis
* Fixes
* lint
* Compile and test raylet TaskTable
* Modify GCS tables to handle unique_ptrs from nested flatbuffers
* Add raylet::TaskTable unit tests to replace ObjectTable ones
* Convert ObjectTable to a log
* Convert ObjectTable tests to the Log
* AppendAt Redis and gcs Log command
* unit test for AppendAt
* Add a Log for task reconstruction data
* Add check for unique entries in TABLE_APPEND
* Documentation
* TABLE_APPEND call
* Convert callbacks back to taking in a string...
* GCS returns flatbuffers, define Log class
* Cleanups
* Modify client table to use the Log interface
* Fix bug where we replied twice from redis
* Fixes
* lint
* Add TableRequestNotifications and TableCancelNotifications to Redis modules
* Add RequestNotifications and CancelNotifications to generic GCS Table
* Add tests for subscribing to specific keys
* Remove TODO!
* Return the current value at the key directly from RequestNotifications instead of through publish
* Add unit test for Lookup failure callback
* Modify tests to account for empty subscription response
* Remove ObjectTable notification methods
* Clean up message parsing and doc in redis context
* Use vectors of DataT in all GCS callbacks
* Clean up SubscriptionCallback
* Move Table definitions into tables.cc
* Refactor and document redis modules
* doc
* Fix new GCS build
* Cleanups
* Revert "Fix new GCS build"
This reverts commit 6e3e69090c67ef60aaf22a9cf62be0290d989e96.
* Use vectors for internal callback interface, user-facing interface takes a reference to a single item
* Fix new GCS build
* Add unit test for Lookup failure callback
* Fix compiler errors
* Cleanup
* Publish the entry ID with the notification
* Check that the ID for a notification matches in client tests
* Define execution dependencies flatbuffer and add to Redis commands
* Convert TaskSpec to TaskExecutionSpec
* Add execution dependencies to Python bindings
* Submitting actor tasks uses execution dependency API instead of dummy argument
* Fix dependency getters and some cleanup for fetching missing dependencies
* C++ convention
* Make TaskExecutionSpec a C++ class
* Convert local scheduler to use TaskExecutionSpec class
* Convert some pointers to references
* Finish conversion to TaskExecutionSpec class
* fix
* Fix
* Fix memory errors?
* Cast flatbuffers GetSize to size_t
* Fixes
* add more retries in global scheduler unit test
* fix linting and cast fbb.GetSize to size_t
* Style and doc
* Fix linting and simplify from_flatbuf.
* Object table lookup returns vector of DBClientID instead of address strings
* Add node IP address to DBClient notification
* DB client cache stores entire DB client, convert addresses to std::string
* get cached db client returns the client
* Expose a call to initialize the redis cache
* Local scheduler filters out dead clients during reconstruction
* Remove node ip address from dbclient, use aux_address for plasma managers
* Get entire db client entry when not found in cache
* Fix common tests
* Fix address in tests
* Push error to driver if driver task did the put
* Address Robert's comments and cleanup
* Remove unused Redis command
* Fix db test
* WIP: removing OL, OI, TT on client exit; no saving yet.
* ray_redis_module.cc: update header comment.
* Cleanup: just the removal.
* Reformat via yapf: use pep8 style instead of google.
* Checkpoint addressing comments (partially)
* Add 'b' marker before strings (py3 compat)
* Add MonitorTest.
* Use `isort` to sort imports.
* Remove some loggings
* Fix flake8 noqa marker runtest.py
* Try to separate tests out to monitor_test.py
* Rework cleanup algorithm: correct logic
* Extend tests to cover multi-shard cases
* Add some small comments and formatting changes.
* Remove race between local scheduler disconnecting and global scheduler
assigning a task
* Fix number of workers started in component failures test
* Fix race between global scheduler retrying a task assignment and monitor
cleaning up task table. The global scheduler should only retry the task
assignment if the local scheduler is still alive.
* Clean up task_table_update callback if failure
* Look up current local scheduler mapping when retrying actor task submission
* Log warning if no subscribers received a task table update
* Clean up database handle memory in local scheduler
* Replace a local scheduler ut_array with a std::vector.
* Replace vector of sizes in local scheduler with std::pair.
* Remove utarray include.
* Replace utarray with std::vector for reading local scheduler input messages.
* Remove more UT data structures.
* Remove UT includes.
* Fix linting.
* Include stdlib.h to find size_t.
* Remove includes of stdbool.h.
* Replace std::pair with TaskQueueEntry.
* Fix redis tests.
* Reinstate tests.
* added log_table function and a test
* fixed log_files and added task_profiles
* fixed formatting
* fixed linting errors
* fixes
* removed file
* more fixes
* hopefully fixed
* Small changes.
* Fix linting.
* Fix bug in log monitor.
* Small changes.
* Fix bug in travis.
* Including data_size and hash in the ResultTableReply.
* Included data_size and hash info in object_table.
* Fixed bugs in ray_redis_module.cc.
* Removing commented out code.
* Fixes
* Freed hash and data_size strings after using, and checked if they're null along with task_id and is_put.
* Changed it so that data_size is set correctly.
* Removed iostream import.
* Included a check to ensure that the Redis string to long long conversion was successful.
* Included separate data_size and hash null checks.
* Fixed bug.
* Made linting changes.
* Another linting error.
* Slight simplication.
* Implement sharding in the Ray core
* Single node Python modifications to do sharding
* Do the sharding in redis.cc
* Pipe num_redis_shards through start_ray.py and worker.py.
* Use multiple redis shards in multinode tests.
* first steps for sharding ray.global_state
* Fix problem in multinode docker test.
* fix runtest.py
* fix some tests
* fix redis shard startup
* fix redis sharding
* fix
* fix bug introduced by the map-iterator being consumed
* fix sharding bug
* shard event table
* update number of Redis clients to be 64K
* Fix object table tests by flushing shards in between unit tests
* Fix local scheduler tests
* Documentation
* Register shard locations in the primary shard
* Add plasma unit tests back to build
* lint
* lint and fix build
* Fix
* Address Robert's comments
* Refactor start_ray_processes to start Redis shard
* lint
* Fix global scheduler python tests
* Fix redis module test
* Fix plasma test
* Fix component failure test
* Fix local scheduler test
* Fix runtest.py
* Fix global scheduler test for python3
* Fix task_table_test_and_update bug, from actor task table submission race
* Fix jenkins tests.
* Retry Redis shard connections
* Fix test cases
* Convert database clients to DBClient struct
* Fix race condition when subscribing to db client table
* Remove unused lines, add APITest for sharded Ray
* Fix
* Fix memory leak
* Suppress ReconstructionTests output
* Suppress output for APITestSharded
* Reissue task table add/update commands if initial command does not publish to any subscribers.
* fix
* Fix linting.
* fix tests
* fix linting
* fix python test
* fix linting
* Fix worker blocked bug
* tmp
* Push an error to the driver on ray.put for non-driver tasks
* Fix result table tests
* Fix test, logging
* Address comments
* Fix suppression bug
* Fix redis module test
* Edit error message
* Get values in chunks during reconstruction
* Test case for driver ray.put errors
* Error for evicting ray.put objects from the driver
* Fix tests
* Reduce verbosity
* Documentation
* Failing test case
* Local scheduler exits cleanly after plasma store dies
* Tolerate one plasma store failure
* Tolerate plasma store failures on all nodes except head node
* Plasma manager heartbeats
* Component failure tests
* Don't run the helper for Python testing
* Fix C test
* Fix hanging plasma transfer test
* Fix python3
* Consolidate ClientConnection code
* Fix valgrind test
* fix c test
* We can restart worker nodes!
* Fix flatbuffers bug
* Address comments
* Only register actual workers with the local scheduler
* Fix bug
* Fix segfaults
* Add test case that tests for driver liveness, fix local scheduler bug
* Clean up after tests
* Allocate retry info on the stack
* Send SIGKILL before waiting
* Relax unit test conditions
* Driver liveness test case and documentation
* Compile the Ray redis module with C++.
* Redo parsing of object table notifications with flatbuffers.
* Update redis module python tests.
* Redo parsing of task table notifications with flatbuffers.
* Fix linting.
* Redo parsing of db client notifications with flatbuffers.
* Redo publishing of local scheduler heartbeats with flatbuffers.
* Fix linting.
* Remove usage of fixed-width formatting of scheduling state in channel name.
* Reply with flatbuffer object to task table queries, also simplify redis string to flatbuffer string conversion.
* Fix linting and tests.
* fix
* cleanup
* simplify logic in ReplyWithTask
