* Enable --scoped-enums in flatbuffer compiler.
* Change enum to c++11 style (enum class).
* Resolve conflicts.
* Solve building failure when RAY_USE_NEW_GCS=on and remove ERROR_INDEX suffix.
* Merge with master and fix CI failure.
* Print error when actor takes too long to start, and refactor error message pushing.
* Print warning every ten seconds.
* Fix linting and tests.
* Fix tests.
* Treat actor creation like a regular task.
* Small cleanups.
* Change semantics of actor resource handling.
* Bug fix.
* Minor linting
* Bug fix
* Fix jenkins test.
* Fix actor tests
* Some cleanups
* Bug fix
* Fix bug.
* Remove cached actor tasks when a driver is removed.
* Add more info to taskspec in global state API.
* Fix cyclic import bug in tune.
* Fix
* Fix linting.
* Fix linting.
* Don't schedule any tasks (especially actor creaiton tasks) on local schedulers with 0 CPUs.
* Bug fix.
* Add test for 0 CPU case
* Fix linting
* Address comments.
* Fix typos and add comment.
* Add assertion and fix test.
* spillback policy implementation: global + local scheduler
* modernize global scheduler policy state; factor out random number engine and generator
* Minimal version.
* Fix test.
* Make load balancing test less strenuous.
* Expose calls to get and set the actor frontier
* Remove fields used for old checkpointing prototype, change actor_checkpoint_failed -> succeeded
* Prototype for actor checkpointing
* Filter out duplicate tasks on the local scheduler
* Clean up some of the Python checkpointing code
* More cleanups
* Documentation
* cleanup and fix unit test
* Allow remote checkpoint calls through actor handle
* Check whether object is local before reconstructing
* Enable checkpointing for distributed actor handles, refactor tests
* Fix local scheduler tests
* lint
* Address comments
* lint
* Skip tests that fail on new GCS
* style
* Don't put same object twice when setting the actor frontier
* Address Philipp's comments, cleaner fbs naming
* Add failing unit test for nondeterministic reconstruction
* Retry scheduling actor tasks if reassigned to local scheduler
* Update execution edges asynchronously upon dispatch for nondeterministic reconstruction
* Fix bug for updating checkpoint task execution dependencies
* Update comments for deterministic reconstruction
* cleanup
* Add (and skip) failing test case for nondeterministic reconstruction
* Suppress test output
* Define execution dependencies flatbuffer and add to Redis commands
* Convert TaskSpec to TaskExecutionSpec
* Add execution dependencies to Python bindings
* Submitting actor tasks uses execution dependency API instead of dummy argument
* Fix dependency getters and some cleanup for fetching missing dependencies
* C++ convention
* Make TaskExecutionSpec a C++ class
* Convert local scheduler to use TaskExecutionSpec class
* Convert some pointers to references
* Finish conversion to TaskExecutionSpec class
* fix
* Fix
* Fix memory errors?
* Cast flatbuffers GetSize to size_t
* Fixes
* add more retries in global scheduler unit test
* fix linting and cast fbb.GetSize to size_t
* Style and doc
* Fix linting and simplify from_flatbuf.
* Enable scheduling with custom resource labels.
* Fix.
* Minor fixes and ref counting fix.
* Linting
* Use .data() instead of .c_str().
* Fix linting.
* Fix ResourcesTest.testGPUIDs test by waiting for workers to start up.
* Sleep in test so that all tasks are submitted before any completes.
* wip
* with test
* add timeout
* also add test for f
* remove on cleanup
* update
* wip
* fix tests
* mark actor removed in redis
* clang-format
* fix bug when no-inprogress tasks
* try to set task status done
* Add comment.
* Plasma client test for plasma abort
* Use ray-project/arrow:abort-objects branch
* Set plasma manager connection cursor to -1 when not in use
* Handle transfer errors between plasma managers, abort unsealed objects
* Add TODO for local scheduler exiting on plasma manager death
* Revert "Plasma client test for plasma abort"
This reverts commit e00fbd58dc4a632f58383549b19fb9057b305a14.
* Upgrade arrow to version with PlasmaClient::Abort
* Fix plasma manager test
* Fix plasma test
* Temporarily use arrow fork for testing
* fix and set arrow commit
* Fix plasma test
* Fix plasma manager test and make write_object_chunk consistent with read_object_chunk
* style
* upgrade arrow
* Initial pass at factoring out C++ configuration into a single file.
* Expose config through Python.
* Forward declarations.
* Fixes with Python extensions
* Remove old code.
* Consistent naming for constants.
* Fixes
* Fix linting.
* More linting.
* Whitespace
* rename config -> _config.
* Move config inside a class.
* update naming convention
* Fix linting.
* More linting
* More linting.
* Add in some more constants.
* Fix linting
* Add actor handle ID to the task spec
* Local scheduler dispatches actor tasks according to a task counter per handle
* Fix python test
* Allow passing actor handles into tasks. Not completely working yet. Also this is very messy.
* Fixes, should be roughly working now.
* Refactor actor handle wrapper
* Fix __init__ tests
* Terminate actor when the original handle goes out of scope
* TODO and a couple test cases
* Make tests for unsupported cases
* Fix Python mode tests
* Linting.
* Cache actor definitions that occur before ray.init() is called.
* Fix export actor class
* Deterministically compute actor handle ID
* Fix __getattribute__
* Fix string encoding for python3
* doc
* Add comment and assertion.
* Compile global scheduler with -Werror -Wall.
* Compile plasma manager with -Werror -Wall.
* Compile local scheduler with -Werror -Wall.
* Compile common code with -Werror -Wall.
* Signed/unsigned comparisons.
* More signed/unsigned fixes.
* More signed/unsigned fixes and added extern keyword.
* Fix linting.
* Don't check strict-aliasing because Python.h doesn't pass.
* Worker reports error in previous task, actor task counter is incremented after task is successful
* Refactor actor task execution
- Return new task counter in GetTaskRequest
- Update worker state for actor tasks inside of the actor method
executor
* Manually invoked checkpoint method
* Scheduling for actor checkpoint methods
* Fix python bugs in checkpointing
* Return task success from worker to local scheduler instead of actor counter
* Kill local schedulers halfway through actor execution instead of waiting for all tasks to execute once
* Remove redundant actor tasks during dispatch, reconstruct missing dependencies for actor tasks
* Make executor for temporary actor methods
* doc
* Set default argument for whether the previous task was a success
* Refactor actor method call
* Simplify checkpoint task submission
* lint
* fix philipp's comments
* Add missing line
* Make actor reconstruction tests run faster
* Unimportant whitespace.
* Unimportant whitespace.
* Update checkpoint method signature
* Documentation and handle exceptions during checkpoint save/resume
* Rename get_task message field to actor_checkpoint_failed
* Fix bug.
* Remove debugging check, redirect test output
* When a task is passed to the global scheduler, if it is not received, then try again.
* Call give_task_to_global_scheduler directly (same with local).
* Fix bug in wait_for_pid_to_exit, add test for actor deletion.
* Fix actor garbage collection by breaking cyclic references
* Add test for calling actor method immediately after actor creation.
* Fix bug, must dispatch tasks when workers are killed.
* Fix python test
* Fix cyclic reference problem by creating ActorMethod objects on the fly.
* Try simply increasing the time allowed for many_drivers_test.py.
* Local scheduler sends a null heartbeat to global scheduler to notify death
* Add whitespace.
* Speed up component failures test
* Free local scheduler state upon plasma manager disconnection
* Revert Python actor reconstruction
* Actor reconstruction using object lineage
* Add dummy arguments and return values for actor tasks
* Pin dummy outputs for actor tasks
* Skip checkpointing test for now
* TODOs
* minor edits
* Generate dummy object dependencies in Python, not C
* Fix linting.
* Move actor counter and dummy objects inside of the actor handle
* Refactor Worker._process_task, suppress exception propagation for
sequential actor tasks
* Remove race between local scheduler disconnecting and global scheduler
assigning a task
* Fix number of workers started in component failures test
* Fix race between global scheduler retrying a task assignment and monitor
cleaning up task table. The global scheduler should only retry the task
assignment if the local scheduler is still alive.
* Clean up task_table_update callback if failure
* Look up current local scheduler mapping when retrying actor task submission
* Log warning if no subscribers received a task table update
* Clean up database handle memory in local scheduler
* adding support for the user-interpretable label(UIR)
* more plumbing for num_uirs further upstream; set to infty when specified on cmd line
* pass default num_uirs for actors; update GlobalStateAPI
* support num_uirs in ray.init()
* local scheduler resource accounting: support num_uirs; prep for vectorized resource accounting
* global scheduler test updated
* Fix bug introduced by rebase.
* Rename UIR -> CustomResource and add test.
* Small changes and use constexpr instead of macros.
* Linting and some renaming.
* Reorder some code.
* Remove cpus_in_use and fix bug.
* Add another test and make a small change.
* Rephrase documentation about feature stability.
* Reconstruct actor state when local schedulers fail.
* Simplify construction of arguments to pass into default_worker.py from local scheduler.
* Remove deprecated ray.actor.
* Simplify actor reconstruction method.
* Fix linting.
* Small fixes.
* Rebase Ray on top of Plasma in Apache Arrow
* add thirdparty building scripts
* use rebased arrow
* fix
* fix build
* fix python visibility
* comment out C tests for now
* fix multithreading
* fix
* reduce logging
* fix plasma manager multithreading
* make sure old and new object IDs can coexist peacefully
* more rebasing
* update
* fixes
* fix
* install pyarrow
* install cython
* fix
* install newer cmake
* fix
* rebase on top of latest arrow
* getting runtest.py run locally (needed to comment out a test for that to work)
* work on plasma tests
* more fixes
* fix local scheduler tests
* fix global scheduler test
* more fixes
* fix python 3 bytes vs string
* fix manager tests valgrind
* fix documentation building
* fix linting
* fix c++ linting
* fix linting
* add tests back in
* Install without sudo.
* Set PKG_CONFIG_PATH in build.sh so that Ray can find plasma.
* Install pkg-config
* Link -lpthread, note that find_package(Threads) doesn't seem to work reliably.
* Comment in testGPUIDs in runtest.py.
* Set PKG_CONFIG_PATH when building pyarrow.
* Pull apache/arrow and not pcmoritz/arrow.
* Fix installation in docker image.
* adapt to changes of the plasma api
* Fix installation of pyarrow module.
* Fix linting.
* Use correct python executable to build pyarrow.
* Don't reconstruct all objects in every fetch request in local scheduler.
* Separate out fetch timer and reconstruction timer.
* Fix bug.
* Bug fix.
* Fix naming convention for global variables.
* Address comments.
* Make reconstruct_counter a static variable.
* Fix linting.
* Redo reconstruct handler using a set of objects to fetch.
* Fix linting.
* Replace set with vector.
* Dispatch regular and actor tasks when resources become available.
* Make actor methods do resource bookkeeping and add test.
* Remove unnecessary field.
* Fix linting.
* Fix actor test.
* Maintain set of actors with pending tasks to speed up task dispatch.
* Exit early from task dispatch if there are no resources available.
* Fix linting.
* Fix error.
* Fix bug related to iterator invalidation.
* When an actor is removed, remove it from the set of actors with pending tasks.
* Replace a local scheduler ut_array with a std::vector.
* Replace vector of sizes in local scheduler with std::pair.
* Remove utarray include.
* Replace utarray with std::vector for reading local scheduler input messages.
* Remove more UT data structures.
* Remove UT includes.
* Fix linting.
* Include stdlib.h to find size_t.
* Remove includes of stdbool.h.
* Replace std::pair with TaskQueueEntry.
* Fix redis tests.
* Reinstate tests.
* Implement sharding in the Ray core
* Single node Python modifications to do sharding
* Do the sharding in redis.cc
* Pipe num_redis_shards through start_ray.py and worker.py.
* Use multiple redis shards in multinode tests.
* first steps for sharding ray.global_state
* Fix problem in multinode docker test.
* fix runtest.py
* fix some tests
* fix redis shard startup
* fix redis sharding
* fix
* fix bug introduced by the map-iterator being consumed
* fix sharding bug
* shard event table
* update number of Redis clients to be 64K
* Fix object table tests by flushing shards in between unit tests
* Fix local scheduler tests
* Documentation
* Register shard locations in the primary shard
* Add plasma unit tests back to build
* lint
* lint and fix build
* Fix
* Address Robert's comments
* Refactor start_ray_processes to start Redis shard
* lint
* Fix global scheduler python tests
* Fix redis module test
* Fix plasma test
* Fix component failure test
* Fix local scheduler test
* Fix runtest.py
* Fix global scheduler test for python3
* Fix task_table_test_and_update bug, from actor task table submission race
* Fix jenkins tests.
* Retry Redis shard connections
* Fix test cases
* Convert database clients to DBClient struct
* Fix race condition when subscribing to db client table
* Remove unused lines, add APITest for sharded Ray
* Fix
* Fix memory leak
* Suppress ReconstructionTests output
* Suppress output for APITestSharded
* Reissue task table add/update commands if initial command does not publish to any subscribers.
* fix
* Fix linting.
* fix tests
* fix linting
* fix python test
* fix linting
* Change local scheduler bookkeeping to use GPU IDs.
* Update actor test.
* Add tests for actors and tasks simultaneously using GPUs.
* Add additional task GPU ID test.
* Fix linting.
* Make redis GPU assignment ignore GPU IDs.
* Small fix.
* copy task specifications put into the actor task cache so it won't get overwritten when the scheduler receives the next task
* cleanup
* cleanup and fix
* linting
* fix jenkins test
* fix linting
