* Replace a local scheduler ut_array with a std::vector.
* Replace vector of sizes in local scheduler with std::pair.
* Remove utarray include.
* Replace utarray with std::vector for reading local scheduler input messages.
* Remove more UT data structures.
* Remove UT includes.
* Fix linting.
* Include stdlib.h to find size_t.
* Remove includes of stdbool.h.
* Replace std::pair with TaskQueueEntry.
* Fix redis tests.
* Reinstate tests.
* Implement sharding in the Ray core
* Single node Python modifications to do sharding
* Do the sharding in redis.cc
* Pipe num_redis_shards through start_ray.py and worker.py.
* Use multiple redis shards in multinode tests.
* first steps for sharding ray.global_state
* Fix problem in multinode docker test.
* fix runtest.py
* fix some tests
* fix redis shard startup
* fix redis sharding
* fix
* fix bug introduced by the map-iterator being consumed
* fix sharding bug
* shard event table
* update number of Redis clients to be 64K
* Fix object table tests by flushing shards in between unit tests
* Fix local scheduler tests
* Documentation
* Register shard locations in the primary shard
* Add plasma unit tests back to build
* lint
* lint and fix build
* Fix
* Address Robert's comments
* Refactor start_ray_processes to start Redis shard
* lint
* Fix global scheduler python tests
* Fix redis module test
* Fix plasma test
* Fix component failure test
* Fix local scheduler test
* Fix runtest.py
* Fix global scheduler test for python3
* Fix task_table_test_and_update bug, from actor task table submission race
* Fix jenkins tests.
* Retry Redis shard connections
* Fix test cases
* Convert database clients to DBClient struct
* Fix race condition when subscribing to db client table
* Remove unused lines, add APITest for sharded Ray
* Fix
* Fix memory leak
* Suppress ReconstructionTests output
* Suppress output for APITestSharded
* Reissue task table add/update commands if initial command does not publish to any subscribers.
* fix
* Fix linting.
* fix tests
* fix linting
* fix python test
* fix linting
* Fix worker blocked bug
* tmp
* Push an error to the driver on ray.put for non-driver tasks
* Fix result table tests
* Fix test, logging
* Address comments
* Fix suppression bug
* Fix redis module test
* Edit error message
* Get values in chunks during reconstruction
* Test case for driver ray.put errors
* Error for evicting ray.put objects from the driver
* Fix tests
* Reduce verbosity
* Documentation
* WARN instead of FATAL for object hash mismatches, push error to driver
* Document the callback signature for object_table_add/remove
* Error table
* Wait for all errors in python test
* Fix doc
* Fix state test
* Implement actor field for tasks
* Implement actor management in local scheduler.
* initial python frontend for actors
* import actors on worker
* IPython code completion and tests
* prepare creating actors through local schedulers
* add actor id to PyTask
* submit actor calls to local scheduler
* starting to integrate
* simple fix
* Fixes from rebasing.
* more work on python actors
* Improve local scheduler actor handlers.
* Pass actor ID to local scheduler when connecting a client.
* first working version of actors
* fixing actors
* fix creating two copies of the same actor
* fix actors
* remove sleep
* get rid of export synchronization
* update
* insert actor methods into the queue in the right order
* remove print statements
* make it compile again after rebase
* Minor updates.
* fix python actor ids
* Pass actor_id to start_worker.
* add test
* Minor changes.
* Update actor tests.
* Temporary plan for import counter.
* Temporarily fix import counters.
* Fix some tests.
* Fixes.
* Make actor creation non-blocking.
* Fix test?
* Fix actors on Python 2.
* fix rare case.
* Fix python 2 test.
* More tests.
* Small fixes.
* Linting.
* Revert tensorflow version to 0.12.0 temporarily.
* Small fix.
* Enhance inheritance test.
* Refactor local scheduler to remove worker indices.
* Change scheduling state enum to int in all function signatures.
* Bug fix, don't use pointers into a resizable array.
* Remove total_num_workers.
* Fix tests.
* First pass at reconstruction in the worker
Modify reconstruction stress testing to start Plasma service before rest of Ray cluster
TODO about reconstructing ray.puts
Fix ray.put error for double creates
Distinguish between empty entry and no entry in object table
Fix test case
Fix Python test
Fix tests
* Only call reconstruct on objects we have not yet received
* Address review comments
* Fix reconstruction for Python3
* remove unused code
* Address Robert's comments, stress tests are crashing
* Test and update the task's scheduling state to suppress duplicate
reconstruction requests.
* Split result table into two lookups, one for task ID and the other as a
test-and-set for the task state
* Fix object table tests
* Fix redis module result_table_lookup test case
* Multinode reconstruction tests
* Fix python3 test case
* rename
* Use new start_redis
* Remove unused code
* lint
* indent
* Address Robert's comments
* Use start_redis from ray.services in state table tests
* Remove unnecessary memset
* Switch to using redis modules for task table.
* Switch to using redis modules for the task table.
* Fix some tests.
* Fix naming and remove code duplication.
* Remove duplication in redis modules and add more cleanups.
* Address comments.
* global scheduler with object transfer cost awareness -- upstream rebase
* debugging global scheduler: multiple subscriptions
* global scheduler: utarray push bug fix; tasks change state to SCHEDULED
* change global scheduler test to be an integraton test
* unit and integration tests are passing for global scheduler
* improve global scheduler test: break up into several
* global scheduler checkpoint: fix photon object id bug in test
* test with timesync between object and task notifications; TODO: handle OoO object+task notifications in GS
* fallback to base policy if no object dependencies are cached (may happen due to OoO object+task notification arrivals
* clean up printfs; handle a missing LS in LS cache
* Minor changes to Python test and factor out some common code.
* refactoring handle task waiting
* addressing comments
* log_info -> log_debug
* Change object ID printing.
* PRId64 merge
* Python 3 fix.
* PRId64.
* Python 3 fix.
* resurrect differentiation between no args and missing object info; spacing
* Valgrind fix.
* Run all global scheduler tests in valgrind.
* clang format
* Comments and documentation changes.
* Minor cleanups.
* fix whitespace
* Fix.
* Documentation fix.
* adding object broadcast channel; published on each object table add
* publishing data size to the bcast channel
* bug fix: objectkey
* update object tests to test for data size: C + py
* remove debug
* clang format
* Minor changes.
* Fix error.
* merging with Robert's comments
* clang format for the object table test upgrade
* First attempt at fixing psubscribe. psubscribe_success_test will fail
* psubscribe test
* SUBSCRIBE returns the number of subscriptions, not success
* Comment out failing test.
* Object reconstruction in Photon and C test cases for Photon
* Fix hanging test case on mac
* Remove unnecessary event from photon tests
* make photon_disconnect not leak file descriptors
* fix some of the memory errors
* Fix valgrind
* lint
* Address Robert's comments and add test case for object reconstruction suppression
* Remove OWNER
* rebase Alexey's PR on top
* rebase on master
* fix test failure waiting for plasma manager to exit
* clang format
* addressing comments
* Minor formatting and naming fixes.
* factoring out object_info for general use by several Ray components
* addressing comments
* Replace SHA256 task hash with MD5
Add object hash to object table (always overwrites)
Support for table operations that span multiple asynchronous Redis
commands
Add a new object location in a transaction, using Redis's optimistic
concurrency
Use Redis GETSET instead of transactions and Python frontend code for object hashing
Remove spurious log message
Fix for object_table_add
Revert "Replace SHA256 task hash with MD5"
This reverts commit e599de473c8dad9189ccb0600429534b469b76a2.
Revert to sha256
Test case for illegal puts
Use SETNX to set object hashes
Initialize digest with zeros
Initialize plasma_request with zeros
* Fixes
* replace SHA256 with a faster hash in the object store
* Fix valgrind
* Address Robert's comments
* Check that plasma_compute_object_hash succeeds.
* Don't run test_illegal_put test with valgrind because it causes an intentional crash which causes valgrind to complain.
* Debugging after rebase.
* handling Robert's comments
* Fix bugs after rebase.
* final fixes for Stephanie's PR
* fix
* Add Python and Redis submodules, and remove old third-party modules
* Update VS projects (WARNING: references files that do not exist yet)
* Update code & add shims for APIs except AF_UNIX/{send,recv}msg()
* Minor style changes.
* Use sizeof(field) instead of sizeof(type) and other fixes.
* Fix formatting.
* Bug fix.
* Zero-initialize structs. There are many more instances of these that I haven't changed yet.
* Bug fix.
* Revert from atexit to signaling to fix valgrind tests.
* Address Philipp's comments.
* Initial scheduler commit
* global scheduler
* add global scheduler
* Implement global scheduler skeleton.
* Formatting.
* Allow local scheduler to be started without a connection to redis so that we can test it without a global scheduler.
* Fail if there are no local schedulers when the global scheduler receives a task.
* Initialize uninitialized value and formatting fix.
* Generalize local scheduler table to db client table.
* Remove code duplication in local scheduler and add flag for whether a task came from the global scheduler or not.
* Queue task specs in the local scheduler instead of tasks.
* Simple global scheduler tests, including valgrind.
* Factor out functions for starting processes.
* Fixes.
* Merge task table and task log
* Fix test in db tests
* Address Robert's comments and some better error checking
* Add a LOG_FATAL that exits the program
* Put infrastructure in place to compute task IDs and object IDs.
* Fix version number for common library.
* Compute task IDs and object IDs deterministically.
* Address Stephanie's comments.
* Update task documentation.
* Fix formatting.
* Add more tests and checks.
* Fix formatting.
* Enable DCHECKs and change CHECKs to DCHECKs.
* Fix socket bind collisions in manager_tests
* bind manager sockets before connecting to the store
* fix memory leak in tests
* fix valgrind early process termination
* fix bind/listen/subscribe race condition
* fix photon
* fix other tests
* make it that all of common is tested
* fix clang-format
* fix
* Ion and Philipp's table retries
* Refactor the retry struct:
- Rename it from retry_struct to retry_info
- Retry information contains the failure callback, not the retry callback
- All functions take in retry information as an arg instead of its expanded fields
* Rename cb -> callback
* Remove prints
* Fix compiler warnings
* Change some CHECKs to greatest ASSERTs
* Key outstanding callbacks hash table with timer ID instead of callback data pointer
* Use the new retry API for table commands
* Memory cleanup in plasma unit tests
* fix Robert's comments
* add valgrind for common
