Shard Redis. (#539)

* Implement sharding in the Ray core

* Single node Python modifications to do sharding

* Do the sharding in redis.cc

* Pipe num_redis_shards through start_ray.py and worker.py.

* Use multiple redis shards in multinode tests.

* first steps for sharding ray.global_state

* Fix problem in multinode docker test.

* fix runtest.py

* fix some tests

* fix redis shard startup

* fix redis sharding

* fix

* fix bug introduced by the map-iterator being consumed

* fix sharding bug

* shard event table

* update number of Redis clients to be 64K

* Fix object table tests by flushing shards in between unit tests

* Fix local scheduler tests

* Documentation

* Register shard locations in the primary shard

* Add plasma unit tests back to build

* lint

* lint and fix build

* Fix

* Address Robert's comments

* Refactor start_ray_processes to start Redis shard

* lint

* Fix global scheduler python tests

* Fix redis module test

* Fix plasma test

* Fix component failure test

* Fix local scheduler test

* Fix runtest.py

* Fix global scheduler test for python3

* Fix task_table_test_and_update bug, from actor task table submission race

* Fix jenkins tests.

* Retry Redis shard connections

* Fix test cases

* Convert database clients to DBClient struct

* Fix race condition when subscribing to db client table

* Remove unused lines, add APITest for sharded Ray

* Fix

* Fix memory leak

* Suppress ReconstructionTests output

* Suppress output for APITestSharded

* Reissue task table add/update commands if initial command does not publish to any subscribers.

* fix

* Fix linting.

* fix tests

* fix linting

* fix python test

* fix linting

python/ray/plasma/plasma.py

@@ -405,7 +405,8 @@ def start_plasma_manager(store_name, redis_address,
                "-m", plasma_manager_name,
                "-h", node_ip_address,
                "-p", str(plasma_manager_port),
-               "-r", redis_address]
+               "-r", redis_address,
+               ]
     if use_valgrind:
       process = subprocess.Popen(["valgrind",
                                   "--track-origins=yes",

python/ray/plasma/test/test.py

@@ -480,7 +480,7 @@ class TestPlasmaManager(unittest.TestCase):
     store_name1, self.p2 = plasma.start_plasma_store(use_valgrind=USE_VALGRIND)
     store_name2, self.p3 = plasma.start_plasma_store(use_valgrind=USE_VALGRIND)
     # Start a Redis server.
-    redis_address = services.address("127.0.0.1", services.start_redis()[0])
+    redis_address, _ = services.start_redis("127.0.0.1")
     # Start two PlasmaManagers.
     manager_name1, self.p4, self.port1 = plasma.start_plasma_manager(
         store_name1, redis_address, use_valgrind=USE_VALGRIND)
@@ -778,8 +778,7 @@ class TestPlasmaManagerRecovery(unittest.TestCase):
     self.store_name, self.p2 = plasma.start_plasma_store(
         use_valgrind=USE_VALGRIND)
     # Start a Redis server.
-    self.redis_address = services.address("127.0.0.1",
-                                          services.start_redis()[0])
+    self.redis_address, _ = services.start_redis("127.0.0.1")
     # Start a PlasmaManagers.
     manager_name, self.p3, self.port1 = plasma.start_plasma_manager(
         self.store_name,