Plasma and worker node failure. (#373)

* Failing test case

* Local scheduler exits cleanly after plasma store dies

* Tolerate one plasma store failure

* Tolerate plasma store failures on all nodes except head node

* Plasma manager heartbeats

* Component failure tests

* Don't run the helper for Python testing

* Fix C test

* Fix hanging plasma transfer test

* Fix python3

* Consolidate ClientConnection code

* Fix valgrind test

* fix c test

* We can restart worker nodes!

* Fix flatbuffers bug

* Address comments

* Only register actual workers with the local scheduler

* Fix bug

* Fix segfaults

* Add test case that tests for driver liveness, fix local scheduler bug

* Clean up after tests

* Allocate retry info on the stack

* Send SIGKILL before waiting

* Relax unit test conditions

* Driver liveness test case and documentation

scripts/start_ray.py

@@ -30,6 +30,14 @@ def check_no_existing_redis_clients(node_ip_address, redis_address):
     assert b"ray_client_id" in info
     assert b"node_ip_address" in info
     assert b"client_type" in info
+    assert b"deleted" in info
+    # Clients that ran on the same node but that are marked dead can be
+    # ignored.
+    deleted = info[b"deleted"]
+    deleted = bool(int(deleted))
+    if deleted:
+      continue
+
     if info[b"node_ip_address"].decode("ascii") == node_ip_address:
       raise Exception("This Redis instance is already connected to clients with this IP address.")