[Issue 2809][xray] Cleanup on driver detach (#2826)

This change addresses issue #2809. Test #2797 has been enabled for raylet and can pass.

The following should happen when a driver exits (either gracefully or ungracefully).

#2797 should be enabled and pass.
Any actors created by the driver that are still running should be killed.
Any workers running tasks for the driver should be killed.
Any tasks for the driver in any node_manager queues should be removed.
Any future tasks received by a node manager for the driver should be ignored.
The driver death notification should only be received once.
This commit is contained in:
Zhijun Fu
2018-09-07 16:11:32 +08:00
committed by Hao Chen
parent 3f6ed537a4
commit 753ba76141
11 changed files with 246 additions and 49 deletions
+4 -2
View File
@@ -239,7 +239,9 @@ def ray_start_head_with_resources():
subprocess.Popen(["ray", "stop"]).wait()
@pytest.mark.skip(reason="This test does not work yet.")
@pytest.mark.skipif(
os.environ.get("RAY_USE_XRAY") != "1",
reason="This test only works with xray.")
def test_drivers_release_resources(ray_start_head_with_resources):
redis_address = ray_start_head_with_resources
@@ -278,7 +280,7 @@ print("success")
driver_script2 = (driver_script1 +
"import sys\nsys.stdout.flush()\ntime.sleep(10 ** 6)\n")
def wait_for_success_output(process_handle, timeout=100):
def wait_for_success_output(process_handle, timeout=10):
# Wait until the process prints "success" and then return.
start_time = time.time()
while time.time() - start_time < timeout: