mirror of
https://github.com/wassname/ray.git
synced 2026-07-07 09:36:21 +08:00
[Issue 2809][xray] Cleanup on driver detach (#2826)
This change addresses issue #2809. Test #2797 has been enabled for raylet and can pass. The following should happen when a driver exits (either gracefully or ungracefully). #2797 should be enabled and pass. Any actors created by the driver that are still running should be killed. Any workers running tasks for the driver should be killed. Any tasks for the driver in any node_manager queues should be removed. Any future tasks received by a node manager for the driver should be ignored. The driver death notification should only be received once.
This commit is contained in:
@@ -239,7 +239,9 @@ def ray_start_head_with_resources():
|
||||
subprocess.Popen(["ray", "stop"]).wait()
|
||||
|
||||
|
||||
@pytest.mark.skip(reason="This test does not work yet.")
|
||||
@pytest.mark.skipif(
|
||||
os.environ.get("RAY_USE_XRAY") != "1",
|
||||
reason="This test only works with xray.")
|
||||
def test_drivers_release_resources(ray_start_head_with_resources):
|
||||
redis_address = ray_start_head_with_resources
|
||||
|
||||
@@ -278,7 +280,7 @@ print("success")
|
||||
driver_script2 = (driver_script1 +
|
||||
"import sys\nsys.stdout.flush()\ntime.sleep(10 ** 6)\n")
|
||||
|
||||
def wait_for_success_output(process_handle, timeout=100):
|
||||
def wait_for_success_output(process_handle, timeout=10):
|
||||
# Wait until the process prints "success" and then return.
|
||||
start_time = time.time()
|
||||
while time.time() - start_time < timeout:
|
||||
|
||||
Reference in New Issue
Block a user