[direct task] Retry tasks on failure and turn on RAY_FORCE_DIRECT for test_multinode_failures.py (#6306)

* multinode failures direct

* Add number of retries allowed for tasks

* Retry tasks

* Add failing test for object reconstruction

* Handle return status and debug

* update

* Retry task unit test

* update

* update

* todo

* Fix max_retries decorator, fix test

* Fix test that flaked

* lint

* comments
This commit is contained in:
Stephanie Wang
2019-12-02 10:20:57 -08:00
committed by GitHub
parent 0b0a16982a
commit da41180dc0
21 changed files with 284 additions and 63 deletions
+2 -1
View File
@@ -86,7 +86,8 @@ cdef extern from "ray/core_worker/core_worker.h" nogil:
CRayStatus SubmitTask(
const CRayFunction &function, const c_vector[CTaskArg] &args,
const CTaskOptions &options, c_vector[CObjectID] *return_ids)
const CTaskOptions &options, c_vector[CObjectID] *return_ids,
int max_retries)
CRayStatus CreateActor(
const CRayFunction &function, const c_vector[CTaskArg] &args,
const CActorCreationOptions &options, CActorID *actor_id)