wassname/ray
1
0
Fork 0
mirror of https://github.com/wassname/ray.git synced 2026-09-09 11:32:43 +08:00
Code Issues Packages Projects Releases Wiki Activity

[core] Enable object reconstruction for retryable actor tasks (#9557)

Browse Source 
* Test actor plasma reconstruction

* Allow resubmission of actor tasks

* doc

* Test for actor constructor

* Kill PID before removing node

* Kill pid before node
This commit is contained in:
Stephanie Wang
2020-07-23 21:15:12 -07:00
committed by GitHub
parent 239196fffc
commit f2705e2c73
8 changed files with 224 additions and 24 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/source/fault-tolerance.rst
+24 -16
View File
@@ -41,20 +41,6 @@ You can experiment with this behavior by running the following code.
except ray.exceptions.RayWorkerError:
print('FAILURE')
Task outputs over a configurable threshold (default 100KB) may be stored in
Ray's distributed object store. Thus, a node failure can cause the loss of a
task output. If this occurs, Ray will automatically attempt to recover the
value by looking for copies of the same object on other nodes. If there are no
other copies left, an ``UnreconstructableError`` will be raised.
When there are no copies of an object left, Ray also provides an option to
automatically recover the value by re-executing the task that created the
value. Arguments to the task are recursively reconstructed with the same
method. This option can be enabled with
``ray.init(enable_object_reconstruction=True)`` in standalone mode or ``ray
start --enable-object-reconstruction`` in cluster mode.
Actors
------
@@ -164,8 +150,8 @@ You can experiment with this behavior by running the following code.
For at-least-once actors, the system will still guarantee execution ordering
according to the initial submission order. For example, any tasks submitted
after a failed actor task will not execute on the actor until the failed actor
task has been successfully retried. The system also will not attempt to
re-execute any tasks that executed successfully before the failure.
task has been successfully retried. The system will not attempt to re-execute
any tasks that executed successfully before the failure (unless :ref:`object reconstruction <object-reconstruction>` is enabled).
At-least-once execution is best suited for read-only actors or actors with
ephemeral state that does not need to be rebuilt after a failure. For actors
@@ -174,3 +160,25 @@ manually restart the actor or automatically restart the actor with at-most-once
semantics. If the actor’s exact state at the time of failure is needed, the
application is responsible for resubmitting all tasks since the last
checkpoint.
.. _object-reconstruction:
Objects
-------
Task outputs over a configurable threshold (default 100KB) may be stored in
Ray's distributed object store. Thus, a node failure can cause the loss of a
task output. If this occurs, Ray will automatically attempt to recover the
value by looking for copies of the same object on other nodes. If there are no
other copies left, an ``UnreconstructableError`` will be raised.
When there are no copies of an object left, Ray also provides an option to
automatically recover the value by re-executing the task that created the
value. Arguments to the task are recursively reconstructed with the same
method. This option can be enabled with
``ray.init(enable_object_reconstruction=True)`` in standalone mode or ``ray
start --enable-object-reconstruction`` in cluster mode.
During reconstruction, each task will only be re-executed up to the specified
number of times, using ``max_retries`` for normal tasks and
``max_task_retries`` for actor tasks. Both limits can be set to infinity with
the value -1.