wassname/ray
1
0
Fork 0
mirror of https://github.com/wassname/ray.git synced 2026-08-15 12:45:23 +08:00
Code Issues Packages Projects Releases Wiki Activity

Update actor fault tolerance documentation. (#3175)

Browse Source
This commit is contained in:
Robert Nishihara
2018-11-01 11:52:05 -07:00
committed by Philipp Moritz
parent 60f28040ea
commit 57d6e98302
1 changed files with 9 additions and 25 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/source/fault-tolerance.rst
+9 -25
View File
@@ -6,16 +6,9 @@ This document describes the handling of failures in Ray.
Machine and Process Failures
----------------------------
Currently, each **local scheduler** and each **plasma manager** send heartbeats
to a **monitor** process. If the monitor does not receive any heartbeats from a
given process for some duration of time (about ten seconds), then it will mark
that process as dead. The monitor process will then clean up the associated
state in the Redis servers. If a manager is marked as dead, the object table
will be updated to remove all occurrences of that manager so that other managers
don't try to fetch objects from the dead manager. If a local scheduler is marked
as dead, all of the tasks that are marked as executing on that local scheduler
in the task table will be marked as lost and all actors associated with that
local scheduler will be recreated by other local schedulers.
Each **raylet** (the scheduler process) sends heartbeats to a **monitor**
process. If the monitor does not receive any heartbeats from a given raylet for
some period of time (about ten seconds), then it will mark that process as dead.
Lost Objects
------------
@@ -23,19 +16,16 @@ Lost Objects
If an object is needed but is lost or was never created, then the task that
created the object will be re-executed to create the object. If necessary, tasks
needed to create the input arguments to the task being re-executed will also be
re-executed.
re-executed. This is the standard *lineage-based fault tolerance* strategy used
by other systems like Spark.
Actors
------
When a local scheduler is marked as dead, all actors associated with that local
scheduler that were still alive will be recreated by other local schedulers. By
default, all of the actor methods will be re-executed in the same order that
they were initially executed. If actor checkpointing is enabled, then the actor
state will be loaded from the most recent checkpoint and the actor methods that
occurred after the checkpoint will be re-executed. Note that actor checkpointing
is currently an experimental feature.
When an actor dies (either because the actor process crashed or because the node
that the actor was on died), by default any attempt to get an object from that
actor that cannot be created will raise an exception. Subsequent releases will
include an option for automatically restarting actors.
Current Limitations
-------------------
@@ -58,9 +48,3 @@ Lost Objects
evicted, and is later needed, Ray will not reconstruct this object.
2. If an object is constructed by an actor method, is then evicted, and is later
needed, Ray will not reconstruct this object.
Actor Reconstruction
~~~~~~~~~~~~~~~~~~~~
1. Actor reconstruction follows the order of initial execution, but new tasks
may get interleaved with the re-executed tasks.