From 57d6e983022ad86559b07f6a863404198ebf2347 Mon Sep 17 00:00:00 2001 From: Robert Nishihara Date: Thu, 1 Nov 2018 11:52:05 -0700 Subject: [PATCH] Update actor fault tolerance documentation. (#3175) --- doc/source/fault-tolerance.rst | 34 +++++++++------------------------- 1 file changed, 9 insertions(+), 25 deletions(-) diff --git a/doc/source/fault-tolerance.rst b/doc/source/fault-tolerance.rst index 112927ce3..6c388c9f8 100644 --- a/doc/source/fault-tolerance.rst +++ b/doc/source/fault-tolerance.rst @@ -6,16 +6,9 @@ This document describes the handling of failures in Ray. Machine and Process Failures ---------------------------- -Currently, each **local scheduler** and each **plasma manager** send heartbeats -to a **monitor** process. If the monitor does not receive any heartbeats from a -given process for some duration of time (about ten seconds), then it will mark -that process as dead. The monitor process will then clean up the associated -state in the Redis servers. If a manager is marked as dead, the object table -will be updated to remove all occurrences of that manager so that other managers -don't try to fetch objects from the dead manager. If a local scheduler is marked -as dead, all of the tasks that are marked as executing on that local scheduler -in the task table will be marked as lost and all actors associated with that -local scheduler will be recreated by other local schedulers. +Each **raylet** (the scheduler process) sends heartbeats to a **monitor** +process. If the monitor does not receive any heartbeats from a given raylet for +some period of time (about ten seconds), then it will mark that process as dead. Lost Objects ------------ @@ -23,19 +16,16 @@ Lost Objects If an object is needed but is lost or was never created, then the task that created the object will be re-executed to create the object. If necessary, tasks needed to create the input arguments to the task being re-executed will also be -re-executed. +re-executed. This is the standard *lineage-based fault tolerance* strategy used +by other systems like Spark. Actors ------ -When a local scheduler is marked as dead, all actors associated with that local -scheduler that were still alive will be recreated by other local schedulers. By -default, all of the actor methods will be re-executed in the same order that -they were initially executed. If actor checkpointing is enabled, then the actor -state will be loaded from the most recent checkpoint and the actor methods that -occurred after the checkpoint will be re-executed. Note that actor checkpointing -is currently an experimental feature. - +When an actor dies (either because the actor process crashed or because the node +that the actor was on died), by default any attempt to get an object from that +actor that cannot be created will raise an exception. Subsequent releases will +include an option for automatically restarting actors. Current Limitations ------------------- @@ -58,9 +48,3 @@ Lost Objects evicted, and is later needed, Ray will not reconstruct this object. 2. If an object is constructed by an actor method, is then evicted, and is later needed, Ray will not reconstruct this object. - -Actor Reconstruction -~~~~~~~~~~~~~~~~~~~~ - -1. Actor reconstruction follows the order of initial execution, but new tasks - may get interleaved with the re-executed tasks.