[tune] Node Fault Tolerance (#3238)

This PR introduces single-node fault tolerance for Tune.

## Previous behavior:
 - Actors will be restarted without checking if resources are available. This can lead to problems if we lose resources.

## New behavior:
 - RUNNING trials will be resumed on another node on a best effort basis (meaning they will run if resources available). 
 - If the cluster is saturated, RUNNING trials on that failed node will become PENDING and queued.
 - During recovery, TrialSchedulers and SearchAlgorithms should receive notification of this (via `trial_runner.stop_trial`) so that they don’t wait/block for a trial that isn’t running.


Remaining questions:
 -  Should `last_result` be consistent during restore?
Yes; but not for earlier trials (trials that are yet to be checkpointed).

 - Waiting for some PRs to merge first (#3239)

Closes #2851.

python/ray/tune/trial_executor.py

@@ -4,7 +4,6 @@ from __future__ import division
 from __future__ import print_function
 
 import logging
-import traceback
 
 from ray.tune.trial import Trial, Checkpoint
 
@@ -61,24 +60,24 @@ class TrialExecutor(object):
                                   "stop_trial() method")
 
     def restart_trial(self, trial, error_msg=None):
-        """Restarts the trial.
+        """Restarts or requeues the trial.
 
-        The state of the trial should restore from the last checkpoint.
+        The state of the trial should restore from the last checkpoint. Trial
+        is requeued if the cluster no longer has resources to accomodate it.
 
         Args:
             error_msg (str): Optional error message.
         """
-        try:
-            logger.info(
-                "Attempting to recover trial state from last checkpoint")
-            self.stop_trial(
-                trial, error=True, error_msg=error_msg, stop_logger=False)
-            trial.result_logger.flush()
+        self.stop_trial(
+            trial,
+            error=error_msg is not None,
+            error_msg=error_msg,
+            stop_logger=False)
+        trial.result_logger.flush()
+        if self.has_resources(trial.resources):
             self.start_trial(trial)
-        except Exception:
-            error_msg = traceback.format_exc()
-            logger.exception("Error recovering trial from checkpoint, abort.")
-            self.stop_trial(trial, error=True, error_msg=error_msg)
+        else:
+            trial.status = Trial.PENDING
 
     def continue_training(self, trial):
         """Continues the training of this trial."""