[tune] Node Fault Tolerance (#3238)

This PR introduces single-node fault tolerance for Tune.

## Previous behavior:
 - Actors will be restarted without checking if resources are available. This can lead to problems if we lose resources.

## New behavior:
 - RUNNING trials will be resumed on another node on a best effort basis (meaning they will run if resources available). 
 - If the cluster is saturated, RUNNING trials on that failed node will become PENDING and queued.
 - During recovery, TrialSchedulers and SearchAlgorithms should receive notification of this (via `trial_runner.stop_trial`) so that they don’t wait/block for a trial that isn’t running.


Remaining questions:
 -  Should `last_result` be consistent during restore?
Yes; but not for earlier trials (trials that are yet to be checkpointed).

 - Waiting for some PRs to merge first (#3239)

Closes #2851.

python/ray/test/cluster_utils.py

@@ -42,9 +42,10 @@ class Cluster(object):
             self.add_node(**head_node_args)
             if connect:
                 redis_password = head_node_args.get("redis_password")
-                ray.init(
+                output_info = ray.init(
                     redis_address=self.redis_address,
                     redis_password=redis_password)
+                logger.info(output_info)
         if shutdown_at_exit:
             atexit.register(self.shutdown)
 
@@ -172,8 +173,10 @@ class Cluster(object):
         for node in all_nodes:
             self.remove_node(node)
 
-        if self.head_node is not None:
+        if self.head_node:
             self.remove_node(self.head_node)
+        else:
+            logger.warning("No headnode exists!")
 
 
 class Node(object):

python/ray/test/test_global_state.py

@@ -6,7 +6,7 @@ import json
 import pytest
 try:
     import pytest_timeout
-except ModuleNotFoundError:
+except ImportError:
     pytest_timeout = None
 import time