[tune] Cluster Fault Tolerance (#3309)

This PR introduces cluster-level fault tolerance for Tune by checkpointing global state. This occurs with relatively high frequency and allows users to easily resume experiments when the cluster crashes.

Note that this PR may affect automated workflows due to auto-prompting, but this is resolvable.

test/multi_node_test.py

@@ -394,6 +394,7 @@ def train_func(config, reporter):  # add a reporter arg
         time.sleep(0.1)
         reporter(timesteps_total=i, mean_accuracy=i+97)  # report metrics
 
+os.environ["TUNE_RESUME_PROMPT_OFF"] = "True"
 ray.init(redis_address="{}")
 ray.tune.register_trainable("train_func", train_func)