Actor checkpointing for distributed actor handles (#1498)

* Expose calls to get and set the actor frontier

* Remove fields used for old checkpointing prototype, change actor_checkpoint_failed -> succeeded

* Prototype for actor checkpointing

* Filter out duplicate tasks on the local scheduler

* Clean up some of the Python checkpointing code

* More cleanups

* Documentation

* cleanup and fix unit test

* Allow remote checkpoint calls through actor handle

* Check whether object is local before reconstructing

* Enable checkpointing for distributed actor handles, refactor tests

* Fix local scheduler tests

* lint

* Address comments

* lint

* Skip tests that fail on new GCS

* style

* Don't put same object twice when setting the actor frontier

* Address Philipp's comments, cleaner fbs naming

python/ray/utils.py

@@ -20,6 +20,28 @@ def _random_string():
     return np.random.bytes(20)
 
 
+def format_error_message(exception_message, task_exception=False):
+    """Improve the formatting of an exception thrown by a remote function.
+
+    This method takes a traceback from an exception and makes it nicer by
+    removing a few uninformative lines and adding some space to indent the
+    remaining lines nicely.
+
+    Args:
+        exception_message (str): A message generated by traceback.format_exc().
+
+    Returns:
+        A string of the formatted exception message.
+    """
+    lines = exception_message.split("\n")
+    if task_exception:
+        # For errors that occur inside of tasks, remove lines 1, 2, 3, and 4,
+        # which are always the same, they just contain information about the
+        # main loop.
+        lines = lines[0:1] + lines[5:]
+    return "\n".join(lines)
+
+
 def push_error_to_driver(redis_client, error_type, message, driver_id=None,
                          data=None):
     """Push an error message to the driver to be printed in the background.