[sgd] fault tolerance for pytorch + revamp documentation (#6465)

doc/source/index.rst

@@ -17,12 +17,13 @@ Ray is packaged with the following libraries for accelerating machine learning w
 
 - `Tune`_: Scalable Hyperparameter Tuning
 - `RLlib`_: Scalable Reinforcement Learning
-- `Distributed Training <distributed_training.html>`__
+- `RaySGD`_: Distributed Training
 
 
 Star us on `on GitHub`_. You can also get started by visiting our `Tutorials <https://github.com/ray-project/tutorial>`_. For the latest wheels (nightlies), see the `installation page <installation.html>`__.
 
 .. _`on GitHub`: https://github.com/ray-project/ray
+.. _`RaySGD`: raysgd/raysgd.html
 
 
 Quick Start
@@ -272,12 +273,16 @@ Getting Involved
    rllib-dev.rst
    rllib-package-ref.rst
 
+.. toctree::
+   :maxdepth: -1
+   :caption: RaySGD
+
+   raysgd/raysgd.rst
+
 .. toctree::
    :maxdepth: -1
    :caption: Experimental
 
-   distributed_training.rst
-   tf_distributed_training.rst
    pandas_on_ray.rst
    projects.rst
    signals.rst

doc/source/raysgd/raysgd.rst

@@ -0,0 +1,22 @@
+RaySGD: Distributed Deep Learning
+=================================
+
+.. image:: raysgdlogo.png
+    :scale: 20%
+    :align: center
+
+RaySGD is a lightweight library for distributed deep learning, providing thin wrappers around framework-native modules for data parallel training.
+
+The main features are:
+
+  - Ease of use: Scale Pytorch's native ``DistributedDataParallel`` and TensorFlow's ``tf.distribute.MirroredStrategy`` without needing to monitor individual nodes.
+  - Composibility: RaySGD is built on top of the Ray Actor API, enabling seamless integration with existing Ray applications such as RLlib, Tune, and Ray.Serve.
+  - Scale up and down: Start on single CPU. Scale up to multi-node, multi-gpu by changing 2 lines of code.
+
+
+
+.. toctree::
+
+   raysgd_pytorch.rst
+   raysgd_tensorflow.rst
+   raysgd_ft.rst

doc/source/raysgd/raysgd_ft.rst

@@ -0,0 +1,33 @@
+RaySGD Fault Tolerance
+======================
+
+.. note:: Fault tolerance is currently only enabled for the PyTorchTrainer.
+
+For distributed deep learning, jobs are often run on infrastructure where nodes can be pre-empted frequently (i.e., spot instances in the cloud). To overcome this, RaySGD provides **fault tolerance** features that enable training to continue regardless of node failures.
+
+.. code-block:: bash
+
+    trainer.train(max_retries=N)
+
+
+How does it work?
+-----------------
+
+During each ``train`` method, each parallel worker iterates through the dataset, synchronizing gradients and parameters at each batch. These synchronization primitives can hang when one or more of the parallel workers becomes unresponsive (i.e., when a node is lost). To address this, we've implemented the following protocol.
+
+  1. If any worker node is lost, Ray will mark the training task as complete (``ray.wait`` will return).
+  2. Ray will throw ``RayActorException`` when fetching the result for any worker, so the Trainer class will call ``ray.get`` on the "finished" training task.
+  3. Upon catching this exception, the Trainer class will kill all of its workers.
+  4. The Trainer will then detect the quantity of available resources (either CPUs or GPUs). It will then restart as many workers as it can, each resuming from the last checkpoint. Note that this may result in fewer workers than initially specified.
+  5. If there are no available resources, the Trainer will apply an exponential backoff before retrying to create workers.
+  6. If there are available resources and the Trainer has fewer workers than initially specified, then it will scale up its worker pool until it reaches the initially specified ``num_workers``.
+
+Note that we assume the Trainer itself is not on a pre-emptible node. It is currently not possible to recover from a Trainer node failure.
+
+Users can set ``checkpoint="auto"`` to always checkpoint the current model before executing a pass over the training dataset.
+
+.. code-block:: bash
+
+    trainer.train(max_retries=N, checkpoint="auto")
+
+

doc/source/raysgd/raysgd_pytorch.rst

@@ -1,5 +1,7 @@
-Distributed Training (Experimental)
-===================================
+RaySGD Pytorch
+==============
+
+.. warning:: This is still an experimental API and is subject to change in the near future.
 
 Ray's ``PyTorchTrainer`` simplifies distributed model training for PyTorch. The ``PyTorchTrainer`` is a wrapper around ``torch.distributed.launch`` with a Python API to easily incorporate distributed training into a larger Python application, as opposed to needing to execute training outside of Python.
 
@@ -84,7 +86,7 @@ PyTorchTrainer Example
 
 Below is an example of using Ray's PyTorchTrainer. Under the hood, ``PytorchTrainer`` will create *replicas* of your model (controlled by ``num_replicas``) which are each managed by a worker.
 
-.. literalinclude:: ../../python/ray/experimental/sgd/examples/train_example.py
+.. literalinclude:: ../../../python/ray/experimental/sgd/examples/train_example.py
    :language: python
    :start-after: __torch_train_example__
 
@@ -94,7 +96,7 @@ Hyperparameter Optimization on Distributed Pytorch
 
 ``PyTorchTrainer`` naturally integrates with Tune via the ``PyTorchTrainable`` interface. The same arguments to ``PyTorchTrainer`` should be passed into the ``tune.run(config=...)`` as shown below.
 
-.. literalinclude:: ../../python/ray/experimental/sgd/examples/tune_example.py
+.. literalinclude:: ../../../python/ray/experimental/sgd/examples/tune_example.py
    :language: python
    :start-after: __torch_tune_example__
 

doc/source/raysgd/raysgd_tensorflow.rst

@@ -1,7 +1,7 @@
-TF Distributed Training
-=======================
+RaySGD TensorFlow
+=================
 
-Ray's ``TFTrainer`` simplifies distributed model training for Tensorflow. The ``TFTrainer`` is a wrapper around ``MultiWorkerMirroredStrategy`` with a Python API to easily incorporate distributed training into a larger Python application, as opposed to write custom logic of setting environments and starting separate processes.
+RaySGD's ``TFTrainer`` simplifies distributed model training for Tensorflow. The ``TFTrainer`` is a wrapper around ``MultiWorkerMirroredStrategy`` with a Python API to easily incorporate distributed training into a larger Python application, as opposed to write custom logic of setting environments and starting separate processes.
 
 .. important:: This API has only been tested with TensorFlow2.0rc and is still highly experimental. Please file bug reports if you run into any - thanks!
 
@@ -67,7 +67,7 @@ TFTrainer Example
 
 Below is an example of using Ray's TFTrainer. Under the hood, ``TFTrainer`` will create *replicas* of your model (controlled by ``num_replicas``) which are each managed by a worker.
 
-.. literalinclude:: ../../python/ray/experimental/sgd/examples/tensorflow_train_example.py
+.. literalinclude:: ../../../python/ray/experimental/sgd/examples/tensorflow_train_example.py
    :language: python