[tune] Update node syncing documentation (#10126)

doc/source/tune/_tutorials/tune-pytorch-cifar.rst

@@ -148,10 +148,10 @@ to decide which hyperparameter configuration lead to the best results. These met
 can also be used to stop bad performing trials early in order to avoid wasting
 resources on those trials.
 
-The checkpoint saving is optional, however, it is necessary if we wanted to use advanced
+The :ref:`checkpoint saving <tune-checkpoint>` is optional. However, it is necessary if we wanted to use advanced
 schedulers like `Population Based Training <https://docs.ray.io/en/master/tune/tutorials/tune-advanced-tutorial.html>`_.
-Also, by saving the checkpoint we can later load the trained models and validate them
-on a test set.
+After training, we can also restore the checkpointed models and validate them on a test set.
+
 
 Full training function
 ~~~~~~~~~~~~~~~~~~~~~~

doc/source/tune/user-guide.rst

@@ -143,14 +143,12 @@ During training, Tune will automatically log the below metrics in addition to th
 Checkpointing
 -------------
 
-When running a hyperparameter search, Tune can automatically and periodically save/checkpoint your model. Checkpointing is used for
+When running a hyperparameter search, Tune can automatically and periodically save/checkpoint your model. This allows you to:
 
- * saving a model throughout training
- * fault-tolerance when using pre-emptible machines.
+ * save intermediate models throughout training
+ * use pre-emptible machines (by automatically restoring from last checkpoint)
  * Pausing trials when using Trial Schedulers such as HyperBand and PBT.
 
-Checkpointing assumes that the model state will be saved to disk on whichever node the Trainable is running on.
-
 To use Tune's checkpointing features, you must expose a ``checkpoint_dir`` argument in the function signature, and call ``tune.checkpoint_dir``:
 
 .. code-block:: python
@@ -194,6 +192,21 @@ You can restore a single trial checkpoint by using ``tune.run(restore=<checkpoin
         config={"env": "CartPole-v0"},
     )
 
+Distributed Checkpointing
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+On a multinode cluster, Tune automatically creates a copy of all trial checkpoints on the head node. This requires the Ray cluster to be started with the :ref:`cluster launcher <ref-automatic-cluster>` and also requires rsync to be installed.
+
+Note that you must use the ``tune.checkpoint_dir`` API to trigger syncing. Also, if running Tune on Kubernetes, be sure to use the :ref:`KubernetesSyncer <tune-kubernetes>` to transfer files between different pods. 
+
+If you do not use the cluster launcher, you should set up a NFS or global file system and
+disable cross-node syncing:
+
+.. code-block:: python
+
+    tune.run(func, sync_to_driver=False)
+
+
 Handling Large Datasets
 -----------------------
 
@@ -540,4 +553,4 @@ You can post questions or issues or feedback through the following channels:
 2. `GitHub Issues`_: For bug reports and feature requests.
 
 .. _`StackOverflow`: https://stackoverflow.com/questions/tagged/ray
-.. _`GitHub Issues`: https://github.com/ray-project/ray/issues
+.. _`GitHub Issues`: https://github.com/ray-project/ray/issues