diff --git a/doc/source/tune/_tutorials/tune-pytorch-cifar.rst b/doc/source/tune/_tutorials/tune-pytorch-cifar.rst index 743f44463..c4e291226 100644 --- a/doc/source/tune/_tutorials/tune-pytorch-cifar.rst +++ b/doc/source/tune/_tutorials/tune-pytorch-cifar.rst @@ -148,10 +148,10 @@ to decide which hyperparameter configuration lead to the best results. These met can also be used to stop bad performing trials early in order to avoid wasting resources on those trials. -The checkpoint saving is optional, however, it is necessary if we wanted to use advanced +The :ref:`checkpoint saving ` is optional. However, it is necessary if we wanted to use advanced schedulers like `Population Based Training `_. -Also, by saving the checkpoint we can later load the trained models and validate them -on a test set. +After training, we can also restore the checkpointed models and validate them on a test set. + Full training function ~~~~~~~~~~~~~~~~~~~~~~ diff --git a/doc/source/tune/user-guide.rst b/doc/source/tune/user-guide.rst index 8f99e24b1..33712aa0d 100644 --- a/doc/source/tune/user-guide.rst +++ b/doc/source/tune/user-guide.rst @@ -143,14 +143,12 @@ During training, Tune will automatically log the below metrics in addition to th Checkpointing ------------- -When running a hyperparameter search, Tune can automatically and periodically save/checkpoint your model. Checkpointing is used for +When running a hyperparameter search, Tune can automatically and periodically save/checkpoint your model. This allows you to: - * saving a model throughout training - * fault-tolerance when using pre-emptible machines. + * save intermediate models throughout training + * use pre-emptible machines (by automatically restoring from last checkpoint) * Pausing trials when using Trial Schedulers such as HyperBand and PBT. -Checkpointing assumes that the model state will be saved to disk on whichever node the Trainable is running on. - To use Tune's checkpointing features, you must expose a ``checkpoint_dir`` argument in the function signature, and call ``tune.checkpoint_dir``: .. code-block:: python @@ -194,6 +192,21 @@ You can restore a single trial checkpoint by using ``tune.run(restore=` and also requires rsync to be installed. + +Note that you must use the ``tune.checkpoint_dir`` API to trigger syncing. Also, if running Tune on Kubernetes, be sure to use the :ref:`KubernetesSyncer ` to transfer files between different pods. + +If you do not use the cluster launcher, you should set up a NFS or global file system and +disable cross-node syncing: + +.. code-block:: python + + tune.run(func, sync_to_driver=False) + + Handling Large Datasets ----------------------- @@ -540,4 +553,4 @@ You can post questions or issues or feedback through the following channels: 2. `GitHub Issues`_: For bug reports and feature requests. .. _`StackOverflow`: https://stackoverflow.com/questions/tagged/ray -.. _`GitHub Issues`: https://github.com/ray-project/ray/issues \ No newline at end of file +.. _`GitHub Issues`: https://github.com/ray-project/ray/issues