From d87c1867213d4f5b73c0dc907a7ec2652f026daa Mon Sep 17 00:00:00 2001 From: Amog Kamsetty Date: Thu, 22 Oct 2020 13:32:27 -0700 Subject: [PATCH] [RaySGD] Docs for SGD+Tune usage (#11479) --- doc/source/index.rst | 1 + doc/source/raysgd/raysgd_pytorch.rst | 14 +------- doc/source/raysgd/raysgd_tune.rst | 51 ++++++++++++++++++++++++++++ 3 files changed, 53 insertions(+), 13 deletions(-) create mode 100644 doc/source/raysgd/raysgd_tune.rst diff --git a/doc/source/index.rst b/doc/source/index.rst index 7ef69749c..b840e0f46 100644 --- a/doc/source/index.rst +++ b/doc/source/index.rst @@ -272,6 +272,7 @@ Papers raysgd/raysgd_pytorch.rst raysgd/raysgd_tensorflow.rst raysgd/raysgd_dataset.rst + raysgd/raysgd_tune.rst raysgd/raysgd_ref.rst .. toctree:: diff --git a/doc/source/raysgd/raysgd_pytorch.rst b/doc/source/raysgd/raysgd_pytorch.rst index 0f09f031e..bcd8379fd 100644 --- a/doc/source/raysgd/raysgd_pytorch.rst +++ b/doc/source/raysgd/raysgd_pytorch.rst @@ -21,7 +21,7 @@ Basic Usage Setting up training ~~~~~~~~~~~~~~~~~~~ -.. tip:: If you want to leverage multi-node data parallel training with PyTorch while using RayTune *without* restructuring your code, check out the :ref:`Tune PyTorch user guide ` and Tune's :ref:`distributed pytorch integrations `. +.. tip:: If you want to leverage multi-node data parallel training with PyTorch while using RayTune *without* using RaySGD, check out the :ref:`Tune PyTorch user guide ` and Tune's :ref:`distributed pytorch integrations `. The :ref:`ref-torch-trainer` can be constructed from a custom :ref:`ref-torch-operator` subclass that defines training components like the model, data, optimizer, loss, and ``lr_scheduler``. These components are all automatically replicated across different machines and devices so that training can be executed in parallel. @@ -467,18 +467,6 @@ During each ``train`` method, each parallel worker iterates through the iterable Note that we assume the Trainer itself is not on a pre-emptible node. To allow the entire Trainer to recover from failure, you must use Tune to execute the training. -Advanced: Hyperparameter Tuning -------------------------------- -``TorchTrainer`` naturally integrates with Tune via the ``BaseTorchTrainable`` interface. Without changing any arguments, you can call ``TorchTrainer.as_trainable(model_creator...)`` to create a Tune-compatible class. See the documentation (:ref:`BaseTorchTrainable-doc`). - -.. literalinclude:: ../../../python/ray/util/sgd/torch/examples/tune_example.py - :language: python - :start-after: __torch_tune_example__ - :end-before: __end_torch_tune_example__ - -You can see the `Tune example script `_ for an end-to-end example. - - Simultaneous Multi-model Training --------------------------------- diff --git a/doc/source/raysgd/raysgd_tune.rst b/doc/source/raysgd/raysgd_tune.rst new file mode 100644 index 000000000..0826ad97b --- /dev/null +++ b/doc/source/raysgd/raysgd_tune.rst @@ -0,0 +1,51 @@ +RaySGD Hyperparameter Tuning +============================ + +RaySGD integrates with :ref:`Ray Tune ` to easily run distributed hyperparameter tuning experiments with your RaySGD Trainer. + +PyTorch +------- + +.. tip:: If you want to leverage multi-node data parallel training with PyTorch while using RayTune *without* using RaySGD, check out the :ref:`Tune PyTorch user guide ` and Tune's lightweight :ref:`distributed pytorch integrations `. + +``TorchTrainer`` naturally integrates with Tune via the ``BaseTorchTrainable`` interface. Without changing any arguments, you can call ``TorchTrainer.as_trainable(...)`` to create a Tune-compatible class. +Then, you can simply pass the returned Trainable class to ``tune.run``. The ``config`` used for each ``Trainable`` in tune will automatically be passed down to the ``TorchTrainer``. +Therefore, each trial will have its own ``TorchTrainable`` that holds an instance of the ``TorchTrainer`` with its own unique hyperparameter configuration. +See the documentation (:ref:`BaseTorchTrainable-doc`) for more info. + +.. literalinclude:: ../../../python/ray/util/sgd/torch/examples/tune_example.py + :language: python + :start-after: __torch_tune_example__ + :end-before: __end_torch_tune_example__ + +By default the training step for the returned ``Trainable`` will run one epoch of training and one epoch of validation, and will report +the combined result dictionaries to Tune. + +By combining RaySGD with Tune, each individual trial will be run in a distributed fashion with ``num_workers`` workers, +but there can be multiple trials running in parallel as well. + +Custom Training Step +~~~~~~~~~~~~~~~~~~~~ +Sometimes it is necessary to provide a custom training step, for example if you want to run more than one epoch of training for +each tune iteration, or you need to manually update the scheduler after validation. Custom training steps can easily be provided by passing +in a ``override_tune_step`` function to ``TorchTrainer.as_trainable(...)``. + +.. literalinclude:: ../../../python/ray/util/sgd/torch/examples/tune_example.py + :language: python + :start-after: __torch_tune_manual_lr_example__ + :end-before: __end_torch_tune_manual_lr_example__ + +Your custom step function should take in two arguments: an instance of the ``TorchTrainer`` and an ``info`` dict containing other potentially +necessary information. + +The info dict contains the following values: + +.. code-block:: python + + # The current Tune iteration. + # This may be different than the number of epochs trained if each tune step does more than one epoch of training. + iteration + +If you would like any other information to be available in the ``info`` dict please file a feature request on `Github Issues `_! + +You can see the `Tune example script `_ for an end-to-end example.