mirror of
https://github.com/wassname/ray.git
synced 2026-09-10 12:38:43 +08:00
[tune] New Doc edits, add Concepts page (#8083)
Co-Authored-By: Sven Mika <sven@anyscale.io>
This commit is contained in:
Richard Liaw
co-authored by
Sven Mika
Sven Mika
parent
69ff7e3e35
commit
b506f87117
@@ -0,0 +1 @@
|
||||
:orphan:
|
||||
@@ -0,0 +1,177 @@
|
||||
.. _tune-guides-overview:
|
||||
|
||||
Tutorials, User Guides, Examples
|
||||
================================
|
||||
|
||||
In this section, you can find material on how to use Tune and its various features. If any of the materials is out of date or broken, or if you'd like to add an example to this page, feel free to raise an issue on our Github repository.
|
||||
|
||||
|
||||
Tutorials
|
||||
---------
|
||||
|
||||
Take a look at any of the below tutorials to get started with Tune.
|
||||
|
||||
.. raw:: html
|
||||
|
||||
<div class="sphx-glr-bigcontainer">
|
||||
|
||||
.. customgalleryitem::
|
||||
:tooltip: A gentle 60 second tour of core Tune concepts.
|
||||
:figure: /images/tune-workflow.png
|
||||
:description: :doc:`A gentle 60 second tour of Tune <tune-60-seconds>`
|
||||
|
||||
.. customgalleryitem::
|
||||
:tooltip: A simple Tune walkthrough.
|
||||
:figure: /images/tune.png
|
||||
:description: :doc:`A walkthrough to setup your first Tune experiment <tune-tutorial>`
|
||||
|
||||
.. raw:: html
|
||||
|
||||
</div>
|
||||
|
||||
.. toctree::
|
||||
:hidden:
|
||||
|
||||
tune-60-seconds.rst
|
||||
tune-tutorial.rst
|
||||
|
||||
|
||||
User Guides
|
||||
-----------
|
||||
|
||||
These pages will demonstrate the various features and configurations of Tune.
|
||||
|
||||
.. raw:: html
|
||||
|
||||
<div class="sphx-glr-bigcontainer">
|
||||
|
||||
.. customgalleryitem::
|
||||
:tooltip: A guide to Tune features.
|
||||
:figure: /images/tune.png
|
||||
:description: :doc:`A guide to Tune features <tune-usage>`
|
||||
|
||||
.. customgalleryitem::
|
||||
:tooltip: A simple guide to Population-based Training
|
||||
:figure: /images/tune-pbt-small.png
|
||||
:description: :doc:`A simple guide to Population-based Training <tune-advanced-tutorial>`
|
||||
|
||||
.. customgalleryitem::
|
||||
:tooltip: A guide to distributed hyperparameter tuning
|
||||
:figure: /images/tune.png
|
||||
:description: :doc:`A guide to distributed hyperparameter tuning <tune-distributed>`
|
||||
|
||||
.. raw:: html
|
||||
|
||||
</div>
|
||||
|
||||
.. toctree::
|
||||
:hidden:
|
||||
|
||||
tune-usage.rst
|
||||
tune-advanced-tutorial.rst
|
||||
tune-distributed.rst
|
||||
|
||||
Colab Exercises
|
||||
---------------
|
||||
|
||||
Learn how to use Tune in your browser with the following Colab-based exercises.
|
||||
|
||||
.. raw:: html
|
||||
|
||||
<table>
|
||||
<tr>
|
||||
<th class="tune-colab">Exercise Description</th>
|
||||
<th class="tune-colab">Library</th>
|
||||
<th class="tune-colab">Colab Link</th>
|
||||
</tr>
|
||||
<tr>
|
||||
<td class="tune-colab">Basics of using Tune.</td>
|
||||
<td class="tune-colab">TF/Keras</td>
|
||||
<td class="tune-colab">
|
||||
<a href="https://colab.research.google.com/github/ray-project/tutorial/blob/master/tune_exercises/exercise_1_basics.ipynb" target="_parent">
|
||||
<img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Tune Tutorial"/>
|
||||
</a>
|
||||
</td>
|
||||
</tr>
|
||||
|
||||
<tr>
|
||||
<td class="tune-colab">Using Search algorithms and Trial Schedulers to optimize your model.</td>
|
||||
<td class="tune-colab">Pytorch</td>
|
||||
<td class="tune-colab">
|
||||
<a href="https://colab.research.google.com/github/ray-project/tutorial/blob/master/tune_exercises/exercise_2_optimize.ipynb" target="_parent">
|
||||
<img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Tune Tutorial"/>
|
||||
</a>
|
||||
</td>
|
||||
</tr>
|
||||
|
||||
<tr>
|
||||
<td class="tune-colab">Using Population-Based Training (PBT).</td>
|
||||
<td class="tune-colab">Pytorch</td>
|
||||
<td class="tune-colab">
|
||||
<a href="https://colab.research.google.com/github/ray-project/tutorial/blob/master/tune_exercises/exercise_3_pbt.ipynb" target="_parent">
|
||||
<img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Tune Tutorial"/>
|
||||
</a>
|
||||
</td>
|
||||
</tr>
|
||||
</table>
|
||||
|
||||
Tutorial source files `can be found here <https://github.com/ray-project/tutorial>`_.
|
||||
|
||||
Tune Examples
|
||||
-------------
|
||||
|
||||
.. Keep this in sync with ray/python/ray/tune/examples/README.rst
|
||||
|
||||
If any example is broken, or if you'd like to add an example to this page, feel free to raise an issue on our Github repository.
|
||||
|
||||
|
||||
General Examples
|
||||
~~~~~~~~~~~~~~~~
|
||||
|
||||
- `async_hyperband_example <https://github.com/ray-project/ray/blob/master/python/ray/tune/examples/async_hyperband_example.py>`__: Example of using a Trainable class with AsyncHyperBandScheduler.
|
||||
- `hyperband_example <https://github.com/ray-project/ray/blob/master/python/ray/tune/examples/hyperband_example.py>`__: Example of using a Trainable class with HyperBandScheduler. Also uses the Experiment class API for specifying the experiment configuration. Also uses the AsyncHyperBandScheduler.
|
||||
- `pbt_example <https://github.com/ray-project/ray/blob/master/python/ray/tune/examples/pbt_example.py>`__: Example of using a Trainable class with PopulationBasedTraining scheduler.
|
||||
- `pbt_ppo_example <https://github.com/ray-project/ray/blob/master/python/ray/tune/examples/pbt_ppo_example.py>`__: Example of optimizing a distributed RLlib algorithm (PPO) with the PopulationBasedTraining scheduler.
|
||||
- `logging_example <https://github.com/ray-project/ray/blob/master/python/ray/tune/examples/logging_example.py>`__: Example of custom loggers and custom trial directory naming.
|
||||
|
||||
Search Algorithm Examples
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
- `Ax example <https://github.com/ray-project/ray/blob/master/python/ray/tune/examples/ax_example.py>`__: Optimize a Hartmann function with `Ax <https://ax.dev>`_ with 4 parallel workers.
|
||||
- `HyperOpt Example <https://github.com/ray-project/ray/blob/master/python/ray/tune/examples/hyperopt_example.py>`__: Optimizes a basic function using the function-based API and the HyperOptSearch (SearchAlgorithm wrapper for HyperOpt TPE).
|
||||
- `Nevergrad example <https://github.com/ray-project/ray/blob/master/python/ray/tune/examples/nevergrad_example.py>`__: Optimize a simple toy function with the gradient-free optimization package `Nevergrad <https://github.com/facebookresearch/nevergrad>`_ with 4 parallel workers.
|
||||
- `Bayesian Optimization example <https://github.com/ray-project/ray/blob/master/python/ray/tune/examples/bayesopt_example.py>`__: Optimize a simple toy function using `Bayesian Optimization <https://github.com/fmfn/BayesianOptimization>`_ with 4 parallel workers.
|
||||
|
||||
Tensorflow/Keras Examples
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
- `tune_mnist_keras <https://github.com/ray-project/ray/blob/master/python/ray/tune/examples/tune_mnist_keras.py>`__: Converts the Keras MNIST example to use Tune with the function-based API and a Keras callback. Also shows how to easily convert something relying on argparse to use Tune.
|
||||
- `pbt_memnn_example <https://github.com/ray-project/ray/blob/master/python/ray/tune/examples/pbt_memnn_example.py>`__: Example of training a Memory NN on bAbI with Keras using PBT.
|
||||
- `Tensorflow 2 Example <https://github.com/ray-project/ray/blob/master/python/ray/tune/examples/tf_mnist_example.py>`__: Converts the Advanced TF2.0 MNIST example to use Tune with the Trainable. This uses `tf.function`. Original code from tensorflow: https://www.tensorflow.org/tutorials/quickstart/advanced
|
||||
|
||||
|
||||
PyTorch Examples
|
||||
~~~~~~~~~~~~~~~~
|
||||
|
||||
- `mnist_pytorch <https://github.com/ray-project/ray/blob/master/python/ray/tune/examples/mnist_pytorch.py>`__: Converts the PyTorch MNIST example to use Tune with the function-based API. Also shows how to easily convert something relying on argparse to use Tune.
|
||||
- `mnist_pytorch_trainable <https://github.com/ray-project/ray/blob/master/python/ray/tune/examples/mnist_pytorch_trainable.py>`__: Converts the PyTorch MNIST example to use Tune with Trainable API. Also uses the HyperBandScheduler and checkpoints the model at the end.
|
||||
|
||||
|
||||
XGBoost Example
|
||||
~~~~~~~~~~~~~~~
|
||||
|
||||
- `xgboost_example <https://github.com/ray-project/ray/blob/master/python/ray/tune/examples/xgboost_example.py>`__: Trains a basic XGBoost model with Tune with the function-based API and an XGBoost callback.
|
||||
|
||||
|
||||
LightGBM Example
|
||||
~~~~~~~~~~~~~~~~
|
||||
|
||||
- `lightgbm_example <https://github.com/ray-project/ray/blob/master/python/ray/tune/examples/lightgbm_example.py>`__: Trains a basic LightGBM model with Tune with the function-based API and a LightGBM callback.
|
||||
|
||||
|
||||
Contributed Examples
|
||||
~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
- `pbt_tune_cifar10_with_keras <https://github.com/ray-project/ray/blob/master/python/ray/tune/examples/pbt_tune_cifar10_with_keras.py>`__: A contributed example of tuning a Keras model on CIFAR10 with the PopulationBasedTraining scheduler.
|
||||
- `genetic_example <https://github.com/ray-project/ray/blob/master/python/ray/tune/examples/genetic_example.py>`__: Optimizing the michalewicz function using the contributed GeneticSearch algorithm with AsyncHyperBandScheduler.
|
||||
- `tune_cifar10_gluon <https://github.com/ray-project/ray/blob/master/python/ray/tune/examples/tune_cifar10_gluon.py>`__: MXNet Gluon example to use Tune with the function-based API on CIFAR-10 dataset.
|
||||
@@ -0,0 +1,193 @@
|
||||
.. _tune-60-seconds:
|
||||
|
||||
Tune in 60 Seconds
|
||||
==================
|
||||
|
||||
Let's quickly walk through the key concepts you need to know to use Tune. In this guide, we'll be covering the following:
|
||||
|
||||
.. contents::
|
||||
:local:
|
||||
:depth: 1
|
||||
|
||||
Tune takes a user-defined Python function or class and evaluates it on a set of hyperparameter configurations. Each hyperparameter configuration evaluation is called a *trial*, and Tune runs multiple trials in parallel, leveraging Search Algorithms and Trial Schedulers to optimize your hyperparameters.
|
||||
|
||||
.. image:: /images/tune-workflow.png
|
||||
|
||||
Trainables
|
||||
----------
|
||||
|
||||
To allow Tune to optimize your model, Tune will need to control your training process. This is done via the Trainable API. Each *trial* corresponds to one instance of a Trainable; Tune will create multiple instances of the Trainable.
|
||||
|
||||
The Trainable API is where you specify how to set up your model and track intermediate training progress. There are two types of Trainables - a **function-based API** is for fast prototyping, and **class-based** API that unlocks many Tune features such as checkpointing, pausing.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from ray import tune
|
||||
|
||||
class Trainable(tune.Trainable):
|
||||
"""Tries to iteratively find the password."""
|
||||
|
||||
def _setup(self, config):
|
||||
self.iter = 0
|
||||
self.password = 1024
|
||||
|
||||
def _train(self):
|
||||
"""Execute one step of 'training'. This function will be called iteratively"""
|
||||
self.iter += 1
|
||||
return {
|
||||
"accuracy": abs(self.iter - self.password),
|
||||
"training_iteration": self.iter # Tune will automatically provide this.
|
||||
}
|
||||
|
||||
def _stop(self):
|
||||
# perform any cleanup necessary.
|
||||
pass
|
||||
|
||||
Function API example:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
def trainable(config):
|
||||
"""
|
||||
Args:
|
||||
config (dict): Parameters provided from the search algorithm
|
||||
or variant generation.
|
||||
"""
|
||||
|
||||
while True:
|
||||
# ...
|
||||
tune.track.log(**kwargs)
|
||||
|
||||
.. tip:: Do not use ``tune.track.log`` within a ``Trainable`` class.
|
||||
|
||||
See the documentation: :ref:`trainable-docs`.
|
||||
|
||||
tune.run
|
||||
--------
|
||||
|
||||
Use ``tune.run`` execute hyperparameter tuning using the core Ray APIs. This function manages your distributed experiment and provides many features such as logging, checkpointing, and early stopping.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
# Pass in a Trainable class or function to tune.run.
|
||||
tune.run(trainable)
|
||||
|
||||
# Run 10 trials (each trial is one instance of a Trainable). Tune runs in
|
||||
# parallel and automatically determines concurrency.
|
||||
tune.run(trainable, num_samples=10)
|
||||
|
||||
# Run 1 trial, stop when trial has reached 10 iterations OR a mean accuracy of 0.98.
|
||||
tune.run(my_trainable, stop={"training_iteration": 10, "mean_accuracy": 0.98})
|
||||
|
||||
# Run 1 trial, search over hyperparameters, stop after 10 iterations.
|
||||
hyperparameters = {"lr": tune.uniform(0, 1), "momentum": tune.uniform(0, 1)}
|
||||
tune.run(my_trainable, config=hyperparameters, stop={"training_iteration": 10})
|
||||
|
||||
This function will report status on the command line until all Trials stop:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
== Status ==
|
||||
Memory usage on this node: 11.4/16.0 GiB
|
||||
Using FIFO scheduling algorithm.
|
||||
Resources requested: 4/12 CPUs, 0/0 GPUs, 0.0/3.17 GiB heap, 0.0/1.07 GiB objects
|
||||
Result logdir: /Users/foo/ray_results/myexp
|
||||
Number of trials: 4 (4 RUNNING)
|
||||
+----------------------+----------+---------------------+-----------+--------+--------+----------------+-------+
|
||||
| Trial name | status | loc | param1 | param2 | acc | total time (s) | iter |
|
||||
|----------------------+----------+---------------------+-----------+--------+--------+----------------+-------|
|
||||
| MyTrainable_a826033a | RUNNING | 10.234.98.164:31115 | 0.303706 | 0.0761 | 0.1289 | 7.54952 | 15 |
|
||||
| MyTrainable_a8263fc6 | RUNNING | 10.234.98.164:31117 | 0.929276 | 0.158 | 0.4865 | 7.0501 | 14 |
|
||||
| MyTrainable_a8267914 | RUNNING | 10.234.98.164:31111 | 0.068426 | 0.0319 | 0.9585 | 7.0477 | 14 |
|
||||
| MyTrainable_a826b7bc | RUNNING | 10.234.98.164:31112 | 0.729127 | 0.0748 | 0.1797 | 7.05715 | 14 |
|
||||
+----------------------+----------+---------------------+-----------+--------+--------+----------------+-------+
|
||||
|
||||
See the documentation: :ref:`tune-run-ref`.
|
||||
|
||||
|
||||
Search Algorithms
|
||||
-----------------
|
||||
|
||||
To optimize the hyperparameters of your training process, you will want to explore a “search space”.
|
||||
|
||||
Search Algorithms are Tune modules that help explore a provided search space. It will use previous results from evaluating different hyperparameters to suggest better hyperparameters. Tune has SearchAlgorithms that integrate with many popular **optimization** libraries, such as `Nevergrad <https://github.com/facebookresearch/nevergrad>`_ and `Hyperopt <https://github.com/hyperopt/hyperopt/>`_.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
# https://github.com/hyperopt/hyperopt/
|
||||
# pip install hyperopt
|
||||
import hyperopt as hp
|
||||
from ray.tune.suggest.hyperopt import HyperOptSearch
|
||||
|
||||
# Create a HyperOpt search space
|
||||
space = {"momentum": hp.uniform("momentum", 0, 20), "lr": hp.uniform("lr", 0, 1)}
|
||||
# Pass the search space into Tune's HyperOpt wrapper and maximize accuracy
|
||||
hyperopt = HyperOptSearch(space, metric="accuracy", mode="max")
|
||||
|
||||
# Execute 20 trials using HyperOpt, stop after 20 iterations
|
||||
max_iters = {"training_iteration": 20}
|
||||
tune.run(trainable, search_alg=hyperopt, num_samples=20, stop=max_iters)
|
||||
|
||||
See the documentation: :ref:`searchalg-ref`.
|
||||
|
||||
Trial Schedulers
|
||||
----------------
|
||||
|
||||
In addition, you can make your training process more efficient by stopping, pausing, or changing the hyperparameters of running trials.
|
||||
|
||||
Trial Schedulers are Tune modules that adjust and change distributed training runs during execution. These modules can stop/pause/tweak the hyperparameters of running trials, making your hyperparameter tuning process much faster. Population-based training and HyperBand are examples of popular optimization algorithms implemented as Trial Schedulers.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from ray.tune.schedulers import HyperBandScheduler
|
||||
|
||||
# Create HyperBand scheduler and maximize accuracy
|
||||
hyperband = HyperBandScheduler(metric="accuracy", mode="max")
|
||||
|
||||
# Execute 20 trials using HyperBand using a search space
|
||||
configs = {"lr": tune.uniform(0, 1), "momentum": tune.uniform(0, 1)}
|
||||
tune.run(MyTrainableClass, num_samples=20, config=configs, scheduler=hyperband)
|
||||
|
||||
Unlike **Search Algorithms**, Trial Schedulers do not select which hyperparameter configurations to evaluate. However, you can use them together.
|
||||
|
||||
See the documentation: :ref:`schedulers-ref`.
|
||||
|
||||
|
||||
Analysis
|
||||
--------
|
||||
|
||||
After running a hyperparameter tuning job, you will want to analyze your results to determine what specific parameters are important and which hyperparameter values are the best.
|
||||
|
||||
``tune.run`` returns an :ref:`Analysis <tune-analysis-docs>` object which has methods you can use for analyzing your results. This object can also retrieve all training runs as dataframes, allowing you to do ad-hoc data analysis over your results.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
analysis = tune.run(trainable, search_alg=algo, stop={"training_iteration": 20})
|
||||
|
||||
# Get the best hyperparameters
|
||||
best_hyperparameters = analysis.get_best_config()
|
||||
|
||||
# Get a dataframe for the max accuracy seen for each trial
|
||||
df = analysis.dataframe(metric="mean_accuracy", mode="max")
|
||||
|
||||
What's Next?
|
||||
~~~~~~~~~~~~
|
||||
|
||||
|
||||
Now that you have a working understanding of Tune, check out:
|
||||
|
||||
* :ref:`Tune Guides and Examples <tune-guides-overview>`: Examples and templates for using Tune with your preferred machine learning library.
|
||||
* :ref:`tune-tutorial`: A simple tutorial that walks you through the process of setting up a Tune experiment.
|
||||
* :ref:`tune-user-guide`: A comprehensive overview of Tune's features.
|
||||
|
||||
|
||||
Further Questions or Issues?
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Reach out to us if you have any questions or issues or feedback through the following channels:
|
||||
|
||||
1. `StackOverflow`_: For questions about how to use Ray.
|
||||
2. `GitHub Issues`_: For bug reports and feature requests.
|
||||
|
||||
.. _`StackOverflow`: https://stackoverflow.com/questions/tagged/ray
|
||||
.. _`GitHub Issues`: https://github.com/ray-project/ray/issues
|
||||
@@ -0,0 +1,204 @@
|
||||
Guide to Population Based Training (PBT)
|
||||
========================================
|
||||
|
||||
Tune includes a distributed implementation of `Population Based Training (PBT) <https://deepmind.com/blog/population-based-training-neural-networks>`__ as
|
||||
a :ref:`scheduler <tune-scheduler-pbt>`.
|
||||
|
||||
.. image:: /images/tune_advanced_paper1.png
|
||||
|
||||
|
||||
PBT starts by training many neural networks in parallel with random hyperparameters, using information from the rest of the population to refine these
|
||||
hyperparameters and allocate resources to promising models. Let's walk through how to use this algorithm.
|
||||
|
||||
.. contents::
|
||||
:local:
|
||||
:backlinks: none
|
||||
|
||||
|
||||
Trainable API with Population Based Training
|
||||
--------------------------------------------
|
||||
|
||||
PBT takes its inspiration from genetic algorithms where each member of the population
|
||||
can exploit information from the remainder of the population. For example, a worker might
|
||||
copy the model parameters from a better performing worker. It can also explore new hyperparameters by
|
||||
changing the current values randomly.
|
||||
|
||||
As the training of the population of neural networks progresses, this process of exploiting and exploring
|
||||
is performed periodically, ensuring that all the workers in the population have a good base level of performance
|
||||
and also that new hyperparameters are consistently explored.
|
||||
|
||||
This means that PBT can quickly exploit good hyperparameters, can dedicate more training time to
|
||||
promising models and, crucially, can adapt the hyperparameter values throughout training,
|
||||
leading to automatic learning of the best configurations.
|
||||
|
||||
First, we define a Trainable that wraps a ConvNet model.
|
||||
|
||||
.. literalinclude:: /../../python/ray/tune/examples/pbt_convnet_example.py
|
||||
:language: python
|
||||
:start-after: __trainable_begin__
|
||||
:end-before: __trainable_end__
|
||||
|
||||
The example reuses some of the functions in ray/tune/examples/mnist_pytorch.py, and is also a good
|
||||
demo for how to decouple the tuning logic and original training code.
|
||||
|
||||
Here, we also override ``reset_config``. This method is optional but can be implemented to speed
|
||||
up algorithms such as PBT, and to allow performance optimizations such as running experiments
|
||||
with ``reuse_actors=True``.
|
||||
|
||||
Then, we define a PBT scheduler:
|
||||
|
||||
.. literalinclude:: /../../python/ray/tune/examples/pbt_convnet_example.py
|
||||
:language: python
|
||||
:start-after: __pbt_begin__
|
||||
:end-before: __pbt_end__
|
||||
|
||||
Some of the most important parameters are:
|
||||
|
||||
- ``hyperparam_mutations`` and ``custom_explore_fn`` are used to mutate the hyperparameters.
|
||||
``hyperparam_mutations`` is a dictionary where each key/value pair specifies the candidates
|
||||
or function for a hyperparameter. custom_explore_fn is applied after built-in perturbations
|
||||
from hyperparam_mutations are applied, and should return config updated as needed.
|
||||
|
||||
- ``resample_probability``: The probability of resampling from the original distribution
|
||||
when applying hyperparam_mutations. If not resampled, the value will be perturbed by a
|
||||
factor of 1.2 or 0.8 if continuous, or changed to an adjacent value if discrete. Note that
|
||||
``resample_probability`` by default is 0.25, thus hyperparameter with a distribution
|
||||
may go out of the specific range.
|
||||
|
||||
Now we can kick off the tuning process by invoking tune.run:
|
||||
|
||||
.. literalinclude:: /../../python/ray/tune/examples/pbt_convnet_example.py
|
||||
:language: python
|
||||
:start-after: __tune_begin__
|
||||
:end-before: __tune_end__
|
||||
|
||||
During the training, we can constantly check the status of the models from console log:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
== Status ==
|
||||
Memory usage on this node: 10.4/16.0 GiB
|
||||
PopulationBasedTraining: 4 checkpoints, 1 perturbs
|
||||
Resources requested: 4/12 CPUs, 0/0 GPUs, 0.0/3.42 GiB heap, 0.0/1.17 GiB objects
|
||||
Number of trials: 4 ({'RUNNING': 4})
|
||||
Result logdir: /Users/yuhao.yang/ray_results/pbt_test
|
||||
+--------------------------+----------+---------------------+----------+------------+--------+------------------+----------+
|
||||
| Trial name | status | loc | lr | momentum | iter | total time (s) | acc |
|
||||
|--------------------------+----------+---------------------+----------+------------+--------+------------------+----------|
|
||||
| PytorchTrainble_3b42d914 | RUNNING | 30.57.180.224:49840 | 0.122032 | 0.302176 | 18 | 3.8689 | 0.8875 |
|
||||
| PytorchTrainble_3b45091e | RUNNING | 30.57.180.224:49835 | 0.505325 | 0.628559 | 18 | 3.90404 | 0.134375 |
|
||||
| PytorchTrainble_3b454c46 | RUNNING | 30.57.180.224:49843 | 0.490228 | 0.969013 | 17 | 3.72111 | 0.0875 |
|
||||
| PytorchTrainble_3b458a9c | RUNNING | 30.57.180.224:49833 | 0.961861 | 0.169701 | 13 | 2.72594 | 0.1125 |
|
||||
+--------------------------+----------+---------------------+----------+------------+--------+------------------+----------+
|
||||
|
||||
In {LOG_DIR}/{MY_EXPERIMENT_NAME}/, all mutations are logged in pbt_global.txt
|
||||
and individual policy perturbations are recorded in pbt_policy_{i}.txt. Tune logs:
|
||||
[target trial tag, clone trial tag, target trial iteration, clone trial iteration,
|
||||
old config, new config] on each perturbation step.
|
||||
|
||||
Checking the accuracy:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
# Plot by wall-clock time
|
||||
dfs = analysis.fetch_trial_dataframes()
|
||||
# This plots everything on the same plot
|
||||
ax = None
|
||||
for d in dfs.values():
|
||||
ax = d.plot("training_iteration", "mean_accuracy", ax=ax, legend=False)
|
||||
|
||||
plt.xlabel("iterations")
|
||||
plt.ylabel("Test Accuracy")
|
||||
|
||||
print('best config:', analysis.get_best_config("mean_accuracy"))
|
||||
|
||||
.. image:: /images/tune_advanced_plot1.png
|
||||
|
||||
DCGAN with Trainable and PBT
|
||||
----------------------------
|
||||
|
||||
The Generative Adversarial Networks (GAN) (Goodfellow et al., 2014) framework learns generative
|
||||
models via a training paradigm consisting of two competing modules – a generator and a
|
||||
discriminator. GAN training can be remarkably brittle and unstable in the face of suboptimal
|
||||
hyperparameter selection with generators often collapsing to a single mode or diverging entirely.
|
||||
|
||||
As presented in `Population Based Training (PBT) <https://deepmind.com/blog/population-based-training-neural-networks>`__,
|
||||
PBT can help with the DCGAN training. We will now walk through how to do this in Tune.
|
||||
Complete code example at `github <https://github.com/ray-project/ray/tree/master/python/ray/tune/examples/pbt_dcgan_mnist>`__
|
||||
|
||||
We define the Generator and Discriminator with standard Pytorch API:
|
||||
|
||||
.. literalinclude:: /../../python/ray/tune/examples/pbt_dcgan_mnist/pbt_dcgan_mnist.py
|
||||
:language: python
|
||||
:start-after: __GANmodel_begin__
|
||||
:end-before: __GANmodel_end__
|
||||
|
||||
To train the model with PBT, we need to define a metric for the scheduler to evaluate
|
||||
the model candidates. For a GAN network, inception score is arguably the most
|
||||
commonly used metric. We trained a mnist classification model (LeNet) and use
|
||||
it to inference the generated images and evaluate the image quality.
|
||||
|
||||
.. literalinclude:: /../../python/ray/tune/examples/pbt_dcgan_mnist/pbt_dcgan_mnist.py
|
||||
:language: python
|
||||
:start-after: __INCEPTION_SCORE_begin__
|
||||
:end-before: __INCEPTION_SCORE_end__
|
||||
|
||||
The ``Trainable`` class includes a Generator and a Discriminator, each with an
|
||||
independent learning rate and optimizer.
|
||||
|
||||
.. literalinclude:: /../../python/ray/tune/examples/pbt_dcgan_mnist/pbt_dcgan_mnist.py
|
||||
:language: python
|
||||
:start-after: __Trainable_begin__
|
||||
:end-before: __Trainable_end__
|
||||
|
||||
We specify inception score as the metric and start the tuning:
|
||||
|
||||
.. literalinclude:: /../../python/ray/tune/examples/pbt_dcgan_mnist/pbt_dcgan_mnist.py
|
||||
:language: python
|
||||
:start-after: __tune_begin__
|
||||
:end-before: __tune_end__
|
||||
|
||||
The trained Generator models can be loaded from log directory, and generate images
|
||||
from noise signals.
|
||||
|
||||
Visualization
|
||||
~~~~~~~~~~~~~
|
||||
|
||||
Below, we visualize the increasing inception score from the training logs.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
lossG = [df['is_score'].tolist() for df in list(analysis.trial_dataframes.values())]
|
||||
|
||||
plt.figure(figsize=(10,5))
|
||||
plt.title("Inception Score During Training")
|
||||
for i, lossg in enumerate(lossG):
|
||||
plt.plot(lossg,label=i)
|
||||
|
||||
plt.xlabel("iterations")
|
||||
plt.ylabel("is_score")
|
||||
plt.legend()
|
||||
plt.show()
|
||||
|
||||
.. image:: /images/tune_advanced_dcgan_inscore.png
|
||||
|
||||
And the Generator loss:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
lossG = [df['lossg'].tolist() for df in list(analysis.trial_dataframes.values())]
|
||||
|
||||
plt.figure(figsize=(10,5))
|
||||
plt.title("Generator Loss During Training")
|
||||
for i, lossg in enumerate(lossG):
|
||||
plt.plot(lossg,label=i)
|
||||
|
||||
plt.xlabel("iterations")
|
||||
plt.ylabel("LossG")
|
||||
plt.legend()
|
||||
plt.show()
|
||||
|
||||
.. image:: /images/tune_advanced_dcgan_Gloss.png
|
||||
|
||||
Training of the MNist Generator takes a couple of minutes. The example can be easily
|
||||
altered to generate images for other datasets, e.g. cifar10 or LSUN.
|
||||
@@ -0,0 +1,368 @@
|
||||
.. _tune-distributed:
|
||||
|
||||
Tune Distributed Experiments
|
||||
============================
|
||||
|
||||
Tune is commonly used for large-scale distributed hyperparameter optimization. This page will overview:
|
||||
|
||||
1. How to setup and launch a distributed experiment,
|
||||
2. :ref:`Commonly used commands <tune-distributed-common>`, including fast file mounting, one-line cluster launching, and result uploading to cloud storage.
|
||||
|
||||
**Quick Summary**: To run a distributed experiment with Tune, you need to:
|
||||
|
||||
1. Make sure your script has ``ray.init(address=...)`` to connect to the existing Ray cluster.
|
||||
2. If a ray cluster does not exist, start a Ray cluster.
|
||||
3. Run the script on the head node (or use ``ray submit``).
|
||||
|
||||
.. contents::
|
||||
:local:
|
||||
:backlinks: none
|
||||
|
||||
Running a distributed experiment
|
||||
--------------------------------
|
||||
|
||||
Running a distributed (multi-node) experiment requires Ray to be started already. You can do this on local machines or on the cloud.
|
||||
|
||||
Across your machines, Tune will automatically detect the number of GPUs and CPUs without you needing to manage ``CUDA_VISIBLE_DEVICES``.
|
||||
|
||||
To execute a distributed experiment, call ``ray.init(address=XXX)`` before ``tune.run``, where ``XXX`` is the Ray redis address, which defaults to ``localhost:6379``. The Tune python script should be executed only on the head node of the Ray cluster.
|
||||
|
||||
One common approach to modifying an existing Tune experiment to go distributed is to set an ``argparse`` variable so that toggling between distributed and single-node is seamless.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
import ray
|
||||
import argparse
|
||||
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--address")
|
||||
args = parser.parse_args()
|
||||
ray.init(address=args.address)
|
||||
|
||||
tune.run(...)
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
# On the head node, connect to an existing ray cluster
|
||||
$ python tune_script.py --ray-address=localhost:XXXX
|
||||
|
||||
If you used a cluster configuration (starting a cluster with ``ray up`` or ``ray submit --start``), use:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
ray submit tune-default.yaml tune_script.py -- --ray-address=localhost:6379
|
||||
|
||||
.. tip::
|
||||
|
||||
1. In the examples, the Ray redis address commonly used is ``localhost:6379``.
|
||||
2. If the Ray cluster is already started, you should not need to run anything on the worker nodes.
|
||||
|
||||
.. _tune-distributed-local:
|
||||
|
||||
Local Cluster Setup
|
||||
-------------------
|
||||
|
||||
If you already have a list of nodes, you can follow the local private cluster setup `instructions here <autoscaling.html#quick-start-private-cluster>`_. Below is an example cluster configuration as ``tune-default.yaml``:
|
||||
|
||||
.. literalinclude:: /../../python/ray/tune/examples/tune-local-default.yaml
|
||||
:language: yaml
|
||||
|
||||
``ray up`` starts Ray on the cluster of nodes.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
ray up tune-default.yaml
|
||||
|
||||
``ray submit`` uploads ``tune_script.py`` to the cluster and runs ``python tune_script.py [args]``.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
ray submit tune-default.yaml tune_script.py -- --ray-address=localhost:6379
|
||||
|
||||
Manual Local Cluster Setup
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
If you run into issues using the local cluster setup (or want to add nodes manually), you can use the manual cluster setup. `Full documentation here <using-ray-on-a-cluster.html>`__. At a glance,
|
||||
|
||||
**On the head node**:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
# If the ``--redis-port`` argument is omitted, Ray will choose a port at random.
|
||||
$ ray start --head --redis-port=6379
|
||||
|
||||
The command will print out the address of the Redis server that was started (and some other address information).
|
||||
|
||||
**Then on all of the other nodes**, run the following. Make sure to replace ``<address>`` with the value printed by the command on the head node (it should look something like ``123.45.67.89:6379``).
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
$ ray start --address=<address>
|
||||
|
||||
Then, you can run your Tune Python script on the head node like:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
# On the head node, execute using existing ray cluster
|
||||
$ python tune_script.py --ray-address=<address>
|
||||
|
||||
.. tune-distributed-cloud:
|
||||
|
||||
Launching a cloud cluster
|
||||
-------------------------
|
||||
|
||||
.. tip::
|
||||
|
||||
If you have already have a list of nodes, go to :ref:`tune-distributed-local`.
|
||||
|
||||
Ray currently supports AWS and GCP. Follow the instructions below to launch nodes on AWS (using the Deep Learning AMI). See the :ref:`cluster setup documentation <ref-automatic-cluster>`. Save the below cluster configuration (``tune-default.yaml``):
|
||||
|
||||
.. literalinclude:: /../../python/ray/tune/examples/tune-default.yaml
|
||||
:language: yaml
|
||||
:name: tune-default.yaml
|
||||
|
||||
``ray up`` starts Ray on the cluster of nodes.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
ray up tune-default.yaml
|
||||
|
||||
``ray submit --start`` starts a cluster as specified by the given cluster configuration YAML file, uploads ``tune_script.py`` to the cluster, and runs ``python tune_script.py [args]``.
|
||||
|
||||
.. note:: You may see a message like: ``bash: cannot set terminal process group (-1): Inappropriate ioctl for device bash: no job control in this shell`` This is a harmless error. If the cluster launcher fails, it is most likely due to some other factor.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
ray submit tune-default.yaml tune_script.py --start -- --ray-address=localhost:6379
|
||||
|
||||
.. image:: /images/tune-upload.png
|
||||
:scale: 50%
|
||||
:align: center
|
||||
|
||||
Analyze your results on TensorBoard by starting TensorBoard on the remote head machine.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
# Go to http://localhost:6006 to access TensorBoard.
|
||||
ray exec tune-default.yaml 'tensorboard --logdir=~/ray_results/ --port 6006' --port-forward 6006
|
||||
|
||||
|
||||
Note that you can customize the directory of results by running: ``tune.run(local_dir=..)``. You can then point TensorBoard to that directory to visualize results. You can also use `awless <https://github.com/wallix/awless>`_ for easy cluster management on AWS.
|
||||
|
||||
Syncing
|
||||
-------
|
||||
|
||||
Tune automatically syncs the trial folder on remote nodes back to the head node. This requires the ray cluster to be started with the :ref:`autoscaler <ref-automatic-cluster>`.
|
||||
By default, local syncing requires rsync to be installed. You can customize the sync command with the ``sync_to_driver`` argument in ``tune.run`` by providing either a function or a string.
|
||||
|
||||
If a string is provided, then it must include replacement fields ``{source}`` and ``{target}``, like ``rsync -savz -e "ssh -i ssh_key.pem" {source} {target}``. Alternatively, a function can be provided with the following signature:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
def custom_sync_func(source, target):
|
||||
sync_cmd = "rsync {source} {target}".format(
|
||||
source=source,
|
||||
target=target)
|
||||
sync_process = subprocess.Popen(sync_cmd, shell=True)
|
||||
sync_process.wait()
|
||||
|
||||
tune.run(
|
||||
MyTrainableClass,
|
||||
name="experiment_name",
|
||||
sync_to_driver=custom_sync_func,
|
||||
)
|
||||
|
||||
When syncing results back to the driver, the source would be a path similar to ``ubuntu@192.0.0.1:/home/ubuntu/ray_results/trial1``, and the target would be a local path.
|
||||
This custom sync command is used to restart trials under failure. The ``sync_to_driver`` is invoked to push a checkpoint to new node for a paused/pre-empted trial to resume.
|
||||
|
||||
|
||||
.. _tune-distributed-spot:
|
||||
|
||||
Pre-emptible Instances (Cloud)
|
||||
------------------------------
|
||||
|
||||
Running on spot instances (or pre-emptible instances) can reduce the cost of your experiment. You can enable spot instances in AWS via the following configuration modification:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
# Provider-specific config for worker nodes, e.g. instance type.
|
||||
worker_nodes:
|
||||
InstanceType: m5.large
|
||||
ImageId: ami-0b294f219d14e6a82 # Deep Learning AMI (Ubuntu) Version 21.0
|
||||
|
||||
# Run workers on spot by default. Comment this out to use on-demand.
|
||||
InstanceMarketOptions:
|
||||
MarketType: spot
|
||||
SpotOptions:
|
||||
MaxPrice: 1.0 # Max Hourly Price
|
||||
|
||||
In GCP, you can use the following configuration modification:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
worker_nodes:
|
||||
machineType: n1-standard-2
|
||||
disks:
|
||||
- boot: true
|
||||
autoDelete: true
|
||||
type: PERSISTENT
|
||||
initializeParams:
|
||||
diskSizeGb: 50
|
||||
# See https://cloud.google.com/compute/docs/images for more images
|
||||
sourceImage: projects/deeplearning-platform-release/global/images/family/tf-1-13-cpu
|
||||
|
||||
# Run workers on preemtible instances.
|
||||
scheduling:
|
||||
- preemptible: true
|
||||
|
||||
Spot instances may be removed suddenly while trials are still running. Often times this may be difficult to deal with when using other distributed hyperparameter optimization frameworks. Tune allows users to mitigate the effects of this by preserving the progress of your model training through checkpointing.
|
||||
|
||||
The easiest way to do this is to subclass the pre-defined ``Trainable`` class and implement ``_save``, and ``_restore`` abstract methods, as seen in the example below:
|
||||
|
||||
.. literalinclude:: /../../python/ray/tune/examples/mnist_pytorch_trainable.py
|
||||
:language: python
|
||||
:start-after: __trainable_example_begin__
|
||||
:end-before: __trainable_example_end__
|
||||
|
||||
This can then be used similarly to the Function API as before:
|
||||
|
||||
.. literalinclude:: /../../python/ray/tune/tests/tutorial.py
|
||||
:language: python
|
||||
:start-after: __trainable_run_begin__
|
||||
:end-before: __trainable_run_end__
|
||||
|
||||
|
||||
Example for using spot instances (AWS)
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Here is an example for running Tune on spot instances. This assumes your AWS credentials have already been setup (``aws configure``):
|
||||
|
||||
1. Download a full example Tune experiment script here. This includes a Trainable with checkpointing: :download:`mnist_pytorch_trainable.py </../../python/ray/tune/examples/mnist_pytorch_trainable.py>`. To run this example, you will need to install the following:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
$ pip install ray torch torchvision filelock
|
||||
|
||||
2. Download an example cluster yaml here: :download:`tune-default.yaml </../../python/ray/tune/examples/tune-default.yaml>`
|
||||
3. Run ``ray submit`` as below to run Tune across them. Append ``[--start]`` if the cluster is not up yet. Append ``[--stop]`` to automatically shutdown your nodes after running.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
ray submit tune-default.yaml mnist_pytorch_trainable.py --start -- --ray-address=localhost:6379
|
||||
|
||||
|
||||
4. Optionally for testing on AWS or GCP, you can use the following to kill a random worker node after all the worker nodes are up
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
$ ray kill-random-node tune-default.yaml --hard
|
||||
|
||||
To summarize, here are the commands to run:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
wget https://raw.githubusercontent.com/ray-project/ray/master/python/ray/tune/examples/mnist_pytorch_trainable.py
|
||||
wget https://raw.githubusercontent.com/ray-project/ray/master/python/ray/tune/tune-default.yaml
|
||||
ray submit tune-default.yaml mnist_pytorch_trainable.py --start -- --ray-address=localhost:6379
|
||||
|
||||
# wait a while until after all nodes have started
|
||||
ray kill-random-node tune-default.yaml --hard
|
||||
|
||||
You should see Tune eventually continue the trials on a different worker node. See the :ref:`Fault Tolerance <tune-fault-tol>` section for more details.
|
||||
|
||||
You can also specify ``tune.run(upload_dir=...)`` to sync results with a cloud storage like S3, allowing you to persist results in case you want to start and stop your cluster automatically.
|
||||
|
||||
.. _tune-fault-tol:
|
||||
|
||||
Fault Tolerance
|
||||
---------------
|
||||
|
||||
Tune will automatically restart trials in case of trial failures/error (if ``max_failures != 0``), both in the single node and distributed setting.
|
||||
|
||||
Tune will restore trials from the latest checkpoint, where available. In the distributed setting, if using the autoscaler with ``rsync`` enabled, Tune will automatically sync the trial folder with the driver. For example, if a node is lost while a trial (specifically, the corresponding Trainable actor of the trial) is still executing on that node and a checkpoint of the trial exists, Tune will wait until available resources are available to begin executing the trial again.
|
||||
|
||||
If the trial/actor is placed on a different node, Tune will automatically push the previous checkpoint file to that node and restore the remote trial actor state, allowing the trial to resume from the latest checkpoint even after failure.
|
||||
|
||||
Recovering From Failures
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Tune automatically persists the progress of your entire experiment (a ``tune.run`` session), so if an experiment crashes or is otherwise cancelled, it can be resumed by passing one of True, False, "LOCAL", "REMOTE", or "PROMPT" to ``tune.run(resume=...)``. Note that this only works if trial checkpoints are detected, whether it be by manual or periodic checkpointing.
|
||||
|
||||
**Settings:**
|
||||
|
||||
- The default setting of ``resume=False`` creates a new experiment.
|
||||
- ``resume="LOCAL"`` and ``resume=True`` restore the experiment from ``local_dir/[experiment_name]``.
|
||||
- ``resume="REMOTE"`` syncs the upload dir down to the local dir and then restores the experiment from ``local_dir/experiment_name``.
|
||||
- ``resume="PROMPT"`` will cause Tune to prompt you for whether you want to resume. You can always force a new experiment to be created by changing the experiment name.
|
||||
|
||||
Note that trials will be restored to their last checkpoint. If trial checkpointing is not enabled, unfinished trials will be restarted from scratch.
|
||||
|
||||
E.g.:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
tune.run(
|
||||
my_trainable,
|
||||
checkpoint_freq=10,
|
||||
local_dir="~/path/to/results",
|
||||
resume=True
|
||||
)
|
||||
|
||||
Upon a second run, this will restore the entire experiment state from ``~/path/to/results/my_experiment_name``. Importantly, any changes to the experiment specification upon resume will be ignored. For example, if the previous experiment has reached its termination, then resuming it with a new stop criterion will not run. The new experiment will terminate immediately after initialization. If you want to change the configuration, such as training more iterations, you can do so restore the checkpoint by setting ``restore=<path-to-checkpoint>`` - note that this only works for a single trial.
|
||||
|
||||
.. warning::
|
||||
|
||||
This feature is still experimental, so any provided Trial Scheduler or Search Algorithm will not be checkpointed and able to resume. Only ``FIFOScheduler`` and ``BasicVariantGenerator`` will be supported.
|
||||
|
||||
.. _tune-distributed-common:
|
||||
|
||||
Common Commands
|
||||
---------------
|
||||
|
||||
Below are some commonly used commands for submitting experiments. Please see the :ref:`Autoscaler page <ref-automatic-cluster>` to see find more comprehensive documentation of commands.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
# Upload `tune_experiment.py` from your local machine onto the cluster. Then,
|
||||
# run `python tune_experiment.py --address=localhost:6379` on the remote machine.
|
||||
$ ray submit CLUSTER.YAML tune_experiment.py -- --address=localhost:6379
|
||||
|
||||
# Start a cluster and run an experiment in a detached tmux session,
|
||||
# and shut down the cluster as soon as the experiment completes.
|
||||
# In `tune_experiment.py`, set `tune.run(upload_dir="s3://...")` to persist results
|
||||
$ ray submit CLUSTER.YAML --tmux --start --stop tune_experiment.py -- --address=localhost:6379
|
||||
|
||||
# To start or update your cluster:
|
||||
$ ray up CLUSTER.YAML [-y]
|
||||
|
||||
# Shut-down all instances of your cluster:
|
||||
$ ray down CLUSTER.YAML [-y]
|
||||
|
||||
# Run Tensorboard and forward the port to your own machine.
|
||||
$ ray exec CLUSTER.YAML 'tensorboard --logdir ~/ray_results/ --port 6006' --port-forward 6006
|
||||
|
||||
# Run Jupyter Lab and forward the port to your own machine.
|
||||
$ ray exec CLUSTER.YAML 'jupyter lab --port 6006' --port-forward 6006
|
||||
|
||||
# Get a summary of all the experiments and trials that have executed so far.
|
||||
$ ray exec CLUSTER.YAML 'tune ls ~/ray_results'
|
||||
|
||||
# Upload and sync file_mounts up to the cluster with this command.
|
||||
$ ray rsync-up CLUSTER.YAML
|
||||
|
||||
# Download the results directory from your cluster head node to your local machine on ``~/cluster_results``.
|
||||
$ ray rsync-down CLUSTER.YAML '~/ray_results' ~/cluster_results
|
||||
|
||||
# Launching multiple clusters using the same configuration.
|
||||
$ ray up CLUSTER.YAML -n="cluster1"
|
||||
$ ray up CLUSTER.YAML -n="cluster2"
|
||||
$ ray up CLUSTER.YAML -n="cluster3"
|
||||
|
||||
Troubleshooting
|
||||
---------------
|
||||
|
||||
Sometimes, your program may freeze. Run this to restart the Ray cluster without running any of the installation commands.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
$ ray up CLUSTER.YAML --restart-only
|
||||
@@ -0,0 +1,129 @@
|
||||
.. _tune-tutorial:
|
||||
|
||||
A Basic Tune Tutorial
|
||||
=====================
|
||||
|
||||
.. image:: /images/tune-api.svg
|
||||
|
||||
This tutorial will walk you through the following process to setup a Tune experiment. Specifically, we'll leverage ASHA and Bayesian Optimization (via HyperOpt) via the following steps:
|
||||
|
||||
1. Integrating Tune into your workflow
|
||||
2. Specifying a TrialScheduler
|
||||
3. Adding a SearchAlgorithm
|
||||
4. Getting the best model and analyzing results
|
||||
|
||||
.. note::
|
||||
|
||||
To run this example, you will need to install the following:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
$ pip install ray torch torchvision
|
||||
|
||||
We first run some imports:
|
||||
|
||||
.. literalinclude:: /../../python/ray/tune/tests/tutorial.py
|
||||
:language: python
|
||||
:start-after: __tutorial_imports_begin__
|
||||
:end-before: __tutorial_imports_end__
|
||||
|
||||
|
||||
Below, we have some boiler plate code for a PyTorch training function.
|
||||
|
||||
.. literalinclude:: /../../python/ray/tune/tests/tutorial.py
|
||||
:language: python
|
||||
:start-after: __train_func_begin__
|
||||
:end-before: __train_func_end__
|
||||
|
||||
Notice that there's a couple helper functions in the above training script. You can take a look at these functions in the imported module `examples/mnist_pytorch <https://github.com/ray-project/ray/blob/master/python/ray/tune/examples/mnist_pytorch.py>`__; there's no black magic happening. For example, ``train`` is simply a for loop over the data loader.
|
||||
|
||||
.. code:: python
|
||||
|
||||
EPOCH_SIZE = 20
|
||||
|
||||
def train(model, optimizer, train_loader):
|
||||
model.train()
|
||||
for batch_idx, (data, target) in enumerate(train_loader):
|
||||
if batch_idx * len(data) > EPOCH_SIZE:
|
||||
return
|
||||
optimizer.zero_grad()
|
||||
output = model(data)
|
||||
loss = F.nll_loss(output, target)
|
||||
loss.backward()
|
||||
optimizer.step()
|
||||
|
||||
Let's run 1 trial, randomly sampling from a uniform distribution for learning rate and momentum.
|
||||
|
||||
.. literalinclude:: /../../python/ray/tune/tests/tutorial.py
|
||||
:language: python
|
||||
:start-after: __eval_func_begin__
|
||||
:end-before: __eval_func_end__
|
||||
|
||||
We can then plot the performance of this trial.
|
||||
|
||||
.. literalinclude:: /../../python/ray/tune/tests/tutorial.py
|
||||
:language: python
|
||||
:start-after: __plot_begin__
|
||||
:end-before: __plot_end__
|
||||
|
||||
.. important:: Tune will automatically run parallel trials across all available cores/GPUs on your machine or cluster. To limit the number of cores that Tune uses, you can call ``ray.init(num_cpus=<int>, num_gpus=<int>)`` before ``tune.run``.
|
||||
|
||||
|
||||
Early Stopping with ASHA
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Let's integrate a Trial Scheduler to our search - ASHA, a scalable algorithm for principled early stopping.
|
||||
|
||||
How does it work? On a high level, it terminates trials that are less promising and
|
||||
allocates more time and resources to more promising trials. See `this blog post <https://blog.ml.cmu.edu/2018/12/12/massively-parallel-hyperparameter-optimization/>`__ for more details.
|
||||
|
||||
We can afford to **increase the search space by 5x**, by adjusting the parameter ``num_samples``. See :ref:`tune-schedulers` for more details of available schedulers and library integrations.
|
||||
|
||||
.. literalinclude:: /../../python/ray/tune/tests/tutorial.py
|
||||
:language: python
|
||||
:start-after: __run_scheduler_begin__
|
||||
:end-before: __run_scheduler_end__
|
||||
|
||||
You can run the below in a Jupyter notebook to visualize trial progress.
|
||||
|
||||
.. literalinclude:: /../../python/ray/tune/tests/tutorial.py
|
||||
:language: python
|
||||
:start-after: __plot_scheduler_begin__
|
||||
:end-before: __plot_scheduler_end__
|
||||
|
||||
.. image:: /images/tune-df-plot.png
|
||||
:scale: 50%
|
||||
:align: center
|
||||
|
||||
You can also use Tensorboard for visualizing results.
|
||||
|
||||
.. code:: bash
|
||||
|
||||
$ tensorboard --logdir {logdir}
|
||||
|
||||
|
||||
Search Algorithms in Tune
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
With Tune you can combine powerful hyperparameter search libraries such as `HyperOpt <https://github.com/hyperopt/hyperopt>`_ and `Ax <https://ax.dev>`_ with state-of-the-art algorithms such as HyperBand without modifying any model training code. Tune allows you to use different search algorithms in combination with different trial schedulers. See :ref:`tune-search-alg` for more details of available algorithms and library integrations.
|
||||
|
||||
.. literalinclude:: /../../python/ray/tune/tests/tutorial.py
|
||||
:language: python
|
||||
:start-after: __run_searchalg_begin__
|
||||
:end-before: __run_searchalg_end__
|
||||
|
||||
|
||||
Evaluate your model
|
||||
~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
You can evaluate best trained model using the Analysis object to retrieve the best model:
|
||||
|
||||
.. literalinclude:: /../../python/ray/tune/tests/tutorial.py
|
||||
:language: python
|
||||
:start-after: __run_analysis_begin__
|
||||
:end-before: __run_analysis_end__
|
||||
|
||||
|
||||
Next Steps
|
||||
----------
|
||||
Take a look at the :ref:`tune-user-guide` for a more comprehensive overview of Tune's features.
|
||||
@@ -0,0 +1,424 @@
|
||||
.. _tune-user-guide:
|
||||
|
||||
Tune User Guide
|
||||
===============
|
||||
|
||||
.. warning:: Before you continue, be sure to have read :ref:`tune-60-seconds`.
|
||||
|
||||
This document provides an overview of the core concepts as well as some of the configurations for running Tune.
|
||||
|
||||
.. contents:: :local:
|
||||
|
||||
Parallelism / GPUs
|
||||
------------------
|
||||
|
||||
.. tip:: To run everything sequentially, use :ref:`Ray Local Mode <tune-debugging>`.
|
||||
|
||||
Parallelism is determined by ``resources_per_trial`` (defaulting to 1 CPU, 0 GPU per trial) and the resources available to Tune (``ray.cluster_resources()``).
|
||||
|
||||
Tune will allocate the specified GPU and CPU from ``resources_per_trial`` to each individual trial. A trial will not be scheduled unless at least that amount of resources is available, preventing the cluster from being overloaded.
|
||||
|
||||
By default, Tune automatically runs N concurrent trials, where N is the number of CPUs (cores) on your machine.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
# If you have 4 CPUs on your machine, this will run 4 concurrent trials at a time.
|
||||
tune.run(trainable, num_samples=10)
|
||||
|
||||
You can override this parallelism with ``resources_per_trial``:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
# If you have 4 CPUs on your machine, this will run 2 concurrent trials at a time.
|
||||
tune.run(trainable, num_samples=10, resources_per_trial={"cpu": 2})
|
||||
|
||||
# If you have 4 CPUs on your machine, this will run 1 trial at a time.
|
||||
tune.run(trainable, num_samples=10, resources_per_trial={"cpu": 4})
|
||||
|
||||
# Fractional values are also supported, (i.e., {"cpu": 0.5}).
|
||||
tune.run(trainable, num_samples=10, resources_per_trial={"cpu": 0.5})
|
||||
|
||||
To leverage GPUs, you must set ``gpu`` in ``resources_per_trial``. This will automatically set ``CUDA_VISIBLE_DEVICES`` for each trial.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
# If you have 8 GPUs, this will run 8 trials at once.
|
||||
tune.run(trainable, num_samples=10, resources_per_trial={"gpu": 1})
|
||||
|
||||
# If you have 4 CPUs on your machine and 1 GPU, this will run 1 trial at a time.
|
||||
tune.run(trainable, num_samples=10, resources_per_trial={"cpu": 2, "gpu": 1})
|
||||
|
||||
You can find an example of this in the `Keras MNIST example <https://github.com/ray-project/ray/blob/master/python/ray/tune/examples/tune_mnist_keras.py>`__.
|
||||
|
||||
.. warning:: If 'gpu' is not set, ``CUDA_VISIBLE_DEVICES`` environment variable will be set as empty, disallowing GPU access.
|
||||
|
||||
To attach to a Ray cluster, simply run ``ray.init`` before ``tune.run``:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
# Connect to an existing distributed Ray cluster
|
||||
ray.init(address=<ray_address>)
|
||||
tune.run(trainable, num_samples=100, resources_per_trial={"cpu": 2, "gpu": 1})
|
||||
|
||||
Search Space (Grid/Random)
|
||||
--------------------------
|
||||
|
||||
.. warning:: If you use a Search Algorithm, you will need to use a different search space API.
|
||||
|
||||
You can specify a grid search or random search via the dict passed into ``tune.run(config=)``.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
parameters = {
|
||||
"qux": tune.sample_from(lambda spec: 2 + 2),
|
||||
"bar": tune.grid_search([True, False]),
|
||||
"foo": tune.grid_search([1, 2, 3]),
|
||||
"baz": "asd", # a constant value
|
||||
}
|
||||
|
||||
tune.run(trainable, config=parameters)
|
||||
|
||||
By default, each random variable and grid search point is sampled once. To take multiple random samples, add ``num_samples: N`` to the experiment config. If `grid_search` is provided as an argument, the grid will be repeated `num_samples` of times.
|
||||
|
||||
.. code-block:: python
|
||||
:emphasize-lines: 13
|
||||
|
||||
# num_samples=10 repeats the 3x3 grid search 10 times, for a total of 90 trials
|
||||
tune.run(
|
||||
my_trainable,
|
||||
name="my_trainable",
|
||||
config={
|
||||
"alpha": tune.uniform(100),
|
||||
"beta": tune.sample_from(lambda spec: spec.config.alpha * np.random.normal()),
|
||||
"nn_layers": [
|
||||
tune.grid_search([16, 64, 256]),
|
||||
tune.grid_search([16, 64, 256]),
|
||||
],
|
||||
},
|
||||
num_samples=10
|
||||
)
|
||||
|
||||
Read about this in the :ref:`Grid/Random Search API <tune-grid-random>` page.
|
||||
|
||||
Reporting Metrics
|
||||
-----------------
|
||||
|
||||
You can log arbitrary values and metrics in both training APIs:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
def trainable(config):
|
||||
num_epochs = 100
|
||||
for i in range(num_epochs):
|
||||
accuracy = model.train()
|
||||
metric_1 = f(model)
|
||||
metric_2 = model.get_loss()
|
||||
tune.track.log(acc=accuracy, metric_foo=random_metric_1, bar=metric_2)
|
||||
|
||||
class Trainable(tune.Trainable):
|
||||
...
|
||||
|
||||
def _train(self): # this is called iteratively
|
||||
accuracy = self.model.train()
|
||||
metric_1 = f(self.model)
|
||||
metric_2 = self.model.get_loss()
|
||||
# don't call track.log here!
|
||||
return dict(acc=accuracy, metric_foo=random_metric_1, bar=metric_2)
|
||||
|
||||
During training, Tune will automatically log the below metrics in addition to the user-provided values. All of these can be used as stopping conditions or passed as a parameter to Trial Schedulers/Search Algorithms.
|
||||
|
||||
.. literalinclude:: ../../../../python/ray/tune/result.py
|
||||
:language: python
|
||||
:start-after: __sphinx_doc_begin__
|
||||
:end-before: __sphinx_doc_end__
|
||||
|
||||
.. _tune-checkpoint:
|
||||
|
||||
Checkpointing
|
||||
-------------
|
||||
|
||||
When running a hyperparameter search, Tune can automatically and periodically save/checkpoint your model. Checkpointing is used for
|
||||
|
||||
* saving a model throughout training
|
||||
* fault-tolerance when using pre-emptible machines.
|
||||
* Pausing trials when using Trial Schedulers such as HyperBand and PBT.
|
||||
|
||||
To enable checkpointing, you must implement a :ref:`Trainable class <trainable-docs>` (the function-based API are not checkpointable, since they never return control back to their caller).
|
||||
|
||||
Checkpointing assumes that the model state will be saved to disk on whichever node the Trainable is running on. You can checkpoint with three different mechanisms: manually, periodically, and at termination.
|
||||
|
||||
**Manual Checkpointing**: A custom Trainable can manually trigger checkpointing by returning ``should_checkpoint: True`` (or ``tune.result.SHOULD_CHECKPOINT: True``) in the result dictionary of `_train`. This can be especially helpful in spot instances:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
def _train(self):
|
||||
# training code
|
||||
result = {"mean_accuracy": accuracy}
|
||||
if detect_instance_preemption():
|
||||
result.update(should_checkpoint=True)
|
||||
return result
|
||||
|
||||
|
||||
**Periodic Checkpointing**: periodic checkpointing can be used to provide fault-tolerance for experiments. This can be enabled by setting ``checkpoint_freq=<int>`` and ``max_failures=<int>`` to checkpoint trials every *N* iterations and recover from up to *M* crashes per trial, e.g.:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
tune.run(
|
||||
my_trainable,
|
||||
checkpoint_freq=10,
|
||||
max_failures=5,
|
||||
)
|
||||
|
||||
**Checkpointing at Termination**: The checkpoint_freq may not coincide with the exact end of an experiment. If you want a checkpoint to be created at the end
|
||||
of a trial, you can additionally set the ``checkpoint_at_end=True``:
|
||||
|
||||
.. code-block:: python
|
||||
:emphasize-lines: 5
|
||||
|
||||
tune.run(
|
||||
my_trainable,
|
||||
checkpoint_freq=10,
|
||||
checkpoint_at_end=True,
|
||||
max_failures=5,
|
||||
)
|
||||
|
||||
The checkpoint will be saved at a path that looks like ``local_dir/exp_name/trial_name/checkpoint_x/``, where the x is the number of iterations so far when the checkpoint is saved. To restore the checkpoint, you can use the ``restore`` argument and specify a checkpoint file. By doing this, you can change whatever experiments' configuration such as the experiment's name, the training iteration or so:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
# Restored previous trial from the given checkpoint
|
||||
tune.run(
|
||||
"PG",
|
||||
name="RestoredExp", # The name can be different.
|
||||
stop={"training_iteration": 10}, # train 5 more iterations than previous
|
||||
restore="~/ray_results/Original/PG_<xxx>/checkpoint_5/checkpoint-5",
|
||||
config={"env": "CartPole-v0"},
|
||||
)
|
||||
|
||||
Handling Large Datasets
|
||||
-----------------------
|
||||
|
||||
You often will want to compute a large object (e.g., training data, model weights) on the driver and use that object within each trial. Tune provides a ``pin_in_object_store`` utility function that can be used to broadcast such large objects. Objects pinned in this way will never be evicted from the Ray object store while the driver process is running, and can be efficiently retrieved from any task via ``get_pinned_object``.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
import ray
|
||||
from ray import tune
|
||||
from ray.tune.utils import pin_in_object_store, get_pinned_object
|
||||
|
||||
import numpy as np
|
||||
|
||||
ray.init()
|
||||
|
||||
# X_id can be referenced in closures
|
||||
X_id = pin_in_object_store(np.random.random(size=100000000))
|
||||
|
||||
def f(config, reporter):
|
||||
X = get_pinned_object(X_id)
|
||||
# use X
|
||||
|
||||
tune.run(f)
|
||||
|
||||
Stopping Trials
|
||||
---------------
|
||||
|
||||
You can control when trials are stopped early by passing the ``stop`` argument to ``tune.run``. This argument takes either a dictionary or a function.
|
||||
|
||||
If a dictionary is passed in, the keys may be any field in the return result of ``tune.track.log`` in the Function API or ``_train()`` (including the results from ``_train`` and auto-filled metrics).
|
||||
|
||||
In the example below, each trial will be stopped either when it completes 10 iterations OR when it reaches a mean accuracy of 0.98. These metrics are assumed to be **increasing**.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
# training_iteration is an auto-filled metric by Tune.
|
||||
tune.run(
|
||||
my_trainable,
|
||||
stop={"training_iteration": 10, "mean_accuracy": 0.98}
|
||||
)
|
||||
|
||||
For more flexibility, you can pass in a function instead. If a function is passed in, it must take ``(trial_id, result)`` as arguments and return a boolean (``True`` if trial should be stopped and ``False`` otherwise).
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
|
||||
def stopper(trial_id, result):
|
||||
return result["mean_accuracy"] / result["training_iteration"] > 5
|
||||
|
||||
tune.run(my_trainable, stop=stopper)
|
||||
|
||||
Finally, you can implement the ``Stopper`` abstract class for stopping entire experiments. For example, the following example stops all trials after the criteria is fulfilled by any individual trial, and prevents new ones from starting:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from ray.tune import Stopper
|
||||
|
||||
class CustomStopper(Stopper):
|
||||
def __init__(self):
|
||||
self.should_stop = False
|
||||
|
||||
def __call__(self, trial_id, result):
|
||||
if not self.should_stop and result['foo'] > 10:
|
||||
self.should_stop = True
|
||||
return self.should_stop
|
||||
|
||||
def stop_all(self):
|
||||
"""Returns whether to stop trials and prevent new ones from starting."""
|
||||
return self.should_stop
|
||||
|
||||
stopper = CustomStopper()
|
||||
tune.run(my_trainable, stop=stopper)
|
||||
|
||||
|
||||
Note that in the above example the currently running trials will not stop immediately but will do so once their current iterations are complete. See the :ref:`tune-stop-ref` documentation.
|
||||
|
||||
Logging/Tensorboard
|
||||
-------------------
|
||||
|
||||
Tune will log the results of each trial to a subfolder under a specified local dir, which defaults to ``~/ray_results``.
|
||||
Tune by default will log results for Tensorboard, CSV, and JSON formats.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
# This logs to 2 different trial folders:
|
||||
# ~/ray_results/trainable_name/trial_name_1 and ~/ray_results/trainable_name/trial_name_2
|
||||
# trainable_name and trial_name are autogenerated.
|
||||
tune.run(trainable, num_samples=2)
|
||||
|
||||
Learn about how to customize logging paths and outputs: :ref:`loggers-docstring`.
|
||||
|
||||
Tune automatically outputs Tensorboard files during ``tune.run``. To visualize learning in tensorboard, install tensorboardX:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
$ pip install tensorboardX
|
||||
|
||||
Then, after you run an experiment, you can visualize your experiment with TensorBoard by specifying the output directory of your results.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
$ tensorboard --logdir=~/ray_results/my_experiment
|
||||
|
||||
If you are running Ray on a remote multi-user cluster where you do not have sudo access, you can run the following commands to make sure tensorboard is able to write to the tmp directory:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
$ export TMPDIR=/tmp/$USER; mkdir -p $TMPDIR; tensorboard --logdir=~/ray_results
|
||||
|
||||
.. image:: ../../ray-tune-tensorboard.png
|
||||
|
||||
If using TF2, Tune also automatically generates TensorBoard HParams output, as shown below:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
tune.run(
|
||||
...,
|
||||
config={
|
||||
"lr": tune.grid_search([1e-5, 1e-4]),
|
||||
"momentum": tune.grid_search([0, 0.9])
|
||||
}
|
||||
)
|
||||
|
||||
.. image:: ../../images/tune-hparams.png
|
||||
|
||||
Console Output
|
||||
--------------
|
||||
|
||||
The following fields will automatically show up on the console output, if provided:
|
||||
|
||||
1. ``episode_reward_mean``
|
||||
2. ``mean_loss``
|
||||
3. ``mean_accuracy``
|
||||
4. ``timesteps_this_iter`` (aggregated into ``timesteps_total``).
|
||||
|
||||
Below is an example of the console output:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
== Status ==
|
||||
Memory usage on this node: 11.4/16.0 GiB
|
||||
Using FIFO scheduling algorithm.
|
||||
Resources requested: 4/12 CPUs, 0/0 GPUs, 0.0/3.17 GiB heap, 0.0/1.07 GiB objects
|
||||
Result logdir: /Users/foo/ray_results/myexp
|
||||
Number of trials: 4 (4 RUNNING)
|
||||
+----------------------+----------+---------------------+-----------+--------+--------+----------------+-------+
|
||||
| Trial name | status | loc | param1 | param2 | acc | total time (s) | iter |
|
||||
|----------------------+----------+---------------------+-----------+--------+--------+----------------+-------|
|
||||
| MyTrainable_a826033a | RUNNING | 10.234.98.164:31115 | 0.303706 | 0.0761 | 0.1289 | 7.54952 | 15 |
|
||||
| MyTrainable_a8263fc6 | RUNNING | 10.234.98.164:31117 | 0.929276 | 0.158 | 0.4865 | 7.0501 | 14 |
|
||||
| MyTrainable_a8267914 | RUNNING | 10.234.98.164:31111 | 0.068426 | 0.0319 | 0.9585 | 7.0477 | 14 |
|
||||
| MyTrainable_a826b7bc | RUNNING | 10.234.98.164:31112 | 0.729127 | 0.0748 | 0.1797 | 7.05715 | 14 |
|
||||
+----------------------+----------+---------------------+-----------+--------+--------+----------------+-------+
|
||||
|
||||
You can use a :ref:`Reporter <tune-reporter-doc>` object to customize the console output.
|
||||
|
||||
|
||||
Uploading Results
|
||||
-----------------
|
||||
|
||||
If an upload directory is provided, Tune will automatically sync results from the ``local_dir`` to the given directory, natively supporting standard S3/gsutil URIs.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
tune.run(
|
||||
MyTrainableClass,
|
||||
local_dir="~/ray_results",
|
||||
upload_dir="s3://my-log-dir"
|
||||
)
|
||||
|
||||
You can customize this to specify arbitrary storages with the ``sync_to_cloud`` argument in ``tune.run``. This argument supports either strings with the same replacement fields OR arbitrary functions.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
tune.run(
|
||||
MyTrainableClass,
|
||||
upload_dir="s3://my-log-dir",
|
||||
sync_to_cloud=custom_sync_str_or_func,
|
||||
)
|
||||
|
||||
If a string is provided, then it must include replacement fields ``{source}`` and ``{target}``, like ``s3 sync {source} {target}``. Alternatively, a function can be provided with the following signature:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
def custom_sync_func(source, target):
|
||||
# do arbitrary things inside
|
||||
sync_cmd = "s3 {source} {target}".format(
|
||||
source=source,
|
||||
target=target)
|
||||
sync_process = subprocess.Popen(sync_cmd, shell=True)
|
||||
sync_process.wait()
|
||||
|
||||
.. _tune-debugging:
|
||||
|
||||
Debugging
|
||||
---------
|
||||
|
||||
By default, Tune will run hyperparameter evaluations on multiple processes. However, if you need to debug your training process, it may be easier to do everything on a single process. You can force all Ray functions to occur on a single process with ``local_mode`` by calling the following before ``tune.run``.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
ray.init(local_mode=True)
|
||||
|
||||
Local mode with multiple configuration evaluations will interleave computation, so it is most naturally used when running a single configuration evaluation.
|
||||
|
||||
Stopping after the first failure
|
||||
--------------------------------
|
||||
|
||||
By default, ``tune.run`` will continue executing until all trials have terminated or errored. To stop the entire Tune run as soon as **any** trial errors:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
tune.run(trainable, fail_fast=True)
|
||||
|
||||
This is useful when you are trying to setup a large hyperparameter experiment.
|
||||
|
||||
|
||||
Further Questions or Issues?
|
||||
----------------------------
|
||||
|
||||
You can post questions or issues or feedback through the following channels:
|
||||
|
||||
1. `StackOverflow`_: For questions about how to use Ray.
|
||||
2. `GitHub Issues`_: For bug reports and feature requests.
|
||||
|
||||
.. _`StackOverflow`: https://stackoverflow.com/questions/tagged/ray
|
||||
.. _`GitHub Issues`: https://github.com/ray-project/ray/issues
|
||||
Reference in New Issue
Block a user