mirror of
https://github.com/wassname/ray.git
synced 2026-08-12 12:20:11 +08:00
[tune] Improve user guides and API docs (#7716)
* create guide gallery for Tune * mods * ok * fix * fix_up_gallery * ok * Apply suggestions from code review Co-Authored-By: Sven Mika <sven@anyscale.io> * Apply suggestions from code review Co-Authored-By: Sven Mika <sven@anyscale.io> Co-authored-by: Sven Mika <sven@anyscale.io>
This commit is contained in:
Richard Liaw
co-authored by
Sven Mika
Sven Mika
parent
22ccc43670
commit
a67edc4051
@@ -4,6 +4,41 @@ Analysis/Logging (tune.analysis / tune.logger)
|
||||
Analyzing Results
|
||||
-----------------
|
||||
|
||||
You can use the ``ExperimentAnalysis`` object for analyzing results. It is returned automatically when calling ``tune.run``.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
analysis = tune.run(
|
||||
trainable,
|
||||
name="example-experiment",
|
||||
num_samples=10,
|
||||
)
|
||||
|
||||
Here are some example operations for obtaining a summary of your experiment:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
# Get a dataframe for the last reported results of all of the trials
|
||||
df = analysis.dataframe()
|
||||
|
||||
# Get a dataframe for the max accuracy seen for each trial
|
||||
df = analysis.dataframe(metric="mean_accuracy", mode="max")
|
||||
|
||||
# Get a dict mapping {trial logdir -> dataframes} for all trials in the experiment.
|
||||
all_dataframes = analysis.trial_dataframes
|
||||
|
||||
# Get a list of trials
|
||||
trials = analysis.trials
|
||||
|
||||
You may want to get a summary of multiple experiments that point to the same ``local_dir``. For this, you can use the ``Analysis`` class.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from ray.tune import Analysis
|
||||
analysis = Analysis("~/ray_results/example-experiment")
|
||||
|
||||
.. _exp-analysis-docstring:
|
||||
|
||||
ExperimentAnalysis
|
||||
~~~~~~~~~~~~~~~~~~
|
||||
|
||||
@@ -11,8 +46,6 @@ ExperimentAnalysis
|
||||
:show-inheritance:
|
||||
:members:
|
||||
|
||||
.. _analysis-docstring:
|
||||
|
||||
Analysis
|
||||
~~~~~~~~
|
||||
|
||||
@@ -24,6 +57,21 @@ Analysis
|
||||
Loggers (tune.logger)
|
||||
---------------------
|
||||
|
||||
Viskit
|
||||
~~~~~~
|
||||
|
||||
Tune automatically integrates with Viskit via the ``CSVLogger`` outputs. To use VisKit (you may have to install some dependencies), run:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
$ git clone https://github.com/rll/rllab.git
|
||||
$ python rllab/rllab/viskit/frontend.py ~/ray_results/my_experiment
|
||||
|
||||
The nonrelevant metrics (like timing stats) can be disabled on the left to show only the relevant ones (like accuracy, loss, etc.).
|
||||
|
||||
.. image:: /ray-tune-viskit.png
|
||||
|
||||
|
||||
.. _logger-interface:
|
||||
|
||||
Logger
|
||||
@@ -54,5 +102,6 @@ CSVLogger
|
||||
MLFLowLogger
|
||||
~~~~~~~~~~~~
|
||||
|
||||
.. autoclass:: ray.tune.logger.MLFLowLogger
|
||||
Tune also provides a default logger for `MLFlow <https://mlflow.org>`_. You can install MLFlow via ``pip install mlflow``. An example can be found `mlflow_example.py <https://github.com/ray-project/ray/blob/master/python/ray/tune/examples/mlflow_example.py>`__. Note that this currently does not include artifact logging support. For this, you can use the native MLFlow APIs inside your Trainable definition.
|
||||
|
||||
.. autoclass:: ray.tune.logger.MLFLowLogger
|
||||
|
||||
@@ -1,10 +1,162 @@
|
||||
.. _tune-grid-random:
|
||||
|
||||
Grid/Random Search
|
||||
==================
|
||||
Overview
|
||||
--------
|
||||
|
||||
Tune has a native interface for specifying a grid search or random search. You can specify the search space via ``tune.run(config=...)``.
|
||||
|
||||
Thereby, you can either use the ``tune.grid_search`` primitive to specify an axis of a grid search...
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
tune.run(
|
||||
trainable,
|
||||
config={"bar": tune.grid_search([True, False])})
|
||||
|
||||
|
||||
... or one of the random sampling primitives to specify distributions (:ref:`tune-sample-docs`):
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
tune.run(
|
||||
trainable,
|
||||
config={
|
||||
"param1": tune.choice([True, False]),
|
||||
"bar": tune.uniform(0, 10),
|
||||
"alpha": tune.sample_from(lambda _: np.random.uniform(100) ** 2),
|
||||
"const": "hello" # It is also ok to specify constant values.
|
||||
})
|
||||
|
||||
|
||||
|
||||
.. caution:: If you use a Search Algorithm, you may not be able to specify lambdas or grid search with this
|
||||
interface, as the search algorithm may require a different search space declaration.
|
||||
|
||||
|
||||
To sample multiple times/run multiple trials, specify ``tune.run(num_samples=N``. If ``grid_search`` is provided as an argument, the *same* grid will be repeated ``N`` times.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
# 13 different configs.
|
||||
tune.run(trainable config={
|
||||
"x": tune.choice([0, 1, 2]),
|
||||
}
|
||||
)
|
||||
|
||||
# 13 different configs.
|
||||
tune.run(trainable, num_samples=13, config={
|
||||
"x": tune.choice([0, 1, 2]),
|
||||
"y": tune.randn([0, 1, 2]),
|
||||
}
|
||||
)
|
||||
|
||||
# 4 different configs.
|
||||
tune.run(trainable, config={"x": tune.grid_search([1, 2, 3, 4])}, num_samples=1)
|
||||
|
||||
# 3 different configs.
|
||||
tune.run(trainable, config={"x": grid_search([1, 2, 3])}, num_samples=1)
|
||||
|
||||
# 6 different configs.
|
||||
tune.run(trainable, config={"x": tune.grid_search([1, 2, 3])}, num_samples=2)
|
||||
|
||||
# 9 different configs.
|
||||
tune.run(trainable, num_samples=1, config={
|
||||
"x": tune.grid_search([1, 2, 3]),
|
||||
"y": tune.grid_search([a, b, c])}
|
||||
)
|
||||
|
||||
# 18 different configs.
|
||||
tune.run(trainable, num_samples=2, config={
|
||||
"x": tune.grid_search([1, 2, 3]),
|
||||
"y": tune.grid_search([a, b, c])}
|
||||
)
|
||||
|
||||
# 45 different configs.
|
||||
tune.run(trainable, num_samples=5, config={
|
||||
"x": tune.grid_search([1, 2, 3]),
|
||||
"y": tune.grid_search([a, b, c])}
|
||||
)
|
||||
|
||||
|
||||
|
||||
Note that grid search and random search primitives are inter-operable. Each can be used independently or in combination with each other.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
# 6 different configs.
|
||||
tune.run(trainable, num_samples=2, config={
|
||||
"x": tune.sample_from(...),
|
||||
"y": tune.grid_search([a, b, c])
|
||||
}
|
||||
)
|
||||
|
||||
In the below example, ``num_samples=10`` repeats the 3x3 grid search 10 times, for a total of 90 trials, each with randomly sampled values of ``alpha`` and ``beta``.
|
||||
|
||||
.. code-block:: python
|
||||
:emphasize-lines: 12
|
||||
|
||||
tune.run(
|
||||
my_trainable,
|
||||
name="my_trainable",
|
||||
# num_samples will repeat the entire config 10 times.
|
||||
num_samples=10
|
||||
config={
|
||||
# ``sample_from`` creates a generator to call the lambda once per trial.
|
||||
"alpha": tune.sample_from(lambda spec: np.random.uniform(100)),
|
||||
# ``sample_from`` also supports "conditional search spaces"
|
||||
"beta": tune.sample_from(lambda spec: spec.config.alpha * np.random.normal()),
|
||||
"nn_layers": [
|
||||
# tune.grid_search will make it so that all values are evaluated.
|
||||
tune.grid_search([16, 64, 256]),
|
||||
tune.grid_search([16, 64, 256]),
|
||||
],
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
Custom/Conditional Search Spaces
|
||||
--------------------------------
|
||||
|
||||
You'll often run into awkward search spaces (i.e., when one hyperparameter depends on another). Use ``tune.sample_from(func)`` to provide a **custom** callable function for generating a search space.
|
||||
|
||||
The parameter ``func`` should take in a ``spec`` object, which has a ``config`` namespace from which you can access other hyperparameters. This is useful for conditional distributions:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
tune.run(
|
||||
...,
|
||||
config={
|
||||
# A random function
|
||||
"alpha": tune.sample_from(lambda _: np.random.uniform(100)),
|
||||
# Use the `spec.config` namespace to access other hyperparameters
|
||||
"beta": tune.sample_from(lambda spec: spec.config.alpha * np.random.normal())
|
||||
}
|
||||
)
|
||||
|
||||
Here's an example showing a grid search over two nested parameters combined with random sampling from two lambda functions, generating 9 different trials. Note that the value of ``beta`` depends on the value of ``alpha``, which is represented by referencing ``spec.config.alpha`` in the lambda function. This lets you specify conditional parameter distributions.
|
||||
|
||||
.. code-block:: python
|
||||
:emphasize-lines: 4-11
|
||||
|
||||
tune.run(
|
||||
my_trainable,
|
||||
name="my_trainable",
|
||||
config={
|
||||
"alpha": tune.sample_from(lambda spec: np.random.uniform(100)),
|
||||
"beta": tune.sample_from(lambda spec: spec.config.alpha * np.random.normal()),
|
||||
"nn_layers": [
|
||||
tune.grid_search([16, 64, 256]),
|
||||
tune.grid_search([16, 64, 256]),
|
||||
],
|
||||
}
|
||||
)
|
||||
|
||||
.. _tune-sample-docs:
|
||||
|
||||
Random Distributions
|
||||
--------------------
|
||||
Random Distributions API
|
||||
------------------------
|
||||
|
||||
tune.randn
|
||||
~~~~~~~~~~
|
||||
@@ -31,14 +183,13 @@ tune.sample_from
|
||||
|
||||
.. autoclass:: ray.tune.sample_from
|
||||
|
||||
Grid Search
|
||||
-----------
|
||||
|
||||
tune.grid_search
|
||||
~~~~~~~~~~~~~~~~
|
||||
Grid Search API
|
||||
---------------
|
||||
|
||||
.. autofunction:: ray.tune.grid_search
|
||||
|
||||
Internals
|
||||
---------
|
||||
|
||||
BasicVariantGenerator
|
||||
~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
@@ -11,11 +11,11 @@ on `Github`_.
|
||||
|
||||
execution.rst
|
||||
trainable.rst
|
||||
reporters.rst
|
||||
analysis.rst
|
||||
grid_random.rst
|
||||
suggestion.rst
|
||||
schedulers.rst
|
||||
internals.rst
|
||||
reporters.rst
|
||||
client.rst
|
||||
cli.rst
|
||||
|
||||
@@ -1,10 +1,149 @@
|
||||
.. _trainable-docs:
|
||||
|
||||
Training (tune.Trainable, tune.track)
|
||||
=====================================
|
||||
|
||||
.. _trainable-docstring:
|
||||
Training can be done with either a **Class API** (``tune.Trainable``) < or **function-based API** (``track.log``).
|
||||
|
||||
You can use the **function-based API** for fast prototyping. On the other hand, the ``tune.Trainable`` interface supports checkpoint/restore functionality and provides more control for advanced algorithms.
|
||||
|
||||
Function-based API
|
||||
------------------
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
def trainable(config):
|
||||
"""
|
||||
Args:
|
||||
config (dict): Parameters provided from the search algorithm
|
||||
or variant generation.
|
||||
"""
|
||||
|
||||
while True:
|
||||
# ...
|
||||
tune.track.log(**kwargs)
|
||||
|
||||
.. tip:: Do not use ``tune.track.log`` within a ``Trainable`` class.
|
||||
|
||||
Tune will run this function on a separate thread in a Ray actor process. Note that this API is not checkpointable, since the thread will never return control back to its caller.
|
||||
|
||||
.. note:: If you have a lambda function that you want to train, you will need to first register the function: ``tune.register_trainable("lambda_id", lambda x: ...)``. You can then use ``lambda_id`` in place of ``my_trainable``.
|
||||
|
||||
Trainable API
|
||||
-------------
|
||||
|
||||
.. caution:: Do not use ``tune.track.log`` within a ``Trainable`` class.
|
||||
|
||||
The Trainable **class API** will require users to subclass ``ray.tune.Trainable``. Here's a naive example of this API:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from ray import tune
|
||||
|
||||
class Guesser(tune.Trainable):
|
||||
"""Randomly picks 10 number from [1, 10000) to find the password."""
|
||||
|
||||
def _setup(self, config):
|
||||
self.config = config
|
||||
self.password = 1024
|
||||
|
||||
def _train(self):
|
||||
"""Execute one step of 'training'."""
|
||||
result_dict = {"diff": abs(self.config['guess'] - self.password)}
|
||||
return result_dict
|
||||
|
||||
def _stop(self):
|
||||
# perform any cleanup necessary.
|
||||
pass
|
||||
|
||||
analysis = tune.run(
|
||||
Guesser,
|
||||
stop={
|
||||
"training_iteration": 1,
|
||||
},
|
||||
num_samples=10,
|
||||
config={
|
||||
"guess": tune.randint(1, 10000)
|
||||
})
|
||||
|
||||
print('best config: ', analysis.get_best_config(metric="diff", mode="min"))
|
||||
|
||||
As a subclass of ``tune.Trainable``, Tune will create a ``Guesser`` object on a separate process (using the Ray Actor API).
|
||||
|
||||
1. ``_setup`` function is invoked once training starts.
|
||||
2. ``_train`` is invoked **multiple times**. Each time, the Guesser object executes one logical iteration of training in the tuning process, which may include one or more iterations of actual training.
|
||||
3. ``_stop`` is invoked when training is finished.
|
||||
|
||||
.. tip:: As a rule of thumb, the execution time of ``_train`` should be large enough to avoid overheads (i.e. more than a few seconds), but short enough to report progress periodically (i.e. at most a few minutes).
|
||||
|
||||
In this example, we only implemented the ``_setup`` and ``_train`` methods for simplification. Next, we'll implement ``_save`` and ``_restore`` for checkpoint and fault tolerance.
|
||||
|
||||
Save and Restore
|
||||
~~~~~~~~~~~~~~~~
|
||||
|
||||
Many Tune features rely on ``_save``, and ``_restore``, including the usage of certain Trial Schedulers, fault tolerance, and checkpointing.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
class MyTrainableClass(Trainable):
|
||||
def _save(self, tmp_checkpoint_dir):
|
||||
checkpoint_path = os.path.join(tmp_checkpoint_dir, "model.pth")
|
||||
torch.save(self.model.state_dict(), checkpoint_path)
|
||||
return tmp_checkpoint_dir
|
||||
|
||||
def _restore(self, tmp_checkpoint_dir):
|
||||
checkpoint_path = os.path.join(tmp_checkpoint_dir, "model.pth")
|
||||
self.model.load_state_dict(torch.load(checkpoint_path))
|
||||
|
||||
Checkpoints will be saved by training iteration to ``local_dir/exp_name/trial_name/checkpoint_<iter>``. You can restore a single trial checkpoint by using ``tune.run(restore=<checkpoint_dir>)``.
|
||||
|
||||
Tune also generates temporary checkpoints for pausing and switching between trials. For this purpose, it is important not to depend on absolute paths in the implementation of ``save``.
|
||||
|
||||
Use ``validate_save_restore`` to catch ``_save``/``_restore`` errors before execution.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from ray.tune.utils import validate_save_restore
|
||||
|
||||
# both of these should return
|
||||
validate_save_restore(MyTrainableClass)
|
||||
validate_save_restore(MyTrainableClass, use_object_store=True)
|
||||
|
||||
Advanced: Reusing Actors
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Your Trainable can often take a long time to start. To avoid this, you can do ``tune.run(reuse_actors=True)`` to reuse the same Trainable Python process and object for multiple hyperparameters.
|
||||
|
||||
This requires you to implement ``Trainable.reset_config``, which provides a new set of hyperparameters. It is up to the user to correctly update the hyperparameters of your trainable.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
class PytorchTrainble(tune.Trainable):
|
||||
"""Train a Pytorch ConvNet."""
|
||||
|
||||
def _setup(self, config):
|
||||
self.train_loader, self.test_loader = get_data_loaders()
|
||||
self.model = ConvNet()
|
||||
self.optimizer = optim.SGD(
|
||||
self.model.parameters(),
|
||||
lr=config.get("lr", 0.01),
|
||||
momentum=config.get("momentum", 0.9))
|
||||
|
||||
def reset_config(self, new_config):
|
||||
for param_group in self.optimizer.param_groups:
|
||||
if "lr" in new_config:
|
||||
param_group["lr"] = new_config["lr"]
|
||||
if "momentum" in new_config:
|
||||
param_group["momentum"] = new_config["momentum"]
|
||||
|
||||
self.model = ConvNet()
|
||||
self.config = new_config
|
||||
return True
|
||||
|
||||
|
||||
tune.Trainable
|
||||
~~~~~~~~~~~~~~
|
||||
--------------
|
||||
|
||||
|
||||
.. autoclass:: ray.tune.Trainable
|
||||
:member-order: groupwise
|
||||
@@ -12,21 +151,28 @@ tune.Trainable
|
||||
:members:
|
||||
|
||||
tune.DurableTrainable
|
||||
~~~~~~~~~~~~~~~~~~~~~
|
||||
---------------------
|
||||
|
||||
.. autoclass:: ray.tune.DurableTrainable
|
||||
|
||||
.. _track-docstring:
|
||||
|
||||
tune.track
|
||||
~~~~~~~~~~
|
||||
----------
|
||||
|
||||
.. automodule:: ray.tune.track
|
||||
:members:
|
||||
:exclude-members: init, shutdown
|
||||
:exclude-members: init,
|
||||
|
||||
KerasCallback
|
||||
-------------
|
||||
|
||||
.. automodule:: ray.tune.integration.keras
|
||||
:members:
|
||||
|
||||
|
||||
StatusReporter
|
||||
~~~~~~~~~~~~~~
|
||||
--------------
|
||||
|
||||
.. autoclass:: ray.tune.function_runner.StatusReporter
|
||||
:members: __call__, logdir
|
||||
|
||||
Reference in New Issue
Block a user