wassname/ray
1
0
Fork 0
mirror of https://github.com/wassname/ray.git synced 2026-08-04 13:14:14 +08:00
Code Issues Packages Projects Releases Wiki Activity

[tune+core] tune lifecycle and starting ray guide (#10813)

Browse Source
This commit is contained in:
Richard Liaw
2020-09-22 23:58:13 +00:00
committed by Barak Michener
parent 3f6038b605
commit cb79510e24
16 changed files with 426 additions and 237 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/source/tune/_tutorials/_faq.rst
+1
View File
@@ -183,6 +183,7 @@ additional outputs:
called
* ``trial_id``: Unique trial ID
How do I set resources?
~~~~~~~~~~~~~~~~~~~~~~~
If you want to allocate specific resources to a trial, you can use the
doc/source/tune/_tutorials/overview.rst
+6
View File
@@ -24,6 +24,11 @@ Take a look at any of the below tutorials to get started with Tune.
:figure: /images/tune.png
:description: :doc:`A walkthrough to setup your first Tune experiment <tune-tutorial>`
.. customgalleryitem::
:tooltip: A deep dive into Tune's workings.
:figure: /images/tune.png
:description: :doc:`How does Tune work? <tune-lifecycle>`
.. customgalleryitem::
:tooltip: A simple guide to Population-based Training
:figure: /images/tune-pbt-small.png
@@ -76,6 +81,7 @@ Take a look at any of the below tutorials to get started with Tune.
tune-tutorial.rst
tune-advanced-tutorial.rst
tune-lifecycle.rst
tune-distributed.rst
tune-sklearn.rst
tune-pytorch-cifar.rst
doc/source/tune/_tutorials/tune-distributed.rst
+51 -71
View File
@@ -3,59 +3,21 @@
Tune Distributed Experiments
============================
Tune is commonly used for large-scale distributed hyperparameter optimization. This page will overview:
1. How to setup and launch a distributed experiment,
2. :ref:`Commonly used commands <tune-distributed-common>`, including fast file mounting, one-line cluster launching, and result uploading to cloud storage.
**Quick Summary**: To run a distributed experiment with Tune, you need to:
1. Make sure your script has ``ray.init(address=...)`` to connect to the existing Ray cluster.
2. If a ray cluster does not exist, start a Ray cluster.
3. Run the script on the head node (or use ``ray submit``).
Tune is commonly used for large-scale distributed hyperparameter optimization. This page will overview how to setup and launch a distributed experiment along with :ref:`commonly used commands <tune-distributed-common>` for Tune when running distributed experiments.
.. contents::
:local:
:backlinks: none
Running a distributed experiment
--------------------------------
Summary
-------
Running a distributed (multi-node) experiment requires Ray to be started already. You can do this on local machines or on the cloud.
To run a distributed experiment with Tune, you need to:
Across your machines, Tune will automatically detect the number of GPUs and CPUs without you needing to manage ``CUDA_VISIBLE_DEVICES``.
1. First, :ref:`start a Ray cluster <cluster-index>` if you have not already.
2. Specify ``ray.init(address=...)`` in your script :ref:`to connect to the existing Ray cluster <using-ray-on-a-cluster>`.
3. Run the script on the head node (or use :ref:`ray submit <ray-submit-doc>`).
To execute a distributed experiment, call ``ray.init(address=XXX)`` before ``tune.run``, where ``XXX`` is the Ray redis address, which defaults to ``localhost:6379``. The Tune python script should be executed only on the head node of the Ray cluster.
One common approach to modifying an existing Tune experiment to go distributed is to set an ``argparse`` variable so that toggling between distributed and single-node is seamless.
.. code-block:: python
import ray
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("--address")
args = parser.parse_args()
ray.init(address=args.address)
tune.run(...)
.. code-block:: bash
# On the head node, connect to an existing ray cluster
$ python tune_script.py --ray-address=localhost:XXXX
If you used a cluster configuration (starting a cluster with ``ray up`` or ``ray submit --start``), use:
.. code-block:: bash
ray submit tune-default.yaml tune_script.py -- --ray-address=localhost:6379
.. tip::
1. In the examples, the Ray redis address commonly used is ``localhost:6379``.
2. If the Ray cluster is already started, you should not need to run anything on the worker nodes.
.. _tune-distributed-local:
@@ -84,28 +46,6 @@ Manual Local Cluster Setup
If you run into issues using the local cluster setup (or want to add nodes manually), you can use :ref:`the manual cluster setup <cluster-index>`. At a glance,
**On the head node**:
.. code-block:: bash
# If the ``--redis-port`` argument is omitted, Ray will choose a port at random.
$ ray start --head --redis-port=6379
The command will print out the address of the Redis server that was started (and some other address information).
**Then on all of the other nodes**, run the following. Make sure to replace ``<address>`` with the value printed by the command on the head node (it should look something like ``123.45.67.89:6379``).
.. code-block:: bash
$ ray start --address=<address>
Then, you can run your Tune Python script on the head node like:
.. code-block:: bash
# On the head node, execute using existing ray cluster
$ python tune_script.py --ray-address=<address>
.. tune-distributed-cloud:
Launching a cloud cluster
@@ -147,6 +87,46 @@ Analyze your results on TensorBoard by starting TensorBoard on the remote head m
Note that you can customize the directory of results by running: ``tune.run(local_dir=..)``. You can then point TensorBoard to that directory to visualize results. You can also use `awless <https://github.com/wallix/awless>`_ for easy cluster management on AWS.
Running a distributed experiment
--------------------------------
Running a distributed (multi-node) experiment requires Ray to be started already. You can do this on local machines or on the cloud.
Across your machines, Tune will automatically detect the number of GPUs and CPUs without you needing to manage ``CUDA_VISIBLE_DEVICES``.
To execute a distributed experiment, call ``ray.init(address=XXX)`` before ``tune.run``, where ``XXX`` is the Ray redis address, which defaults to ``localhost:6379``. The Tune python script should be executed only on the head node of the Ray cluster.
One common approach to modifying an existing Tune experiment to go distributed is to set an ``argparse`` variable so that toggling between distributed and single-node is seamless.
.. code-block:: python
import ray
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("--address")
args = parser.parse_args()
ray.init(address=args.address)
tune.run(...)
.. code-block:: bash
# On the head node, connect to an existing ray cluster
$ python tune_script.py --ray-address=localhost:XXXX
If you used a cluster configuration (starting a cluster with ``ray up`` or ``ray submit --start``), use:
.. code-block:: bash
ray submit tune-default.yaml tune_script.py -- --ray-address=localhost:6379
.. tip::
1. In the examples, the Ray redis address commonly used is ``localhost:6379``.
2. If the Ray cluster is already started, you should not need to run anything on the worker nodes.
Syncing
-------
@@ -281,10 +261,10 @@ Tune automatically persists the progress of your entire experiment (a ``tune.run
**Settings:**
- The default setting of ``resume=False`` creates a new experiment.
- ``resume="LOCAL"`` and ``resume=True`` restore the experiment from ``local_dir/[experiment_name]``.
- ``resume="REMOTE"`` syncs the upload dir down to the local dir and then restores the experiment from ``local_dir/experiment_name``.
- ``resume="PROMPT"`` will cause Tune to prompt you for whether you want to resume. You can always force a new experiment to be created by changing the experiment name.
- The default setting of ``resume=False`` creates a new experiment.
- ``resume="LOCAL"`` and ``resume=True`` restore the experiment from ``local_dir/[experiment_name]``.
- ``resume="REMOTE"`` syncs the upload dir down to the local dir and then restores the experiment from ``local_dir/experiment_name``.
- ``resume="PROMPT"`` will cause Tune to prompt you for whether you want to resume. You can always force a new experiment to be created by changing the experiment name.
Note that trials will be restored to their last checkpoint. If trial checkpointing is not enabled, unfinished trials will be restarted from scratch.
doc/source/tune/_tutorials/tune-lifecycle.rst
+94
View File
@@ -0,0 +1,94 @@
.. _tune-lifecycle:
How does Tune work?
===================
This page provides an overview of Tune's inner workings.
.. tip:: Before you continue, be sure to have read :ref:`the Tune Key Concepts page <tune-60-seconds>`.
Definitions
-----------
**Trainable**
The function passed to tune.run.
**Trial**
Logical representation of a single hyperparameter configuration. Each trial is associated with an instance of a Trainable.
**Driver/worker process**
The driver process is the python process that calls ``tune.run`` (which calls ``ray.init()`` underneath the hood).
The Tune driver process runs on the node where you run your script (which calls ``tune.run``), while Ray Tune trainable "actors" run on any node (either on the same node or on worker nodes (distributed Ray only)).
**Ray Actors**
Tune uses Ray Actors as worker processes to evaluate multiple Trainables in parallel.
:ref:`Ray Actors <actor-guide>` allow you to parallelize an instance of a class in Python. When you instantiate a class that is a Ray actor, Ray will start a instance of that class on a separate process either on the same machine (or another distributed machine, if running a Ray cluster). This actor can then asynchronously execute method calls and maintain its own internal state.
What happens in ``tune.run``?
-----------------------------
When calling the following:
.. code-block:: python
space = {"x": tune.uniform(0, 1)}
tune.run(my_trainable, config=space, num_samples=10)
The provided function/trainable is evaluated multiple times in parallel with different hyperparameters (sampled from ``uniform(0, 1)``).
Every Tune run consists of "driver process" and many "worker processes". As mentioned in the Definitions section, the driver process (Tune Driver) is the python process in which you call ``tune.run``.
The driver spawns parallel worker processes (:ref:`Ray actors <actor-guide>`)
that are responsible for evaluating each trial using its hyperparameter configuration and the provided trainable (see the `trial executor source code <https://github.com/ray-project/ray/blob/master/python/ray/tune/trial_executor.py>`__).
While the Trainable is executing (:ref:`trainable-execution`), the Tune Driver communicates with each actor via actor methods to receive intermediate training results and pause/stop actors (see :ref:`trial-lifecycle`).
When the Trainable terminates (or is stopped), the actor is also terminated.
.. _trainable-execution:
The execution of a trainable
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Tune uses :ref:`Ray actors <actor-guide>` to parallelize the evaluation of multiple hyperparameter configurations. Each actor is a Python process that executes an instance of the user-provided Trainable.
The definition of the user-provided Trainable will be :ref:`serialized via cloudpickle <serialization-guide>`) and sent to each actor process. Each Ray actor will start an instance of the Trainable to be executed.
If the Trainable is a class, it will be executed iteratively by calling ``train/step``. After each invocation, the driver is notified that a "result dict" is ready. The driver will then pull the result via ``ray.get``.
If the trainable is a callable or a function, it will be executed on the Ray actor process on a separate execution thread. Whenever ``tune.report`` is called, the execution thread is paused and waits for the driver to pull a result (see `function_runner.py <https://github.com/ray-project/ray/blob/master/python/ray/tune/function_runner.py>`__. After pulling, the actor’s execution thread will automatically resume.
Resource Management in Tune
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Before running a trial, the Tune Driver will check whether there are available resources on the cluster (see :ref:`resource-requirements`). It will compare the available resources with the resources required by the trial.
If there is space on the cluster, then the Tune Driver will start a Ray actor (worker). This actor will be scheduled and executed on some node where the resources are available.
See :ref:`tune-parallelism` for more information.
.. _trial-lifecycle:
Lifecycle of a trial
--------------------
A trial's life cycle consists of 6 stages:
* **Initialization** (generation): A trial is first generated as a hyperparameter sample, and its parameters are configured according to what was provided in tune.run. Trials are then placed into a queue to be executed (with status PENDING).
* **PENDING**: A pending trial is a trial to be executed on the machine. Every trial is configured with resource values. Whenever the trial’s resource values are available, tune will run the trial (by starting a ray actor holding the config and the training function.
* **RUNNING**: A running trial is assigned a Ray Actor. There can be multiple running trials in parallel. See the :ref:`trainable execution <trainable-execution>` section for more details.
* **ERRORED**: If a running trial throws an exception, Tune will catch that exception and mark the trial as errored. Note that exceptions can be propagated from an actor to the main Tune driver process. If max_retries is set, Tune will set the trial back into "PENDING" and later start it from the last checkpoint.
* **TERMINATED**: A trial is terminated if it is stopped by a Stopper/Scheduler. If using the Function API, the trial is also terminated when the function stops.
* **PAUSED**: A trial can be paused by a Trial scheduler. This means that the trial’s actor will be stopped. A paused trial can later be resumed from the most recent checkpoint.
doc/source/tune/_tutorials/tune-sklearn.py
+5 -5
View File
@@ -145,11 +145,11 @@ print(tune_search.best_params_)
#
# Check out more detailed examples and get started with tune-sklearn!
#
# * `Skorch with tune-sklearn <https://github.com/ray-project/tune-sklearn/blob/master/examples/torch_nn.py>`_
# * `Scikit-Learn Pipelines with tune-sklearn <https://github.com/ray-project/tune-sklearn/blob/master/examples/sklearn_pipeline.py>`_
# * `XGBoost with tune-sklearn <https://github.com/ray-project/tune-sklearn/blob/master/examples/xgbclassifier.py>`_
# * `KerasClassifier with tune-sklearn <https://github.com/ray-project/tune-sklearn/blob/master/examples/keras_example.py>`_
# * `LightGBM with tune-sklearn <https://github.com/ray-project/tune-sklearn/blob/master/examples/lgbm.py>`_
# * `Skorch with tune-sklearn <https://github.com/ray-project/tune-sklearn/blob/master/examples/torch_nn.py>`_
# * `Scikit-Learn Pipelines with tune-sklearn <https://github.com/ray-project/tune-sklearn/blob/master/examples/sklearn_pipeline.py>`_
# * `XGBoost with tune-sklearn <https://github.com/ray-project/tune-sklearn/blob/master/examples/xgbclassifier.py>`_
# * `KerasClassifier with tune-sklearn <https://github.com/ray-project/tune-sklearn/blob/master/examples/keras_example.py>`_
# * `LightGBM with tune-sklearn <https://github.com/ray-project/tune-sklearn/blob/master/examples/lgbm.py>`_
#
#
# Further Reading