mirror of
https://github.com/wassname/ray.git
synced 2026-09-09 11:32:43 +08:00
[rllib] Include config dicts in the sphinx docs (#3064)
This commit is contained in:
Eric Liang
@@ -38,6 +38,13 @@ SpaceInvaders 646 ~300
|
||||
|
||||
Ape-X using 32 workers in RLlib vs vanilla DQN (orange) and A3C (blue) on PongNoFrameskip-v4.
|
||||
|
||||
**Ape-X specific configs** (see also `common configs <rllib-training.html#common-parameters>`__):
|
||||
|
||||
.. literalinclude:: ../../python/ray/rllib/agents/dqn/apex.py
|
||||
:language: python
|
||||
:start-after: __sphinx_doc_begin__
|
||||
:end-before: __sphinx_doc_end__
|
||||
|
||||
Importance Weighted Actor-Learner Architecture (IMPALA)
|
||||
-------------------------------------------------------
|
||||
|
||||
@@ -73,6 +80,13 @@ SpaceInvaders 843 ~300
|
||||
|
||||
IMPALA solves Atari several times faster than A2C / A3C, with similar sample efficiency. Here IMPALA scales from 16 to 128 workers to solve PongNoFrameskip-v4 in ~8 minutes.
|
||||
|
||||
**IMPALA-specific configs** (see also `common configs <rllib-training.html#common-parameters>`__):
|
||||
|
||||
.. literalinclude:: ../../python/ray/rllib/agents/impala/impala.py
|
||||
:language: python
|
||||
:start-after: __sphinx_doc_begin__
|
||||
:end-before: __sphinx_doc_end__
|
||||
|
||||
Gradient-based
|
||||
~~~~~~~~~~~~~~
|
||||
|
||||
@@ -97,6 +111,13 @@ Qbert 3620 ~1000
|
||||
SpaceInvaders 692 ~600
|
||||
============= ======================== ==============================
|
||||
|
||||
**A3C-specific configs** (see also `common configs <rllib-training.html#common-parameters>`__):
|
||||
|
||||
.. literalinclude:: ../../python/ray/rllib/agents/a3c/a3c.py
|
||||
:language: python
|
||||
:start-after: __sphinx_doc_begin__
|
||||
:end-before: __sphinx_doc_end__
|
||||
|
||||
Deep Deterministic Policy Gradients (DDPG)
|
||||
------------------------------------------
|
||||
`[paper] <https://arxiv.org/abs/1509.02971>`__ `[implementation] <https://github.com/ray-project/ray/blob/master/python/ray/rllib/agents/ddpg/ddpg.py>`__
|
||||
@@ -104,6 +125,13 @@ DDPG is implemented similarly to DQN (below). The algorithm can be scaled by inc
|
||||
|
||||
Tuned examples: `Pendulum-v0 <https://github.com/ray-project/ray/blob/master/python/ray/rllib/tuned_examples/pendulum-ddpg.yaml>`__, `MountainCarContinuous-v0 <https://github.com/ray-project/ray/blob/master/python/ray/rllib/tuned_examples/mountaincarcontinuous-ddpg.yaml>`__, `HalfCheetah-v2 <https://github.com/ray-project/ray/blob/master/python/ray/rllib/tuned_examples/halfcheetah-ddpg.yaml>`__
|
||||
|
||||
**DDPG-specific configs** (see also `common configs <rllib-training.html#common-parameters>`__):
|
||||
|
||||
.. literalinclude:: ../../python/ray/rllib/agents/ddpg/ddpg.py
|
||||
:language: python
|
||||
:start-after: __sphinx_doc_begin__
|
||||
:end-before: __sphinx_doc_end__
|
||||
|
||||
Deep Q Networks (DQN, Rainbow)
|
||||
------------------------------
|
||||
`[paper] <https://arxiv.org/abs/1312.5602>`__ `[implementation] <https://github.com/ray-project/ray/blob/master/python/ray/rllib/agents/dqn/dqn.py>`__
|
||||
@@ -125,12 +153,26 @@ Qbert 3921 7968 15780
|
||||
SpaceInvaders 650 1001 1025 ~500
|
||||
============= ======================== ============================= ============================== ===============================
|
||||
|
||||
**DQN-specific configs** (see also `common configs <rllib-training.html#common-parameters>`__):
|
||||
|
||||
.. literalinclude:: ../../python/ray/rllib/agents/dqn/dqn.py
|
||||
:language: python
|
||||
:start-after: __sphinx_doc_begin__
|
||||
:end-before: __sphinx_doc_end__
|
||||
|
||||
Policy Gradients
|
||||
----------------
|
||||
`[paper] <https://papers.nips.cc/paper/1713-policy-gradient-methods-for-reinforcement-learning-with-function-approximation.pdf>`__ `[implementation] <https://github.com/ray-project/ray/blob/master/python/ray/rllib/agents/pg/pg.py>`__ We include a vanilla policy gradients implementation as an example algorithm. This is usually outperformed by PPO.
|
||||
|
||||
Tuned examples: `CartPole-v0 <https://github.com/ray-project/ray/blob/master/python/ray/rllib/tuned_examples/regression_tests/cartpole-pg.yaml>`__
|
||||
|
||||
**PG-specific configs** (see also `common configs <rllib-training.html#common-parameters>`__):
|
||||
|
||||
.. literalinclude:: ../../python/ray/rllib/agents/pg/pg.py
|
||||
:language: python
|
||||
:start-after: __sphinx_doc_begin__
|
||||
:end-before: __sphinx_doc_end__
|
||||
|
||||
Proximal Policy Optimization (PPO)
|
||||
----------------------------------
|
||||
`[paper] <https://arxiv.org/abs/1707.06347>`__ `[implementation] <https://github.com/ray-project/ray/blob/master/python/ray/rllib/agents/ppo/ppo.py>`__
|
||||
@@ -158,6 +200,13 @@ SpaceInvaders 671 944 ~800
|
||||
|
||||
RLlib's multi-GPU PPO scales to multiple GPUs and hundreds of CPUs on solving the Humanoid-v1 task. Here we compare against a reference MPI-based implementation.
|
||||
|
||||
**PPO-specific configs** (see also `common configs <rllib-training.html#common-parameters>`__):
|
||||
|
||||
.. literalinclude:: ../../python/ray/rllib/agents/ppo/ppo.py
|
||||
:language: python
|
||||
:start-after: __sphinx_doc_begin__
|
||||
:end-before: __sphinx_doc_end__
|
||||
|
||||
Derivative-free
|
||||
~~~~~~~~~~~~~~~
|
||||
|
||||
@@ -168,6 +217,13 @@ ARS is a random search method for training linear policies for continuous contro
|
||||
|
||||
Tuned examples: `CartPole-v0 <https://github.com/ray-project/ray/blob/master/python/ray/rllib/tuned_examples/regression_tests/cartpole-ars.yaml>`__, `Swimmer-v2 <https://github.com/ray-project/ray/blob/master/python/ray/rllib/tuned_examples/swimmer-ars.yaml>`__
|
||||
|
||||
**ARS-specific configs** (see also `common configs <rllib-training.html#common-parameters>`__):
|
||||
|
||||
.. literalinclude:: ../../python/ray/rllib/agents/ars/ars.py
|
||||
:language: python
|
||||
:start-after: __sphinx_doc_begin__
|
||||
:end-before: __sphinx_doc_end__
|
||||
|
||||
Evolution Strategies
|
||||
--------------------
|
||||
`[paper] <https://arxiv.org/abs/1703.03864>`__ `[implementation] <https://github.com/ray-project/ray/blob/master/python/ray/rllib/agents/es/es.py>`__
|
||||
@@ -181,3 +237,10 @@ Tuned examples: `Humanoid-v1 <https://github.com/ray-project/ray/blob/master/pyt
|
||||
:width: 500px
|
||||
|
||||
RLlib's ES implementation scales further and is faster than a reference Redis implementation on solving the Humanoid-v1 task.
|
||||
|
||||
**ES-specific configs** (see also `common configs <rllib-training.html#common-parameters>`__):
|
||||
|
||||
.. literalinclude:: ../../python/ray/rllib/agents/es/es.py
|
||||
:language: python
|
||||
:start-after: __sphinx_doc_begin__
|
||||
:end-before: __sphinx_doc_end__
|
||||
|
||||
@@ -17,6 +17,15 @@ In addition, if you set ``"model": {"use_lstm": true}``, then the model output w
|
||||
|
||||
For preprocessors, RLlib tries to pick one of its built-in preprocessor based on the environment's observation space. Discrete observations are one-hot encoded, Atari observations downscaled, and Tuple observations flattened (there isn't native tuple support yet, but you can reshape the flattened observation in a custom model). Note that for Atari, RLlib defaults to using the `DeepMind preprocessors <https://github.com/ray-project/ray/blob/master/python/ray/rllib/env/atari_wrappers.py>`__, which are also used by the OpenAI baselines library.
|
||||
|
||||
Built-in Model Parameters
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The following is a list of the built-in model hyperparameters:
|
||||
|
||||
.. literalinclude:: ../../python/ray/rllib/models/catalog.py
|
||||
:language: python
|
||||
:start-after: __sphinx_doc_begin__
|
||||
:end-before: __sphinx_doc_end__
|
||||
|
||||
Custom Models
|
||||
-------------
|
||||
|
||||
@@ -37,26 +37,6 @@ with ``--env`` (any OpenAI gym environment including ones registered by the user
|
||||
can be used) and for choosing the algorithm with ``--run``
|
||||
(available options are ``PPO``, ``PG``, ``A2C``, ``A3C``, ``IMPALA``, ``ES``, ``DDPG``, ``DQN``, ``APEX``, and ``APEX_DDPG``).
|
||||
|
||||
Specifying Parameters
|
||||
~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Each algorithm has specific hyperparameters that can be set with ``--config``, in addition to a number of `common hyperparameters <https://github.com/ray-project/ray/blob/master/python/ray/rllib/agents/agent.py>`__. See the
|
||||
`algorithms documentation <rllib-algorithms.html>`__ for more information.
|
||||
|
||||
In an example below, we train A2C by specifying 8 workers through the config flag. We also set ``"monitor": true`` to save episode videos to the result dir:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
python ray/python/ray/rllib/train.py --env=PongDeterministic-v4 \
|
||||
--run=A2C --config '{"num_workers": 8, "monitor": true}'
|
||||
|
||||
.. image:: rllib-config.svg
|
||||
|
||||
Specifying Resources
|
||||
~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
You can control the degree of parallelism used by setting the ``num_workers`` hyperparameter for most agents. Many agents also provide a ``num_gpus`` or ``gpu`` option. In addition, you can allocate a fraction of a GPU by setting ``gpu_fraction: f``. For example, with DQN you can pack five agents onto one GPU by setting ``gpu_fraction: 0.2``. Note that fractional GPU support requires enabling the experimental Xray backend by setting the environment variable ``RAY_USE_XRAY=1``.
|
||||
|
||||
Evaluating Trained Agents
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
@@ -77,6 +57,39 @@ The ``rollout.py`` helper script reconstructs a DQN agent from the checkpoint
|
||||
located at ``~/ray_results/default/DQN_CartPole-v0_0upjmdgr0/checkpoint-1``
|
||||
and renders its behavior in the environment specified by ``--env``.
|
||||
|
||||
Configuration
|
||||
-------------
|
||||
|
||||
Specifying Parameters
|
||||
~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Each algorithm has specific hyperparameters that can be set with ``--config``, in addition to a number of `common hyperparameters <https://github.com/ray-project/ray/blob/master/python/ray/rllib/agents/agent.py>`__. See the
|
||||
`algorithms documentation <rllib-algorithms.html>`__ for more information.
|
||||
|
||||
In an example below, we train A2C by specifying 8 workers through the config flag. We also set ``"monitor": true`` to save episode videos to the result dir:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
python ray/python/ray/rllib/train.py --env=PongDeterministic-v4 \
|
||||
--run=A2C --config '{"num_workers": 8, "monitor": true}'
|
||||
|
||||
.. image:: rllib-config.svg
|
||||
|
||||
Specifying Resources
|
||||
~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
You can control the degree of parallelism used by setting the ``num_workers`` hyperparameter for most agents. Many agents also provide a ``num_gpus`` or ``gpu`` option. In addition, you can allocate a fraction of a GPU by setting ``gpu_fraction: f``. For example, with DQN you can pack five agents onto one GPU by setting ``gpu_fraction: 0.2``. Note that fractional GPU support requires enabling the experimental X-ray backend by setting the environment variable ``RAY_USE_XRAY=1``.
|
||||
|
||||
Common Parameters
|
||||
~~~~~~~~~~~~~~~~~
|
||||
|
||||
The following is a list of the common agent hyperparameters:
|
||||
|
||||
.. literalinclude:: ../../python/ray/rllib/agents/agent.py
|
||||
:language: python
|
||||
:start-after: __sphinx_doc_begin__
|
||||
:end-before: __sphinx_doc_end__
|
||||
|
||||
Tuned Examples
|
||||
~~~~~~~~~~~~~~
|
||||
|
||||
@@ -154,7 +167,7 @@ Tune will schedule the trials to run in parallel on your Ray cluster:
|
||||
== Status ==
|
||||
Using FIFO scheduling algorithm.
|
||||
Resources requested: 4/4 CPUs, 0/0 GPUs
|
||||
Result logdir: /home/eric/ray_results/my_experiment
|
||||
Result logdir: ~/ray_results/my_experiment
|
||||
PENDING trials:
|
||||
- PPO_CartPole-v0_2_sgd_stepsize=0.0001: PENDING
|
||||
RUNNING trials:
|
||||
|
||||
@@ -27,6 +27,7 @@ You might also want to clone the Ray repo for convenient access to RLlib helper
|
||||
Training APIs
|
||||
-------------
|
||||
* `Command-line <rllib-training.html>`__
|
||||
* `Configuration <rllib-training.html#configuration>`__
|
||||
* `Python API <rllib-training.html#python-api>`__
|
||||
* `REST API <rllib-training.html#rest-api>`__
|
||||
|
||||
|
||||
Reference in New Issue
Block a user