mirror of
https://github.com/wassname/ray.git
synced 2026-08-07 11:27:43 +08:00
[rllib] [docs] Cleanup RLlib API and make docs consistent with upcoming blog post (#1708)
* wip * more work * fix apex * docs * apex doc * pool comment * clean up * make wrap stack pluggable * Mon Mar 12 21:45:50 PDT 2018 * clean up comment * table * Mon Mar 12 22:51:57 PDT 2018 * Mon Mar 12 22:53:05 PDT 2018 * Mon Mar 12 22:55:03 PDT 2018 * Mon Mar 12 22:56:18 PDT 2018 * Mon Mar 12 22:59:54 PDT 2018 * Update apex_optimizer.py * Update index.rst * Update README.rst * Update README.rst * comments * Wed Mar 14 19:01:02 PDT 2018
This commit is contained in:
Eric Liang
@@ -320,4 +320,7 @@ texinfo_documents = [
|
||||
# pcmoritz: To make the following work, you have to run
|
||||
# sudo pip install recommonmark
|
||||
|
||||
# Python methods should be presented in source code order
|
||||
autodoc_member_order = 'bysource'
|
||||
|
||||
# see also http://searchvoidstar.tumblr.com/post/125486358368/making-pdfs-from-markdown-on-readthedocsorg-using
|
||||
|
||||
@@ -41,7 +41,7 @@ View the `codebase on GitHub`_.
|
||||
Ray comes with libraries that accelerate deep learning and reinforcement learning development:
|
||||
|
||||
- `Ray Tune`_: Hyperparameter Optimization Framework
|
||||
- `Ray RLlib`_: A Scalable Reinforcement Learning Library
|
||||
- `Ray RLlib`_: Scalable Reinforcement Learning
|
||||
|
||||
.. _`Ray Tune`: tune.html
|
||||
.. _`Ray RLlib`: rllib.html
|
||||
@@ -78,6 +78,7 @@ Ray comes with libraries that accelerate deep learning and reinforcement learnin
|
||||
:caption: Ray RLlib
|
||||
|
||||
rllib.rst
|
||||
policy-optimizers.rst
|
||||
rllib-dev.rst
|
||||
|
||||
.. toctree::
|
||||
|
||||
@@ -0,0 +1,67 @@
|
||||
Policy Optimizers
|
||||
=================
|
||||
|
||||
RLlib supports using its distributed policy optimizer implementations from external algorithms.
|
||||
|
||||
Example of constructing and using a policy optimizer `(link to full example) <https://github.com/ericl/baselines/blob/rllib-example/baselines/deepq/run_simple_loop.py>`__:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
ray.init()
|
||||
env_creator = lambda env_config: gym.make("PongNoFrameskip-v4")
|
||||
optimizer = LocalSyncReplayOptimizer.make(
|
||||
YourEvaluatorClass, [env_creator], num_workers=0, optimizer_config={})
|
||||
|
||||
i = 0
|
||||
while optimizer.num_steps_sampled < 100000:
|
||||
i += 1
|
||||
print("== optimizer step {} ==".format(i))
|
||||
optimizer.step()
|
||||
print("optimizer stats", optimizer.stats())
|
||||
print("local evaluator stats", optimizer.local_evaluator.stats())
|
||||
|
||||
Here are the steps for using a RLlib policy optimizer with an existing algorithm.
|
||||
|
||||
1. Implement the `Policy evaluator interface <rllib-dev.html#policy-evaluators-and-optimizers>`__.
|
||||
|
||||
- Here is an example of porting a `PyTorch Rainbow implementation <https://github.com/ericl/Rainbow/blob/rllib-example/rainbow_evaluator.py>`__.
|
||||
|
||||
- Another example porting a `TensorFlow DQN implementation <https://github.com/ericl/baselines/blob/rllib-example/baselines/deepq/dqn_evaluator.py>`__.
|
||||
|
||||
2. Pick a `Policy optimizer class <https://github.com/ray-project/ray/tree/master/python/ray/rllib/optimizers>`__. The `LocalSyncOptimizer <https://github.com/ray-project/ray/blob/master/python/ray/rllib/optimizers/local_sync.py>`__ is a reasonable choice for local testing. You can also implement your own. Policy optimizers can be constructed using their ``make`` method (e.g., ``LocalSyncOptimizer.make(evaluator_cls, evaluator_args, num_workers, optimizer_config)``), or you can construct them by passing in a list of evaluators instantiated as Ray actors.
|
||||
|
||||
- Here is code showing the `simple Policy Gradient agent <https://github.com/ray-project/ray/blob/master/python/ray/rllib/pg/pg.py>`__ using ``make()``.
|
||||
|
||||
- A different example showing an `A3C agent <https://github.com/ray-project/ray/blob/master/python/ray/rllib/a3c/a3c.py>`__ passing in Ray actors directly.
|
||||
|
||||
3. Decide how you want to drive the training loop.
|
||||
|
||||
- Option 1: call ``optimizer.step()`` from some existing training code. Training statistics can be retrieved by querying the ``optimizer.local_evaluator`` evaluator instance, or mapping over the remote evaluators (e.g., ``ray.get([ev.some_fn.remote() for ev in optimizer.remote_evaluators])``) if you are running with multiple workers.
|
||||
|
||||
- Option 2: define a full RLlib `Agent class <https://github.com/ray-project/ray/blob/master/python/ray/rllib/agent.py>`__. This might be preferable if you don't have an existing training harness or want to use features provided by `Ray Tune <tune.html>`__.
|
||||
|
||||
Available Policy Optimizers
|
||||
---------------------------
|
||||
|
||||
+-----------------------------+---------------------+-----------------+------------------------------+
|
||||
| **Policy optimizer class** | **Operating range** | **Works with** | **Description** |
|
||||
+=============================+=====================+=================+==============================+
|
||||
|AsyncOptimizer |1-10s of CPUs |(any) |Asynchronous gradient-based |
|
||||
| | | |optimization (e.g., A3C) |
|
||||
+-----------------------------+---------------------+-----------------+------------------------------+
|
||||
|LocalSyncOptimizer |0-1 GPUs + |(any) |Synchronous gradient-based |
|
||||
| |1-100s of CPUs | |optimization with parallel |
|
||||
| | | |sample collection |
|
||||
+-----------------------------+---------------------+-----------------+------------------------------+
|
||||
|LocalSyncReplayOptimizer |0-1 GPUs + | Off-policy |Adds a replay buffer |
|
||||
| |1-100s of CPUs | algorithms |to LocalSyncOptimizer |
|
||||
+-----------------------------+---------------------+-----------------+------------------------------+
|
||||
|LocalMultiGPUOptimizer |0-10 GPUs + | Algorithms |Implements data-parallel |
|
||||
| |1-100s of CPUs | written in |optimization over multiple |
|
||||
| | | TensorFlow |GPUs, e.g., for PPO |
|
||||
+-----------------------------+---------------------+-----------------+------------------------------+
|
||||
|ApexOptimizer |1 GPU + | Off-policy |Implements the Ape-X |
|
||||
| |10-100s of CPUs | algorithms |distributed prioritization |
|
||||
| | | w/sample |algorithm |
|
||||
| | | prioritization | |
|
||||
+-----------------------------+---------------------+-----------------+------------------------------+
|
||||
@@ -42,10 +42,10 @@ a common base class:
|
||||
Policy Evaluators and Optimizers
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. autoclass:: ray.rllib.optimizers.evaluator.Evaluator
|
||||
.. autoclass:: ray.rllib.optimizers.policy_evaluator.PolicyEvaluator
|
||||
:members:
|
||||
|
||||
.. autoclass:: ray.rllib.optimizers.optimizer.Optimizer
|
||||
.. autoclass:: ray.rllib.optimizers.policy_optimizer.PolicyOptimizer
|
||||
:members:
|
||||
|
||||
Sample Batches
|
||||
|
||||
+19
-16
@@ -1,21 +1,16 @@
|
||||
Ray RLlib: A Scalable Reinforcement Learning Library
|
||||
====================================================
|
||||
Ray RLlib: Scalable Reinforcement Learning
|
||||
==========================================
|
||||
|
||||
Ray RLlib is a reinforcement learning library that aims to provide both performance and composability:
|
||||
Ray RLlib is an RL execution toolkit built on the Ray distributed execution framework. RLlib implements a collection of distributed *policy optimizers* that make it easy to use a variety of training strategies with existing RL algorithms written in frameworks such as PyTorch, TensorFlow, and Theano. This enables complex architectures for RL training (e.g., Ape-X, IMPALA), to be implemented once and reused many times across different RL algorithms and libraries.
|
||||
|
||||
- Performance
|
||||
- High performance algorithm implementions
|
||||
- Pluggable distributed RL execution strategies
|
||||
You can find the code for RLlib `here on GitHub <https://github.com/ray-project/ray/tree/master/python/ray/rllib>`__, and the paper `here <https://arxiv.org/abs/1712.09381>`__.
|
||||
|
||||
- Composability
|
||||
- Integration with the `Ray Tune <tune.html>`__ hyperparam tuning tool
|
||||
- Support for multiple frameworks (TensorFlow, PyTorch)
|
||||
- Scalable primitives for developing new algorithms
|
||||
- Shared models between algorithms
|
||||
.. note::
|
||||
|
||||
You can find the code for RLlib `here on GitHub <https://github.com/ray-project/ray/tree/master/python/ray/rllib>`__, and the NIPS symposium paper `here <https://arxiv.org/abs/1712.09381>`__.
|
||||
To use RLlib's policy optimizers outside of RLlib, see the `RLlib policy optimizers documentation <policy-optimizers.html>`__.
|
||||
|
||||
RLlib currently provides the following algorithms:
|
||||
|
||||
RLlib's policy optimizers serve as the basis for RLlib's reference algorithms, which include:
|
||||
|
||||
- `Proximal Policy Optimization (PPO) <https://arxiv.org/abs/1707.06347>`__ which
|
||||
is a proximal variant of `TRPO <https://arxiv.org/abs/1502.05477>`__.
|
||||
@@ -24,6 +19,8 @@ RLlib currently provides the following algorithms:
|
||||
|
||||
- `Deep Q Networks (DQN) <https://arxiv.org/abs/1312.5602>`__.
|
||||
|
||||
- `Ape-X Distributed Prioritized Experience Replay <https://arxiv.org/abs/1803.00933>`__.
|
||||
|
||||
- Evolution Strategies, as described in `this
|
||||
paper <https://arxiv.org/abs/1703.03864>`__. Our implementation
|
||||
is adapted from
|
||||
@@ -80,7 +77,7 @@ The ``train.py`` script has a number of options you can show by running
|
||||
The most important options are for choosing the environment
|
||||
with ``--env`` (any OpenAI gym environment including ones registered by the user
|
||||
can be used) and for choosing the algorithm with ``--run``
|
||||
(available options are ``PPO``, ``A3C``, ``ES`` and ``DQN``).
|
||||
(available options are ``PPO``, ``A3C``, ``ES``, ``DQN`` and ``APEX``).
|
||||
|
||||
Specifying Parameters
|
||||
~~~~~~~~~~~~~~~~~~~~~
|
||||
@@ -89,8 +86,9 @@ Each algorithm has specific hyperparameters that can be set with ``--config`` -
|
||||
``DEFAULT_CONFIG`` variable in
|
||||
`PPO <https://github.com/ray-project/ray/blob/master/python/ray/rllib/ppo/ppo.py>`__,
|
||||
`A3C <https://github.com/ray-project/ray/blob/master/python/ray/rllib/a3c/a3c.py>`__,
|
||||
`ES <https://github.com/ray-project/ray/blob/master/python/ray/rllib/es/es.py>`__ and
|
||||
`DQN <https://github.com/ray-project/ray/blob/master/python/ray/rllib/dqn/dqn.py>`__.
|
||||
`ES <https://github.com/ray-project/ray/blob/master/python/ray/rllib/es/es.py>`__,
|
||||
`DQN <https://github.com/ray-project/ray/blob/master/python/ray/rllib/dqn/dqn.py>`__ and
|
||||
`APEX <https://github.com/ray-project/ray/blob/master/python/ray/rllib/dqn/apex.py>`__.
|
||||
|
||||
In an example below, we train A3C by specifying 8 workers through the config flag.
|
||||
function that creates the env to refer to it by name. The contents of the env_config agent config field will be passed to that function to allow the environment to be configured. The return type should be an OpenAI gym.Env. For example:
|
||||
@@ -325,6 +323,11 @@ in the ``config`` section of the experiments.
|
||||
For an advanced example of using Population Based Training (PBT) with RLlib,
|
||||
see the `PPO + PBT Walker2D training example <https://github.com/ray-project/ray/blob/master/python/ray/tune/examples/pbt_ppo_example.py>`__.
|
||||
|
||||
Using Policy Optimizers outside of RLlib
|
||||
----------------------------------------
|
||||
|
||||
See the `RLlib policy optimizers documentation <policy-optimizers.html>`__.
|
||||
|
||||
Contributing to RLlib
|
||||
---------------------
|
||||
|
||||
|
||||
Reference in New Issue
Block a user