mirror of
https://github.com/wassname/ray.git
synced 2026-08-11 11:24:51 +08:00
[rllib] Update docs with api and components overview figures (#1443)
This commit is contained in:
Eric Liang
File diff suppressed because one or more lines are too long
|
After Width: | Height: | Size: 72 KiB |
File diff suppressed because one or more lines are too long
|
After Width: | Height: | Size: 100 KiB |
@@ -10,49 +10,23 @@ Recipe for an RLlib algorithm
|
||||
|
||||
Here are the steps for implementing a new algorithm in RLlib:
|
||||
|
||||
1. Define an algorithm-specific `Evaluator class <#evaluators-and-optimizers>`__ (the core of the algorithm). Evaluators encapsulate framework-specific components such as the policy and loss functions. For an example, see the `A3C Evaluator implementation <https://github.com/ray-project/ray/blob/master/python/ray/rllib/a3c/a3c_evaluator.py>`__.
|
||||
1. Define an algorithm-specific `Policy evaluator class <#policy-evaluators-and-optimizers>`__ (the core of the algorithm). Evaluators encapsulate framework-specific components such as the policy and loss functions. For an example, see the `A3C Evaluator implementation <https://github.com/ray-project/ray/blob/master/python/ray/rllib/a3c/a3c_evaluator.py>`__.
|
||||
|
||||
|
||||
2. Pick an appropriate `RLlib optimizer class <#evaluators-and-optimizers>`__. Optimizers manage the parallel execution of the algorithm. RLlib provides several built-in optimizers for gradient-based algorithms. Advanced algorithms may find it beneficial to implement their own optimizers.
|
||||
2. Pick an appropriate `Policy optimizer class <#policy-evaluators-and-optimizers>`__. Optimizers manage the parallel execution of the algorithm. RLlib provides several built-in optimizers for gradient-based algorithms. Advanced algorithms may find it beneficial to implement their own optimizers.
|
||||
|
||||
|
||||
3. Wrap the two up in an `Agent class <#agents>`__. Agents are the user-facing API of RLlib. They provide the necessary "glue" and implement accessory functionality such as statistics reporting and checkpointing.
|
||||
|
||||
To help with implementation, RLlib provides common action distributions, preprocessors, and neural network models, found in `catalog.py <https://github.com/ray-project/ray/blob/master/python/ray/rllib/models/catalog.py>`__, which are shared by all algorithms. Note that most of these utilities are currently Tensorflow specific.
|
||||
|
||||
Defining a custom model
|
||||
-----------------------
|
||||
|
||||
Often you will want to plug in your own neural network into an existing RLlib algorithm.
|
||||
This can be easily done by defining your own `Model class <#models-and-preprocessors>`__ and registering it in the RLlib catalog, after which it will be available for use by all RLlib algorithms.
|
||||
|
||||
An example usage of a custom model looks like this:
|
||||
|
||||
::
|
||||
|
||||
from ray.rllib.models import ModelCatalog, Model
|
||||
|
||||
class MyModelClass(Model):
|
||||
def _init(self, inputs, num_outputs, options):
|
||||
layer1 = slim.fully_connected(inputs, 64, ...)
|
||||
layer2 = slim.fully_connected(inputs, 64, ...)
|
||||
...
|
||||
return layerN, layerN_minus_1
|
||||
|
||||
ModelCatalog.register_custom_model("my_model", MyModelClass)
|
||||
|
||||
alg = ppo.PPOAgent(env="CartPole-v0", config={
|
||||
"custom_model": "my_model",
|
||||
})
|
||||
|
||||
|
||||
Note that if you need to reference large data objects as part of the computation, e.g. weights, you can put them into the Ray object store with ``ray.put`` and then retrieve them from inside your model class.
|
||||
.. image:: rllib-api.svg
|
||||
|
||||
|
||||
The Developer API
|
||||
-----------------
|
||||
|
||||
The following APIs are the building blocks of RLlib algorithms. Note that they are not yet considered stable.
|
||||
The following APIs are the building blocks of RLlib algorithms (also take a look at the `user components overview <rllib.html#components-user-customizable-and-internal>`__).
|
||||
|
||||
Agents
|
||||
~~~~~~
|
||||
@@ -65,8 +39,8 @@ a common base class:
|
||||
.. autoclass:: ray.rllib.agent.Agent
|
||||
:members:
|
||||
|
||||
Evaluators and Optimizers
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
Policy Evaluators and Optimizers
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. autoclass:: ray.rllib.optimizers.evaluator.Evaluator
|
||||
:members:
|
||||
@@ -123,7 +97,7 @@ Currently we support the following action distributions:
|
||||
The Model Catalog
|
||||
~~~~~~~~~~~~~~~~~
|
||||
|
||||
The Model Catalog is the mechanism for algorithms to get preprocessors, models, and action distributions for varying gym environments. It enables sharing of these components across different algorithms.
|
||||
The Model Catalog is the mechanism for algorithms to get canonical preprocessors, models, and action distributions for varying gym environments. It enables easy reuse of these components across different algorithms.
|
||||
|
||||
.. autoclass:: ray.rllib.models.ModelCatalog
|
||||
:members:
|
||||
|
||||
+75
-28
@@ -35,11 +35,12 @@ including custom ones written and registered by the user.
|
||||
Installation
|
||||
------------
|
||||
|
||||
RLlib has extra dependencies on top of **ray**:
|
||||
RLlib has extra dependencies on top of **ray**. You might also want to clone the Ray repo for convenient access to RLlib helper scripts:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
pip install 'ray[rllib]'
|
||||
git clone https://github.com/ray-project/ray
|
||||
|
||||
For usage of PyTorch models, visit the `PyTorch website <http://pytorch.org/>`__
|
||||
for instructions on installing PyTorch.
|
||||
@@ -47,6 +48,12 @@ for instructions on installing PyTorch.
|
||||
Getting Started
|
||||
---------------
|
||||
|
||||
At a high level, RLlib provides an ``Agent`` class which
|
||||
holds a policy for environment interaction. Through the agent interface, the policy can
|
||||
be trained, checkpointed, or an action computed.
|
||||
|
||||
.. image:: rllib-api.svg
|
||||
|
||||
You can train a simple DQN agent with the following command
|
||||
|
||||
::
|
||||
@@ -88,7 +95,8 @@ Each algorithm has specific hyperparameters that can be set with ``--config`` -
|
||||
In an example below, we train A3C by specifying 8 workers through the config flag.
|
||||
::
|
||||
|
||||
python ray/python/ray/rllib/train.py --env=PongDeterministic-v4 --run=A3C --config '{"num_workers": 8}'
|
||||
python ray/python/ray/rllib/train.py --env=PongDeterministic-v4 \
|
||||
--run=A3C --config '{"num_workers": 8}'
|
||||
|
||||
Evaluating Trained Agents
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
@@ -98,34 +106,33 @@ set ``--checkpoint-freq`` (number of training iterations between checkpoints)
|
||||
when running ``train.py``.
|
||||
|
||||
|
||||
You can evaluate a simple DQN agent with the following command
|
||||
An example of evaluating a previously trained DQN agent is as follows:
|
||||
|
||||
::
|
||||
|
||||
python ray/python/ray/rllib/eval.py \
|
||||
/tmp/ray/default/DQN_CartPole-v0_0upjmdgr0/checkpoint-1 \
|
||||
~/ray_results/default/DQN_CartPole-v0_0upjmdgr0/checkpoint-1 \
|
||||
--run DQN --env CartPole-v0
|
||||
|
||||
|
||||
By default, the script reconstructs a DQN agent from the checkpoint
|
||||
located at ``/tmp/ray/default/DQN_CartPole-v0_0upjmdgr0/checkpoint-1``
|
||||
The ``eval.py`` helper script reconstructs a DQN agent from the checkpoint
|
||||
located at ``~/ray_results/default/DQN_CartPole-v0_0upjmdgr0/checkpoint-1``
|
||||
and renders its behavior in the environment specified by ``--env``.
|
||||
Checkpoints are be found within the experiment directory,
|
||||
specified by ``--local-dir`` and ``--experiment-name`` when running ``train.py``.
|
||||
|
||||
Tuned Examples
|
||||
--------------
|
||||
|
||||
Some good hyperparameters and settings are available in
|
||||
`the repository <https://github.com/ray-project/ray/blob/master/python/ray/rllib/test/tuned_examples.sh>`__
|
||||
`the repository <https://github.com/ray-project/ray/blob/master/python/ray/rllib/tuned_examples>`__
|
||||
(some of them are tuned to run on GPUs). If you find better settings or tune
|
||||
an algorithm on a different domain, consider submitting a Pull Request!
|
||||
|
||||
Python User API
|
||||
---------------
|
||||
|
||||
You will be using this part of the API if you run the existing algorithms
|
||||
on a new problem. Here is an example how to use it:
|
||||
The Python API provides the needed flexibility for applying RLlib to new problems. You will need to use this API if you wish to use custom environments, preprocesors, or models with RLlib.
|
||||
|
||||
Here is an example of the basic usage:
|
||||
|
||||
::
|
||||
|
||||
@@ -133,23 +140,38 @@ on a new problem. Here is an example how to use it:
|
||||
import ray.rllib.ppo as ppo
|
||||
|
||||
ray.init()
|
||||
|
||||
config = ppo.DEFAULT_CONFIG.copy()
|
||||
alg = ppo.PPOAgent(config=config, env="CartPole-v1")
|
||||
agent = ppo.PPOAgent(config=config, env="CartPole-v0")
|
||||
|
||||
# Can optionally call alg.restore(path) to load a checkpoint.
|
||||
# Can optionally call agent.restore(path) to load a checkpoint.
|
||||
|
||||
for i in range(10):
|
||||
# Perform one iteration of the algorithm.
|
||||
result = alg.train()
|
||||
for i in range(1000):
|
||||
# Perform one iteration of training the policy with PPO
|
||||
result = agent.train()
|
||||
print("result: {}".format(result))
|
||||
print("checkpoint saved at path: {}".format(alg.save()))
|
||||
|
||||
if i % 100 == 0:
|
||||
checkpoint = agent.save()
|
||||
print("checkpoint saved at", checkpoint)
|
||||
|
||||
Components: User-customizable and Internal
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The following diagram provides a conceptual overview of data flow between different components in RLlib. We start with an ``Environment``, which given an action produces an observation. The observation is preprocessed by a ``Preprocessor`` and ``Filter`` (e.g. for running mean normalization) before being sent to a neural network ``Model``. The model output is in turn interpreted by an ``ActionDistribution`` to determine the next action.
|
||||
|
||||
.. image:: rllib-components.svg
|
||||
|
||||
The components highlighted in green above are *User-customizable*, which means RLlib provides APIs for swapping in user-defined implementations, as described in the next sections. The purple components are *RLlib internal*, which means they currently can only be modified by changing the RLlib source code.
|
||||
|
||||
For more information about these components, also see the `RLlib Developer Guide <rllib-dev.html>`__.
|
||||
|
||||
Custom Environments
|
||||
~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
To train against a custom environment, i.e. one not in the gym catalog, you
|
||||
can register a function that creates the env to refer to it by name. For example:
|
||||
can register a function that creates the env to refer to it by name. The contents of the
|
||||
``env_config`` agent config field will be passed to that function to allow the
|
||||
environment to be configured. For example:
|
||||
|
||||
::
|
||||
|
||||
@@ -157,32 +179,50 @@ can register a function that creates the env to refer to it by name. For example
|
||||
from ray.tune.registry import register_env
|
||||
from ray.rllib import ppo
|
||||
|
||||
env_creator = lambda env_config: create_my_env()
|
||||
env_creator = lambda env_config: MyCustomEnv(env_config)
|
||||
env_creator_name = "custom_env"
|
||||
register_env(env_creator_name, env_creator)
|
||||
|
||||
ray.init()
|
||||
alg = ppo.PPOAgent(env=env_creator_name)
|
||||
agent = ppo.PPOAgent(env=env_creator_name, config={
|
||||
"env_config": {}, # config to pass to env creator
|
||||
})
|
||||
|
||||
|
||||
Custom Models and Preprocessors
|
||||
Custom Preprocessors and Models
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
RLlib includes default neural network models and preprocessors for common gym
|
||||
environments, but you can also specify your own as follows. The interfaces for
|
||||
custom model and preprocessor classes are documented in the
|
||||
RLlib includes default preprocessors and models for common gym
|
||||
environments, but you can also specify your own as follows. At a high level, your neural
|
||||
network model needs to take an input tensor of the preprocessed observation shape and
|
||||
output a vector of the size specified in the constructor. The interfaces for
|
||||
these custom classes can be found in the
|
||||
`RLlib Developer Guide <rllib-dev.html>`__.
|
||||
|
||||
::
|
||||
|
||||
import ray
|
||||
from ray.rllib.models import ModelCatalog
|
||||
from ray.rllib.models import ModelCatalog, Model
|
||||
from ray.rllib.models.preprocessors import Preprocessor
|
||||
|
||||
class MyPreprocessorClass(Preprocessor):
|
||||
def _init(self):
|
||||
self.shape = ...
|
||||
|
||||
def transform(self, observation):
|
||||
return ...
|
||||
|
||||
class MyModelClass(Model):
|
||||
def _init(self, inputs, num_outputs, options):
|
||||
layer1 = slim.fully_connected(inputs, 64, ...)
|
||||
layer2 = slim.fully_connected(inputs, 64, ...)
|
||||
...
|
||||
return layerN, layerN_minus_1
|
||||
|
||||
ModelCatalog.register_custom_preprocessor("my_prep", MyPreprocessorClass)
|
||||
ModelCatalog.register_custom_model("my_model", MyModelClass)
|
||||
|
||||
ray.init()
|
||||
alg = ppo.PPOAgent(env="CartPole-v0", config={
|
||||
agent = ppo.PPOAgent(env="CartPole-v0", config={
|
||||
"model": {
|
||||
"custom_preprocessor": "my_prep",
|
||||
"custom_model": "my_model",
|
||||
@@ -190,6 +230,13 @@ custom model and preprocessor classes are documented in the
|
||||
},
|
||||
})
|
||||
|
||||
For a full example of a custom model in code, see the `Carla RLlib model <https://github.com/ray-project/ray/blob/master/examples/carla/models.py>`__ and associated `training scripts <https://github.com/ray-project/ray/tree/master/examples/carla>`__. The ``CarlaModel`` class defined there operates over a composite (Tuple) observation space including both images and scalar measurements.
|
||||
|
||||
External Data API
|
||||
~~~~~~~~~~~~~~~~~
|
||||
*coming soon!*
|
||||
|
||||
|
||||
Using RLlib with Ray.tune
|
||||
-------------------------
|
||||
|
||||
|
||||
@@ -33,8 +33,8 @@ All RLlib algorithms implement a common training API (agent.py), which enables m
|
||||
# Integration with ray.tune for hyperparam evaluation
|
||||
python train.py -f tuned_examples/cartpole-grid-search-example.yaml
|
||||
|
||||
Evaluator and Optimizer abstractions
|
||||
------------------------------------
|
||||
Policy Evaluator and Optimizer abstractions
|
||||
-------------------------------------------
|
||||
|
||||
RLlib's gradient-based algorithms are composed using two abstractions: Evaluators (evaluator.py) and Optimizers (optimizers/optimizer.py). Optimizers encapsulate a particular distributed optimization strategy for RL. Evaluators encapsulate the model graph, and once implemented, any Optimizer may be "plugged in" to any algorithm that implements the Evaluator interface.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user