diff --git a/doc/source/index.rst b/doc/source/index.rst
index 70c28fade..600dbaa91 100644
--- a/doc/source/index.rst
+++ b/doc/source/index.rst
@@ -200,6 +200,7 @@ The following are good places to discuss Ray.
:caption: RLlib
rllib.rst
+ rllib-toc.rst
rllib-training.rst
rllib-env.rst
rllib-models.rst
diff --git a/doc/source/multi-flat.svg b/doc/source/multi-flat.svg
new file mode 100644
index 000000000..c6aceecc1
--- /dev/null
+++ b/doc/source/multi-flat.svg
@@ -0,0 +1 @@
+
\ No newline at end of file
diff --git a/doc/source/rllib-toc.rst b/doc/source/rllib-toc.rst
new file mode 100644
index 000000000..8c7c841b9
--- /dev/null
+++ b/doc/source/rllib-toc.rst
@@ -0,0 +1,138 @@
+RLlib Table of Contents
+=======================
+
+Training APIs
+-------------
+* `Command-line `__
+* `Configuration `__
+* `Python API `__
+* `Debugging `__
+* `REST API `__
+
+Environments
+------------
+* `RLlib Environments Overview `__
+* `Feature Compatibility Matrix `__
+* `OpenAI Gym `__
+* `Vectorized `__
+* `Multi-Agent and Hierarchical `__
+* `Interfacing with External Agents `__
+* `Advanced Integrations `__
+
+Models, Preprocessors, and Action Distributions
+-----------------------------------------------
+* `RLlib Models, Preprocessors, and Action Distributions Overview `__
+* `TensorFlow Models `__
+* `PyTorch Models `__
+* `Custom Preprocessors `__
+* `Custom Action Distributions `__
+* `Supervised Model Losses `__
+* `Variable-length / Parametric Action Spaces `__
+* `Autoregressive Action Distributions `__
+
+Algorithms
+----------
+
+* High-throughput architectures
+
+ - `Distributed Prioritized Experience Replay (Ape-X) `__
+
+ - `Importance Weighted Actor-Learner Architecture (IMPALA) `__
+
+ - `Asynchronous Proximal Policy Optimization (APPO) `__
+
+* Gradient-based
+
+ - `Advantage Actor-Critic (A2C, A3C) `__
+
+ - `Deep Deterministic Policy Gradients (DDPG, TD3) `__
+
+ - `Deep Q Networks (DQN, Rainbow, Parametric DQN) `__
+
+ - `Policy Gradients `__
+
+ - `Proximal Policy Optimization (PPO) `__
+
+ - `Soft Actor Critic (SAC) `__
+
+* Derivative-free
+
+ - `Augmented Random Search (ARS) `__
+
+ - `Evolution Strategies `__
+
+* Multi-agent specific
+
+ - `QMIX Monotonic Value Factorisation (QMIX, VDN, IQN) `__
+ - `Multi-Agent Deep Deterministic Policy Gradient (contrib/MADDPG) `__
+
+* Offline
+
+ - `Advantage Re-Weighted Imitation Learning (MARWIL) `__
+
+Offline Datasets
+----------------
+* `Working with Offline Datasets `__
+* `Input Pipeline for Supervised Losses `__
+* `Input API `__
+* `Output API `__
+
+Concepts and Custom Algorithms
+------------------------------
+* `Policies `__
+
+ - `Policies in Multi-Agent `__
+
+ - `Building Policies in TensorFlow `__
+
+ - `Building Policies in TensorFlow Eager `__
+
+ - `Building Policies in PyTorch `__
+
+ - `Extending Existing Policies `__
+
+* `Policy Evaluation `__
+* `Policy Optimization `__
+* `Trainers `__
+
+Examples
+--------
+
+* `Tuned Examples `__
+* `Training Workflows `__
+* `Custom Envs and Models `__
+* `Serving and Offline `__
+* `Multi-Agent and Hierarchical `__
+* `Community Examples `__
+
+Development
+-----------
+
+* `Development Install `__
+* `API Stability `__
+* `Features `__
+* `Benchmarks `__
+* `Contributing Algorithms `__
+
+Package Reference
+-----------------
+* `ray.rllib.agents `__
+* `ray.rllib.env `__
+* `ray.rllib.evaluation `__
+* `ray.rllib.models `__
+* `ray.rllib.optimizers `__
+* `ray.rllib.utils `__
+
+Troubleshooting
+---------------
+
+If you encounter errors like
+`blas_thread_init: pthread_create: Resource temporarily unavailable` when using many workers,
+try setting ``OMP_NUM_THREADS=1``. Similarly, check configured system limits with
+`ulimit -a` for other resource limit errors.
+
+If you encounter out-of-memory errors, consider setting ``redis_max_memory`` and ``object_store_memory`` in ``ray.init()`` to reduce memory usage.
+
+For debugging unexpected hangs or performance problems, you can run ``ray stack`` to dump
+the stack traces of all Ray workers on the current node, and ``ray timeline`` to dump
+a timeline visualization of tasks to a file.
diff --git a/doc/source/rllib.rst b/doc/source/rllib.rst
index 6bc278940..373cf50f7 100644
--- a/doc/source/rllib.rst
+++ b/doc/source/rllib.rst
@@ -7,155 +7,109 @@ RLlib is an open-source library for reinforcement learning that offers both high
To get started, take a look over the `custom env example `__ and the `API documentation `__. If you're looking to develop custom algorithms with RLlib, also check out `concepts and custom algorithms `__.
-Installation
-------------
+RLlib in 60 seconds
+-------------------
+
+The following is a whirlwind overview of RLlib. See also the full `table of contents `__ for a more in-depth guide including the `list of built-in algorithms `__.
+
+Running RLlib
+~~~~~~~~~~~~~
RLlib has extra dependencies on top of ``ray``. First, you'll need to install either `PyTorch `__ or `TensorFlow `__. Then, install the RLlib module:
.. code-block:: bash
- pip install tensorflow # or tensorflow-gpu
pip install ray[rllib] # also recommended: ray[debug]
-You might also want to clone the `Ray repo `__ for convenient access to RLlib helper scripts:
+Then, you can try out training in the following equivalent ways:
.. code-block:: bash
- git clone https://github.com/ray-project/ray
- cd ray/rllib
+ rllib train --run=PPO --env=CartPole-v0
-Training APIs
--------------
-* `Command-line `__
-* `Configuration `__
-* `Python API `__
-* `Debugging `__
-* `REST API `__
+.. code-block:: python
-Environments
-------------
-* `RLlib Environments Overview `__
-* `Feature Compatibility Matrix `__
-* `OpenAI Gym `__
-* `Vectorized `__
-* `Multi-Agent and Hierarchical `__
-* `Interfacing with External Agents `__
-* `Advanced Integrations `__
+ from ray import tune
+ from ray.rllib.agents.ppo import PPOTrainer
+ tune.run(PPOTrainer, config={"env": "CartPole-v0"})
-Models, Preprocessors, and Action Distributions
------------------------------------------------
-* `RLlib Models, Preprocessors, and Action Distributions Overview `__
-* `TensorFlow Models `__
-* `PyTorch Models `__
-* `Custom Preprocessors `__
-* `Custom Action Distributions `__
-* `Supervised Model Losses `__
-* `Variable-length / Parametric Action Spaces `__
-* `Autoregressive Action Distributions `__
+Next, we'll cover three key concepts in RLlib: Policies, Samples, and Trainers.
-Algorithms
-----------
+Policies
+~~~~~~~~
-* High-throughput architectures
+`Policies `__ are a core concept in RLlib. In a nutshell, policies are Python classes that define how an agent acts in an environment. `Rollout workers `__ query the policy to determine agent actions. In a `gym `__ environment, there is a single agent and policy. In `vector envs `__, policy inference is for multiple agents at once, and in `multi-agent `__, there may be multiple policies, each controlling one or more agents:
- - `Distributed Prioritized Experience Replay (Ape-X) `__
+.. image:: multi-flat.svg
- - `Importance Weighted Actor-Learner Architecture (IMPALA) `__
+Policies can be implemented using `any framework `__. However, for TensorFlow and PyTorch, RLlib has `build_tf_policy `__ and `build_torch_policy `__ helper functions that let you define a trainable policy with a functional-style API, for example:
- - `Asynchronous Proximal Policy Optimization (APPO) `__
+.. code-block:: python
-* Gradient-based
+ def policy_gradient_loss(policy, batch_tensors):
+ actions = batch_tensors[SampleBatch.ACTIONS]
+ rewards = batch_tensors[SampleBatch.REWARDS]
+ return -tf.reduce_mean(policy.action_dist.logp(actions) * rewards)
- - `Advantage Actor-Critic (A2C, A3C) `__
+ #
+ MyTFPolicy = build_tf_policy(
+ name="MyTFPolicy",
+ loss_fn=policy_gradient_loss)
- - `Deep Deterministic Policy Gradients (DDPG, TD3) `__
+Sample Batches
+~~~~~~~~~~~~~~
- - `Deep Q Networks (DQN, Rainbow, Parametric DQN) `__
+Whether running in a single process or `large cluster `__, all data interchange in RLlib is in the form of `sample batches `__. Sample batches encode one or more fragments of a trajectory. Typically, RLlib collects batches of size ``sample_batch_size`` from rollout workers, and concatenates one or more of these batches into a batch of size ``train_batch_size`` that is the input to SGD.
- - `Policy Gradients `__
+A typical sample batch looks something like the following when summarized. Since all values are kept in arrays, this allows for efficient encoding and transmission across the network:
- - `Proximal Policy Optimization (PPO) `__
+.. code-block:: python
- - `Soft Actor Critic (SAC) `__
+ { 'action_logp': np.ndarray((200,), dtype=float32, min=-0.701, max=-0.685, mean=-0.694),
+ 'actions': np.ndarray((200,), dtype=int64, min=0.0, max=1.0, mean=0.495),
+ 'dones': np.ndarray((200,), dtype=bool, min=0.0, max=1.0, mean=0.055),
+ 'infos': np.ndarray((200,), dtype=object, head={}),
+ 'new_obs': np.ndarray((200, 4), dtype=float32, min=-2.46, max=2.259, mean=0.018),
+ 'obs': np.ndarray((200, 4), dtype=float32, min=-2.46, max=2.259, mean=0.016),
+ 'rewards': np.ndarray((200,), dtype=float32, min=1.0, max=1.0, mean=1.0),
+ 't': np.ndarray((200,), dtype=int64, min=0.0, max=34.0, mean=9.14)}
-* Derivative-free
+In `multi-agent mode `__, sample batches are collected separately for each individual policy.
- - `Augmented Random Search (ARS) `__
+Training
+~~~~~~~~
- - `Evolution Strategies `__
+Policies each define a ``learn_on_batch()`` method that improves the policy given a sample batch of input. For TF and Torch policies, this is implemented using a `loss function` that takes as input sample batch tensors and outputs a scalar loss. Here are a few example loss functions:
-* Multi-agent specific
+- Simple `policy gradient loss `__
+- Simple `Q-function loss `__
+- Importance-weighted `APPO surrogate loss `__
- - `QMIX Monotonic Value Factorisation (QMIX, VDN, IQN) `__
- - `Multi-Agent Deep Deterministic Policy Gradient (contrib/MADDPG) `__
+RLlib `Trainer classes `__ coordinate the distributed workflow of running rollouts and optimizing policies. They do this by leveraging `policy optimizers `__ that implement the desired computation pattern (i.e., synchronous or asynchronous sampling, distributed replay, etc):
-* Offline
+.. figure:: a2c-arch.svg
- - `Advantage Re-Weighted Imitation Learning (MARWIL) `__
+ Synchronous Sampling (e.g., A2C, PG, PPO)
-Offline Datasets
-----------------
-* `Working with Offline Datasets `__
-* `Input Pipeline for Supervised Losses `__
-* `Input API `__
-* `Output API `__
+.. figure:: dqn-arch.svg
-Concepts and Custom Algorithms
-------------------------------
-* `Policies `__
+ Synchronous Replay (e.g., DQN, DDPG, TD3)
- - `Policies in Multi-Agent `__
+.. figure:: impala-arch.svg
- - `Building Policies in TensorFlow `__
+ Asynchronous Sampling (e.g., IMPALA, APPO)
- - `Building Policies in TensorFlow Eager `__
+.. figure:: apex-arch.svg
- - `Building Policies in PyTorch `__
+ Asynchronous Replay (e.g., Ape-X)
- - `Extending Existing Policies `__
+RLlib uses `Ray actors `__ to scale these architectures from a single core to many thousands of cores in a cluster. You can `configure the parallelism `__ used for training by changing the ``num_workers`` parameter.
-* `Policy Evaluation `__
-* `Policy Optimization `__
-* `Trainers `__
+Customization
+~~~~~~~~~~~~~
-Examples
---------
+RLlib provides ways to customize almost all aspects of training, including the `environment `__, `neural network model `__, `action distribution `__, and `policy definitions `__:
-* `Tuned Examples `__
-* `Training Workflows `__
-* `Custom Envs and Models `__
-* `Serving and Offline `__
-* `Multi-Agent and Hierarchical `__
-* `Community Examples `__
+.. image:: rllib-components.svg
-Development
------------
-
-* `Development Install `__
-* `API Stability `__
-* `Features `__
-* `Benchmarks `__
-* `Contributing Algorithms `__
-
-Package Reference
------------------
-* `ray.rllib.agents `__
-* `ray.rllib.env `__
-* `ray.rllib.evaluation `__
-* `ray.rllib.models `__
-* `ray.rllib.optimizers `__
-* `ray.rllib.utils `__
-
-Troubleshooting
----------------
-
-If you encounter errors like
-`blas_thread_init: pthread_create: Resource temporarily unavailable` when using many workers,
-try setting ``OMP_NUM_THREADS=1``. Similarly, check configured system limits with
-`ulimit -a` for other resource limit errors.
-
-If you encounter out-of-memory errors, consider setting ``redis_max_memory`` and ``object_store_memory`` in ``ray.init()`` to reduce memory usage.
-
-For debugging unexpected hangs or performance problems, you can run ``ray stack`` to dump
-the stack traces of all Ray workers on the current node, and ``ray timeline`` to dump
-a timeline visualization of tasks to a file.
+To learn more, proceed to the `table of contents `__.
diff --git a/rllib/policy/sample_batch.py b/rllib/policy/sample_batch.py
index a9515eeea..01c1f9b8e 100644
--- a/rllib/policy/sample_batch.py
+++ b/rllib/policy/sample_batch.py
@@ -14,77 +14,6 @@ from ray.rllib.utils.memory import concat_aligned
DEFAULT_POLICY_ID = "default_policy"
-@PublicAPI
-class MultiAgentBatch(object):
- """A batch of experiences from multiple policies in the environment.
-
- Attributes:
- policy_batches (dict): Mapping from policy id to a normal SampleBatch
- of experiences. Note that these batches may be of different length.
- count (int): The number of timesteps in the environment this batch
- contains. This will be less than the number of transitions this
- batch contains across all policies in total.
- """
-
- @PublicAPI
- def __init__(self, policy_batches, count):
- self.policy_batches = policy_batches
- self.count = count
-
- @staticmethod
- @PublicAPI
- def wrap_as_needed(batches, count):
- if len(batches) == 1 and DEFAULT_POLICY_ID in batches:
- return batches[DEFAULT_POLICY_ID]
- return MultiAgentBatch(batches, count)
-
- @staticmethod
- @PublicAPI
- def concat_samples(samples):
- policy_batches = collections.defaultdict(list)
- total_count = 0
- for s in samples:
- assert isinstance(s, MultiAgentBatch)
- for policy_id, batch in s.policy_batches.items():
- policy_batches[policy_id].append(batch)
- total_count += s.count
- out = {}
- for policy_id, batches in policy_batches.items():
- out[policy_id] = SampleBatch.concat_samples(batches)
- return MultiAgentBatch(out, total_count)
-
- @PublicAPI
- def copy(self):
- return MultiAgentBatch(
- {k: v.copy()
- for (k, v) in self.policy_batches.items()}, self.count)
-
- @PublicAPI
- def total(self):
- ct = 0
- for batch in self.policy_batches.values():
- ct += batch.count
- return ct
-
- @DeveloperAPI
- def compress(self, bulk=False, columns=frozenset(["obs", "new_obs"])):
- for batch in self.policy_batches.values():
- batch.compress(bulk=bulk, columns=columns)
-
- @DeveloperAPI
- def decompress_if_needed(self, columns=frozenset(["obs", "new_obs"])):
- for batch in self.policy_batches.values():
- batch.decompress_if_needed(columns)
-
- def __str__(self):
- return "MultiAgentBatch({}, count={})".format(
- str(self.policy_batches), self.count)
-
- def __repr__(self):
- return "MultiAgentBatch({}, count={})".format(
- str(self.policy_batches), self.count)
-
-
@PublicAPI
class SampleBatch(object):
"""Wrapper around a dictionary with string keys and array-like values.
@@ -294,3 +223,74 @@ class SampleBatch(object):
def __contains__(self, x):
return x in self.data
+
+
+@PublicAPI
+class MultiAgentBatch(object):
+ """A batch of experiences from multiple policies in the environment.
+
+ Attributes:
+ policy_batches (dict): Mapping from policy id to a normal SampleBatch
+ of experiences. Note that these batches may be of different length.
+ count (int): The number of timesteps in the environment this batch
+ contains. This will be less than the number of transitions this
+ batch contains across all policies in total.
+ """
+
+ @PublicAPI
+ def __init__(self, policy_batches, count):
+ self.policy_batches = policy_batches
+ self.count = count
+
+ @staticmethod
+ @PublicAPI
+ def wrap_as_needed(batches, count):
+ if len(batches) == 1 and DEFAULT_POLICY_ID in batches:
+ return batches[DEFAULT_POLICY_ID]
+ return MultiAgentBatch(batches, count)
+
+ @staticmethod
+ @PublicAPI
+ def concat_samples(samples):
+ policy_batches = collections.defaultdict(list)
+ total_count = 0
+ for s in samples:
+ assert isinstance(s, MultiAgentBatch)
+ for policy_id, batch in s.policy_batches.items():
+ policy_batches[policy_id].append(batch)
+ total_count += s.count
+ out = {}
+ for policy_id, batches in policy_batches.items():
+ out[policy_id] = SampleBatch.concat_samples(batches)
+ return MultiAgentBatch(out, total_count)
+
+ @PublicAPI
+ def copy(self):
+ return MultiAgentBatch(
+ {k: v.copy()
+ for (k, v) in self.policy_batches.items()}, self.count)
+
+ @PublicAPI
+ def total(self):
+ ct = 0
+ for batch in self.policy_batches.values():
+ ct += batch.count
+ return ct
+
+ @DeveloperAPI
+ def compress(self, bulk=False, columns=frozenset(["obs", "new_obs"])):
+ for batch in self.policy_batches.values():
+ batch.compress(bulk=bulk, columns=columns)
+
+ @DeveloperAPI
+ def decompress_if_needed(self, columns=frozenset(["obs", "new_obs"])):
+ for batch in self.policy_batches.values():
+ batch.decompress_if_needed(columns)
+
+ def __str__(self):
+ return "MultiAgentBatch({}, count={})".format(
+ str(self.policy_batches), self.count)
+
+ def __repr__(self):
+ return "MultiAgentBatch({}, count={})".format(
+ str(self.policy_batches), self.count)