mirror of
https://github.com/wassname/ray.git
synced 2026-08-16 10:34:09 +08:00
[rllib] Adds eager support with a generic TFEagerPolicy class (#5436)
This commit is contained in:
gehring
File diff suppressed because one or more lines are too long
|
Before Width: | Height: | Size: 101 KiB After Width: | Height: | Size: 102 KiB |
@@ -120,12 +120,14 @@ To start, you first have to define a loss function. In RLlib, loss functions are
|
||||
import tensorflow as tf
|
||||
from ray.rllib.policy.sample_batch import SampleBatch
|
||||
|
||||
def policy_gradient_loss(policy, batch_tensors):
|
||||
actions = batch_tensors[SampleBatch.ACTIONS]
|
||||
rewards = batch_tensors[SampleBatch.REWARDS]
|
||||
return -tf.reduce_mean(policy.action_dist.logp(actions) * rewards)
|
||||
def policy_gradient_loss(policy, model, dist_class, train_batch):
|
||||
actions = train_batch[SampleBatch.ACTIONS]
|
||||
rewards = train_batch[SampleBatch.REWARDS]
|
||||
logits, _ = model.from_batch(train_batch)
|
||||
action_dist = dist_class(logits, model)
|
||||
return -tf.reduce_mean(action_dist.logp(actions) * rewards)
|
||||
|
||||
In the above snippet, ``actions`` is a Tensor placeholder of shape ``[batch_size, action_dim...]``, and ``rewards`` is a placeholder of shape ``[batch_size]``. The ``policy.action_dist`` object is an `ActionDistribution <rllib-package-ref.html#ray.rllib.models.ActionDistribution>`__ that represents the output of the neural network policy model. Passing this loss function to ``build_tf_policy`` is enough to produce a very basic TF policy:
|
||||
In the above snippet, ``actions`` is a Tensor placeholder of shape ``[batch_size, action_dim...]``, and ``rewards`` is a placeholder of shape ``[batch_size]``. The ``action_dist`` object is an `ActionDistribution <rllib-package-ref.html#ray.rllib.models.ActionDistribution>`__ that is parameterized by the output of the neural network policy model. Passing this loss function to ``build_tf_policy`` is enough to produce a very basic TF policy:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
@@ -181,10 +183,12 @@ Let's modify our policy loss to include rewards summed over time. To enable this
|
||||
return compute_advantages(
|
||||
sample_batch, 0.0, policy.config["gamma"], use_gae=False)
|
||||
|
||||
def policy_gradient_loss(policy, batch_tensors):
|
||||
actions = batch_tensors[SampleBatch.ACTIONS]
|
||||
advantages = batch_tensors[Postprocessing.ADVANTAGES]
|
||||
return -tf.reduce_mean(policy.action_dist.logp(actions) * advantages)
|
||||
def policy_gradient_loss(policy, model, dist_class, train_batch):
|
||||
logits, _ = model.from_batch(train_batch)
|
||||
action_dist = dist_class(logits, model)
|
||||
return -tf.reduce_mean(
|
||||
action_dist.logp(train_batch[SampleBatch.ACTIONS]) *
|
||||
train_batch[Postprocessing.ADVANTAGES])
|
||||
|
||||
MyTFPolicy = build_tf_policy(
|
||||
name="MyTFPolicy",
|
||||
@@ -193,7 +197,7 @@ Let's modify our policy loss to include rewards summed over time. To enable this
|
||||
|
||||
The ``postprocess_advantages()`` function above uses calls RLlib's ``compute_advantages`` function to compute advantages for each timestep. If you re-run the trainer with this improved policy, you'll find that it quickly achieves the max reward of 200.
|
||||
|
||||
You might be wondering how RLlib makes the advantages placeholder automatically available as ``batch_tensors[Postprocessing.ADVANTAGES]``. When building your policy, RLlib will create a "dummy" trajectory batch where all observations, actions, rewards, etc. are zeros. It then calls your ``postprocess_fn``, and generates TF placeholders based on the numpy shapes of the postprocessed batch. RLlib tracks which placeholders that ``loss_fn`` and ``stats_fn`` access, and then feeds the corresponding sample data into those placeholders during loss optimization. You can also access these placeholders via ``policy.get_placeholder(<name>)`` after loss initialization.
|
||||
You might be wondering how RLlib makes the advantages placeholder automatically available as ``train_batch[Postprocessing.ADVANTAGES]``. When building your policy, RLlib will create a "dummy" trajectory batch where all observations, actions, rewards, etc. are zeros. It then calls your ``postprocess_fn``, and generates TF placeholders based on the numpy shapes of the postprocessed batch. RLlib tracks which placeholders that ``loss_fn`` and ``stats_fn`` access, and then feeds the corresponding sample data into those placeholders during loss optimization. You can also access these placeholders via ``policy.get_placeholder(<name>)`` after loss initialization.
|
||||
|
||||
**Example 1: Proximal Policy Optimization**
|
||||
|
||||
@@ -290,9 +294,9 @@ The ``update_kl`` method on the policy is defined in `PPOTFPolicy <https://githu
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
def kl_and_loss_stats(policy, batch_tensors):
|
||||
def kl_and_loss_stats(policy, train_batch):
|
||||
policy.explained_variance = explained_variance(
|
||||
batch_tensors[Postprocessing.VALUE_TARGETS], policy.value_function)
|
||||
train_batch[Postprocessing.VALUE_TARGETS], policy.model.value_function())
|
||||
|
||||
stats_fetches = {
|
||||
"cur_kl_coeff": policy.kl_coeff,
|
||||
@@ -307,14 +311,14 @@ The ``update_kl`` method on the policy is defined in `PPOTFPolicy <https://githu
|
||||
|
||||
return stats_fetches
|
||||
|
||||
``extra_actions_fetches_fn``: This function defines extra outputs that will be recorded when generating actions with the policy. For example, this enables saving the raw policy logits in the experience batch, which e.g. means it can be referenced in the PPO loss function via ``batch_tensors[BEHAVIOUR_LOGITS]``. Other values such as the current value prediction can also be emitted for debugging or optimization purposes:
|
||||
``extra_actions_fetches_fn``: This function defines extra outputs that will be recorded when generating actions with the policy. For example, this enables saving the raw policy logits in the experience batch, which e.g. means it can be referenced in the PPO loss function via ``batch[BEHAVIOUR_LOGITS]``. Other values such as the current value prediction can also be emitted for debugging or optimization purposes:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
def vf_preds_and_logits_fetches(policy):
|
||||
return {
|
||||
SampleBatch.VF_PREDS: policy.value_function,
|
||||
BEHAVIOUR_LOGITS: policy.model.outputs,
|
||||
SampleBatch.VF_PREDS: policy.model.value_function(),
|
||||
BEHAVIOUR_LOGITS: policy.model.last_output(),
|
||||
}
|
||||
|
||||
``gradients_fn``: If defined, this function returns TF gradients for the loss function. You'd typically only want to override this to apply transformations such as gradient clipping:
|
||||
@@ -323,12 +327,10 @@ The ``update_kl`` method on the policy is defined in `PPOTFPolicy <https://githu
|
||||
|
||||
def clip_gradients(policy, optimizer, loss):
|
||||
if policy.config["grad_clip"] is not None:
|
||||
policy.var_list = tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES,
|
||||
tf.get_variable_scope().name)
|
||||
grads = tf.gradients(loss, policy.var_list)
|
||||
grads = tf.gradients(loss, policy.model.trainable_variables())
|
||||
policy.grads, _ = tf.clip_by_global_norm(grads,
|
||||
policy.config["grad_clip"])
|
||||
clipped_grads = list(zip(policy.grads, policy.var_list))
|
||||
clipped_grads = list(zip(policy.grads, policy.model.trainable_variables()))
|
||||
return clipped_grads
|
||||
else:
|
||||
return optimizer.compute_gradients(
|
||||
@@ -416,31 +418,11 @@ Finally, note that you do not have to use ``build_tf_policy`` to define a Tensor
|
||||
Building Policies in TensorFlow Eager
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
While RLlib runs all TF operations in graph mode, you can still leverage TensorFlow eager using `tf.py_function <https://www.tensorflow.org/api_docs/python/tf/py_function>`__. However, note that eager and non-eager tensors cannot be mixed within the ``py_function``. Here's an example of embedding eager execution within a policy loss function:
|
||||
Policies built with ``build_tf_policy`` (most of the reference algorithms are) can be run in eager mode by setting the ``"eager": True`` config option or using ``rllib train --eager``. This will tell RLlib to execute the model forward pass, action distribution, loss, and stats functions in eager mode.
|
||||
|
||||
.. code-block:: python
|
||||
Eager mode makes debugging much easier, since you can now use normal Python functions such as ``print()`` to inspect intermediate tensor values. However, it is slower than graph mode.
|
||||
|
||||
def eager_loss(policy, batch_tensors):
|
||||
"""Example of using embedded eager execution in a custom loss.
|
||||
|
||||
Here `compute_penalty` prints the actions and rewards for debugging, and
|
||||
also computes a (dummy) penalty term to add to the loss.
|
||||
"""
|
||||
|
||||
def compute_penalty(actions, rewards):
|
||||
penalty = tf.reduce_mean(tf.cast(actions, tf.float32))
|
||||
if random.random() > 0.9:
|
||||
print("The eagerly computed penalty is", penalty, actions, rewards)
|
||||
return penalty
|
||||
|
||||
actions = batch_tensors[SampleBatch.ACTIONS]
|
||||
rewards = batch_tensors[SampleBatch.REWARDS]
|
||||
penalty = tf.py_function(
|
||||
compute_penalty, [actions, rewards], Tout=tf.float32)
|
||||
|
||||
return penalty - tf.reduce_mean(policy.action_dist.logp(actions) * rewards)
|
||||
|
||||
You can find a runnable file for the above eager execution example `here <https://github.com/ray-project/ray/blob/master/rllib/examples/eager_execution.py>`__.
|
||||
You can also selectively leverage eager operations within graph mode execution with `tf.py_function <https://www.tensorflow.org/api_docs/python/tf/py_function>`__. Here's an example of using eager ops embedded `within a loss function <https://github.com/ray-project/ray/blob/master/rllib/examples/eager_execution.py>`__.
|
||||
|
||||
Building Policies in PyTorch
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
@@ -452,13 +434,11 @@ Defining a policy in PyTorch is quite similar to that for TensorFlow (and the pr
|
||||
from ray.rllib.policy.sample_batch import SampleBatch
|
||||
from ray.rllib.policy.torch_policy_template import build_torch_policy
|
||||
|
||||
def policy_gradient_loss(policy, batch_tensors):
|
||||
logits, _, values, _ = policy.model({
|
||||
SampleBatch.CUR_OBS: batch_tensors[SampleBatch.CUR_OBS]
|
||||
}, [])
|
||||
action_dist = policy.dist_class(logits)
|
||||
log_probs = action_dist.logp(batch_tensors[SampleBatch.ACTIONS])
|
||||
return -batch_tensors[SampleBatch.REWARDS].dot(log_probs)
|
||||
def policy_gradient_loss(policy, model, dist_class, train_batch):
|
||||
logits, _ = model.from_batch(train_batch)
|
||||
action_dist = dist_class(logits)
|
||||
log_probs = action_dist.logp(train_batch[SampleBatch.ACTIONS])
|
||||
return -train_batch[SampleBatch.REWARDS].dot(log_probs)
|
||||
|
||||
# <class 'ray.rllib.policy.torch_policy_template.MyTorchPolicy'>
|
||||
MyTorchPolicy = build_torch_policy(
|
||||
@@ -480,17 +460,16 @@ Now, building on the TF examples above, let's look at how the `A3C torch policy
|
||||
optimizer_fn=torch_optimizer,
|
||||
mixins=[ValueNetworkMixin])
|
||||
|
||||
``loss_fn``: Similar to the TF example, the actor critic loss is defined over ``batch_tensors``. We imperatively execute the forward pass by calling ``policy.model()`` on the observations followed by ``policy.dist_class()`` on the output logits. The output Tensors are saved as attributes of the policy object (e.g., ``policy.entropy = dist.entropy.mean()``), and we return the scalar loss:
|
||||
``loss_fn``: Similar to the TF example, the actor critic loss is defined over ``batch``. We imperatively execute the forward pass by calling ``model()`` on the observations followed by ``dist_class()`` on the output logits. The output Tensors are saved as attributes of the policy object (e.g., ``policy.entropy = dist.entropy.mean()``), and we return the scalar loss:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
def actor_critic_loss(policy, batch_tensors):
|
||||
logits, _, values, _ = policy.model({
|
||||
SampleBatch.CUR_OBS: batch_tensors[SampleBatch.CUR_OBS]
|
||||
}, [])
|
||||
dist = policy.dist_class(logits)
|
||||
log_probs = dist.logp(batch_tensors[SampleBatch.ACTIONS])
|
||||
policy.entropy = dist.entropy().mean()
|
||||
def actor_critic_loss(policy, model, dist_class, train_batch):
|
||||
logits, _ = model.from_batch(train_batch)
|
||||
values = model.value_function()
|
||||
action_dist = dist_class(logits)
|
||||
log_probs = action_dist.logp(train_batch[SampleBatch.ACTIONS])
|
||||
policy.entropy = action_dist.entropy().mean()
|
||||
...
|
||||
return overall_err
|
||||
|
||||
@@ -498,19 +477,19 @@ Now, building on the TF examples above, let's look at how the `A3C torch policy
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
def loss_and_entropy_stats(policy, batch_tensors):
|
||||
def loss_and_entropy_stats(policy, train_batch):
|
||||
return {
|
||||
"policy_entropy": policy.entropy.item(),
|
||||
"policy_loss": policy.pi_err.item(),
|
||||
"vf_loss": policy.value_err.item(),
|
||||
}
|
||||
|
||||
``extra_action_out_fn``: We save value function predictions given model outputs. This makes the value function predictions of the model available in the trajectory as ``batch_tensors[SampleBatch.VF_PREDS]``:
|
||||
``extra_action_out_fn``: We save value function predictions given model outputs. This makes the value function predictions of the model available in the trajectory as ``batch[SampleBatch.VF_PREDS]``:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
def model_value_predictions(policy, input_dict, state_batches, model_out):
|
||||
return {SampleBatch.VF_PREDS: model_out[2].cpu().numpy()}
|
||||
def model_value_predictions(policy, input_dict, state_batches, model):
|
||||
return {SampleBatch.VF_PREDS: model.value_function().cpu().numpy()}
|
||||
|
||||
``postprocess_fn`` and ``mixins``: Similar to the PPO example, we need access to the value function during postprocessing (i.e., ``add_advantages`` below calls ``policy._value()``. The value function is exposed through a mixin class that defines the method:
|
||||
|
||||
@@ -537,7 +516,7 @@ Now, building on the TF examples above, let's look at how the `A3C torch policy
|
||||
|
||||
You can find the full policy definition in `a3c_torch_policy.py <https://github.com/ray-project/ray/blob/master/rllib/agents/a3c/a3c_torch_policy.py>`__.
|
||||
|
||||
In summary, the main differences between the PyTorch and TensorFlow policy builder functions is that the TF loss and stats functions are built symbolically when the policy is initialized, whereas for PyTorch these functions are called imperatively each time they are used.
|
||||
In summary, the main differences between the PyTorch and TensorFlow policy builder functions is that the TF loss and stats functions are built symbolically when the policy is initialized, whereas for PyTorch (or TensorFlow Eager) these functions are called imperatively each time they are used.
|
||||
|
||||
Extending Existing Policies
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
+33
-33
@@ -231,39 +231,6 @@ The `rock_paper_scissors_multiagent.py <https://github.com/ray-project/ray/blob/
|
||||
|
||||
TensorBoard output of running the rock-paper-scissors example, where a learned policy faces off between a random selection of the same-move and beat-last-move heuristics. Here the performance of heuristic policies vs the learned policy is compared with LSTM enabled (blue) and a plain feed-forward policy (red). While the feedforward policy can easily beat the same-move heuristic by simply avoiding the last move taken, it takes a LSTM policy to distinguish between and consistently beat both policies.
|
||||
|
||||
Hierarchical Environments
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Hierarchical training can sometimes be implemented as a special case of multi-agent RL. For example, consider a three-level hierarchy of policies, where a top-level policy issues high level actions that are executed at finer timescales by a mid-level and low-level policy. The following timeline shows one step of the top-level policy, which corresponds to two mid-level actions and five low-level actions:
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
top_level ---------------------------------------------------------------> top_level --->
|
||||
mid_level_0 -------------------------------> mid_level_0 ----------------> mid_level_1 ->
|
||||
low_level_0 -> low_level_0 -> low_level_0 -> low_level_1 -> low_level_1 -> low_level_2 ->
|
||||
|
||||
This can be implemented as a multi-agent environment with three types of agents. Each higher-level action creates a new lower-level agent instance with a new id (e.g., ``low_level_0``, ``low_level_1``, ``low_level_2`` in the above example). These lower-level agents pop in existence at the start of higher-level steps, and terminate when their higher-level action ends. Their experiences are aggregated by policy, so from RLlib's perspective it's just optimizing three different types of policies. The configuration might look something like this:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
"multiagent": {
|
||||
"policies": {
|
||||
"top_level": (custom_policy or None, ...),
|
||||
"mid_level": (custom_policy or None, ...),
|
||||
"low_level": (custom_policy or None, ...),
|
||||
},
|
||||
"policy_mapping_fn":
|
||||
lambda agent_id:
|
||||
"low_level" if agent_id.startswith("low_level_") else
|
||||
"mid_level" if agent_id.startswith("mid_level_") else "top_level"
|
||||
"policies_to_train": ["top_level"],
|
||||
},
|
||||
|
||||
|
||||
In this setup, the appropriate rewards for training lower-level agents must be provided by the multi-agent env implementation. The environment class is also responsible for routing between the agents, e.g., conveying `goals <https://arxiv.org/pdf/1703.01161.pdf>`__ from higher-level agents to lower-level agents as part of the lower-level agent observation.
|
||||
|
||||
See this file for a runnable example: `hierarchical_training.py <https://github.com/ray-project/ray/blob/master/rllib/examples/hierarchical_training.py>`__.
|
||||
|
||||
Variable-Sharing Between Policies
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
@@ -323,6 +290,39 @@ It is common to have groups of agents in multi-agent RL. RLlib treats agent grou
|
||||
|
||||
For environments with multiple groups, or mixtures of agent groups and individual agents, you can use grouping in conjunction with the policy mapping API described in prior sections.
|
||||
|
||||
Hierarchical Environments
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Hierarchical training can sometimes be implemented as a special case of multi-agent RL. For example, consider a three-level hierarchy of policies, where a top-level policy issues high level actions that are executed at finer timescales by a mid-level and low-level policy. The following timeline shows one step of the top-level policy, which corresponds to two mid-level actions and five low-level actions:
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
top_level ---------------------------------------------------------------> top_level --->
|
||||
mid_level_0 -------------------------------> mid_level_0 ----------------> mid_level_1 ->
|
||||
low_level_0 -> low_level_0 -> low_level_0 -> low_level_1 -> low_level_1 -> low_level_2 ->
|
||||
|
||||
This can be implemented as a multi-agent environment with three types of agents. Each higher-level action creates a new lower-level agent instance with a new id (e.g., ``low_level_0``, ``low_level_1``, ``low_level_2`` in the above example). These lower-level agents pop in existence at the start of higher-level steps, and terminate when their higher-level action ends. Their experiences are aggregated by policy, so from RLlib's perspective it's just optimizing three different types of policies. The configuration might look something like this:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
"multiagent": {
|
||||
"policies": {
|
||||
"top_level": (custom_policy or None, ...),
|
||||
"mid_level": (custom_policy or None, ...),
|
||||
"low_level": (custom_policy or None, ...),
|
||||
},
|
||||
"policy_mapping_fn":
|
||||
lambda agent_id:
|
||||
"low_level" if agent_id.startswith("low_level_") else
|
||||
"mid_level" if agent_id.startswith("mid_level_") else "top_level"
|
||||
"policies_to_train": ["top_level"],
|
||||
},
|
||||
|
||||
|
||||
In this setup, the appropriate rewards for training lower-level agents must be provided by the multi-agent env implementation. The environment class is also responsible for routing between the agents, e.g., conveying `goals <https://arxiv.org/pdf/1703.01161.pdf>`__ from higher-level agents to lower-level agents as part of the lower-level agent observation.
|
||||
|
||||
See this file for a runnable example: `hierarchical_training.py <https://github.com/ray-project/ray/blob/master/rllib/examples/hierarchical_training.py>`__.
|
||||
|
||||
Interfacing with External Agents
|
||||
--------------------------------
|
||||
|
||||
|
||||
@@ -14,7 +14,7 @@ You can train a simple DQN trainer with the following command:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
rllib train --run DQN --env CartPole-v0
|
||||
rllib train --run DQN --env CartPole-v0 # add --eager for eager execution
|
||||
|
||||
By default, the results will be logged to a subdirectory of ``~/ray_results``.
|
||||
This subdirectory will contain a file ``params.json`` which contains the
|
||||
@@ -122,6 +122,7 @@ Here is an example of the basic usage (for a more complete example, see `custom_
|
||||
config = ppo.DEFAULT_CONFIG.copy()
|
||||
config["num_gpus"] = 0
|
||||
config["num_workers"] = 1
|
||||
config["eager"] = False
|
||||
trainer = ppo.PPOTrainer(config=config, env="CartPole-v0")
|
||||
|
||||
# Can optionally call trainer.restore(path) to load a checkpoint.
|
||||
@@ -156,6 +157,7 @@ All RLlib trainers are compatible with the `Tune API <tune-usage.html>`__. This
|
||||
"num_gpus": 0,
|
||||
"num_workers": 1,
|
||||
"lr": tune.grid_search([0.01, 0.001, 0.0001]),
|
||||
"eager": False,
|
||||
},
|
||||
)
|
||||
|
||||
@@ -370,7 +372,9 @@ The ``"monitor": true`` config can be used to save Gym episode videos to the res
|
||||
TensorFlow Eager
|
||||
~~~~~~~~~~~~~~~~
|
||||
|
||||
While RLlib uses TF graph mode for all computations, you can still leverage TF eager to inspect the intermediate state of computations using `tf.py_function <https://www.tensorflow.org/api_docs/python/tf/py_function>`__. Here's an example of using eager mode in `a custom RLlib model and loss <https://github.com/ray-project/ray/blob/master/rllib/examples/eager_execution.py>`__.
|
||||
Policies built with ``build_tf_policy`` can be also run in eager mode by setting the ``"eager": True`` config option or using ``rllib train --eager``. This will tell RLlib to execute the model forward pass, action distribution, loss, and stats functions in eager mode.
|
||||
|
||||
Eager mode makes debugging much easier, since you can now use normal Python functions such as ``print()`` to inspect intermediate tensor values. However, it is slower than graph mode.
|
||||
|
||||
Episode Traces
|
||||
~~~~~~~~~~~~~~
|
||||
|
||||
+10
-21
@@ -1,7 +1,7 @@
|
||||
RLlib: Scalable Reinforcement Learning
|
||||
======================================
|
||||
|
||||
RLlib is an open-source library for reinforcement learning that offers both high scalability and a unified API for a variety of applications.
|
||||
RLlib is an open-source library for reinforcement learning that offers both high scalability and a unified API for a variety of applications. RLlib natively supports TensorFlow, TensorFlow Eager, and PyTorch, but most of its internals are framework agnostic.
|
||||
|
||||
.. image:: rllib-stack.svg
|
||||
|
||||
@@ -25,13 +25,13 @@ Then, you can try out training in the following equivalent ways:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
rllib train --run=PPO --env=CartPole-v0
|
||||
rllib train --run=PPO --env=CartPole-v0 # add --eager for eager execution
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from ray import tune
|
||||
from ray.rllib.agents.ppo import PPOTrainer
|
||||
tune.run(PPOTrainer, config={"env": "CartPole-v0"})
|
||||
tune.run(PPOTrainer, config={"env": "CartPole-v0"}) # "eager": True for eager execution
|
||||
|
||||
Next, we'll cover three key concepts in RLlib: Policies, Samples, and Trainers.
|
||||
|
||||
@@ -46,10 +46,11 @@ Policies can be implemented using `any framework <https://github.com/ray-project
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
def policy_gradient_loss(policy, batch_tensors):
|
||||
actions = batch_tensors[SampleBatch.ACTIONS]
|
||||
rewards = batch_tensors[SampleBatch.REWARDS]
|
||||
return -tf.reduce_mean(policy.action_dist.logp(actions) * rewards)
|
||||
def policy_gradient_loss(policy, model, dist_class, train_batch):
|
||||
logits, _ = model.from_batch(train_batch)
|
||||
action_dist = dist_class(logits, model)
|
||||
return -tf.reduce_mean(
|
||||
action_dist.logp(train_batch["actions"]) * train_batch["rewards"])
|
||||
|
||||
# <class 'ray.rllib.policy.tf_policy_template.MyTFPolicy'>
|
||||
MyTFPolicy = build_tf_policy(
|
||||
@@ -85,25 +86,13 @@ Policies each define a ``learn_on_batch()`` method that improves the policy give
|
||||
- Simple `Q-function loss <https://github.com/ray-project/ray/blob/a1d2e1762325cd34e14dc411666d63bb15d6eaf0/rllib/agents/dqn/simple_q_policy.py#L136>`__
|
||||
- Importance-weighted `APPO surrogate loss <https://github.com/ray-project/ray/blob/master/rllib/agents/ppo/appo_policy.py>`__
|
||||
|
||||
RLlib `Trainer classes <rllib-concepts.html#trainers>`__ coordinate the distributed workflow of running rollouts and optimizing policies. They do this by leveraging `policy optimizers <rllib-concepts.html#policy-optimization>`__ that implement the desired computation pattern (i.e., synchronous or asynchronous sampling, distributed replay, etc):
|
||||
RLlib `Trainer classes <rllib-concepts.html#trainers>`__ coordinate the distributed workflow of running rollouts and optimizing policies. They do this by leveraging `policy optimizers <rllib-concepts.html#policy-optimization>`__ that implement the desired computation pattern. The following figure shows *synchronous sampling*, the simplest of `these patterns <rllib-algorithms.html>`__:
|
||||
|
||||
.. figure:: a2c-arch.svg
|
||||
|
||||
Synchronous Sampling (e.g., A2C, PG, PPO)
|
||||
|
||||
.. figure:: dqn-arch.svg
|
||||
|
||||
Synchronous Replay (e.g., DQN, DDPG, TD3)
|
||||
|
||||
.. figure:: impala-arch.svg
|
||||
|
||||
Asynchronous Sampling (e.g., IMPALA, APPO)
|
||||
|
||||
.. figure:: apex-arch.svg
|
||||
|
||||
Asynchronous Replay (e.g., Ape-X)
|
||||
|
||||
RLlib uses `Ray actors <actors.html>`__ to scale these architectures from a single core to many thousands of cores in a cluster. You can `configure the parallelism <rllib-training.html#specifying-resources>`__ used for training by changing the ``num_workers`` parameter.
|
||||
RLlib uses `Ray actors <actors.html>`__ to scale training from a single core to many thousands of cores in a cluster. You can `configure the parallelism <rllib-training.html#specifying-resources>`__ used for training by changing the ``num_workers`` parameter.
|
||||
|
||||
Customization
|
||||
~~~~~~~~~~~~~
|
||||
|
||||
Reference in New Issue
Block a user