mirror of
https://github.com/wassname/ray.git
synced 2026-07-09 04:00:44 +08:00
Eric Liang
7dee2c6735
## What do these changes do?
**Vectorized envs**: Users can either implement `VectorEnv`, or alternatively set `num_envs=N` to auto-vectorize gym envs (this vectorizes just the action computation part).
```
# CartPole-v0 on single core with 64x64 MLP:
# vector_width=1:
Actions per second 2720.1284458322966
# vector_width=8:
Actions per second 13773.035334888269
# vector_width=64:
Actions per second 37903.20472563333
```
**Async envs**: The more general form of `VectorEnv` is `AsyncVectorEnv`, which allows agents to execute out of lockstep. We use this as an adapter to support `ServingEnv`. Since we can convert any other form of env to `AsyncVectorEnv`, utils.sampler has been rewritten to run against this interface.
**Policy serving**: This provides an env which is not stepped. Rather, the env executes in its own thread, querying the policy for actions via `self.get_action(obs)`, and reporting results via `self.log_returns(rewards)`. We also support logging of off-policy actions via `self.log_action(obs, action)`. This is a more convenient API for some use cases, and also provides parallelizable support for policy serving (for example, if you start a HTTP server in the env) and ingest of offline logs (if the env reads from serving logs).
Any of these types of envs can be passed to RLlib agents. RLlib handles conversions internally in CommonPolicyEvaluator, for example:
```
gym.Env => rllib.VectorEnv => rllib.AsyncVectorEnv
rllib.ServingEnv => rllib.AsyncVectorEnv
```
153 lines
4.3 KiB
Python
153 lines
4.3 KiB
Python
from __future__ import absolute_import
|
|
from __future__ import division
|
|
from __future__ import print_function
|
|
|
|
import os
|
|
|
|
|
|
class PolicyEvaluator(object):
|
|
"""Algorithms implement this interface to leverage policy optimizers.
|
|
|
|
Policy evaluators are the "data plane" of an algorithm.
|
|
|
|
Any algorithm that implements Evaluator can plug in any PolicyOptimizer,
|
|
e.g. async SGD, Ape-X, local multi-GPU SGD, etc.
|
|
"""
|
|
|
|
def sample(self):
|
|
"""Returns a batch of experience sampled from this evaluator.
|
|
|
|
This method must be implemented by subclasses.
|
|
|
|
Returns:
|
|
SampleBatch: A columnar batch of experiences (e.g., tensors).
|
|
|
|
Examples:
|
|
>>> print(ev.sample())
|
|
SampleBatch({"obs": [1, 2, 3], "action": [0, 1, 0], ...})
|
|
"""
|
|
|
|
raise NotImplementedError
|
|
|
|
def compute_gradients(self, samples):
|
|
"""Returns a gradient computed w.r.t the specified samples.
|
|
|
|
This method must be implemented by subclasses.
|
|
|
|
Returns:
|
|
object: A gradient that can be applied on a compatible evaluator.
|
|
info: dictionary of extra metadata.
|
|
|
|
Examples:
|
|
>>> batch = ev.sample()
|
|
>>> grads, info = ev2.compute_gradients(samples)
|
|
"""
|
|
|
|
raise NotImplementedError
|
|
|
|
def apply_gradients(self, grads):
|
|
"""Applies the given gradients to this evaluator's weights.
|
|
|
|
This method must be implemented by subclasses.
|
|
|
|
Examples:
|
|
>>> samples = ev1.sample()
|
|
>>> grads, info = ev2.compute_gradients(samples)
|
|
>>> ev1.apply_gradients(grads)
|
|
"""
|
|
|
|
raise NotImplementedError
|
|
|
|
def get_weights(self):
|
|
"""Returns the model weights of this Evaluator.
|
|
|
|
This method must be implemented by subclasses.
|
|
|
|
Returns:
|
|
object: weights that can be set on a compatible evaluator.
|
|
info: dictionary of extra metadata.
|
|
|
|
Examples:
|
|
>>> weights = ev1.get_weights()
|
|
"""
|
|
|
|
raise NotImplementedError
|
|
|
|
def set_weights(self, weights):
|
|
"""Sets the model weights of this Evaluator.
|
|
|
|
This method must be implemented by subclasses.
|
|
|
|
Examples:
|
|
>>> weights = ev1.get_weights()
|
|
>>> ev2.set_weights(weights)
|
|
"""
|
|
|
|
raise NotImplementedError
|
|
|
|
def compute_apply(self, samples):
|
|
"""Fused compute gradients and apply gradients call.
|
|
|
|
Returns:
|
|
info: dictionary of extra metadata from compute_gradients().
|
|
|
|
Examples:
|
|
>>> batch = ev.sample()
|
|
>>> ev.compute_apply(samples)
|
|
"""
|
|
|
|
grads, info = self.compute_gradients(samples)
|
|
self.apply_gradients(grads)
|
|
return info
|
|
|
|
def get_host(self):
|
|
"""Returns the hostname of the process running this evaluator."""
|
|
|
|
return os.uname()[1]
|
|
|
|
def apply(self, func, *args):
|
|
"""Apply the given function to this evaluator instance."""
|
|
|
|
return func(self, *args)
|
|
|
|
|
|
class TFMultiGPUSupport(PolicyEvaluator):
|
|
"""The multi-GPU TF optimizer requires additional TF-specific support.
|
|
|
|
Attributes:
|
|
sess (Session): the tensorflow session associated with this evaluator.
|
|
"""
|
|
|
|
def tf_loss_inputs(self):
|
|
"""Returns a list of the input placeholders required for the loss.
|
|
|
|
For example, the following calls should work:
|
|
|
|
Returns:
|
|
list: a (name, placeholder) tuple for each loss input argument.
|
|
Each placeholder name must correspond to one of the SampleBatch
|
|
column keys returned by sample().
|
|
|
|
Examples:
|
|
>>> print(ev.tf_loss_inputs())
|
|
[("action", action_placeholder), ("reward", reward_placeholder)]
|
|
|
|
>>> print(ev.sample()[0].data.keys())
|
|
["action", "reward"]
|
|
"""
|
|
|
|
raise NotImplementedError
|
|
|
|
def build_tf_loss(self, input_placeholders):
|
|
"""Returns a new loss tensor graph for the specified inputs.
|
|
|
|
The graph must share vars with this Evaluator's policy model, so that
|
|
the multi-gpu optimizer can update the weights.
|
|
|
|
Examples:
|
|
>>> loss_inputs = ev.tf_loss_inputs()
|
|
>>> ev.build_tf_loss([ph for _, ph in loss_inputs])
|
|
"""
|
|
|
|
raise NotImplementedError
|