wassname/ray
1
0
Fork 0
mirror of https://github.com/wassname/ray.git synced 2026-07-09 11:43:56 +08:00
Code Issues Packages Projects Releases Wiki Activity

[rllib] Misc fixes: set lr for PG, better error message for LSTM/PPO, fix multi-agent/APEX (#3697)

Browse Source 
* fix

* update test

* better error

* compute

* eps fix

* add get_policy() api

* Update agent.py

* better err msg

* fix

* pass in rew
This commit is contained in:
Eric Liang
2019-01-06 19:37:35 -08:00
committed by GitHub
parent df0733cafb
commit e78562b2e8
12 changed files with 111 additions and 34 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/source/rllib-training.rst
+7 -9
View File
@@ -179,23 +179,21 @@ Accessing Policy State
~~~~~~~~~~~~~~~~~~~~~~
It is common to need to access an agent's internal state, e.g., to set or get internal weights. In RLlib an agent's state is replicated across multiple *policy evaluators* (Ray actors) in the cluster. However, you can easily get and update this state between calls to ``train()`` via ``agent.optimizer.foreach_evaluator()`` or ``agent.optimizer.foreach_evaluator_with_index()``. These functions take a lambda function that is applied with the evaluator as an arg. You can also return values from these functions and those will be returned as a list.
You can also access just the "master" copy of the agent state through ``agent.local_evaluator``, but note that updates here may not be immediately reflected in remote replicas if you have configured ``num_workers > 0``. For example, to access the weights of a local TF policy, you can run ``agent.local_evaluator.policy_map["default"].get_weights()``. This is also equivalent to ``agent.local_evaluator.for_policy(lambda p: p.get_weights())``:
You can also access just the "master" copy of the agent state through ``agent.get_policy()`` or ``agent.local_evaluator``, but note that updates here may not be immediately reflected in remote replicas if you have configured ``num_workers > 0``. For example, to access the weights of a local TF policy, you can run ``agent.get_policy().get_weights()``. This is also equivalent to ``agent.local_evaluator.policy_map["default"].get_weights()``:
.. code-block:: python
# Get weights of the local policy
# Get weights of the default local policy
agent.get_policy().get_weights()
# Same as above
agent.local_evaluator.policy_map["default"].get_weights()
# Same as above
agent.local_evaluator.for_policy(lambda p: p.get_weights())
# Get list of weights of each evaluator, including remote replicas
agent.optimizer.foreach_evaluator(
lambda ev: ev.for_policy(lambda p: p.get_weights()))
agent.optimizer.foreach_evaluator(lambda ev: ev.get_policy().get_weights())
# Same as above
agent.optimizer.foreach_evaluator_with_index(
lambda ev, i: ev.for_policy(lambda p: p.get_weights()))
agent.optimizer.foreach_evaluator_with_index(lambda ev, i: ev.get_policy().get_weights())
Global Coordination
~~~~~~~~~~~~~~~~~~~