[rllib] Document "v2" APIs (#2316)

* re * wip * wip * a3c working * torch support * pg works * lint * rm v2 * consumer id * clean up pg * clean up more * fix python 2.7 * tf session management * docs * dqn wip * fix compile * dqn * apex runs * up * impotrs * ddpg * quotes * fix tests * fix last r * fix tests * lint * pass checkpoint restore * kwar * nits * policy graph * fix yapf * com * class * pyt * vectorization * update * test cpe * unit test * fix ddpg2 * changes * wip * args * faster test * common * fix * add alg option * batch mode and policy serving * multi serving test * todo * wip * serving test * doc async env * num envs * comments * thread * remove init hook * update * fix ppo * comments1 * fix * updates * add jenkins tests * fix * fix pytorch * fix * fixes * fix a3c policy * fix squeeze * fix trunc on apex * fix squeezing for real * update * remove horizon test for now * multiagent wip * update * fix race condition * fix ma * t * doc * st * wip * example * wip * working * cartpole * wip * batch wip * fix bug * make other_batches None default * working * debug * nit * warn * comments * fix ppo * fix obs filter * update * wip * tf * update * fix * cleanup * cleanup * spacing * model * fix * dqn * fix ddpg * doc * keep names * update * fix * com * docs * clarify model outputs * Update torch_policy_graph.py * fix obs filter * pass thru worker index * fix * rename * vlad torch comments * fix log action * debug name * fix lstm * remove unused ddpg net * remove conv net * revert lstm * wip * wip * cast * wip * works * fix a3c * works * lstm util test * doc * clean up * update * fix lstm check * move to end * fix sphinx * fix cmd * remove bad doc * envs * vec * doc prep * models * rl * alg * up * clarify * copy * async sa * fix * comments * fix a3c conf * tune lstm * fix reshape * fix * back to 16 * tuned a3c update * update * tuned * optional * merge * wip * fix up * move pg class * rename env * wip * update * tip * alg * readme * fix catalog * readme * doc * context * remove prep * comma * add env * link to paper * paper * update * rnn * update * wip * clean up ev creation * fix * fix * fix * fix lint * up * no comma * ma * Update run_multi_node_tests.sh * fix * sphinx is stupid * sphinx is stupid * clarify torch graph * no horizon * fix config * sb * Update test_optimizers.py
2026-09-11 12:43:20 +08:00 · 2018-07-01 00:05:08 -07:00
parent 762bdf646e
commit 8aa56c12e6
113 changed files with 1148 additions and 1141 deletions
@@ -0,0 +1,64 @@
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import tensorflow as tf
+
+import ray
+from ray.rllib.models.catalog import ModelCatalog
+from ray.rllib.evaluation.postprocessing import compute_advantages
+from ray.rllib.evaluation.tf_policy_graph import TFPolicyGraph
+
+
+class PGLoss(object):
+    def __init__(self, action_dist, actions, advantages):
+        self.loss = -tf.reduce_mean(action_dist.logp(actions) * advantages)
+
+
+class PGPolicyGraph(TFPolicyGraph):
+    def __init__(self, obs_space, action_space, config):
+        config = dict(ray.rllib.agents.pg.pg.DEFAULT_CONFIG, **config)
+        self.config = config
+
+        # Setup policy
+        obs = tf.placeholder(tf.float32, shape=[None] + list(obs_space.shape))
+        dist_class, self.logit_dim = ModelCatalog.get_action_dist(
+            action_space, self.config["model"])
+        self.model = ModelCatalog.get_model(
+            obs, self.logit_dim, options=self.config["model"])
+        action_dist = dist_class(self.model.outputs)  # logit for each action
+
+        # Setup policy loss
+        actions = ModelCatalog.get_action_placeholder(action_space)
+        advantages = tf.placeholder(tf.float32, [None], name="adv")
+        loss = PGLoss(action_dist, actions, advantages).loss
+
+        # Initialize TFPolicyGraph
+        sess = tf.get_default_session()
+        loss_in = [
+            ("obs", obs),
+            ("actions", actions),
+            ("advantages", advantages),
+        ]
+
+        # LSTM support
+        for i, ph in enumerate(self.model.state_in):
+            loss_in.append(("state_in_{}".format(i), ph))
+
+        is_training = tf.placeholder_with_default(True, ())
+        TFPolicyGraph.__init__(
+            self, obs_space, action_space, sess, obs_input=obs,
+            action_sampler=action_dist.sample(), loss=loss,
+            loss_inputs=loss_in, is_training=is_training,
+            state_inputs=self.model.state_in,
+            state_outputs=self.model.state_out,
+            seq_lens=self.model.seq_lens,
+            max_seq_len=config["model"]["max_seq_len"])
+        sess.run(tf.global_variables_initializer())
+
+    def postprocess_trajectory(self, sample_batch, other_agent_batches=None):
+        return compute_advantages(
+            sample_batch, 0.0, self.config["gamma"], use_gae=False)
+
+    def get_initial_state(self):
+        return self.model.state_init