From 4ef9d15315baa338771215636a8ba077255a2271 Mon Sep 17 00:00:00 2001
From: Eric Liang <ekhliang@gmail.com>
Date: Sun, 8 Jul 2018 18:46:52 -0700
Subject: [PATCH] [rllib] Add concepts section of docs (#2373)

This fills in the rllib concepts documentation.
---
 doc/source/index.rst          |  1 +
 doc/source/rllib-concepts.rst | 27 +++++++++++++++++++++++++++
 doc/source/rllib-env.rst      |  2 +-
 doc/source/rllib.rst          | 14 +++++++-------
 4 files changed, 36 insertions(+), 8 deletions(-)
 create mode 100644 doc/source/rllib-concepts.rst

diff --git a/doc/source/index.rst b/doc/source/index.rst
index f1b205f6c..14865b002 100644
--- a/doc/source/index.rst
+++ b/doc/source/index.rst
@@ -83,6 +83,7 @@ Ray comes with libraries that accelerate deep learning and reinforcement learnin
    rllib-env.rst
    rllib-algorithms.rst
    rllib-models.rst
+   rllib-concepts.rst
    rllib-package-ref.rst
 
 .. toctree::
diff --git a/doc/source/rllib-concepts.rst b/doc/source/rllib-concepts.rst
new file mode 100644
index 000000000..9e9937c83
--- /dev/null
+++ b/doc/source/rllib-concepts.rst
@@ -0,0 +1,27 @@
+RLlib Concepts
+==============
+
+.. note::
+
+    To learn more about these concepts, see also the `ICML paper <https://arxiv.org/abs/1712.09381>`__.
+
+Policy Graphs
+-------------
+
+Policy graph classes encapsulate the core numerical components of RL algorithms. This typically includes the policy model that determines actions to take, a trajectory postprocessor for experiences, and a loss function to improve the policy given postprocessed experiences. For a simple example, see the policy gradients `graph definition <https://github.com/ray-project/ray/blob/master/python/ray/rllib/agents/pg/pg_policy_graph.py>`__.
+
+Most interaction with deep learning frameworks is isolated to the `PolicyGraph interface <https://github.com/ray-project/ray/blob/master/python/ray/rllib/evaluation/policy_graph.py>`__, allowing RLlib to support multiple frameworks. To simplify the definition of policy graphs, RLlib includes `Tensorflow <https://github.com/ray-project/ray/blob/master/python/ray/rllib/evaluation/tf_policy_graph.py>`__ and `PyTorch-specific <https://github.com/ray-project/ray/blob/master/python/ray/rllib/evaluation/torch_policy_graph.py>`__ templates.
+
+Policy Evaluation
+-----------------
+
+Given an environment and policy graph, policy evaluation produces `batches <https://github.com/ray-project/ray/blob/master/python/ray/rllib/evaluation/sample_batch.py>`__ of experiences. This is your classic "environment interaction loop". Efficient policy evaluation can be burdensome to get right, especially when leveraging vectorization, RNNs, or when operating in a multi-agent environment. RLlib provides a `PolicyEvaluator <https://github.com/ray-project/ray/blob/master/python/ray/rllib/evaluation/policy_evaluator.py>`__ class that manages all of this, and this class is used in most RLlib algorithm.
+
+You can also use policy evaluation standalone to produce batches of experiences. This can be done by calling ``ev.sample()`` on an evaluator instance, or ``ev.sample.remote()`` in parallel on evaluator instances created as Ray actors (see ``PolicyEvalutor.as_remote()``).
+
+Policy Optimization
+-------------------
+
+Similar to how a `gradient-descent optimizer <https://www.tensorflow.org/api_docs/python/tf/train/GradientDescentOptimizer>`__ can be used to improve a model, RLlib's `policy optimizers <https://github.com/ray-project/ray/tree/master/python/ray/rllib/optimizers>`__ implement different strategies for improving a policy graph.
+
+For example, in A3C you'd want to compute gradient asynchronously on different workers, and apply them to a central policy graph replica. This strategy is implemented by the `AsyncGradientsOptimizer <https://github.com/ray-project/ray/blob/master/python/ray/rllib/optimizers/async_gradients_optimizer.py>`__. Another alternative is to gather experiences synchronously in parallel and optimize the model centrally, as in `SyncSamplesOptimizer <https://github.com/ray-project/ray/blob/master/python/ray/rllib/optimizers/sync_samples_optimizer.py>`__. Policy optimizers abstract these strategies away into reusable modules.
diff --git a/doc/source/rllib-env.rst b/doc/source/rllib-env.rst
index 20e6eed3b..95f6b088e 100644
--- a/doc/source/rllib-env.rst
+++ b/doc/source/rllib-env.rst
@@ -19,7 +19,7 @@ In the high-level agent APIs, environments are identified with string names. By
 
     register_env("my_env", env_creator)
     ray.init()
-    trainer = ppo.PPOAgent(env="my-env", config={
+    trainer = ppo.PPOAgent(env="my_env", config={
         "env_config": {},  # config to pass to env creator
     })
 
diff --git a/doc/source/rllib.rst b/doc/source/rllib.rst
index 7316fc127..1e1d0d915 100644
--- a/doc/source/rllib.rst
+++ b/doc/source/rllib.rst
@@ -34,7 +34,7 @@ Environments
 ------------
 * `RLlib Environments Overview <rllib-env.html>`__
 * `OpenAI Gym <rllib-env.html#openai-gym>`__
-* `Vectorized (Batch) <rllib-env.html#vectorized>`__
+* `Vectorized <rllib-env.html#vectorized>`__
 * `Multi-Agent <rllib-env.html#multi-agent>`__
 * `Serving (Agent-oriented) <rllib-env.html#serving>`__
 * `Offline Data Ingest <rllib-env.html#offline-data>`__ 
@@ -51,17 +51,17 @@ Algorithms
 * `Proximal Policy Optimization <rllib-algorithms.html#proximal-policy-optimization>`__
 
 Models and Preprocessors
--------------------------------
+------------------------
 * `RLlib Models and Preprocessors Overview <rllib-models.html>`__
 * `Built-in Models and Preprocessors <rllib-models.html#built-in-models-and-preprocessors>`__
 * `Custom Models <rllib-models.html#custom-models>`__
 * `Custom Preprocessors <rllib-models.html#custom-preprocessors>`__
 
-RL Building Blocks
-------------------
-* Policy Models, Losses, Postprocessing
-* Policy Evaluation
-* Policy Optimization
+RLlib Concepts
+--------------
+* `Policy Graphs <rllib-concepts.html>`__
+* `Policy Evaluation <rllib-concepts.html#policy-evaluation>`__
+* `Policy Optimization <rllib-concepts.html#policy-optimization>`__
 
 Package Reference
 -----------------