[rllib] Implement custom metrics (#3144)

doc/source/rllib-env.rst

@@ -74,7 +74,7 @@ Performance
 
 There are two ways to scale experience collection with Gym environments:
 
-    1. **Vectorization within a single process:** Though many envs can very achieve high frame rates per core, their throughput is limited in practice by policy evaluation between steps. For example, even small TensorFlow models incur a couple milliseconds of latency to evaluate. This can be worked around by creating multiple envs per process and batching policy evaluations across these envs.
+    1. **Vectorization within a single process:** Though many envs can achieve high frame rates per core, their throughput is limited in practice by policy evaluation between steps. For example, even small TensorFlow models incur a couple milliseconds of latency to evaluate. This can be worked around by creating multiple envs per process and batching policy evaluations across these envs.
 
       You can configure ``{"num_envs_per_worker": M}`` to have RLlib create ``M`` concurrent environments per worker. RLlib auto-vectorizes Gym environments via `VectorEnv.wrap() <https://github.com/ray-project/ray/blob/master/python/ray/rllib/env/vector_env.py>`__.
 

doc/source/rllib-training.rst

@@ -66,12 +66,12 @@ Specifying Parameters
 Each algorithm has specific hyperparameters that can be set with ``--config``, in addition to a number of `common hyperparameters <https://github.com/ray-project/ray/blob/master/python/ray/rllib/agents/agent.py>`__. See the
 `algorithms documentation <rllib-algorithms.html>`__ for more information.
 
-In an example below, we train A2C by specifying 8 workers through the config flag. We also set ``"monitor": true`` to save episode videos to the result dir:
+In an example below, we train A2C by specifying 8 workers through the config flag.
 
 .. code-block:: bash
 
     python ray/python/ray/rllib/train.py --env=PongDeterministic-v4 \
-        --run=A2C --config '{"num_workers": 8, "monitor": true}'
+        --run=A2C --config '{"num_workers": 8}'
 
 .. image:: rllib-config.svg
 
@@ -224,6 +224,79 @@ Sometimes, it is necessary to coordinate between pieces of code that live in dif
 
 Ray actors provide high levels of performance, so in more complex cases they can be used implement communication patterns such as parameter servers and allreduce.
 
+Debugging
+---------
+
+Gym Monitor
+~~~~~~~~~~~
+
+The ``"monitor": true`` config can be used to save Gym episode videos to the result dir. For example:
+
+.. code-block:: bash
+
+    python ray/python/ray/rllib/train.py --env=PongDeterministic-v4 \
+        --run=A2C --config '{"num_workers": 2, "monitor": true}'
+
+    # videos will be saved in the ~/ray_results/<experiment> dir, for example
+    openaigym.video.0.31401.video000000.meta.json
+    openaigym.video.0.31401.video000000.mp4
+    openaigym.video.0.31403.video000000.meta.json
+    openaigym.video.0.31403.video000000.mp4
+
+Log Verbosity
+~~~~~~~~~~~~~
+
+You can control the agent log level via the ``"log_level"`` flag. Valid values are "INFO" (default), "DEBUG", "WARN", and "ERROR". This can be used to increase or decrease the verbosity of internal logging. For example:
+
+.. code-block:: bash
+
+    python ray/python/ray/rllib/train.py --env=PongDeterministic-v4 \
+        --run=A2C --config '{"num_workers": 2, "log_level": "DEBUG"}'
+
+Callbacks and Custom Metrics
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+You can provide callback functions to be called at points during policy evaluation. These functions have access to an info dict containing state for the current `episode <https://github.com/ray-project/ray/blob/master/python/ray/rllib/evaluation/episode.py>`__. Custom state can be stored for the `episode <https://github.com/ray-project/ray/blob/master/python/ray/rllib/evaluation/episode.py>`__ in the ``info["episode"].user_data`` dict, and custom scalar metrics reported by saving values to the ``info["episode"].custom_metrics`` dict. These custom metrics will be averaged and reported as part of training results. The following example (full code `here <https://github.com/ray-project/ray/blob/master/python/ray/rllib/examples/custom_metrics_and_callbacks.py>`__) logs a custom metric from the environment:
+
+.. code-block:: python
+
+    def on_episode_start(info):
+        print(info.keys())  # -> "env", 'episode"
+        episode = info["episode"]
+        print("episode {} started".format(episode.episode_id))
+        episode.user_data["pole_angles"] = []
+
+    def on_episode_step(info):
+        episode = info["episode"]
+        pole_angle = abs(episode.last_observation_for()[2])
+        episode.user_data["pole_angles"].append(pole_angle)
+
+    def on_episode_end(info):
+        episode = info["episode"]
+        mean_pole_angle = np.mean(episode.user_data["pole_angles"])
+        print("episode {} ended with length {} and pole angles {}".format(
+            episode.episode_id, episode.length, mean_pole_angle))
+        episode.custom_metrics["mean_pole_angle"] = mean_pole_angle
+
+    ray.init()
+    trials = tune.run_experiments({
+        "test": {
+            "env": "CartPole-v0",
+            "run": "PG",
+            "config": {
+                "callbacks": {
+                    "on_episode_start": tune.function(on_episode_start),
+                    "on_episode_step": tune.function(on_episode_step),
+                    "on_episode_end": tune.function(on_episode_end),
+                },
+            },
+        }
+    })
+
+Custom metrics can be accessed and visualized like any other training result:
+
+.. image:: custom_metric.png
+
 REST API
 --------
 

doc/source/rllib.rst

@@ -29,6 +29,7 @@ Training APIs
 * `Command-line <rllib-training.html>`__
 * `Configuration <rllib-training.html#configuration>`__
 * `Python API <rllib-training.html#python-api>`__
+* `Debugging <rllib-training.html#debugging>`__
 * `REST API <rllib-training.html#rest-api>`__
 
 Environments