[tune] Expose progress reporter to users (#6915)

* Pluggable progress reporter

* Fix types

* Fix bug, address comments

* lint

* Add convenience function and test

* lint

* Use trials instead of trial_runner

* Add docs

* Update docs

* Fix doc examples

* More doc updates

* Address comments, add configurable frequency

* use reward

doc/source/tune-usage.rst

@@ -708,6 +708,80 @@ By default, Tune will run hyperparameter evaluations on multiple processes. Howe
 
 Note that some behavior such as writing to files by depending on the current working directory in a Trainable and setting global process variables may not work as expected. Local mode with multiple configuration evaluations will interleave computation, so it is most naturally used when running a single configuration evaluation.
 
+CLI Progress Reporting
+----------------------
+
+By default, Tune reports experiment progress periodically to the command-line as follows.
+
+.. code-block:: bash
+
+    == Status ==
+    Memory usage on this node: 11.4/16.0 GiB
+    Using FIFO scheduling algorithm.
+    Resources requested: 4/12 CPUs, 0/0 GPUs, 0.0/3.17 GiB heap, 0.0/1.07 GiB objects
+    Result logdir: /Users/foo/ray_results/myexp
+    Number of trials: 4 (4 RUNNING)
+    +----------------------+----------+---------------------+-----------+--------+--------+--------+--------+------------------+-------+
+    | Trial name           | status   | loc                 |    param1 | param2 | param3 |    acc |   loss |   total time (s) |  iter |
+    |----------------------+----------+---------------------+-----------+--------+--------+--------+--------+------------------+-------|
+    | MyTrainable_a826033a | RUNNING  | 10.234.98.164:31115 | 0.303706  | 0.0761 | 0.4328 | 0.1289 | 1.8572 |          7.54952 |    15 |
+    | MyTrainable_a8263fc6 | RUNNING  | 10.234.98.164:31117 | 0.929276  | 0.158  | 0.3417 | 0.4865 | 1.6307 |          7.0501  |    14 |
+    | MyTrainable_a8267914 | RUNNING  | 10.234.98.164:31111 | 0.068426  | 0.0319 | 0.1147 | 0.9585 | 1.9603 |          7.0477  |    14 |
+    | MyTrainable_a826b7bc | RUNNING  | 10.234.98.164:31112 | 0.729127  | 0.0748 | 0.1784 | 0.1797 | 1.7161 |          7.05715 |    14 |
+    +----------------------+----------+---------------------+-----------+--------+--------+--------+--------+------------------+-------+
+
+Note that columns will be hidden if they are completely empty. The output can be configured in various ways by instantiating a ``CLIReporter`` instance (or ``JupyterNotebookReporter`` if you're using jupyter notebook). Here's an example:
+
+.. code-block:: python
+
+    from ray.tune import CLIReporter
+
+    # Limit the number of rows.
+    reporter = CLIReporter(max_progress_rows=10)
+    # Add a custom metric column, in addition to the default metrics.
+    # Note that this must be a metric that is returned in your training results.
+    reporter.add_metric_column("custom_metric")
+    tune.run(my_trainable, progress_reporter=reporter)
+
+Extending ``CLIReporter`` lets you control reporting frequency. For example:
+
+.. code-block:: python
+
+    class ExperimentTerminationReporter(CLIReporter):
+        def should_report(self, trials, done=False):
+            """Reports only on experiment termination."""
+            return done
+
+    tune.run(my_trainable, progress_reporter=ExperimentTerminationReporter())
+
+    class TrialTerminationReporter(CLIReporter):
+        def __init__(self):
+            self.num_terminated = 0
+
+        def should_report(self, trials, done=False):
+            """Reports only on trial termination events."""
+            old_num_terminated = self.num_terminated
+            self.num_terminated = len([t for t in trials if t.status == Trial.TERMINATED])
+            return self.num_terminated > old_num_terminated
+
+    tune.run(my_trainable, progress_reporter=TrialTerminationReporter())
+
+The default reporting style can also be overriden more broadly by extending the ``ProgressReporter`` interface directly. Note that you can print to any output stream, file etc.
+
+.. code-block:: python
+
+    from ray.tune import ProgressReporter
+
+    class CustomReporter(ProgressReporter):
+
+        def should_report(self, trials, done=False):
+            return True
+
+        def report(self, trials, *sys_info):
+            print(*sys_info)
+            print("\n".join([str(trial) for trial in trials]))
+
+    tune.run(my_trainable, progress_reporter=CustomReporter())
 
 Tune CLI (Experimental)
 -----------------------