[tune] New Doc edits, add Concepts page (#8083)

Co-Authored-By: Sven Mika <sven@anyscale.io>

doc/source/tune/api_docs/analysis.rst

@@ -1,5 +1,7 @@
-Analysis/Logging (tune.analysis / tune.logger)
-==============================================
+.. _tune-analysis-docs:
+
+Analysis (tune.analysis)
+========================
 
 Analyzing Results
 -----------------
@@ -52,56 +54,3 @@ Analysis
 .. autoclass:: ray.tune.Analysis
     :members:
 
-.. _loggers-docstring:
-
-Loggers (tune.logger)
----------------------
-
-Viskit
-~~~~~~
-
-Tune automatically integrates with Viskit via the ``CSVLogger`` outputs. To use VisKit (you may have to install some dependencies), run:
-
-.. code-block:: bash
-
-    $ git clone https://github.com/rll/rllab.git
-    $ python rllab/rllab/viskit/frontend.py ~/ray_results/my_experiment
-
-The nonrelevant metrics (like timing stats) can be disabled on the left to show only the relevant ones (like accuracy, loss, etc.).
-
-.. image:: /ray-tune-viskit.png
-
-
-.. _logger-interface:
-
-Logger
-~~~~~~
-
-.. autoclass:: ray.tune.logger.Logger
-
-UnifiedLogger
-~~~~~~~~~~~~~
-
-.. autoclass:: ray.tune.logger.UnifiedLogger
-
-TBXLogger
-~~~~~~~~~
-
-.. autoclass:: ray.tune.logger.TBXLogger
-
-JsonLogger
-~~~~~~~~~~
-
-.. autoclass:: ray.tune.logger.JsonLogger
-
-CSVLogger
-~~~~~~~~~
-
-.. autoclass:: ray.tune.logger.CSVLogger
-
-MLFLowLogger
-~~~~~~~~~~~~
-
-Tune also provides a default logger for `MLFlow <https://mlflow.org>`_. You can install MLFlow via ``pip install mlflow``. An example can be found `mlflow_example.py <https://github.com/ray-project/ray/blob/master/python/ray/tune/examples/mlflow_example.py>`__. Note that this currently does not include artifact logging support. For this, you can use the native MLFlow APIs inside your Trainable definition.
-
-.. autoclass:: ray.tune.logger.MLFLowLogger

doc/source/tune/api_docs/execution.rst

@@ -1,6 +1,8 @@
 Training (tune.run, tune.Experiment)
 ====================================
 
+.. _tune-run-ref:
+
 tune.run
 --------
 
@@ -16,6 +18,7 @@ tune.Experiment
 
 .. autofunction:: ray.tune.Experiment
 
+.. _tune-stop-ref:
 
 Stopper (tune.Stopper)
 ----------------------

doc/source/tune/api_docs/logging.rst

@@ -0,0 +1,120 @@
+.. _loggers-docstring:
+
+Loggers (tune.logger)
+=====================
+
+Tune has default loggers for Tensorboard, CSV, and JSON formats.
+
+Logging Path
+------------
+
+Tune will log the results of each trial to a subfolder under a specified local dir, which defaults to ``~/ray_results``.
+
+.. code-block:: python
+
+    # This logs to 2 different trial folders:
+    # ~/ray_results/trainable_name/trial_name_1 and ~/ray_results/trainable_name/trial_name_2
+    # trainable_name and trial_name are autogenerated.
+    tune.run(trainable, num_samples=2)
+
+You can specify the ``local_dir`` and ``trainable_name``:
+
+.. code-block:: python
+
+    # This logs to 2 different trial folders:
+    # ./results/test_experiment/trial_name_1 and ./results/test_experiment/trial_name_2
+    # Only trial_name is autogenerated.
+    tune.run(trainable, num_samples=2, local_dir="./results", name="test_experiment")
+
+To specify custom trial folder names, you can pass use the ``trial_name_creator`` argument
+to `tune.run`.  This takes a function with the following signature:
+
+.. code-block:: python
+
+    def trial_name_string(trial):
+        """
+        Args:
+            trial (Trial): A generated trial object.
+
+        Returns:
+            trial_name (str): String representation of Trial.
+        """
+        return str(trial)
+
+    tune.run(
+        MyTrainableClass,
+        name="example-experiment",
+        num_samples=1,
+        trial_name_creator=trial_name_string
+    )
+
+See the documentation on Trials: :ref:`trial-docstring`.
+
+
+Custom Loggers
+--------------
+
+You can pass in your own logging mechanisms to output logs in custom formats as follows:
+
+.. code-block:: python
+
+    from ray.tune.logger import DEFAULT_LOGGERS
+
+    tune.run(
+        MyTrainableClass,
+        name="experiment_name",
+        loggers=DEFAULT_LOGGERS + (CustomLogger1, CustomLogger2)
+    )
+
+These loggers will be called along with the default Tune loggers. All loggers must inherit the Logger interface (:ref:`logger-interface`).  You can also check out `logger.py <https://github.com/ray-project/ray/blob/master/python/ray/tune/logger.py>`__ for implementation details.
+
+An example can be found in `logging_example.py <https://github.com/ray-project/ray/blob/master/python/ray/tune/examples/logging_example.py>`__.
+
+Viskit
+------
+
+Tune automatically integrates with `Viskit <https://github.com/vitchyr/viskit>`_ via the ``CSVLogger`` outputs. To use VisKit (you may have to install some dependencies), run:
+
+.. code-block:: bash
+
+    $ git clone https://github.com/rll/rllab.git
+    $ python rllab/rllab/viskit/frontend.py ~/ray_results/my_experiment
+
+The nonrelevant metrics (like timing stats) can be disabled on the left to show only the relevant ones (like accuracy, loss, etc.).
+
+.. image:: /ray-tune-viskit.png
+
+
+.. _logger-interface:
+
+Logger
+------
+
+.. autoclass:: ray.tune.logger.Logger
+
+UnifiedLogger
+-------------
+
+.. autoclass:: ray.tune.logger.UnifiedLogger
+
+TBXLogger
+---------
+
+.. autoclass:: ray.tune.logger.TBXLogger
+
+JsonLogger
+----------
+
+.. autoclass:: ray.tune.logger.JsonLogger
+
+CSVLogger
+---------
+
+.. autoclass:: ray.tune.logger.CSVLogger
+
+MLFLowLogger
+------------
+
+Tune also provides a default logger for `MLFlow <https://mlflow.org>`_. You can install MLFlow via ``pip install mlflow``. An example can be found `mlflow_example.py <https://github.com/ray-project/ray/blob/master/python/ray/tune/examples/mlflow_example.py>`__. Note that this currently does not include artifact logging support. For this, you can use the native MLFlow APIs inside your Trainable definition.
+
+.. autoclass:: ray.tune.logger.MLFLowLogger

doc/source/tune/api_docs/overview.rst

@@ -1,3 +1,5 @@
+.. _tune-api-ref:
+
 Tune API Reference
 ==================
 
@@ -16,6 +18,7 @@ on `Github`_.
    grid_random.rst
    suggestion.rst
    schedulers.rst
+   logging.rst
    internals.rst
    client.rst
    cli.rst

doc/source/tune/api_docs/reporters.rst

@@ -1,3 +1,5 @@
+.. _tune-reporter-doc:
+
 Console Output (Reporters)
 ==========================
 
@@ -73,20 +75,22 @@ The default reporting style can also be overriden more broadly by extending the
 
     tune.run(my_trainable, progress_reporter=CustomReporter())
 
-ProgressReporter
-----------------
-
-.. autoclass:: ray.tune.ProgressReporter
-    :members:
 
 CLIReporter
 -----------
 
 .. autoclass:: ray.tune.CLIReporter
+    :members: add_metric_column
 
 JupyterNotebookReporter
 -----------------------
 
 .. autoclass:: ray.tune.JupyterNotebookReporter
+    :members: add_metric_column
 
 
+ProgressReporter
+----------------
+
+.. autoclass:: ray.tune.ProgressReporter
+    :members:

doc/source/tune/api_docs/schedulers.rst

@@ -1,5 +1,7 @@
-Schedulers (tune.schedulers)
-============================
+.. _schedulers-ref:
+
+Trial Schedulers (tune.schedulers)
+==================================
 
 FIFOScheduler
 ~~~~~~~~~~~~~

doc/source/tune/api_docs/suggestion.rst

@@ -1,3 +1,5 @@
+.. _searchalg-ref:
+
 Search Algorithms (tune.suggest)
 ================================
 

doc/source/tune/api_docs/trainable.rst

@@ -3,7 +3,7 @@
 Training (tune.Trainable, tune.track)
 =====================================
 
-Training can be done with either a **Class API** (``tune.Trainable``) < or **function-based API** (``track.log``).
+Training can be done with either a **Class API** (``tune.Trainable``) or **function-based API** (``track.log``).
 
 You can use the **function-based API** for fast prototyping. On the other hand, the ``tune.Trainable`` interface supports checkpoint/restore functionality and provides more control for advanced algorithms.
 
@@ -41,26 +41,26 @@ The Trainable **class API** will require users to subclass ``ray.tune.Trainable`
     from ray import tune
 
     class Guesser(tune.Trainable):
-        """Randomly picks 10 number from [1, 10000) to find the password."""
+        """Randomly picks a number from [1, 10000) to find the password."""
 
         def _setup(self, config):
-            self.config = config
+            self.guess = config["guess"]
+            self.iter = 0
             self.password = 1024
 
         def _train(self):
-            """Execute one step of 'training'."""
-            result_dict = {"diff": abs(self.config['guess'] - self.password)}
-            return result_dict
+            """Execute one step of 'training'. This function will be called iteratively"""
+            self.iter += 1
+            self.guess += 1
+            return {
+                "accuracy": abs(self.guess - self.password),
+                "training_iteration": self.iter  # Tune will automatically provide this.
+            }
 
-        def _stop(self):
-            # perform any cleanup necessary.
-            pass
 
     analysis = tune.run(
         Guesser,
-        stop={
-            "training_iteration": 1,
-        },
+        stop={"training_iteration": 10},
         num_samples=10,
         config={
             "guess": tune.randint(1, 10000)
@@ -109,6 +109,28 @@ Use ``validate_save_restore`` to catch ``_save``/``_restore`` errors before exec
     validate_save_restore(MyTrainableClass)
     validate_save_restore(MyTrainableClass, use_object_store=True)
 
+
+Advanced Resource Allocation
+----------------------------
+
+Trainables can themselves be distributed. If your trainable function / class creates further Ray actors or tasks that also consume CPU / GPU resources, you will want to set ``extra_cpu`` or ``extra_gpu`` inside ``tune.run`` to reserve extra resource slots. For example, if a trainable class requires 1 GPU itself, but also launches 4 actors, each using another GPU, then you should set ``"gpu": 1, "extra_gpu": 4``.
+
+.. code-block:: python
+   :emphasize-lines: 4-8
+
+    tune.run(
+        my_trainable,
+        name="my_trainable",
+        resources_per_trial={
+            "cpu": 1,
+            "gpu": 1,
+            "extra_gpu": 4
+        }
+    )
+
+The ``Trainable`` also provides the ``default_resource_requests`` interface to automatically declare the ``resources_per_trial`` based on the given configuration.
+
+
 Advanced: Reusing Actors
 ~~~~~~~~~~~~~~~~~~~~~~~~