[tune] Support yield and return statements (#10857)

* Support `yield` and `return` statements in Tune trainable functions

* Support anonymous metric with ``tune.report(value)``

* Raise on invalid return/yield value

* Fix end to end reporter test

doc/source/tune/api_docs/trainable.rst

@@ -17,7 +17,6 @@ For the sake of example, let's maximize this objective function:
 Function API
 ------------
 
-
 Here is a simple example of using the function API. You can report intermediate metrics by simply calling ``tune.report`` within the provided function.
 
 .. code-block:: python
@@ -28,7 +27,7 @@ Here is a simple example of using the function API. You can report intermediate
         for x in range(20):
             intermediate_score = objective(x, config["a"], config["b"])
 
-            tune.report(value=intermediate_score)  # This sends the score to Tune.
+            tune.report(score=intermediate_score)  # This sends the score to Tune.
 
     analysis = tune.run(
         trainable,
@@ -41,7 +40,56 @@ Here is a simple example of using the function API. You can report intermediate
 
 Tune will run this function on a separate thread in a Ray actor process.
 
-.. tip:: If you want to leverage multi-node data parallel training with PyTorch while using parallel hyperparameter tuning, check out our :ref:PyTorch user guide and Tune's :ref:distributed pytorch integrations.
+.. tip:: If you want to leverage multi-node data parallel training with PyTorch while using parallel hyperparameter tuning, check out our :ref:`PyTorch <tune-pytorch-cifar>` user guide and Tune's :ref:`distributed pytorch integrations <tune-integration-torch>`.
+
+Function API return and yield values
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Instead of using ``tune.report()``, you can also use Python's ``yield``
+statement to report metrics to Ray Tune:
+
+
+.. code-block:: python
+
+    def trainable(config):
+        # config (dict): A dict of hyperparameters.
+
+        for x in range(20):
+            intermediate_score = objective(x, config["a"], config["b"])
+
+            yield {"score": intermediate_score}  # This sends the score to Tune.
+
+    analysis = tune.run(
+        trainable,
+        config={"a": 2, "b": 4}
+    )
+
+    print("best config: ", analysis.get_best_config(metric="score", mode="max"))
+
+If you yield a dictionary object, this will work just as ``tune.report()``.
+If you yield a number, if will be reported to Ray Tune with the key ``_metric``, i.e.
+as if you had called ``tune.report(_metric=value)``.
+
+Ray Tune supports the same functionality for return values if you only
+report metrics at the end of each run:
+
+.. code-block:: python
+
+    def trainable(config):
+        # config (dict): A dict of hyperparameters.
+
+        final_score = 0
+        for x in range(20):
+            final_score = objective(x, config["a"], config["b"])
+
+        return {"score": final_score}  # This sends the score to Tune.
+
+    analysis = tune.run(
+        trainable,
+        config={"a": 2, "b": 4}
+    )
+
+    print("best config: ", analysis.get_best_config(metric="score", mode="max"))
+
 
 .. _tune-function-checkpointing: