[tune] TensorFlow Distributed Trainable (#11876)

Co-authored-by: Richard Liaw <rliaw@berkeley.edu>

doc/source/tune/api_docs/trainable.rst

@@ -352,7 +352,7 @@ Utilities
 Distributed Torch
 -----------------
 
-Ray also offers lightweight integrations to distribute your model training on Ray Tune.
+Ray offers lightweight integrations to distribute your PyTorch training on Ray Tune.
 
 
 .. autofunction:: ray.tune.integration.torch.DistributedTrainableCreator
@@ -364,6 +364,17 @@ Ray also offers lightweight integrations to distribute your model training on Ra
 .. autofunction:: ray.tune.integration.torch.is_distributed_trainable
    :noindex:
 
+.. _tune-dist-tf-doc:
+
+Distributed TensorFlow
+----------------------
+
+Ray also offers lightweight integrations to distribute your TensorFlow training on Ray Tune.
+
+
+.. autofunction:: ray.tune.integration.tensorflow.DistributedTrainableCreator
+   :noindex:
+
 tune.DurableTrainable
 ---------------------
 

doc/source/tune/user-guide.rst

@@ -87,6 +87,8 @@ See :ref:`limiter` for more details.
 Distributed Tuning
 ~~~~~~~~~~~~~~~~~~
 
+.. tip:: This section covers how to run Tune across multiple machines. See :ref:`Distributed Training <tune-dist-training>` for guidance in tuning distributed training jobs.
+
 To attach to a Ray cluster, simply run ``ray.init`` before ``tune.run``. See :ref:`start-ray-cli` for more information about ``ray.init``:
 
 .. code-block:: python
@@ -97,6 +99,28 @@ To attach to a Ray cluster, simply run ``ray.init`` before ``tune.run``. See :re
 
 Read more in the Tune :ref:`distributed experiments guide <tune-distributed>`.
 
+.. _tune-dist-training:
+
+Tune Distributed Training
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To tune distributed training jobs, Tune provides a set of ``DistributedTrainableCreator`` for different training frameworks.
+Below is an example for tuning distributed TensorFlow jobs:
+
+.. code-block:: python
+
+    # Please refer to full example in tf_distributed_keras_example.py
+    from ray.tune.integration.tensorflow import DistributedTrainableCreator
+    tf_trainable = DistributedTrainableCreator(
+        train_mnist,
+        use_gpu=args.use_gpu,
+        num_workers=2)
+    tune.run(tf_trainable,
+             num_samples=1)
+
+Read more about tuning :ref:`distributed PyTorch <tune-ddp-doc>`, :ref:`TensorFlow <tune-dist-tf-doc>` and :ref:`Horovod <tune-integration-horovod>` jobs.
+
+
 .. _tune-default-search-space:
 
 Search Space (Grid/Random)