diff --git a/doc/source/index.rst b/doc/source/index.rst index 89779dfdb..d932cfa7b 100644 --- a/doc/source/index.rst +++ b/doc/source/index.rst @@ -161,10 +161,13 @@ Academic Papers :maxdepth: -1 :caption: Ray Tune - tune.rst - Tutorials, Guides, Examples + tune/index.rst + tune/key-concepts.rst + tune/user-guide.rst + tune/tutorials/overview.rst + tune/examples/index.rst tune/api_docs/overview.rst - tune-contrib.rst + tune/contrib.rst .. toctree:: :hidden: diff --git a/doc/source/ray-overview/basics.rst b/doc/source/ray-overview/basics.rst index 514581267..84de84166 100644 --- a/doc/source/ray-overview/basics.rst +++ b/doc/source/ray-overview/basics.rst @@ -19,7 +19,7 @@ Ray accomplishes this mission by: On top of **Ray Core** are several libraries for solving problems in machine learning: -- :ref:`tune-index` +- :doc:`../tune/index` - :ref:`rllib-index` - :ref:`sgd-index` - :ref:`rayserve` diff --git a/doc/source/ray-overview/index.rst b/doc/source/ray-overview/index.rst index 768385848..67d96f422 100644 --- a/doc/source/ray-overview/index.rst +++ b/doc/source/ray-overview/index.rst @@ -68,7 +68,7 @@ An Overview of the Ray Libraries Ray has a rich ecosystem of libraries and frameworks built on top of it. The main ones being: -- :ref:`tune-index` +- :doc:`../tune/index` - :ref:`rllib-index` - :ref:`sgd-index` - :ref:`rayserve` diff --git a/doc/source/rllib-training.rst b/doc/source/rllib-training.rst index 1c2b0e007..4623dc8a2 100644 --- a/doc/source/rllib-training.rst +++ b/doc/source/rllib-training.rst @@ -180,9 +180,9 @@ Here is an example of the basic usage (for a more complete example, see `custom_ .. note:: - It's recommended that you run RLlib trainers with :ref:`Tune `, for easy experiment management and visualization of results. Just set ``"run": ALG_NAME, "env": ENV_NAME`` in the experiment config. + It's recommended that you run RLlib trainers with :doc:`Tune `, for easy experiment management and visualization of results. Just set ``"run": ALG_NAME, "env": ENV_NAME`` in the experiment config. -All RLlib trainers are compatible with the :ref:`Tune API `. This enables them to be easily used in experiments with :ref:`Tune `. For example, the following code performs a simple hyperparam sweep of PPO: +All RLlib trainers are compatible with the :ref:`Tune API `. This enables them to be easily used in experiments with :doc:`Tune `. For example, the following code performs a simple hyperparam sweep of PPO: .. code-block:: python diff --git a/doc/source/tune/_tutorials/overview.rst b/doc/source/tune/_tutorials/overview.rst index 084b61a49..30d72dd55 100644 --- a/doc/source/tune/_tutorials/overview.rst +++ b/doc/source/tune/_tutorials/overview.rst @@ -1,16 +1,12 @@ -.. _tune-guides-overview: +.. _tune-guides: -Tutorials, User Guides, Examples -================================ +Tutorials +========= .. tip:: We'd love to hear your feedback on using Tune - fill out a `short survey `_! In this section, you can find material on how to use Tune and its various features. If any of the materials is out of date or broken, or if you'd like to add an example to this page, feel free to raise an issue on our Github repository. - -Tutorials ---------- - Take a look at any of the below tutorials to get started with Tune. .. raw:: html @@ -18,40 +14,15 @@ Take a look at any of the below tutorials to get started with Tune.
.. customgalleryitem:: - :tooltip: Tune concepts in 60 seconds. + :tooltip: Key concepts in 60 seconds. :figure: /images/tune-workflow.png - :description: :doc:`Tune concepts in 60 seconds ` + :description: :doc:`Key concepts in 60 seconds ` .. customgalleryitem:: :tooltip: A simple Tune walkthrough. :figure: /images/tune.png :description: :doc:`A walkthrough to setup your first Tune experiment ` -.. raw:: html - -
- -.. toctree:: - :hidden: - - tune-60-seconds.rst - tune-tutorial.rst - - -User Guides ------------ - -These pages will demonstrate the various features and configurations of Tune. - -.. raw:: html - -
- -.. customgalleryitem:: - :tooltip: Tune User Guide - :figure: /images/tune.png - :description: :doc:`Tune User Guide ` - .. customgalleryitem:: :tooltip: A simple guide to Population-based Training :figure: /images/tune-pbt-small.png @@ -67,16 +38,16 @@ These pages will demonstrate the various features and configurations of Tune. :figure: /images/tune-sklearn.png :description: :doc:`Tune's Scikit-Learn Adapters ` -.. customgalleryitem:: - :tooltip: Tuning PyTorch Lightning modules - :figure: /images/pytorch_lightning_small.png - :description: :doc:`Tuning PyTorch Lightning modules ` - .. customgalleryitem:: :tooltip: How to use Tune with PyTorch :figure: /images/pytorch_logo.png :description: :doc:`How to use Tune with PyTorch ` +.. customgalleryitem:: + :tooltip: Tuning PyTorch Lightning modules + :figure: /images/pytorch_lightning_small.png + :description: :doc:`Tuning PyTorch Lightning modules ` + .. customgalleryitem:: :tooltip: Tuning XGBoost parameters. :figure: /images/xgboost_logo.png @@ -87,10 +58,11 @@ These pages will demonstrate the various features and configurations of Tune.
+ .. toctree:: :hidden: - tune-usage.rst + tune-tutorial.rst tune-advanced-tutorial.rst tune-distributed.rst tune-sklearn.rst @@ -144,76 +116,10 @@ Learn how to use Tune in your browser with the following Colab-based exercises. Tutorial source files `can be found here `_. -Tune Examples +What's Next? ------------- -.. Keep this in sync with ray/python/ray/tune/examples/README.rst +Check out: -If any example is broken, or if you'd like to add an example to this page, feel free to raise an issue on our Github repository. - -.. _tune-general-examples: - -General Examples -~~~~~~~~~~~~~~~~ - -- :doc:`/tune/examples/async_hyperband_example`: Example of using a Trainable class with AsyncHyperBandScheduler. -- :doc:`/tune/examples/hyperband_example`: Example of using a Trainable class with HyperBandScheduler. Also uses the Experiment class API for specifying the experiment configuration. Also uses the AsyncHyperBandScheduler. -- :doc:`/tune/examples/pbt_example`: Example of using a Trainable class with PopulationBasedTraining scheduler. -- :doc:`/tune/examples/pbt_function`: Example of using the function API with a PopulationBasedTraining scheduler. -- :doc:`/tune/examples/pbt_ppo_example`: Example of optimizing a distributed RLlib algorithm (PPO) with the PopulationBasedTraining scheduler. -- :doc:`/tune/examples/logging_example`: Example of custom loggers and custom trial directory naming. - -Search Algorithm Examples -~~~~~~~~~~~~~~~~~~~~~~~~~ - -- :doc:`/tune/examples/ax_example`: Optimize a Hartmann function with `Ax `_ with 4 parallel workers. -- :doc:`/tune/examples/hyperopt_example`: Optimizes a basic function using the function-based API and the HyperOptSearch (SearchAlgorithm wrapper for HyperOpt TPE). -- :doc:`/tune/examples/nevergrad_example`: Optimize a simple toy function with the gradient-free optimization package `Nevergrad `_ with 4 parallel workers. -- :doc:`/tune/examples/bayesopt_example`: Optimize a simple toy function using `Bayesian Optimization `_ with 4 parallel workers. - -Tensorflow/Keras Examples -~~~~~~~~~~~~~~~~~~~~~~~~~ - -- :doc:`/tune/examples/tune_mnist_keras`: Converts the Keras MNIST example to use Tune with the function-based API and a Keras callback. Also shows how to easily convert something relying on argparse to use Tune. -- :doc:`/tune/examples/pbt_memnn_example`: Example of training a Memory NN on bAbI with Keras using PBT. -- :doc:`/tune/examples/tf_mnist_example`: Converts the Advanced TF2.0 MNIST example to use Tune with the Trainable. This uses `tf.function`. Original code from tensorflow: https://www.tensorflow.org/tutorials/quickstart/advanced - - -PyTorch Examples -~~~~~~~~~~~~~~~~ - -- :doc:`/tune/examples/mnist_pytorch`: Converts the PyTorch MNIST example to use Tune with the function-based API. Also shows how to easily convert something relying on argparse to use Tune. -- :doc:`/tune/examples/mnist_pytorch_trainable`: Converts the PyTorch MNIST example to use Tune with Trainable API. Also uses the HyperBandScheduler and checkpoints the model at the end. - - -XGBoost Example -~~~~~~~~~~~~~~~ - -- :ref:`XGBoost tutorial `: A guide to tuning XGBoost parameters with Tune. -- :doc:`/tune/examples/xgboost_example`: Trains a basic XGBoost model with Tune with the function-based API and an XGBoost callback. - - -LightGBM Example -~~~~~~~~~~~~~~~~ - -- :doc:`/tune/examples/lightgbm_example`: Trains a basic LightGBM model with Tune with the function-based API and a LightGBM callback. - - -Contributed Examples -~~~~~~~~~~~~~~~~~~~~ - -- :doc:`/tune/examples/pbt_tune_cifar10_with_keras`: A contributed example of tuning a Keras model on CIFAR10 with the PopulationBasedTraining scheduler. -- :doc:`/tune/examples/genetic_example`: Optimizing the michalewicz function using the contributed GeneticSearch algorithm with AsyncHyperBandScheduler. -- :doc:`/tune/examples/tune_cifar10_gluon`: MXNet Gluon example to use Tune with the function-based API on CIFAR-10 dataset. - -Open Source Projects using Tune -------------------------------- - -Here are some of the popular open source repositories and research projects that leverage Tune. Feel free to submit a pull-request adding (or requesting a removal!) of a listed project. - - - `Softlearning `_: Softlearning is a reinforcement learning framework for training maximum entropy policies in continuous domains. Includes the official implementation of the Soft Actor-Critic algorithm. - - `Flambe `_: An ML framework to accelerate research and its path to production. See `flambe.ai `_. - - `Population Based Augmentation `_: Population Based Augmentation (PBA) is a algorithm that quickly and efficiently learns data augmentation functions for neural network training. PBA matches state-of-the-art results on CIFAR with one thousand times less compute. - - `Fast AutoAugment by Kakao `_: Fast AutoAugment (Accepted at NeurIPS 2019) learns augmentation policies using a more efficient search strategy based on density matching. - - `Allentune `_: Hyperparameter Search for AllenNLP from AllenAI. - - `machinable `_: A modular configuration system for machine learning research. See `machinable.org `_. + * :doc:`/tune/user-guide`: A comprehensive overview of Tune's features. + * :doc:`/tune/examples/index`: End-to-end examples and templates for using Tune with your preferred machine learning library. diff --git a/doc/source/tune/_tutorials/tune-tutorial.rst b/doc/source/tune/_tutorials/tune-tutorial.rst index 1808d77fe..1ea414f9d 100644 --- a/doc/source/tune/_tutorials/tune-tutorial.rst +++ b/doc/source/tune/_tutorials/tune-tutorial.rst @@ -134,6 +134,7 @@ You can evaluate best trained model using the :ref:`Analysis object ` for guides on using Tune with your preferred machine learning library. * Browse our :ref:`gallery of examples ` to see how to use Tune with PyTorch, XGBoost, Tensorflow, etc. * `Let us know `__ if you ran into issues or have any questions by opening an issue on our Github. diff --git a/doc/source/tune-contrib.rst b/doc/source/tune/contrib.rst similarity index 100% rename from doc/source/tune-contrib.rst rename to doc/source/tune/contrib.rst diff --git a/doc/source/tune/examples/index.rst b/doc/source/tune/examples/index.rst new file mode 100644 index 000000000..b83873a3d --- /dev/null +++ b/doc/source/tune/examples/index.rst @@ -0,0 +1,76 @@ +======== +Examples +======== + +.. Keep this in sync with ray/python/ray/tune/examples/README.rst + +If any example is broken, or if you'd like to add an example to this page, feel free to raise an issue on our Github repository. + +.. tip:: Check out :ref:`the Tune tutorials page ` for guides on how to use Tune with your preferred machine learning library. + +.. _tune-general-examples: + +General Examples +~~~~~~~~~~~~~~~~ + +- :doc:`/tune/examples/async_hyperband_example`: Example of using a Trainable class with AsyncHyperBandScheduler. +- :doc:`/tune/examples/hyperband_example`: Example of using a Trainable class with HyperBandScheduler. Also uses the Experiment class API for specifying the experiment configuration. Also uses the AsyncHyperBandScheduler. +- :doc:`/tune/examples/pbt_example`: Example of using a Trainable class with PopulationBasedTraining scheduler. +- :doc:`/tune/examples/pbt_function`: Example of using the function API with a PopulationBasedTraining scheduler. +- :doc:`/tune/examples/pbt_ppo_example`: Example of optimizing a distributed RLlib algorithm (PPO) with the PopulationBasedTraining scheduler. +- :doc:`/tune/examples/logging_example`: Example of custom loggers and custom trial directory naming. + +Search Algorithm Examples +~~~~~~~~~~~~~~~~~~~~~~~~~ + +- :doc:`/tune/examples/ax_example`: Optimize a Hartmann function with `Ax `_ with 4 parallel workers. +- :doc:`/tune/examples/hyperopt_example`: Optimizes a basic function using the function-based API and the HyperOptSearch (SearchAlgorithm wrapper for HyperOpt TPE). +- :doc:`/tune/examples/nevergrad_example`: Optimize a simple toy function with the gradient-free optimization package `Nevergrad `_ with 4 parallel workers. +- :doc:`/tune/examples/bayesopt_example`: Optimize a simple toy function using `Bayesian Optimization `_ with 4 parallel workers. + +Tensorflow/Keras Examples +~~~~~~~~~~~~~~~~~~~~~~~~~ + +- :doc:`/tune/examples/tune_mnist_keras`: Converts the Keras MNIST example to use Tune with the function-based API and a Keras callback. Also shows how to easily convert something relying on argparse to use Tune. +- :doc:`/tune/examples/pbt_memnn_example`: Example of training a Memory NN on bAbI with Keras using PBT. +- :doc:`/tune/examples/tf_mnist_example`: Converts the Advanced TF2.0 MNIST example to use Tune with the Trainable. This uses `tf.function`. Original code from tensorflow: https://www.tensorflow.org/tutorials/quickstart/advanced + + +PyTorch Examples +~~~~~~~~~~~~~~~~ + +- :doc:`/tune/examples/mnist_pytorch`: Converts the PyTorch MNIST example to use Tune with the function-based API. Also shows how to easily convert something relying on argparse to use Tune. +- :doc:`/tune/examples/mnist_pytorch_trainable`: Converts the PyTorch MNIST example to use Tune with Trainable API. Also uses the HyperBandScheduler and checkpoints the model at the end. + + +XGBoost Example +~~~~~~~~~~~~~~~ + +- :ref:`XGBoost tutorial `: A guide to tuning XGBoost parameters with Tune. +- :doc:`/tune/examples/xgboost_example`: Trains a basic XGBoost model with Tune with the function-based API and an XGBoost callback. + + +LightGBM Example +~~~~~~~~~~~~~~~~ + +- :doc:`/tune/examples/lightgbm_example`: Trains a basic LightGBM model with Tune with the function-based API and a LightGBM callback. + + +Contributed Examples +~~~~~~~~~~~~~~~~~~~~ + +- :doc:`/tune/examples/pbt_tune_cifar10_with_keras`: A contributed example of tuning a Keras model on CIFAR10 with the PopulationBasedTraining scheduler. +- :doc:`/tune/examples/genetic_example`: Optimizing the michalewicz function using the contributed GeneticSearch algorithm with AsyncHyperBandScheduler. +- :doc:`/tune/examples/tune_cifar10_gluon`: MXNet Gluon example to use Tune with the function-based API on CIFAR-10 dataset. + +Open Source Projects using Tune +------------------------------- + +Here are some of the popular open source repositories and research projects that leverage Tune. Feel free to submit a pull-request adding (or requesting a removal!) of a listed project. + + - `Softlearning `_: Softlearning is a reinforcement learning framework for training maximum entropy policies in continuous domains. Includes the official implementation of the Soft Actor-Critic algorithm. + - `Flambe `_: An ML framework to accelerate research and its path to production. See `flambe.ai `_. + - `Population Based Augmentation `_: Population Based Augmentation (PBA) is a algorithm that quickly and efficiently learns data augmentation functions for neural network training. PBA matches state-of-the-art results on CIFAR with one thousand times less compute. + - `Fast AutoAugment by Kakao `_: Fast AutoAugment (Accepted at NeurIPS 2019) learns augmentation policies using a more efficient search strategy based on density matching. + - `Allentune `_: Hyperparameter Search for AllenNLP from AllenAI. + - `machinable `_: A modular configuration system for machine learning research. See `machinable.org `_. diff --git a/doc/source/tune.rst b/doc/source/tune/index.rst similarity index 56% rename from doc/source/tune.rst rename to doc/source/tune/index.rst index dfb3d16a3..d4404ed51 100644 --- a/doc/source/tune.rst +++ b/doc/source/tune/index.rst @@ -1,23 +1,21 @@ -.. _tune-index: - Tune: Scalable Hyperparameter Tuning ==================================== -.. image:: images/tune.png +.. image:: /images/tune.png :scale: 30% :align: center Tune is a Python library for experiment execution and hyperparameter tuning at any scale. Core features: * Launch a multi-node :ref:`distributed hyperparameter sweep ` in less than 10 lines of code. - * Supports any machine learning framework, :ref:`including PyTorch, XGBoost, MXNet, and Keras`. + * Supports any machine learning framework, :ref:`including PyTorch, XGBoost, MXNet, and Keras `. * Automatically manages :ref:`checkpoints ` and logging to :ref:`TensorBoard `. * Choose among state of the art algorithms such as :ref:`Population Based Training (PBT) `, :ref:`BayesOptSearch `, :ref:`HyperBand/ASHA `. * Move your models from training to serving on the same infrastructure with `Ray Serve`_. .. _`Ray Serve`: serve/index.html -**Want to get started?** Head over to the :ref:`60 second Tune tutorial `. +**Want to get started?** Head over to the :doc:`Key Concepts page `. .. tip:: We'd love to hear your feedback on using Tune - fill out a `short survey `_! @@ -31,7 +29,7 @@ To run this example, install the following: ``pip install 'ray[tune]' torch torc This example runs a small grid search to train a convolutional neural network using PyTorch and Tune. -.. literalinclude:: ../../python/ray/tune/tests/example.py +.. literalinclude:: ../../../python/ray/tune/tests/example.py :language: python :start-after: __quick_start_begin__ :end-before: __quick_start_end__ @@ -43,13 +41,13 @@ If TensorBoard is installed, automatically visualize all trial results: tensorboard --logdir ~/ray_results -.. image:: images/tune-start-tb.png +.. image:: /images/tune-start-tb.png :scale: 30% :align: center If using TF2 and TensorBoard, Tune will also automatically generate TensorBoard HParams output: -.. image:: images/tune-hparams-coord.png +.. image:: /images/tune-hparams-coord.png :scale: 20% :align: center @@ -59,14 +57,44 @@ Why choose Tune? There are many other hyperparameter optimization libraries out there. If you're new to Tune, you're probably wondering, "what makes Tune different?" -.. include:: tune/why_tune.rst +Cutting-edge optimization algorithms +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +As a user, you're probably looking into hyperparameter optimization because you want to quickly increase your model performance. + +Tune enables you to leverage a variety of these cutting edge optimization algorithms, reducing the cost of tuning by `aggressively terminating bad hyperparameter evaluations `_, intelligently :ref:`choosing better parameters to evaluate `, or even :ref:`changing the hyperparameters during training ` to optimize hyperparameter schedules. + +First-class Developer Productivity +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +A key problem with machine learning frameworks is the need to restructure all of your code to fit the framework. + +With Tune, you can optimize your model just by :ref:`adding a few code snippets `. + +Further, Tune actually removes boilerplate from your code training workflow, automatically :ref:`managing checkpoints ` and :ref:`logging results to tools ` such as MLFlow and TensorBoard. + + +Multi-GPU & distributed training out of the box +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Hyperparameter tuning is known to be highly time-consuming, so it is often necessary to parallelize this process. Most other tuning frameworks require you to implement your own multi-process framework or build your own distributed system to speed up hyperparameter tuning. + +However, Tune allows you to transparently :ref:`parallelize across multiple GPUs and multiple nodes `. Tune even has seamless :ref:`fault tolerance and cloud support `, allowing you to scale up your hyperparameter search by 100x while reducing costs by up to 10x by using cheap preemptible instances. + +What if I'm already doing hyperparameter tuning? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +You might be already using an existing hyperparameter tuning tool such as HyperOpt or Bayesian Optimization. + +In this situation, Tune actually allows you to power up your existing workflow. Tune's :ref:`Search Algorithms ` integrate with a variety of popular hyperparameter tuning libraries (such as Nevergrad or HyperOpt) and allow you to seamlessly scale up your optimization process -- without sacrificing performance. + Reference Materials ------------------- Here are some reference materials for Tune: - * :ref:`Tune Tutorials, Guides, and Examples ` + * :doc:`/tune/user-guide` * `Code `__: GitHub repository for Tune Below are some blog posts and talks about Tune: diff --git a/doc/source/tune/_tutorials/tune-60-seconds.rst b/doc/source/tune/key-concepts.rst similarity index 95% rename from doc/source/tune/_tutorials/tune-60-seconds.rst rename to doc/source/tune/key-concepts.rst index fe2c46dea..0d7ae38b5 100644 --- a/doc/source/tune/_tutorials/tune-60-seconds.rst +++ b/doc/source/tune/key-concepts.rst @@ -1,7 +1,8 @@ .. _tune-60-seconds: -Tune in 60 Seconds -================== +============ +Key Concepts +============ Let's quickly walk through the key concepts you need to know to use Tune. In this guide, we'll be covering the following: @@ -186,13 +187,14 @@ This object can also retrieve all training runs as dataframes, allowing you to d df = analysis.dataframe(metric="score", mode="max") What's Next? -~~~~~~~~~~~~ +------------- Now that you have a working understanding of Tune, check out: - * :ref:`Tune Guides and Examples `: Examples and templates for using Tune with your preferred machine learning library. + * :doc:`/tune/user-guide`: A comprehensive overview of Tune's features. + * :ref:`tune-guides`: Tutorials for using Tune with your preferred machine learning library. + * :doc:`/tune/examples/index`: End-to-end examples and templates for using Tune with your preferred machine learning library. * :ref:`tune-tutorial`: A simple tutorial that walks you through the process of setting up a Tune experiment. - * :ref:`tune-user-guide`: A comprehensive overview of Tune's features. Further Questions or Issues? diff --git a/doc/source/tune/_tutorials/tune-usage.rst b/doc/source/tune/user-guide.rst similarity index 97% rename from doc/source/tune/_tutorials/tune-usage.rst rename to doc/source/tune/user-guide.rst index f456dca97..694f315c0 100644 --- a/doc/source/tune/_tutorials/tune-usage.rst +++ b/doc/source/tune/user-guide.rst @@ -1,7 +1,9 @@ -.. _tune-user-guide: +============================= +User Guide & Configuring Tune +============================= + +These pages will demonstrate the various features and configurations of Tune. -Tune User Guide -=============== .. warning:: Before you continue, be sure to have read :ref:`tune-60-seconds`. @@ -131,7 +133,7 @@ You can log arbitrary values and metrics in both training APIs: During training, Tune will automatically log the below metrics in addition to the user-provided values. All of these can be used as stopping conditions or passed as a parameter to Trial Schedulers/Search Algorithms. -.. literalinclude:: ../../../../python/ray/tune/result.py +.. literalinclude:: ../../../python/ray/tune/result.py :language: python :start-after: __sphinx_doc_begin__ :end-before: __sphinx_doc_end__ @@ -344,7 +346,7 @@ If you are running Ray on a remote multi-user cluster where you do not have sudo $ export TMPDIR=/tmp/$USER; mkdir -p $TMPDIR; tensorboard --logdir=~/ray_results -.. image:: ../../ray-tune-tensorboard.png +.. image:: ../ray-tune-tensorboard.png If using TF2, Tune also automatically generates TensorBoard HParams output, as shown below: @@ -358,7 +360,7 @@ If using TF2, Tune also automatically generates TensorBoard HParams output, as s } ) -.. image:: ../../images/tune-hparams.png +.. image:: ../images/tune-hparams.png Console Output -------------- @@ -463,4 +465,4 @@ You can post questions or issues or feedback through the following channels: 2. `GitHub Issues`_: For bug reports and feature requests. .. _`StackOverflow`: https://stackoverflow.com/questions/tagged/ray -.. _`GitHub Issues`: https://github.com/ray-project/ray/issues +.. _`GitHub Issues`: https://github.com/ray-project/ray/issues \ No newline at end of file diff --git a/doc/source/tune/why_tune.rst b/doc/source/tune/why_tune.rst deleted file mode 100644 index 1bd1d9db1..000000000 --- a/doc/source/tune/why_tune.rst +++ /dev/null @@ -1,30 +0,0 @@ -Tune offers cutting-edge optimization algorithms. -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -As a user, you're probably looking into hyperparameter optimization because you want to quickly increase your model performance. - -Tune enables you to leverage a variety of these cutting edge optimization algorithms, reducing the cost of tuning by `aggressively terminating bad hyperparameter evaluations `_, intelligently :ref:`choosing better parameters to evaluate `, or even :ref:`changing the hyperparameters during training ` to optimize hyperparameter schedules. - -Tune simplifies your workflow. -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -A key problem with machine learning frameworks is the need to restructure all of your code to fit the framework. - -With Tune, you can optimize your model just by :ref:`adding a few code snippets `. - -Further, Tune actually removes boilerplate from your code training workflow, automatically :ref:`managing checkpoints ` and :ref:`logging results to tools ` such as MLFlow and TensorBoard. - - -Tune provides first-class multi-GPU & distributed training support. -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Hyperparameter tuning is known to be highly time-consuming, so it is often necessary to parallelize this process. Most other tuning frameworks require you to implement your own multi-process framework or build your own distributed system to speed up hyperparameter tuning. - -However, Tune allows you to transparently :ref:`parallelize across multiple GPUs and multiple nodes `. Tune even has seamless :ref:`fault tolerance and cloud support `, allowing you to scale up your hyperparameter search by 100x while reducing costs by up to 10x by using cheap preemptible instances. - -What if I'm already doing hyperparameter tuning? -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -You might be already using an existing hyperparameter tuning tool such as HyperOpt or Bayesian Optimization. - -In this situation, Tune actually allows you to power up your existing workflow. Tune's :ref:`Search Algorithms ` integrate with a variety of popular hyperparameter tuning libraries (such as Nevergrad or HyperOpt) and allow you to seamlessly scale up your optimization process -- without sacrificing performance.