Merge branch 'master' into py39

doc/source/advanced.rst

@@ -392,3 +392,39 @@ To get information about the current available resource capacity of your cluster
 
 .. autofunction:: ray.available_resources
     :noindex:
+
+Object Spilling
+---------------
+
+Ray 1.2.0+ has *beta* support for spilling objects to external storage once the capacity
+of the object store is used up. Please file a `GitHub issue <https://github.com/ray-project/ray/issues/>`__
+if you encounter any problems with this new feature. Eventually, object spilling will be
+enabled by default, but for now you need to enable it manually:
+
+To enable object spilling to the local filesystem (single node clusters only):
+
+.. code-block:: python
+
+    ray.init(
+        _system_config={
+            "automatic_object_spilling_enabled": True,
+            "object_spilling_config": json.dumps(
+                {"type": "filesystem", "params": {"directory_path": "/tmp/spill"}},
+            )
+        },
+    )
+
+To enable object spilling to remote storage (any URI supported by `smart_open <https://pypi.org/project/smart-open/>`__):
+
+.. code-block:: python
+
+    ray.init(
+        _system_config={
+            "automatic_object_spilling_enabled": True,
+            "max_io_workers": 4,  # More IO workers for remote storage.
+            "min_spilling_size": 100 * 1024 * 1024,  # Spill at least 100MB at a time.
+            "object_spilling_config": json.dumps(
+                {"type": "smart_open", "params": {"uri": "s3:///bucket/path"}},
+            )
+        },
+    )

doc/source/conf.py

@@ -41,6 +41,7 @@ MOCK_MODULES = [
     "horovod",
     "horovod.ray",
     "kubernetes",
+    "mlflow",
     "mxnet",
     "mxnet.model",
     "psutil",

doc/source/ray-debugging.rst

@@ -89,6 +89,112 @@ The Ray debugger supports the
 `same commands as PDB
 <https://docs.python.org/3/library/pdb.html#debugger-commands>`_.
 
+Stepping between Ray tasks
+--------------------------
+
+You can use the debugger to step between Ray tasks. Let's take the
+following recursive function as an example:
+
+.. code-block:: python
+
+    import ray
+
+    ray.init()
+
+    @ray.remote
+    def fact(n):
+        if n == 1:
+            return n
+        else:
+            n_id = fact.remote(n - 1)
+            return n * ray.get(n_id)
+
+    ray.util.pdb.set_trace()
+    result_ref = fact.remote(5)
+    result = ray.get(result_ref)
+
+
+After running the program by executing the Python file and calling
+``ray debug``, you can select the breakpoint by pressing ``0`` and
+enter. This will result in the following output:
+
+.. code-block:: python
+
+    Enter breakpoint index or press enter to refresh: 0
+    > /Users/pcmoritz/tmp/stepping.py(14)<module>()
+    -> result_ref = fact.remote(5)
+    (Pdb) 
+
+You can jump into the call with the ``remote`` command in Ray's debugger.
+Inside the function, print the value of `n` with ``p(n)``, resulting in
+the following output:
+
+.. code-block:: python
+
+    -> result_ref = fact.remote(5)
+    (Pdb) remote
+    *** Connection closed by remote host ***
+    Continuing pdb session in different process...
+    --Call--
+    > /Users/pcmoritz/tmp/stepping.py(5)fact()
+    -> @ray.remote
+    (Pdb) ll
+      5  ->	@ray.remote
+      6  	def fact(n):
+      7  	    if n == 1:
+      8  	        return n
+      9  	    else:
+     10  	        n_id = fact.remote(n - 1)
+     11  	        return n * ray.get(n_id)
+    (Pdb) p(n)
+    5
+    (Pdb) 
+
+Now step into the next remote call again with
+``remote`` and print `n`. You an now either continue recursing into
+the function by calling ``remote`` a few more times, or you can jump
+to the location where ``ray.get`` is called on the result by using the
+``get`` debugger comand. Use ``get`` again to jump back to the original
+call site and use ``p(result)`` to print the result:
+
+.. code-block:: python
+
+    Enter breakpoint index or press enter to refresh: 0
+    > /Users/pcmoritz/tmp/stepping.py(14)<module>()
+    -> result_ref = fact.remote(5)
+    (Pdb) remote
+    *** Connection closed by remote host ***
+    Continuing pdb session in different process...
+    --Call--
+    > /Users/pcmoritz/tmp/stepping.py(5)fact()
+    -> @ray.remote
+    (Pdb) p(n)
+    5
+    (Pdb) remote
+    *** Connection closed by remote host ***
+    Continuing pdb session in different process...
+    --Call--
+    > /Users/pcmoritz/tmp/stepping.py(5)fact()
+    -> @ray.remote
+    (Pdb) p(n)
+    4
+    (Pdb) get
+    *** Connection closed by remote host ***
+    Continuing pdb session in different process...
+    --Return--
+    > /Users/pcmoritz/tmp/stepping.py(5)fact()->120
+    -> @ray.remote
+    (Pdb) get
+    *** Connection closed by remote host ***
+    Continuing pdb session in different process...
+    --Return--
+    > /Users/pcmoritz/tmp/stepping.py(14)<module>()->None
+    -> result_ref = fact.remote(5)
+    (Pdb) p(result)
+    120
+    (Pdb) 
+
+
 Post Mortem Debugging
 ---------------------
 

doc/source/serve/advanced.rst

@@ -267,6 +267,26 @@ That's it. Let's take a look at an example:
 
 .. literalinclude:: ../../../python/ray/serve/examples/doc/snippet_model_composition.py
 
+
+.. _serve-sync-async-handles:
+
+Sync and Async Handles
+======================
+
+Ray Serve offers two types of ``ServeHandle``. You can use the ``client.get_handle(..., sync=True|False)``
+flag to toggle between them.
+
+- When you set ``sync=True`` (the default), a synchronous handle is returned.
+  Calling ``handle.remote()`` should return a Ray ObjectRef.
+- When you set ``sync=False``, an asyncio based handle is returned. You need to
+  Call it with ``await handle.remote()`` to return a Ray ObjectRef. To use ``await``,
+  you have to run ``client.get_handle`` and ``handle.remote`` in Python asyncio event loop.
+
+The async handle has performance advantage because it uses asyncio directly; as compared
+to the sync handle, which talks to an asyncio event loop in a thread. To learn more about
+the reasoning behind these, checkout our `architecture documentation <./architecture.html>`_.
+
+
 Monitoring
 ==========
 
@@ -327,3 +347,11 @@ as shown below.
   :mod:`client.create_backend <ray.serve.api.Client.create_backend>` by
   default.
 
+The dependencies required in the backend may be different than
+the dependencies installed in the driver program (the one running Serve API
+calls). In this case, you can use an
+:mod:`ImportedBackend <ray.serve.backends.ImportedBackend>` to specify a
+backend based on a class that is installed in the Python environment that
+the workers will run in. Example:
+
+.. literalinclude:: ../../../python/ray/serve/examples/doc/imported_backend.py

doc/source/serve/faq.rst

@@ -117,7 +117,7 @@ policies <serve-split-traffic>`, finding the next available replica, and
 batching requests together.
 
 When the request arrives in the model, you can access the data similarly to how
-you would with HTTP request. Here are some examples how ServeRequest mirrors Flask.Request:
+you would with HTTP request. Here are some examples how ServeRequest mirrors Starlette.Request:
 
 .. list-table::
    :header-rows: 1
@@ -125,25 +125,25 @@ you would with HTTP request. Here are some examples how ServeRequest mirrors Fla
    * - HTTP
      - ServeHandle
      - | Request
-       | (Flask.Request and ServeRequest)
+       | (Starlette.Request and ServeRequest)
    * - ``requests.get(..., headers={...})``
      - ``handle.options(http_headers={...})``
      - ``request.headers``
    * - ``requests.post(...)``
      - ``handle.options(http_method="POST")``
-     - ``requests.method``
-   * - ``request.get(..., json={...})``
+     - ``request.method``
+   * - ``requests.get(..., json={...})``
      - ``handle.remote({...})``
-     - ``request.json``
-   * - ``request.get(..., form={...})``
+     - ``await request.json()``
+   * - ``requests.get(..., form={...})``
      - ``handle.remote({...})``
-     - ``request.form``
-   * - ``request.get(..., params={"a":"b"})``
+     - ``await request.form()``
+   * - ``requests.get(..., params={"a":"b"})``
      - ``handle.remote(a="b")``
-     - ``request.args``
-   * - ``request.get(..., data="long string")``
+     - ``request.query_params``
+   * - ``requests.get(..., data="long string")``
      - ``handle.remote("long string")``
-     - ``request.data``
+     - ``await request.body()``
    * - ``N/A``
      - ``handle.remote(python_object)``
      - ``request.data``
@@ -157,9 +157,9 @@ you would with HTTP request. Here are some examples how ServeRequest mirrors Fla
 
     .. code-block:: python
 
-        import flask
+        import starlette.requests
 
-        if isinstance(request, flask.Request):
+        if isinstance(request, starlette.requests.Request):
             print("Request coming from web!")
         elif isinstance(request, ServeRequest):
             print("Request coming from Python!")
@@ -170,10 +170,10 @@ you would with HTTP request. Here are some examples how ServeRequest mirrors Fla
 
     .. code-block:: python
 
-        handle.remote(flask_request)
+        handle.remote(starlette_request)
 
     In this case, Serve will `not` wrap it in ServeRequest. You can directly
-    process the request as a ``flask.Request``.
+    process the request as a ``starlette.requests.Request``.
 
 How fast is Ray Serve?
 ----------------------
@@ -187,13 +187,6 @@ You can checkout our `microbenchmark instruction <https://github.com/ray-project
 to benchmark on your hardware.
 
 
-Does Ray Serve use Flask?
--------------------------
-Flask is only used as a web request object for servable to consume the data.
-We actually use the fastest Python web server: `Uvicorn <https://www.uvicorn.org/>`_ as our web server,
-alongside with the power of Python asyncio.
-**Flask is ONLY the request object that we are using, Uvicorn (not flask) provides the webserver.**
-
 Can I use asyncio along with Ray Serve?
 ---------------------------------------
 Yes! You can make your servable methods ``async def`` and Serve will run them

doc/source/serve/index.rst

@@ -33,6 +33,9 @@ Since Serve is built on Ray, it also allows you to scale to many machines, in yo
   If you want to try out Serve, join our `community slack <https://forms.gle/9TSdDYUgxYs8SA9e8>`_
   and discuss in the #serve channel.
 
+.. note::
+  Starting with Ray version 1.3.0, Ray Serve backends must take in a Starlette Request object instead of a Flask Request object.  
+  See the `migration guide <https://docs.google.com/document/d/1CG4y5WTTc4G_MRQGyjnb_eZ7GK3G9dUX6TNLKLnKRAc/edit?usp=sharing>`_ for details.
 
 Installation
 ============

doc/source/serve/key-concepts.rst

@@ -19,10 +19,8 @@ Backends
 Backends define the implementation of your business logic or models that will handle requests when queries come in to :ref:`serve-endpoint`.
 In order to support seamless scalability backends can have many replicas, which are individual processes running in the Ray cluster to handle requests.
 To define a backend, first you must define the "handler" or the business logic you'd like to respond with.
-The handler should take as input a `Flask Request object <https://flask.palletsprojects.com/en/1.1.x/api/?highlight=request#flask.Request>`_. 
-The handler should return any JSON-serializable object as output.  For a more customizable response type, the handler may return a 
+The handler should take as input a `Starlette Request object <https://www.starlette.io/requests/>`_ and return any JSON-serializable object as output.  For a more customizable response type, the handler may return a 
 `Starlette Response object <https://www.starlette.io/responses/>`_.  
-In the future, Ray Serve will support `Starlette Request objects <https://www.starlette.io/requests/>`_ as input as well.
 
 A backend is defined using :mod:`client.create_backend <ray.serve.api.Client.create_backend>`, and the implementation can be defined as either a function or a class.
 Use a function when your response is stateless and a class when you might need to maintain some state (like a model).
@@ -32,7 +30,7 @@ A backend consists of a number of *replicas*, which are individual copies of the
 
 .. code-block:: python
 
-  def handle_request(flask_request):
+  def handle_request(starlette_request):
     return "hello world"
 
   class RequestHandler:
@@ -40,7 +38,7 @@ A backend consists of a number of *replicas*, which are individual copies of the
     def __init__(self, msg):
         self.msg = msg
 
-    def __call__(self, flask_request):
+    def __call__(self, starlette_request):
         return self.msg
 
   client.create_backend("simple_backend", handle_request)

doc/source/serve/package-ref.rst

@@ -23,7 +23,7 @@ Handle API
     :members: remote, options
 
 When calling from Python, the backend implementation will receive ``ServeRequest``
-objects instead of Flask requests.
+objects instead of Starlette requests.
 
 .. autoclass:: ray.serve.utils.ServeRequest
     :members:
@@ -31,3 +31,6 @@ objects instead of Flask requests.
 Batching Requests
 -----------------
 .. autofunction:: ray.serve.accept_batch
+
+Built-in Backends
+.. autoclass:: ray.serve.backends.ImportedBackend

doc/source/serve/tutorials/batch.rst

@@ -30,13 +30,13 @@ You can use the ``@serve.accept_batch`` decorator to annotate a function or a cl
 This annotation is needed because batched backends have different APIs compared
 to single request backends. In a batched backend, the inputs are a list of values.
 
-For single query backend, the input type is a single Flask request or
+For single query backend, the input type is a single Starlette request or
 :mod:`ServeRequest <ray.serve.utils.ServeRequest>`:
 
 .. code-block:: python
 
     def single_request(
-        request: Union[Flask.Request, ServeRequest],
+        request: Union[starlette.requests.Request, ServeRequest],
     ):
         pass
 
@@ -47,7 +47,7 @@ types:
 
     @serve.accept_batch
     def batched_request(
-        request: List[Union[Flask.Request, ServeRequest]],
+        request: List[Union[starlette.requests.Request, ServeRequest]],
     ):
         pass
 
@@ -84,8 +84,8 @@ Ray Serve was able to evaluate them in batches.
 
 What if you want to evaluate a whole batch in Python? Ray Serve allows you to send
 queries via the Python API. A batch of queries can either come from the web server
-or the Python API. Requests coming from the Python API will have the similar API
-as Flask.Request. See more on the API :ref:`here<serve-handle-explainer>`.
+or the Python API. Requests coming from the Python API will have a similar API
+to Starlette Request. See more on the API :ref:`here<serve-handle-explainer>`.
 
 .. literalinclude:: ../../../../python/ray/serve/examples/doc/tutorial_batch.py
     :start-after: __doc_define_servable_v1_begin__

doc/source/tune/_tutorials/overview.rst

@@ -70,6 +70,11 @@ Take a look at any of the below tutorials to get started with Tune.
    :figure: /images/wandb_logo.png
    :description: :doc:`Track your experiment process with the Weights & Biases tools <tune-wandb>`
 
+.. customgalleryitem::
+    :tooltip: Use MLFlow with Ray Tune.
+    :figure: /images/mlflow.png
+    :description: :doc:`Log and track your hyperparameter sweep with MLFlow Tracking & AutoLogging <tune-mlflow>`
+
 
 .. raw:: html
 
@@ -81,12 +86,13 @@ Take a look at any of the below tutorials to get started with Tune.
 
    tune-tutorial.rst
    tune-advanced-tutorial.rst
-   tune-lifecycle.rst
    tune-distributed.rst
-   tune-sklearn.rst
+   tune-lifecycle.rst
+   tune-mlflow.rst
    tune-pytorch-cifar.rst
    tune-pytorch-lightning.rst
    tune-serve-integration-mnist.rst
+   tune-sklearn.rst
    tune-xgboost.rst
    tune-wandb.rst
 
@@ -156,4 +162,4 @@ Check out:
 
 .. _tune-faq:
 
-.. include:: _faq.rst
+.. include:: _faq.rst

doc/source/tune/_tutorials/tune-mlflow.rst

@@ -0,0 +1,47 @@
+.. _tune-mlflow:
+
+Using MLFlow with Tune
+======================
+
+`MLFlow <https://mlflow.org/>`_ is an open source platform to manage the ML lifecycle, including experimentation,
+reproducibility, deployment, and a central model registry. It currently offers four components, including
+MLFlow Tracking to record and query experiments, including code, data, config, and results.
+
+.. image:: /images/mlflow.png
+  :height: 80px
+  :alt: MLflow
+  :align: center
+  :target: https://www.mlflow.org/
+
+Ray Tune currently offers two lightweight integrations for MLFlow Tracking.
+One is the :ref:`MLFlowLoggerCallback <tune-mlflow-logger>`, which automatically logs
+metrics reported to Tune to the MLFlow Tracking API.
+
+The other one is the :ref:`@mlflow_mixin <tune-mlflow-mixin>` decorator, which can be
+used with the function API. It automatically
+initializes the MLFlow API with Tune's training information and creates a run for each Tune trial.
+Then within your training function, you can just use the
+MLFlow like you would normally do, e.g. using ``mlflow.log_metrics()`` or even ``mlflow.autolog()``
+to log to your training process.
+
+Please :doc:`see here </tune/examples/mlflow_example>` for a full example on how you can use either the
+MLFlowLoggerCallback or the mlflow_mixin.
+
+MLFlow AutoLogging
+------------------
+You can also check out :doc:`here </tune/examples/mlflow_ptl_example>` for an example on how you can leverage MLflow
+autologging, in this case with Pytorch Lightning
+
+MLFlow Logger API
+-----------------
+.. _tune-mlflow-logger:
+
+.. autoclass:: ray.tune.integration.mlflow.MLFlowLoggerCallback
+   :noindex:
+
+MLFlow Mixin API
+----------------
+.. _tune-mlflow-mixin:
+
+.. autofunction:: ray.tune.integration.mlflow.mlflow_mixin
+   :noindex:

doc/source/tune/api_docs/logging.rst

@@ -70,11 +70,7 @@ An example of creating a custom logger can be found in :doc:`/tune/examples/logg
 Trainable Logging
 -----------------
 
-By default, Tune only logs the *training result dictionaries* from your Trainable. However, you may want to visualize the model weights, model graph, or use a custom logging library that requires multi-process logging. For example, you may want to do this if:
-
- * you're using `Weights and Biases <https://www.wandb.com/>`_
- * you're using `MLFlow <https://github.com/mlflow/mlflow/>`__
- * you're trying to log images to Tensorboard.
+By default, Tune only logs the *training result dictionaries* from your Trainable. However, you may want to visualize the model weights, model graph, or use a custom logging library that requires multi-process logging. For example, you may want to do this if you're trying to log images to Tensorboard.
 
 You can do this in the trainable, as shown below:
 
@@ -163,12 +159,17 @@ CSVLogger
 
 .. autoclass:: ray.tune.logger.CSVLoggerCallback
 
-MLFLowLogger
+MLFlowLogger
 ------------
 
-Tune also provides a default logger for `MLFlow <https://mlflow.org>`_. You can install MLFlow via ``pip install mlflow``. An example can be found in :doc:`/tune/examples/mlflow_example`. Note that this currently does not include artifact logging support. For this, you can use the native MLFlow APIs inside your Trainable definition.
+Tune also provides a default logger for `MLFlow <https://mlflow.org>`_. You can install MLFlow via ``pip install mlflow``.
+You can see the :doc:`tutorial here </tune/tutorials/tune-mlflow>`.
 
-.. autoclass:: ray.tune.logger.MLFLowLogger
+WandbLogger
+-----------
+
+Tune also provides a default logger for `Weights & Biases <https://www.wandb.com/>`_. You can install Wandb via ``pip install wandb``.
+You can see the :doc:`tutorial here </tune/tutorials/tune-wandb>`
 
 
 .. _logger-interface:

doc/source/tune/api_docs/search_space.rst

@@ -192,10 +192,18 @@ For a high-level overview, see this example:
         # Sample a integer uniformly between -9 (inclusive) and 15 (exclusive)
         "randint": tune.randint(-9, 15),
 
+        # Sample a integer uniformly between 1 (inclusive) and 10 (exclusive),
+        # while sampling in log space
+        "lograndint": tune.lograndint(1, 10),
+
         # Sample a random uniformly between -21 (inclusive) and 12 (inclusive (!))
         # rounding to increments of 3 (includes 12)
         "qrandint": tune.qrandint(-21, 12, 3),
 
+        # Sample a integer uniformly between 1 (inclusive) and 10 (inclusive (!)),
+        # while sampling in log space and rounding to increments of 2
+        "qlograndint": tune.qlograndint(1, 10, 2),
+
         # Sample an option uniformly from the specified choices
         "choice": tune.choice(["a", "b", "c"]),
 
@@ -263,10 +271,7 @@ Grid Search API
 
 .. autofunction:: ray.tune.grid_search
 
-Internals
----------
+References
+----------
 
-BasicVariantGenerator
-~~~~~~~~~~~~~~~~~~~~~
-
-.. autoclass:: ray.tune.suggest.BasicVariantGenerator
+See also :ref:`tune-basicvariant`.

doc/source/tune/api_docs/suggestion.rst

@@ -22,6 +22,10 @@ Summary
      - Summary
      - Website
      - Code Example
+   * - :ref:`Random search/grid search <tune-basicvariant>`
+     - Random search/grid search
+     -
+     - :doc:`/tune/examples/tune_basic_example`
    * - :ref:`AxSearch <tune-ax>`
      - Bayesian/Bandit Optimization
      - [`Ax <https://ax.dev/>`__]
@@ -123,6 +127,21 @@ identifier.
 
 .. note:: This is currently not implemented for: AxSearch, TuneBOHB, SigOptSearch, and DragonflySearch.
 
+.. _tune-basicvariant:
+
+Random search and grid search (tune.suggest.basic_variant.BasicVariantGenerator)
+--------------------------------------------------------------------------------
+
+The default and most basic way to do hyperparameter search is via random and grid search.
+Ray Tune does this through the :class:`BasicVariantGenerator <ray.tune.suggest.basic_variant.BasicVariantGenerator>`
+class that generates trial variants given a search space definition.
+
+The :class:`BasicVariantGenerator <ray.tune.suggest.basic_variant.BasicVariantGenerator>` is used per
+default if no search algorithm is passed to
+:func:`tune.run() <ray.tune.run>`.
+
+.. autoclass:: ray.tune.suggest.basic_variant.BasicVariantGenerator
+
 .. _tune-ax:
 
 Ax (tune.suggest.ax.AxSearch)

doc/source/tune/examples/index.rst

@@ -13,7 +13,7 @@ If any example is broken, or if you'd like to add an example to this page, feel
 General Examples
 ----------------
 
-
+- :doc:`/tune/examples/tune_basic_example`: Simple example for doing a basic random and grid search.
 - :doc:`/tune/examples/async_hyperband_example`: Example of using a simple tuning function with AsyncHyperBandScheduler.
 - :doc:`/tune/examples/hyperband_function_example`: Example of using a Trainable function with HyperBandScheduler.  Also uses the AsyncHyperBandScheduler.
 - :doc:`/tune/examples/pbt_function`: Example of using the function API with a PopulationBasedTraining scheduler.
@@ -88,6 +88,7 @@ Wandb, MLFlow
 - :ref:`Tutorial <tune-wandb>` for using `wandb <https://www.wandb.com/>`__ with Ray Tune
 - :doc:`/tune/examples/wandb_example`: Example for using `Weights and Biases <https://www.wandb.com/>`__ with Ray Tune.
 - :doc:`/tune/examples/mlflow_example`: Example for using `MLFlow <https://github.com/mlflow/mlflow/>`__ with Ray Tune.
+- :doc:`/tune/examples/mlflow_ptl_example`: Example for using `MLFlow <https://github.com/mlflow/mlflow/>`__ and `Pytorch Lightning <https://github.com/PyTorchLightning/pytorch-lightning>`_ with Ray Tune.
 
 Tensorflow/Keras
 ~~~~~~~~~~~~~~~~

doc/source/tune/examples/mlflow_ptl_example.rst

@@ -0,0 +1,6 @@
+:orphan:
+
+mlflow_ptl_example
+~~~~~~~~~~~~~~~~~~
+
+.. literalinclude:: /../../python/ray/tune/examples/mlflow_ptl.py

doc/source/tune/examples/tune_basic_example.rst

@@ -0,0 +1,6 @@
+:orphan:
+
+tune_basic_example
+~~~~~~~~~~~~~~~~~~
+
+.. literalinclude:: /../../python/ray/tune/examples/tune_basic_example.py

doc/source/walkthrough.rst

@@ -417,6 +417,8 @@ actors.
     to the object ref returned by the put exists. This only applies to the specific
     ref returned by put, not refs in general or copies of that refs.
 
+See also: `object spilling <advanced.html#object-spilling>`__.
+
 Remote Classes (Actors)
 -----------------------