[Serve] Implement ServeHandle refactoring (#10527)

doc/source/serve/advanced.rst

@@ -348,3 +348,66 @@ You can follow the same pattern for other Starlette middlewares.
             Middleware(
                 CORSMiddleware, allow_origins=["*"], allow_methods=["*"])
         ])
+
+
+.. _serve-handle-explainer:
+
+How do ``ServeHandle`` and ``ServeRequest`` work?
+---------------------------------------------------
+
+Ray Serve enables you to query models both from HTTP and Python. This feature
+enables seamless :ref:`model composition<serve-model-composition>`. You can
+get a ``ServeHandle`` corresponding to an ``endpoint``, similar how you can
+reach an endpoint through HTTP via a specific route. When you issue a request
+to an endpoint through ``ServeHandle``, the request goes through the same code
+path as an HTTP request would: choosing backends through :ref:`traffic
+policies <serve-split-traffic>`, finding the next available replica, and
+batching requests together.
+
+When the request arrives in the model, you can access the data similarly to how
+you would with HTTP request. Here are some examples how ServeRequest mirrors Flask.Request:
+
+.. list-table::
+   :header-rows: 1
+
+   * - HTTP
+     - ServeHandle
+     - | Request
+       | (Flask.Request and ServeRequest)
+   * - ``requests.get(..., headers={...})``
+     - ``handle.options(http_headers={...})``
+     - ``request.headers``
+   * - ``requests.post(...)``
+     - ``handle.options(http_method="POST")``
+     - ``requests.method``
+   * - ``request.get(..., json={...})``
+     - ``handle.remote({...})``
+     - ``request.json``
+   * - ``request.get(..., form={...})``
+     - ``handle.remote({...})``
+     - ``request.form``
+   * - ``request.get(..., params={"a":"b"})``
+     - ``handle.remote(a="b")``
+     - ``request.args``
+   * - ``request.get(..., data="long string")``
+     - ``request.remote("long string")``
+     - ``request.data``
+   * - ``N/A``
+     - ``request.remote(python_object)``
+     - ``request.data``
+
+.. note::
+
+    You might have noticed that the last row of the table shows that ServeRequest supports
+    Python object pass through the handle. This is not possible in HTTP. If you
+    need to distinguish if the origin of the request is from Python or HTTP, you can do an ``isinstance``
+    check:
+
+    .. code-block:: python
+
+        import flask
+
+        if isinstance(request, flask.Request):
+            print("Request coming from web!")
+        elif isinstance(request, ServeRequest):
+            print("Request coming from Python!")

doc/source/serve/package-ref.rst

@@ -20,6 +20,12 @@ Handle API
 .. autoclass:: ray.serve.handle.RayServeHandle
     :members: remote, options
 
+When calling from Python, the backend implementation will receive ``ServeRequest``
+objects instead of Flask requests.
+
+.. autoclass:: ray.serve.utils.ServeRequest
+    :members:
+
 Batching Requests
 -----------------
 .. autofunction:: ray.serve.accept_batch

doc/source/serve/tutorials/batch.rst

@@ -30,28 +30,24 @@ You can use the ``@serve.accept_batch`` decorator to annotate a function or a cl
 This annotation is needed because batched backends have different APIs compared
 to single request backends. In a batched backend, the inputs are a list of values.
 
-For single query backend, the input types are single flask request or Python
-argument:
+For single query backend, the input type is a single Flask request or
+:mod:`ServeRequest <ray.serve.utils.ServeRequest>`:
 
 .. code-block:: python
 
     def single_request(
-        flask_request: Flask.Request,
-        *,
-        python_arg: int = 0
+        request: Union[Flask.Request, ServeRequest],
     ):
         pass
 
-For batched backend, the inputs types are converted to list of their original
+For batched backends, the input types are converted to list of their original
 types:
 
 .. code-block:: python
 
     @serve.accept_batch
     def batched_request(
-        flask_request: List[Flask.Request],
-        *,
-        python_arg: List[int]
+        request: List[Union[Flask.Request, ServeRequest]],
     ):
         pass
 
@@ -70,6 +66,8 @@ configuration option limits the maximum possible batch size send to the backend.
     Ray Serve performs *opportunistic batching*. When a worker is free to evaluate
     the next batch, Ray Serve will look at the pending queries and take
     ``max(number_of_pending_queries, max_batch_size)`` queries to form a batch.
+    You can provide :mod:`batch_wait_timeout <ray.serve.BackendConfig>` to override
+    this behavior to wait for a full batch to arrive before executing (under a timeout).
 
 .. literalinclude:: ../../../../python/ray/serve/examples/doc/tutorial_batch.py
     :start-after: __doc_deploy_begin__
@@ -85,17 +83,9 @@ Ray Serve was able to evaluate them in batches.
     :end-before: __doc_query_end__
 
 What if you want to evaluate a whole batch in Python? Ray Serve allows you to send
-queries via the Python API. You can use the boolean value ``serve.context.web`` to
-distinguish the origin of the queries. A batch of queries can either come from
-the web server or the Python API. Ray Serve will guarantee there won't be queries
-with mixed origins.
-
-When the batch of requests comes from the web API, Ray Serve will fill the first
-argument ``flask_requests`` with a list of ``Flask.Request`` objects and set
-``serve.context.web = True``. When the batch of requests comes from the Python API,
-Ray Serve will fill ``flask_requests`` arguments with placeholders, and directly inject
-Python objects into the keyword arguments. In this case, the ``numbers`` argument
-will be a list of Python integers.
+queries via the Python API. A batch of queries can either come from the web server
+or the Python API. Requests coming from the Python API will have the similar API
+as Flask.Request. See more on the API :ref:`here<serve-handle-explainer>`.
 
 .. literalinclude:: ../../../../python/ray/serve/examples/doc/tutorial_batch.py
     :start-after: __doc_define_servable_v1_begin__
@@ -110,7 +100,7 @@ Let's deploy the new version to the same endpoint. Don't forget to set
 
 To query the backend via Python API, we can use ``serve.get_handle`` to receive
 a handle to the corresponding "endpoint". To enqueue a query, you can call
-``handle.remote(argument_name=argument_value)``. This call returns immediately
+``handle.remote(data, argument_name=argument_value)``. This call returns immediately
 with a :ref:`Ray ObjectRef<ray-object-refs>`. You can call `ray.get` to retrieve
 the result.