wassname/ray
1
0
Fork 0
mirror of https://github.com/wassname/ray.git synced 2026-07-16 11:21:10 +08:00
Code Issues Packages Projects Releases Wiki Activity

[Serve] Implement ServeHandle refactoring (#10527)

Browse Source
This commit is contained in:
Simon Mo
2020-09-04 15:50:56 -07:00
committed by GitHub
parent 6b6780a108
commit 55b6c19d98
21 changed files with 396 additions and 384 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/source/serve/tutorials/batch.rst
+11 -21
View File
@@ -30,28 +30,24 @@ You can use the ``@serve.accept_batch`` decorator to annotate a function or a cl
This annotation is needed because batched backends have different APIs compared
to single request backends. In a batched backend, the inputs are a list of values.
For single query backend, the input types are single flask request or Python
argument:
For single query backend, the input type is a single Flask request or
:mod:`ServeRequest <ray.serve.utils.ServeRequest>`:
.. code-block:: python
def single_request(
flask_request: Flask.Request,
*,
python_arg: int = 0
request: Union[Flask.Request, ServeRequest],
):
pass
For batched backend, the inputs types are converted to list of their original
For batched backends, the input types are converted to list of their original
types:
.. code-block:: python
@serve.accept_batch
def batched_request(
flask_request: List[Flask.Request],
*,
python_arg: List[int]
request: List[Union[Flask.Request, ServeRequest]],
):
pass
@@ -70,6 +66,8 @@ configuration option limits the maximum possible batch size send to the backend.
Ray Serve performs *opportunistic batching*. When a worker is free to evaluate
the next batch, Ray Serve will look at the pending queries and take
``max(number_of_pending_queries, max_batch_size)`` queries to form a batch.
You can provide :mod:`batch_wait_timeout <ray.serve.BackendConfig>` to override
this behavior to wait for a full batch to arrive before executing (under a timeout).
.. literalinclude:: ../../../../python/ray/serve/examples/doc/tutorial_batch.py
:start-after: __doc_deploy_begin__
@@ -85,17 +83,9 @@ Ray Serve was able to evaluate them in batches.
:end-before: __doc_query_end__
What if you want to evaluate a whole batch in Python? Ray Serve allows you to send
queries via the Python API. You can use the boolean value ``serve.context.web`` to
distinguish the origin of the queries. A batch of queries can either come from
the web server or the Python API. Ray Serve will guarantee there won't be queries
with mixed origins.
When the batch of requests comes from the web API, Ray Serve will fill the first
argument ``flask_requests`` with a list of ``Flask.Request`` objects and set
``serve.context.web = True``. When the batch of requests comes from the Python API,
Ray Serve will fill ``flask_requests`` arguments with placeholders, and directly inject
Python objects into the keyword arguments. In this case, the ``numbers`` argument
will be a list of Python integers.
queries via the Python API. A batch of queries can either come from the web server
or the Python API. Requests coming from the Python API will have the similar API
as Flask.Request. See more on the API :ref:`here<serve-handle-explainer>`.
.. literalinclude:: ../../../../python/ray/serve/examples/doc/tutorial_batch.py
:start-after: __doc_define_servable_v1_begin__
@@ -110,7 +100,7 @@ Let's deploy the new version to the same endpoint. Don't forget to set
To query the backend via Python API, we can use ``serve.get_handle`` to receive
a handle to the corresponding "endpoint". To enqueue a query, you can call
``handle.remote(argument_name=argument_value)``. This call returns immediately
``handle.remote(data, argument_name=argument_value)``. This call returns immediately
with a :ref:`Ray ObjectRef<ray-object-refs>`. You can call `ray.get` to retrieve
the result.