wassname/ray
1
0
Fork 0
mirror of https://github.com/wassname/ray.git synced 2026-08-01 12:51:09 +08:00
Code Issues Packages Projects Releases Wiki Activity

[Serve] [Doc] Improve batching doc (#13389)

Browse Source
This commit is contained in:
Simon Mo
2021-01-13 14:39:42 -08:00
committed by GitHub
parent 6de5711690
commit 44acbdd82a
1 changed files with 7 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/source/serve/tutorials/batch.rst
+7 -4
View File
@@ -64,10 +64,13 @@ configuration option limits the maximum possible batch size sent to the backend.
.. note::
Ray Serve performs *opportunistic batching*. When a replica is free to evaluate
the next batch, Ray Serve will look at the pending queries and take
``max(number_of_pending_queries, max_batch_size)`` queries to form a batch.
You can provide :mod:`batch_wait_timeout <ray.serve.BackendConfig>` to override
this behavior to wait for a full batch to arrive before executing (under a timeout).
the next batch, Ray Serve will look at the pending queries and try to take
:mod:`max_batch_size <ray.serve.BackendConfig>` queries to form a batch.
If the batch has less than ``max_batch_size`` queries, the backend will
wait for :mod:`batch_wait_timeout <ray.serve.BackendConfig>`
seconds to wait for a full batch to arrive. The default wait is ``0s`` to
minimize query latency. You can increase the timeout to improve throughput
and increase utilization at the cost of some additional latency.
.. literalinclude:: ../../../../python/ray/serve/examples/doc/tutorial_batch.py
:start-after: __doc_deploy_begin__