From 44acbdd82a35b7c5cd708728250f80fea1c1c133 Mon Sep 17 00:00:00 2001 From: Simon Mo Date: Wed, 13 Jan 2021 14:39:42 -0800 Subject: [PATCH] [Serve] [Doc] Improve batching doc (#13389) --- doc/source/serve/tutorials/batch.rst | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/doc/source/serve/tutorials/batch.rst b/doc/source/serve/tutorials/batch.rst index 48fbc5344..343efcb5d 100644 --- a/doc/source/serve/tutorials/batch.rst +++ b/doc/source/serve/tutorials/batch.rst @@ -64,10 +64,13 @@ configuration option limits the maximum possible batch size sent to the backend. .. note:: Ray Serve performs *opportunistic batching*. When a replica is free to evaluate - the next batch, Ray Serve will look at the pending queries and take - ``max(number_of_pending_queries, max_batch_size)`` queries to form a batch. - You can provide :mod:`batch_wait_timeout ` to override - this behavior to wait for a full batch to arrive before executing (under a timeout). + the next batch, Ray Serve will look at the pending queries and try to take + :mod:`max_batch_size ` queries to form a batch. + If the batch has less than ``max_batch_size`` queries, the backend will + wait for :mod:`batch_wait_timeout ` + seconds to wait for a full batch to arrive. The default wait is ``0s`` to + minimize query latency. You can increase the timeout to improve throughput + and increase utilization at the cost of some additional latency. .. literalinclude:: ../../../../python/ray/serve/examples/doc/tutorial_batch.py :start-after: __doc_deploy_begin__