mirror of
https://github.com/wassname/ray.git
synced 2026-07-14 11:17:54 +08:00
[serve] Serve client refactor (#10409)
This commit is contained in:
Edward Oakes
@@ -21,11 +21,11 @@ To scale out a backend to multiple workers, simplify configure the number of rep
|
||||
.. code-block:: python
|
||||
|
||||
config = {"num_replicas": 10}
|
||||
serve.create_backend("my_scaled_endpoint_backend", handle_request, config=config)
|
||||
client.create_backend("my_scaled_endpoint_backend", handle_request, config=config)
|
||||
|
||||
# scale it back down...
|
||||
config = {"num_replicas": 2}
|
||||
serve.update_backend_config("my_scaled_endpoint_backend", config)
|
||||
client.update_backend_config("my_scaled_endpoint_backend", config)
|
||||
|
||||
This will scale up or down the number of workers that can accept requests.
|
||||
|
||||
@@ -42,7 +42,7 @@ following:
|
||||
.. code-block:: python
|
||||
|
||||
config = {"num_gpus": 1}
|
||||
serve.create_backend("my_gpu_backend", handle_request, ray_actor_options=config)
|
||||
client.create_backend("my_gpu_backend", handle_request, ray_actor_options=config)
|
||||
|
||||
Configuring Parallelism with OMP_NUM_THREADS
|
||||
--------------------------------------------
|
||||
@@ -64,7 +64,7 @@ If you *do* want to enable this parallelism in your Serve backend, just set OMP_
|
||||
os.environ["OMP_NUM_THREADS"] = parallelism
|
||||
# Download model weights, initialize model, etc.
|
||||
|
||||
serve.create_backend("parallel_backend", MyBackend, 12)
|
||||
client.create_backend("parallel_backend", MyBackend, 12)
|
||||
|
||||
.. _serve-batching:
|
||||
|
||||
@@ -90,8 +90,8 @@ You can also have Ray Serve batch requests for performance. In order to do use t
|
||||
return responses
|
||||
|
||||
config = {"max_batch_size": 5}
|
||||
serve.create_backend("counter1", BatchingExample, config=config)
|
||||
serve.create_endpoint("counter1", backend="counter1", route="/increment")
|
||||
client.create_backend("counter1", BatchingExample, config=config)
|
||||
client.create_endpoint("counter1", backend="counter1", route="/increment")
|
||||
|
||||
Please take a look at :ref:`Batching Tutorial<serve-batch-tutorial>` for a deep
|
||||
dive.
|
||||
@@ -102,17 +102,17 @@ Splitting Traffic Between Backends
|
||||
==================================
|
||||
|
||||
At times it may be useful to expose a single endpoint that is served by multiple backends.
|
||||
You can do this by splitting the traffic for an endpoint between backends using :mod:`set_traffic <ray.serve.set_traffic>`.
|
||||
When calling :mod:`set_traffic <ray.serve.set_traffic>`, you provide a dictionary of backend name to a float value that will be used to randomly route that portion of traffic (out of a total of 1.0) to the given backend.
|
||||
You can do this by splitting the traffic for an endpoint between backends using :mod:`client.set_traffic <ray.serve.api.Client.set_traffic>`.
|
||||
When calling :mod:`client.set_traffic <ray.serve.api.Client.set_traffic>`, you provide a dictionary of backend name to a float value that will be used to randomly route that portion of traffic (out of a total of 1.0) to the given backend.
|
||||
For example, here we split traffic 50/50 between two backends:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
serve.create_backend("backend1", MyClass1)
|
||||
serve.create_backend("backend2", MyClass2)
|
||||
client.create_backend("backend1", MyClass1)
|
||||
client.create_backend("backend2", MyClass2)
|
||||
|
||||
serve.create_endpoint("fifty-fifty", backend="backend1", route="/fifty")
|
||||
serve.set_traffic("fifty-fifty", {"backend1": 0.5, "backend2": 0.5})
|
||||
client.create_endpoint("fifty-fifty", backend="backend1", route="/fifty")
|
||||
client.set_traffic("fifty-fifty", {"backend1": 0.5, "backend2": 0.5})
|
||||
|
||||
Each request is routed randomly between the backends in the traffic dictionary according to the provided weights.
|
||||
Please see :ref:`session-affinity` for details on how to ensure that clients or users are consistently mapped to the same backend.
|
||||
@@ -120,52 +120,52 @@ Please see :ref:`session-affinity` for details on how to ensure that clients or
|
||||
Canary Deployments
|
||||
------------------
|
||||
|
||||
:mod:`set_traffic <ray.serve.set_traffic>` can be used to implement canary deployments, where one backend serves the majority of traffic, while a small fraction is routed to a second backend. This is especially useful for "canary testing" a new model on a small percentage of users, while the tried and true old model serves the majority. Once you are satisfied with the new model, you can reroute all traffic to it and remove the old model:
|
||||
:mod:`client.set_traffic <ray.serve.api.Client.set_traffic>` can be used to implement canary deployments, where one backend serves the majority of traffic, while a small fraction is routed to a second backend. This is especially useful for "canary testing" a new model on a small percentage of users, while the tried and true old model serves the majority. Once you are satisfied with the new model, you can reroute all traffic to it and remove the old model:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
serve.create_backend("default_backend", MyClass)
|
||||
client.create_backend("default_backend", MyClass)
|
||||
|
||||
# Initially, set all traffic to be served by the "default" backend.
|
||||
serve.create_endpoint("canary_endpoint", backend="default_backend", route="/canary-test")
|
||||
client.create_endpoint("canary_endpoint", backend="default_backend", route="/canary-test")
|
||||
|
||||
# Add a second backend and route 1% of the traffic to it.
|
||||
serve.create_backend("new_backend", MyNewClass)
|
||||
serve.set_traffic("canary_endpoint", {"default_backend": 0.99, "new_backend": 0.01})
|
||||
client.create_backend("new_backend", MyNewClass)
|
||||
client.set_traffic("canary_endpoint", {"default_backend": 0.99, "new_backend": 0.01})
|
||||
|
||||
# Add a third backend that serves another 1% of the traffic.
|
||||
serve.create_backend("new_backend2", MyNewClass2)
|
||||
serve.set_traffic("canary_endpoint", {"default_backend": 0.98, "new_backend": 0.01, "new_backend2": 0.01})
|
||||
client.create_backend("new_backend2", MyNewClass2)
|
||||
client.set_traffic("canary_endpoint", {"default_backend": 0.98, "new_backend": 0.01, "new_backend2": 0.01})
|
||||
|
||||
# Route all traffic to the new, better backend.
|
||||
serve.set_traffic("canary_endpoint", {"new_backend": 1.0})
|
||||
client.set_traffic("canary_endpoint", {"new_backend": 1.0})
|
||||
|
||||
# Or, if not so succesful, revert to the "default" backend for all traffic.
|
||||
serve.set_traffic("canary_endpoint", {"default_backend": 1.0})
|
||||
client.set_traffic("canary_endpoint", {"default_backend": 1.0})
|
||||
|
||||
Incremental Rollout
|
||||
-------------------
|
||||
|
||||
:mod:`set_traffic <ray.serve.set_traffic>` can also be used to implement incremental rollout.
|
||||
:mod:`client.set_traffic <ray.serve.api.Client.set_traffic>` can also be used to implement incremental rollout.
|
||||
Here, we want to replace an existing backend with a new implementation by gradually increasing the proportion of traffic that it serves.
|
||||
In the example below, we do this repeatedly in one script, but in practice this would likely happen over time across multiple scripts.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
serve.create_backend("existing_backend", MyClass)
|
||||
client.create_backend("existing_backend", MyClass)
|
||||
|
||||
# Initially, all traffic is served by the existing backend.
|
||||
serve.create_endpoint("incremental_endpoint", backend="existing_backend", route="/incremental")
|
||||
client.create_endpoint("incremental_endpoint", backend="existing_backend", route="/incremental")
|
||||
|
||||
# Then we can slowly increase the proportion of traffic served by the new backend.
|
||||
serve.create_backend("new_backend", MyNewClass)
|
||||
serve.set_traffic("incremental_endpoint", {"existing_backend": 0.9, "new_backend": 0.1})
|
||||
serve.set_traffic("incremental_endpoint", {"existing_backend": 0.8, "new_backend": 0.2})
|
||||
serve.set_traffic("incremental_endpoint", {"existing_backend": 0.5, "new_backend": 0.5})
|
||||
serve.set_traffic("incremental_endpoint", {"new_backend": 1.0})
|
||||
client.create_backend("new_backend", MyNewClass)
|
||||
client.set_traffic("incremental_endpoint", {"existing_backend": 0.9, "new_backend": 0.1})
|
||||
client.set_traffic("incremental_endpoint", {"existing_backend": 0.8, "new_backend": 0.2})
|
||||
client.set_traffic("incremental_endpoint", {"existing_backend": 0.5, "new_backend": 0.5})
|
||||
client.set_traffic("incremental_endpoint", {"new_backend": 1.0})
|
||||
|
||||
# At any time, we can roll back to the existing backend.
|
||||
serve.set_traffic("incremental_endpoint", {"existing_backend": 1.0})
|
||||
client.set_traffic("incremental_endpoint", {"existing_backend": 1.0})
|
||||
|
||||
.. _session-affinity:
|
||||
|
||||
@@ -175,7 +175,7 @@ Session Affinity
|
||||
Splitting traffic randomly among backends for each request is is general and simple, but it can be an issue when you want to ensure that a given user or client is served by the same backend repeatedly.
|
||||
To address this, Serve offers a "shard key" can be specified for each request that will deterministically map to a backend.
|
||||
In practice, this should be something that uniquely identifies the entity that you want to consistently map, like a client ID or session ID.
|
||||
The shard key can either be specified via the X-SERVE-SHARD-KEY HTTP header or ``handle.options(shard_key="key")``.
|
||||
The shard key can either be specified via the X-SERVE-SHARD-KEY HTTP header or :mod:`handle.options(shard_key="key") <ray.serve.handle.RayServeHandle.options>`.
|
||||
|
||||
.. note:: The mapping from shard key to backend may change when you update the traffic policy for an endpoint.
|
||||
|
||||
@@ -185,7 +185,7 @@ The shard key can either be specified via the X-SERVE-SHARD-KEY HTTP header or `
|
||||
requests.get("127.0.0.1:8000/api", headers={"X-SERVE-SHARD-KEY": session_id})
|
||||
|
||||
# Specifying the shard key in a call made via serve handle.
|
||||
handle = serve.get_handle("api_endpoint")
|
||||
handle = client.get_handle("api_endpoint")
|
||||
handler.options(shard_key=session_id).remote(args)
|
||||
|
||||
.. _serve-shadow-testing:
|
||||
@@ -194,33 +194,33 @@ Shadow Testing
|
||||
--------------
|
||||
|
||||
Sometimes when deploying a new backend, you may want to test it out without affecting the results seen by users.
|
||||
You can do this with :mod:`shadow_traffic <ray.serve.shadow_traffic>`, which allows you to duplicate requests to multiple backends for testing while still having them served by the set of backends specified via :mod:`set_traffic <ray.serve.set_traffic>`.
|
||||
You can do this with :mod:`client.shadow_traffic <ray.serve.api.Client.shadow_traffic>`, which allows you to duplicate requests to multiple backends for testing while still having them served by the set of backends specified via :mod:`client.set_traffic <ray.serve.api.Client.set_traffic>`.
|
||||
Metrics about these requests are recorded as usual so you can use them to validate model performance.
|
||||
This is demonstrated in the example below, where we create an endpoint serviced by a single backend but shadow traffic to two other backends for testing.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
serve.create_backend("existing_backend", MyClass)
|
||||
client.create_backend("existing_backend", MyClass)
|
||||
|
||||
# All traffic is served by the existing backend.
|
||||
serve.create_endpoint("shadowed_endpoint", backend="existing_backend", route="/shadow")
|
||||
client.create_endpoint("shadowed_endpoint", backend="existing_backend", route="/shadow")
|
||||
|
||||
# Create two new backends that we want to test.
|
||||
serve.create_backend("new_backend_1", MyNewClass)
|
||||
serve.create_backend("new_backend_2", MyNewClass)
|
||||
client.create_backend("new_backend_1", MyNewClass)
|
||||
client.create_backend("new_backend_2", MyNewClass)
|
||||
|
||||
# Shadow traffic to the two new backends. This does not influence the result
|
||||
# of requests to the endpoint, but a proportion of requests are
|
||||
# *additionally* sent to these backends.
|
||||
|
||||
# Send 50% of all queries to the endpoint new_backend_1.
|
||||
serve.shadow_traffic("shadowed_endpoint", "new_backend_1", 0.5)
|
||||
client.shadow_traffic("shadowed_endpoint", "new_backend_1", 0.5)
|
||||
# Send 10% of all queries to the endpoint new_backend_2.
|
||||
serve.shadow_traffic("shadowed_endpoint", "new_backend_2", 0.1)
|
||||
client.shadow_traffic("shadowed_endpoint", "new_backend_2", 0.1)
|
||||
|
||||
# Stop shadowing traffic to the backends.
|
||||
serve.shadow_traffic("shadowed_endpoint", "new_backend_1", 0)
|
||||
serve.shadow_traffic("shadowed_endpoint", "new_backend_2", 0)
|
||||
client.shadow_traffic("shadowed_endpoint", "new_backend_1", 0)
|
||||
client.shadow_traffic("shadowed_endpoint", "new_backend_2", 0)
|
||||
|
||||
.. _serve-model-composition:
|
||||
|
||||
@@ -267,31 +267,35 @@ See :doc:`deployment` for information about how to deploy serve.
|
||||
How do I delete backends and endpoints?
|
||||
---------------------------------------
|
||||
|
||||
To delete a backend, you can use :mod:`serve.delete_backend <ray.serve.delete_backend>`.
|
||||
To delete a backend, you can use :mod:`client.delete_backend <ray.serve.api.Client.delete_backend>`.
|
||||
Note that the backend must not be use by any endpoints in order to be delete.
|
||||
Once a backend is deleted, its tag can be reused.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
serve.delete_backend("simple_backend")
|
||||
client.delete_backend("simple_backend")
|
||||
|
||||
|
||||
To delete a endpoint, you can use :mod:`serve.delete_endpoint <ray.serve.delete_endpoint>`.
|
||||
To delete a endpoint, you can use :mod:`client.delete_endpoint <ray.serve.api.Client.delete_endpoint>`.
|
||||
Note that the endpoint will no longer work and return a 404 when queried.
|
||||
Once a endpoint is deleted, its tag can be reused.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
serve.delete_endpoint("simple_endpoint")
|
||||
client.delete_endpoint("simple_endpoint")
|
||||
|
||||
How do I call an endpoint from Python code?
|
||||
-------------------------------------------
|
||||
|
||||
use the following to get a "handle" to that endpoint.
|
||||
Use :mod:`client.get_handle <ray.serve.api.Client.get_handle>` to get a handle to the endpoint,
|
||||
then use :mod:`handle.remote <ray.serve.handle.RayServeHandle.remote>` to send requests to that
|
||||
endpoint. This returns a Ray ObjectRef whose result can be waited for or retrieved using
|
||||
``ray.wait`` or ``ray.get``, respectively.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
handle = serve.get_handle("api_endpoint")
|
||||
handle = client.get_handle("api_endpoint")
|
||||
ray.get(handle.remote(request))
|
||||
|
||||
|
||||
How do I call a method on my backend class besides __call__?
|
||||
@@ -299,7 +303,7 @@ How do I call a method on my backend class besides __call__?
|
||||
|
||||
To call a method via HTTP use the header field ``X-SERVE-CALL-METHOD``.
|
||||
|
||||
To call a method via Python, do the following:
|
||||
To call a method via Python, use :mod:`handle.options <ray.serve.handle.RayServeHandle.options>`:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
@@ -314,7 +318,7 @@ To call a method via Python, do the following:
|
||||
self.count += inc
|
||||
return True
|
||||
|
||||
handle = serve.get_handle("backend_name")
|
||||
handle = client.get_handle("backend_name")
|
||||
handle.options(method_name="other_method").remote(5)
|
||||
|
||||
How do I enable CORS and other HTTP features?
|
||||
|
||||
@@ -13,10 +13,10 @@ Serve runs on Ray and utilizes :ref:`Ray actors<actor-guide>`.
|
||||
|
||||
There are three kinds of actors that are created to make up a Serve instance:
|
||||
|
||||
- Controller: A global actor unique to each :ref:`Serve instance <serve-instance>` that manages
|
||||
- Controller: A global actor unique to each Serve instance that manages
|
||||
the control plane. The Controller is responsible for creating, updating, and
|
||||
destroying other actors. Serve API calls like :mod:`serve.create_backend <ray.serve.create_backend>`,
|
||||
:mod:`serve.create_endpoint <ray.serve.create_endpoint>` make remote calls to the Controller.
|
||||
destroying other actors. Serve API calls like :mod:`client.create_backend <ray.serve.api.Client.create_backend>`,
|
||||
:mod:`client.create_endpoint <ray.serve.api.Client.create_endpoint>` make remote calls to the Controller.
|
||||
- Router: There is one router per node. Each router is a `Uvicorn <https://www.uvicorn.org/>`_ HTTP
|
||||
server that accepts incoming requests, forwards them to the worker replicas, and
|
||||
responds once they are completed.
|
||||
|
||||
@@ -6,33 +6,27 @@ In the :doc:`key-concepts`, you saw some of the basics of how to write serve app
|
||||
This section will dive a bit deeper into how Ray Serve runs on a Ray cluster and how you're able
|
||||
to deploy and update your serve application over time.
|
||||
|
||||
To deploy a Ray Serve instance you're going to need several things.
|
||||
|
||||
1. A running Ray cluster (you can deploy one on your local machine for testing). To learn more about Ray clusters see :ref:`cluster-index`.
|
||||
2. A Ray Serve instance.
|
||||
3. Your Ray Serve endpoint(s) and backend(s).
|
||||
|
||||
.. contents:: Deploying Ray Serve
|
||||
|
||||
.. _serve-deploy-tutorial:
|
||||
|
||||
Deploying a Model with Ray Serve
|
||||
Lifetime of a Ray Serve Instance
|
||||
================================
|
||||
|
||||
Let's get started deploying our first Ray Serve application. The first thing you'll need
|
||||
to do is start a Ray cluster. You can do that using the Ray cluster launcher, but in our case
|
||||
we'll create it on our local machine. To learn more about Ray Clusters see :ref:`cluster-index`.
|
||||
Ray Serve instances run on top of Ray clusters and are started using :mod:`serve.start <ray.serve.start>`.
|
||||
:mod:`serve.start <ray.serve.start>` returns a :mod:`Client <ray.serve.api.Client>` object that can be used to create the backends and endpoints
|
||||
that will be used to serve your Python code (including ML models).
|
||||
The Serve instance will be torn down when the client object goes out of scope or the script exits.
|
||||
|
||||
Starting the Cluster
|
||||
--------------------
|
||||
We do that by running:
|
||||
When running on a long-lived Ray cluster (e.g., one started using ``ray start`` and connected
|
||||
to using ``ray.init(address="auto")``, you can also deploy a Ray Serve instance as a long-running
|
||||
service using ``serve.start(detached=True)``. In this case, the Serve instance will continue to
|
||||
run on the Ray cluster even after the script that calls it exits. If you want to run another script
|
||||
to update the Serve instance, you can run another script that connects to the Ray cluster and then
|
||||
calls :mod:`serve.connect <ray.serve.connect>`. Note that there can only be one detached Serve instance on each Ray cluster.
|
||||
|
||||
.. code::
|
||||
|
||||
ray start --head
|
||||
|
||||
That starts a cluster on our local machine. We can shut that down by running ``ray stop``. You should
|
||||
run this after we complete this tutorial.
|
||||
Deploying a Model with Ray Serve
|
||||
================================
|
||||
|
||||
Setup: Training a Model
|
||||
-----------------------
|
||||
@@ -58,14 +52,14 @@ Creating a Model and Serving it
|
||||
In the following snippet we will complete two things:
|
||||
|
||||
1. Define a servable model by instantiating a class and defining the ``__call__`` method.
|
||||
2. Connect to our running Ray cluster(``ray.init(...)``) and then start or connect to the Ray Serve instance on that
|
||||
cluster(:mod:`serve.init(...) <ray.serve.init>`).
|
||||
2. Start a local Ray cluster and a Ray Serve instance on top of it
|
||||
(:mod:`serve.start(...) <ray.serve.start>`).
|
||||
|
||||
|
||||
You can see that defining the model is straightforward and simple, we're simply instantiating
|
||||
the model like we might a typical Python class.
|
||||
|
||||
Configuring our model to accept traffic is specified via :mod:`.set_traffic <ray.serve.set_traffic>` after we created
|
||||
Configuring our model to accept traffic is specified via :mod:`client.set_traffic <ray.serve.api.Client.set_traffic>` after we created
|
||||
a backend in serve for our model (and versioned it with a string).
|
||||
|
||||
.. literalinclude:: ../../../python/ray/serve/examples/doc/tutorial_deploy.py
|
||||
@@ -75,7 +69,7 @@ a backend in serve for our model (and versioned it with a string).
|
||||
What serve does when we run this code is store the model as a Ray actor
|
||||
and route traffic to it as the endpoint is queried, in this case over HTTP.
|
||||
Note that in order for this endpoint to be accessible from other machines, we
|
||||
need to specify ``http_host="0.0.0.0"`` in :mod:`serve.init <ray.serve.init>` like we did here.
|
||||
need to specify ``http_host="0.0.0.0"`` in :mod:`serve.start <ray.serve.start>` like we did here.
|
||||
|
||||
Now let's query our endpoint to see the result.
|
||||
|
||||
@@ -111,7 +105,7 @@ these two models.
|
||||
|
||||
.. code::
|
||||
|
||||
serve.set_traffic("iris_classifier", {"lr:v2": 0.25, "lr:v1": 0.75})
|
||||
client.set_traffic("iris_classifier", {"lr:v2": 0.25, "lr:v1": 0.75})
|
||||
|
||||
While this is a simple operation, you may want to see :ref:`serve-split-traffic` for more information.
|
||||
One thing you may want to consider as well is
|
||||
@@ -232,13 +226,13 @@ With the cluster now running, we can run a simple script to start Ray Serve and
|
||||
# Connect to the running Ray cluster.
|
||||
ray.init(address="auto")
|
||||
# Bind on 0.0.0.0 to expose the HTTP server on external IPs.
|
||||
serve.init(http_host="0.0.0.0")
|
||||
client = serve.start(http_host="0.0.0.0")
|
||||
|
||||
def hello():
|
||||
return "hello world"
|
||||
|
||||
serve.create_backend("hello_backend", hello)
|
||||
serve.create_endpoint("hello_endpoint", backend="hello_backend", route="/hello")
|
||||
client.create_backend("hello_backend", hello)
|
||||
client.create_endpoint("hello_endpoint", backend="hello_backend", route="/hello")
|
||||
|
||||
Save this script locally as ``deploy.py`` and run it on the head node using ``ray submit``:
|
||||
|
||||
@@ -276,28 +270,3 @@ One thing you may notice is that we never have to declare a ``while True`` loop
|
||||
something to keep the Ray Serve process running. In general, we don't recommend using forever loops and therefore
|
||||
opt for launching a Ray Cluster locally. Specify a Ray cluster like we did in :ref:`serve-deploy-tutorial`.
|
||||
To learn more, in general, about Ray Clusters see :ref:`cluster-index`.
|
||||
|
||||
|
||||
.. _serve-instance:
|
||||
|
||||
Deploying Multiple Serve Instances on a Single Ray Cluster
|
||||
----------------------------------------------------------
|
||||
|
||||
You can run multiple serve instances on the same Ray cluster by providing a ``name`` in :mod:`serve.init() <ray.serve.init>`.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
# Create a first cluster whose HTTP server listens on 8000.
|
||||
serve.init(name="cluster1", http_port=8000)
|
||||
|
||||
# Create a second cluster whose HTTP server listens on 8001.
|
||||
serve.init(name="cluster2", http_port=8001)
|
||||
|
||||
# Create a backend that will be served on the second cluster.
|
||||
serve.create_backend("backend2", function)
|
||||
serve.create_endpoint("endpoint2", backend="backend2", route="/increment")
|
||||
|
||||
# Switch back to the first cluster and create the same backend on it.
|
||||
serve.init(name="cluster1")
|
||||
serve.create_backend("backend1", function)
|
||||
serve.create_endpoint("endpoint1", backend="backend1", route="/increment")
|
||||
|
||||
@@ -9,7 +9,7 @@ To follow along, you'll need to make the necessary imports.
|
||||
.. code-block:: python
|
||||
|
||||
from ray import serve
|
||||
serve.init() # Initializes Ray and Ray Serve.
|
||||
client = serve.start() # Starts Ray and initializes a Ray Serve instance.
|
||||
|
||||
.. _`serve-backend`:
|
||||
|
||||
@@ -17,11 +17,12 @@ Backends
|
||||
========
|
||||
|
||||
Backends define the implementation of your business logic or models that will handle requests when queries come in to :ref:`serve-endpoint`.
|
||||
In order to support seamless scalability backends can have many replicas, which are individual processes running in the Ray cluster to handle requests.
|
||||
To define a backend, first you must define the "handler" or the business logic you'd like to respond with.
|
||||
The handler should take as input a `Flask Request object <https://flask.palletsprojects.com/en/1.1.x/api/?highlight=request#flask.Request>`_ and return any JSON-serializable object as output.
|
||||
A backend is defined using :mod:`serve.create_backend <ray.serve.create_backend>`, and the implementation can be defined as either a function or a class.
|
||||
A backend is defined using :mod:`client.create_backend <ray.serve.api.Client.create_backend>`, and the implementation can be defined as either a function or a class.
|
||||
Use a function when your response is stateless and a class when you might need to maintain some state (like a model).
|
||||
When using a class, you can specify arguments to be passed to the constructor in :mod:`serve.create_backend <ray.serve.create_backend>`, shown below.
|
||||
When using a class, you can specify arguments to be passed to the constructor in :mod:`client.create_backend <ray.serve.api.Client.create_backend>`, shown below.
|
||||
|
||||
A backend consists of a number of *replicas*, which are individual copies of the function or class that are started in separate worker processes.
|
||||
|
||||
@@ -38,23 +39,23 @@ A backend consists of a number of *replicas*, which are individual copies of the
|
||||
def __call__(self, flask_request):
|
||||
return self.msg
|
||||
|
||||
serve.create_backend("simple_backend", handle_request)
|
||||
client.create_backend("simple_backend", handle_request)
|
||||
# Pass in the message that the backend will return as an argument.
|
||||
# If we call this backend, it will respond with "hello, world!".
|
||||
serve.create_backend("simple_backend_class", RequestHandler, "hello, world!")
|
||||
client.create_backend("simple_backend_class", RequestHandler, "hello, world!")
|
||||
|
||||
We can also list all available backends and delete them to reclaim resources.
|
||||
Note that a backend cannot be deleted while it is in use by an endpoint because then traffic to an endpoint may not be able to be handled.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
>> serve.list_backends()
|
||||
>> client.list_backends()
|
||||
{
|
||||
'simple_backend': {'accepts_batches': False, 'num_replicas': 1, 'max_batch_size': None},
|
||||
'simple_backend_class': {'accepts_batches': False, 'num_replicas': 1, 'max_batch_size': None},
|
||||
}
|
||||
>> serve.delete_backend("simple_backend")
|
||||
>> serve.list_backends()
|
||||
>> client.delete_backend("simple_backend")
|
||||
>> client.list_backends()
|
||||
{
|
||||
'simple_backend_class': {'accepts_batches': False, 'num_replicas': 1, 'max_batch_size': None},
|
||||
}
|
||||
@@ -67,12 +68,12 @@ Endpoints
|
||||
While backends define the implementation of your request handling logic, endpoints allow you to expose them via HTTP.
|
||||
Endpoints are "logical" and can have one or multiple backends that serve requests to them.
|
||||
To create an endpoint, we simply need to specify a name for the endpoint, the name of a backend to handle requests to the endpoint, and the route and methods where it will be accesible.
|
||||
By default endpoints are serviced only by the backend provided to :mod:`serve.create_endpoint <ray.serve.create_endpoint>`, but in some cases you may want to specify multiple backends for an endpoint, e.g., for A/B testing or incremental rollout.
|
||||
By default endpoints are serviced only by the backend provided to :mod:`client.create_endpoint <ray.serve.api.Client.create_endpoint>`, but in some cases you may want to specify multiple backends for an endpoint, e.g., for A/B testing or incremental rollout.
|
||||
For information on how to do this, please see :ref:`serve-split-traffic`.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
serve.create_endpoint("simple_endpoint", backend="simple_backend", route="/simple", methods=["GET"])
|
||||
client.create_endpoint("simple_endpoint", backend="simple_backend", route="/simple", methods=["GET"])
|
||||
|
||||
After creating the endpoint, it is now exposed by the HTTP server and handles requests using the specified backend.
|
||||
We can query the model to verify that it's working.
|
||||
@@ -82,17 +83,17 @@ We can query the model to verify that it's working.
|
||||
import requests
|
||||
print(requests.get("http://127.0.0.1:8000/simple").text)
|
||||
|
||||
To view all of the existing endpoints that have created, use :mod:`serve.list_endpoints <ray.serve.list_endpoints>`.
|
||||
To view all of the existing endpoints that have created, use :mod:`client.list_endpoints <ray.serve.api.Client.list_endpoints>`.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
>>> serve.list_endpoints()
|
||||
>>> client.list_endpoints()
|
||||
{'simple_endpoint': {'route': '/simple', 'methods': ['GET'], 'traffic': {}}}
|
||||
|
||||
You can also delete an endpoint using :mod:`serve.delete_endpoint <ray.serve.delete_endpoint>`.
|
||||
You can also delete an endpoint using :mod:`client.delete_endpoint <ray.serve.api.Client.delete_endpoint>`.
|
||||
Endpoints and backends are independent, so deleting an endpoint will not delete its backends.
|
||||
However, an endpoint must be deleted in order to delete the backends that serve its traffic.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
serve.delete_endpoint("simple_endpoint")
|
||||
client.delete_endpoint("simple_endpoint")
|
||||
|
||||
@@ -1,39 +1,25 @@
|
||||
Package Reference
|
||||
=================
|
||||
Serve API Reference
|
||||
===================
|
||||
|
||||
Basic APIs
|
||||
Start or Connect to a Cluster
|
||||
-----------------------------
|
||||
.. autofunction:: ray.serve.start
|
||||
.. autofunction:: ray.serve.connect
|
||||
|
||||
Client API
|
||||
----------
|
||||
.. autofunction:: ray.serve.init
|
||||
.. autofunction:: ray.serve.shutdown
|
||||
.. autoclass:: ray.serve.api.Client
|
||||
:members: create_backend, list_backends, delete_backend, get_backend_config, update_backend_config, create_endpoint, list_endpoints, delete_endpoint, set_traffic, shadow_traffic, get_handle, shutdown
|
||||
|
||||
Backend Configuration
|
||||
---------------------
|
||||
.. autoclass:: ray.serve.BackendConfig
|
||||
.. autofunction:: ray.serve.create_backend
|
||||
.. autofunction:: ray.serve.create_endpoint
|
||||
|
||||
|
||||
APIs for Managing Endpoints
|
||||
---------------------------
|
||||
.. autofunction:: ray.serve.list_endpoints
|
||||
.. autofunction:: ray.serve.delete_endpoint
|
||||
.. autofunction:: ray.serve.set_traffic
|
||||
.. autofunction:: ray.serve.shadow_traffic
|
||||
|
||||
|
||||
APIs for Managing Backends
|
||||
--------------------------
|
||||
.. autofunction:: ray.serve.list_backends
|
||||
.. autofunction:: ray.serve.delete_backend
|
||||
.. autofunction:: ray.serve.get_backend_config
|
||||
.. autofunction:: ray.serve.update_backend_config
|
||||
|
||||
Advanced APIs
|
||||
-------------
|
||||
|
||||
``serve.get_handle`` enables calling endpoints from Python.
|
||||
|
||||
.. autofunction:: ray.serve.get_handle
|
||||
Handle API
|
||||
----------
|
||||
.. autoclass:: ray.serve.handle.RayServeHandle
|
||||
:members: remote, options
|
||||
|
||||
|
||||
``serve.accept_batch`` marks your backend API does accept list of input instead
|
||||
of just single input.
|
||||
Batching Requests
|
||||
-----------------
|
||||
.. autofunction:: ray.serve.accept_batch
|
||||
|
||||
Reference in New Issue
Block a user