mirror of
https://github.com/wassname/ray.git
synced 2026-08-07 11:27:43 +08:00
[Kubernetes][Docs] GPU usage (#13325)
* gpu-note * gpu-note * More info * lint? * Update doc/source/cluster/kubernetes.rst Co-authored-by: Richard Liaw <rliaw@berkeley.edu> * Update doc/source/cluster/kubernetes.rst Co-authored-by: Richard Liaw <rliaw@berkeley.edu> * Update doc/source/cluster/kubernetes.rst Co-authored-by: Richard Liaw <rliaw@berkeley.edu> * Update doc/source/cluster/kubernetes.rst Co-authored-by: Richard Liaw <rliaw@berkeley.edu> * GKE->Kubernetes Co-authored-by: Richard Liaw <rliaw@berkeley.edu>
This commit is contained in:
Dmitri Gekhtman
co-authored by
Richard Liaw
Richard Liaw
parent
a5ddc27bab
commit
7166949194
@@ -7,14 +7,19 @@ Deploying on Kubernetes
|
||||
|
||||
This document is mainly for advanced Kubernetes usage. The easiest way to run a Ray cluster on Kubernetes is by using the built-in Cluster Launcher. Please see the :ref:`Cluster Launcher documentation <ray-launch-k8s>` for details.
|
||||
|
||||
|
||||
|
||||
This document assumes that you have access to a Kubernetes cluster and have
|
||||
``kubectl`` installed locally and configured to access the cluster. It will
|
||||
first walk you through how to deploy a Ray cluster on your existing Kubernetes
|
||||
cluster, then explore a few different ways to run programs on the Ray cluster.
|
||||
|
||||
|
||||
To learn about deploying an autoscaling Ray cluster using :ref:`Ray's Kubernetes operator<k8s-operator>`, read
|
||||
:ref:`here<k8s-operator>`.
|
||||
|
||||
For information on using GPUs with Ray on Kubernetes, see :ref:`here<k8s-gpus>`.
|
||||
|
||||
The configuration ``yaml`` files used here are provided in the `Ray repository`_
|
||||
as examples to get you started. When deploying real applications, you will probably
|
||||
want to build and use your own container images, add more worker nodes to the
|
||||
@@ -292,6 +297,80 @@ To delete a running Ray cluster, you can run the following command:
|
||||
|
||||
kubectl delete -f ray/doc/kubernetes/ray-cluster.yaml
|
||||
|
||||
.. _k8s-gpus:
|
||||
|
||||
Using GPUs
|
||||
----------
|
||||
|
||||
To use GPUs on Kubernetes, you will need to configure both your Kubernetes setup and add additional values to your Ray cluster configuration.
|
||||
|
||||
For relevant documentation for GPU usage on different clouds, see instructions for `GKE`_, for `EKS`_, and for `AKS`_.
|
||||
|
||||
The `Ray Docker Hub <https://hub.docker.com/r/rayproject/>`_ hosts CUDA-based images packaged with Ray for use in Kubernetes pods.
|
||||
For example, the image ``rayproject/ray-ml:nightly-gpu`` is ideal for running GPU-based ML workloads with the most recent nightly build of Ray.
|
||||
Read :ref:`here<docker-images>` for further details on Ray images.
|
||||
|
||||
Using Nvidia GPUs requires specifying the relevant resource `limits` in the container fields of your Kubernetes configurations.
|
||||
(Kubernetes `sets <https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus/#using-device-plugins>`_
|
||||
the GPU request equal to the limit.) The configuration for a pod running a Ray GPU image and
|
||||
using one Nvidia GPU looks like this:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
generateName: example-cluster-ray-worker
|
||||
spec:
|
||||
...
|
||||
containers:
|
||||
- name: ray-node
|
||||
image: rayproject/ray:nightly-gpu
|
||||
...
|
||||
resources:
|
||||
cpu: 1000m
|
||||
memory: 512Mi
|
||||
limits:
|
||||
memory: 512Mi
|
||||
nvidia.com/gpu: 1
|
||||
|
||||
GPU taints and tolerations
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
.. note::
|
||||
|
||||
Users using a managed Kubernetes service probably don't need to worry about this section.
|
||||
|
||||
The `Nvidia gpu plugin`_ for Kubernetes applies `taints`_ to GPU nodes; these taints prevent non-GPU pods from being scheduled on GPU nodes.
|
||||
Managed Kubernetes services like GKE, EKS, and AKS automatically apply matching `tolerations`_
|
||||
to pods requesting GPU resources. Tolerations are applied by means of Kubernetes's `ExtendedResourceToleration`_ `admission controller`_.
|
||||
If this admission controller is not enabled for your Kubernetes cluster, you may need to manually add a GPU toleration each of to your GPU pod configurations. For example,
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
generateName: example-cluster-ray-worker
|
||||
spec:
|
||||
...
|
||||
tolerations:
|
||||
- effect: NoSchedule
|
||||
key: nvidia.com/gpu
|
||||
operator: Exists
|
||||
...
|
||||
containers:
|
||||
- name: ray-node
|
||||
image: rayproject/ray:nightly-gpu
|
||||
...
|
||||
|
||||
Further reference and discussion
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
Read about Kubernetes device plugins `here <https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/>`__,
|
||||
about Kubernetes GPU plugins `here <https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus>`__,
|
||||
and about Nvidia's GPU plugin for Kubernetes `here <https://github.com/NVIDIA/k8s-device-plugin>`__.
|
||||
|
||||
If you run into problems setting up GPUs for your Ray cluster on Kubernetes, please reach out to us at `<https://discuss.ray.io>`_.
|
||||
|
||||
Questions or Issues?
|
||||
--------------------
|
||||
|
||||
@@ -303,3 +382,14 @@ Questions or Issues?
|
||||
.. _`Kubernetes Service`: https://kubernetes.io/docs/concepts/services-networking/service/
|
||||
.. _`Kubernetes Deployment`: https://kubernetes.io/docs/concepts/workloads/controllers/deployment/
|
||||
.. _`Kubernetes Job`: https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/
|
||||
|
||||
.. _`Discussion Board`: https://discuss.ray.io/
|
||||
.. _`GKE`: https://cloud.google.com/kubernetes-engine/docs/how-to/gpus
|
||||
.. _`EKS`: https://docs.aws.amazon.com/eks/latest/userguide/eks-optimized-ami.html
|
||||
.. _`AKS`: https://docs.microsoft.com/en-us/azure/aks/gpu-cluster
|
||||
|
||||
.. _`tolerations`: https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/
|
||||
.. _`taints`: https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/
|
||||
.. _`Nvidia gpu plugin`: https://github.com/NVIDIA/k8s-device-plugin
|
||||
.. _`admission controller`: https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/
|
||||
.. _`ExtendedResourceToleration`: https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#extendedresourcetoleration
|
||||
|
||||
@@ -229,6 +229,7 @@ Installing from ``pip`` should be sufficient for most Ray users.
|
||||
However, should you need to build from source, follow :ref:`these instructions for building <building-ray>` Ray.
|
||||
|
||||
|
||||
.. _docker-images:
|
||||
|
||||
Docker Source Images
|
||||
--------------------
|
||||
|
||||
Reference in New Issue
Block a user