From 7166949194365593bb38813843d8a8ab6be38e06 Mon Sep 17 00:00:00 2001 From: Dmitri Gekhtman <62982571+DmitriGekhtman@users.noreply.github.com> Date: Mon, 11 Jan 2021 21:36:31 -0800 Subject: [PATCH] [Kubernetes][Docs] GPU usage (#13325) * gpu-note * gpu-note * More info * lint? * Update doc/source/cluster/kubernetes.rst Co-authored-by: Richard Liaw * Update doc/source/cluster/kubernetes.rst Co-authored-by: Richard Liaw * Update doc/source/cluster/kubernetes.rst Co-authored-by: Richard Liaw * Update doc/source/cluster/kubernetes.rst Co-authored-by: Richard Liaw * GKE->Kubernetes Co-authored-by: Richard Liaw --- doc/source/cluster/kubernetes.rst | 90 +++++++++++++++++++++++++++++++ doc/source/installation.rst | 1 + 2 files changed, 91 insertions(+) diff --git a/doc/source/cluster/kubernetes.rst b/doc/source/cluster/kubernetes.rst index 6793fae0a..ec3eeb87b 100644 --- a/doc/source/cluster/kubernetes.rst +++ b/doc/source/cluster/kubernetes.rst @@ -7,14 +7,19 @@ Deploying on Kubernetes This document is mainly for advanced Kubernetes usage. The easiest way to run a Ray cluster on Kubernetes is by using the built-in Cluster Launcher. Please see the :ref:`Cluster Launcher documentation ` for details. + + This document assumes that you have access to a Kubernetes cluster and have ``kubectl`` installed locally and configured to access the cluster. It will first walk you through how to deploy a Ray cluster on your existing Kubernetes cluster, then explore a few different ways to run programs on the Ray cluster. + To learn about deploying an autoscaling Ray cluster using :ref:`Ray's Kubernetes operator`, read :ref:`here`. +For information on using GPUs with Ray on Kubernetes, see :ref:`here`. + The configuration ``yaml`` files used here are provided in the `Ray repository`_ as examples to get you started. When deploying real applications, you will probably want to build and use your own container images, add more worker nodes to the @@ -292,6 +297,80 @@ To delete a running Ray cluster, you can run the following command: kubectl delete -f ray/doc/kubernetes/ray-cluster.yaml +.. _k8s-gpus: + +Using GPUs +---------- + +To use GPUs on Kubernetes, you will need to configure both your Kubernetes setup and add additional values to your Ray cluster configuration. + +For relevant documentation for GPU usage on different clouds, see instructions for `GKE`_, for `EKS`_, and for `AKS`_. + +The `Ray Docker Hub `_ hosts CUDA-based images packaged with Ray for use in Kubernetes pods. +For example, the image ``rayproject/ray-ml:nightly-gpu`` is ideal for running GPU-based ML workloads with the most recent nightly build of Ray. +Read :ref:`here` for further details on Ray images. + +Using Nvidia GPUs requires specifying the relevant resource `limits` in the container fields of your Kubernetes configurations. +(Kubernetes `sets `_ +the GPU request equal to the limit.) The configuration for a pod running a Ray GPU image and +using one Nvidia GPU looks like this: + +.. code-block:: yaml + + apiVersion: v1 + kind: Pod + metadata: + generateName: example-cluster-ray-worker + spec: + ... + containers: + - name: ray-node + image: rayproject/ray:nightly-gpu + ... + resources: + cpu: 1000m + memory: 512Mi + limits: + memory: 512Mi + nvidia.com/gpu: 1 + +GPU taints and tolerations +~~~~~~~~~~~~~~~~~~~~~~~~~~ +.. note:: + + Users using a managed Kubernetes service probably don't need to worry about this section. + +The `Nvidia gpu plugin`_ for Kubernetes applies `taints`_ to GPU nodes; these taints prevent non-GPU pods from being scheduled on GPU nodes. +Managed Kubernetes services like GKE, EKS, and AKS automatically apply matching `tolerations`_ +to pods requesting GPU resources. Tolerations are applied by means of Kubernetes's `ExtendedResourceToleration`_ `admission controller`_. +If this admission controller is not enabled for your Kubernetes cluster, you may need to manually add a GPU toleration each of to your GPU pod configurations. For example, + +.. code-block:: yaml + + apiVersion: v1 + kind: Pod + metadata: + generateName: example-cluster-ray-worker + spec: + ... + tolerations: + - effect: NoSchedule + key: nvidia.com/gpu + operator: Exists + ... + containers: + - name: ray-node + image: rayproject/ray:nightly-gpu + ... + +Further reference and discussion +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Read about Kubernetes device plugins `here `__, +about Kubernetes GPU plugins `here `__, +and about Nvidia's GPU plugin for Kubernetes `here `__. + +If you run into problems setting up GPUs for your Ray cluster on Kubernetes, please reach out to us at ``_. + Questions or Issues? -------------------- @@ -303,3 +382,14 @@ Questions or Issues? .. _`Kubernetes Service`: https://kubernetes.io/docs/concepts/services-networking/service/ .. _`Kubernetes Deployment`: https://kubernetes.io/docs/concepts/workloads/controllers/deployment/ .. _`Kubernetes Job`: https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/ + +.. _`Discussion Board`: https://discuss.ray.io/ +.. _`GKE`: https://cloud.google.com/kubernetes-engine/docs/how-to/gpus +.. _`EKS`: https://docs.aws.amazon.com/eks/latest/userguide/eks-optimized-ami.html +.. _`AKS`: https://docs.microsoft.com/en-us/azure/aks/gpu-cluster + +.. _`tolerations`: https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/ +.. _`taints`: https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/ +.. _`Nvidia gpu plugin`: https://github.com/NVIDIA/k8s-device-plugin +.. _`admission controller`: https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/ +.. _`ExtendedResourceToleration`: https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#extendedresourcetoleration diff --git a/doc/source/installation.rst b/doc/source/installation.rst index b85ff20d2..237d62104 100644 --- a/doc/source/installation.rst +++ b/doc/source/installation.rst @@ -229,6 +229,7 @@ Installing from ``pip`` should be sufficient for most Ray users. However, should you need to build from source, follow :ref:`these instructions for building ` Ray. +.. _docker-images: Docker Source Images --------------------