wassname/ray
1
0
Fork 0
mirror of https://github.com/wassname/ray.git synced 2026-07-31 12:41:01 +08:00
Code Issues Packages Projects Releases Wiki Activity

[docs] Revised Cluster documentation (#9062)

Browse Source 
Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>
This commit is contained in:
Richard Liaw
2020-06-26 09:29:22 -07:00
committed by GitHub
co-authored by Edward Oakes
parent af1203b9df
commit 56d934bc18
24 changed files with 834 additions and 636 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/examples/plot_example-lm.rst
+4 -4
View File
@@ -4,18 +4,19 @@ Fault-Tolerant Fairseq Training
This document provides a walkthrough of adapting the `Fairseq library <https://github.com/pytorch/fairseq>`__ to perform fault-tolerant distributed training on AWS.
As an example, we use the WikiText-103 dataset to pretrain the RoBERTa model following `this tutorial <https://github.com/pytorch/fairseq/blob/master/examples/roberta/README.pretraining.md>`__. The pipeline and configurations in this document will work for other models supported by Fairseq, such as sequence-to-sequence machine translation models.
To run this example, you will need to install Ray on your local machine to use Ray Autoscaler.
To run this example, you will need to install Ray on your local machine to use the Ray cluster launcher.
You can view the `code for this example`_.
.. _`code for this example`: https://github.com/ray-project/ray/tree/master/doc/examples/lm
To use Ray Autoscaler on AWS, install boto (``pip install boto3``) and configure your AWS credentials in ``~/.aws/credentials`` as described on `Automatic Cluster Setup page <../autoscaling.html>`__. We provide an `example config file <https://github.com/ray-project/ray/tree/master/doc/examples/lm/lm-cluster.yaml>`__ (``lm-cluster.yaml``).
To use Ray cluster launcher on AWS, install boto (``pip install boto3``) and configure your AWS credentials in ``~/.aws/credentials`` as described on the :ref:`Automatic Cluster Setup page <ref-automatic-cluster>`.
We provide an `example config file <https://github.com/ray-project/ray/tree/master/doc/examples/lm/lm-cluster.yaml>`__ (``lm-cluster.yaml``).
In the example config file, we use an ``m5.xlarge`` on-demand instance as the head node, and use ``p3.2xlarge`` GPU spot instances as the worker nodes. We set the minimal number of workers to 1 and maximum workers to 2 in the config, which can be modified according to your own demand.
We also mount `Amazon EFS <../autoscaling.html#using-amazon-efs>`__ to store code, data and checkpoints.
We also mount :ref:`Amazon EFS <aws-cluster-efs>` to store code, data and checkpoints.
.. note::
@@ -308,4 +309,3 @@ To test the fault-tolerance, you can run the following command on your local mac
.. code-block:: bash
ray kill-random-node lm-cluster.yaml
doc/source/autoscaling.rst
-446
View File
@@ -1,446 +0,0 @@
.. _ref-automatic-cluster:
Automatic Cluster Setup
=======================
Ray comes with a built-in autoscaler that makes deploying a Ray cluster simple, just run ``ray up`` from your local machine to start or update a cluster in the cloud or on an on-premise cluster. Once the Ray cluster is running, you can manually SSH into it or use provided commands like ``ray attach``, ``ray rsync-up``, and ``ray-exec`` to access it and run Ray programs.
Setup
-----
This section provides instructions for configuring the autoscaler to launch a Ray cluster on AWS/Azure/GCP, an existing Kubernetes cluster, or on a private cluster of host machines.
Once you have finished configuring the autoscaler to create a cluster, see the Quickstart guide below for more details on how to get started running Ray programs on it.
AWS
~~~
First, install boto (``pip install boto3``) and configure your AWS credentials in ``~/.aws/credentials``,
as described in `the boto docs <http://boto3.readthedocs.io/en/latest/guide/configuration.html>`__.
Once boto is configured to manage resources on your AWS account, you should be ready to run the autoscaler. The provided `ray/python/ray/autoscaler/aws/example-full.yaml <https://github.com/ray-project/ray/tree/master/python/ray/autoscaler/aws/example-full.yaml>`__ cluster config file will create a small cluster with an m5.large head node (on-demand) configured to autoscale up to two m5.large `spot workers <https://aws.amazon.com/ec2/spot/>`__.
Test that it works by running the following commands from your local machine:
.. code-block:: bash
# Create or update the cluster. When the command finishes, it will print
# out the command that can be used to SSH into the cluster head node.
$ ray up ray/python/ray/autoscaler/aws/example-full.yaml
# Get a remote screen on the head node.
$ ray attach ray/python/ray/autoscaler/aws/example-full.yaml
$ source activate tensorflow_p36
$ # Try running a Ray program with 'ray.init(address="auto")'.
# Tear down the cluster.
$ ray down ray/python/ray/autoscaler/aws/example-full.yaml
.. tip:: For the AWS node configuration, you can set ``"ImageId: latest_dlami"`` to automatically use the newest `Deep Learning AMI <https://aws.amazon.com/machine-learning/amis/>`_ for your region. For example, ``head_node: {InstanceType: c5.xlarge, ImageId: latest_dlami}``.
Azure
~~~~~
First, install the Azure CLI (``pip install azure-cli azure-core``) then login using (``az login``).
Set the subscription to use from the command line (``az account set -s <subscription_id>``) or by modifying the provider section of the config provided e.g: `ray/python/ray/autoscaler/azure/example-full.yaml`
Once the Azure CLI is configured to manage resources on your Azure account, you should be ready to run the autoscaler. The provided `ray/python/ray/autoscaler/azure/example-full.yaml <https://github.com/ray-project/ray/tree/master/python/ray/autoscaler/azure/example-full.yaml>`__ cluster config file will create a small cluster with a Standard DS2v3 head node (on-demand) configured to autoscale up to two Standard DS2v3 `spot workers <https://docs.microsoft.com/en-us/azure/virtual-machines/windows/spot-vms>`__. Note that you'll need to fill in your resource group and location in those templates.
Test that it works by running the following commands from your local machine:
.. code-block:: bash
# Create or update the cluster. When the command finishes, it will print
# out the command that can be used to SSH into the cluster head node.
$ ray up ray/python/ray/autoscaler/azure/example-full.yaml
# Get a remote screen on the head node.
$ ray attach ray/python/ray/autoscaler/azure/example-full.yaml
# test ray setup
# enable conda environment
$ exec bash -l
$ conda activate py37_tensorflow
$ python -c 'import ray; ray.init()'
$ exit
# Tear down the cluster.
$ ray down ray/python/ray/autoscaler/azure/example-full.yaml
Azure Portal
~~~~~~~~~~~~
Alternatively, you can deploy a cluster using Azure portal directly. Please note that auto scaling is done using Azure VM Scale Sets and not through
the Ray autoscaler. This will deploy `Azure Data Science VMs (DSVM) <https://azure.microsoft.com/en-us/services/virtual-machines/data-science-virtual-machines/>`_
for both the head node and the auto-scalable cluster managed by `Azure Virtual Machine Scale Sets <https://azure.microsoft.com/en-us/services/virtual-machine-scale-sets/>`_.
The head node conveniently exposes both SSH as well as JupyterLab.
.. image:: https://aka.ms/deploytoazurebutton
:target: https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2Fray-project%2Fray%2Fmaster%2Fdoc%2Fazure%2Fazure-ray-template.json
:alt: Deploy to Azure
Once the template is successfully deployed the deployment output page provides the ssh command to connect and the link to the JupyterHub on the head node (username/password as specified on the template input).
Use the following code in a Jupyter notebook to connect to the Ray cluster.
.. code-block:: python
import ray
ray.init(address='auto')
Note that on each node the `azure-init.sh <https://github.com/ray-project/ray/blob/master/doc/azure/azure-init.sh>`_ script is executed and performs the following actions:
1. Activates one of the conda environments available on DSVM
2. Installs Ray and any other user-specified dependencies
3. Sets up a systemd task (``/lib/systemd/system/ray.service``) to start Ray in head or worker mode
GCP
~~~
First, install the Google API client (``pip install google-api-python-client``), set up your GCP credentials, and create a new GCP project.
Once the API client is configured to manage resources on your GCP account, you should be ready to run the autoscaler. The provided `ray/python/ray/autoscaler/gcp/example-full.yaml <https://github.com/ray-project/ray/tree/master/python/ray/autoscaler/gcp/example-full.yaml>`__ cluster config file will create a small cluster with a n1-standard-2 head node (on-demand) configured to autoscale up to two n1-standard-2 `preemptible workers <https://cloud.google.com/preemptible-vms/>`__. Note that you'll need to fill in your project id in those templates.
Test that it works by running the following commands from your local machine:
.. code-block:: bash
# Create or update the cluster. When the command finishes, it will print
# out the command that can be used to SSH into the cluster head node.
$ ray up ray/python/ray/autoscaler/gcp/example-full.yaml
# Get a remote screen on the head node.
$ ray attach ray/python/ray/autoscaler/gcp/example-full.yaml
$ source activate tensorflow_p36
$ # Try running a Ray program with 'ray.init(address="auto")'.
# Tear down the cluster.
$ ray down ray/python/ray/autoscaler/gcp/example-full.yaml
Kubernetes
~~~~~~~~~~
The autoscaler can also be used to start Ray clusters on an existing Kubernetes cluster. First, install the Kubernetes API client (``pip install kubernetes``), then make sure your Kubernetes credentials are set up properly to access the cluster (if a command like ``kubectl get pods`` succeeds, you should be good to go).
Once you have ``kubectl`` configured locally to access the remote cluster, you should be ready to run the autoscaler. The provided `ray/python/ray/autoscaler/kubernetes/example-full.yaml <https://github.com/ray-project/ray/tree/master/python/ray/autoscaler/kubernetes/example-full.yaml>`__ cluster config file will create a small cluster of one pod for the head node configured to autoscale up to two worker node pods, with all pods requiring 1 CPU and 0.5GiB of memory.
Test that it works by running the following commands from your local machine:
.. code-block:: bash
# Create or update the cluster. When the command finishes, it will print
# out the command that can be used to get a remote shell into the head node.
$ ray up ray/python/ray/autoscaler/kubernetes/example-full.yaml
# List the pods running in the cluster. You shoud only see one head node
# until you start running an application, at which point worker nodes
# should be started. Don't forget to include the Ray namespace in your
# 'kubectl' commands ('ray' by default).
$ kubectl -n ray get pods
# Get a remote screen on the head node.
$ ray attach ray/python/ray/autoscaler/kubernetes/example-full.yaml
$ # Try running a Ray program with 'ray.init(address="auto")'.
# Tear down the cluster
$ ray down ray/python/ray/autoscaler/kubernetes/example-full.yaml
Private Cluster
~~~~~~~~~~~~~~~
The autoscaler can also be used to run a Ray cluster on a private cluster of hosts, specified as a list of machine IP addresses to connect to. You can get started by filling out the fields in the provided `ray/python/ray/autoscaler/local/example-full.yaml <https://github.com/ray-project/ray/tree/master/python/ray/autoscaler/local/example-full.yaml>`__.
Be sure to specify the proper ``head_ip``, list of ``worker_ips``, and the ``ssh_user`` field.
Test that it works by running the following commands from your local machine:
.. code-block:: bash
# Create or update the cluster. When the command finishes, it will print
# out the command that can be used to get a remote shell into the head node.
$ ray up ray/python/ray/autoscaler/local/example-full.yaml
# Get a remote screen on the head node.
$ ray attach ray/python/ray/autoscaler/local/example-full.yaml
$ # Try running a Ray program with 'ray.init(address="auto")'.
# Tear down the cluster
$ ray down ray/python/ray/autoscaler/local/example-full.yaml
External Node Provider
~~~~~~~~~~~~~~~~~~~~~~
Ray also supports external node providers (check `node_provider.py <https://github.com/ray-project/ray/tree/master/python/ray/autoscaler/node_provider.py>`__ implementation).
You can specify the external node provider using the yaml config:
.. code-block:: yaml
provider:
type: external
module: mypackage.myclass
The module needs to be in the format `package.provider_class` or `package.sub_package.provider_class`.
Additional Cloud Providers
~~~~~~~~~~~~~~~~~~~~~~~~~~
To use Ray autoscaling on other Cloud providers or cluster management systems, you can implement the ``NodeProvider`` interface (~100 LOC) and register it in `node_provider.py <https://github.com/ray-project/ray/tree/master/python/ray/autoscaler/node_provider.py>`__. Contributions are welcome!
Quickstart
----------
Starting and updating a cluster
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
When you run ``ray up`` with an existing cluster, the command checks if the local configuration differs from the applied configuration of the cluster. This includes any changes to synced files specified in the ``file_mounts`` section of the config. If so, the new files and config will be uploaded to the cluster. Following that, Ray services will be restarted.
You can also run ``ray up`` to restart a cluster if it seems to be in a bad state (this will restart all Ray services even if there are no config changes).
If you don't want the update to restart services (e.g., because the changes don't require a restart), pass ``--no-restart`` to the update call.
.. code-block:: bash
# Replace '<your_backend>' with one of: 'aws', 'gcp', 'kubernetes', or 'local'.
$ BACKEND=<your_backend>
# Create or update the cluster.
$ ray up ray/python/ray/autoscaler/$BACKEND/example-full.yaml
# Reconfigure autoscaling behavior without interrupting running jobs.
$ ray up ray/python/ray/autoscaler/$BACKEND/example-full.yaml \
--max-workers=N --no-restart
# Tear down the cluster.
$ ray down ray/python/ray/autoscaler/$BACKEND/example-full.yaml
Running commands on new and existing clusters
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You can use ``ray exec`` to conveniently run commands on clusters. Note that scripts you run should connect to Ray via ``ray.init(address="auto")``.
.. code-block:: bash
# Run a command on the cluster
$ ray exec cluster.yaml 'echo "hello world"'
# Run a command on the cluster, starting it if needed
$ ray exec cluster.yaml 'echo "hello world"' --start
# Run a command on the cluster, stopping the cluster after it finishes
$ ray exec cluster.yaml 'echo "hello world"' --stop
# Run a command on a new cluster called 'experiment-1', stopping it after
$ ray exec cluster.yaml 'echo "hello world"' \
--start --stop --cluster-name experiment-1
# Run a command in a detached tmux session
$ ray exec cluster.yaml 'echo "hello world"' --tmux
# Run a command in a screen (experimental)
$ ray exec cluster.yaml 'echo "hello world"' --screen
You can also use ``ray submit`` to execute Python scripts on clusters. This will ``rsync`` the designated file onto the cluster and execute it with the given arguments.
.. code-block:: bash
# Run a Python script in a detached tmux session
$ ray submit cluster.yaml --tmux --start --stop tune_experiment.py
Attaching to a running cluster
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You can use ``ray attach`` to attach to an interactive screen session on the cluster.
.. code-block:: bash
# Open a screen on the cluster
$ ray attach cluster.yaml
# Open a screen on a new cluster called 'session-1'
$ ray attach cluster.yaml --start --cluster-name=session-1
# Attach to tmux session on cluster (creates a new one if none available)
$ ray attach cluster.yaml --tmux
Port-forwarding applications
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If you want to run applications on the cluster that are accessible from a web browser (e.g., Jupyter notebook), you can use the ``--port-forward`` option for ``ray exec``. The local port opened is the same as the remote port.
Note: For Kubernetes clusters, the ``port-forward`` option cannot be used while executing a command. To port forward and run a command you need to call ``ray exec`` twice separately.
.. code-block:: bash
$ ray exec cluster.yaml --port-forward=8899 'source ~/anaconda3/bin/activate tensorflow_p36 && jupyter notebook --port=8899'
Manually synchronizing files
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To download or upload files to the cluster head node, use ``ray rsync_down`` or ``ray rsync_up``:
.. code-block:: bash
$ ray rsync_down cluster.yaml '/path/on/cluster' '/local/path'
$ ray rsync_up cluster.yaml '/local/path' '/path/on/cluster'
Security
~~~~~~~~
On cloud providers, nodes will be launched into their own security group by default, with traffic allowed only between nodes in the same group. A new SSH key will also be created and saved to your local machine for access to the cluster.
Autoscaling
~~~~~~~~~~~
Ray clusters come with a load-based autoscaler. When cluster resource usage exceeds a configurable threshold (80% by default), new nodes will be launched up the specified ``max_workers`` limit. When nodes are idle for more than a timeout, they will be removed, down to the ``min_workers`` limit. The head node is never removed.
The default idle timeout is 5 minutes. This is to prevent excessive node churn which could impact performance and increase costs (in AWS / GCP there is a minimum billing charge of 1 minute per instance, after which usage is billed by the second).
Monitoring cluster status
~~~~~~~~~~~~~~~~~~~~~~~~~
The ray also comes with an online dashboard. The dashboard is accessible via HTTP on the head node (by default it listens on ``localhost:8265``). To access it locally, you'll need to forward the port to your local machine. You can also use the built-in ``ray dashboard`` to do this automatically.
You can monitor cluster usage and auto-scaling status by tailing the autoscaling
logs in ``/tmp/ray/session_*/logs/monitor*``.
The Ray autoscaler also reports per-node status in the form of instance tags. In your cloud provider console, you can click on a Node, go the the "Tags" pane, and add the ``ray-node-status`` tag as a column. This lets you see per-node statuses at a glance:
.. image:: autoscaler-status.png
Customizing cluster setup
~~~~~~~~~~~~~~~~~~~~~~~~~
You are encouraged to copy the example YAML file and modify it to your needs. This may include adding additional setup commands to install libraries or sync local data files.
.. note:: After you have customized the nodes, it is also a good idea to create a new machine image (or docker container) and use that in the config file. This reduces worker setup time, improving the efficiency of auto-scaling.
The setup commands you use should ideally be *idempotent*, that is, can be run more than once. This allows Ray to update nodes after they have been created. You can usually make commands idempotent with small modifications, e.g. ``git clone foo`` can be rewritten as ``test -e foo || git clone foo`` which checks if the repo is already cloned first.
Most of the example YAML file is optional. Here is a `reference minimal YAML file <https://github.com/ray-project/ray/tree/master/python/ray/autoscaler/aws/example-minimal.yaml>`__, and you can find the defaults for `optional fields in this YAML file <https://github.com/ray-project/ray/tree/master/python/ray/autoscaler/aws/example-full.yaml>`__.
Syncing git branches
~~~~~~~~~~~~~~~~~~~~
A common use case is syncing a particular local git branch to all workers of the cluster. However, if you just put a `git checkout <branch>` in the setup commands, the autoscaler won't know when to rerun the command to pull in updates. There is a nice workaround for this by including the git SHA in the input (the hash of the file will change if the branch is updated):
.. code-block:: yaml
file_mounts: {
"/tmp/current_branch_sha": "/path/to/local/repo/.git/refs/heads/<YOUR_BRANCH_NAME>",
}
setup_commands:
- test -e <REPO_NAME> || git clone https://github.com/<REPO_ORG>/<REPO_NAME>.git
- cd <REPO_NAME> && git fetch && git checkout `cat /tmp/current_branch_sha`
This tells ``ray up`` to sync the current git branch SHA from your personal computer to a temporary file on the cluster (assuming you've pushed the branch head already). Then, the setup commands read that file to figure out which SHA they should checkout on the nodes. Note that each command runs in its own session. The final workflow to update the cluster then becomes just this:
1. Make local changes to a git branch
2. Commit the changes with ``git commit`` and ``git push``
3. Update files on your Ray cluster with ``ray up``
Using Amazon EFS
~~~~~~~~~~~~~~~~
To use Amazon EFS, install some utilities and mount the EFS in ``setup_commands``. Note that these instructions only work if you are using the AWS Autoscaler.
.. note::
You need to replace the ``{{FileSystemId}}`` to your own EFS ID before using the config. You may also need to set correct ``SecurityGroupIds`` for the instances in the config file.
.. code-block:: yaml
setup_commands:
- sudo kill -9 `sudo lsof /var/lib/dpkg/lock-frontend | awk '{print $2}' | tail -n 1`;
sudo pkill -9 apt-get;
sudo pkill -9 dpkg;
sudo dpkg --configure -a;
sudo apt-get -y install binutils;
cd $HOME;
git clone https://github.com/aws/efs-utils;
cd $HOME/efs-utils;
./build-deb.sh;
sudo apt-get -y install ./build/amazon-efs-utils*deb;
cd $HOME;
mkdir efs;
sudo mount -t efs {{FileSystemId}}:/ efs;
sudo chmod 777 efs;
Common cluster configurations
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The ``example-full.yaml`` configuration is enough to get started with Ray, but for more compute intensive workloads you will want to change the instance types to e.g. use GPU or larger compute instance by editing the yaml file. Here are a few common configurations:
**GPU single node**: use Ray on a single large GPU instance.
.. code-block:: yaml
max_workers: 0
head_node:
InstanceType: p2.8xlarge
**Docker**: Specify docker image. This executes all commands on all nodes in the docker container,
and opens all the necessary ports to support the Ray cluster. It will also automatically install
Docker if Docker is not installed. This currently does not have GPU support.
.. code-block:: yaml
docker:
image: tensorflow/tensorflow:1.5.0-py3
container_name: ray_docker
**Mixed GPU and CPU nodes**: for RL applications that require proportionally more
CPU than GPU resources, you can use additional CPU workers with a GPU head node.
.. code-block:: yaml
max_workers: 10
head_node:
InstanceType: p2.8xlarge
worker_nodes:
InstanceType: m4.16xlarge
**Autoscaling CPU cluster**: use a small head node and have Ray auto-scale
workers as needed. This can be a cost-efficient configuration for clusters with
bursty workloads. You can also request spot workers for additional cost savings.
.. code-block:: yaml
min_workers: 0
max_workers: 10
head_node:
InstanceType: m4.large
worker_nodes:
InstanceMarketOptions:
MarketType: spot
InstanceType: m4.16xlarge
**Autoscaling GPU cluster**: similar to the autoscaling CPU cluster, but
with GPU worker nodes instead.
.. code-block:: yaml
min_workers: 0 # NOTE: older Ray versions may need 1+ GPU workers (#2106)
max_workers: 10
head_node:
InstanceType: m4.large
worker_nodes:
InstanceMarketOptions:
MarketType: spot
InstanceType: p2.xlarge
Questions or Issues?
~~~~~~~~~~~~~~~~~~~~
You can post questions or issues or feedback through the following channels:
1. `ray-dev@googlegroups.com`_: For discussions about development or any general
questions and feedback.
2. `StackOverflow`_: For questions about how to use Ray.
3. `GitHub Issues`_: For bug reports and feature requests.
.. _`ray-dev@googlegroups.com`: https://groups.google.com/forum/#!forum/ray-dev
.. _`StackOverflow`: https://stackoverflow.com/questions/tagged/ray
.. _`GitHub Issues`: https://github.com/ray-project/ray/issues
doc/source/cluster-index.rst
-15
View File
@@ -1,15 +0,0 @@
Deploying Ray
=============
How to setup your cluster and use Ray most effectively.
.. toctree::
:maxdepth: 2
:caption: Cluster Setup
autoscaling.rst
using-ray-on-a-cluster.rst
deploy-on-yarn.rst
deploy-on-kubernetes.rst
deploying-on-slurm.rst
projects.rst
doc/source/cluster/cloud.rst
+230
View File
@@ -0,0 +1,230 @@
.. _cluster-cloud:
Launching clusters on the cloud
===============================
This section provides instructions for configuring the Ray Cluster Launcher to use with AWS/Azure/GCP, an existing Kubernetes cluster, or on a private cluster of host machines.
.. contents::
:local:
:backlinks: none
See this blog post for a `step by step guide`_ to using the Ray Cluster Launcher.
.. _`step by step guide`: https://medium.com/distributed-computing-with-ray/a-step-by-step-guide-to-scaling-your-first-python-application-in-the-cloud-8761fe331ef1
AWS (EC2)
---------
First, install boto (``pip install boto3``) and configure your AWS credentials in ``~/.aws/credentials``,
as described in `the boto docs <http://boto3.readthedocs.io/en/latest/guide/configuration.html>`__.
Once boto is configured to manage resources on your AWS account, you should be ready to launch your cluster. The provided `ray/python/ray/autoscaler/aws/example-full.yaml <https://github.com/ray-project/ray/tree/master/python/ray/autoscaler/aws/example-full.yaml>`__ cluster config file will create a small cluster with an m5.large head node (on-demand) configured to autoscale up to two m5.large `spot workers <https://aws.amazon.com/ec2/spot/>`__.
Test that it works by running the following commands from your local machine:
.. code-block:: bash
# Create or update the cluster. When the command finishes, it will print
# out the command that can be used to SSH into the cluster head node.
$ ray up ray/python/ray/autoscaler/aws/example-full.yaml
# Get a remote screen on the head node.
$ ray attach ray/python/ray/autoscaler/aws/example-full.yaml
$ source activate tensorflow_p36
$ # Try running a Ray program with 'ray.init(address="auto")'.
# Tear down the cluster.
$ ray down ray/python/ray/autoscaler/aws/example-full.yaml
.. tip:: For the AWS node configuration, you can set ``"ImageId: latest_dlami"`` to automatically use the newest `Deep Learning AMI <https://aws.amazon.com/machine-learning/amis/>`_ for your region. For example, ``head_node: {InstanceType: c5.xlarge, ImageId: latest_dlami}``.
.. _aws-cluster-efs:
Using Amazon EFS
~~~~~~~~~~~~~~~~
To use Amazon EFS, install some utilities and mount the EFS in ``setup_commands``. Note that these instructions only work if you are using the AWS Autoscaler.
.. note::
You need to replace the ``{{FileSystemId}}`` to your own EFS ID before using the config. You may also need to set correct ``SecurityGroupIds`` for the instances in the config file.
.. code-block:: yaml
setup_commands:
- sudo kill -9 `sudo lsof /var/lib/dpkg/lock-frontend | awk '{print $2}' | tail -n 1`;
sudo pkill -9 apt-get;
sudo pkill -9 dpkg;
sudo dpkg --configure -a;
sudo apt-get -y install binutils;
cd $HOME;
git clone https://github.com/aws/efs-utils;
cd $HOME/efs-utils;
./build-deb.sh;
sudo apt-get -y install ./build/amazon-efs-utils*deb;
cd $HOME;
mkdir efs;
sudo mount -t efs {{FileSystemId}}:/ efs;
sudo chmod 777 efs;
Azure
-----
First, install the Azure CLI (``pip install azure-cli azure-core``) then login using (``az login``).
Set the subscription to use from the command line (``az account set -s <subscription_id>``) or by modifying the provider section of the config provided e.g: `ray/python/ray/autoscaler/azure/example-full.yaml`
Once the Azure CLI is configured to manage resources on your Azure account, you should be ready to launch your cluster. The provided `ray/python/ray/autoscaler/azure/example-full.yaml <https://github.com/ray-project/ray/tree/master/python/ray/autoscaler/azure/example-full.yaml>`__ cluster config file will create a small cluster with a Standard DS2v3 head node (on-demand) configured to autoscale up to two Standard DS2v3 `spot workers <https://docs.microsoft.com/en-us/azure/virtual-machines/windows/spot-vms>`__. Note that you'll need to fill in your resource group and location in those templates.
Test that it works by running the following commands from your local machine:
.. code-block:: bash
# Create or update the cluster. When the command finishes, it will print
# out the command that can be used to SSH into the cluster head node.
$ ray up ray/python/ray/autoscaler/azure/example-full.yaml
# Get a remote screen on the head node.
$ ray attach ray/python/ray/autoscaler/azure/example-full.yaml
# test ray setup
# enable conda environment
$ exec bash -l
$ conda activate py37_tensorflow
$ python -c 'import ray; ray.init()'
$ exit
# Tear down the cluster.
$ ray down ray/python/ray/autoscaler/azure/example-full.yaml
Azure Portal
------------
Alternatively, you can deploy a cluster using Azure portal directly. Please note that autoscaling is done using Azure VM Scale Sets and not through
the Ray autoscaler. This will deploy `Azure Data Science VMs (DSVM) <https://azure.microsoft.com/en-us/services/virtual-machines/data-science-virtual-machines/>`_
for both the head node and the auto-scalable cluster managed by `Azure Virtual Machine Scale Sets <https://azure.microsoft.com/en-us/services/virtual-machine-scale-sets/>`_.
The head node conveniently exposes both SSH as well as JupyterLab.
.. image:: https://aka.ms/deploytoazurebutton
:target: https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2Fray-project%2Fray%2Fmaster%2Fdoc%2Fazure%2Fazure-ray-template.json
:alt: Deploy to Azure
Once the template is successfully deployed the deployment output page provides the ssh command to connect and the link to the JupyterHub on the head node (username/password as specified on the template input).
Use the following code in a Jupyter notebook to connect to the Ray cluster.
.. code-block:: python
import ray
ray.init(address='auto')
Note that on each node the `azure-init.sh <https://github.com/ray-project/ray/blob/master/doc/azure/azure-init.sh>`_ script is executed and performs the following actions:
1. Activates one of the conda environments available on DSVM
2. Installs Ray and any other user-specified dependencies
3. Sets up a systemd task (``/lib/systemd/system/ray.service``) to start Ray in head or worker mode
GCP
---
First, install the Google API client (``pip install google-api-python-client``), set up your GCP credentials, and create a new GCP project.
Once the API client is configured to manage resources on your GCP account, you should be ready to launch your cluster. The provided `ray/python/ray/autoscaler/gcp/example-full.yaml <https://github.com/ray-project/ray/tree/master/python/ray/autoscaler/gcp/example-full.yaml>`__ cluster config file will create a small cluster with a n1-standard-2 head node (on-demand) configured to autoscale up to two n1-standard-2 `preemptible workers <https://cloud.google.com/preemptible-vms/>`__. Note that you'll need to fill in your project id in those templates.
Test that it works by running the following commands from your local machine:
.. code-block:: bash
# Create or update the cluster. When the command finishes, it will print
# out the command that can be used to SSH into the cluster head node.
$ ray up ray/python/ray/autoscaler/gcp/example-full.yaml
# Get a remote screen on the head node.
$ ray attach ray/python/ray/autoscaler/gcp/example-full.yaml
$ source activate tensorflow_p36
$ # Try running a Ray program with 'ray.init(address="auto")'.
# Tear down the cluster.
$ ray down ray/python/ray/autoscaler/gcp/example-full.yaml
.. _ray-launch-k8s:
Kubernetes
----------
The cluster launcher can also be used to start Ray clusters on an existing Kubernetes cluster. First, install the Kubernetes API client (``pip install kubernetes``), then make sure your Kubernetes credentials are set up properly to access the cluster (if a command like ``kubectl get pods`` succeeds, you should be good to go).
Once you have ``kubectl`` configured locally to access the remote cluster, you should be ready to launch your cluster. The provided `ray/python/ray/autoscaler/kubernetes/example-full.yaml <https://github.com/ray-project/ray/tree/master/python/ray/autoscaler/kubernetes/example-full.yaml>`__ cluster config file will create a small cluster of one pod for the head node configured to autoscale up to two worker node pods, with all pods requiring 1 CPU and 0.5GiB of memory.
Test that it works by running the following commands from your local machine:
.. code-block:: bash
# Create or update the cluster. When the command finishes, it will print
# out the command that can be used to get a remote shell into the head node.
$ ray up ray/python/ray/autoscaler/kubernetes/example-full.yaml
# List the pods running in the cluster. You shoud only see one head node
# until you start running an application, at which point worker nodes
# should be started. Don't forget to include the Ray namespace in your
# 'kubectl' commands ('ray' by default).
$ kubectl -n ray get pods
# Get a remote screen on the head node.
$ ray attach ray/python/ray/autoscaler/kubernetes/example-full.yaml
$ # Try running a Ray program with 'ray.init(address="auto")'.
# Tear down the cluster
$ ray down ray/python/ray/autoscaler/kubernetes/example-full.yaml
.. tip:: This section describes the easiest way to launch a Ray cluster on Kubernetes. See this :ref:`document for advanced usage <ray-k8s-deploy>` of Kubernetes with Ray.
.. _cluster-private-setup:
Private Cluster (List of nodes)
-------------------------------
The most preferable way to run a Ray cluster on a private cluster of hosts is via the Ray Cluster Launcher.
You can get started by filling out the fields in the provided `ray/python/ray/autoscaler/local/example-full.yaml <https://github.com/ray-project/ray/tree/master/python/ray/autoscaler/local/example-full.yaml>`__.
Be sure to specify the proper ``head_ip``, list of ``worker_ips``, and the ``ssh_user`` field.
Test that it works by running the following commands from your local machine:
.. code-block:: bash
# Create or update the cluster. When the command finishes, it will print
# out the command that can be used to get a remote shell into the head node.
$ ray up ray/python/ray/autoscaler/local/example-full.yaml
# Get a remote screen on the head node.
$ ray attach ray/python/ray/autoscaler/local/example-full.yaml
$ # Try running a Ray program with 'ray.init(address="auto")'.
# Tear down the cluster
$ ray down ray/python/ray/autoscaler/local/example-full.yaml
External Node Provider
----------------------
Ray also supports external node providers (check `node_provider.py <https://github.com/ray-project/ray/tree/master/python/ray/autoscaler/node_provider.py>`__ implementation).
You can specify the external node provider using the yaml config:
.. code-block:: yaml
provider:
type: external
module: mypackage.myclass
The module needs to be in the format `package.provider_class` or `package.sub_package.provider_class`.
Additional Cloud Providers
--------------------------
To use Ray autoscaling on other Cloud providers or cluster management systems, you can implement the ``NodeProvider`` interface (100 LOC) and register it in `node_provider.py <https://github.com/ray-project/ray/tree/master/python/ray/autoscaler/node_provider.py>`__. Contributions are welcome!
Security
--------
On cloud providers, nodes will be launched into their own security group by default, with traffic allowed only between nodes in the same group. A new SSH key will also be created and saved to your local machine for access to the cluster.
doc/source/cluster/config.rst
+113
View File
@@ -0,0 +1,113 @@
.. _cluster-config:
Configuring the Cluster
=======================
The Ray Cluster Launcher requires a *cluster configuration file*, which specifies some important details about the cluster. At a minimum, we need to specify:
* the name of our cluster,
* the number of workers in the cluster
* the cloud provider
* any setup commands that should run on the node upon launch.
Here is an example cluster configuration file:
.. code-block:: yaml
# A unique identifier for this cluster.
cluster_name: basic-ray
# The maximum number of workers nodes to launch in addition to the head
# node.
max_workers: 0 # this means zero workers
# Cloud-provider specific configuration.
provider:
type: aws
region: us-west-2
availability_zone: us-west-2a
# How Ray will authenticate with newly launched nodes.
auth:
ssh_user: ubuntu
setup_commands:
- pip install ray[all]
# The following line demonstrate that you can specify arbitrary
# startup scripts on the cluster.
- touch /tmp/some_file.txt
Most of the example YAML file is optional. Here is a `reference minimal YAML file <https://github.com/ray-project/ray/tree/master/python/ray/autoscaler/aws/example-minimal.yaml>`__, and you can find the defaults for `optional fields in this YAML file <https://github.com/ray-project/ray/tree/master/python/ray/autoscaler/aws/example-full.yaml>`__.
You are encouraged to copy the example YAML file and modify it to your needs. This may include adding additional setup commands to install libraries or sync local data files.
Setup Commands
--------------
.. note:: After you have customized the nodes, it is also a good idea to create a new machine image (or docker container) and use that in the config file. This reduces worker setup time, improving the efficiency of auto-scaling.
The setup commands you use should ideally be *idempotent*, that is, can be run more than once. This allows Ray to update nodes after they have been created. You can usually make commands idempotent with small modifications, e.g. ``git clone foo`` can be rewritten as ``test -e foo || git clone foo`` which checks if the repo is already cloned first.
Common cluster configurations
-----------------------------
The ``example-full.yaml`` configuration is enough to get started with Ray, but for more compute intensive workloads you will want to change the instance types to e.g. use GPU or larger compute instance by editing the yaml file. Here are a few common configurations:
**GPU single node**: use Ray on a single large GPU instance.
.. code-block:: yaml
max_workers: 0
head_node:
InstanceType: p2.8xlarge
**Docker**: Specify docker image. This executes all commands on all nodes in the docker container,
and opens all the necessary ports to support the Ray cluster. It will also automatically install
Docker if Docker is not installed. This currently does not have GPU support.
.. code-block:: yaml
docker:
image: tensorflow/tensorflow:1.5.0-py3
container_name: ray_docker
**Mixed GPU and CPU nodes**: for RL applications that require proportionally more
CPU than GPU resources, you can use additional CPU workers with a GPU head node.
.. code-block:: yaml
max_workers: 10
head_node:
InstanceType: p2.8xlarge
worker_nodes:
InstanceType: m4.16xlarge
**Autoscaling CPU cluster**: use a small head node and have Ray auto-scale
workers as needed. This can be a cost-efficient configuration for clusters with
bursty workloads. You can also request spot workers for additional cost savings.
.. code-block:: yaml
min_workers: 0
max_workers: 10
head_node:
InstanceType: m4.large
worker_nodes:
InstanceMarketOptions:
MarketType: spot
InstanceType: m4.16xlarge
**Autoscaling GPU cluster**: similar to the autoscaling CPU cluster, but
with GPU worker nodes instead.
.. code-block:: yaml
min_workers: 0 # NOTE: older Ray versions may need 1+ GPU workers (#2106)
max_workers: 10
head_node:
InstanceType: m4.large
worker_nodes:
InstanceMarketOptions:
MarketType: spot
InstanceType: p2.xlarge
doc/source/cluster/deploy.rst
+16
View File
@@ -0,0 +1,16 @@
.. _ref-cluster-setup:
Ray with Cluster Managers
=========================
.. note::
If you're using AWS, Azure or GCP you can use the :ref:`Ray Cluster Launcher <ref-automatic-cluster>` to simplify the cluster setup process.
.. toctree::
:maxdepth: 2
kubernetes.rst
yarn.rst
slurm.rst
doc/source/cluster/index.rst
+105
View File
@@ -0,0 +1,105 @@
.. _cluster-index:
Multi-node Ray Overview
=======================
One of Ray's strengths is the ability to leverage multiple machines in the same program. You can do this by creating and connecting to a Ray cluster.
Key Concepts
------------
* **Ray Nodes**: A Ray cluster consists of a **head node** and a set of **worker nodes**. The head node needs to be started first, and the worker nodes are given the address of the head node to form the cluster. The Ray cluster itself can also "auto-scale," meaning that it can interact with a Cloud Provider to request or release instances according to application workload.
* **Ports**: Ray processes communicate via TCP ports. When starting a Ray cluster, either on prem or on the cloud, it is important to open the right ports so that Ray functions correctly.
* **Ray Cluster Launcher**: The :ref:`Ray Cluster Launcher <ref-automatic-cluster>` is a simple tool that automatically provisions machines and launches a multi-node Ray cluster. You can use the cluster launcher on GCP, Amazon EC2, Azure, or even Kubernetes.
Starting a Ray cluster
----------------------
Clusters can be started with the :ref:`Cluster Launcher <ref-automatic-cluster>` or :ref:`manually <manual-cluster>`. You can also create a Ray cluster using a standard cluster manager such as :ref:`Kubernetes <ray-k8s-deploy>`, :ref:`YARN <ray-yarn-deploy>`, or :ref:`SLURM <ray-slurm-deploy>`.
Here is an example of using the Cluster Launcher on AWS:
.. code-block:: shell
# First, run `pip install boto3` and `aws configure`
#
# Create or update the cluster. When the command finishes, it will print
# out the command that can be used to SSH into the cluster head node.
$ ray up ray/python/ray/autoscaler/aws/example-full.yaml
Running a Ray program on the Ray cluster
----------------------------------------
To run a distributed Ray program, you'll need to execute your program on the same machine as one of the nodes. For example, start up Python on one of the nodes in the cluster:
.. code-block:: python
import ray
ray.init(address="auto")
.. note:: A common mistake is setting the address to be a cluster node while running the script on your laptop. This will not work because the script needs to be started/executed on one of the Ray nodes.
To verify that the correct number of nodes have joined the cluster, you can run the following.
.. code-block:: python
import time
@ray.remote
def f():
time.sleep(0.01)
return ray.services.get_node_ip_address()
# Get a list of the IP addresses of the nodes that have joined the cluster.
set(ray.get([f.remote() for _ in range(1000)]))
.. _manual-cluster:
Manual Ray Cluster Setup
------------------------
The most preferable way to run a Ray cluster is via the Ray Cluster Launcher. However, it is also possible to start a Ray cluster by hand.
This section assumes that you have a list of machines and that the nodes in the cluster can communicate with each other. It also assumes that Ray is installed
on each machine. To install Ray, follow the `installation instructions`_.
.. _`installation instructions`: http://docs.ray.io/en/latest/installation.html
Starting Ray on each machine
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
On the head node (just choose some node to be the head node), run the following.
If the ``--port`` argument is omitted, Ray will choose port 6379, falling back to a
random port.
.. code-block:: bash
ray start --head --port=6379
The command will print out the address of the Redis server that was started
(and some other address information).
**Then on all of the other nodes**, run the following. Make sure to replace
``<address>`` with the value printed by the command on the head node (it
should look something like ``123.45.67.89:6379``).
.. code-block:: bash
ray start --address=<address>
If you wish to specify that a machine has 10 CPUs and 1 GPU, you can do this
with the flags ``--num-cpus=10`` and ``--num-gpus=1``. See the :ref:`Configuration <configuring-ray>` page for more information.
Now we've started all of the Ray processes on each node Ray. This includes
- Some worker processes on each machine.
- An object store on each machine.
- A raylet on each machine.
- Multiple Redis servers (on the head node).
Stopping Ray
~~~~~~~~~~~~
When you want to stop the Ray processes, run ``ray stop`` on each node.
doc/source/deploy-on-kubernetes.rst → doc/source/cluster/kubernetes.rst
+3 -10
View File
@@ -1,18 +1,11 @@
.. _ray-k8s-deploy:
Deploying on Kubernetes
=======================
.. note::
The easiest way to run a Ray cluster is by using the built-in autoscaler,
which has support for running on top of Kubernetes. Please see the `autoscaler
documentation <autoscaling.html>`__ for details.
.. warning::
Running Ray on Kubernetes is still a work in progress. If you have a
suggestion for how to improve this documentation or want to request a
missing feature, please get in touch using one of the channels in the
`Questions or Issues?`_ section below.
This document is mainly for advanced Kubernetes usage. The easiest way to run a Ray cluster on Kubernetes is by using the built-in Cluster Launcher. Please see the :ref:`Cluster Launcher documentation <ray-launch-k8s>` for details.
This document assumes that you have access to a Kubernetes cluster and have
``kubectl`` installed locally and configured to access the cluster. It will
doc/source/cluster/launcher-usage.rst
+205
View File
@@ -0,0 +1,205 @@
.. _launcher-usage:
Cluster Launcher Usage Guide
============================
To launch a cluster, you must first create a *cluster configuration file*, which specifies some important details about the cluster. Here is an example cluster configuration file for AWS:
.. code-block:: yaml
# A unique identifier for this cluster.
cluster_name: basic-ray
# The maximum number of workers nodes to launch in addition to the head
# node.
max_workers: 0 # this means zero workers
# Cloud-provider specific configuration.
provider:
type: aws
region: us-west-2
availability_zone: us-west-2a
# How Ray will authenticate with newly launched nodes.
auth:
ssh_user: ubuntu
setup_commands:
- pip install ray[all]
# The following line demonstrate that you can specify arbitrary
# startup scripts on the cluster.
- touch /tmp/some_file.txt
In another example, the `AWS example configuration file <https://github.com/ray-project/ray/tree/master/python/ray/autoscaler/aws/example-full.yaml>`__ cluster config file will create a small cluster with an m5.large head node (on-demand) configured to autoscale up to two m5.large `spot workers <https://aws.amazon.com/ec2/spot/>`__.
See the :ref:`Cluster Configuration <cluster-config>` docs on how to customize the configuration file.
Here are the commands covered in this document:
.. contents::
:local:
:backlinks: none
Launching a cluster (``ray up``)
--------------------------------
This will start up the machines in the cloud, install your dependencies and run any setup commands that you have, configure the Ray cluster automatically, and prepare you to scale your distributed system. See :ref:`the documentation <ray-up-doc>` for ``ray up``.
.. tip:: The worker nodes will start only after the head node has finished starting. To monitor the progress of the cluster setup, you can run `ray monitor <cluster yaml>`.
.. code-block:: shell
# Replace '<your_backend>' with one of: 'aws', 'gcp', 'kubernetes', or 'local'.
$ BACKEND=<your_backend>
# Create or update the cluster.
$ ray up ray/python/ray/autoscaler/$BACKEND/example-full.yaml
# Tear down the cluster.
$ ray down ray/python/ray/autoscaler/$BACKEND/example-full.yaml
Updating an existing cluster (``ray up``)
-----------------------------------------
If you want to update your cluster configuration (add more files, change dependencies), run ``ray up`` again on the existing cluster.
This command checks if the local configuration differs from the applied configuration of the cluster. This includes any changes to synced files specified in the ``file_mounts`` section of the config. If so, the new files and config will be uploaded to the cluster. Following that, Ray services/processes will be restarted.
.. tip:: Don't do this for the cloud provider specifications (e.g., change from AWS to GCP on a running cluster) or change the cluster name (as this will just start a new cluster and orphan the original one).
You can also run ``ray up`` to restart a cluster if it seems to be in a bad state (this will restart all Ray services even if there are no config changes).
If you don't want the update to restart services (e.g., because the changes don't require a restart), pass ``--no-restart`` to the update call.
See :ref:`the documentation <ray-up-doc>` for ``ray up``.
.. code-block:: shell
# Reconfigure autoscaling behavior without interrupting running jobs.
$ ray up ray/python/ray/autoscaler/$BACKEND/example-full.yaml \
--max-workers=N --no-restart
Running shell commands on the cluster (``ray exec``)
----------------------------------------------------
You can use ``ray exec`` to conveniently run commands on clusters. Note that python scripts that you want to scale should connect to Ray via ``ray.init(address="auto")``. See :ref:`the documentation <ray-exec-doc>` for ``ray exec``.
.. code-block:: shell
# Run a command on the cluster
$ ray exec cluster.yaml 'echo "hello world"'
# Run a command on the cluster, starting it if needed
$ ray exec cluster.yaml 'echo "hello world"' --start
# Run a command on the cluster, stopping the cluster after it finishes
$ ray exec cluster.yaml 'echo "hello world"' --stop
# Run a command on a new cluster called 'experiment-1', stopping it after
$ ray exec cluster.yaml 'echo "hello world"' \
--start --stop --cluster-name experiment-1
# Run a command in a detached tmux session
$ ray exec cluster.yaml 'echo "hello world"' --tmux
# Run a command in a screen (experimental)
$ ray exec cluster.yaml 'echo "hello world"' --screen
If you want to run applications on the cluster that are accessible from a web browser (e.g., Jupyter notebook), you can use the ``--port-forward``. The local port opened is the same as the remote port.
.. code-block:: shell
$ ray exec cluster.yaml --port-forward=8899 'source ~/anaconda3/bin/activate tensorflow_p36 && jupyter notebook --port=8899'
.. note:: For Kubernetes clusters, the ``port-forward`` option cannot be used while executing a command. To port forward and run a command you need to call ``ray exec`` twice separately.
Running Ray scripts on the cluster (``ray submit``)
---------------------------------------------------
You can also use ``ray submit`` to execute Python scripts on clusters. This will ``rsync`` the designated file onto the cluster and execute it with the given arguments. See :ref:`the documentation <ray-submit-doc>` for ``ray submit``.
.. code-block:: shell
# Run a Python script in a detached tmux session
$ ray submit cluster.yaml --tmux --start --stop tune_experiment.py
Attaching to a running cluster (``ray attach``)
-----------------------------------------------
You can use ``ray attach`` to attach to an interactive screen session on the cluster. See :ref:`the documentation <ray-attach-doc>` for ``ray attach`` or run ``ray attach --help``.
.. code-block:: shell
# Open a screen on the cluster
$ ray attach cluster.yaml
# Open a screen on a new cluster called 'session-1'
$ ray attach cluster.yaml --start --cluster-name=session-1
# Attach to tmux session on cluster (creates a new one if none available)
$ ray attach cluster.yaml --tmux
Synchronizing files from the cluster (``ray rsync-up/down``)
------------------------------------------------------------
To download or upload files to the cluster head node, use ``ray rsync_down`` or ``ray rsync_up``:
.. code-block:: shell
$ ray rsync_down cluster.yaml '/path/on/cluster' '/local/path'
$ ray rsync_up cluster.yaml '/local/path' '/path/on/cluster'
.. _monitor-cluster:
Monitoring cluster status (``ray dashboard/monitor``)
-----------------------------------------------------
The ray also comes with an online dashboard. The dashboard is accessible via HTTP on the head node (by default it listens on ``localhost:8265``). You can also use the built-in ``ray dashboard`` to do this automatically.
.. code-block:: shell
$ ray dashboard cluster.yaml
You can monitor cluster usage and auto-scaling status by tailing the autoscaling
logs in ``/tmp/ray/session_*/logs/monitor*``.
.. code-block:: shell
$ ray monitor cluster.yaml
The Ray autoscaler also reports per-node status in the form of instance tags. In your cloud provider console, you can click on a Node, go the the "Tags" pane, and add the ``ray-node-status`` tag as a column. This lets you see per-node statuses at a glance:
.. image:: /images/autoscaler-status.png
Common Workflow: Syncing git branches
-------------------------------------
A common use case is syncing a particular local git branch to all workers of the cluster. However, if you just put a `git checkout <branch>` in the setup commands, the autoscaler won't know when to rerun the command to pull in updates. There is a nice workaround for this by including the git SHA in the input (the hash of the file will change if the branch is updated):
.. code-block:: yaml
file_mounts: {
"/tmp/current_branch_sha": "/path/to/local/repo/.git/refs/heads/<YOUR_BRANCH_NAME>",
}
setup_commands:
- test -e <REPO_NAME> || git clone https://github.com/<REPO_ORG>/<REPO_NAME>.git
- cd <REPO_NAME> && git fetch && git checkout `cat /tmp/current_branch_sha`
This tells ``ray up`` to sync the current git branch SHA from your personal computer to a temporary file on the cluster (assuming you've pushed the branch head already). Then, the setup commands read that file to figure out which SHA they should checkout on the nodes. Note that each command runs in its own session. The final workflow to update the cluster then becomes just this:
1. Make local changes to a git branch
2. Commit the changes with ``git commit`` and ``git push``
3. Update files on your Ray cluster with ``ray up``
Autoscaling
-----------
The Ray Cluster Launcher will automatically enable a load-based autoscaler. When cluster resource usage exceeds a configurable threshold (80% by default), new nodes will be launched up the specified ``max_workers`` limit (in the cluster config). When nodes are idle for more than a timeout, they will be removed, down to the ``min_workers`` limit. The head node is never removed.
The default idle timeout is 5 minutes, which can be set in the cluster config. This is to prevent excessive node churn which could impact performance and increase costs (in AWS / GCP there is a minimum billing charge of 1 minute per instance, after which usage is billed by the second).
doc/source/cluster/launcher.rst
+41
View File
@@ -0,0 +1,41 @@
.. _ref-automatic-cluster:
Ray Cluster Launcher
====================
Ray comes with a built-in cluster launcher that makes deploying a Ray cluster simple.
.. code-block:: shell
# First, run `pip install boto3` and `aws configure`
#
# Create or update the cluster. When the command finishes, it will print
# out the command that can be used to SSH into the cluster head node.
$ ray up ray/python/ray/autoscaler/aws/example-full.yaml
This will provision resources from a node provider (like AWS EC2 or Kubernetes) to instantiate the specified cluster, and start a Ray cluster on the provisioned resources. Once the Ray cluster is running, you can manually SSH into it or use provided commands like ``ray attach``, ``ray rsync-up``, and ``ray exec`` to access it and run Ray programs. Check out :ref:`the Usage Guide <launcher-usage>` for instructions on how to use the cluster launcher.
You can configure the Ray Cluster Launcher to use with :ref:`a cloud provider <cluster-cloud>`, an existing :ref:`Kubernetes cluster <ray-launch-k8s>`, or a private cluster of machines.
Check out the below for more information about how to configure and use the cluster launcher.
.. toctree::
:maxdepth: 2
launcher-usage.rst
cloud.rst
config.rst
Questions or Issues?
--------------------
You can post questions or issues or feedback through the following channels:
1. `ray-dev@googlegroups.com`_: For discussions about development or any general
questions and feedback.
2. `StackOverflow`_: For questions about how to use Ray.
3. `GitHub Issues`_: For bug reports and feature requests.
.. _`ray-dev@googlegroups.com`: https://groups.google.com/forum/#!forum/ray-dev
.. _`StackOverflow`: https://stackoverflow.com/questions/tagged/ray
.. _`GitHub Issues`: https://github.com/ray-project/ray/issues
doc/source/deploying-on-slurm.rst → doc/source/cluster/slurm.rst
+3 -1
View File
@@ -1,3 +1,5 @@
.. _ray-slurm-deploy:
Deploying on Slurm
==================
@@ -33,7 +35,7 @@ Clusters managed by Slurm may require that Ray is initialized as a part of the s
srun --nodes=1 --ntasks=1 -w $node1 ray start --block --head --redis-port=6379 --redis-password=$redis_password & # Starting the head
sleep 5
# Make sure the head successfully starts before any worker does, otherwise
# the worker will not be able to connect to redis. In case of longer delay,
# the worker will not be able to connect to redis. In case of longer delay,
# adjust the sleeptime above to ensure proper order.
for (( i=1; i<=$worker_num; i++ ))
doc/source/deploy-on-yarn.rst → doc/source/cluster/yarn.rst
+6 -4
View File
@@ -1,3 +1,5 @@
.. _ray-yarn-deploy:
Deploying on YARN
=================
@@ -129,7 +131,7 @@ Clean up all started processes even if the application fails or is killed.
Putting things together, we have:
.. literalinclude:: ../yarn/ray-skein.yaml
.. literalinclude:: /../yarn/ray-skein.yaml
:language: yaml
:start-after: # Head service
:end-before: # Worker service
@@ -152,7 +154,7 @@ Start all of the processes needed on a ray worker node, blocking until killed by
Putting things together, we have:
.. literalinclude:: ../yarn/ray-skein.yaml
.. literalinclude:: /../yarn/ray-skein.yaml
:language: yaml
:start-after: # Worker service
@@ -161,7 +163,7 @@ Running a Job
Within your Ray script, use the following to connect to the started Ray cluster:
.. literalinclude:: ../yarn/example.py
.. literalinclude:: /../yarn/example.py
:language: python
:start-after: if __name__ == "__main__"
@@ -173,7 +175,7 @@ You can use the following command to launch the application as specified by the
Once it has been submitted, you can see the job running on the YARN dashboard.
.. image:: images/yarn-job.png
.. image:: /images/yarn-job.png
Cleaning Up
-----------
doc/source/autoscaler-status.png → doc/source/images/autoscaler-status.png
View File
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 20 KiB

After

Width:  |  Height:  |  Size: 20 KiB

doc/source/index.rst
+9 -1
View File
@@ -132,12 +132,19 @@ Academic Papers
using-ray.rst
configure.rst
ray-dashboard.rst
cluster-index.rst
Tutorial and Examples <auto_examples/overview.rst>
package-ref.rst
.. toctree::
:hidden:
:maxdepth: -1
:caption: Ray Cluster
cluster/index.rst
cluster/launcher.rst
cluster/deploy.rst
.. toctree::
:maxdepth: -1
:caption: Ray Serve
@@ -195,6 +202,7 @@ Academic Papers
joblib.rst
iter.rst
pandas_on_ray.rst
projects.rst
.. toctree::
:hidden:
doc/source/package-ref.rst
+26
View File
@@ -57,54 +57,80 @@ Experimental APIs
The Ray Command Line API
------------------------
.. _ray-start-doc:
.. click:: ray.scripts.scripts:start
:prog: ray start
:show-nested:
.. _ray-stop-doc:
.. click:: ray.scripts.scripts:stop
:prog: ray stop
:show-nested:
.. _ray-up-doc:
.. click:: ray.scripts.scripts:create_or_update
:prog: ray up
:show-nested:
.. _ray-down-doc:
.. click:: ray.scripts.scripts:teardown
:prog: ray down
:show-nested:
.. _ray-exec-doc:
.. click:: ray.scripts.scripts:exec_cmd
:prog: ray exec
:show-nested:
.. _ray-submit-doc:
.. click:: ray.scripts.scripts:submit
:prog: ray submit
:show-nested:
.. _ray-attach-doc:
.. click:: ray.scripts.scripts:attach
:prog: ray attach
:show-nested:
.. _ray-get_head_ip-doc:
.. click:: ray.scripts.scripts:get_head_ip
:prog: ray get_head_ip
:show-nested:
.. _ray-stack-doc:
.. click:: ray.scripts.scripts:stack
:prog: ray stack
:show-nested:
.. _ray-stat-doc:
.. click:: ray.scripts.scripts:stat
:prog: ray stat
:show-nested:
.. _ray-memory-doc:
.. click:: ray.scripts.scripts:memory
:prog: ray memory
:show-nested:
.. _ray-globalgc-doc:
.. click:: ray.scripts.scripts:globalgc
:prog: ray globalgc
:show-nested:
.. _ray-timeline-doc:
.. click:: ray.scripts.scripts:timeline
:prog: ray timeline
:show-nested:
doc/source/projects.rst
+1 -1
View File
@@ -186,4 +186,4 @@ Cluster file format (cluster.yaml)
----------------------------------
This is the same as for the autoscaler, see
`Cluster Launch page <autoscaling.html>`_.
:ref:`Cluster Launch page <ref-automatic-cluster>`.
doc/source/ray-dashboard.rst
+40 -39
View File
@@ -1,6 +1,6 @@
Ray Dashboard
=============
Ray's built-in dashboard provides metrics, charts, and other features that help
Ray's built-in dashboard provides metrics, charts, and other features that help
Ray users to understand Ray clusters and libraries.
Through the dashboard, you can
@@ -19,7 +19,7 @@ Getting Started
You can access the dashboard through its default URL, **localhost:8265**.
(Note that the port number increases if the default port is not available).
If you prefer to explicitly set the port on which the dashboard will run, you can pass
the ``--dashboard-port`` argument with ``ray start`` in the command line, or you can pass the
the ``--dashboard-port`` argument with ``ray start`` in the command line, or you can pass the
keyword argument ``dashboard_port`` in your call to ``ray.init()``.
The URL is printed when ``ray.init()`` is called.
@@ -28,8 +28,7 @@ The URL is printed when ``ray.init()`` is called.
INFO services.py:1093 -- View the Ray dashboard at localhost:8265
The dashboard is also available when using the autoscaler. Read about how to
`use the dashboard with the autoscaler <autoscaling.html#monitoring-cluster-status>`_.
The dashboard is also available :ref:`when using the cluster launcher <monitor-cluster>`.
Views
-----
@@ -71,6 +70,7 @@ The memory view shows you:
Ray Config
~~~~~~~~~~
The ray config tab shows you the current autoscaler configuration.
.. image:: https://raw.githubusercontent.com/ray-project/Images/master/docs/dashboard/Ray-config-basic.png
@@ -100,10 +100,10 @@ Debugging a Blocked Actor
~~~~~~~~~~~~~~~~~~~~~~~~~
You can find hanging actors through the Logical View tab.
If creating an actor requires resources (e.g., CPUs, GPUs, or other custom resources)
that are not currently available, the actor cannot be created until those resources are
added to the cluster or become available. This can cause an application to hang. To alert
you to this issue, infeasible tasks are shown in red in the dashboard, and pending tasks
If creating an actor requires resources (e.g., CPUs, GPUs, or other custom resources)
that are not currently available, the actor cannot be created until those resources are
added to the cluster or become available. This can cause an application to hang. To alert
you to this issue, infeasible tasks are shown in red in the dashboard, and pending tasks
are shown in yellow.
Below is an example.
@@ -111,7 +111,7 @@ Below is an example.
.. code-block:: python
import ray
ray.init(num_gpus=2)
@ray.remote(num_gpus=1)
@@ -131,11 +131,11 @@ Below is an example.
.. image:: https://raw.githubusercontent.com/ray-project/Images/master/docs/dashboard/dashboard-pending-infeasible-actors.png
:align: center
This cluster has two GPUs, and so it only has room to create two copies of ``Actor1``.
This cluster has two GPUs, and so it only has room to create two copies of ``Actor1``.
As a result, the rest of ``Actor1`` will be pending.
You can also see it is infeasible to create ``Actor2`` because it requires 4 GPUs which
is bigger than the total gpus available in this cluster (2 GPUs).
You can also see it is infeasible to create ``Actor2`` because it requires 4 GPUs which
is bigger than the total gpus available in this cluster (2 GPUs).
Debugging ObjectStoreFullError and Memory Leak
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -144,25 +144,25 @@ Note that this is the same information as displayed in the `ray memory command <
Inspect Memory Usage
~~~~~~~~~~~~~~~~~~~~
You can detect local memory anomalies through the Logical View tab. If NumObjectIdsInScope,
NumLocalObjects, or UsedLocalObjectMemory keeps growing without bound, it can lead to out
of memory errors or eviction of objectIDs that your program still wants to use.
You can detect local memory anomalies through the Logical View tab. If NumObjectIdsInScope,
NumLocalObjects, or UsedLocalObjectMemory keeps growing without bound, it can lead to out
of memory errors or eviction of objectIDs that your program still wants to use.
Profiling (Experimental)
~~~~~~~~~~~~~~~~~~~~~~~~
Use profiling features when you want to find bottlenecks in your Ray applications.
Use profiling features when you want to find bottlenecks in your Ray applications.
.. image:: https://raw.githubusercontent.com/ray-project/images/master/docs/dashboard/dashboard-profiling-buttons.png
:align: center
Clicking one of the profiling buttons on the dashboard launches py-spy, which will profile
Clicking one of the profiling buttons on the dashboard launches py-spy, which will profile
your actor process for the given duration. Once the profiling has been done, you can click the "profiling result" button to visualize the profiling information as a flamegraph.
This visualization can help reveal computational bottlenecks.
.. note::
The profiling button currently only works when you use **passwordless** ``sudo``.
The profiling button currently only works when you use **passwordless** ``sudo``.
It is still experimental. Please report any issues you run into.
More information on how to interpret the flamegraph is available at https://github.com/jlfwong/speedscope#usage.
@@ -176,8 +176,8 @@ References
Machine View
~~~~~~~~~~~~
**Machine/Worker Hierarchy**: The dashboard visualizes hierarchical relationship of
workers (processes) and machines (nodes). Each host consists of many workers, and
**Machine/Worker Hierarchy**: The dashboard visualizes hierarchical relationship of
workers (processes) and machines (nodes). Each host consists of many workers, and
you can see them by clicking the + button.
.. image:: https://raw.githubusercontent.com/ray-project/Images/master/docs/dashboard/Machine-view-reference-1.png
@@ -194,12 +194,12 @@ You can hide it again by clicking the - button.
:align: center
Resource configuration is represented as ``([Resource]: [Used Resources] / [Configured Resources])``.
For example, when a Ray cluster is configured with 4 cores, ``ray.init(num_cpus=4)``, you can see (CPU: 0 / 4).
For example, when a Ray cluster is configured with 4 cores, ``ray.init(num_cpus=4)``, you can see (CPU: 0 / 4).
.. image:: https://raw.githubusercontent.com/ray-project/Images/master/docs/dashboard/resource-allocation-row-configured-1.png
:align: center
When you spawn a new actor that uses 1 CPU, you can see this will be (CPU: 1/4).
When you spawn a new actor that uses 1 CPU, you can see this will be (CPU: 1/4).
Below is an example.
@@ -220,7 +220,7 @@ Below is an example.
**Host**: If it is a node, it shows host information. If it is a worker, it shows a pid.
**Workers**: If it is a node, it shows a number of workers and virtual cores.
**Workers**: If it is a node, it shows a number of workers and virtual cores.
Note that number of workers can exceed number of cores.
**Uptime**: Uptime of each worker and process.
@@ -244,7 +244,7 @@ Logical View (Experimental)
~~~~~~~~~~~~~~~~~~~~~~~~~~~
**Actor Titles**: Name of an actor and its arguments.
**State**: State of an actor.
**State**: State of an actor.
- 0: Alive
- 1: Restarting
@@ -254,31 +254,31 @@ Logical View (Experimental)
**Excuted**: A number of executed tasks for this actor.
**NumObjectIdsInScope**: Number of object IDs in scope for this actor. object IDs
**NumObjectIdsInScope**: Number of object IDs in scope for this actor. object IDs
in scope will not be evicted unless object stores are full.
**NumLocalObjects**: Number of object IDs that are in this actor's local memory.
Only big objects (>100KB) are residing in plasma object stores, and other small
**NumLocalObjects**: Number of object IDs that are in this actor's local memory.
Only big objects (>100KB) are residing in plasma object stores, and other small
objects are staying in local memory.
**UsedLocalObjectMemory**: Used memory used by local objects.
**kill actor**: A button to kill an actor in a cluster. It is corresponding to ``ray.kill``.
**kill actor**: A button to kill an actor in a cluster. It is corresponding to ``ray.kill``.
**profile for**: A button to run profiling. We currently support profiling for 10s,
**profile for**: A button to run profiling. We currently support profiling for 10s,
30s and 60s. It requires passwordless ``sudo``.
**Infeasible Actor Creation**: Actor creation is infeasible when an actor
requires more resources than a Ray cluster can provide. This is depicted
**Infeasible Actor Creation**: Actor creation is infeasible when an actor
requires more resources than a Ray cluster can provide. This is depicted
as a red colored actor.
**Pending Actor Creation**: Actor creation is pending when there are no
available resources for this actor because they are already taken by other
**Pending Actor Creation**: Actor creation is pending when there are no
available resources for this actor because they are already taken by other
tasks and actors. This is depicted as a yellow colored actor.
**Actor Hierarchy**: The logical view renders actor information in a tree format.
**Actor Hierarchy**: The logical view renders actor information in a tree format.
To illustrate this, in the code block below, the ``Parent`` actor creates
To illustrate this, in the code block below, the ``Parent`` actor creates
two ``Child`` actors and each ``Child`` actor creates one ``GrandChild`` actor.
This relationship is visible in the dashboard *Logical View* tab.
@@ -296,7 +296,7 @@ This relationship is visible in the dashboard *Logical View* tab.
class Child:
def __init__(self):
self.grandchild_handle = Grandchild.remote()
@ray.remote
class Parent:
def __init__(self):
@@ -304,7 +304,7 @@ This relationship is visible in the dashboard *Logical View* tab.
parent_handle = Parent.remote()
You can see that the dashboard shows the parent/child relationship as expected.
You can see that the dashboard shows the parent/child relationship as expected.
.. image:: https://raw.githubusercontent.com/ray-project/Images/master/docs/dashboard/Logical-view-basic.png
:align: center
@@ -329,7 +329,8 @@ Memory
Ray Config
~~~~~~~~~~~~
If you are using the autoscaler, this Configuration defined at ``cluster.yaml`` is shown.
If you are using the cluster launcher, this Configuration defined at ``cluster.yaml`` is shown.
See `Cluster.yaml reference <https://github.com/ray-project/ray/blob/master/python/ray/autoscaler/aws/example-full.yaml>`_ for more details.
Tune (Experimental)
@@ -342,5 +343,5 @@ Tune (Experimental)
**Start Time**: Start time of each trial.
**Hyperparameters**: There are many hyperparameter users specify. All of values will
**Hyperparameters**: There are many hyperparameter users specify. All of values will
be visible at the dashboard.
doc/source/ray-overview/index.rst
+5 -5
View File
@@ -14,7 +14,7 @@ Parallelizing Python Functions with Ray Tasks
=============================================
First, import ray and ``init`` the Ray service.
Then decorate your function with ``@ray.remote`` to declare that you want to run this function
Then decorate your function with ``@ray.remote`` to declare that you want to run this function
remotely. Lastly, call that function with ``.remote()`` instead of calling it normally. This remote call yields a future, or ``ObjectID`` that you can then
fetch with ``ray.get``.
@@ -37,15 +37,15 @@ Parallelizing Python Classes with Ray Actors
==============================================
Ray provides actors to allow you to parallelize an instance of a class in Python.
When you instantiate a class that is a Ray actor, Ray will start a remote instance
of that class in the cluster. This actor can then execute remote method calls and
When you instantiate a class that is a Ray actor, Ray will start a remote instance
of that class in the cluster. This actor can then execute remote method calls and
maintain its own internal state.
.. code-block:: python
import ray
ray.init() # Only call this once.
@ray.remote
class Counter(object):
def __init__(self):
@@ -157,4 +157,4 @@ Ray programs can run on a single machine, and can also seamlessly scale to large
``ray submit [CLUSTER.YAML] example.py --start``
Read more about `launching clusters <autoscaling.html>`_.
Read more about `:ref:launching clusters <cluster-index>`.
doc/source/raysgd/raysgd_pytorch.rst
+1 -1
View File
@@ -441,7 +441,7 @@ Distributed Multi-node Training
You can scale your training to multiple nodes without making any modifications to your training code.
To train across a cluster, first make sure that the Ray cluster is started. You can start a Ray cluster `via the Ray cluster launcher <autoscaling.html>`_ or `manually <using-ray-on-a-cluster.html>`_.
To train across a cluster, first make sure that the Ray cluster is started (see :ref:`cluster-index` for more details).
Then, in your program, you'll need to connect to this cluster via ``ray.init``:
doc/source/raysgd/raysgd_tensorflow.rst
+1 -1
View File
@@ -36,7 +36,7 @@ Wrap your training with this:
})
Then, start a Ray cluster `via autoscaler <autoscaling.html>`_ or `manually <using-ray-on-a-cluster.html>`_.
Then, start a :ref:`Ray cluster <cluster-index>`.
.. code-block:: bash
doc/source/serve/deployment.rst
+20 -20
View File
@@ -3,12 +3,12 @@ Deploying Ray Serve
===================
In the :doc:`key-concepts`, you saw some of the basics of how to write serve applications.
This section will dive a bit deeper into how Ray Serve runs on a Ray cluster and how you're able
This section will dive a bit deeper into how Ray Serve runs on a Ray cluster and how you're able
to deploy and update your serve application over time.
To deploy a Ray Serve instance you're going to need several things.
1. A running Ray cluster (you can deploy one on your local machine for testing). To learn more about Ray clusters see :doc:`../cluster-index`.
1. A running Ray cluster (you can deploy one on your local machine for testing). To learn more about Ray clusters see :ref:`cluster-index`.
2. A Ray Serve instance.
3. Your Ray Serve endpoint(s) and backend(s).
@@ -20,8 +20,8 @@ Deploying a Model with Ray Serve
================================
Let's get started deploying our first Ray Serve application. The first thing you'll need
to do is start a Ray cluster. You can do that using the Ray autoscaler, but in our case
we'll create it on our local machine. To learn more about Ray Clusters see :doc:`../cluster-index`.
to do is start a Ray cluster. You can do that using the Ray cluster launcher, but in our case
we'll create it on our local machine. To learn more about Ray Clusters see :ref:`cluster-index`.
Starting the Cluster
--------------------
@@ -31,8 +31,8 @@ We do that by running:
ray start --head
That starts a cluster on our local machine. We can shut that down by running ``ray stop``. You should
run this after we complete this tutorial.
That starts a cluster on our local machine. We can shut that down by running ``ray stop``. You should
run this after we complete this tutorial.
Setup: Training a Model
-----------------------
@@ -72,7 +72,7 @@ a backend in serve for our model (and versioned it with a string).
:start-after: __doc_create_deploy_begin__
:end-before: __doc_create_deploy_end__
What serve does when we run this code is store the model as a Ray actor
What serve does when we run this code is store the model as a Ray actor
and route traffic to it as the endpoint is queried, in this case over HTTP.
Note that in order for this endpoint to be accessible from other machines, we
need to specify ``http_host="0.0.0.0"`` in :mod:`serve.init <ray.serve.init>` like we did here.
@@ -95,8 +95,8 @@ this model with a new set of code.
Updating Your Model Over Time
=============================
Updating our model is as simple as deploying the first one. While the code snippet includes
a lot of information, all that we're doing is we are defining a new model, saving it, then loading
Updating our model is as simple as deploying the first one. While the code snippet includes
a lot of information, all that we're doing is we are defining a new model, saving it, then loading
it into serve. The key lines are at the end.
.. literalinclude:: ../../../python/ray/serve/examples/doc/tutorial_deploy.py
@@ -113,12 +113,12 @@ these two models.
serve.set_traffic("iris_classifier", {"lr:v2": 0.25, "lr:v1": 0.75})
While this is a simple operation, you may want to see :ref:`serve-split-traffic` for more information.
While this is a simple operation, you may want to see :ref:`serve-split-traffic` for more information.
One thing you may want to consider as well is
:ref:`session-affinity` which gives you the ability to ensure that queries from users/clients always get mapped to the same backend.
versions.
Now that we're up and running serving two models in production, let's query
Now that we're up and running serving two models in production, let's query
our results several times to see some results. You'll notice that we're now splitting
traffic between these two different models.
@@ -131,10 +131,10 @@ We'll use the requests library to query our endpoint and be able to get a result
:start-after: __doc_query_begin__
:end-before: __doc_query_end__
If you run this code several times, you'll notice that the output will change - this
If you run this code several times, you'll notice that the output will change - this
is due to us running the two models in parallel that we created above.
Upon concluding the above tutorial, you'll want to run ``ray stop`` to
Upon concluding the above tutorial, you'll want to run ``ray stop`` to
shutdown the Ray cluster on your local machine.
Deploying as a Kubernetes Service
@@ -146,10 +146,10 @@ In order to deploy Ray Serve on Kubernetes, we need to do the following:
2. Expose the head node of the cluster as a `Service`_.
3. Start Ray Serve on the cluster.
There are multiple ways to start a Ray cluster on Kubernetes, see :doc:`../deploy-on-kubernetes` for more information.
Here, we will be using the :doc:`../autoscaling` tool, which has support for Kubernetes as a backend.
There are multiple ways to start a Ray cluster on Kubernetes, see :ref:`ray-k8s-deploy` for more information.
Here, we will be using the :ref:`Ray Cluster Launcher <ref-automatic-cluster>` tool, which has support for Kubernetes as a backend.
The autoscaler takes in a yaml config file that describes the cluster.
The cluster launcher takes in a yaml config file that describes the cluster.
Here, we'll be using the `Kubernetes default config`_ with a few small modifications.
First, we need to make sure that the head node of the cluster, where Ray Serve will run its HTTP server, is exposed as a Kubernetes `Service`_.
There is already a default head node service defined in the ``services`` field of the config, so we just need to make sure that it's exposing the right port: 8000, which Ray Serve binds on by default.
@@ -228,7 +228,7 @@ With the cluster now running, we can run a simple script to start Ray Serve and
import ray
from ray import serve
# Connect to the running Ray cluster.
ray.init(address="auto")
# Bind on 0.0.0.0 to expose the HTTP server on external IPs.
@@ -272,10 +272,10 @@ Deployment FAQ
Best practices for local development
------------------------------------
One thing you may notice is that we never have to declare a ``while True`` loop or
something to keep the Ray Serve process running. In general, we don't recommend using forever loops and therefore
One thing you may notice is that we never have to declare a ``while True`` loop or
something to keep the Ray Serve process running. In general, we don't recommend using forever loops and therefore
opt for launching a Ray Cluster locally. Specify a Ray cluster like we did in :ref:`serve-deploy-tutorial`.
To learn more, in general, about Ray Clusters see :doc:`../cluster-index`.
To learn more, in general, about Ray Clusters see :ref:`cluster-index`.
Deploying Multiple Serve Instaces on a Single Ray Cluster
doc/source/tune/_tutorials/tune-distributed.rst
+4 -4
View File
@@ -62,7 +62,7 @@ If you used a cluster configuration (starting a cluster with ``ray up`` or ``ray
Local Cluster Setup
-------------------
If you already have a list of nodes, you can follow the local private cluster setup `instructions here <autoscaling.html#quick-start-private-cluster>`_. Below is an example cluster configuration as ``tune-default.yaml``:
If you already have a list of nodes, you can follow the local :ref:`private cluster setup <cluster-private-setup>`. Below is an example cluster configuration as ``tune-default.yaml``:
.. literalinclude:: /../../python/ray/tune/examples/tune-local-default.yaml
:language: yaml
@@ -82,7 +82,7 @@ If you already have a list of nodes, you can follow the local private cluster se
Manual Local Cluster Setup
~~~~~~~~~~~~~~~~~~~~~~~~~~
If you run into issues using the local cluster setup (or want to add nodes manually), you can use the manual cluster setup. `Full documentation here <using-ray-on-a-cluster.html>`__. At a glance,
If you run into issues using the local cluster setup (or want to add nodes manually), you can use :ref:`the manual cluster setup <cluster-index>`. At a glance,
**On the head node**:
@@ -150,7 +150,7 @@ Note that you can customize the directory of results by running: ``tune.run(loca
Syncing
-------
Tune automatically syncs the trial folder on remote nodes back to the head node. This requires the ray cluster to be started with the :ref:`autoscaler <ref-automatic-cluster>`.
Tune automatically syncs the trial folder on remote nodes back to the head node. This requires the ray cluster to be started with the :ref:`cluster launcher <ref-automatic-cluster>`.
By default, local syncing requires rsync to be installed. You can customize the sync command with the ``sync_to_driver`` argument in ``tune.run`` by providing either a function or a string.
If a string is provided, then it must include replacement fields ``{source}`` and ``{target}``, like ``rsync -savz -e "ssh -i ssh_key.pem" {source} {target}``. Alternatively, a function can be provided with the following signature:
@@ -277,7 +277,7 @@ Fault Tolerance
Tune will automatically restart trials in case of trial failures/error (if ``max_failures != 0``), both in the single node and distributed setting.
Tune will restore trials from the latest checkpoint, where available. In the distributed setting, if using the autoscaler with ``rsync`` enabled, Tune will automatically sync the trial folder with the driver. For example, if a node is lost while a trial (specifically, the corresponding Trainable actor of the trial) is still executing on that node and a checkpoint of the trial exists, Tune will wait until available resources are available to begin executing the trial again.
Tune will restore trials from the latest checkpoint, where available. In the distributed setting, if using the cluster launcher with ``rsync`` enabled, Tune will automatically sync the trial folder with the driver. For example, if a node is lost while a trial (specifically, the corresponding Trainable actor of the trial) is still executing on that node and a checkpoint of the trial exists, Tune will wait until available resources are available to begin executing the trial again.
If the trial/actor is placed on a different node, Tune will automatically push the previous checkpoint file to that node and restore the remote trial actor state, allowing the trial to resume from the latest checkpoint even after failure.
doc/source/using-ray-on-a-cluster.rst
-83
View File
@@ -1,83 +0,0 @@
.. _ref-cluster-setup:
Manual Cluster Setup
====================
.. note::
If you're using AWS, Azure or GCP you should use the automated `setup commands <autoscaling.html>`_.
The instructions in this document work well for small clusters. For larger
clusters, consider using the pssh package: ``sudo apt-get install pssh`` or
the `setup commands for private clusters <autoscaling.html#quick-start-private-cluster>`_.
Deploying Ray on a Cluster
--------------------------
This section assumes that you have a cluster running and that the nodes in the
cluster can communicate with each other. It also assumes that Ray is installed
on each machine. To install Ray, follow the `installation instructions`_.
.. _`installation instructions`: http://docs.ray.io/en/latest/installation.html
Starting Ray on each machine
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
On the head node (just choose some node to be the head node), run the following.
If the ``--port`` argument is omitted, Ray will choose port 6379, falling back to a
random port.
.. code-block:: bash
ray start --head --port=6379
The command will print out the address of the Redis server that was started
(and some other address information).
**Then on all of the other nodes**, run the following. Make sure to replace
``<address>`` with the value printed by the command on the head node (it
should look something like ``123.45.67.89:6379``).
.. code-block:: bash
ray start --address=<address>
If you wish to specify that a machine has 10 CPUs and 1 GPU, you can do this
with the flags ``--num-cpus=10`` and ``--num-gpus=1``. See the `Configuration <configure.html>`__ page for more information.
Now we've started all of the Ray processes on each node Ray. This includes
- Some worker processes on each machine.
- An object store on each machine.
- A raylet on each machine.
- Multiple Redis servers (on the head node).
To run some commands, start up Python on one of the nodes in the cluster, and do
the following.
.. code-block:: python
import ray
ray.init(address="<address>")
Now you can define remote functions and execute tasks. For example, to verify
that the correct number of nodes have joined the cluster, you can run the
following.
.. code-block:: python
import time
@ray.remote
def f():
time.sleep(0.01)
return ray.services.get_node_ip_address()
# Get a list of the IP addresses of the nodes that have joined the cluster.
set(ray.get([f.remote() for _ in range(1000)]))
Stopping Ray
~~~~~~~~~~~~
When you want to stop the Ray processes, run ``ray stop`` on each node.
doc/source/walkthrough.rst
+1 -1
View File
@@ -32,7 +32,7 @@ You can start Ray on a single machine by adding this to your python script.
Ray will then be able to utilize all cores of your machine. Find out how to configure the number of cores Ray will use at :ref:`configuring-ray`.
To start a multi-node Ray cluster, see the `cluster setup page <using-ray-on-a-cluster.html>`__.
To start a multi-node Ray cluster, see the :ref:`cluster setup page <cluster-index>`.
.. _ray-remote-functions: