diff --git a/doc/source/index.rst b/doc/source/index.rst index 84473e9dc..b2c6e19f9 100644 --- a/doc/source/index.rst +++ b/doc/source/index.rst @@ -44,6 +44,6 @@ Ray :maxdepth: 1 :caption: Cluster Usage - using-ray-on-a-cluster.md - using-ray-on-a-large-cluster.md + using-ray-on-a-cluster.rst + using-ray-on-a-large-cluster.rst using-ray-and-docker-on-a-cluster.md diff --git a/doc/source/using-ray-on-a-cluster.md b/doc/source/using-ray-on-a-cluster.md deleted file mode 100644 index 6c401f725..000000000 --- a/doc/source/using-ray-on-a-cluster.md +++ /dev/null @@ -1,72 +0,0 @@ -# Using Ray on a cluster - -Deploying Ray on a cluster currently requires a bit of manual work. - -## Deploying Ray on a cluster. - -This section assumes that you have a cluster running and that the node in the -cluster can communicate with each other. It also assumes that Ray is installed -on each machine. To install Ray, follow the instructions for [installation on -Ubuntu](install-on-ubuntu.md). - -### Starting Ray on each machine. - -On the head node (just choose some node to be the head node), run the following, -replacing `` with a port of your choice, e.g., `6379`. - -``` -./ray/scripts/start_ray.sh --head --redis-port -``` - -The `--redis-port` arugment is optional, and if not provided Ray starts Redis -on a port selected at random. -In either case, the command will print out the address of the Redis server -that was started (and some other address information). - -Then on all of the other nodes, run the following. Make sure to replace -`` with the value printed by the command on the head node (it -should look something like `123.45.67.89:6379`). - -``` -./ray/scripts/start_ray.sh --redis-address -``` - -To specify the number of processes to start, use the flag `--num-workers`, as -follows: - -``` -./ray/scripts/start_ray.sh --num-workers -``` - -Now we've started all of the Ray processes on each node Ray. This includes - -- Some worker processes on each machine. -- An object store on each machine. -- A local scheduler on each machine. -- One Redis server (on the head node). -- One global scheduler (on the head node). -- Optionally, this may start up some processes for visualizing the system state - through a web UI. - -To run some commands, start up Python on one of the nodes in the cluster, and do -the following. - -```python -import ray -ray.init(redis_address="") -``` - -Now you can define remote functions and execute tasks. For example: - -```python -@ray.remote -def f(x): - return x - -ray.get([f.remote(f.remote(f.remote(0))) for _ in range(1000)]) -``` - -### Stopping Ray - -When you want to stop the Ray processes, run `./ray/scripts/stop_ray.sh` -on each node. diff --git a/doc/source/using-ray-on-a-cluster.rst b/doc/source/using-ray-on-a-cluster.rst new file mode 100644 index 000000000..97af9d23c --- /dev/null +++ b/doc/source/using-ray-on-a-cluster.rst @@ -0,0 +1,81 @@ +Using Ray on a Cluster +====================== + +The instructions in this document work well for small clusters. For larger +clusters, follow the instructions for `managing a cluster with parallel ssh`_. + +.. _`managing a cluster with parallel ssh`: http://ray.readthedocs.io/en/latest/using-ray-on-a-large-cluster.html + +Deploying Ray on a Cluster +-------------------------- + +This section assumes that you have a cluster running and that the nodes in the +cluster can communicate with each other. It also assumes that Ray is installed +on each machine. To install Ray, follow the instructions for +`installation on Ubuntu`_. + +.. _`installation on Ubuntu`: http://ray.readthedocs.io/en/latest/install-on-ubuntu.html + +Starting Ray on each machine +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +On the head node (just choose some node to be the head node), run the following. +If the ``--redis-port`` argument is omitted, Ray will choose a port at random. + +.. code-block:: bash + + ./ray/scripts/start_ray.sh --head --redis-port=6379 + +The command will print out the address of the Redis server that was started +(and some other address information). + +Then on all of the other nodes, run the following. Make sure to replace +```` with the value printed by the command on the head node (it +should look something like ``123.45.67.89:6379``). + +.. code-block:: bash + + ./ray/scripts/start_ray.sh --redis-address= + +If you wish to specify that a machine has 10 CPUs and 1 GPU, you can do this +with the flags ``--num-cpus=10`` and ``--num-gpus=1``. If these flags are not +used, then Ray will detect the number of CPUs automatically and will assume +there are 0 GPUs. + +Now we've started all of the Ray processes on each node Ray. This includes + +- Some worker processes on each machine. +- An object store on each machine. +- A local scheduler on each machine. +- Multiple Redis servers (on the head node). +- One global scheduler (on the head node). + +To run some commands, start up Python on one of the nodes in the cluster, and do +the following. + +.. code-block:: python + + import ray + ray.init(redis_address="") + +Now you can define remote functions and execute tasks. For example, to verify +that the correct number of nodes have joined the cluster, you can run the +following. + +.. code-block:: python + + import time + + @ray.remote + def f(): + time.sleep(0.01) + return ray.services.get_node_ip_address() + + # Get a list of the IP addresses of the nodes that have joined the cluster. + set(ray.get([f.remote() for _ in range(1000)])) + +Stopping Ray +~~~~~~~~~~~~ + +When you want to stop the Ray processes, run ``./ray/scripts/stop_ray.sh`` on +each node. diff --git a/doc/source/using-ray-on-a-large-cluster.md b/doc/source/using-ray-on-a-large-cluster.md deleted file mode 100644 index ea101fa72..000000000 --- a/doc/source/using-ray-on-a-large-cluster.md +++ /dev/null @@ -1,273 +0,0 @@ -# Using Ray on a large cluster - -Deploying Ray on a cluster currently requires a bit of manual work. The -instructions here illustrate how to use parallel ssh commands to simplify the -process of running commands and scripts on many machines simultaneously. - -## Booting up a cluster on EC2 - -* Create an EC2 instance running Ray following instructions for [installation on -Ubuntu](install-on-ubuntu.md). - * Add any packages that you may need for running your application. - * Install the pssh package: `sudo apt-get install pssh` -* [Create an AMI Image](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/creating-an-ami-ebs.html) -of your installation. -* Use the EC2 console to launch additional instances using the AMI created. - -## Deploying Ray on a cluster. - -This section assumes that you have a cluster of machines running and that these -nodes have network connectivity to one another. It also assumes that Ray is -installed on each machine. - -Additional assumptions: - -* All of the following commands are run from a machine designated as - the _head node_. -* The head node will run Redis and the global scheduler. -* The head node is the launching point for driver programs and for - administrative tasks. -* The head node has ssh access to all other nodes. -* All nodes are accessible via ssh keys -* Ray is checked out on each node at the location `$HOME/ray`. - -**Note:** The commands below will probably need to be customized for your specific -setup. - -### Connect to the head node - -In order to initiate ssh commands from the cluster head node we suggest enabling -ssh agent forwarding. This will allow the session that you initiate with the -head node to connect to other nodes in the cluster to run scripts on them. You -can enable ssh forwarding by running the following command (replacing -`` with the path to the private key that you would use when logging in -to the nodes in the cluster). - -``` -ssh-add -``` - -Now log in to the head node with the following command, where -`` is the public IP address of the head node (just choose -one of the nodes to be the head node). - -``` -ssh -A ubuntu@ -``` - -### Build a list of node IP addresses - -Populate a file `workers.txt` with one IP address on each line. Do not include -the head node IP address in this file. These IP addresses should typically be -private network IP addresses, but any IP addresses which the head node can use -to ssh to worker nodes will work here. - -### Confirm that you can ssh to all nodes - -```bash -for host in $(cat workers.txt); do - ssh $host uptime -done -``` - -You may be prompted to verify the host keys during this process. - -### Starting Ray - -#### Starting Ray on the head node - -On the head node (just choose some node to be the head node), run the following: - -``` -./ray/scripts/start_ray.sh --head --num-workers= --redis-port -``` - -Replace `` with a port of your choice, e.g., `6379`. Also, replace -`` with the number of workers that you wish to start. - - -#### Start Ray on the worker nodes - -Create a file `start_worker.sh` that contains something like the following: - -```bash -# Make sure the SSH session has the correct version of Python on its path. -export PATH=/home/ubuntu/anaconda2/bin/:$PATH -ray/scripts/start_ray.sh --num-workers= --redis-address=: -``` - -This script, when run on the worker nodes, will start up Ray. You will need to -replace `` with the IP address that worker nodes will use to -connect to the head node (most likely a **private IP address**). In this -example we also export the path to the Python installation since our remote -commands will not be executing in a login shell. - -**Warning:** You may need to manually export the correct path to Python (you -will need to change the first line of `start_worker.sh` to find the version of -Python that Ray was built against). This is necessary because the `PATH` -environment variable used by `parallel-ssh` can differ from the `PATH` -environment variable that gets set when you `ssh` to the machine. - -**Warning:** If the `parallel-ssh` command below appears to hang, `head-node-ip` -may need to be a private IP address instead of a public IP address (e.g., if you -are using EC2). - -Now use `parallel-ssh` to start up Ray on each worker node. - -``` -parallel-ssh -h workers.txt -P -I < start_worker.sh -``` - -Note that on some distributions the `parallel-ssh` command may be called `pssh`. - -#### Verification - -Now you have started all of the Ray processes on each node. These include: - -- Some worker processes on each machine. -- An object store on each machine. -- A local scheduler on each machine. -- One Redis server (on the head node). -- One global scheduler (on the head node). - -To confirm that the Ray cluster setup is working, start up Python on one of the -nodes in the cluster and enter the following commands to connect to the Ray -cluster. - -```python -import ray -ray.init(redis_address="") -``` - -Here `` should have the form `:`. - -Now you can define remote functions and execute tasks. For example: - -```python -@ray.remote -def f(x): - return x - -ray.get([f.remote(f.remote(f.remote(0))) for _ in range(1000)]) -``` - -### Stopping Ray - -#### Stop Ray on worker nodes - -``` -parallel-ssh -h workers.txt -P ray/scripts/stop_ray.sh -``` - -This command will execute the `stop_ray.sh` script on each of the worker nodes. - -#### Stop Ray on the head node - -``` -ray/scripts/stop_ray.sh -``` - -### Upgrading Ray - -Ray remains under active development so you may at times want to upgrade the -cluster to take advantage of improvements and fixes. - -#### Create an upgrade script - -On the head node, create a file called `upgrade.sh` that contains the commands necessary -to upgrade Ray. It should look something like the following: - -```bash -# Make sure the SSH session has the correct version of Python on its path. -export PATH=/home/ubuntu/anaconda2/bin/:$PATH -# Do pushd/popd to make sure we end up in the same directory. -pushd . -# Upgrade Ray. -cd ray -git remote set-url origin https://github.com/ray-project/ray -git checkout master -git pull -cd python -python setup.py install --user -popd -``` - -This script executes a series of git commands to update the Ray source code, then builds -and installs Ray. - -#### Stop Ray on the cluster - -Follow the instructions for [stopping Ray](#stopping-ray). - - -#### Run the upgrade script on the cluster - -First run the upgrade script on the head node. This will upgrade the head node and -help confirm that the upgrade script is working properly. - -``` -bash upgrade.sh -``` - -Next run the upgrade script on the worker nodes. - -``` -parallel-ssh -h workers.txt -P -t 0 -I < upgrade.sh -``` - -Note here that we use the `-t 0` option to set the timeout to infinite. - -#### Start Ray on the cluster - -Follow the instructions for [starting Ray](#starting-ray). - -## Sync Application Files to other nodes - -If you are running an application that reads input files or uses python libraries then you may find it useful to copy a directory on the head to the worker nodes. - - -You can do this using the `parallel-rsync` command: - -``` -parallel-rsync -h workers.txt -r /home/ubuntu/ -``` - -where `` is the directory you want to synchronize. -Note that the destination argument for this command must represent an absolute path on the worker node. - -## Troubleshooting - -### Problems with parallel-ssh - -If any of the above commands fail, verify that the head node has SSH access to -the other nodes by running - -```bash -for host in $(cat workers.txt); do - ssh $host uptime -done -``` - -If you get a permission denied error, then make sure you have SSH'ed to the head -node with agent forwarding enabled. This is done as follows. - -``` -ssh-add -ssh -A ubuntu@ -``` - -### Configuring EC2 instances to increase the number of allowed Redis clients - -This section can be ignored unless you run into problems with the maximum -number of Redis clients. - -* Ensure that the hard limit for the number of open file descriptors is set - to a large number (e.g., 65536). This only needs to be done on instances - where Redis shards will run --- by default, just the _head node_. - - Check the hard ulimit for open file descriptors with `ulimit -Hn` - - If that number is smaller than 65536, set the hard ulimit for open file descriptors - system-wide: - ``` - sudo bash -c "echo $USER hard nofile 65536 >> /etc/security/limits.conf" - ``` - - Logout and log back in diff --git a/doc/source/using-ray-on-a-large-cluster.rst b/doc/source/using-ray-on-a-large-cluster.rst new file mode 100644 index 000000000..f788ac37e --- /dev/null +++ b/doc/source/using-ray-on-a-large-cluster.rst @@ -0,0 +1,314 @@ +Using Ray on a Large Cluster +============================ + +Deploying Ray on a cluster requires a bit of manual work. The instructions here +illustrate how to use parallel ssh commands to simplify the process of running +commands and scripts on many machines simultaneously. + +Booting up a cluster on EC2 +--------------------------- + +* Create an EC2 instance running Ray following instructions for + `installation on Ubuntu`_. + + * Add any packages that you may need for running your application. + * Install the pssh package: ``sudo apt-get install pssh``. +* `Create an AMI`_ with Ray installed and with whatever code and libraries you + want on the cluster. +* Use the EC2 console to launch additional instances using the AMI you created. +* Configure the instance security groups so that they machines can all + communicate with one another. + +.. _`installation on Ubuntu`: http://ray.readthedocs.io/en/latest/install-on-ubuntu.html +.. _`Create an AMI`: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/creating-an-ami-ebs.html + +Deploying Ray on a Cluster +-------------------------- + +This section assumes that you have a cluster of machines running and that these +nodes have network connectivity to one another. It also assumes that Ray is +installed on each machine. + +Additional assumptions: + +* All of the following commands are run from a machine designated as + the **head node**. +* The head node will run Redis and the global scheduler. +* The head node has ssh access to all other nodes. +* All nodes are accessible via ssh keys +* Ray is checked out on each node at the location `$HOME/ray`. + +**Note:** The commands below will probably need to be customized for your +specific setup. + +Connect to the head node +~~~~~~~~~~~~~~~~~~~~~~~~ + +In order to initiate ssh commands from the cluster head node we suggest enabling +ssh agent forwarding. This will allow the session that you initiate with the +head node to connect to other nodes in the cluster to run scripts on them. You +can enable ssh forwarding by running the following command before connecting to +the head node (replacing ```` with the path to the private key that you +would use when logging in to the nodes in the cluster). + +.. code-block:: bash + + ssh-add + +Now log in to the head node with the following command, where +```` is the public IP address of the head node (just choose +one of the nodes to be the head node). + +.. code-block:: bash + + ssh -A ubuntu@ + +Build a list of node IP addresses +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +On the head node, populate a file ``workers.txt`` with one IP address on each +line. Do not include the head node IP address in this file. These IP addresses +should typically be private network IP addresses, but any IP addresses which the +head node can use to ssh to worker nodes will work here. This should look +something like the following. + +.. code-block:: bash + + 172.31.27.16 + 172.31.29.173 + 172.31.24.132 + 172.31.29.224 + +Confirm that you can ssh to all nodes +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: bash + + for host in $(cat workers.txt); do + ssh $host uptime + done + +You may need to verify the host keys during this process. If so, run this step +again to verify that it worked. If you see a **permission denied** error, you +most likely forgot to run ``ssh-add `` before connecting to the head +node. + +Starting Ray +~~~~~~~~~~~~ + +**Start Ray on the head node** + +On the head node, run the following: + +.. code-block:: bash + + ./ray/scripts/start_ray.sh --head --redis-port=6379 + + +**Start Ray on the worker nodes** + +Create a file ``start_worker.sh`` that contains something like the following: + +.. code-block:: bash + + # Make sure the SSH session has the correct version of Python on its path. + # You will probably have to change the line below. + export PATH=/home/ubuntu/anaconda3/bin/:$PATH + ray/scripts/start_ray.sh --redis-address=:6379 + +This script, when run on the worker nodes, will start up Ray. You will need to +replace ```` with the IP address that worker nodes will use to +connect to the head node (most likely a **private IP address**). In this +example we also export the path to the Python installation since our remote +commands will not be executing in a login shell. + +**Warning:** You will probably need to manually export the correct path to +Python (you will need to change the first line of ``start_worker.sh`` to find +the version of Python that Ray was built against). This is necessary because the +``PATH`` environment variable used by ``parallel-ssh`` can differ from the +``PATH`` environment variable that gets set when you ``ssh`` to the machine. + +**Warning:** If the ``parallel-ssh`` command below appears to hang or otherwise +fails, ``head-node-ip`` may need to be a private IP address instead of a public +IP address (e.g., if you are using EC2). It's also possible that you forgot to +run ``ssh-add `` or that you forgot the ``-A`` flag when connecting to +the head node. + +Now use ``parallel-ssh`` to start up Ray on each worker node. + +.. code-block:: bash + + parallel-ssh -h workers.txt -P -I < start_worker.sh + +Note that on some distributions the ``parallel-ssh`` command may be called +``pssh``. + +**Verification** + +Now you have started all of the Ray processes on each node. These include: + +- Some worker processes on each machine. +- An object store on each machine. +- A local scheduler on each machine. +- Multiple Redis servers (on the head node). +- One global scheduler (on the head node). + +To confirm that the Ray cluster setup is working, start up Python on one of the +nodes in the cluster and enter the following commands to connect to the Ray +cluster. + +.. code-block:: python + + import ray + ray.init(redis_address="") + +Here ```` should have the form ``:6379``. + +Now you can define remote functions and execute tasks. For example, to verify +that the correct number of nodes have joined the cluster, you can run the +following. + +.. code-block:: python + + import time + + @ray.remote + def f(): + time.sleep(0.01) + return ray.services.get_node_ip_address() + + # Get a list of the IP addresses of the nodes that have joined the cluster. + set(ray.get([f.remote() for _ in range(1000)])) + + +Stopping Ray +~~~~~~~~~~~~ + +**Stop Ray on worker nodes** + +.. code-block:: bash + + parallel-ssh -h workers.txt -P ray/scripts/stop_ray.sh + +This command will execute the ``stop_ray.sh`` script on each of the worker +nodes. + +**Stop Ray on the head node** + +.. code-block:: bash + + ray/scripts/stop_ray.sh + +Upgrading Ray +~~~~~~~~~~~~~ + +Ray remains under active development so you may at times want to upgrade the +cluster to take advantage of improvements and fixes. + +**Create an upgrade script** + +On the head node, create a file called ``upgrade.sh`` that contains the commands +necessary to upgrade Ray. It should look something like the following: + +.. code-block:: bash + + # Make sure the SSH session has the correct version of Python on its path. + # You will probably have to change the line below. + export PATH=/home/ubuntu/anaconda3/bin/:$PATH + # Do pushd/popd to make sure we end up in the same directory. + pushd . + # Upgrade Ray. + cd ray + git remote set-url origin https://github.com/ray-project/ray + git checkout master + git pull + cd python + python setup.py install --user + popd + +This script executes a series of git commands to update the Ray source code, then builds +and installs Ray. + +**Stop Ray on the cluster** + +Follow the instructions for `Stopping Ray`_. + +**Run the upgrade script on the cluster** + +First run the upgrade script on the head node. This will upgrade the head node +and help confirm that the upgrade script is working properly. + +.. code-block:: bash + + bash upgrade.sh + +Next run the upgrade script on the worker nodes. + +.. code-block:: bash + + parallel-ssh -h workers.txt -P -t 0 -I < upgrade.sh + +Note here that we use the ``-t 0`` option to set the timeout to infinite. + +It is probably a good idea to ssh to one of the other nodes and verify that the +upgrade script ran as expected. + +Sync Application Files to other nodes +------------------------------------- + +If you are running an application that reads input files or uses python +libraries then you may find it useful to copy a directory on the head node to +the worker nodes. + +You can do this using the ``parallel-rsync`` command: + +.. code-block:: bash + + parallel-rsync -h workers.txt -r /home/ubuntu/ + +where ```` is the directory you want to synchronize. Note that the +destination argument for this command must represent an absolute path on the +worker node. + +Troubleshooting +--------------- + +Problems with parallel-ssh +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If any of the above commands fail, verify that the head node has SSH access to +the other nodes by running + +.. code-block:: bash + + for host in $(cat workers.txt); do + ssh $host uptime + done + +If you get a permission denied error, then make sure you have SSH'ed to the head +node with agent forwarding enabled. This is done as follows. + +.. code-block:: bash + + ssh-add + ssh -A ubuntu@ + +Configuring EC2 instances to increase the number of allowed Redis clients +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This section can be ignored unless you run into problems with the maximum +number of Redis clients. + +* Ensure that the hard limit for the number of open file descriptors is set + to a large number (e.g., 65536). This only needs to be done on instances + where Redis shards will run --- by default, just the head node. + + * Check the hard ulimit for open file descriptors with ``ulimit -Hn``. + * If that number is smaller than 65536, set the hard ulimit for open file + descriptors system-wide: + + .. code-block:: bash + + sudo bash -c "echo $USER hard nofile 65536 >> /etc/security/limits.conf" + + * Logout and log back in.