From a77b19e4f2ab743d1ebe464083b1d915babbd7ce Mon Sep 17 00:00:00 2001 From: Robert Zangnan Yu Date: Mon, 27 Apr 2020 09:30:16 -0700 Subject: [PATCH] [docs] Comments on potential srun orders during Slurm Deployment (#8183) --- doc/source/deploying-on-slurm.rst | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/doc/source/deploying-on-slurm.rst b/doc/source/deploying-on-slurm.rst index e65ee4ee4..04508747d 100644 --- a/doc/source/deploying-on-slurm.rst +++ b/doc/source/deploying-on-slurm.rst @@ -32,11 +32,15 @@ Clusters managed by Slurm may require that Ray is initialized as a part of the s srun --nodes=1 --ntasks=1 -w $node1 ray start --block --head --redis-port=6379 --redis-password=$redis_password & # Starting the head sleep 5 + # Make sure the head successfully starts before any worker does, otherwise + # the worker will not be able to connect to redis. In case of longer delay, + # adjust the sleeptime above to ensure proper order. for (( i=1; i<=$worker_num; i++ )) do node2=${nodes_array[$i]} srun --nodes=1 --ntasks=1 -w $node2 ray start --block --address=$ip_head --redis-password=$redis_password & # Starting the workers + # Flag --block will keep ray process alive on each compute node. sleep 5 done