From 6147b6a1a3cc76a02928914804931c2e9d0479bf Mon Sep 17 00:00:00 2001 From: dHannasch Date: Wed, 4 Nov 2020 14:48:03 -0700 Subject: [PATCH] [docs] Note that the printed IP address can be incorrect. (#11804) * If the head node is on a subnet with NAT, then you will need a different IP address. * Specify what you are checking firewall settings and network configuration *for*. * reword following @amogkam * Give the full error message. --- doc/source/cluster/index.rst | 57 +++++++++++++++++++++++++++++++++--- 1 file changed, 53 insertions(+), 4 deletions(-) diff --git a/doc/source/cluster/index.rst b/doc/source/cluster/index.rst index 224bb2f19..b69cb5134 100644 --- a/doc/source/cluster/index.rst +++ b/doc/source/cluster/index.rst @@ -102,6 +102,13 @@ The command will print out the address of the Redis server that was started ``
`` with the value printed by the command on the head node (it should look something like ``123.45.67.89:6379``). +Note that if your compute nodes are on their own subnetwork with Network +Address Translation, to connect from a regular machine outside that subnetwork, +the command printed by the head node will not work. You need to find the +address that will reach the head node from the second machine. If the head node +has a domain address like compute04.berkeley.edu, you can simply use that in +place of an IP address and rely on the DNS. + .. code-block:: bash $ ray start --address=
--redis-password='' @@ -115,11 +122,53 @@ should look something like ``123.45.67.89:6379``). If you wish to specify that a machine has 10 CPUs and 1 GPU, you can do this with the flags ``--num-cpus=10`` and ``--num-gpus=1``. See the :ref:`Configuration ` page for more information. +If you see ``Unable to connect to Redis. If the Redis instance is on a +different machine, check that your firewall is configured properly.``, +this means the ``--port`` is inaccessible at the given IP address (because, for +example, the head node is not actually running Ray, or you have the wrong IP +address). + If you see ``Ray runtime started.``, then the node successfully connected to -the ``
``. If the ``
`` is inaccessible (because, for example, -the head node is not actually running), then you will get an error such as -``Unable to connect to Redis. If the Redis instance is on a different machine, -check that your firewall is configured properly.`` +the IP address at the ``--port``. You should now be able to connect to the +cluster with ``ray.init(address='auto')``. + +If ``ray.init(address='auto')`` keeps repeating +``redis_context.cc:303: Failed to connect to Redis, retrying.``, then the node +is failing to connect to some other port(s) besides the main port. + +.. code-block:: bash + + If connection fails, check your firewall settings and network configuration. + +If the connection fails, to check whether each port can be reached from a node, +you can use a tool such as ``nmap`` or ``nc``. + +.. code-block:: bash + + $ nmap -sV --reason -p $PORT $HEAD_ADDRESS + Nmap scan report for compute04.berkeley.edu (123.456.78.910) + Host is up, received echo-reply ttl 60 (0.00087s latency). + rDNS record for 123.456.78.910: compute04.berkeley.edu + PORT STATE SERVICE REASON VERSION + 6379/tcp open redis syn-ack ttl 60 Redis key-value store + Service detection performed. Please report any incorrect results at https://nmap.org/submit/ . + $ nc -vv -z $HEAD_ADDRESS $PORT + Connection to compute04.berkeley.edu 6379 port [tcp/*] succeeded! + +If the node cannot access that port at that IP address, you might see + +.. code-block:: bash + + $ nmap -sV --reason -p $PORT $HEAD_ADDRESS + Nmap scan report for compute04.berkeley.edu (123.456.78.910) + Host is up (0.0011s latency). + rDNS record for 123.456.78.910: compute04.berkeley.edu + PORT STATE SERVICE REASON VERSION + 6379/tcp closed redis reset ttl 60 + Service detection performed. Please report any incorrect results at https://nmap.org/submit/ . + $ nc -vv -z $HEAD_ADDRESS $PORT + nc: connect to compute04.berkeley.edu port 6379 (tcp) failed: Connection refused + Stopping Ray ~~~~~~~~~~~~