Fixes and cleanups for the multinode setting. (#143)

* Add function for driver to get address info from Redis.

* Use Redis address instead of Redis port.

* Configure Redis to run in unprotected mode.

* Add method for starting Ray processes on non-head node.

* Pass in correct node ip address to start_plasma_manager.

* Script for starting Ray processes.

* Handle the case where an object already exists in the store. Maybe this should also compare the object hashes.

* Have driver get info from Redis when start_ray_local=False.

* Fix.

* Script for killing ray processes.

* Catch some errors when the main_loop in a worker throws an exception.

* Allow redirecting stdout and stderr to /dev/null.

* Wrap start_ray.py in a shell script.

* More helpful error messages.

* Fixes.

* Wait for redis server to start up before configuring it.

* Allow seeding of deterministic object ID generation.

* Small change.

src/common/redis_module/ray_redis_module.c

@@ -61,14 +61,25 @@ RedisModuleKey *OpenPrefixedKey(RedisModuleCtx *ctx,
 /**
  * Register a client with Redis. This is called from a client with the command:
  *
- *     RAY.CONNECT <client type> <address> <ray client id> <aux address>
+ *     RAY.CONNECT <ray client id> <node ip address> <client type> <field 1>
+ *         <value 1> <field 2> <value 2> ...
+ *
+ * The command can take an arbitrary number of pairs of field names and keys,
+ * and these will be stored in a hashmap associated with this client. Several
+ * fields are singled out for special treatment:
+ *
+ *     address: This is provided by plasma managers and it should be an address
+ *         like "127.0.0.1:1234". It is returned by RAY.GET_CLIENT_ADDRESS so
+ *         that other plasma managers know how to fetch objects.
+ *     aux_address: This is provided by local schedulers and should be the
+ *         address of the plasma manager that the local scheduler is connected
+ *         to. This is published to the "db_clients" channel by the RAY.CONNECT
+ *         command and is used by the global scheduler to determine which plasma
+ *         managers and local schedulers are connected.
  *
- * @param client_type The type of the client (e.g., plasma_manager).
- * @param address The address of the client.
  * @param ray_client_id The db client ID of the client.
- * @param aux_address An auxiliary address. This is currently just used by the
- *        local scheduler to record the address of the plasma manager that it is
- *        connected to.
+ * @param node_ip_address The IP address of the node the client is on.
+ * @param client_type The type of the client (e.g., plasma_manager).
  * @return OK if the operation was successful.
  */
 int Connect_RedisCommand(RedisModuleCtx *ctx,
@@ -95,7 +106,8 @@ int Connect_RedisCommand(RedisModuleCtx *ctx,
       RedisModule_CreateString(ctx, "aux_address", strlen("aux_address"));
 
   RedisModule_HashSet(db_client_table_key, REDISMODULE_HASH_CFIELDS,
-                      "node_ip_address", node_ip_address, NULL);
+                      "ray_client_id", ray_client_id, "node_ip_address",
+                      node_ip_address, "client_type", client_type, NULL);
 
   for (int i = 4; i < argc; i += 2) {
     RedisModuleString *key = argv[i];

src/common/state/redis.c

@@ -160,11 +160,19 @@ db_handle *db_connect(const char *db_address,
                       "could not establish synchronous connection to redis "
                       "%s:%d",
                       db_address, db_port);
-  /* Enable keyspace events. */
-  reply = redisCommand(context, "CONFIG SET notify-keyspace-events AKE");
+  /* Configure Redis to generate keyspace notifications for list events. This
+   * should only need to be done once (by whoever started Redis), but since
+   * Redis may be started in multiple places (e.g., for testing or when starting
+   * processes by hand), it is easier to do it multiple times. */
+  reply = redisCommand(context, "CONFIG SET notify-keyspace-events Kl");
   CHECKM(reply != NULL, "db_connect failed on CONFIG SET");
   freeReplyObject(reply);
-  /* Add new client using optimistic locking. */
+  /* Also configure Redis to not run in protected mode, so clients on other
+   * hosts can connect to it. */
+  reply = redisCommand(context, "CONFIG SET protected-mode no");
+  CHECKM(reply != NULL, "db_connect failed on CONFIG SET");
+  freeReplyObject(reply);
+  /* Create a client ID for this client. */
   db_client_id client = globally_unique_id();
 
   /* Construct the argument arrays for RAY.CONNECT. */