Add multinode tests by simulating multiple nodes using Docker. (#378)

* run test workloads for a Docker cluster

* better manage docker image versions

* Changes to make multinode docker tests work with Python 3.

* option to mount local test directory on head node to speed development

* Attempt to simplify multinode test setup.

* Small change.

* Add in development-mode to run multinode docker tests more easily during development.

* add jenkins test script that links to Docker hash

* Read docker SHA from build_docker.sh and add test that should fail.

* Consolidate implementations and remove duplicate files.

* Allow test to retry if it fails to schedule on all nodes.

* Remove sleep when in docker multinode tests.

test/jenkins_tests/multi_node_tests/test_0.py

@@ -0,0 +1,22 @@
+import os
+import time
+
+import ray
+
+@ray.remote
+def f():
+  time.sleep(0.1)
+  return ray.services.get_node_ip_address()
+
+if __name__ == "__main__":
+  ray.init(redis_address=os.environ["RAY_REDIS_ADDRESS"])
+  # Check that tasks are scheduled on all nodes.
+  num_attempts = 30
+  for i in range(num_attempts):
+    ip_addresses = ray.get([f.remote() for i in range(1000)])
+    distinct_addresses = set(ip_addresses)
+    counts = [ip_addresses.count(address) for address in distinct_addresses]
+    print("Counts are {}".format(counts))
+    if len(counts) == 5:
+      break
+  assert len(counts) == 5