Automatically add relevant directories to Python paths of workers (#380)

* Make ray.init set python paths of workers.

* Decouple starting cluster from copying user source code

* also add current directory to path

* Add comments about deallocation.

* Add test for new code path.
This commit is contained in:
Robert Nishihara
2016-08-16 14:53:55 -07:00
committed by Philipp Moritz
parent 7246013008
commit e06311d415
12 changed files with 222 additions and 64 deletions
+9 -12
View File
@@ -127,14 +127,9 @@ the tests.
python test/array_test.py # This tests some array libraries.
```
6. Start the cluster with `cluster.start_ray()`. If you would like to deploy
source code to it, you can pass in the local path to the directory that contains
your Python code. For example, `cluster.start_ray("~/example_ray_code")`. This
will copy your source code to each node on the cluster, placing it in a
directory on the PYTHONPATH.
The `cluster.start_ray` command will start the Ray scheduler, object stores, and
workers, and before finishing it will print instructions for connecting to the
cluster via ssh.
6. Start the cluster with `cluster.start_ray()`. The `cluster.start_ray` command
will start the Ray scheduler, object stores, and workers, and before finishing
it will print instructions for connecting to the cluster via ssh.
7. To connect to the cluster (either with a Python shell or with a script), ssh
to the cluster's head node (as described by the output of the
@@ -146,7 +141,6 @@ to the cluster's head node (as described by the output of the
Then run the following commands.
cd $HOME/ray
source $HOME/ray/setup-env.sh # Add Ray to your Python path.
Then within a Python interpreter, run the following commands.
@@ -177,11 +171,14 @@ need to install a few more Python packages. This can be done, within
- `cluster.install_ray()` - This pulls the Ray source code on each node,
builds all of the third party libraries, and builds the project itself.
- `cluster.start_ray(user_source_directory=None, num_workers_per_node=10)` -
This starts a scheduler process on the head node, and it starts an object
store and some workers on each node.
- `cluster.start_ray(num_workers_per_node=10)` - This starts a scheduler
process on the head node, and it starts an object store and some workers
on each node.
- `cluster.stop_ray()` - This shuts down the cluster (killing all of the
processes).
- `cluster.copy_code_to_cluster(user_source_directory)` - This copies the
contents of `user_source_directory` locally to the cluster under
`~/ray_source_files/`.
- `cluster.update_ray()` - This pulls the latest Ray source code and builds
it.
- `cluster.run_command_over_ssh_on_all_nodes_in_parallel(command)` - This