wassname/ray
1
0
Fork 0
mirror of https://github.com/wassname/ray.git synced 2026-09-12 12:51:15 +08:00
Code Issues Packages Projects Releases Wiki Activity

Move documentation to ReadTheDocs. (#326)

Browse Source
This commit is contained in:
Robert Nishihara
2017-02-27 21:14:31 -08:00
committed by Philipp Moritz
parent 1ae7e7d29e
commit 1a997ed279
32 changed files with 106 additions and 236 deletions
Download Patch File Download Diff File Expand all files Collapse all files
examples/hyperopt/README.md
-163
View File
@@ -1,163 +0,0 @@
# Hyperparameter Optimization
This document provides a walkthrough of the hyperparameter optimization example.
To run the application, first install this dependency.
- [TensorFlow](https://www.tensorflow.org/)
Then from the directory `ray/examples/hyperopt/` run the following.
```
python driver.py
```
Machine learning algorithms often have a number of *hyperparameters* whose
values must be chosen by the practitioner. For example, an optimization
algorithm may have a step size, a decay rate, and a regularization coefficient.
In a deep network, the network parameterization itself (e.g., the number of
layers and the number of units per layer) can be considered a hyperparameter.
Choosing these parameters can be challenging, and so a common practice is to
search over the space of hyperparameters. One approach that works surprisingly
well is to randomly sample different options.
## The serial version
Suppose that we want to train a convolutional network, but we aren't sure how to
choose the following hyperparameters:
- the learning rate
- the batch size
- the dropout probability
- the standard deviation of the distribution from which to initialize the
network weights
Suppose that we've defined a Python function `train_cnn_and_compute_accuracy`,
which takes values for these hyperparameters as its input (along with the
dataset), trains a convolutional network using those hyperparameters, and
returns the accuracy of the trained model on a validation set.
```python
def train_cnn_and_compute_accuracy(hyperparameters, train_images, train_labels, validation_images, validation_labels):
# Construct a deep network, train it, and return the validation accuracy.
# The argument hyperparameters is a dictionary with keys:
# - "learning_rate"
# - "batch_size"
# - "dropout"
# - "stddev"
return validation_accuracy
```
Something that works surprisingly well is to try random values for the
hyperparameters. For example, we can write the following.
```python
def generate_random_params():
# Randomly choose values for the hyperparameters
learning_rate = 10 ** np.random.uniform(-5, 5)
batch_size = np.random.randint(1, 100)
dropout = np.random.uniform(0, 1)
stddev = 10 ** np.random.uniform(-5, 5)
return {"learning_rate": learning_rate, "batch_size": batch_size, "dropout": dropout, "stddev": stddev}
results = []
for _ in range(100):
params = generate_random_params()
accuracy = train_cnn_and_compute_accuracy(randparams, train_images, train_labels, validation_images, validation_labels)
results.append(accuracy)
```
Then we can inspect the contents of `results` and see which set of
hyperparameters worked the best.
Of course, as there are no dependencies between the different invocations of
`train_cnn_and_compute_accuracy`, this computation could easily be parallelized
over multiple cores or multiple machines. Let's do that now.
## The distributed version
First, let's turn `train_cnn_and_compute_accuracy` into a remote function in Ray
by writing it as follows. In this example application, a slightly more
complicated version of this remote function is defined in
[hyperopt.py](hyperopt.py).
```python
@ray.remote
def train_cnn_and_compute_accuracy(hyperparameters, train_images, train_labels, validation_images, validation_labels):
# Actual work omitted.
return validation_accuracy
```
The only difference is that we added the `@ray.remote` decorator.
Now a call to `train_cnn_and_compute_accuracy` does not execute the function. It
submits the task to the scheduler and returns an object ID for the output
of the eventual computation. The scheduler, at its leisure, will schedule the
task on a worker (which may live on the same machine or on a different machine
in the cluster).
Now the for loop runs almost instantaneously because it does not do any actual
computation. Instead, it simply submits a number of tasks to the scheduler.
```python
result_ids = []
# Launch 100 tasks.
for _ in range(100):
params = generate_random_params()
accuracy_id = train_cnn_and_compute_accuracy.remote(randparams, train_images, train_labels, validation_images, validation_labels)
result_ids.append(accuracy_id)
```
If we wish to wait until the results have all been retrieved, we can retrieve
their values with `ray.get`.
```python
results = ray.get(result_ids)
```
One drawback of the above approach is that nothing will be printed until all of
the experiments have finished. What we'd really like is to start processing
the results of certain experiments as soon as they finish (and possibly launch
more experiments based on the outcomes of the first ones). To do this, we can
use `ray.wait`, which takes a list of object IDs and returns two lists of object
IDs.
```python
ready_ids, remaining_ids = ray.wait(result_ids, num_returns=3, timeout=10)
```
In the above, `result_ids` is a list of object IDs. The command `ray.wait` will
return as soon as either three of the object IDs in `result_ids` are ready (that
is, the task that created the corresponding object finished executing and stored
the object in the object store) or ten seconds pass, whichever comes first. To
wait indefinitely, omit the timeout argument. Now, we can rewrite the script as
follows.
```python
remaining_ids = []
# Launch 100 tasks.
for _ in range(100):
params = generate_random_params()
accuracy_id = train_cnn_and_compute_accuracy.remote(randparams, train_images, train_labels, validation_images, validation_labels)
remaining_ids.append(accuracy_id)
# Process the tasks one at a time.
while len(remaining_ids) > 0:
# Process the next task that finishes.
ready_ids, remaining_ids = ray.wait(remaining_ids, num_returns=1)
# Get the accuracy corresponding to the ready object ID.
accuracy = ray.get(ready_ids[0])
print("Accuracy {}".format(accuracy))
```
Note that the above example does not associate the accuracy with the parameters
that produced that accuracy, but this is done in the actual script.
## Additional notes
**Early Stopping:** Sometimes when running an optimization, it is clear early on
that the hyperparameters being used are bad (for example, the loss function may
start diverging). In these situations, it makes sense to end that particular run
early to save resources. This is implemented within the remote function
`train_cnn_and_compute_accuracy`. If it detects that the optimization is going
poorly, it returns early.
examples/lbfgs/README.md
-150
View File
@@ -1,150 +0,0 @@
# Batch L-BFGS
This document provides a walkthrough of the L-BFGS example. To run the
application, first install these dependencies.
- SciPy
- [TensorFlow](https://www.tensorflow.org/)
Then from the directory `ray/examples/lbfgs/` run the following.
```
python driver.py
```
Optimization is at the heart of many machine learning algorithms. Much of
machine learning involves specifying a loss function and finding the parameters
that minimize the loss. If we can compute the gradient of the loss function,
then we can apply a variety of gradient-based optimization algorithms. L-BFGS is
one such algorithm. It is a quasi-Newton method that uses gradient information
to approximate the inverse Hessian of the loss function in a computationally
efficient manner.
## The serial version
First we load the data in batches. Here, each element in `batches` is a tuple
whose first component is a batch of `100` images and whose second component is a
batch of the `100` corresponding labels. For simplicity, we use TensorFlow's
built in methods for loading the data.
```python
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets("MNIST_data/", one_hot=True)
batch_size = 100
num_batches = mnist.train.num_examples // batch_size
batches = [mnist.train.next_batch(batch_size) for _ in range(num_batches)]
```
Now, suppose we have defined a function which takes a set of model parameters
`theta` and a batch of data (both images and labels) and computes the loss for
that choice of model parameters on that batch of data. Similarly, suppose we've
also defined a function that takes the same arguments and computes the gradient
of the loss for that choice of model parameters.
```python
def loss(theta, xs, ys):
# compute the loss on a batch of data
return loss
def grad(theta, xs, ys):
# compute the gradient on a batch of data
return grad
def full_loss(theta):
# compute the loss on the full data set
return sum([loss(theta, xs, ys) for (xs, ys) in batches])
def full_grad(theta):
# compute the gradient on the full data set
return sum([grad(theta, xs, ys) for (xs, ys) in batches])
```
Since we are working with a small dataset, we don't actually need to separate
these methods into the part that operates on a batch and the part that operates
on the full dataset, but doing so will make the distributed version clearer.
Now, if we wish to optimize the loss function using L-BFGS, we simply plug these
functions, along with an initial choice of model parameters, into
`scipy.optimize.fmin_l_bfgs_b`.
```python
theta_init = 1e-2 * np.random.normal(size=dim)
result = scipy.optimize.fmin_l_bfgs_b(full_loss, theta_init, fprime=full_grad)
```
## The distributed version
In this example, the computation of the gradient itself can be done in parallel
on a number of workers or machines.
First, let's turn the data into a collection of remote objects.
```python
batch_ids = [(ray.put(xs), ray.put(ys)) for (xs, ys) in batches]
```
We can load the data on the driver and distribute it this way because MNIST
easily fits on a single machine. However, for larger data sets, we will need to
use remote functions to distribute the loading of the data.
Now, lets turn `loss` and `grad` into remote functions.
```python
@ray.remote
def loss(theta, xs, ys):
# compute the loss
return loss
@ray.remote
def grad(theta, xs, ys):
# compute the gradient
return grad
```
The only difference is that we added the `@ray.remote` decorator.
Now, it is easy to speed up the computation of the full loss and the full
gradient.
```python
def full_loss(theta):
theta_id = ray.put(theta)
loss_ids = [loss.remote(theta_id, xs_id, ys_id) for (xs_id, ys_id) in batch_ids]
return sum(ray.get(loss_ids))
def full_grad(theta):
theta_id = ray.put(theta)
grad_ids = [grad.remote(theta_id, xs_id, ys_id) for (xs_id, ys_id) in batch_ids]
return sum(ray.get(grad_ids)).astype("float64") # This conversion is necessary for use with fmin_l_bfgs_b.
```
Note that we turn `theta` into a remote object with the line `theta_id =
ray.put(theta)` before passing it into the remote functions. If we had written
```python
[loss.remote(theta, xs_id, ys_id) for (xs_id, ys_id) in batch_ids]
```
instead of
```python
theta_id = ray.put(theta)
[loss.remote(theta_id, xs_id, ys_id) for (xs_id, ys_id) in batch_ids]
```
then each task that got sent to the scheduler (one for every element of
`batch_ids`) would have had a copy of `theta` serialized inside of it. Since
`theta` here consists of the parameters of a potentially large model, this is
inefficient. *Large objects should be passed by object ID to remote functions
and not by value*.
We use remote functions and remote objects internally in the implementation of
`full_loss` and `full_grad`, but the user-facing behavior of these methods is
identical to the behavior in the serial version.
We can now optimize the objective with the same function call as before.
```python
theta_init = 1e-2 * np.random.normal(size=dim)
result = scipy.optimize.fmin_l_bfgs_b(full_loss, theta_init, fprime=full_grad)
```
examples/rl_pong/README.md
-113
View File
@@ -1,113 +0,0 @@
# Learning to Play Pong
In this example, we'll be training a neural network to play Pong using the
OpenAI Gym. This application is adapted, with minimal modifications, from Andrej
Karpathy's
[code](https://gist.github.com/karpathy/a4166c7fe253700972fcbc77e4ea32c5) (see
the accompanying [blog post](http://karpathy.github.io/2016/05/31/rl/)). To run
the application, first install this dependency.
- [Gym](https://gym.openai.com/)
Then from the directory `ray/examples/rl_pong/` run the following.
```
python driver.py
```
## The distributed version
At the core of [Andrej's
code](https://gist.github.com/karpathy/a4166c7fe253700972fcbc77e4ea32c5), a
neural network is used to define a "policy" for playing Pong (that is, a
function that chooses an action given a state). In the loop, the network
repeatedly plays games of Pong and records a gradient from each game. Every ten
games, the gradients are combined together and used to update the network.
This example is easy to parallelize because the network can play ten games in
parallel and no information needs to be shared between the games. We define a
remote function `compute_gradient`, which plays a game of pong and returns an
estimate of the gradient. Below is a simplified pseudocode version of this
function.
```python
@ray.remote(num_return_vals=2)
def compute_gradient(model):
# Retrieve the game environment.
env = ray.env.env
# Reset the game.
observation = env.reset()
while not done:
# Choose an action using policy_forward.
# Take the action and observe the new state of the world.
# Compute a gradient using policy_backward. Return the gradient and reward.
return gradient, reward_sum
```
Calling this remote function inside of a for loop, we launch multiple tasks to
perform rollouts and compute gradients. If we have at least ten worker
processes, then these tasks will all be executed in parallel.
```python
model_id = ray.put(model)
grads, reward_sums = [], []
# Launch tasks to compute gradients from multiple rollouts in parallel.
for i in range(10):
grad_id, reward_sum_id = compute_gradient.remote(model_id)
grads.append(grad_id)
reward_sums.append(reward_sum_id)
```
### Reusing the Gym environment
Workers are long-running Python processes, and though we'd like to think of
workers as being stateless, sometimes it's important to have a variable that
gets shared between different tasks on the same worker (perhaps because it is
expensive to initialize the variable).
In this example, we'd like each worker to have access to a Pong environment. The
Pong environment has state that gets mutated by the task, and this state is
shared between tasks that run on the same worker, so there is some danger that
the output of the overall program will depend on which tasks are scheduled on
which workers. This can be avoided if the state of the Pong environment is reset
between tasks.
To accomplish this, the user must mark the Pong environment as an environment
variable. This is done by providing a method for initializing the gym, and
storing it in `ray.env`.
```python
# Function for initializing the gym environment.
def env_initializer():
return gym.make("Pong-v0")
# Create an environment variable for the gym environment.
ray.env.env = ray.EnvironmentVariable(env_initializer)
```
A remote task can then call `ray.env.env` to retrieve the variable.
By default, whenever a task uses the `ray.env.env` variable, the worker
that the task was scheduled on will rerun the initialization code
`env_initializer` after the task has finished so that state will not leak
between the tasks.
However, sometimes the initialization code is expensive, and there may be a
faster way to reinitialize the variable (or maybe no reinitialization is needed
at all). In these cases, the user can provide a custom **reinitializer**, which
gets run after any task that uses the variable.
```python
# Function for initializing the gym environment.
def env_initializer():
return gym.make("Pong-v0")
# Function for reinitializing the gym environment in order to guarantee that
# the state of the game is reset after each remote task.
def env_reinitializer(env):
env.reset()
return env
# Create an environment variable for the gym environment.
ray.env.env = ray.EnvironmentVariable(env_initializer, env_reinitializer)
```