wassname/ray
1
0
Fork 0
mirror of https://github.com/wassname/ray.git synced 2026-07-24 13:20:22 +08:00
Code Issues Packages Projects Releases Wiki Activity

Update hyperparameter optimization example. (#332)

Browse Source 
* Update hyperparameter optimization example.

* Remove early stopping.
This commit is contained in:
Robert Nishihara
2017-03-04 10:45:15 -08:00
committed by Philipp Moritz
parent 41b8675d04
commit 0a233b7144
6 changed files with 453 additions and 218 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/source/example-hyperopt.md
-163
View File
@@ -1,163 +0,0 @@
# Hyperparameter Optimization
This document provides a walkthrough of the hyperparameter optimization example.
To run the application, first install this dependency.
- [TensorFlow](https://www.tensorflow.org/)
Then from the directory `ray/examples/hyperopt/` run the following.
```
python driver.py
```
Machine learning algorithms often have a number of *hyperparameters* whose
values must be chosen by the practitioner. For example, an optimization
algorithm may have a step size, a decay rate, and a regularization coefficient.
In a deep network, the network parameterization itself (e.g., the number of
layers and the number of units per layer) can be considered a hyperparameter.
Choosing these parameters can be challenging, and so a common practice is to
search over the space of hyperparameters. One approach that works surprisingly
well is to randomly sample different options.
## The serial version
Suppose that we want to train a convolutional network, but we aren't sure how to
choose the following hyperparameters:
- the learning rate
- the batch size
- the dropout probability
- the standard deviation of the distribution from which to initialize the
network weights
Suppose that we've defined a Python function `train_cnn_and_compute_accuracy`,
which takes values for these hyperparameters as its input (along with the
dataset), trains a convolutional network using those hyperparameters, and
returns the accuracy of the trained model on a validation set.
```python
def train_cnn_and_compute_accuracy(hyperparameters, train_images, train_labels, validation_images, validation_labels):
# Construct a deep network, train it, and return the validation accuracy.
# The argument hyperparameters is a dictionary with keys:
# - "learning_rate"
# - "batch_size"
# - "dropout"
# - "stddev"
return validation_accuracy
```
Something that works surprisingly well is to try random values for the
hyperparameters. For example, we can write the following.
```python
def generate_random_params():
# Randomly choose values for the hyperparameters
learning_rate = 10 ** np.random.uniform(-5, 5)
batch_size = np.random.randint(1, 100)
dropout = np.random.uniform(0, 1)
stddev = 10 ** np.random.uniform(-5, 5)
return {"learning_rate": learning_rate, "batch_size": batch_size, "dropout": dropout, "stddev": stddev}
results = []
for _ in range(100):
params = generate_random_params()
accuracy = train_cnn_and_compute_accuracy(randparams, train_images, train_labels, validation_images, validation_labels)
results.append(accuracy)
```
Then we can inspect the contents of `results` and see which set of
hyperparameters worked the best.
Of course, as there are no dependencies between the different invocations of
`train_cnn_and_compute_accuracy`, this computation could easily be parallelized
over multiple cores or multiple machines. Let's do that now.
## The distributed version
First, let's turn `train_cnn_and_compute_accuracy` into a remote function in Ray
by writing it as follows. In this example application, a slightly more
complicated version of this remote function is defined in
[hyperopt.py](hyperopt.py).
```python
@ray.remote
def train_cnn_and_compute_accuracy(hyperparameters, train_images, train_labels, validation_images, validation_labels):
# Actual work omitted.
return validation_accuracy
```
The only difference is that we added the `@ray.remote` decorator.
Now a call to `train_cnn_and_compute_accuracy` does not execute the function. It
submits the task to the scheduler and returns an object ID for the output
of the eventual computation. The scheduler, at its leisure, will schedule the
task on a worker (which may live on the same machine or on a different machine
in the cluster).
Now the for loop runs almost instantaneously because it does not do any actual
computation. Instead, it simply submits a number of tasks to the scheduler.
```python
result_ids = []
# Launch 100 tasks.
for _ in range(100):
params = generate_random_params()
accuracy_id = train_cnn_and_compute_accuracy.remote(randparams, train_images, train_labels, validation_images, validation_labels)
result_ids.append(accuracy_id)
```
If we wish to wait until the results have all been retrieved, we can retrieve
their values with `ray.get`.
```python
results = ray.get(result_ids)
```
One drawback of the above approach is that nothing will be printed until all of
the experiments have finished. What we'd really like is to start processing
the results of certain experiments as soon as they finish (and possibly launch
more experiments based on the outcomes of the first ones). To do this, we can
use `ray.wait`, which takes a list of object IDs and returns two lists of object
IDs.
```python
ready_ids, remaining_ids = ray.wait(result_ids, num_returns=3, timeout=10)
```
In the above, `result_ids` is a list of object IDs. The command `ray.wait` will
return as soon as either three of the object IDs in `result_ids` are ready (that
is, the task that created the corresponding object finished executing and stored
the object in the object store) or ten seconds pass, whichever comes first. To
wait indefinitely, omit the timeout argument. Now, we can rewrite the script as
follows.
```python
remaining_ids = []
# Launch 100 tasks.
for _ in range(100):
params = generate_random_params()
accuracy_id = train_cnn_and_compute_accuracy.remote(randparams, train_images, train_labels, validation_images, validation_labels)
remaining_ids.append(accuracy_id)
# Process the tasks one at a time.
while len(remaining_ids) > 0:
# Process the next task that finishes.
ready_ids, remaining_ids = ray.wait(remaining_ids, num_returns=1)
# Get the accuracy corresponding to the ready object ID.
accuracy = ray.get(ready_ids[0])
print("Accuracy {}".format(accuracy))
```
Note that the above example does not associate the accuracy with the parameters
that produced that accuracy, but this is done in the actual script.
## Additional notes
**Early Stopping:** Sometimes when running an optimization, it is clear early on
that the hyperparameters being used are bad (for example, the loss function may
start diverging). In these situations, it makes sense to end that particular run
early to save resources. This is implemented within the remote function
`train_cnn_and_compute_accuracy`. If it detects that the optimization is going
poorly, it returns early.
doc/source/example-hyperopt.rst
+240
View File
@@ -0,0 +1,240 @@
Hyperparameter Optimization
===========================
This document provides a walkthrough of the hyperparameter optimization example.
To run the application, first install some dependencies.
.. code-block:: bash
pip install tensorflow
You can view the `code for this example`_.
.. _`code for this example`: https://github.com/ray-project/ray/tree/master/examples/hyperopt
The simple script that processes results as they become available and launches
new experiments can be run as follows.
.. code-block:: bash
python ray/examples/hyperopt/hyperopt_simple.py --trials=5 --steps=10
The variant that divides training into multiple segments and aggressively
terminates poorly performing models can be run as follows.
.. code-block:: bash
python ray/examples/hyperopt/hyperopt_adaptive.py --num-starting-segments=5 \
--num-segments=10 \
--steps-per-segment=20
Machine learning algorithms often have a number of *hyperparameters* whose
values must be chosen by the practitioner. For example, an optimization
algorithm may have a step size, a decay rate, and a regularization coefficient.
In a deep network, the network parameterization itself (e.g., the number of
layers and the number of units per layer) can be considered a hyperparameter.
Choosing these parameters can be challenging, and so a common practice is to
search over the space of hyperparameters. One approach that works surprisingly
well is to randomly sample different options.
Problem Setup
-------------
Suppose that we want to train a convolutional network, but we aren't sure how to
choose the following hyperparameters:
- the learning rate
- the batch size
- the dropout probability
- the standard deviation of the distribution from which to initialize the
network weights
Suppose that we've defined a remote function ``train_cnn_and_compute_accuracy``,
which takes values for these hyperparameters as its input (along with the
dataset), trains a convolutional network using those hyperparameters, and
returns the accuracy of the trained model on a validation set.
.. code-block:: python
import numpy as np
import ray
@ray.remote
def train_cnn_and_compute_accuracy(hyperparameters,
train_images,
train_labels,
validation_images,
validation_labels):
# Construct a deep network, train it, and return the accuracy on the
# validation data.
return np.random.uniform(0, 1)
Basic random search
-------------------
Something that works surprisingly well is to try random values for the
hyperparameters. For example, we can write a function that randomly generates
hyperparameter configurations.
.. code-block:: python
def generate_hyperparameters():
# Randomly choose values for the hyperparameters.
return {"learning_rate": 10 ** np.random.uniform(-5, 5),
"batch_size": np.random.randint(1, 100),
"dropout": np.random.uniform(0, 1),
"stddev": 10 ** np.random.uniform(-5, 5)}
In addition, let's assume that we've started Ray and loaded some data.
.. code-block:: python
import ray
ray.init()
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets("MNIST_data", one_hot=True)
train_images = ray.put(mnist.train.images)
train_labels = ray.put(mnist.train.labels)
validation_images = ray.put(mnist.validation.images)
validation_labels = ray.put(mnist.validation.labels)
Then basic random hyperparameter search looks something like this. We launch a
bunch of experiments, and we get the results.
.. code-block:: python
# Generate a bunch of hyperparameter configurations.
hyperparameter_configurations = [generate_hyperparameters() for _ in range(20)]
# Launch some experiments.
results = []
for hyperparameters in hyperparameter_configurations:
results.append(train_cnn_and_compute_accuracy.remote(hyperparameters,
train_images,
train_labels,
validation_images,
validation_labels))
# Get the results.
accuracies = ray.get(results)
Then we can inspect the contents of `accuracies` and see which set of
hyperparameters worked the best. Note that in the above example, the for loop
will run instantaneously and the program will block in the call to ``ray.get``,
which will wait until all of the experiments have finished.
Processing results as they become available
-------------------------------------------
One problem with the above approach is that you have to wait for all of the
experiments to finish before you can process the results. Instead, you may want
to process the results as they become available, perhaps in order to adaptively
choose new experiments to run, or perhaps simply so you know how well the
experiments are doing. To process the results as they become available, we can
use the ``ray.wait`` primitive.
The most simple usage is the following. This example is implemented in more
detail in driver.py_.
.. code-block:: python
# Launch some experiments.
remaining_ids = []
for hyperparameters in hyperparameter_configurations:
remaining_ids.append(train_cnn_and_compute_accuracy.remote(hyperparameters,
train_images,
train_labels,
validation_images,
validation_labels))
# Whenever a new experiment finishes, print the value and start a new
# experiment.
for i in range(100):
ready_ids, remaining_ids = ray.wait(remaining_ids, num_returns=1)
accuracy = ray.get(ready_ids[0])
print("Accuracy is {}".format(accuracy))
# Start a new experiment.
new_hyperparameters = generate_hyperparameters()
remaining_ids.append(train_cnn_and_compute_accuracy.remote(new_hyperparameters,
train_images,
train_labels,
validation_images,
validation_labels))
.. _driver.py: https://github.com/ray-project/ray/blob/master/examples/hyperopt/driver.py
More sophisticated hyperparameter search
----------------------------------------
Hyperparameter search algorithms can get much more sophisticated. So far, we've
been treating the function ``train_cnn_and_compute_accuracy`` as a black box,
that we can choose its inputs and inspect its outputs, but once we decide to run
it, we have to run it until it finishes.
However, there is often more structure to be exploited. For example, if the
training procedure is going poorly, we can end the session early and invest more
resources in the more promising hyperparameter experiments. And if we've saved
the state of the training procedure, we can always restart it again later.
This is one of the ideas of the Hyperband_ algorithm. Start with a huge number
of hyperparameter configurations, aggressively stop the bad ones, and invest
more resources in the promising experiments.
To implement this, we can first adapt our training method to optionally take a
model and to return the updated model.
.. code-block:: python
@ray.remote
def train_cnn_and_compute_accuracy(hyperparameters, model=None):
# Construct a deep network, train it, and return the accuracy on the
# validation data as well as the latest version of the model. If the model
# argument is not None, this will continue training an existing model.
validation_accuracy = np.random.uniform(0, 1)
new_model = model
return validation_accuracy, new_model
Here's a different variant that uses the same principles. Divide each training
session into a series of shorter training sessions. Whenever a short session
finishes, if it still looks promising, then continue running it. If it isn't
doing well, then terminate it and start a new experiment.
.. code-block:: python
import numpy as np
def is_promising(model):
# Return true if the model is doing well and false otherwise. In practice,
# this function will want more information than just the model.
return np.random.choice([True, False])
# Start 10 experiments.
remaining_ids = []
for _ in range(10):
experiment_id = train_cnn_and_compute_accuracy.remote(hyperparameters, model=None)
remaining_ids.append(experiment_id)
accuracies = []
for i in range(100):
# Whenever a segment of an experiment finishes, decide if it looks promising
# or not.
ready_ids, remaining_ids = ray.wait(remaining_ids, num_returns=1)
experiment_id = ready_ids[0]
current_accuracy, current_model = ray.get(experiment_id)
accuracies.append(current_accuracy)
if is_promising(experiment_id):
# Continue running the experiment.
experiment_id = train_cnn_and_compute_accuracy.remote(hyperparameters,
model=current_model)
else:
# Start a new experiment.
experiment_id = train_cnn_and_compute_accuracy.remote(hyperparameters)
remaining_ids.append(experiment_id)
.. _Hyperband: https://arxiv.org/abs/1603.06560
doc/source/index.rst
+1 -1
View File
@@ -18,7 +18,7 @@ learning and reinforcement learning applications.*
:maxdepth: 1
:caption: Examples
example-hyperopt.md
example-hyperopt.rst
example-lbfgs.md
example-rl-pong.md
using-ray-with-tensorflow.md
examples/hyperopt/hyperopt_adaptive.py
+141
View File
@@ -0,0 +1,141 @@
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
import argparse
from collections import defaultdict
import numpy as np
import ray
import tensorflow as tf
from tensorflow.examples.tutorials.mnist import input_data
import objective
parser = argparse.ArgumentParser(description="Run the hyperparameter optimization example.")
parser.add_argument("--num-starting-segments", default=5, type=int, help="The number of training segments to start in parallel.")
parser.add_argument("--num-segments", default=10, type=int, help="The number of additional training segments to perform.")
parser.add_argument("--steps-per-segment", default=20, type=int, help="The number of steps of training to do per training segment.")
parser.add_argument("--redis-address", default=None, type=str, help="The Redis address of the cluster.")
if __name__ == "__main__":
args = parser.parse_args()
ray.init(redis_address=args.redis_address)
# The number of training passes over the dataset to use for network.
steps = args.steps_per_segment
# Load the mnist data and turn the data into remote objects.
print("Downloading the MNIST dataset. This may take a minute.")
mnist = input_data.read_data_sets("MNIST_data", one_hot=True)
train_images = ray.put(mnist.train.images)
train_labels = ray.put(mnist.train.labels)
validation_images = ray.put(mnist.validation.images)
validation_labels = ray.put(mnist.validation.labels)
# Keep track of the accuracies that we've seen at different numbers of
# iterations.
accuracies_by_num_steps = defaultdict(lambda: [])
# Define a method to determine if an experiment looks promising or not.
def is_promising(experiment_info):
accuracies = experiment_info["accuracies"]
total_num_steps = experiment_info["total_num_steps"]
comparable_accuracies = accuracies_by_num_steps[total_num_steps]
if len(comparable_accuracies) == 0:
if len(accuracies) == 1:
# This means that we haven't seen anything finish yet, so keep running
# this experiment.
return True
else:
# The experiment is promising if the second half of the accuracies are
# better than the first half of the accuracies.
return np.mean(accuracies[:len(accuracies) // 2]) < np.mean(accuracies[len(accuracies) // 2:])
# Otherwise, continue running the experiment if it is in the top half of
# experiments we've seen so far at this point in time.
return np.mean(accuracy > np.array(comparable_accuracies)) > 0.5
# Keep track of all of the experiment segments that we're running. This
# dictionary uses the object ID of the experiment as the key.
experiment_info = {}
# Keep track of the curently running experiment IDs.
remaining_ids = []
# Keep track of the best hyperparameters and the best accuracy.
best_hyperparameters = None
best_accuracy = 0
# A function for generating random hyperparameters.
def generate_hyperparameters():
return {"learning_rate": 10 ** np.random.uniform(-5, 5),
"batch_size": np.random.randint(1, 100),
"dropout": np.random.uniform(0, 1),
"stddev": 10 ** np.random.uniform(-5, 5)}
# Launch some initial experiments.
for _ in range(args.num_starting_segments):
hyperparameters = generate_hyperparameters()
experiment_id = objective.train_cnn_and_compute_accuracy.remote(
hyperparameters, steps, train_images, train_labels, validation_images,
validation_labels)
experiment_info[experiment_id] = {"hyperparameters": hyperparameters,
"total_num_steps": steps,
"accuracies": []}
remaining_ids.append(experiment_id)
for _ in range(args.num_segments):
# Wait for a segment of an experiment to finish.
ready_ids, remaining_ids = ray.wait(remaining_ids, num_returns=1)
experiment_id = ready_ids[0]
# Get the accuracy and the weights.
accuracy, weights = ray.get(experiment_id)
# Update the experiment info.
previous_info = experiment_info[experiment_id]
previous_info["accuracies"].append(accuracy)
# Update the best accuracy and best hyperparameters.
if accuracy > best_accuracy:
best_hyperparameters = hyperparameters
best_accuracy = accuracy
if is_promising(previous_info):
# If the experiment still looks promising, then continue running it.
print("Continuing to run the experiment with hyperparameters {}.".format(
previous_info["hyperparameters"]))
new_hyperparameters = previous_info["hyperparameters"]
new_info = {"hyperparameters": new_hyperparameters,
"total_num_steps": previous_info["total_num_steps"] + steps,
"accuracies": previous_info["accuracies"].copy()}
starting_weights = weights
else:
# If the experiment does not look promising, start a new experiment.
print("Ending the experiment with hyperparameters {}.".format(
previous_info["hyperparameters"]))
new_hyperparameters = generate_hyperparameters()
new_info = {"hyperparameters": new_hyperparameters,
"total_num_steps": steps,
"accuracies": []}
starting_weights = None
# Start running the next segment.
new_experiment_id = objective.train_cnn_and_compute_accuracy.remote(
new_hyperparameters, steps, train_images, train_labels,
validation_images, validation_labels, weights=starting_weights)
experiment_info[new_experiment_id] = new_info
remaining_ids.append(new_experiment_id)
# Update the set of all accuracies that we've seen.
accuracies_by_num_steps[previous_info["total_num_steps"]].append(accuracy)
# Record the best performing set of hyperparameters.
print("""Best accuracy was {:.3} with
learning_rate: {:.2}
batch_size: {}
dropout: {:.2}
stddev: {:.2}
""".format(100 * best_accuracy,
best_hyperparameters["learning_rate"],
best_hyperparameters["batch_size"],
best_hyperparameters["dropout"],
best_hyperparameters["stddev"]))
examples/hyperopt/driver.py → examples/hyperopt/hyperopt_simple.py
+31 -24
View File
@@ -1,6 +1,3 @@
# Most of the tensorflow code is adapted from Tensorflow's tutorial on using CNNs to train MNIST
# https://www.tensorflow.org/versions/r0.9/tutorials/mnist/pros/index.html#build-a-multilayer-convolutional-network
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
@@ -12,16 +9,17 @@ import argparse
import tensorflow as tf
from tensorflow.examples.tutorials.mnist import input_data
import hyperopt
import objective
parser = argparse.ArgumentParser(description="Run the hyperparameter optimization example.")
parser.add_argument("--trials", default=2, type=int, help="The number of random trials to do.")
parser.add_argument("--steps", default=10, type=int, help="The number of steps of training to do per network.")
parser.add_argument("--redis-address", default=None, type=str, help="The Redis address of the cluster.")
if __name__ == "__main__":
args = parser.parse_args()
ray.init(num_workers=10)
ray.init(redis_address=args.redis_address)
# The number of sets of random hyperparameters to try.
trials = args.trials
@@ -36,31 +34,32 @@ if __name__ == "__main__":
validation_images = ray.put(mnist.validation.images)
validation_labels = ray.put(mnist.validation.labels)
# Keep track of the best parameters and the best accuracy.
best_params = None
# Keep track of the best hyperparameters and the best accuracy.
best_hyperparamemeters = None
best_accuracy = 0
# This list holds the object IDs for all of the experiments that we have
# launched and that have not yet been processed.
remaining_ids = []
# This is a dictionary mapping the object ID of an experiment to the
# parameters used for that experiment.
params_mapping = {}
# hyerparameters used for that experiment.
hyperparameters_mapping = {}
# A function for generating random hyperparameters.
def generate_random_params():
learning_rate = 10 ** np.random.uniform(-5, 5)
batch_size = np.random.randint(1, 100)
dropout = np.random.uniform(0, 1)
stddev = 10 ** np.random.uniform(-5, 5)
return {"learning_rate": learning_rate, "batch_size": batch_size, "dropout": dropout, "stddev": stddev}
def generate_hyperparameters():
return {"learning_rate": 10 ** np.random.uniform(-5, 5),
"batch_size": np.random.randint(1, 100),
"dropout": np.random.uniform(0, 1),
"stddev": 10 ** np.random.uniform(-5, 5)}
# Randomly generate some hyperparameters, and launch a task for each set.
for i in range(trials):
params = generate_random_params()
accuracy_id = hyperopt.train_cnn_and_compute_accuracy.remote(params, steps, train_images, train_labels, validation_images, validation_labels)
hyperparameters = generate_hyperparameters()
accuracy_id = objective.train_cnn_and_compute_accuracy.remote(
hyperparameters, steps, train_images, train_labels, validation_images,
validation_labels)
remaining_ids.append(accuracy_id)
# Keep track of which parameters correspond to this experiment.
params_mapping[accuracy_id] = params
# Keep track of which hyperparameters correspond to this experiment.
hyperparameters_mapping[accuracy_id] = hyperparameters
# Fetch and print the results of the tasks in the order that they complete.
for i in range(trials):
@@ -68,16 +67,20 @@ if __name__ == "__main__":
ready_ids, remaining_ids = ray.wait(remaining_ids)
# Process the output of this task.
result_id = ready_ids[0]
params = params_mapping[result_id]
accuracy = ray.get(result_id)
hyperparameters = hyperparameters_mapping[result_id]
accuracy, _ = ray.get(result_id)
print("""We achieve accuracy {:.3}% with
learning_rate: {:.2}
batch_size: {}
dropout: {:.2}
stddev: {:.2}
""".format(100 * accuracy, params["learning_rate"], params["batch_size"], params["dropout"], params["stddev"]))
""".format(100 * accuracy,
hyperparameters["learning_rate"],
hyperparameters["batch_size"],
hyperparameters["dropout"],
hyperparameters["stddev"]))
if accuracy > best_accuracy:
best_params = params
best_hyperparameters = hyperparameters
best_accuracy = accuracy
# Record the best performing set of hyperparameters.
@@ -86,4 +89,8 @@ if __name__ == "__main__":
batch_size: {}
dropout: {:.2}
stddev: {:.2}
""".format(trials, 100 * best_accuracy, best_params["learning_rate"], best_params["batch_size"], best_params["dropout"], best_params["stddev"]))
""".format(trials, 100 * best_accuracy,
best_hyperparameters["learning_rate"],
best_hyperparameters["batch_size"],
best_hyperparameters["dropout"],
best_hyperparameters["stddev"]))
examples/hyperopt/hyperopt.py → examples/hyperopt/objective.py
+40 -30
View File
@@ -1,3 +1,7 @@
# Most of the tensorflow code is adapted from Tensorflow's tutorial on using
# CNNs to train MNIST
# https://www.tensorflow.org/versions/r0.9/tutorials/mnist/pros/index.html#build-a-multilayer-convolutional-network.
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
@@ -51,42 +55,48 @@ def cnn_setup(x, y, keep_prob, lr, stddev):
y_conv = tf.nn.softmax(tf.matmul(h_fc1_drop, W_fc2) + b_fc2)
cross_entropy = tf.reduce_mean(-tf.reduce_sum(y * tf.log(y_conv), reduction_indices=[1]))
correct_pred = tf.equal(tf.argmax(y_conv, 1), tf.argmax(y, 1))
return tf.train.AdamOptimizer(lr).minimize(cross_entropy), tf.reduce_mean(tf.cast(correct_pred, tf.float32))
return tf.train.AdamOptimizer(lr).minimize(cross_entropy), tf.reduce_mean(tf.cast(correct_pred, tf.float32)), cross_entropy
# Define a remote function that takes a set of hyperparameters as well as the
# data, consructs and trains a network, and returns the validation accuracy.
@ray.remote
def train_cnn_and_compute_accuracy(params, steps, train_images, train_labels, validation_images, validation_labels):
def train_cnn_and_compute_accuracy(params, steps, train_images, train_labels,
validation_images, validation_labels,
weights=None):
# Extract the hyperparameters from the params dictionary.
learning_rate = params["learning_rate"]
batch_size = params["batch_size"]
keep = 1 - params["dropout"]
stddev = params["stddev"]
# Create the input placeholders for the network.
x = tf.placeholder(tf.float32, shape=[None, 784])
y = tf.placeholder(tf.float32, shape=[None, 10])
keep_prob = tf.placeholder(tf.float32)
# Create the network.
train_step, accuracy = cnn_setup(x, y, keep_prob, learning_rate, stddev)
# Do the training and evaluation.
with tf.Session() as sess:
# Initialize the network weights.
sess.run(tf.global_variables_initializer())
for i in range(1, steps + 1):
# Fetch the next batch of data.
image_batch = get_batch(train_images, i, batch_size)
label_batch = get_batch(train_labels, i, batch_size)
# Do one step of training.
sess.run(train_step, feed_dict={x: image_batch, y: label_batch, keep_prob: keep})
if i % 100 == 0:
# Estimate the training accuracy every once in a while.
train_ac = accuracy.eval(feed_dict={x: image_batch, y: label_batch, keep_prob: 1.0})
# If the training accuracy is too low, stop early in order to avoid
# wasting computation.
if train_ac < 0.25:
# Compute the validation accuracy and return.
totalacc = accuracy.eval(feed_dict={x: validation_images, y: validation_labels, keep_prob: 1.0})
return float(totalacc)
# Training is done, compute the validation accuracy and return.
totalacc = accuracy.eval(feed_dict={x: validation_images, y: validation_labels, keep_prob: 1.0})
return float(totalacc)
# Create the network and related variables.
with tf.Graph().as_default():
# Create the input placeholders for the network.
x = tf.placeholder(tf.float32, shape=[None, 784])
y = tf.placeholder(tf.float32, shape=[None, 10])
keep_prob = tf.placeholder(tf.float32)
# Create the network.
train_step, accuracy, loss = cnn_setup(x, y, keep_prob, learning_rate, stddev)
# Do the training and evaluation.
with tf.Session() as sess:
# Use the TensorFlowVariables utility. This is only necessary if we want to
# set and get the weights.
variables = ray.experimental.TensorFlowVariables(loss, sess)
# Initialize the network weights.
sess.run(tf.global_variables_initializer())
# If some network weights were passed in, set those.
if weights is not None:
variables.set_weights(weights)
# Do some steps of training.
for i in range(1, steps + 1):
# Fetch the next batch of data.
image_batch = get_batch(train_images, i, batch_size)
label_batch = get_batch(train_labels, i, batch_size)
# Do one step of training.
sess.run(train_step, feed_dict={x: image_batch, y: label_batch, keep_prob: keep})
# Training is done, so compute the validation accuracy and the current
# weights and return.
totalacc = accuracy.eval(feed_dict={x: validation_images,
y: validation_labels,
keep_prob: 1.0})
new_weights = variables.get_weights()
return float(totalacc), new_weights