Robert Nishihara
90f88af902
* Fix bug in which worker import counters were treated incorrectly. * Fix bug in which cached functions-to-run were double counted as exports. This also runs the functions-to-run on the driver only after ray.init is called. * Only define reusable variables locally after ray.init has been called. * Remove flaky reference counting tests. It's not clear that these tests make sense. * Make numbuf pip install verbose. * Export cached reusable variables before cached remote functions. * Fix bug causing the worker to hang sometimes. This happens when the worker is trying to run a task, but it hasn't imported enough imports to run the task, so it continually acquires and releases a lock while checking if it has enough imports. However, for some reason, the import thread is waiting to acquire the same lock and never does so (or takes a very long time to do so). By dropping the lock before sleeping, this makes it easier for other threads to acquire the lock. * Acquire locks using 'with' statements. * Fix possible test failure. * Try to start Redis multiple times with different random ports if the original attempt failed. * Fix test in which we redefine a remote function.
Ray
Ray is an experimental distributed extension of Python. It is under development and not ready to be used.
The goal of Ray is to make it easy to write machine learning applications that run on a cluster while providing the development and debugging experience of working on a single machine.
Before jumping into the details, here's a simple Python example for doing a Monte Carlo estimation of pi (using multiple cores or potentially multiple machines).
import ray
import numpy as np
# Start a scheduler, an object store, and some workers.
ray.init(start_ray_local=True, num_workers=10)
# Define a remote function for estimating pi.
@ray.remote
def estimate_pi(n):
x = np.random.uniform(size=n)
y = np.random.uniform(size=n)
return 4 * np.mean(x ** 2 + y ** 2 < 1)
# Launch 10 tasks, each of which estimates pi.
result_ids = []
for _ in range(10):
result_ids.append(estimate_pi.remote(100))
# Fetch the results of the tasks and print their average.
estimate = np.mean(ray.get(result_ids))
print "Pi is approximately {}.".format(estimate)
Within the for loop, each call to estimate_pi.remote(100) sends a message to
the scheduler asking it to schedule the task of running estimate_pi with the
argument 100. This call returns right away without waiting for the actual
estimation of pi to take place. Instead of returning a float, it returns an
object ID, which represents the eventual output of the computation (this is
a similar to a Future).
The call to ray.get(result_id) takes an object ID and returns the actual
estimate of pi (waiting until the computation has finished if necessary).