mirror of
https://github.com/wassname/ray.git
synced 2026-09-09 11:32:43 +08:00
Add ray.util package and move libraries from experimental (#7100)
This commit is contained in:
Eric Liang
@@ -197,19 +197,19 @@ If we instantiate an actor, we can pass the handle around to various tasks.
|
||||
print(ray.get(counter.get_counter.remote()))
|
||||
|
||||
|
||||
Actor Pool (Experimental)
|
||||
-------------------------
|
||||
Actor Pool
|
||||
----------
|
||||
|
||||
The ``ray.experimental`` module contains a utility class, ``ActorPool``.
|
||||
The ``ray.util`` module contains a utility class, ``ActorPool``.
|
||||
This class is similar to multiprocessing.Pool and lets you schedule Ray tasks over a fixed pool of actors.
|
||||
|
||||
.. code-block::
|
||||
|
||||
from ray.experimental import ActorPool
|
||||
from ray.util import ActorPool
|
||||
|
||||
a1, a2 = Actor.remote(), Actor.remote()
|
||||
pool = ActorPool([a1, a2])
|
||||
print(pool.map(lambda a, v: a.double.remote(v), [1, 2, 3, 4]))
|
||||
# [2, 4, 6, 8]
|
||||
|
||||
See the `package reference <package-ref.html#ray.experimental.ActorPool>`_ for more information.
|
||||
See the `package reference <package-ref.html#ray.util.ActorPool>`_ for more information.
|
||||
|
||||
@@ -259,7 +259,7 @@ driver:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
counter = ray.experimental.get_actor("CounterActor")
|
||||
counter = ray.util.get_actor("CounterActor")
|
||||
print(ray.get(counter.get_counter.remote()))
|
||||
|
||||
Note that just creating a named actor is allowed, this actor will be cleaned
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
Async API (Experimental)
|
||||
========================
|
||||
AsyncIO / Concurrency for Actors
|
||||
================================
|
||||
|
||||
Since Python 3.5, it is possible to write concurrent code using the
|
||||
``async/await`` `syntax <https://docs.python.org/3/library/asyncio.html>`__.
|
||||
|
||||
@@ -12,3 +12,4 @@ How to setup your cluster and use Ray most effectively.
|
||||
deploy-on-yarn.rst
|
||||
deploy-on-kubernetes.rst
|
||||
deploying-on-slurm.rst
|
||||
projects.rst
|
||||
|
||||
@@ -237,12 +237,8 @@ scripts. Some of the examples include:
|
||||
* ``bazel test --build_tests_only //:all``
|
||||
|
||||
* Ray serving test commands:
|
||||
* ``python -m pytest python/ray/experimental/serve/tests``
|
||||
* ``python python/ray/experimental/serve/examples/echo_full.py``
|
||||
|
||||
* Ray test commands:
|
||||
* ``python/ray/experimental/test/async_test.py``
|
||||
* ``python/ray/tests/py3_test.py``
|
||||
* ``python -m pytest python/ray/serve/tests``
|
||||
* ``python python/ray/serve/examples/echo_full.py``
|
||||
|
||||
If a Travis-CI build exception doesn't appear to be related to your change,
|
||||
please visit `this link <https://ray-travis-tracker.herokuapp.com/>`_ to
|
||||
|
||||
+9
-14
@@ -16,7 +16,7 @@ Ray is packaged with the following libraries for accelerating machine learning w
|
||||
|
||||
- `Tune`_: Scalable Hyperparameter Tuning
|
||||
- `RLlib`_: Scalable Reinforcement Learning
|
||||
- `RaySGD`_: Distributed Training
|
||||
- `RaySGD`_: Distributed Training Wrappers
|
||||
|
||||
|
||||
Star us on `on GitHub`_. You can also get started by visiting our `Tutorials <https://github.com/ray-project/tutorial>`_. For the latest wheels (nightlies), see the `installation page <installation.html>`__.
|
||||
@@ -239,8 +239,7 @@ Getting Involved
|
||||
using-ray.rst
|
||||
configure.rst
|
||||
cluster-index.rst
|
||||
Tutorials <https://github.com/ray-project/tutorial>
|
||||
Examples <auto_examples/overview.rst>
|
||||
Tutorial and Examples <auto_examples/overview.rst>
|
||||
package-ref.rst
|
||||
|
||||
.. toctree::
|
||||
@@ -254,9 +253,9 @@ Getting Involved
|
||||
tune-distributed.rst
|
||||
tune-schedulers.rst
|
||||
tune-searchalg.rst
|
||||
tune-package-ref.rst
|
||||
tune-design.rst
|
||||
tune-examples.rst
|
||||
tune-package-ref.rst
|
||||
tune-contrib.rst
|
||||
|
||||
.. toctree::
|
||||
@@ -272,31 +271,27 @@ Getting Involved
|
||||
rllib-offline.rst
|
||||
rllib-concepts.rst
|
||||
rllib-examples.rst
|
||||
rllib-dev.rst
|
||||
rllib-package-ref.rst
|
||||
rllib-dev.rst
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: -1
|
||||
:caption: RaySGD
|
||||
:caption: Ray SGD
|
||||
|
||||
raysgd/raysgd.rst
|
||||
raysgd/raysgd_pytorch.rst
|
||||
raysgd/raysgd_tensorflow.rst
|
||||
raysgd/raysgd_ref.rst
|
||||
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: -1
|
||||
:caption: Experimental
|
||||
:caption: Other Libraries
|
||||
|
||||
pandas_on_ray.rst
|
||||
projects.rst
|
||||
signals.rst
|
||||
async_api.rst
|
||||
serve.rst
|
||||
iter.rst
|
||||
multiprocessing.rst
|
||||
joblib.rst
|
||||
iter.rst
|
||||
pandas_on_ray.rst
|
||||
serve.rst
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: -1
|
||||
|
||||
+21
-15
@@ -1,13 +1,19 @@
|
||||
Parallel Iterator API (Experimental)
|
||||
====================================
|
||||
Distributed Iterators
|
||||
=====================
|
||||
|
||||
``ray.experimental.iter`` provides a parallel iterator API for simple data ingest
|
||||
and processing. It can be thought of as syntactic sugar around Ray actors and ``ray.wait`` loops.
|
||||
.. _`issue on GitHub`: https://github.com/ray-project/ray/issues
|
||||
|
||||
``ray.util.iter`` provides a parallel iterator API for simple data ingest and processing. It can be thought of as syntactic sugar around Ray actors and ``ray.wait`` loops.
|
||||
|
||||
Parallel iterators are lazy and can operate over infinite sequences of items. Iterator
|
||||
transformations are only executed when the user calls ``next()`` to fetch the next output
|
||||
item from the iterator.
|
||||
|
||||
.. note::
|
||||
|
||||
This API is new and may be revised in future Ray releases. If you encounter
|
||||
any bugs, please file an `issue on GitHub`_.
|
||||
|
||||
Concepts
|
||||
--------
|
||||
|
||||
@@ -18,20 +24,20 @@ create a worker actor that produces the data for each shard of the iterator:
|
||||
.. code-block:: python
|
||||
|
||||
# Create an iterator with 2 worker actors over the list [1, 2, 3, 4].
|
||||
>>> it = ray.experimental.iter.from_items([1, 2, 3, 4], num_shards=2)
|
||||
>>> it = ray.util.iter.from_items([1, 2, 3, 4], num_shards=2)
|
||||
ParallelIterator[from_items[int, 4, shards=2]]
|
||||
|
||||
# Create an iterator with 32 worker actors over range(1000000).
|
||||
>>> it = ray.experimental.iter.from_range(1000000, num_shards=32)
|
||||
>>> it = ray.util.iter.from_range(1000000, num_shards=32)
|
||||
ParallelIterator[from_range[1000000, shards=32]]
|
||||
|
||||
# Create an iterator over two range(10) generators.
|
||||
>>> it = ray.experimental.iter.from_iterators([range(10), range(10)])
|
||||
>>> it = ray.util.iter.from_iterators([range(10), range(10)])
|
||||
ParallelIterator[from_iterators[shards=2]]
|
||||
|
||||
# Create an iterator from existing worker actors. These actors must
|
||||
# implement the ParallelIteratorWorker interface.
|
||||
>>> it = ray.experimental.iter.from_actors([a1, a2, a3, a4])
|
||||
>>> it = ray.util.iter.from_actors([a1, a2, a3, a4])
|
||||
ParallelIterator[from_actors[shards=4]]
|
||||
|
||||
Simple transformations can be chained on the iterator, such as mapping,
|
||||
@@ -58,7 +64,7 @@ correspond to ``ray.get`` and ``ray.wait`` loops over the actors respectively:
|
||||
.. code-block:: python
|
||||
|
||||
# Gather items synchronously (deterministic round robin across shards):
|
||||
>>> it = ray.experimental.iter.from_range(1000000, 1)
|
||||
>>> it = ray.util.iter.from_range(1000000, 1)
|
||||
>>> it = it.gather_sync()
|
||||
LocalIterator[ParallelIterator[from_range[1000000, shards=1]].gather_sync()]
|
||||
|
||||
@@ -72,7 +78,7 @@ correspond to ``ray.get`` and ``ray.wait`` loops over the actors respectively:
|
||||
[0, 2, 4, 6, 8]
|
||||
|
||||
# Async gather can be used for better performance, but it is non-deterministic.
|
||||
>>> it = ray.experimental.iter.from_range(1000, 4).gather_async()
|
||||
>>> it = ray.util.iter.from_range(1000, 4).gather_async()
|
||||
>>> it.take(5)
|
||||
[0, 250, 500, 750, 1]
|
||||
|
||||
@@ -83,7 +89,7 @@ each shard should only be read by one process at a time:
|
||||
.. code-block:: python
|
||||
|
||||
# Get local iterators representing the shards of this ParallelIterator:
|
||||
>>> it = ray.experimental.iter.from_range(10000, 3)
|
||||
>>> it = ray.util.iter.from_range(10000, 3)
|
||||
>>> [s0, s1, s2] = it.shards()
|
||||
[LocalIterator[from_range[10000, shards=3].shard[0]],
|
||||
LocalIterator[from_range[10000, shards=3].shard[1]],
|
||||
@@ -122,7 +128,7 @@ This means that you can pass a stateful callable to ``.foreach()``:
|
||||
self.total += x
|
||||
return (self.total, x)
|
||||
|
||||
it = ray.experimental.iter.from_range(5, 1)
|
||||
it = ray.util.iter.from_range(5, 1)
|
||||
for x in it.for_each(CumulativeSum()).gather_sync():
|
||||
print(x)
|
||||
|
||||
@@ -150,7 +156,7 @@ streaming grep:
|
||||
|
||||
file_list = glob.glob("/var/log/syslog*.gz")
|
||||
it = (
|
||||
ray.experimental.iter.from_items(file_list, num_shards=4)
|
||||
ray.util.iter.from_items(file_list, num_shards=4)
|
||||
.for_each(lambda f: gzip.open(f).readlines())
|
||||
.flatten()
|
||||
.for_each(lambda line: line.decode("utf-8"))
|
||||
@@ -184,7 +190,7 @@ distributed training:
|
||||
print("train on", batch) # perform model update with batch
|
||||
|
||||
it = (
|
||||
ray.experimental.iter.from_range(1000000, num_shards=4, repeat=True)
|
||||
ray.util.iter.from_range(1000000, num_shards=4, repeat=True)
|
||||
.batch(1024)
|
||||
.for_each(np.array)
|
||||
)
|
||||
@@ -195,7 +201,7 @@ distributed training:
|
||||
API Reference
|
||||
-------------
|
||||
|
||||
.. automodule:: ray.experimental.iter
|
||||
.. automodule:: ray.util.iter
|
||||
:members:
|
||||
:show-inheritance:
|
||||
:special-members:
|
||||
|
||||
+9
-11
@@ -1,12 +1,5 @@
|
||||
sklearn Ray Backend API (Experimental)
|
||||
=======================================
|
||||
|
||||
.. warning::
|
||||
|
||||
Support for running scikit-learn on Ray is an experimental feature,
|
||||
so it may be changed at any time without warning. If you encounter any
|
||||
bugs/shortcomings/incompatibilities, please file an `issue on GitHub`_.
|
||||
Contributions are always welcome!
|
||||
Distributed Scikit-learn / Joblib
|
||||
=================================
|
||||
|
||||
.. _`issue on GitHub`: https://github.com/ray-project/ray/issues
|
||||
|
||||
@@ -15,6 +8,11 @@ implementing a Ray backend for `joblib`_ using `Ray Actors <actors.html>`__
|
||||
instead of local processes. This makes it easy to scale existing applications
|
||||
that use scikit-learn from a single node to a cluster.
|
||||
|
||||
.. note::
|
||||
|
||||
This API is new and may be revised in future Ray releases. If you encounter
|
||||
any bugs, please file an `issue on GitHub`_.
|
||||
|
||||
.. _`joblib`: https://joblib.readthedocs.io
|
||||
.. _`scikit-learn`: https://scikit-learn.org
|
||||
|
||||
@@ -22,7 +20,7 @@ Quickstart
|
||||
----------
|
||||
|
||||
To get started, first `install Ray <installation.html>`__, then use
|
||||
``from ray.experimental.joblib import register_ray`` and run ``register_ray()``.
|
||||
``from ray.util.joblib import register_ray`` and run ``register_ray()``.
|
||||
This will register Ray as a joblib backend for scikit-learn to use.
|
||||
Then run your original scikit-learn code inside
|
||||
``with joblib.parallel_backend('ray')``. This will start a local Ray cluster.
|
||||
@@ -46,7 +44,7 @@ a multi-node Ray cluster instead.
|
||||
search = RandomizedSearchCV(model, param_space, cv=5, n_iter=300, verbose=10)
|
||||
|
||||
import joblib
|
||||
from ray.experimental.joblib import register_ray
|
||||
from ray.util.joblib import register_ray
|
||||
register_ray()
|
||||
with joblib.parallel_backend('ray'):
|
||||
search.fit(digits.data, digits.target)
|
||||
|
||||
@@ -84,10 +84,11 @@ Heap memory quota
|
||||
|
||||
When Ray starts, it queries the available memory on a node / container not reserved for Redis and the object store or being used by other applications. This is considered "available memory" that actors and tasks can request memory out of. You can also set ``memory=<bytes>`` on Ray init to tell Ray explicitly how much memory is available.
|
||||
|
||||
.. note::
|
||||
.. important::
|
||||
|
||||
Setting available memory for the node does not impose any limits on memory usage
|
||||
of tasks. To set per-task limits, see the following sections.
|
||||
Setting available memory for the node does NOT impose any limits on memory usage
|
||||
unless you specify memory resource requirements in decorators. By default, tasks
|
||||
and actors request no memory (and hence have no limit).
|
||||
|
||||
To tell the Ray scheduler a task or actor requires a certain amount of available memory to run, set the ``memory`` argument. The Ray scheduler will then reserve the specified amount of available memory during scheduling, similar to how it handles CPU and GPU resources:
|
||||
|
||||
|
||||
@@ -1,12 +1,5 @@
|
||||
multiprocessing.Pool API
|
||||
========================
|
||||
|
||||
.. warning::
|
||||
|
||||
Support for the multiprocessing.Pool API on Ray is an experimental feature,
|
||||
so it may be changed at any time without warning. If you encounter any
|
||||
bugs/shortcomings/incompatibilities, please file an `issue on GitHub`_.
|
||||
Contributions are always welcome!
|
||||
Distributed multiprocessing.Pool
|
||||
================================
|
||||
|
||||
.. _`issue on GitHub`: https://github.com/ray-project/ray/issues
|
||||
|
||||
@@ -15,6 +8,11 @@ using `Ray Actors <actors.html>`__ instead of local processes. This makes it eas
|
||||
to scale existing applications that use ``multiprocessing.Pool`` from a single node
|
||||
to a cluster.
|
||||
|
||||
.. note::
|
||||
|
||||
This API is new and may be revised in future Ray releases. If you encounter
|
||||
any bugs, please file an `issue on GitHub`_.
|
||||
|
||||
.. _`multiprocessing.Pool API`: https://docs.python.org/3/library/multiprocessing.html#module-multiprocessing.pool
|
||||
|
||||
Quickstart
|
||||
|
||||
@@ -1,9 +1,7 @@
|
||||
RaySGD: Distributed Deep Learning
|
||||
=================================
|
||||
RaySGD: Distributed Training Wrappers
|
||||
=====================================
|
||||
|
||||
.. image:: raysgdlogo.png
|
||||
:scale: 20%
|
||||
:align: center
|
||||
.. _`issue on GitHub`: https://github.com/ray-project/ray/issues
|
||||
|
||||
RaySGD is a lightweight library for distributed deep learning, providing thin wrappers around PyTorch and TensorFlow native modules for data parallel training.
|
||||
|
||||
@@ -13,7 +11,10 @@ The main features are:
|
||||
- **Composability**: RaySGD is built on top of the Ray Actor API, enabling seamless integration with existing Ray applications such as RLlib, Tune, and Ray.Serve.
|
||||
- **Scale up and down**: Start on single CPU. Scale up to multi-node, multi-CPU, or multi-GPU clusters by changing 2 lines of code.
|
||||
|
||||
.. tip:: We need your feedback! RaySGD is currently early in its development, and we're hoping to get feedback from people using or considering it. We'd love `to get in touch <https://forms.gle/26EMwdahdgm7Lscy9>`_!
|
||||
.. note::
|
||||
|
||||
This API is new and may be revised in future Ray releases. If you encounter
|
||||
any bugs, please file an `issue on GitHub`_.
|
||||
|
||||
|
||||
Getting Started
|
||||
@@ -28,8 +29,8 @@ You can start a ``PyTorchTrainer`` with the following:
|
||||
import torch.nn as nn
|
||||
from torch import distributed
|
||||
|
||||
from ray.experimental.sgd import PyTorchTrainer
|
||||
from ray.experimental.sgd.examples.train_example import LinearDataset
|
||||
from ray.util.sgd import PyTorchTrainer
|
||||
from ray.util.sgd.examples.train_example import LinearDataset
|
||||
|
||||
|
||||
def model_creator(config):
|
||||
@@ -61,3 +62,5 @@ You can start a ``PyTorchTrainer`` with the following:
|
||||
print(stats)
|
||||
trainer1.shutdown()
|
||||
print("success!")
|
||||
|
||||
.. tip:: Get in touch with us if you're using or considering using `RaySGD <https://forms.gle/26EMwdahdgm7Lscy9>`_!
|
||||
|
||||
@@ -1,8 +1,5 @@
|
||||
RaySGD Pytorch
|
||||
==============
|
||||
|
||||
.. image:: raysgd-pytorch.svg
|
||||
:align: center
|
||||
Distributed PyTorch
|
||||
===================
|
||||
|
||||
The RaySGD ``PyTorchTrainer`` simplifies distributed model training for PyTorch. The ``PyTorchTrainer`` is a wrapper around ``torch.distributed.launch`` with a Python API to easily incorporate distributed training into a larger Python application, as opposed to needing to wrap your training code in bash scripts.
|
||||
|
||||
@@ -16,7 +13,7 @@ For end to end examples leveraging RaySGD PyTorchTrainer, jump to :ref:`raysgd-p
|
||||
Setting up training
|
||||
-------------------
|
||||
|
||||
.. tip:: We need your feedback! RaySGD is currently early in its development, and we're hoping to get feedback from people using or considering it. We'd love `to get in touch <https://forms.gle/26EMwdahdgm7Lscy9>`_!
|
||||
.. tip:: Get in touch with us if you're using or considering using `RaySGD <https://forms.gle/26EMwdahdgm7Lscy9>`_!
|
||||
|
||||
The ``PyTorchTrainer`` can be constructed with functions that wrap components of the training script. Specifically, it requires constructors for the Model, Data, Optimizer, Loss, and ``lr_scheduler`` to create replicated copies across different devices and machines.
|
||||
|
||||
@@ -154,8 +151,6 @@ After training, you may want to reappropriate the Ray cluster. To release Ray re
|
||||
Initialization Functions
|
||||
------------------------
|
||||
|
||||
.. warning:: This is still an experimental API and is subject to change without warning.
|
||||
|
||||
You may want to run some initializers on each worker when they are started. This may be something like setting an environment variable or downloading some data. You can do this via the ``initialization_hook`` parameter:
|
||||
|
||||
.. code-block:: python
|
||||
@@ -305,11 +300,9 @@ Users can set ``checkpoint="auto"`` to always checkpoint the current model befor
|
||||
Advanced: Hyperparameter Tuning
|
||||
-------------------------------
|
||||
|
||||
.. warning:: This is still an experimental API and is subject to change without warning.
|
||||
|
||||
``PyTorchTrainer`` naturally integrates with Tune via the ``PyTorchTrainable`` interface. The same arguments to ``PyTorchTrainer`` should be passed into the ``tune.run(config=...)`` as shown below.
|
||||
|
||||
.. literalinclude:: ../../../python/ray/experimental/sgd/pytorch/examples/tune_example.py
|
||||
.. literalinclude:: ../../../python/ray/util/sgd/pytorch/examples/tune_example.py
|
||||
:language: python
|
||||
:start-after: __torch_tune_example__
|
||||
|
||||
@@ -321,7 +314,7 @@ In certain scenarios such as training GANs, you may want to use multiple models
|
||||
|
||||
If multiple models, optimizers, or schedulers are returned, you will need to provide a custom training function (and custom validation function if you plan to call ``validate``).
|
||||
|
||||
You can see the `DCGAN script <https://github.com/ray-project/ray/blob/master/python/ray/experimental/sgd/pytorch/examples/dcgan.py>`_ for an end-to-end example.
|
||||
You can see the `DCGAN script <https://github.com/ray-project/ray/blob/master/python/ray/util/sgd/pytorch/examples/dcgan.py>`_ for an end-to-end example.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
@@ -362,8 +355,6 @@ You can see the `DCGAN script <https://github.com/ray-project/ray/blob/master/py
|
||||
Custom Training and Validation Functions
|
||||
----------------------------------------
|
||||
|
||||
.. warning:: This is still an experimental API and is subject to change in the near future.
|
||||
|
||||
``PyTorchTrainer`` allows you to run a custom training and validation step in parallel on each worker, providing a flexibility similar to using PyTorch natively. This is done via the ``train_function`` and ``validation_function`` parameters.
|
||||
|
||||
Note that this is needed if the model creator returns multiple models, optimizers, or schedulers.
|
||||
@@ -477,13 +468,13 @@ PyTorchTrainer Examples
|
||||
Here are some examples of using RaySGD for training PyTorch models. If you'd like
|
||||
to contribute an example, feel free to create a `pull request here <https://github.com/ray-project/ray/>`_.
|
||||
|
||||
- `PyTorch training example <https://github.com/ray-project/ray/blob/master/python/ray/experimental/sgd/pytorch/examples/train_example.py>`__:
|
||||
- `PyTorch training example <https://github.com/ray-project/ray/blob/master/python/ray/util/sgd/pytorch/examples/train_example.py>`__:
|
||||
Simple example of using Ray's PyTorchTrainer.
|
||||
|
||||
- `CIFAR10 example <https://github.com/ray-project/ray/blob/master/python/ray/experimental/sgd/pytorch/examples/cifar_pytorch_example.py>`__:
|
||||
- `CIFAR10 example <https://github.com/ray-project/ray/blob/master/python/ray/util/sgd/pytorch/examples/cifar_pytorch_example.py>`__:
|
||||
Training a ResNet18 model on CIFAR10. It uses a custom training
|
||||
function, a custom validation function, and custom initialization code for each worker.
|
||||
|
||||
- `DCGAN example <https://github.com/ray-project/ray/blob/master/python/ray/experimental/sgd/pytorch/examples/dcgan.py>`__:
|
||||
- `DCGAN example <https://github.com/ray-project/ray/blob/master/python/ray/util/sgd/pytorch/examples/dcgan.py>`__:
|
||||
Training a Deep Convolutional GAN on MNIST. It constructs
|
||||
two models and two optimizers and uses a custom training and validation function.
|
||||
|
||||
@@ -6,7 +6,7 @@ Package Reference
|
||||
PyTorchTrainer
|
||||
--------------
|
||||
|
||||
.. autoclass:: ray.experimental.sgd.pytorch.PyTorchTrainer
|
||||
.. autoclass:: ray.util.sgd.pytorch.PyTorchTrainer
|
||||
:members:
|
||||
|
||||
.. automethod:: __init__
|
||||
@@ -15,13 +15,13 @@ PyTorchTrainer
|
||||
PyTorchTrainable
|
||||
----------------
|
||||
|
||||
.. autoclass:: ray.experimental.sgd.pytorch.PyTorchTrainable
|
||||
.. autoclass:: ray.util.sgd.pytorch.PyTorchTrainable
|
||||
:members:
|
||||
|
||||
TFTrainer
|
||||
---------
|
||||
|
||||
.. autoclass:: ray.experimental.sgd.tf.TFTrainer
|
||||
.. autoclass:: ray.util.sgd.tf.TFTrainer
|
||||
:members:
|
||||
|
||||
.. automethod:: __init__
|
||||
|
||||
@@ -1,9 +1,12 @@
|
||||
RaySGD TensorFlow
|
||||
=================
|
||||
Distributed TensorFlow
|
||||
======================
|
||||
|
||||
RaySGD's ``TFTrainer`` simplifies distributed model training for Tensorflow. The ``TFTrainer`` is a wrapper around ``MultiWorkerMirroredStrategy`` with a Python API to easily incorporate distributed training into a larger Python application, as opposed to write custom logic of setting environments and starting separate processes.
|
||||
|
||||
.. important:: This API has only been tested with TensorFlow2.0rc and is still highly experimental. Please file bug reports if you run into any - thanks!
|
||||
Under the hood, ``TFTrainer`` will create *replicas* of your model (controlled by ``num_replicas``), each of which is managed by a Ray actor.
|
||||
|
||||
.. image:: raysgd-actors.svg
|
||||
:align: center
|
||||
|
||||
.. tip:: We need your feedback! RaySGD is currently early in its development, and we're hoping to get feedback from people using or considering it. We'd love `to get in touch <https://forms.gle/26EMwdahdgm7Lscy9>`_!
|
||||
|
||||
@@ -69,6 +72,6 @@ TFTrainer Example
|
||||
|
||||
Below is an example of using Ray's TFTrainer. Under the hood, ``TFTrainer`` will create *replicas* of your model (controlled by ``num_replicas``) which are each managed by a worker.
|
||||
|
||||
.. literalinclude:: ../../../python/ray/experimental/sgd/tf/examples/tensorflow_train_example.py
|
||||
.. literalinclude:: ../../../python/ray/util/sgd/tf/examples/tensorflow_train_example.py
|
||||
:language: python
|
||||
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
RLlib Development
|
||||
=================
|
||||
Contributing to RLlib
|
||||
=====================
|
||||
|
||||
Development Install
|
||||
-------------------
|
||||
|
||||
@@ -442,7 +442,7 @@ Sometimes, it is necessary to coordinate between pieces of code that live in dif
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from ray.experimental import named_actors
|
||||
from ray.util import named_actors
|
||||
|
||||
@ray.remote
|
||||
class Counter:
|
||||
|
||||
+10
-7
@@ -1,20 +1,23 @@
|
||||
Ray Serve (Experimental)
|
||||
========================
|
||||
Ray Serve
|
||||
=========
|
||||
|
||||
.. _`issue on GitHub`: https://github.com/ray-project/ray/issues
|
||||
|
||||
Ray Serve is a serving library that exposes python function/classes to HTTP.
|
||||
It has built-in support for flexible traffic policy. This means you can easy
|
||||
split incoming traffic to multiple implementations.
|
||||
|
||||
With Ray Serve, you can deploy your services at any scale.
|
||||
|
||||
.. warning::
|
||||
Ray Serve is Python 3 only.
|
||||
|
||||
Ray Serve is under development and its API may be revised in future Ray releases. If you encounter any bugs, please file an `issue on GitHub`_.
|
||||
|
||||
With Ray Serve, you can deploy your services at any scale.
|
||||
|
||||
Quickstart
|
||||
----------
|
||||
.. literalinclude:: ../../python/ray/experimental/serve/examples/echo_full.py
|
||||
.. literalinclude:: ../../python/ray/serve/examples/echo_full.py
|
||||
|
||||
API
|
||||
---
|
||||
.. automodule:: ray.experimental.serve
|
||||
.. automodule:: ray.serve
|
||||
:members:
|
||||
|
||||
@@ -1,168 +0,0 @@
|
||||
Signal API (Experimental)
|
||||
=========================
|
||||
|
||||
This experimental API allows tasks and actors to generate signals which can
|
||||
be received by other tasks and actors. In addition, task failures and actor
|
||||
method failures generate error signals. The error signals enable applications
|
||||
to detect failures and potentially recover from failures.
|
||||
|
||||
.. autofunction:: ray.experimental.signal.send
|
||||
|
||||
Here is a simple example of a remote function that sends a user-defined signal.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
import ray.experimental.signal as signal
|
||||
|
||||
# Define an application level signal.
|
||||
class UserSignal(signal.Signal):
|
||||
def __init__(self, value):
|
||||
self.value = value
|
||||
|
||||
def get_value(self):
|
||||
return self.value
|
||||
|
||||
# Define a remote function that sends a user-defined signal.
|
||||
@ray.remote
|
||||
def send_signal(value):
|
||||
signal.send(UserSignal(value))
|
||||
|
||||
.. autofunction:: ray.experimental.signal.receive
|
||||
|
||||
Here is a simple example of how to receive signals from an actor or task identified
|
||||
by ``a``. Note that an actor is identified by its handle, and a task by one of its
|
||||
object ID return values.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
import ray.experimental.signal as signal
|
||||
|
||||
# This returns a possibly empty list of all signals that have been sent by 'a'
|
||||
# since the last invocation of signal.receive from within this process. If 'a'
|
||||
# did not send any signals, then this will wait for up to 10 seconds to receive
|
||||
# a signal from 'a'.
|
||||
signal_list = signal.receive([a], timeout=10)
|
||||
|
||||
.. autofunction:: ray.experimental.signal.reset
|
||||
|
||||
|
||||
Example: sending a user signal
|
||||
------------------------------
|
||||
|
||||
The code below show a simple example in which a task, called ``send_signal()``
|
||||
sends a user signal and the driver gets it by invoking ``signal.receive()``.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
import ray.experimental.signal as signal
|
||||
|
||||
# Define a user signal.
|
||||
class UserSignal(signal.Signal):
|
||||
def __init__(self, value):
|
||||
self.value = value
|
||||
|
||||
def get_value(self):
|
||||
return self.value
|
||||
|
||||
@ray.remote
|
||||
def send_signal(value):
|
||||
signal.send(UserSignal(value))
|
||||
return
|
||||
|
||||
signal_value = 'simple signal'
|
||||
object_id = send_signal.remote(signal_value)
|
||||
# Wait up to 10sec to receive a signal from the task. Note the task is
|
||||
# identified by the object_id it returns.
|
||||
result_list = signal.receive([object_id], timeout=10)
|
||||
# Print signal values. This should print "simple_signal".
|
||||
# Note that result_list[0] is the signal we expect from the task.
|
||||
# The signal is a tuple where the first element is the first object ID
|
||||
# returned by the task and the second element is the signal object.
|
||||
print(result_list[0][1].get_value())
|
||||
|
||||
Example: Getting an error signals
|
||||
---------------------------------
|
||||
|
||||
This is a simple example in which a driver gets an error signal caused
|
||||
by the failure of ``task()``.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
@ray.remote
|
||||
def task():
|
||||
raise Exception('exception message')
|
||||
|
||||
object_id = task.remote()
|
||||
try:
|
||||
ray.get(object_id)
|
||||
except Exception as e:
|
||||
pass
|
||||
finally:
|
||||
result_list = signal.receive([object_id], timeout=10)
|
||||
# Expected signal is 'ErrorSignal'.
|
||||
assert type(result_list[0][1]) == signal.ErrorSignal
|
||||
# Print the error.
|
||||
print(result_list[0][1].get_error())
|
||||
|
||||
|
||||
Example: Sending signals between multiple actors
|
||||
------------------------------------------------
|
||||
|
||||
This is a more involved example in which two actors ``a1`` and ``a2`` each
|
||||
generate five signals, and another actor ``b`` waits to receive all signals
|
||||
generated by ``a1`` and ``a2``, respectively. Note that ``b`` recursively calls
|
||||
its own method ``get_signals()`` until it gets all signals it expects.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
@ray.remote
|
||||
class ActorSendSignals(object):
|
||||
def send_signals(self, value, count):
|
||||
for i in range(count):
|
||||
signal.send(UserSignal(value + str(i)))
|
||||
|
||||
@ray.remote
|
||||
class ActorGetAllSignals(object):
|
||||
def __init__(self, num_expected_signals, *source_ids):
|
||||
self.received_signals = []
|
||||
self.num_expected_signals = num_expected_signals
|
||||
self.source_ids = source_ids
|
||||
|
||||
def register_handle(self, handle):
|
||||
self.this_actor = handle
|
||||
|
||||
def get_signals(self):
|
||||
new_signals = signal.receive(self.source_ids, timeout=10)
|
||||
self.received_signals.extend(new_signals)
|
||||
if len(self.received_signals) < self.num_expected_signals:
|
||||
self.this_actor.get_signals.remote()
|
||||
|
||||
def get_count(self):
|
||||
return len(self.received_signals)
|
||||
|
||||
# Create two actors to send signals.
|
||||
a1 = ActorSendSignals.remote()
|
||||
a2 = ActorSendSignals.remote()
|
||||
signal_value = 'simple signal'
|
||||
count = 5
|
||||
# Each actor sends five signals.
|
||||
a1.send_signals.remote(signal_value, count)
|
||||
a2.send_signals.remote(signal_value, count)
|
||||
|
||||
# Create an actor that waits for all five signals sent by each actor.
|
||||
b = ActorGetAllSignals.remote(2 * count, *[a1, a2])
|
||||
# Provide actor to its own handle, so it can recursively call itself
|
||||
# to get all signals from a1, and a2, respectively. This enables the actor
|
||||
# execute other methods if needed.
|
||||
ray.get(b.register_handle.remote(b))
|
||||
b.get_signals.remote()
|
||||
# Print total number of signals. This should be 2*count = 10.
|
||||
print(ray.get(b.get_count.remote()))
|
||||
|
||||
Note
|
||||
----
|
||||
|
||||
A failed actor (e.g., an actor that crashed) generates an error message only
|
||||
when another actor or task invokes one of its methods.
|
||||
|
||||
Please `let us know <https://github.com/ray-project/ray/issues>`__ any issues you encounter.
|
||||
+3
-3
@@ -1,11 +1,11 @@
|
||||
Tune: A Scalable Hyperparameter Tuning Library
|
||||
==============================================
|
||||
Tune: Scalable Hyperparameter Tuning
|
||||
====================================
|
||||
|
||||
.. image:: images/tune.png
|
||||
:scale: 30%
|
||||
:align: center
|
||||
|
||||
Tune is a Python library for hyperparameter tuning at any scale. Core features:
|
||||
Tune is a Python library for experiment execution and hyperparameter tuning at any scale. Core features:
|
||||
|
||||
* Launch a multi-node `distributed hyperparameter sweep <tune-distributed.html>`_ in less than 10 lines of code.
|
||||
* Supports any machine learning framework, including PyTorch, XGBoost, MXNet, and Keras. See `examples here <tune-examples.html>`_.
|
||||
|
||||
@@ -12,6 +12,7 @@ Finally, we've also included some content on using core Ray APIs with `Tensorflo
|
||||
|
||||
starting-ray.rst
|
||||
actors.rst
|
||||
async_api.rst
|
||||
using-ray-with-gpus.rst
|
||||
serialization.rst
|
||||
memory-management.rst
|
||||
|
||||
Reference in New Issue
Block a user