wassname/ray
1
0
Fork 0
mirror of https://github.com/wassname/ray.git synced 2026-08-13 12:30:18 +08:00
Code Issues Packages Projects Releases Wiki Activity

[docs] Make Contributions/Building pages more prominent (#9054)

Browse Source 
Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>
This commit is contained in:
Richard Liaw
2020-06-25 12:38:12 -07:00
committed by GitHub
co-authored by Edward Oakes
parent 07655036d2
commit 75e6775b36
6 changed files with 417 additions and 452 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/source/debugging.rst
+100
View File
@@ -0,0 +1,100 @@
Debugging
=========
Starting processes in a debugger
--------------------------------
When processes are crashing, it is often useful to start them in a debugger.
Ray currently allows processes to be started in the following:
- valgrind
- the valgrind profiler
- the perftools profiler
- gdb
- tmux
To use any of these tools, please make sure that you have them installed on
your machine first (``gdb`` and ``valgrind`` on MacOS are known to have issues).
Then, you can launch a subset of ray processes by adding the environment
variable ``RAY_{PROCESS_NAME}_{DEBUGGER}=1``. For instance, if you wanted to
start the raylet in ``valgrind``, then you simply need to set the environment
variable ``RAY_RAYLET_VALGRIND=1``.
To start a process inside of ``gdb``, the process must also be started inside of
``tmux``. So if you want to start the raylet in ``gdb``, you would start your
Python script with the following:
.. code-block:: bash
RAY_RAYLET_GDB=1 RAY_RAYLET_TMUX=1 python
You can then list the ``tmux`` sessions with ``tmux ls`` and attach to the
appropriate one.
You can also get a core dump of the ``raylet`` process, which is especially
useful when filing `issues`_. The process to obtain a core dump is OS-specific,
but usually involves running ``ulimit -c unlimited`` before starting Ray to
allow core dump files to be written.
Inspecting Redis shards
-----------------------
To inspect Redis, you can use the global state API. The easiest way to do this
is to start or connect to a Ray cluster with ``ray.init()``, then query the API
like so:
.. code-block:: python
ray.init()
ray.nodes()
# Returns current information about the nodes in the cluster, such as:
# [{'ClientID': '2a9d2b34ad24a37ed54e4fcd32bf19f915742f5b',
# 'IsInsertion': True,
# 'NodeManagerAddress': '1.2.3.4',
# 'NodeManagerPort': 43280,
# 'ObjectManagerPort': 38062,
# 'ObjectStoreSocketName': '/tmp/ray/session_2019-01-21_16-28-05_4216/sockets/plasma_store',
# 'RayletSocketName': '/tmp/ray/session_2019-01-21_16-28-05_4216/sockets/raylet',
# 'Resources': {'CPU': 8.0, 'GPU': 1.0}}]
To inspect the primary Redis shard manually, you can also query with commands
like the following.
.. code-block:: python
r_primary = ray.worker.global_worker.redis_client
r_primary.keys("*")
To inspect other Redis shards, you will need to create a new Redis client.
For example (assuming the relevant IP address is ``127.0.0.1`` and the
relevant port is ``1234``), you can do this as follows.
.. code-block:: python
import redis
r = redis.StrictRedis(host='127.0.0.1', port=1234)
You can find a list of the relevant IP addresses and ports by running
.. code-block:: python
r_primary.lrange('RedisShards', 0, -1)
.. _backend-logging:
Backend logging
---------------
The ``raylet`` process logs detailed information about events like task
execution and object transfers between nodes. To set the logging level at
runtime, you can set the ``RAY_BACKEND_LOG_LEVEL`` environment variable before
starting Ray. For example, you can do:
.. code-block:: shell
export RAY_BACKEND_LOG_LEVEL=debug
ray start
This will print any ``RAY_LOG(DEBUG)`` lines in the source code to the
``raylet.err`` file, which you can find in the `Temporary Files`_.
.. _`issues`: https://github.com/ray-project/ray/issues
.. _`Temporary Files`: http://docs.ray.io/en/latest/tempfile.html
doc/source/development.rst
+101 -289
View File
@@ -1,45 +1,105 @@
Development Tips
================
.. _building-ray:
Building Ray from Source
=========================
For majority of Ray users, installing Ray via the latest wheels or pip package is usually enough. However, you may want to build the latest master branch.
.. tip:: If you are only editing Python files, follow instructions for :ref:`python-develop` to avoid long build times.
.. contents::
:local:
.. _python-develop:
Building Ray (Python Only)
--------------------------
.. note:: Unless otherwise stated, directory and file paths are relative to the project root directory.
RLlib, Tune, Autoscaler, and most Python files do not require you to build and compile Ray. Follow these instructions to develop Ray's Python files locally without building Ray.
Compilation
-----------
To speed up compilation, be sure to install Ray with the following commands:
1. Pip install the **latest Ray wheels.** See :ref:`install-nightlies` for instructions.
.. code-block:: shell
cd python
pip install -e . --verbose
pip install -U https://s3-us-west-2.amazonaws.com/ray-wheels/latest/ray-0.9.0.dev0-cp38-cp38-manylinux1_x86_64.whl
2. Fork and clone the project to your machine. Connect your repository to the upstream (main project) ray repository.
.. code-block:: shell
git clone https://github.com/[your username]/ray.git
cd ray
git remote add upstream https://github.com/ray-project/ray.git
# Make sure you are up-to-date on master.
4. Replace Python files in the installed package with your local editable copy. We provide a simple script to help you do this: ``ray/python/ray/setup-dev.py``. Running the script will remove the ``ray/tune``, ``ray/rllib``, ``ray/autoscaler`` dir (among other directories) bundled with the ``ray`` pip package, and replace them with links to your local code.
.. code-block:: shell
cd ray
python python/ray/setup-dev.py
# This replaces miniconda3/lib/python3.7/site-packages/ray/tune
# with your local `ray/python/ray/tune`.
.. warning:: Do not run ``pip uninstall ray`` or ``pip install -U`` (for Ray or Ray wheels) if setting up your environment this way. To uninstall or upgrade, you must ``rm -rf`` the installation site (usually a ``site-packages/ray`` location).
Building Ray (full)
-------------------
.. tip:: If you are only editing Tune/RLlib/Autoscaler files, follow instructions for :ref:`python-develop` to avoid long build times.
To build Ray, first install the following dependencies.
For Ubuntu, run the following commands:
.. code-block:: bash
sudo apt-get update
sudo apt-get install -y build-essential curl unzip psmisc
pip install cython==0.29.0 pytest
For MacOS, run the following commands:
.. code-block:: bash
brew update
brew install wget
pip install cython==0.29.0 pytest
For Windows, see the :ref:`Windows Dependencies <windows-dependencies>` section.
Ray can be built from the repository as follows.
.. code-block:: bash
git clone https://github.com/ray-project/ray.git
# Install Bazel.
ray/ci/travis/install-bazel.sh
# Optionally build the dashboard
# (requires Node.js, see https://nodejs.org/ for more information).
pushd ray/python/ray/dashboard/client
npm ci
npm run build
popd
# Install Ray.
cd ray/python
pip install -e . --verbose # Add --user if you see a permission denied error.
The ``-e`` means "editable", so changes you make to files in the Ray
directory will take effect without reinstalling the package. In contrast, if
you do ``python setup.py install``, files will be copied from the Ray
directory to a directory of Python packages (often something like
``$HOME/anaconda3/lib/python3.6/site-packages/ray``). This means that
changes you make to files in the Ray directory will not have any effect.
directory will take effect without reinstalling the package.
If you run into **Permission Denied** errors when running ``pip install``,
you can try adding ``--user``. You may also need to run something like ``sudo
chown -R "$USER" ~/anaconda3`` (substituting in the appropriate path).
.. warning:: if you run ``python setup.py install``, files will be copied from the Ray directory to a directory of Python packages (``/lib/python3.6/site-packages/ray``). This means that changes you make to files in the Ray directory will not have any effect.
If you make changes to the C++ or Python files, you will need to run the
build so C++ code is recompiled and/or Python files are redeployed in
the ``python`` directory. However, you do not need to rerun
``pip install -e .``. Instead, you can recompile much more quickly by running
the following:
.. code-block:: shell
bash build.sh
This command is not enough to recompile all C++ unit tests. To do so, see
`Testing locally`_.
Fast, Debug, and Optimized Builds
---------------------------------
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Currently, Ray is built with optimizations, which can take a long time and
interfere with debugging. To perform fast, debug, or optimized builds, you can
@@ -67,29 +127,24 @@ all of your development in the future.
Using ``dbg`` instead of ``fastbuild`` generates more debug information,
which can make it easier to debug with a debugger like ``gdb``.
.. _python-develop:
Building the Docs
-----------------
Developing Ray (Python Only)
----------------------------
If you make changes that require documentation changes, don't forget to
update the documentation!
.. note:: Unless otherwise stated, directory and file paths are relative to the project root directory.
RLlib, Tune, Autoscaler, and most Python files do not require you to build and compile Ray. Follow these instructions to develop Ray's Python files locally.
1. Pip install the **latest Ray wheels.** See :ref:`install-nightlies` for instructions.
2. Fork and clone the project to your machine. Connect your repository to the upstream (main project) ray repository.
When you make documentation changes, build them locally to verify they render
correctly. `Sphinx <http://sphinx-doc.org/>`_ is used to generate the documentation.
.. code-block:: shell
git clone https://github.com/[your username]/ray.git
cd ray
git remote add upstream https://github.com/ray-project/ray.git
# Make sure you are up-to-date on master.
cd doc
pip install -r requirements-doc.txt
make html
4. Run ``python python/ray/setup-dev.py``. This sets up links between the ``tune`` dir (among other directories) in your local repo and the one bundled with the ``ray`` package.
.. warning:: Do not run ``pip uninstall ray`` or ``pip install -U`` (for Ray or Ray wheels) if setting up your environment this way. To uninstall or upgrade, you must ``rm -rf`` the installation site (usually a ``site-packages/ray`` location).
Once done, the docs will be in ``doc/_build/html``. For example, on Mac
OSX, you can open the docs (assuming you are still in the ``doc``
directory) using ``open _build/html/index.html``.
Using a local repository for dependencies
@@ -117,247 +172,4 @@ If the dependency already has a Bazel build file in it, you can use
To test switching back to the original rule, change ``False`` to ``True``.
Debugging
---------
Starting processes in a debugger
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
When processes are crashing, it is often useful to start them in a debugger.
Ray currently allows processes to be started in the following:
- valgrind
- the valgrind profiler
- the perftools profiler
- gdb
- tmux
To use any of these tools, please make sure that you have them installed on
your machine first (``gdb`` and ``valgrind`` on MacOS are known to have issues).
Then, you can launch a subset of ray processes by adding the environment
variable ``RAY_{PROCESS_NAME}_{DEBUGGER}=1``. For instance, if you wanted to
start the raylet in ``valgrind``, then you simply need to set the environment
variable ``RAY_RAYLET_VALGRIND=1``.
To start a process inside of ``gdb``, the process must also be started inside of
``tmux``. So if you want to start the raylet in ``gdb``, you would start your
Python script with the following:
.. code-block:: bash
RAY_RAYLET_GDB=1 RAY_RAYLET_TMUX=1 python
You can then list the ``tmux`` sessions with ``tmux ls`` and attach to the
appropriate one.
You can also get a core dump of the ``raylet`` process, which is especially
useful when filing `issues`_. The process to obtain a core dump is OS-specific,
but usually involves running ``ulimit -c unlimited`` before starting Ray to
allow core dump files to be written.
Inspecting Redis shards
~~~~~~~~~~~~~~~~~~~~~~~
To inspect Redis, you can use the global state API. The easiest way to do this
is to start or connect to a Ray cluster with ``ray.init()``, then query the API
like so:
.. code-block:: python
ray.init()
ray.nodes()
# Returns current information about the nodes in the cluster, such as:
# [{'ClientID': '2a9d2b34ad24a37ed54e4fcd32bf19f915742f5b',
# 'IsInsertion': True,
# 'NodeManagerAddress': '1.2.3.4',
# 'NodeManagerPort': 43280,
# 'ObjectManagerPort': 38062,
# 'ObjectStoreSocketName': '/tmp/ray/session_2019-01-21_16-28-05_4216/sockets/plasma_store',
# 'RayletSocketName': '/tmp/ray/session_2019-01-21_16-28-05_4216/sockets/raylet',
# 'Resources': {'CPU': 8.0, 'GPU': 1.0}}]
To inspect the primary Redis shard manually, you can also query with commands
like the following.
.. code-block:: python
r_primary = ray.worker.global_worker.redis_client
r_primary.keys("*")
To inspect other Redis shards, you will need to create a new Redis client.
For example (assuming the relevant IP address is ``127.0.0.1`` and the
relevant port is ``1234``), you can do this as follows.
.. code-block:: python
import redis
r = redis.StrictRedis(host='127.0.0.1', port=1234)
You can find a list of the relevant IP addresses and ports by running
.. code-block:: python
r_primary.lrange('RedisShards', 0, -1)
.. _backend-logging:
Backend logging
~~~~~~~~~~~~~~~
The ``raylet`` process logs detailed information about events like task
execution and object transfers between nodes. To set the logging level at
runtime, you can set the ``RAY_BACKEND_LOG_LEVEL`` environment variable before
starting Ray. For example, you can do:
.. code-block:: shell
export RAY_BACKEND_LOG_LEVEL=debug
ray start
This will print any ``RAY_LOG(DEBUG)`` lines in the source code to the
``raylet.err`` file, which you can find in the `Temporary Files`_.
Testing locally
---------------
Testing for Python development
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Suppose that one of the tests in a file of tests, e.g.,
``python/ray/tests/test_basic.py``, is failing. You can run just that
test file locally as follows:
.. code-block:: shell
python -m pytest -v python/ray/tests/test_basic.py
However, this will run all of the tests in the file, which can take some
time. To run a specific test that is failing, you can do the following
instead:
.. code-block:: shell
python -m pytest -v python/ray/tests/test_basic.py::test_keyword_args
When running tests, usually only the first test failure matters. A single
test failure often triggers the failure of subsequent tests in the same
file.
Testing for C++ development
~~~~~~~~~~~~~~~~~~~~~~~~~~~
To compile and run all C++ tests, you can run:
.. code-block:: shell
bazel test $(bazel query 'kind(cc_test, ...)')
Alternatively, you can also run one specific C++ test. You can use:
.. code-block:: shell
bazel test $(bazel query 'kind(cc_test, ...)') --test_filter=ClientConnectionTest --test_output=streamed
Building the Docs
-----------------
If you make changes that require documentation changes, don't forget to
update the documentation!
When you make documentation changes, build them locally to verify they render
correctly. `Sphinx <http://sphinx-doc.org/>`_ is used to generate the documentation.
.. code-block:: shell
cd doc
pip install -r requirements-doc.txt
make html
Once done, the docs will be in ``doc/_build/html``. For example, on Mac
OSX, you can open the docs (assuming you are still in the ``doc``
directory) using ``open _build/html/index.html``.
Creating a pull request
-----------------------
To create a pull request (PR) for your change, first go through the
`PR template`_ checklist and ensure you've completed all the steps.
When you push changes to GitHub, the formatting and verification script
``ci/travis/format.sh`` is run first. For pushing to your fork, you can
skip this step with ``git push --no-verify``.
Before submitting the PR, you should run this script. If it fails, the
push operation will not proceed. This script requires *specific versions*
of the following tools. Installation commands are shown for convenience:
* `yapf <https://github.com/google/yapf>`_ version ``0.23.0`` (``pip install yapf==0.23.0``)
* `flake8 <https://flake8.pycqa.org/en/latest/>`_ version ``3.7.7`` (``pip install flake8==3.7.7``)
* `flake8-quotes <https://github.com/zheller/flake8-quotes>`_ (``pip install flake8-quotes``)
* `clang-format <https://www.kernel.org/doc/html/latest/process/clang-format.html>`_ version ``7.0.0`` (download this version of Clang from `here <http://releases.llvm.org/download.html>`_)
**Note:** On MacOS X, don't use HomeBrew to install ``clang-format``, as the only version available is too new.
The Ray project automatically runs continuous integration (CI) tests once a PR
is opened using `Travis-CI <https://travis-ci.com/ray-project/ray/>`_ with
multiple CI test jobs.
Understand CI test jobs
-----------------------
The `Travis CI`_ test folder contains all integration test scripts and they
invoke other test scripts via ``pytest``, ``bazel``-based test or other bash
scripts. Some of the examples include:
* Raylet integration tests commands:
* ``bazel test //:core_worker_test``
* ``src/ray/test/run_object_manager_tests.sh``
* Bazel test command:
* ``bazel test --build_tests_only //:all``
* Ray serving test commands:
* ``python -m pytest python/ray/serve/tests``
* ``python python/ray/serve/examples/echo_full.py``
If a Travis-CI build exception doesn't appear to be related to your change,
please visit `this link <https://ray-travis-tracker.herokuapp.com/>`_ to
check recent tests known to be flaky.
Format and Linting
------------------
Installation instructions for the tools mentioned here are discussed above in
`Creating a pull request`_.
**Running the linter locally:** To run the Python linter on a specific file, run
``flake8`` as in this example, ``flake8 python/ray/worker.py``.
**Autoformatting code**. We use `yapf <https://github.com/google/yapf>`_ for
linting. The config file is ``.style.yapf``. We recommend running
``scripts/yapf.sh`` prior to pushing a PR to format any changed files. Note
that some projects, such as dataframes and rllib, are currently excluded.
**Running CI linter:** The Travis CI linter script has multiple components to
run. We recommend running ``ci/travis/format.sh``, which runs both linters for
Python and C++ codes. In addition, there are other formatting checkers for
components like the following:
* Python REAME format:
.. code-block:: shell
cd python
python setup.py check --restructuredtext --strict --metadata
* Bazel format:
.. code-block:: shell
./ci/travis/bazel-format.sh
.. _`issues`: https://github.com/ray-project/ray/issues
.. _`Temporary Files`: http://docs.ray.io/en/latest/tempfile.html
.. _`PR template`: https://github.com/ray-project/ray/blob/master/.github/PULL_REQUEST_TEMPLATE.md
.. _`Travis CI`: https://github.com/ray-project/ray/tree/master/ci/travis
doc/source/getting-involved.rst
+134 -20
View File
@@ -1,5 +1,19 @@
Getting Involved
================
.. _getting-involved:
Getting Involved / Contributing
===============================
Ray is more than a framework for distributed applications but also an active community of developers,
researchers, and folks that love machine learning.
.. tip:: Join our `community slack <https://forms.gle/9TSdDYUgxYs8SA9e8>`_ to discuss Ray! The community is extremely active in helping people succeed in building their ray applications.
You can join (and Star!) us on `on GitHub`_.
.. _`on GitHub`: https://github.com/ray-project/ray
Contributing to Ray
-------------------
We welcome (and encourage!) all forms of contributions to Ray, including and not limited to:
@@ -18,6 +32,12 @@ What can I work on?
We use Github to track issues, feature requests, and bugs. Take a look at the
ones labeled `"good first issue" <https://github.com/ray-project/ray/issues?utf8=%E2%9C%93&q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22>`__ and `"help wanted" <https://github.com/ray-project/ray/issues?q=is%3Aopen+is%3Aissue+label%3A%22help+wanted%22>`__ for a place to start.
Setting up your development environment
---------------------------------------
To edit the Ray source code, you'll want to checkout the repository and also build Ray from source. Follow :ref:`these instructions for building <building-ray>` a local copy of Ray to easily make changes.
Submitting and Merging a Contribution
-------------------------------------
@@ -42,7 +62,6 @@ There are a couple steps to merge a contribution.
6. Reviewers will merge and approve the pull request; be sure to ping them if
the pull request is getting stale.
Testing
-------
@@ -57,13 +76,123 @@ burden and speedup review process.
Documentation should be documented in `Google style <https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html>`__ format.
Testing for Python development
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Suppose that one of the tests in a file of tests, e.g.,
``python/ray/tests/test_basic.py``, is failing. You can run just that
test file locally as follows:
.. code-block:: shell
python -m pytest -v python/ray/tests/test_basic.py
However, this will run all of the tests in the file, which can take some
time. To run a specific test that is failing, you can do the following
instead:
.. code-block:: shell
pytest test_file.py -v -k [test substring]
When running tests, usually only the first test failure matters. A single
test failure often triggers the failure of subsequent tests in the same
file.
.. code-block:: shell
# Stop after first failure.
pytest test_file.py -x
Testing for C++ development
~~~~~~~~~~~~~~~~~~~~~~~~~~~
To compile and run all C++ tests, you can run:
.. code-block:: shell
bazel test $(bazel query 'kind(cc_test, ...)')
Alternatively, you can also run one specific C++ test. You can use:
.. code-block:: shell
bazel test $(bazel query 'kind(cc_test, ...)') --test_filter=ClientConnectionTest --test_output=streamed
Lint and Formatting
~~~~~~~~~~~~~~~~~~~
We also have tests for code formatting and linting that need to pass before merge.
Install ``yapf==0.23, flake8, flake8-quotes``. You can run the following locally:
Install ``yapf==0.23, flake8, flake8-quotes``.
* `yapf <https://github.com/google/yapf>`_ version ``0.23.0`` (``pip install yapf==0.23.0``)
* `flake8 <https://flake8.pycqa.org/en/latest/>`_ version ``3.7.7`` (``pip install flake8==3.7.7``)
* `flake8-quotes <https://github.com/zheller/flake8-quotes>`_ (``pip install flake8-quotes``)
* If developing for C++, you will need `clang-format <https://www.kernel.org/doc/html/latest/process/clang-format.html>`_ version ``7.0.0`` (download this version of Clang from `here <http://releases.llvm.org/download.html>`_)
.. note:: On MacOS X, don't use HomeBrew to install ``clang-format``, as the only version available is too new.
You can run the following locally:
.. code-block:: shell
ray/scripts/format.sh
An output like the following indicates failure:
.. code-block:: shell
WARNING: clang-format is not installed! # This is harmless
From https://github.com/ray-project/ray
* branch master -> FETCH_HEAD
python/ray/util/sgd/tf/tf_runner.py:4:1: F401 'numpy as np' imported but unused # Below is the failure
In addition, there are other formatting checkers for components like the following:
* Python README format:
.. code-block:: shell
cd python
python setup.py check --restructuredtext --strict --metadata
* Bazel format:
.. code-block:: shell
./ci/travis/bazel-format.sh
Understanding CI test jobs
--------------------------
The Ray project automatically runs continuous integration (CI) tests once a PR
is opened using `Travis-CI <https://travis-ci.com/ray-project/ray/>`_ with
multiple CI test jobs.
The `Travis CI`_ test folder contains all integration test scripts and they
invoke other test scripts via ``pytest``, ``bazel``-based test or other bash
scripts. Some of the examples include:
* Raylet integration tests commands:
* ``bazel test //:core_worker_test``
* ``src/ray/test/run_object_manager_tests.sh``
* Bazel test command:
* ``bazel test --build_tests_only //:all``
* Ray serving test commands:
* ``pytest python/ray/serve/tests``
* ``python python/ray/serve/examples/echo_full.py``
If a Travis-CI build exception doesn't appear to be related to your change,
please visit `this link <https://ray-travis-tracker.herokuapp.com/>`_ to
check recent tests known to be flaky.
.. _`Travis CI`: https://github.com/ray-project/ray/tree/master/ci/travis
Becoming a Reviewer
-------------------
@@ -79,22 +208,7 @@ solicited by current reviewers.
More Resources for Getting Involved
-----------------------------------
- `ray-dev@googlegroups.com`_: For discussions about development or any general
questions.
- `StackOverflow`_: For questions about how to use Ray.
- `GitHub Issues`_: For reporting bugs and feature requests.
- `Pull Requests`_: For submitting code contributions.
- `Meetup Group`_: Join our meetup group.
- `Community Slack`_: Join our Slack workspace.
- `Twitter`_: Follow updates on Twitter.
.. _`ray-dev@googlegroups.com`: https://groups.google.com/forum/#!forum/ray-dev
.. _`GitHub Issues`: https://github.com/ray-project/ray/issues
.. _`StackOverflow`: https://stackoverflow.com/questions/tagged/ray
.. _`Pull Requests`: https://github.com/ray-project/ray/pulls
.. _`Meetup Group`: https://www.meetup.com/Bay-Area-Ray-Meetup/
.. _`Community Slack`: https://forms.gle/9TSdDYUgxYs8SA9e8
.. _`Twitter`: https://twitter.com/raydistributed
.. include:: ray-overview/involvement.rst
.. note::
doc/source/index.rst
+27 -28
View File
@@ -13,6 +13,8 @@ Ray uses Tasks (functions) and Actors (Classes) to allow you to parallelize your
.. code-block:: python
# First, run `pip install ray`.
import ray
ray.init()
@@ -39,25 +41,22 @@ Ray uses Tasks (functions) and Actors (Classes) to allow you to parallelize your
futures = [c.read.remote() for c in counters]
print(ray.get(futures)) # [1, 1, 1, 1]
The Ray Community
-----------------
Ray is more than a framework for distributed applications but also an active community of developers,
researchers, and folks that love machine learning.
.. tip:: Join our `community slack <https://forms.gle/9TSdDYUgxYs8SA9e8>`_ to discuss Ray! The community is extremely active in helping people succeed in building their ray applications.
You can join (and Star!) us on `on GitHub`_.
You can also get started by visiting our `Tutorials <https://github.com/ray-project/tutorial>`_. For the latest wheels (nightlies), see the `installation page <installation.html>`__.
.. _`on GitHub`: https://github.com/ray-project/ray
Getting Involved
================
.. include:: ray-overview/involvement.rst
If you're interested in contributing to Ray, visit our page on :ref:`Getting Involved <getting-involved>` to read about the contribution process and see what you can work on!
More Information
================
Here are some talks, papers, and press coverage involving Ray and its libraries. Please raise an issue if any of the below links are broken!
Here are some talks, papers, and press coverage involving Ray and its libraries. Please raise an issue if any of the below links are broken, or if you'd like to add your own talk!
Blog and Press
--------------
@@ -113,25 +112,11 @@ Academic Papers
.. _`RLlib paper`: https://arxiv.org/abs/1712.09381
.. _`Tune paper`: https://arxiv.org/abs/1807.05118
Getting Involved
================
- `ray-dev@googlegroups.com`_: For discussions about development or any general
questions.
- `StackOverflow`_: For questions about how to use Ray.
- `GitHub Issues`_: For reporting bugs and feature requests.
- `Pull Requests`_: For submitting code contributions.
.. _`ray-dev@googlegroups.com`: https://groups.google.com/forum/#!forum/ray-dev
.. _`GitHub Issues`: https://github.com/ray-project/ray/issues
.. _`StackOverflow`: https://stackoverflow.com/questions/tagged/ray
.. _`Pull Requests`: https://github.com/ray-project/ray/pulls
.. toctree::
:hidden:
:maxdepth: -1
:caption: Overview of Ray
@@ -139,6 +124,7 @@ Getting Involved
installation.rst
.. toctree::
:hidden:
:maxdepth: -1
:caption: Ray Core
@@ -151,6 +137,7 @@ Getting Involved
package-ref.rst
.. toctree::
:hidden:
:maxdepth: -1
:caption: Ray Serve
@@ -162,6 +149,7 @@ Getting Involved
serve/package-ref.rst
.. toctree::
:hidden:
:maxdepth: -1
:caption: Ray Tune
@@ -171,6 +159,7 @@ Getting Involved
tune-contrib.rst
.. toctree::
:hidden:
:maxdepth: -1
:caption: RLlib
@@ -187,6 +176,7 @@ Getting Involved
rllib-dev.rst
.. toctree::
:hidden:
:maxdepth: -1
:caption: Ray SGD
@@ -197,6 +187,7 @@ Getting Involved
raysgd/raysgd_ref.rst
.. toctree::
:hidden:
:maxdepth: -1
:caption: Other Libraries
@@ -206,10 +197,18 @@ Getting Involved
pandas_on_ray.rst
.. toctree::
:hidden:
:maxdepth: -1
:caption: Development and Internals
:caption: Contributing
getting-involved.rst
.. toctree::
:hidden:
:maxdepth: -1
:caption: Development and Ray Internals
development.rst
debugging.rst
profiling.rst
fault-tolerance.rst
getting-involved.rst
doc/source/installation.rst
+38 -115
View File
@@ -70,6 +70,33 @@ For example, here are the Ray 0.9.0.dev0 wheels for Python 3.5, MacOS for commit
pip install https://ray-wheels.s3-us-west-2.amazonaws.com/master/a0ba4499ac645c9d3e82e68f3a281e48ad57f873/ray-0.9.0.dev0-cp35-cp35m-macosx_10_13_intel.whl
Installing Dashboard
--------------------
The dashboard requires a few additional Python packages, which can be installed
via pip.
.. code-block:: bash
pip install ray[dashboard]
The command ``ray.init()`` or ``ray start --head`` will print out the address of
the dashboard. For example,
.. code-block:: python
>>> import ray
>>> ray.init()
======================================================================
View the dashboard at http://127.0.0.1:8265.
Note: If Ray is running on a remote node, you will need to set up an
SSH tunnel with local port forwarding in order to access the dashboard
in your browser, e.g. by running 'ssh -L 8265:127.0.0.1:8265
<username>@<host>'. Alternatively, you can set dashboard_host="0.0.0.0" in
the call to ray.init() to allow direct access from external machines.
======================================================================
.. _windows-support:
Windows Support
@@ -105,100 +132,6 @@ the runtime library files (e.g. ``VCRUNTIME140_1.dll``):
.. _`Visual C++ Runtime`: https://support.microsoft.com/en-us/help/2977003/the-latest-supported-visual-c-downloads
.. _`install link`: https://aka.ms/vs/16/release/vc_redist.x64.exe
Building Ray from Source
------------------------
Installing from ``pip`` should be sufficient for most Ray users.
However, should you need to build from source, follow instructions below for
both Linux and MacOS.
Dependencies
~~~~~~~~~~~~
To build Ray, first install the following dependencies.
For Ubuntu, run the following commands:
.. code-block:: bash
sudo apt-get update
sudo apt-get install -y build-essential curl unzip psmisc
pip install cython==0.29.0 pytest
For MacOS, run the following commands:
.. code-block:: bash
brew update
brew install wget
pip install cython==0.29.0 pytest
For Windows, see the :ref:`Windows Dependencies <windows-dependencies>` section.
Install Ray
~~~~~~~~~~~
Ray can be built from the repository as follows.
We recommend avoiding paths with spaces or other special characters to avoid potential problems.
However, should you encounter any related issues, please let us know.
.. code-block:: bash
git clone https://github.com/ray-project/ray.git
# Install Bazel.
ray/ci/travis/install-bazel.sh
# Optionally build the dashboard (requires Node.js, see below for more information).
pushd ray/python/ray/dashboard/client
npm ci
npm run build
popd
# Install Ray.
cd ray/python
pip install -e . --verbose # Add --user if you see a permission denied error.
[Optional] Dashboard support
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If you would like to use the dashboard, you will additionally need to install
`Node.js`_ and build the dashboard before installing Ray. The relevant build
steps are included in the installation instructions above.
(Note that the dashboard may not yet work on Windows.)
.. _`Node.js`: https://nodejs.org/
The dashboard requires a few additional Python packages, which can be installed
via pip.
.. code-block:: bash
pip install ray[dashboard]
The command ``ray.init()`` or ``ray start --head`` will print out the address of
the dashboard. For example,
.. code-block:: python
>>> import ray
>>> ray.init()
======================================================================
View the dashboard at http://127.0.0.1:8265.
Note: If Ray is running on a remote node, you will need to set up an
SSH tunnel with local port forwarding in order to access the dashboard
in your browser, e.g. by running 'ssh -L 8265:127.0.0.1:8265
<username>@<host>'. Alternatively, you can set dashboard_host="0.0.0.0" in
the call to ray.init() to allow direct access from external machines.
======================================================================
Installing Ray on Arch Linux
----------------------------
@@ -244,6 +177,17 @@ Use ``pip list`` to confirm that ``ray`` is installed.
.. _`Anaconda`: https://www.anaconda.com/
Building Ray from Source
------------------------
Installing from ``pip`` should be sufficient for most Ray users.
However, should you need to build from source, follow :ref:`these instructions for building <building-ray>` Ray.
Docker Source Images
--------------------
@@ -312,24 +256,3 @@ that you've cloned the git repository.
.. code-block:: bash
python -m pytest -v python/ray/tests/test_mini.py
Troubleshooting installing Arrow
--------------------------------
Some candidate possibilities.
You have a different version of Flatbuffers installed
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Arrow pulls and builds its own copy of Flatbuffers, but if you already have
Flatbuffers installed, Arrow may find the wrong version. If a directory like
``/usr/local/include/flatbuffers`` shows up in the output, this may be the
problem. To solve it, get rid of the old version of flatbuffers.
There is some problem with Boost
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If a message like ``Unable to find the requested Boost libraries`` appears when
installing Arrow, there may be a problem with Boost. This can happen if you
installed Boost using MacPorts. This is sometimes solved by using Brew instead.
doc/source/ray-overview/involvement.rst
+17
View File
@@ -0,0 +1,17 @@
Ray is more than a framework for distributed applications but also an active community of developers,
researchers, and folks that love machine learning. Here's a list of tips for getting involved with the Ray community:
- Join our `community slack <https://forms.gle/9TSdDYUgxYs8SA9e8>`_ to discuss Ray! The community is extremely active in helping people succeed in building their ray applications.
- Star and follow us on `on GitHub`_.
- Join our `Meetup Group`_ to connect with others in the community!
- Use the `[ray]` tag on `StackOverflow`_ to ask and answer questions about Ray usage
- Subscribe to `ray-dev@googlegroups.com`_ to join development discussions.
- Follow us and spread the word on `Twitter`_!
.. _`ray-dev@googlegroups.com`: https://groups.google.com/forum/#!forum/ray-dev
.. _`GitHub Issues`: https://github.com/ray-project/ray/issues
.. _`StackOverflow`: https://stackoverflow.com/questions/tagged/ray
.. _`Pull Requests`: https://github.com/ray-project/ray/pulls
.. _`Twitter`: https://twitter.com/raydistributed
.. _`Meetup Group`: https://www.meetup.com/Bay-Area-Ray-Meetup/
.. _`on GitHub`: https://github.com/ray-project/ray