wassname/ray
1
0
Fork 0
mirror of https://github.com/wassname/ray.git synced 2026-08-20 12:40:44 +08:00
Code Issues Packages Projects Releases Wiki Activity

Improve release process from 0.8.2 (#7303)

Browse Source
This commit is contained in:
Simon Mo
2020-02-24 21:18:53 -08:00
committed by GitHub
parent f2faf8d26e
commit 29b08ddc09
31 changed files with 594 additions and 443 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/dev/RELEASE_PROCESS.rst
+31 -25
View File
@@ -24,9 +24,10 @@ This document describes the process for creating new releases.
For a new micro release (e.g., 0.7.1): No action is required.
4. **Testing:** Before releasing, the following sets of tests should be run. The results
of each of these tests for previous releases are checked in under ``doc/dev/release_tests``,
and should be compared against to identify any regressions.
4. **Testing:** Before releasing, the following sets of tests should be run.
The results of each of these tests for previous releases are checked in
under ``doc/dev/release_tests``, and should be compared against to identify
any regressions.
1. Long-running tests
@@ -38,16 +39,16 @@ This document describes the process for creating new releases.
These tests should run for at least 24 hours (printing new iterations and CPU load
stable in the AWS console).
The last hundred lines or so printed by each test should be checked in under
``doc/dev/release_logs/<version>``.
2. Multi-node regression tests
2. Stress tests
Follow the same instruction as long running stress tests. The large scale distributed
regression tests identify potential performance regression in distributed environment.
The following test should be ran:
.. code-block:: bash
ray/ci/stress_tests/run_stress_tests.sh <release-version> <release-commit>
ray/ci/stress_tests/run_application_stress_tests.sh <release-version> <release-commit>
rllib train -f rllib/tuned_examples/compact-regression-test.yaml
- ``ci/regression_test/rllib_regression-tests`` run the compact regression test for rllib.
- ``ci/regression_test/rllib_stress_tests`` run multinode 8hr IMPALA trial.
- ``ci/regression_test/stress_tests`` contains two tests: ``many_tasks`` and ``dead_actors``.
Each of the test runs on 105 spot instances.
Make sure that these pass. For the RLlib regression tests, see the comment on the
file for the pass criteria. For the rest, it will be obvious if they passed.
@@ -59,12 +60,9 @@ This document describes the process for creating new releases.
3. Microbenchmarks
.. code-block:: bash
ray microbenchmark
Run `ray microbenchmark` on an `m4.16xl` instance running `Ubuntu 18.04` with `Python 3` to get the
latest microbenchmark numbers.
Run the ``ci/microbenchmark`` with the commit. Under the hood, the session will
run `ray microbenchmark` on an `m4.16xl` instance running `Ubuntu 18.04` with `Python 3`
to get the latest microbenchmark numbers.
The results should be checked in under ``doc/dev/release_logs/<version>``.
@@ -82,10 +80,11 @@ This document describes the process for creating new releases.
changes/updates/bugfixes and their PR numbers. Once you have a draft, send it
out to other Ray developers (especially those who contributed heavily during
this release) for feedback. At the end of the release note, you should also
add a list of contributors.
add a list of contributors. Make sure Ray, Tune, RLLib, Autoscaler are
capitalized correctly.
Run ``doc/dev/get_contributors.py`` to generate the list of commits corresponding
to this release and the formatted list of contributors.
to this release and the formatted list of contributors.
You will need to provide a GitHub personal access token
(github.com -> settings -> developer settings -> personal access tokens).
@@ -107,14 +106,16 @@ This document describes the process for creating new releases.
export RAY_HASH=... # e.g., 618147f57fb40368448da3b2fb4fd213828fa12b
export RAY_VERSION=... # e.g., 0.7.0
pip install -U https://s3-us-west-2.amazonaws.com/ray-wheels/releases/$RAY_VERSION/$RAY_HASH/ray-$RAY_VERSION-cp27-cp27mu-manylinux1_x86_64.whl
# Linux Wheels
pip install -U https://s3-us-west-2.amazonaws.com/ray-wheels/releases/$RAY_VERSION/$RAY_HASH/ray-$RAY_VERSION-cp35-cp35m-manylinux1_x86_64.whl
pip install -U https://s3-us-west-2.amazonaws.com/ray-wheels/releases/$RAY_VERSION/$RAY_HASH/ray-$RAY_VERSION-cp36-cp36m-manylinux1_x86_64.whl
pip install -U https://s3-us-west-2.amazonaws.com/ray-wheels/releases/$RAY_VERSION/$RAY_HASH/ray-$RAY_VERSION-cp37-cp37m-manylinux1_x86_64.whl
pip install -U https://s3-us-west-2.amazonaws.com/ray-wheels/releases/$RAY_VERSION/$RAY_HASH/ray-$RAY_VERSION-cp27-cp27m-macosx_10_6_intel.whl
pip install -U https://s3-us-west-2.amazonaws.com/ray-wheels/releases/$RAY_VERSION/$RAY_HASH/ray-$RAY_VERSION-cp35-cp35m-macosx_10_6_intel.whl
pip install -U https://s3-us-west-2.amazonaws.com/ray-wheels/releases/$RAY_VERSION/$RAY_HASH/ray-$RAY_VERSION-cp36-cp36m-macosx_10_6_intel.whl
pip install -U https://s3-us-west-2.amazonaws.com/ray-wheels/releases/$RAY_VERSION/$RAY_HASH/ray-$RAY_VERSION-cp37-cp37m-macosx_10_6_intel.whl
# Mac Wheels
pip install -U https://s3-us-west-2.amazonaws.com/ray-wheels/releases/$RAY_VERSION/$RAY_HASH/ray-$RAY_VERSION-cp35-cp35m-macosx_10_13_intel.whl
pip install -U https://s3-us-west-2.amazonaws.com/ray-wheels/releases/$RAY_VERSION/$RAY_HASH/ray-$RAY_VERSION-cp36-cp36m-macosx_10_13_intel.whl
pip install -U https://s3-us-west-2.amazonaws.com/ray-wheels/releases/$RAY_VERSION/$RAY_HASH/ray-$RAY_VERSION-cp37-cp37m-macosx_10_13_intel.whl
8. **Upload to PyPI Test:** Upload the wheels to the PyPI test site using
``twine``.
@@ -164,9 +165,14 @@ This document describes the process for creating new releases.
pip install -U ray
10. **Improve the release process:** Find some way to improve the release
10. **Create a point release on readthedocs page:** In the `read the docs project page`_,
mark the release branch as "active" so there is a point release for the documentation.
Add @richardliaw to add you if you don't have access.
11. **Improve the release process:** Find some way to improve the release
process so that whoever manages the release next will have an easier time.
.. _`sample PR for bumping a minor release version`: https://github.com/ray-project/ray/pull/6303
.. _`sample commit for bumping the release branch version`: https://github.com/ray-project/ray/commit/a39325d818339970e51677708d5596f4b8f790ce
.. _`GitHub release`: https://github.com/ray-project/ray/releases
.. _`read the docs project page`: https://readthedocs.org/projects/ray/
doc/dev/download_wheels.sh
+3 -3
View File
@@ -1,6 +1,6 @@
wget https://s3-us-west-2.amazonaws.com/ray-wheels/releases/$RAY_VERSION/$RAY_HASH/ray-$RAY_VERSION-cp35-cp35m-manylinux1_x86_64.whl
wget https://s3-us-west-2.amazonaws.com/ray-wheels/releases/$RAY_VERSION/$RAY_HASH/ray-$RAY_VERSION-cp36-cp36m-manylinux1_x86_64.whl
wget https://s3-us-west-2.amazonaws.com/ray-wheels/releases/$RAY_VERSION/$RAY_HASH/ray-$RAY_VERSION-cp37-cp37m-manylinux1_x86_64.whl
wget https://s3-us-west-2.amazonaws.com/ray-wheels/releases/$RAY_VERSION/$RAY_HASH/ray-$RAY_VERSION-cp35-cp35m-macosx_10_6_intel.whl
wget https://s3-us-west-2.amazonaws.com/ray-wheels/releases/$RAY_VERSION/$RAY_HASH/ray-$RAY_VERSION-cp36-cp36m-macosx_10_6_intel.whl
wget https://s3-us-west-2.amazonaws.com/ray-wheels/releases/$RAY_VERSION/$RAY_HASH/ray-$RAY_VERSION-cp37-cp37m-macosx_10_6_intel.whl
wget https://s3-us-west-2.amazonaws.com/ray-wheels/releases/$RAY_VERSION/$RAY_HASH/ray-$RAY_VERSION-cp35-cp35m-macosx_10_13_intel.whl
wget https://s3-us-west-2.amazonaws.com/ray-wheels/releases/$RAY_VERSION/$RAY_HASH/ray-$RAY_VERSION-cp36-cp36m-macosx_10_13_intel.whl
wget https://s3-us-west-2.amazonaws.com/ray-wheels/releases/$RAY_VERSION/$RAY_HASH/ray-$RAY_VERSION-cp37-cp37m-macosx_10_13_intel.whl
doc/dev/release_logs/0.8.2/microbenchmark.txt
+18
View File
@@ -0,0 +1,18 @@
# NOTE: Make sure to run this with OMP_NUM_THREADS=64, otherwise the put gigabytes per
# seconds will be reduced. Put latency was reduced due to extra ipc call to raylet
# for ref counting.
single client get calls per second 11743.14 +- 2062.85
single client put calls per second 3133.08 +- 89.81
single client put gigabytes per second 10.33 +- 7.96
multi client put calls per second 3590.16 +- 22.04
multi client put gigabytes per second 23.38 +- 0.63
single client tasks sync per second 1263.59 +- 63.16
single client tasks async per second 13959.14 +- 393.16
multi client tasks async per second 42285.81 +- 238.55
1:1 actor calls sync per second 2159.21 +- 112.97
1:1 actor calls async per second 7048.53 +- 63.8
1:1 actor calls concurrent per second 6167.01 +- 75.67
1:n actor calls async per second 12241.67 +- 62.13
n:n actor calls async per second 41766.33 +- 672.14
n:n actor calls with arg async per second 13134.22 +- 71.68
doc/dev/release_logs/0.8.2/rllib_regression.txt
+36
View File
@@ -0,0 +1,36 @@
== Status ==
Memory usage on this node: 43.4/480.3 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/64 CPUs, 0.0/8 GPUs, 0.0/440.23 GiB heap, 0.0/12.84 GiB objects
Result logdir: /home/ubuntu/ray_results/apex
Result logdir: /home/ubuntu/ray_results/atari-a2c
Result logdir: /home/ubuntu/ray_results/atari-basic-dqn
Result logdir: /home/ubuntu/ray_results/atari-impala
Result logdir: /home/ubuntu/ray_results/atari-ppo-tf
Result logdir: /home/ubuntu/ray_results/atari-ppo-torch
Number of trials: 24 (24 TERMINATED)
Table truncated to 20 rows. 4 trials (4 TERMINATED) not shown.
+--------------------------------------+------------+-------+----------+------------------+---------+--------+
| Trial name | status | loc | reward | total time (s) | ts | iter |
|--------------------------------------+------------+-------+----------+------------------+---------+--------|
| A2C_BreakoutNoFrameskip-v4_c8ad5a48 | TERMINATED | | 139.19 | 3606.77 | 3686000 | 352 |
| A2C_BreakoutNoFrameskip-v4_c8ad1c54 | TERMINATED | | 75.56 | 3601.57 | 2932000 | 349 |
| A2C_BreakoutNoFrameskip-v4_c8acd28a | TERMINATED | | 131.97 | 3603.39 | 2928000 | 349 |
| A2C_BreakoutNoFrameskip-v4_c8ac8d16 | TERMINATED | | 105.42 | 3601.03 | 2901500 | 349 |
| DQN_BreakoutNoFrameskip-v4_c8af8a02 | TERMINATED | | 15.81 | 3665.65 | 270000 | 27 |
| DQN_BreakoutNoFrameskip-v4_c8af079e | TERMINATED | | 11.32 | 3612.1 | 270000 | 27 |
| APEX_BreakoutNoFrameskip-v4_c8ac4694 | TERMINATED | | 50.56 | 3627.89 | 5786880 | 115 |
| DQN_BreakoutNoFrameskip-v4_c8ae61ae | TERMINATED | | 7.14 | 3620.61 | 270000 | 27 |
| DQN_BreakoutNoFrameskip-v4_c8adbcea | TERMINATED | | 11.24 | 3640.35 | 270000 | 27 |
| APEX_BreakoutNoFrameskip-v4_c8abef3c | TERMINATED | | 94.5 | 3625.19 | 5820800 | 115 |
| PPO_BreakoutNoFrameskip-v4_c8ab0572 | TERMINATED | | 25.26 | 3603.23 | 1335000 | 267 |
| PPO_BreakoutNoFrameskip-v4_c8aabf36 | TERMINATED | | 18.2 | 3603.36 | 1300000 | 260 |
| APEX_BreakoutNoFrameskip-v4_c8abaa86 | TERMINATED | | 90.98 | 3627.03 | 7350400 | 116 |
| PPO_BreakoutNoFrameskip-v4_c8aa6f5e | TERMINATED | | 17.01 | 3611.01 | 1555000 | 311 |
| PPO_BreakoutNoFrameskip-v4_c8aa27e2 | TERMINATED | | 22.41 | 3609.64 | 1545000 | 309 |
| PPO_BreakoutNoFrameskip-v4_c8a9e39a | TERMINATED | | 61.25 | 3602.17 | 4475000 | 895 |
| PPO_BreakoutNoFrameskip-v4_c8a97978 | TERMINATED | | 28.19 | 3601.33 | 4415000 | 883 |
| PPO_BreakoutNoFrameskip-v4_c8a904ca | TERMINATED | | 41.3 | 3600.42 | 4515000 | 903 |
| APEX_BreakoutNoFrameskip-v4_c8ab5108 | TERMINATED | | 62.46 | 3626.37 | 5091840 | 114 |
| PPO_BreakoutNoFrameskip-v4_c8a88004 | TERMINATED | | 60.44 | 3602.52 | 3380000 | 676 |
+--------------------------------------+------------+-------+----------+------------------+---------+--------+
doc/dev/release_logs/0.8.2/stress_tests/application_stress_test.txt
+14
View File
@@ -0,0 +1,14 @@
== Status ==
Memory usage on this node: 34.6/480.3 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/640 CPUs, 0/8 GPUs, 0.0/2541.21 GiB heap, 0.0/128.42 GiB objects
Result logdir: /home/ubuntu/ray_results/atari-impala
Number of trials: 4 (4 TERMINATED)
+---------------------------------------------+------------+-------+-----------------------------+----------+------------------+----------+--------+
| Trial name | status | loc | env | reward | total time (s) | ts | iter |
|---------------------------------------------+------------+-------+-----------------------------+----------+------------------+----------+--------|
| IMPALA_BreakoutNoFrameskip-v4_2565545c | TERMINATED | | BreakoutNoFrameskip-v4 | 451.07 | 22555.3 | 30039500 | 381 |
| IMPALA_BeamRiderNoFrameskip-v4_2565e804 | TERMINATED | | BeamRiderNoFrameskip-v4 | 3124.8 | 24121.2 | 30057000 | 408 |
| IMPALA_QbertNoFrameskip-v4_256671de | TERMINATED | | QbertNoFrameskip-v4 | 8388.25 | 25163.5 | 30080000 | 453 |
| IMPALA_SpaceInvadersNoFrameskip-v4_256725ac | TERMINATED | | SpaceInvadersNoFrameskip-v4 | 780.65 | 23148.1 | 30026500 | 384 |
+---------------------------------------------+------------+-------+-----------------------------+----------+------------------+----------+--------+
doc/dev/release_logs/0.8.2/stress_tests/test_dead_actors.txt
+4
View File
@@ -0,0 +1,4 @@
Finished in: 98.49777579307556s
Average iteration time: 0.9849753308296204s
Max iteration time: 2.9459526538848877s
Min iteration time: 0.08075928688049316s
doc/dev/release_logs/0.8.2/stress_tests/test_many_tasks.txt
+15
View File
@@ -0,0 +1,15 @@
Stage 0 results:
Total time: 22.579216480255127
Stage 1 results:
Total time: 154.41431832313538
Average iteration time: 15.441423058509827
Max iteration time: 15.943994760513306
Min iteration time: 15.029884099960327
Stage 2 results:
Total time: 646.7662391662598
Average iteration time: 129.35279755592347
Max iteration time: 134.80017256736755
Min iteration time: 121.44297170639038
Stage 3 results:
Actor creation time: 0.0635519027709961
Total time: 3464.0461547374725