diff --git a/doc/dev/RELEASE_PROCESS.rst b/doc/dev/RELEASE_PROCESS.rst index 90de97faa..4d5eae734 100644 --- a/doc/dev/RELEASE_PROCESS.rst +++ b/doc/dev/RELEASE_PROCESS.rst @@ -49,7 +49,7 @@ This document describes the process for creating new releases. These suite of tests are similar to the standard long running tests, except these actually run in a multi-node cluster instead of just a simulated one. These tests should also run for at least 24 hours without erroring or hanging. - 2. Multi-node regression tests + 3. Multi-node regression tests Follow the same instruction as long running stress tests. The large scale distributed regression tests identify potential performance regression in distributed environment. @@ -68,7 +68,7 @@ This document describes the process for creating new releases. The summaries printed by each test should be checked in under ``doc/dev/release_logs/``. - 3. Microbenchmarks + 4. Microbenchmarks Run the ``ci/microbenchmark`` with the commit. Under the hood, the session will run `ray microbenchmark` on an `m4.16xl` instance running `Ubuntu 18.04` with `Python 3` @@ -79,6 +79,11 @@ This document describes the process for creating new releases. You can also get the performance change rate from the previous version using microbenchmark_analysis.py + 5. ASAN tests + + Run the ``ci/asan_tests`` with the commit. This will enable ASAN build and run the + whole Python tests to detect memory leaks. + 5. **Resolve release-blockers:** If a release blocking issue arises, there are two ways the issue can be resolved: 1) Fix the issue on the master branch and cherry-pick the relevant commit (using ``git cherry-pick``) onto the release @@ -160,12 +165,13 @@ This document describes the process for creating new releases. scripts. Make sure that it is finding the version of Ray that you just installed by checking ``ray.__version__`` and ``ray.__file__``. - Do this at least for MacOS and Linux. + Do this for MacOS, Linux, and Windows. This process is automated. Run ./bin/pip_download_test.sh. This will download the ray from the test pypi repository and run the minimum sanity check from all the Python version supported. (3.6, 3.7, 3.8) + Windows sanity check test is currently not automated. 9. **Upload to PyPI:** Now that you've tested the wheels on the PyPI test repository, they can be uploaded to the main PyPI repository. Be careful, diff --git a/doc/dev/download_wheels.sh b/doc/dev/download_wheels.sh index f55c6b839..08cefd351 100644 --- a/doc/dev/download_wheels.sh +++ b/doc/dev/download_wheels.sh @@ -21,3 +21,7 @@ wget "https://s3-us-west-2.amazonaws.com/ray-wheels/releases/$RAY_VERSION/$RAY_H wget "https://s3-us-west-2.amazonaws.com/ray-wheels/releases/$RAY_VERSION/$RAY_HASH/ray-$RAY_VERSION-cp37-cp37m-macosx_10_13_intel.whl" # Wheel name convention has been changed from Python 3.8. wget "https://s3-us-west-2.amazonaws.com/ray-wheels/releases/$RAY_VERSION/$RAY_HASH/ray-$RAY_VERSION-cp38-cp38-macosx_10_13_x86_64.whl" +# Make sure Windows wheels are downloadable without errors. +wget https://ray-wheels.s3-us-west-2.amazonaws.com/releases/$RAY_VERSION/$RAY_HASH/ray-$RAY_VERSION-cp36-cp36m-win_amd64.whl +wget https://ray-wheels.s3-us-west-2.amazonaws.com/releases/$RAY_VERSION/$RAY_HASH/ray-$RAY_VERSION-cp37-cp37m-win_amd64.whl +wget https://ray-wheels.s3-us-west-2.amazonaws.com/releases/$RAY_VERSION/$RAY_HASH/ray-$RAY_VERSION-cp38-cp38-win_amd64.whl diff --git a/doc/dev/release_logs/0.8.6/microbenchmark.txt b/doc/dev/release_logs/0.8.6/microbenchmark.txt new file mode 100644 index 000000000..ce3d7c4b9 --- /dev/null +++ b/doc/dev/release_logs/0.8.6/microbenchmark.txt @@ -0,0 +1,19 @@ +single client get calls per second 8746.49 +- 454.49 +single client put calls per second 5449.42 +- 86.57 +single client put gigabytes per second 13.59 +- 3.27 +multi client put calls per second 11662.1 +- 153.21 +multi client put gigabytes per second 17.26 +- 7.94 +single client tasks sync per second 1272.57 +- 33.26 +single client tasks async per second 14621.87 +- 200.32 +multi client tasks async per second 41588.79 +- 4142.82 +1:1 actor calls sync per second 1914.85 +- 20.58 +1:1 actor calls async per second 6541.67 +- 127.36 +1:1 actor calls concurrent per second 5663.49 +- 155.04 +1:n actor calls async per second 10495.3 +- 133.15 +n:n actor calls async per second 34453.12 +- 329.25 +n:n actor calls with arg async per second 12500.39 +- 167.03 +1:1 async-actor calls sync per second 1291.48 +- 48.9 +1:1 async-actor calls async per second 3510.1 +- 48.84 +1:1 async-actor calls with args async per second 2321.3 +- 70.18 +1:n async-actor calls async per second 8961.23 +- 111.43 +n:n async-actor calls async per second 27483.48 +- 113.91 diff --git a/doc/dev/release_logs/0.8.6/rllib_regression.txt b/doc/dev/release_logs/0.8.6/rllib_regression.txt new file mode 100644 index 000000000..7099ed781 --- /dev/null +++ b/doc/dev/release_logs/0.8.6/rllib_regression.txt @@ -0,0 +1,39 @@ +== Status == +Memory usage on this node: 14.3/480.3 GiB +Using FIFO scheduling algorithm. +Resources requested: 0/64 CPUs, 0.0/8 GPUs, 0.0/323.49 GiB heap, 0.0/98.39 GiB objects +Result logdir: /home/ubuntu/ray_results/apex +Result logdir: /home/ubuntu/ray_results/atari-a2c +Result logdir: /home/ubuntu/ray_results/atari-basic-dqn +Result logdir: /home/ubuntu/ray_results/atari-impala +Result logdir: /home/ubuntu/ray_results/atari-ppo-tf +Result logdir: /home/ubuntu/ray_results/atari-ppo-torch +Number of trials: 24 (24 TERMINATED) ++-------------------------------------------+------------+-------+--------+------------------+---------+----------+ +| Trial name | status | loc | iter | total time (s) | ts | reward | +|-------------------------------------------+------------+-------+--------+------------------+---------+----------| +| IMPALA_BreakoutNoFrameskip-v4_926d2_00000 | TERMINATED | | 357 | 3601.43 | 7523500 | 315.74 | +| IMPALA_BreakoutNoFrameskip-v4_926d2_00001 | TERMINATED | | 356 | 3603.43 | 7455000 | 340.84 | +| IMPALA_BreakoutNoFrameskip-v4_926d2_00002 | TERMINATED | | 357 | 3601.72 | 7622250 | 306.88 | +| IMPALA_BreakoutNoFrameskip-v4_926d2_00003 | TERMINATED | | 357 | 3601.02 | 7561500 | 331.75 | +| PPO_BreakoutNoFrameskip-v4_926d2_00004 | TERMINATED | | 692 | 3600.28 | 3460000 | 28.44 | +| PPO_BreakoutNoFrameskip-v4_926d2_00005 | TERMINATED | | 946 | 3600.06 | 4730000 | 53.61 | +| PPO_BreakoutNoFrameskip-v4_926d2_00006 | TERMINATED | | 943 | 3602.39 | 4715000 | 37.03 | +| PPO_BreakoutNoFrameskip-v4_926d2_00007 | TERMINATED | | 944 | 3602.4 | 4720000 | 36.35 | +| PPO_BreakoutNoFrameskip-v4_926d2_00008 | TERMINATED | | 27 | 3600.14 | 135000 | 2.02 | +| PPO_BreakoutNoFrameskip-v4_926d2_00009 | TERMINATED | | 27 | 3723.31 | 135000 | 5.37 | +| PPO_BreakoutNoFrameskip-v4_926d2_00010 | TERMINATED | | 21 | 3704.86 | 105000 | 1.5 | +| PPO_BreakoutNoFrameskip-v4_926d2_00011 | TERMINATED | | 22 | 3630.76 | 110000 | 1.98 | +| APEX_BreakoutNoFrameskip-v4_926d2_00012 | TERMINATED | | 106 | 3607.17 | 5372960 | 74.44 | +| APEX_BreakoutNoFrameskip-v4_926d2_00013 | TERMINATED | | 107 | 3603.72 | 8074080 | 87.35 | +| APEX_BreakoutNoFrameskip-v4_926d2_00014 | TERMINATED | | 90 | 3633.64 | 6787680 | 59.02 | +| APEX_BreakoutNoFrameskip-v4_926d2_00015 | TERMINATED | | 90 | 3616.73 | 6784160 | 61.9 | +| A2C_BreakoutNoFrameskip-v4_926d2_00016 | TERMINATED | | 354 | 3603.13 | 3347000 | 74.53 | +| A2C_BreakoutNoFrameskip-v4_926d2_00017 | TERMINATED | | 353 | 3601.95 | 3374000 | 174.69 | +| A2C_BreakoutNoFrameskip-v4_926d2_00018 | TERMINATED | | 353 | 3601.02 | 3377000 | 149.69 | +| A2C_BreakoutNoFrameskip-v4_926d2_00019 | TERMINATED | | 353 | 3605.72 | 3732000 | 203.06 | +| DQN_BreakoutNoFrameskip-v4_926d2_00020 | TERMINATED | | 27 | 3695.93 | 280000 | 14.71 | +| DQN_BreakoutNoFrameskip-v4_926d2_00021 | TERMINATED | | 26 | 3613.54 | 270000 | 16.37 | +| DQN_BreakoutNoFrameskip-v4_926d2_00022 | TERMINATED | | 27 | 3661.86 | 280000 | 15.43 | +| DQN_BreakoutNoFrameskip-v4_926d2_00023 | TERMINATED | | 27 | 3680.85 | 280000 | 15.62 | ++-------------------------------------------+------------+-------+--------+------------------+---------+----------+ diff --git a/doc/dev/release_logs/0.8.6/stress_tests/application_stress_test.txt b/doc/dev/release_logs/0.8.6/stress_tests/application_stress_test.txt new file mode 100644 index 000000000..309927648 --- /dev/null +++ b/doc/dev/release_logs/0.8.6/stress_tests/application_stress_test.txt @@ -0,0 +1,14 @@ +== Status == +Memory usage on this node: 19.9/480.3 GiB +Using FIFO scheduling algorithm. +Resources requested: 0/640 CPUs, 0/8 GPUs, 0.0/1906.49 GiB heap, 0.0/566.21 GiB objects +Result logdir: /home/ubuntu/ray_results/atari-impala +Number of trials: 4 (4 TERMINATED) ++------------------------------------------------+------------+-------+-----------------------------+--------+------------------+----------+----------+ +| Trial name | status | loc | env | iter | total time (s) | ts | reward | +|------------------------------------------------+------------+-------+-----------------------------+--------+------------------+----------+----------| +| IMPALA_BreakoutNoFrameskip-v4_a4e31_00000 | TERMINATED | | BreakoutNoFrameskip-v4 | 680 | 6847.72 | 30047500 | 358.37 | +| IMPALA_BeamRiderNoFrameskip-v4_a4e31_00001 | TERMINATED | | BeamRiderNoFrameskip-v4 | 694 | 6987.42 | 30001500 | 2657.04 | +| IMPALA_QbertNoFrameskip-v4_a4e31_00002 | TERMINATED | | QbertNoFrameskip-v4 | 676 | 6821.02 | 30016500 | 3606.75 | +| IMPALA_SpaceInvadersNoFrameskip-v4_a4e31_00003 | TERMINATED | | SpaceInvadersNoFrameskip-v4 | 684 | 6889.83 | 30025000 | 653.4 | ++------------------------------------------------+------------+-------+-----------------------------+--------+------------------+----------+----------+ diff --git a/doc/dev/release_logs/0.8.6/stress_tests/test_dead_actors.txt b/doc/dev/release_logs/0.8.6/stress_tests/test_dead_actors.txt new file mode 100644 index 000000000..450ba360a --- /dev/null +++ b/doc/dev/release_logs/0.8.6/stress_tests/test_dead_actors.txt @@ -0,0 +1,4 @@ +Finished in: 457.8229465484619s +Average iteration time: 4.578227622509003s +Max iteration time: 28.17513918876648s +Min iteration time: 0.31386494636535645s diff --git a/doc/dev/release_logs/0.8.6/stress_tests/test_many_tasks.txt b/doc/dev/release_logs/0.8.6/stress_tests/test_many_tasks.txt new file mode 100644 index 000000000..cb618992e --- /dev/null +++ b/doc/dev/release_logs/0.8.6/stress_tests/test_many_tasks.txt @@ -0,0 +1,16 @@ +Stage 0 results: + Total time: 21.636702299118042 +Stage 1 results: + Total time: 150.684504032135 + Average iteration time: 15.068440675735474 + Max iteration time: 16.537473440170288 + Min iteration time: 14.317448616027832 +Stage 2 results: + Total time: 1103.2014956474304 + Average iteration time: 220.63988456726074 + Max iteration time: 229.98777270317078 + Min iteration time: 203.2540054321289 +Stage 3 results: + Actor creation time: 0.09029483795166016 + Total time: 3138.855129480362 + \ No newline at end of file