wassname/ray
1
0
Fork 0
mirror of https://github.com/wassname/ray.git synced 2026-07-01 11:10:02 +08:00
Code Issues Packages Projects Releases Wiki Activity

087release update (#10651)

Browse Source 
* [BRING BACK TO MASTER] Release test result for microbenchmark.

* [BRING BACK TO MASTER] Update result and fix sanity check bugs.
This commit is contained in:
SangBin Cho
2020-09-08 15:45:06 -07:00
committed by GitHub
parent 87c4f36f02
commit 30fff71d96
17 changed files with 137 additions and 32 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/dev/bin/pip_download_test.sh
+11 -2
View File
@@ -3,6 +3,10 @@
# This script automatically download ray and run the sanity check (sanity_check.py)
# in various Python version. This script requires conda command to exist.
unset RAY_ADDRESS
export RAY_HASH=$RAY_HASH
export RAY_VERSION=$RAY_VERSION
if [[ -z "$RAY_HASH" ]]; then
echo "RAY_HASH env var should be provided"
exit 1
@@ -27,7 +31,7 @@ pip install --upgrade pip
# This is required to use conda activate
source "$(conda info --base)/etc/profile.d/conda.sh"
for PYTHON_VERSION in "3.5" "3.6" "3.7" "3.8"
for PYTHON_VERSION in "3.6" "3.7" "3.8"
do
env_name="${RAY_VERSION}-${PYTHON_VERSION}-env"
conda create -y -n "${env_name}" python=${PYTHON_VERSION}
@@ -41,7 +45,12 @@ do
printf "\n\n\n"
pip install redis==3.3.2
pip install msgpack==0.6.2
pip install msgpack==1.0.0
pip install aioredis
pip install colorful
pip install prometheus-client==0.7.1
pip install opencensus
pip install gpustat
pip install ray
pip uninstall -y ray
pip install --index-url https://test.pypi.org/simple/ ray
doc/dev/bin/ray-project/cluster.yaml
+6 -3
View File
@@ -5,7 +5,7 @@ cluster_name: sanity-check
# The maximum number of workers nodes to launch in addition to the head
# node. This takes precedence over min_workers. min_workers defaults to 0.
max_workers: 1
max_workers: 0
# Cloud-provider specific configuration.
provider:
@@ -23,13 +23,16 @@ head_setup_commands:
- wget --quiet https://repo.continuum.io/archive/Anaconda3-5.0.1-Linux-x86_64.sh || true
- bash Anaconda3-5.0.1-Linux-x86_64.sh -b -p $HOME/anaconda3 || true
- echo 'export PATH="$HOME/anaconda3/bin:$PATH"' >> ~/.bashrc
- echo 'RAY_VERSION=0.8.7' >> ~/.bashrc
- echo 'RAY_HASH=56810e136587432ed4758d882bf8894b930355e3' >> ~/.bashrc
- chmod +x ./pip_download_test.sh || true
- pip install -U pip
# Custom commands that will be run on worker nodes after common setup.
worker_setup_commands: []
# Command to start ray on the head node. You don't need to change this.
head_start_ray_commands: []
head_start_ray_commands: [ray stop]
# Command to start ray on worker nodes. You don't need to change this.
worker_start_ray_commands: []
worker_start_ray_commands: [ray stop]
doc/dev/download_wheels.sh
-2
View File
@@ -11,12 +11,10 @@ if [[ -z "$RAY_VERSION" ]]; then
fi
# Make sure Linux wheels are downloadable without errors.
wget "https://s3-us-west-2.amazonaws.com/ray-wheels/releases/$RAY_VERSION/$RAY_HASH/ray-$RAY_VERSION-cp35-cp35m-manylinux1_x86_64.whl"
wget "https://s3-us-west-2.amazonaws.com/ray-wheels/releases/$RAY_VERSION/$RAY_HASH/ray-$RAY_VERSION-cp36-cp36m-manylinux1_x86_64.whl"
wget "https://s3-us-west-2.amazonaws.com/ray-wheels/releases/$RAY_VERSION/$RAY_HASH/ray-$RAY_VERSION-cp37-cp37m-manylinux1_x86_64.whl"
wget "https://s3-us-west-2.amazonaws.com/ray-wheels/releases/$RAY_VERSION/$RAY_HASH/ray-$RAY_VERSION-cp38-cp38-manylinux1_x86_64.whl"
# Make sure macos wheels are downloadable without errors.
wget "https://s3-us-west-2.amazonaws.com/ray-wheels/releases/$RAY_VERSION/$RAY_HASH/ray-$RAY_VERSION-cp35-cp35m-macosx_10_13_intel.whl"
wget "https://s3-us-west-2.amazonaws.com/ray-wheels/releases/$RAY_VERSION/$RAY_HASH/ray-$RAY_VERSION-cp36-cp36m-macosx_10_13_intel.whl"
wget "https://s3-us-west-2.amazonaws.com/ray-wheels/releases/$RAY_VERSION/$RAY_HASH/ray-$RAY_VERSION-cp37-cp37m-macosx_10_13_intel.whl"
# Wheel name convention has been changed from Python 3.8.
doc/dev/microbenchmark_analysis.py
-1
View File
@@ -44,7 +44,6 @@ def main():
task_type = get_task_type(line)
task_performance = get_task_performance(line)
task_standard_deviation = get_task_std(line)
task_info[task_type].append(task_performance)
task_std_info[task_type].append(task_standard_deviation)
doc/dev/release_logs/0.7.7/microbenchmark.txt
+3 -3
View File
@@ -1,7 +1,7 @@
single client get calls per second 28595.02 +- 580.33
single client put calls per second 6313.62 +- 66.88
single client get calls (Plasma Store) per second 28595.02 +- 580.33
single client put calls (Plasma Store) per second 6313.62 +- 66.88
single client put gigabytes per second 11.6 +- 6.86
multi client put calls per second 16800.89 +- 381.69
multi client put calls (Plasma Store) per second 16800.89 +- 381.69
multi client put gigabytes per second 23.33 +- 0.96
single client tasks sync per second 1963.72 +- 48.48
single client tasks async per second 5181.29 +- 30.0
doc/dev/release_logs/0.8.0/microbenchmark.txt
+3 -3
View File
@@ -1,7 +1,7 @@
single client get calls per second 30526.52 +- 1121.22
single client put calls per second 6428.11 +- 51.25
single client get calls (Plasma Store) per second 30526.52 +- 1121.22
single client put calls (Plasma Store) per second 6428.11 +- 51.25
single client put gigabytes per second 12.85 +- 5.45
multi client put calls per second 16976.5 +- 60.13
multi client put calls (Plasma Store) per second 16976.5 +- 60.13
multi client put gigabytes per second 22.54 +- 0.86
single client tasks sync per second 1223.63 +- 18.36
single client tasks async per second 13715.93 +- 229.68
doc/dev/release_logs/0.8.1/microbenchmark.txt
+3 -3
View File
@@ -5,10 +5,10 @@
# and will be fixed in the future by caching buffers after the first ray.get()
# on each object ID.
single client get calls per second 12550.57 +- 1835.19
single client put calls per second 6791.78 +- 176.65
single client get calls (Plasma Store) per second 12550.57 +- 1835.19
single client put calls (Plasma Store) per second 6791.78 +- 176.65
single client put gigabytes per second 13.36 +- 5.7
multi client put calls per second 13503.59 +- 179.1
multi client put calls (Plasma Store) per second 13503.59 +- 179.1
multi client put gigabytes per second 16.22 +- 1.36
single client tasks sync per second 1295.56 +- 42.81
single client tasks async per second 14825.7 +- 358.92
doc/dev/release_logs/0.8.2/microbenchmark.txt
+3 -3
View File
@@ -2,10 +2,10 @@
# seconds will be reduced. Put latency was reduced due to extra ipc call to raylet
# for ref counting.
single client get calls per second 11743.14 +- 2062.85
single client put calls per second 3133.08 +- 89.81
single client get calls (Plasma Store) per second 11743.14 +- 2062.85
single client put calls (Plasma Store) per second 3133.08 +- 89.81
single client put gigabytes per second 10.33 +- 7.96
multi client put calls per second 3590.16 +- 22.04
multi client put calls (Plasma Store) per second 3590.16 +- 22.04
multi client put gigabytes per second 23.38 +- 0.63
single client tasks sync per second 1263.59 +- 63.16
single client tasks async per second 13959.14 +- 393.16
doc/dev/release_logs/0.8.3/microbenchmark.txt
+3 -3
View File
@@ -1,7 +1,7 @@
single client get calls per second 9609.58 +- 1094.38
single client put calls per second 5244.46 +- 96.12
single client get calls (Plasma Store) per second 9609.58 +- 1094.38
single client put calls (Plasma Store) per second 5244.46 +- 96.12
single client put gigabytes per second 13.5 +- 3.12
multi client put calls per second 11147.13 +- 25.58
multi client put calls (Plasma Store) per second 11147.13 +- 25.58
multi client put gigabytes per second 19.08 +- 7.08
single client tasks sync per second 1271.4 +- 14.02
single client tasks async per second 12516.66 +- 392.73
doc/dev/release_logs/0.8.4/microbenchmark.txt
+3 -3
View File
@@ -1,7 +1,7 @@
single client get calls per second 9887.7 +- 1181.06
single client put calls per second 5061.57 +- 89.58
single client get calls (Plasma Store) per second 9887.7 +- 1181.06
single client put calls (Plasma Store) per second 5061.57 +- 89.58
single client put gigabytes per second 13.23 +- 2.84
multi client put calls per second 10954.66 +- 107.58
multi client put calls (Plasma Store) per second 10954.66 +- 107.58
multi client put gigabytes per second 17.98 +- 8.51
single client tasks sync per second 1246.16 +- 64.56
single client tasks async per second 12836.66 +- 196.27
doc/dev/release_logs/0.8.5/microbenchmark.txt
+3 -3
View File
@@ -1,7 +1,7 @@
single client get calls per second 8214.4 +- 892.64
single client put calls per second 5356.26 +- 36.53
single client get calls (Plasma Store) per second 8214.4 +- 892.64
single client put calls (Plasma Store) per second 5356.26 +- 36.53
single client put gigabytes per second 13.46 +- 2.94
multi client put calls per second 11351.56 +- 62.33
multi client put calls (Plasma Store) per second 11351.56 +- 62.33
multi client put gigabytes per second 16.44 +- 8.49
single client tasks sync per second 1311.81 +- 66.92
single client tasks async per second 15531.87 +- 220.75
doc/dev/release_logs/0.8.6/microbenchmark.txt
+3 -3
View File
@@ -1,7 +1,7 @@
single client get calls per second 8746.49 +- 454.49
single client put calls per second 5449.42 +- 86.57
single client get calls (Plasma Store) per second 8746.49 +- 454.49
single client put calls (Plasma Store) per second 5449.42 +- 86.57
single client put gigabytes per second 13.59 +- 3.27
multi client put calls per second 11662.1 +- 153.21
multi client put calls (Plasma Store) per second 11662.1 +- 153.21
multi client put gigabytes per second 17.26 +- 7.94
single client tasks sync per second 1272.57 +- 33.26
single client tasks async per second 14621.87 +- 200.32
doc/dev/release_logs/0.8.7/microbenchmark.txt
+22
View File
@@ -0,0 +1,22 @@
single client get calls per second 28723.85 +- 2122.5
single client put calls per second 28457.3 +- 199.18
multi client put calls per second 136112.46 +- 2957.42
single client get calls (Plasma Store) per second 7913.41 +- 625.96
single client put calls (Plasma Store) per second 5012.85 +- 26.89
multi client put calls (Plasma Store) per second 8767.68 +- 122.88
single client put gigabytes per second 12.41 +- 11.96
multi client put gigabytes per second 39.28 +- 2.8
single client tasks sync per second 1058.16 +- 6.87
single client tasks async per second 14787.24 +- 190.62
multi client tasks async per second 36551.97 +- 3364.28
1:1 actor calls sync per second 1498.51 +- 41.25
1:1 actor calls async per second 6701.16 +- 95.16
1:1 actor calls concurrent per second 5864.73 +- 68.68
1:n actor calls async per second 14372.61 +- 542.3
n:n actor calls async per second 38479.59 +- 920.62
n:n actor calls with arg async per second 10603.63 +- 95.77
1:1 async-actor calls sync per second 1074.72 +- 34.12
1:1 async-actor calls async per second 3755.41 +- 53.77
1:1 async-actor calls with args async per second 2340.26 +- 62.86
1:n async-actor calls async per second 13353.84 +- 446.48
n:n async-actor calls async per second 29600.63 +- 284.26
doc/dev/release_logs/0.8.7/rllib_regression.txt
+41
View File
@@ -0,0 +1,41 @@
== Status ==
Memory usage on this node: 9.5/480.3 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/64 CPUs, 0.0/8 GPUs, 0.0/325.78 GiB heap, 0.0/99.07 GiB objects
Result logdir: /home/ubuntu/ray_results/apex
Result logdir: /home/ubuntu/ray_results/atari-a2c
Result logdir: /home/ubuntu/ray_results/atari-basic-dqn
Result logdir: /home/ubuntu/ray_results/atari-impala
Result logdir: /home/ubuntu/ray_results/atari-ppo-tf
Result logdir: /home/ubuntu/ray_results/atari-ppo-torch
Number of trials: 24 (24 TERMINATED)
+-------------------------------------------+------------+-------+--------+------------------+---------+----------+
| Trial name | status | loc | iter | total time (s) | ts | reward |
|-------------------------------------------+------------+-------+--------+------------------+---------+----------|
| IMPALA_BreakoutNoFrameskip-v4_b13b3_00000 | TERMINATED | | 355 | 3601.74 | 7491750 | 381.68 |
| IMPALA_BreakoutNoFrameskip-v4_b13b3_00001 | TERMINATED | | 355 | 3603.97 | 7482000 | 367.12 |
| IMPALA_BreakoutNoFrameskip-v4_b13b3_00002 | TERMINATED | | 356 | 3607.98 | 7528500 | 375.77 |
| IMPALA_BreakoutNoFrameskip-v4_b13b3_00003 | TERMINATED | | 355 | 3605.97 | 7411000 | 328.21 |
| PPO_BreakoutNoFrameskip-v4_b13b3_00004 | TERMINATED | | 664 | 3603.23 | 3320000 | 31.47 |
| PPO_BreakoutNoFrameskip-v4_b13b3_00005 | TERMINATED | | 906 | 3603.53 | 4530000 | 41.31 |
| PPO_BreakoutNoFrameskip-v4_b13b3_00006 | TERMINATED | | 897 | 3600.83 | 4485000 | 60.32 |
| PPO_BreakoutNoFrameskip-v4_b13b3_00007 | TERMINATED | | 913 | 3600.05 | 4565000 | 51.28 |
| PPO_BreakoutNoFrameskip-v4_b13b3_00008 | TERMINATED | | 492 | 3606.93 | 2460000 | 35.75 |
| PPO_BreakoutNoFrameskip-v4_b13b3_00009 | TERMINATED | | 480 | 3600.3 | 2400000 | 13.71 |
| PPO_BreakoutNoFrameskip-v4_b13b3_00010 | TERMINATED | | 416 | 3606.47 | 2080000 | 7.49 |
| PPO_BreakoutNoFrameskip-v4_b13b3_00011 | TERMINATED | | 422 | 3602.06 | 2110000 | 13.29 |
| APEX_BreakoutNoFrameskip-v4_b13b3_00012 | TERMINATED | | 106 | 3614.12 | 5371360 | 48.41 |
| APEX_BreakoutNoFrameskip-v4_b13b3_00013 | TERMINATED | | 103 | 3618.46 | 7777760 | 82.33 |
| APEX_BreakoutNoFrameskip-v4_b13b3_00014 | TERMINATED | | 94 | 3627.11 | 6155840 | 62.79 |
| APEX_BreakoutNoFrameskip-v4_b13b3_00015 | TERMINATED | | 92 | 3615.04 | 6159520 | 63.5 |
| A2C_BreakoutNoFrameskip-v4_b13b3_00016 | TERMINATED | | 352 | 3606.65 | 3135000 | 62.67 |
| A2C_BreakoutNoFrameskip-v4_b13b3_00017 | TERMINATED | | 351 | 3605.31 | 3098000 | 124.24 |
| A2C_BreakoutNoFrameskip-v4_b13b3_00018 | TERMINATED | | 351 | 3607.44 | 3113500 | 129.7 |
| A2C_BreakoutNoFrameskip-v4_b13b3_00019 | TERMINATED | | 353 | 3601.78 | 4086000 | 271.04 |
| DQN_BreakoutNoFrameskip-v4_b13b3_00020 | TERMINATED | | 24 | 3682.43 | 250000 | 11.27 |
| DQN_BreakoutNoFrameskip-v4_b13b3_00021 | TERMINATED | | 24 | 3681.4 | 250000 | 15.9 |
| DQN_BreakoutNoFrameskip-v4_b13b3_00022 | TERMINATED | | 24 | 3677.74 | 250000 | 11.62 |
| DQN_BreakoutNoFrameskip-v4_b13b3_00023 | TERMINATED | | 24 | 3664.08 | 250000 | 7.47 |
+-------------------------------------------+------------+-------+--------+------------------+---------+----------+
doc/dev/release_logs/0.8.7/stress_tests/application_stress_test.txt
+14
View File
@@ -0,0 +1,14 @@
== Status ==
Memory usage on this node: 22.0/480.3 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/640 CPUs, 0/8 GPUs, 0.0/1906.54 GiB heap, 0.0/566.21 GiB objects (0/1.0 GPUType:V100)
Result logdir: /home/ubuntu/ray_results/atari-impala
Number of trials: 4 (4 TERMINATED)
+------------------------------------------------+------------+-------+-----------------------------+--------+------------------+----------+----------+
| Trial name | status | loc | env | iter | total time (s) | ts | reward |
|------------------------------------------------+------------+-------+-----------------------------+--------+------------------+----------+----------|
| IMPALA_BreakoutNoFrameskip-v4_b783d_00000 | TERMINATED | | BreakoutNoFrameskip-v4 | 746 | 7521.77 | 30009000 | 462.43 |
| IMPALA_BeamRiderNoFrameskip-v4_b783d_00001 | TERMINATED | | BeamRiderNoFrameskip-v4 | 761 | 7671.4 | 30024500 | 2318.14 |
| IMPALA_QbertNoFrameskip-v4_b783d_00002 | TERMINATED | | QbertNoFrameskip-v4 | 739 | 7457.03 | 30022000 | 10025.5 |
| IMPALA_SpaceInvadersNoFrameskip-v4_b783d_00003 | TERMINATED | | SpaceInvadersNoFrameskip-v4 | 766 | 7718.84 | 30037000 | 786.8 |
+------------------------------------------------+------------+-------+-----------------------------+--------+------------------+----------+----------+
doc/dev/release_logs/0.8.7/stress_tests/test_dead_actors.txt
+4
View File
@@ -0,0 +1,4 @@
Finished in: 456.6799967288971s
Average iteration time: 4.566798083782196s
Max iteration time: 10.93368911743164s
Min iteration time: 1.0819299221038818s
doc/dev/release_logs/0.8.7/stress_tests/test_many_tasks.txt
+15
View File
@@ -0,0 +1,15 @@
Stage 0 results:
Total time: 22.120722770690918
Stage 1 results:
Total time: 174.56333804130554
Average iteration time: 17.456315088272095
Max iteration time: 18.66864323616028
Min iteration time: 15.951467514038086
Stage 2 results:
Total time: 1324.5760805606842
Average iteration time: 264.91476254463197
Max iteration time: 278.47786712646484
Min iteration time: 248.30649852752686
Stage 3 results:
Actor creation time: 3.133239269256592
Total time: 2001.6567075252533