wassname/ray
1
0
Fork 0
mirror of https://github.com/wassname/ray.git synced 2026-07-16 11:21:10 +08:00
Code Issues Packages Projects Releases Wiki Activity

[Release] Add 1.1.0 release test logs (#13054)

Browse Source 
* Add microbenchmark to release logs

* check in many_tasks stress test result

* Add results of placement group stress test for 1.1.0

* Add result for test_dead_actors test and correct the name of test_many_tasks.txt

* Add rllib regression test result

* Add pytorch test results for rllib

* remove extraneous log entries
This commit is contained in:
Max Fitton
2021-01-06 11:03:16 -08:00
committed by GitHub
parent 01d4638b49
commit 0d61ea9b06
6 changed files with 78 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
release/release_logs/1.1.0/microbenchmark.txt
+22
View File
@@ -0,0 +1,22 @@
single client get calls per second 46362.64 +- 176.71
single client put calls per second 35583.29 +- 698.24
multi client put calls per second 169525.0 +- 2121.99
single client get calls (Plasma Store) per second 9459.96 +- 182.31
single client put calls (Plasma Store) per second 5694.77 +- 111.86
multi client put calls (Plasma Store) per second 9544.79 +- 347.95
single client put gigabytes per second 8.61 +- 8.12
multi client put gigabytes per second 2.8 +- 0.58
single client tasks sync per second 1668.7 +- 64.61
single client tasks async per second 14355.15 +- 501.59
multi client tasks async per second 46795.34 +- 2017.89
1:1 actor calls sync per second 2686.96 +- 118.0
1:1 actor calls async per second 7997.53 +- 264.71
1:1 actor calls concurrent per second 7066.6 +- 289.54
1:n actor calls async per second 13856.64 +- 841.56
n:n actor calls async per second 48892.03 +- 290.8
n:n actor calls with arg async per second 13447.94 +- 66.48
1:1 async-actor calls sync per second 1876.75 +- 67.75
1:1 async-actor calls async per second 4028.29 +- 149.49
1:1 async-actor calls with args async per second 2833.48 +- 118.95
1:n async-actor calls async per second 14861.94 +- 1054.37
n:n async-actor calls async per second 39168.35 +- 870.17
release/release_logs/1.1.0/rllib_regression_tf.txt
+16
View File
@@ -0,0 +1,16 @@
+-------------------------------------------+------------+-------+--------+------------------+---------+----------+----------------------+----------------------+--------------------+
| Trial name | status | loc | iter | total time (s) | ts | reward | episode_reward_max | episode_reward_min | episode_len_mean |
|-------------------------------------------+------------+-------+--------+------------------+---------+----------+----------------------+----------------------+--------------------|
| A2C_BreakoutNoFrameskip-v4_ed940_00000 | TERMINATED | | 354 | 3601.99 | 3929000 | 142.56 | 330 | 24 | 5552.71 |
| A2C_BreakoutNoFrameskip-v4_ed940_00001 | TERMINATED | | 354 | 3609.69 | 3975000 | 164.16 | 396 | 26 | 5796.55 |
| APEX_BreakoutNoFrameskip-v4_ed940_00002 | TERMINATED | | 92 | 3616.25 | 6282880 | 12.33 | 37 | 4 | 1924.38 |
| APEX_BreakoutNoFrameskip-v4_ed940_00003 | TERMINATED | | 92 | 3600.6 | 6269280 | 13.9 | 30 | 0 | 2131.13 |
| DQN_BreakoutNoFrameskip-v4_ed940_00004 | TERMINATED | | 30 | 3648.1 | 310000 | 16.68 | 39 | 4 | 2077.06 |
| DQN_BreakoutNoFrameskip-v4_ed940_00005 | TERMINATED | | 30 | 3633.36 | 310000 | 17.14 | 35 | 4 | 2253.31 |
| IMPALA_BreakoutNoFrameskip-v4_ed940_00006 | TERMINATED | | 349 | 3609.73 | 2854500 | 140.22 | 345 | 23 | 5830.45 |
| IMPALA_BreakoutNoFrameskip-v4_ed940_00007 | TERMINATED | | 350 | 3609.66 | 2911500 | 113.99 | 327 | 28 | 5303.02 |
| PPO_BreakoutNoFrameskip-v4_ed940_00008 | TERMINATED | | 1311 | 3601.25 | 6555000 | 73.95 | 309 | 10 | 4028.25 |
| PPO_BreakoutNoFrameskip-v4_ed940_00009 | TERMINATED | | 1308 | 3601.78 | 6540000 | 38.9 | 243 | 9 | 2943.55 |
| SAC_HalfCheetahBulletEnv-v0_ed940_00010 | TERMINATED | | 80 | 3609.46 | 89000 | 590.35 | 693.03 | 514.213 | 1000 |
| SAC_HalfCheetahBulletEnv-v0_ed940_00011 | TERMINATED | | 81 | 3629.88 | 90000 | 699.174 | 724.238 | 676.54 | 1000 |
+-------------------------------------------+------------+-------+--------+------------------+---------+----------+----------------------+----------------------+--------------------+
release/release_logs/1.1.0/rllib_regression_torch.txt
+16
View File
@@ -0,0 +1,16 @@
+-------------------------------------------+------------+-------+--------+------------------+---------+----------+----------------------+----------------------+--------------------+
| Trial name | status | loc | iter | total time (s) | ts | reward | episode_reward_max | episode_reward_min | episode_len_mean |
|-------------------------------------------+------------+-------+--------+------------------+---------+----------+----------------------+----------------------+--------------------|
| A2C_BreakoutNoFrameskip-v4_dba56_00000 | TERMINATED | | 352 | 3605.57 | 2314000 | 17.75 | 37 | 5 | 2581.43 |
| A2C_BreakoutNoFrameskip-v4_dba56_00001 | TERMINATED | | 348 | 3604.47 | 2380500 | 16.59 | 42 | 5 | 2399.54 |
| APEX_BreakoutNoFrameskip-v4_dba56_00002 | TERMINATED | | 84 | 3610.08 | 4221440 | 12.55 | 27 | 5 | 1945.59 |
| APEX_BreakoutNoFrameskip-v4_dba56_00003 | TERMINATED | | 84 | 3617.05 | 4216160 | 12.89 | 31 | 3 | 2047.31 |
| DQN_BreakoutNoFrameskip-v4_dba56_00004 | TERMINATED | | 23 | 3682.82 | 240000 | 15.08 | 33 | 5 | 2024.58 |
| DQN_BreakoutNoFrameskip-v4_dba56_00005 | TERMINATED | | 22 | 3604.6 | 230000 | 11.38 | 42 | 3 | 1723.24 |
| IMPALA_BreakoutNoFrameskip-v4_dba56_00006 | TERMINATED | | 358 | 3602.74 | 6764500 | 322.21 | 424 | 25 | 7547.53 |
| IMPALA_BreakoutNoFrameskip-v4_dba56_00007 | TERMINATED | | 358 | 3608.42 | 6601500 | 339.23 | 431 | 21 | 8027.7 |
| PPO_BreakoutNoFrameskip-v4_dba56_00008 | TERMINATED | | 785 | 3603.32 | 3925000 | 40.95 | 381 | 5 | 3177.28 |
| PPO_BreakoutNoFrameskip-v4_dba56_00009 | TERMINATED | | 786 | 3601.17 | 3930000 | 71.25 | 394 | 10 | 2852.6 |
| SAC_HalfCheetahBulletEnv-v0_dba56_00010 | TERMINATED | | 62 | 3606.85 | 71000 | 743.684 | 920.212 | 524.94 | 1000 |
| SAC_HalfCheetahBulletEnv-v0_dba56_00011 | TERMINATED | | 63 | 3643.78 | 72000 | 598.945 | 646.041 | 531.009 | 1000 |
+-------------------------------------------+------------+-------+--------+------------------+---------+----------+----------------------+----------------------+--------------------+
release/release_logs/1.1.0/stress_tests/test_dead_actors.txt
+4
View File
@@ -0,0 +1,4 @@
Finished in: 101.62001442909241s
Average iteration time: 1.016197898387909s
Max iteration time: 3.120382308959961s
Min iteration time: 0.14046645164489746s
release/release_logs/1.1.0/stress_tests/test_many_tasks.txt
+17
View File
@@ -0,0 +1,17 @@
Stage 0 results:
Total time: 56.82723140716553
Stage 1 results:
Total time: 210.9983310699463
Average iteration time: 21.099819946289063
Max iteration time: 41.857728004455566
Min iteration time: 17.929903984069824
Stage 2 results:
Total time: 270.56019592285156
Average iteration time: 54.11184158325195
Max iteration time: 55.14220356941223
Min iteration time: 51.832639932632446
Stage 3 results:
Actor creation time: 0.4038546085357666
Total time: 2379.7233917713165
Stage 4 results:
Scheduling spread: 91.36037818586006.
release/release_logs/1.1.0/stress_tests/test_placement_group_results.txt
+3
View File
@@ -0,0 +1,3 @@
Avg placement group creating time: 0.28472538888888843 ms
Avg placement group removing time: 0.6085673033033048 ms
Stress Test succeed.