wassname/ray
1
0
Fork 0
mirror of https://github.com/wassname/ray.git synced 2026-08-12 12:20:11 +08:00
Code Issues Packages Projects Releases Wiki Activity

Fix release directory & RELEASE_PROCESS.md (#12269)

Browse Source
This commit is contained in:
Edward Oakes
2020-11-23 14:28:59 -06:00
committed by GitHub
parent 40428c9b05
commit 32d159a2ed
61 changed files with 217 additions and 304 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/dev/RELEASE_PROCESS.rst
-218
View File
@@ -1,218 +0,0 @@
Release Process
===============
This document describes the process for creating new releases.
1. **Create a release branch:** Create the branch from the desired commit on master
In order to create the branch, locally checkout the commit ID i.e.,
``git checkout <hash>``. Then checkout a new branch of the format
``releases/<release-version>``. Then push that branch to the ray repo:
``git push upstream releases/<release-version>``.
2. **Update the release branch version:** Push a commit directly to the
newly-created release branch that increments the Python package version in
python/ray/__init__.py and src/ray/raylet/main.cc. See this
`sample commit for bumping the release branch version`_.
3. **Update the master branch version:**
For a new minor release (e.g., 0.7.0): Create a pull request to
increment the dev version in of the master branch. See this
`sample PR for bumping a minor release version`_. **NOTE:** Not all of
the version numbers should be replaced. For example, ``0.7.0`` appears in
this file but should not be updated.
For a new micro release (e.g., 0.7.1): No action is required.
4. **Testing:** Before releasing, the following sets of tests should be run.
The results of each of these tests for previous releases are checked in
under ``doc/dev/release_logs``, and should be compared against to identify
any regressions.
1. Long-running tests
.. code-block:: bash
ray/ci/long_running_tests/README.rst
Follow the instructions to kick off the tests and check the status of the workloads.
These tests should run for at least 24 hours without erroring or hanging (ensure that it is printing new iterations and CPU load is
stable in the AWS console).
2. Long-running multi-node tests
.. code-block:: bash
ray/ci/long_running_distributed_tests/README.rst
Follow the instructions to kick off the tests and check the status of the workloads.
These suite of tests are similar to the standard long running tests, except these actually run in a multi-node cluster instead of just a simulated one.
These tests should also run for at least 24 hours without erroring or hanging.
**IMPORTANT**: check that the test are actually running (printing output regularly) and aren't
just stuck at an iteration. You must also check that the node CPU usage is stable
(and not increasing or decreasing over time, which indicates a leak). You can see the head node
and worker node CPU utilizations in the AWS console.
3. Multi-node regression tests
Follow the same instruction as long running stress tests. The large scale distributed
regression tests identify potential performance regression in distributed environment.
The following test should be ran:
- ``ci/regression_test/rllib_regression-tests`` run the compact regression test for rllib.
- ``ci/regression_test/rllib_stress_tests`` run multinode 8hr IMPALA trial.
- ``ci/regression_test/stress_tests`` contains two tests: ``many_tasks`` and ``dead_actors``.
Each of the test runs on 105 spot instances.
- ``ci/regression_test/stress_tests/placement_group`` contains a Python script to run tests.
It currently uses ``cluster_util`` to emulate the cluster testing. It will be converted to
real multi-node tests in the future. For now, just make sure the test succeed locally.
Make sure that these pass. For the RLlib regression tests, there shouldn't be any errors
and the rewards should be similar to previous releases. For the rest, it will be obvious if
they passed. This will use the autoscaler to start a bunch of machines and run some tests.
**IMPORTANT**: You must get signoff from the RLlib team for the RLlib test results.
The summaries printed by each test should be checked in under
``doc/dev/release_logs/<version>`` on the **master** branch (make a pull request).
4. Microbenchmarks
Run the ``ci/microbenchmark`` with the commit. Under the hood, the session will
run `ray microbenchmark` on an `m4.16xl` instance running `Ubuntu 18.04` with `Python 3`
to get the latest microbenchmark numbers.
The results should be checked in under ``doc/dev/release_logs/<version>``.
You can also get the performance change rate from the previous version using
microbenchmark_analysis.py
5. ASAN tests
Run the ``ci/asan_tests`` with the commit. This will enable ASAN build and run the
whole Python tests to detect memory leaks.
5. **Resolve release-blockers:** If a release blocking issue arises, there are
two ways the issue can be resolved: 1) Fix the issue on the master branch and
cherry-pick the relevant commit (using ``git cherry-pick``) onto the release
branch (recommended). 2) Revert the commit that introduced the bug on the
release branch (using ``git revert``), but not on the master (not recommended).
These changes should then be pushed directly to the release branch.
6. **Create a GitHub release:** Create a `GitHub release`_. This should include
**release notes**. Copy the style and formatting used by previous releases.
Create a draft of the release notes containing information about substantial
changes/updates/bugfixes and their PR numbers. Once you have a draft, send it
out to other Ray developers (especially those who contributed heavily during
this release) for feedback. At the end of the release note, you should also
add a list of contributors. Make sure Ray, Tune, RLLib, Autoscaler are
capitalized correctly.
Run ``doc/dev/get_contributors.py`` to generate the list of commits corresponding
to this release and the formatted list of contributors.
You will need to provide a GitHub personal access token
(github.com -> settings -> developer settings -> personal access tokens).
.. code-block:: bash
# Must be run from inside the Ray repository.
pip install PyGitHub tqdm
python get_contributors.py --help
python get_contributors.py \
--access-token=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx \
--prev-release-commit="<COMMIT_SHA>" \
--curr-release-commit="<COMMIT_SHA>"
7. **Download all the wheels:** Now the release is ready to begin final
testing. The wheels are automatically uploaded to S3, even on the release
branch. To test, ``pip install`` from the following URLs:
.. code-block:: bash
export RAY_HASH=... # e.g., 618147f57fb40368448da3b2fb4fd213828fa12b
export RAY_VERSION=... # e.g., 0.7.0
# Linux Wheels
pip install -U https://s3-us-west-2.amazonaws.com/ray-wheels/releases/$RAY_VERSION/$RAY_HASH/ray-$RAY_VERSION-cp36-cp36m-manylinux2014_x86_64.whl
pip install -U https://s3-us-west-2.amazonaws.com/ray-wheels/releases/$RAY_VERSION/$RAY_HASH/ray-$RAY_VERSION-cp37-cp37m-manylinux2014_x86_64.whl
pip install -U https://s3-us-west-2.amazonaws.com/ray-wheels/releases/$RAY_VERSION/$RAY_HASH/ray-$RAY_VERSION-cp38-cp38-manylinux2014_x86_64.whl
# Mac Wheels
pip install -U https://s3-us-west-2.amazonaws.com/ray-wheels/releases/$RAY_VERSION/$RAY_HASH/ray-$RAY_VERSION-cp36-cp36m-macosx_10_13_intel.whl
pip install -U https://s3-us-west-2.amazonaws.com/ray-wheels/releases/$RAY_VERSION/$RAY_HASH/ray-$RAY_VERSION-cp37-cp37m-macosx_10_13_intel.whl
pip install -U https://s3-us-west-2.amazonaws.com/ray-wheels/releases/$RAY_VERSION/$RAY_HASH/ray-$RAY_VERSION-cp38-cp38-macosx_10_13_x86_64.whl
This can be tested if you use the script source ./bin/download_wheels.sh
8. **Upload to PyPI Test:** Upload the wheels to the PyPI test site using
``twine``.
.. code-block:: bash
# Downloads all of the wheels to the current directory.
RAY_VERSION=<version> RAY_HASH=<commit_sha> bash download_wheels.sh
# Will ask for your PyPI test credentials and require that you're a maintainer
# on PyPI test. If you are not, ask @robertnishihara to add you.
pip install twine
twine upload --repository-url https://test.pypi.org/legacy/ *.whl
Test that you can install the wheels with pip from the PyPI test repository:
.. code-block:: bash
# First install ray normally because installing from test.pypi.org won't
# be able to install some of the other dependencies.
pip install ray
pip uninstall ray
pip install --index-url https://test.pypi.org/simple/ ray
Then start Python, make sure you can ``import ray`` and run some simple Ray
scripts. Make sure that it is finding the version of Ray that you just
installed by checking ``ray.__version__`` and ``ray.__file__``.
Do this for MacOS, Linux, and Windows.
This process is automated. Run ./bin/pip_download_test.sh.
This will download the ray from the test pypi repository and run the minimum
sanity check from all the Python version supported. (3.6, 3.7, 3.8)
Windows sanity check test is currently not automated.
9. **Upload to PyPI:** Now that you've tested the wheels on the PyPI test
repository, they can be uploaded to the main PyPI repository. Be careful,
**it will not be possible to modify wheels once you upload them**, so any
mistake will require a new release.
.. code-block:: bash
# Will ask for your real PyPI credentials and require that you're a maintainer
# on real PyPI. If you are not, ask @robertnishihara to add you.
twine upload --repository-url https://upload.pypi.org/legacy/ *.whl
Now, try installing from the real PyPI mirror. Verify that the correct version is
installed and that you can run some simple scripts.
.. code-block:: bash
pip install -U ray
10. **Create a point release on readthedocs page:** Go to the `Ray Readthedocs version page`_.
Scroll to "Activate a version" and mark the *release branch* as "active" and "public". This creates a point release for the documentation.
Message @richardliaw to add you if you don't have access.
11. **Update 'Default Branch' on the readthedocs page:** Go to the `Ray Readthedocs advanced settings page`_.
In 'Global Settings', set the 'Default Branch' to the *release branch*. This redirects the documentation to the latest pip release.
Message @richardliaw to add you if you don't have access.
12. **Improve the release process:** Find some way to improve the release
process so that whoever manages the release next will have an easier time.
.. _`sample PR for bumping a minor release version`: https://github.com/ray-project/ray/pull/6303
.. _`sample commit for bumping the release branch version`: https://github.com/ray-project/ray/commit/a39325d818339970e51677708d5596f4b8f790ce
.. _`GitHub release`: https://github.com/ray-project/ray/releases
.. _`Ray Readthedocs version page`: https://readthedocs.org/projects/ray/versions/
.. _`Ray Readthedocs advanced settings page`: https://readthedocs.org/dashboard/ray/advanced/
doc/dev/bin/pip_download_test.sh
-75
View File
@@ -1,75 +0,0 @@
#!/usr/bin/env bash
# This script automatically download ray and run the sanity check (sanity_check.py)
# in various Python version. This script requires conda command to exist.
unset RAY_ADDRESS
export RAY_HASH=$RAY_HASH
export RAY_VERSION=$RAY_VERSION
if [[ -z "$RAY_HASH" ]]; then
echo "RAY_HASH env var should be provided"
exit 1
fi
if [[ -z "$RAY_VERSION" ]]; then
echo "RAY_VERSION env var should be provided"
exit 1
fi
if ! [ -x "$(command -v conda)" ]; then
echo "conda doesn't exist. Please download conda for this machine"
exit 1
else
echo "conda exists"
fi
echo "Start downloading Ray version ${RAY_VERSION} of commit ${RAY_HASH}"
pip install --upgrade pip
# This is required to use conda activate
source "$(conda info --base)/etc/profile.d/conda.sh"
for PYTHON_VERSION in "3.6" "3.7" "3.8"
do
env_name="${RAY_VERSION}-${PYTHON_VERSION}-env"
conda create -y -n "${env_name}" python=${PYTHON_VERSION}
conda activate "${env_name}"
printf "\n\n\n"
echo "========================================================="
echo "Python version."
python --version
echo "This should be equal to ${PYTHON_VERSION}"
echo "========================================================="
printf "\n\n\n"
pip install redis==3.3.2
pip install msgpack==1.0.0
pip install aioredis
pip install colorful
pip install prometheus-client==0.7.1
pip install opencensus
pip install gpustat
pip install ray
pip uninstall -y ray
pip install --index-url https://test.pypi.org/simple/ ray
failed=false
printf "\n\n\n"
echo "========================================================="
if python sanity_check.py; then
echo "PYTHON ${PYTHON_VERSION} succeed sanity check."
else
failed=true
fi
echo "========================================================="
printf "\n\n\n"
conda deactivate
conda remove -y --name "${env_name}" --all
if [ "$failed" = true ]; then
echo "PYTHON ${PYTHON_VERSION} failed sanity check."
exit 1
fi
done
doc/dev/bin/ray-project/cluster.yaml
-38
View File
@@ -1,38 +0,0 @@
# This file is generated by `ray project create`.
# A unique identifier for the head node and workers of this cluster.
cluster_name: sanity-check
# The maximum number of workers nodes to launch in addition to the head
# node. This takes precedence over min_workers. min_workers defaults to 0.
max_workers: 0
# Cloud-provider specific configuration.
provider:
type: aws
region: us-west-2
availability_zone: us-west-2a
# How Ray will authenticate with newly launched nodes.
auth:
ssh_user: ubuntu
# Custom commands that will be run on the head node after common setup.
head_setup_commands:
# Install Anaconda.
- wget --quiet https://repo.continuum.io/archive/Anaconda3-5.0.1-Linux-x86_64.sh || true
- bash Anaconda3-5.0.1-Linux-x86_64.sh -b -p $HOME/anaconda3 || true
- echo 'export PATH="$HOME/anaconda3/bin:$PATH"' >> ~/.bashrc
- echo 'RAY_VERSION=0.8.7' >> ~/.bashrc
- echo 'RAY_HASH=56810e136587432ed4758d882bf8894b930355e3' >> ~/.bashrc
- chmod +x ./pip_download_test.sh || true
- pip install -U pip
# Custom commands that will be run on worker nodes after common setup.
worker_setup_commands: []
# Command to start ray on the head node. You don't need to change this.
head_start_ray_commands: [ray stop]
# Command to start ray on worker nodes. You don't need to change this.
worker_start_ray_commands: [ray stop]
doc/dev/bin/ray-project/project.yaml
-36
View File
@@ -1,36 +0,0 @@
# This file is generated by `ray project create`.
name: sanity-check
# description: A short description of the project.
# The URL of the repo this project is part of.
# repo: ...
cluster:
config: ray-project/cluster.yaml
commands:
- name: sanity-check
help: "Do release sanity check with pip"
command: |
export RAY_VERSION={{ray_version}}
export RAY_HASH={{commit}}
chmod +x ./pip_download_test.sh
./pip_download_test.sh
params:
- name: ray_version # Ray version string.
default: "1.1.0.dev0"
- name: commit # Ray commit SHA string.
default: "FILL ME IN"
# Pathnames for files and directories that should be saved
# in a snapshot but that should not be synced with a# session. Pathnames can be relative to the project
# directory or absolute. Generally, this should be files
# that were created by an active session, such as
# application checkpoints and logs.
output_files: [
# For example, uncomment this to save the logs from the
# last ray job.
# "/tmp/ray/session_latest",
]
doc/dev/bin/sanity_check.py
-31
View File
@@ -1,31 +0,0 @@
import os
import ray
import sys
RAY_VERSION = "RAY_VERSION"
RAY_COMMIT = "RAY_HASH"
ray_version = os.getenv(RAY_VERSION)
ray_commit = os.getenv(RAY_COMMIT)
if __name__ == "__main__":
print("Sanity check python version: {}".format(sys.version))
assert ray_version == ray.__version__, (
"Given Ray version {} is not matching with downloaded "
"version {}".format(ray_version, ray.__version__))
assert ray_commit == ray.__commit__, (
"Given Ray commit {} is not matching with downloaded "
"version {}".format(ray_commit, ray.__commit__))
assert ray.__file__ is not None
ray.init()
assert ray.is_initialized()
@ray.remote
def return_arg(arg):
return arg
val = 3
print("Running basic sanity check.")
assert ray.get(return_arg.remote(val)) == val
ray.shutdown()
doc/dev/download_wheels.sh
-25
View File
@@ -1,25 +0,0 @@
#!/usr/bin/env bash
if [[ -z "$RAY_HASH" ]]; then
echo "RAY_HASH env var should be provided"
exit 1
fi
if [[ -z "$RAY_VERSION" ]]; then
echo "RAY_VERSION env var should be provided"
exit 1
fi
# Make sure Linux wheels are downloadable without errors.
wget "https://s3-us-west-2.amazonaws.com/ray-wheels/releases/$RAY_VERSION/$RAY_HASH/ray-$RAY_VERSION-cp36-cp36m-manylinux2014_x86_64.whl"
wget "https://s3-us-west-2.amazonaws.com/ray-wheels/releases/$RAY_VERSION/$RAY_HASH/ray-$RAY_VERSION-cp37-cp37m-manylinux2014_x86_64.whl"
wget "https://s3-us-west-2.amazonaws.com/ray-wheels/releases/$RAY_VERSION/$RAY_HASH/ray-$RAY_VERSION-cp38-cp38-manylinux2014_x86_64.whl"
# Make sure macos wheels are downloadable without errors.
wget "https://s3-us-west-2.amazonaws.com/ray-wheels/releases/$RAY_VERSION/$RAY_HASH/ray-$RAY_VERSION-cp36-cp36m-macosx_10_13_intel.whl"
wget "https://s3-us-west-2.amazonaws.com/ray-wheels/releases/$RAY_VERSION/$RAY_HASH/ray-$RAY_VERSION-cp37-cp37m-macosx_10_13_intel.whl"
# Wheel name convention has been changed from Python 3.8.
wget "https://s3-us-west-2.amazonaws.com/ray-wheels/releases/$RAY_VERSION/$RAY_HASH/ray-$RAY_VERSION-cp38-cp38-macosx_10_13_x86_64.whl"
# Make sure Windows wheels are downloadable without errors.
wget "https://ray-wheels.s3-us-west-2.amazonaws.com/releases/$RAY_VERSION/$RAY_HASH/ray-$RAY_VERSION-cp36-cp36m-win_amd64.whl"
wget "https://ray-wheels.s3-us-west-2.amazonaws.com/releases/$RAY_VERSION/$RAY_HASH/ray-$RAY_VERSION-cp37-cp37m-win_amd64.whl"
wget "https://ray-wheels.s3-us-west-2.amazonaws.com/releases/$RAY_VERSION/$RAY_HASH/ray-$RAY_VERSION-cp38-cp38-win_amd64.whl"
doc/dev/get_contributors.py
-90
View File
@@ -1,90 +0,0 @@
from github import Github
from subprocess import check_output
import shlex
from tqdm import tqdm
import click
from collections import defaultdict
@click.command()
@click.option(
"--access-token",
required=True,
help="""
Github Access token that has repo:public_repo and user:read:user permission.
Create them at https://github.com/settings/tokens/new
""",
)
@click.option(
"--prev-release-commit",
required=True,
help="Last commit SHA of the previous release.")
@click.option(
"--curr-release-commit",
required=True,
help="Last commit SHA of the current release.")
def run(access_token, prev_release_commit, curr_release_commit):
print("Writing commit descriptions to 'commits.txt'...")
check_output(
(f"git log {prev_release_commit}..{curr_release_commit} "
f"--pretty=format:'%s' > commits.txt"),
shell=True)
# Generate command
cmd = []
cmd.append((f"git log {prev_release_commit}..{curr_release_commit} "
f"--pretty=format:\"%s\" "
f" | grep -Eo \"#(\d+)\""))
joined = " && ".join(cmd)
cmd = f"bash -c '{joined}'"
cmd = shlex.split(cmd)
print("Executing", cmd)
# Sort the PR numbers
pr_numbers = [
int(l.lstrip("#")) for l in check_output(cmd).decode().split()
]
print("PR numbers", pr_numbers)
# Use Github API to fetch the
g = Github(access_token)
ray_repo = g.get_repo("ray-project/ray")
logins = set()
for num in tqdm(pr_numbers):
try:
logins.add(ray_repo.get_pull(num).user.login)
except Exception as e:
print(e)
print()
print("Here's the list of contributors")
print("=" * 10)
print()
print("@" + ", @".join(logins))
print()
print("=" * 10)
# Organize commits
NO_CATEGORY = "[NO_CATEGORY]"
def get_category(line):
if line[0] == "[":
return (line.split("]")[0].strip(" ") + "]").upper()
else:
return NO_CATEGORY
commits = defaultdict(list)
with open("commits.txt") as file:
for line in file.readlines():
commits[get_category(line)].append(line.strip())
with open("commits.txt", "a") as file:
for category, commit_msgs in commits.items():
file.write("\n{}\n".format(category))
for commit_msg in commit_msgs:
file.write("{}\n".format(commit_msg))
if __name__ == "__main__":
run()
doc/dev/microbenchmark_analysis.py
-64
View File
@@ -1,64 +0,0 @@
"""
Compare the last two versions and output the change rate.
This can also be used to draw graph, which is not
implemented yet.
Usage: python microbenchmark_analysis.py
"""
import glob
from collections import defaultdict
FRIST_VERSION = 0
LAST_VERSION = 5
FILES = sorted(
glob.glob("./release_logs/[0-9].[0-9].[0-9]/microbenchmark.txt"))
task_info = defaultdict(list)
task_std_info = defaultdict(list)
version_list = []
def get_task_type(line):
return line.split("per")[0]
def get_task_performance(line):
return float(line.split(" ")[-3])
def get_task_std(line):
return float(line.split(" ")[-1])
def main():
for file_name in FILES:
version = file_name.split("/")[1]
version_list.append(version)
with open(file_name) as file:
for line in file.readlines():
if line.startswith("#") or line.startswith("\n"):
continue
line = line.strip()
task_type = get_task_type(line)
task_performance = get_task_performance(line)
task_standard_deviation = get_task_std(line)
task_info[task_type].append(task_performance)
task_std_info[task_type].append(task_standard_deviation)
for task_type, task_performance_list in task_info.items():
# Newly introduced fields are not going to be compared.
if len(task_performance_list) < 2:
continue
latest_perf = task_performance_list[-1]
second_latest_perf = task_performance_list[-2]
change_rate = (
latest_perf - second_latest_perf) / second_latest_perf * 100
print("{} performance change rate: {}%".format(task_type,
round(change_rate, 2)))
if __name__ == "__main__":
main()
doc/dev/release_logs/0.7.7/microbenchmark.txt
-14
View File
@@ -1,14 +0,0 @@
single client get calls (Plasma Store) per second 28595.02 +- 580.33
single client put calls (Plasma Store) per second 6313.62 +- 66.88
single client put gigabytes per second 11.6 +- 6.86
multi client put calls (Plasma Store) per second 16800.89 +- 381.69
multi client put gigabytes per second 23.33 +- 0.96
single client tasks sync per second 1963.72 +- 48.48
single client tasks async per second 5181.29 +- 30.0
multi client tasks async per second 5566.7 +- 280.72
1:1 actor calls sync per second 1595.47 +- 38.32
1:1 actor calls async per second 2496.26 +- 37.62
1:1 direct actor calls async per second 7233.63 +- 205.75
n:n actor calls async per second 5357.63 +- 116.9
n:n direct actor calls async per second 90703.32 +- 805.56
n:n direct actor calls with arg async per second 13300.47 +- 532.66
doc/dev/release_logs/0.7.7/rllib_regression.txt
-24
View File
@@ -1,24 +0,0 @@
+----------------------------------------+------------+-------+--------+------------------+-------------+----------+
| Trial name | status | loc | iter | total time (s) | timesteps | reward |
|----------------------------------------+------------+-------+--------+------------------+-------------+----------|
| IMPALA_BreakoutNoFrameskip-v4_4445a400 | TERMINATED | | 294 | 3600.28 | 6183500 | 352.15 |
| IMPALA_BreakoutNoFrameskip-v4_44461b92 | TERMINATED | | 292 | 3610.1 | 6178500 | 147.14 |
| IMPALA_BreakoutNoFrameskip-v4_44467b28 | TERMINATED | | 293 | 3611.05 | 6200500 | 275.1 |
| IMPALA_BreakoutNoFrameskip-v4_4446e1bc | TERMINATED | | 294 | 3611.16 | 6161500 | 322.95 |
| PPO_BreakoutNoFrameskip-v4_44474a4e | TERMINATED | | 549 | 3603.09 | 2745000 | 36.41 |
| PPO_BreakoutNoFrameskip-v4_4447b740 | TERMINATED | | 545 | 3604.43 | 2725000 | 16.83 |
| PPO_BreakoutNoFrameskip-v4_44481ea6 | TERMINATED | | 549 | 3605.4 | 2745000 | 52.36 |
| PPO_BreakoutNoFrameskip-v4_444885a8 | TERMINATED | | 546 | 3605.27 | 2730000 | 34.94 |
| APEX_BreakoutNoFrameskip-v4_4448feac | TERMINATED | | 113 | 3629.93 | 3552960 | 13.47 |
| APEX_BreakoutNoFrameskip-v4_44497684 | TERMINATED | | 112 | 3615.5 | 3539360 | 28.65 |
| APEX_BreakoutNoFrameskip-v4_4449f01e | TERMINATED | | 113 | 3621.85 | 3524640 | 28.17 |
| APEX_BreakoutNoFrameskip-v4_444a5dd8 | TERMINATED | | 112 | 3616.41 | 3447520 | 20.56 |
| A2C_BreakoutNoFrameskip-v4_444acb7e | TERMINATED | | 349 | 3604.38 | 2981000 | 108.8 |
| A2C_BreakoutNoFrameskip-v4_444b3a50 | TERMINATED | | 349 | 3602.3 | 2967500 | 145.82 |
| A2C_BreakoutNoFrameskip-v4_444b9ebe | TERMINATED | | 349 | 3605.71 | 2990500 | 127.3 |
| A2C_BreakoutNoFrameskip-v4_444c0016 | TERMINATED | | 349 | 3602.58 | 2970000 | 112.92 |
| DQN_BreakoutNoFrameskip-v4_444c6830 | TERMINATED | | 30 | 3662.81 | 300000 | 18.71 |
| DQN_BreakoutNoFrameskip-v4_444cdeb4 | TERMINATED | | 30 | 3608.95 | 300000 | 17.45 |
| DQN_BreakoutNoFrameskip-v4_444d5038 | TERMINATED | | 31 | 3658.65 | 310000 | 17.43 |
| DQN_BreakoutNoFrameskip-v4_444dbfb4 | TERMINATED | | 31 | 3696.8 | 310000 | 18.63 |
+----------------------------------------+------------+-------+--------+------------------+-------------+----------+
doc/dev/release_logs/0.7.7/stress_tests/test_dead_actors.txt
-4
View File
@@ -1,4 +0,0 @@
Finished in: 361.19257402420044s
Average iteration time: 3.6119208979606627s
Max iteration time: 6.086421489715576s
Min iteration time: 0.23141980171203613s
doc/dev/release_logs/0.7.7/stress_tests/test_many_tasks.txt
-13
View File
@@ -1,13 +0,0 @@
Stage 1 results:
Total time: 562.7756896018982
Average iteration time: 56.277525544166565
Max iteration time: 59.43245506286621
Min iteration time: 46.18883228302002
Stage 2 results:
Total time: 650.9956197738647
Average iteration time: 130.19871163368225
Max iteration time: 134.73312878608704
Min iteration time: 126.5669572353363
Stage 3 results:
Actor creation time: 0.12409853935241699
Total time: 3345.6912751197815
doc/dev/release_logs/0.8.0/microbenchmark.txt
-14
View File
@@ -1,14 +0,0 @@
single client get calls (Plasma Store) per second 30526.52 +- 1121.22
single client put calls (Plasma Store) per second 6428.11 +- 51.25
single client put gigabytes per second 12.85 +- 5.45
multi client put calls (Plasma Store) per second 16976.5 +- 60.13
multi client put gigabytes per second 22.54 +- 0.86
single client tasks sync per second 1223.63 +- 18.36
single client tasks async per second 13715.93 +- 229.68
multi client tasks async per second 38344.2 +- 4336.41
1:1 actor calls sync per second 2318.72 +- 24.19
1:1 actor calls async per second 7530.01 +- 222.72
1:1 actor calls concurrent per second 6981.48 +- 228.29
1:n actor calls async per second 13974.54 +- 41.26
n:n actor calls async per second 43835.4 +- 400.04
n:n actor calls with arg async per second 14775.69 +- 295.24
doc/dev/release_logs/0.8.0/rllib_regression.txt
-24
View File
@@ -1,24 +0,0 @@
+----------------------------------------+------------+--------------------+--------+------------------+-------------+----------+
| Trial name | status | loc | iter | total time (s) | timesteps | reward |
|----------------------------------------+------------+--------------------+--------+------------------+-------------+----------|
| IMPALA_BreakoutNoFrameskip-v4_e25c5ce8 | TERMINATED | | 310 | 3604.3 | 9648000 | 399.75 |
| IMPALA_BreakoutNoFrameskip-v4_e25cdfec | TERMINATED | | 310 | 3611.22 | 9552000 | 403.49 |
| IMPALA_BreakoutNoFrameskip-v4_e25d65de | TERMINATED | | 310 | 3604.05 | 9604500 | 390.47 |
| IMPALA_BreakoutNoFrameskip-v4_e25ddf0a | TERMINATED | | 310 | 3604.18 | 9447000 | 365.57 |
| PPO_BreakoutNoFrameskip-v4_e25e4ddc | TERMINATED | | 604 | 3605.24 | 3020000 | 42.59 |
| PPO_BreakoutNoFrameskip-v4_e25eb8da | TERMINATED | | 613 | 3603.35 | 3065000 | 50.58 |
| PPO_BreakoutNoFrameskip-v4_e25f4520 | TERMINATED | | 608 | 3600.01 | 3040000 | 51.2 |
| PPO_BreakoutNoFrameskip-v4_e25fd44a | TERMINATED | | 613 | 3604.97 | 3065000 | 26.19 |
| APEX_BreakoutNoFrameskip-v4_e2603fa2 | TERMINATED | | 115 | 3622.22 | 5169280 | 48.56 |
| APEX_BreakoutNoFrameskip-v4_e260d462 | TERMINATED | | 115 | 3616.47 | 5201280 | 45.01 |
| APEX_BreakoutNoFrameskip-v4_e2615a90 | TERMINATED | | 115 | 3615.15 | 5134080 | 25.02 |
| APEX_BreakoutNoFrameskip-v4_e261cbec | TERMINATED | | 115 | 3625.55 | 5189280 | 47.01 |
| A2C_BreakoutNoFrameskip-v4_e26260a2 | TERMINATED | | 351 | 3602.43 | 3478000 | 145.84 |
| A2C_BreakoutNoFrameskip-v4_e26331f8 | TERMINATED | | 351 | 3601.93 | 3577000 | 178.73 |
| A2C_BreakoutNoFrameskip-v4_e2639e68 | TERMINATED | | 351 | 3607.28 | 3521500 | 185.45 |
| A2C_BreakoutNoFrameskip-v4_e2642630 | TERMINATED | | 351 | 3606.35 | 3460000 | 122.32 |
| DQN_BreakoutNoFrameskip-v4_99c588f2 | TERMINATED | | 32 | 3643.26 | 320000 | 19.24 |
| DQN_BreakoutNoFrameskip-v4_99c62de8 | TERMINATED | | 31 | 3619.43 | 310000 | 19.97 |
| DQN_BreakoutNoFrameskip-v4_99c6c01e | TERMINATED | | 33 | 3680.95 | 330000 | 16.91 |
| DQN_BreakoutNoFrameskip-v4_99c73f44 | TERMINATED | | 31 | 3625.12 | 310000 | 16.23 |
+----------------------------------------+------------+--------------------+--------+------------------+-------------+----------+
doc/dev/release_logs/0.8.0/stress_tests/test_dead_actors.txt
-4
View File
@@ -1,4 +0,0 @@
Finished in: 220.28446006774902s
Average iteration time: 2.2028422021865843s
Max iteration time: 4.1706767082214355s
Min iteration time: 1.1767382621765137s
doc/dev/release_logs/0.8.0/stress_tests/test_many_tasks.txt
-15
View File
@@ -1,15 +0,0 @@
Stage 0 results:
Total time: 18.916658878326416
Stage 1 results:
Total time: 112.98085951805115
Average iteration time: 11.298071336746215
Max iteration time: 11.726166009902954
Min iteration time: 11.037099123001099
Stage 2 results:
Total time: 1492.7644929885864
Average iteration time: 298.5524812221527
Max iteration time: 334.69453477859497
Min iteration time: 233.40272068977356
Stage 3 results:
Actor creation time: 0.03908824920654297
Total time: 2432.014177799225
doc/dev/release_logs/0.8.1/microbenchmark.txt
-21
View File
@@ -1,21 +0,0 @@
# NOTE: performance decreases are related to enabling reference counting and
# pinning by default for objects passed using the ray API. 'single client get'
# results dropped because we are no longer keeping a buffer reference in object
# IDs for ray.put() objects. This shouldn't have a large impact on applications
# and will be fixed in the future by caching buffers after the first ray.get()
# on each object ID.
single client get calls (Plasma Store) per second 12550.57 +- 1835.19
single client put calls (Plasma Store) per second 6791.78 +- 176.65
single client put gigabytes per second 13.36 +- 5.7
multi client put calls (Plasma Store) per second 13503.59 +- 179.1
multi client put gigabytes per second 16.22 +- 1.36
single client tasks sync per second 1295.56 +- 42.81
single client tasks async per second 14825.7 +- 358.92
multi client tasks async per second 43699.93 +- 627.98
1:1 actor calls sync per second 2194.35 +- 57.18
1:1 actor calls async per second 6873.68 +- 87.3
1:1 actor calls concurrent per second 7285.91 +- 50.57
1:n actor calls async per second 13290.25 +- 140.2
n:n actor calls async per second 45354.88 +- 678.84
n:n actor calls with arg async per second 13668.97 +- 105.62
doc/dev/release_logs/0.8.1/rllib_regression.txt
-24
View File
@@ -1,24 +0,0 @@
+----------------------------------------+------------+-------+--------+------------------+-------------+----------+
| Trial name | status | loc | iter | total time (s) | timesteps | reward |
|----------------------------------------+------------+-------+--------+------------------+-------------+----------|
| IMPALA_BreakoutNoFrameskip-v4_84d6d8e6 | TERMINATED | | 297 | 3606.54 | 6304000 | 362.33 |
| IMPALA_BreakoutNoFrameskip-v4_84d75776 | TERMINATED | | 295 | 3605.31 | 6274500 | 361.79 |
| IMPALA_BreakoutNoFrameskip-v4_84d7c2f6 | TERMINATED | | 294 | 3606.04 | 6267500 | 298.03 |
| IMPALA_BreakoutNoFrameskip-v4_84d9bfa2 | TERMINATED | | 294 | 3609.24 | 6256500 | 358.66 |
| PPO_BreakoutNoFrameskip-v4_84dafff2 | TERMINATED | | 550 | 3605.93 | 2750000 | 34.19 |
| PPO_BreakoutNoFrameskip-v4_84dc19e6 | TERMINATED | | 546 | 3604.47 | 2730000 | 39.9 |
| PPO_BreakoutNoFrameskip-v4_84dc8be2 | TERMINATED | | 545 | 3603.96 | 2725000 | 18.28 |
| PPO_BreakoutNoFrameskip-v4_84dda946 | TERMINATED | | 549 | 3601.85 | 2745000 | 28.13 |
| APEX_BreakoutNoFrameskip-v4_84e092fa | TERMINATED | | 113 | 3622.29 | 3535840 | 18.75 |
| APEX_BreakoutNoFrameskip-v4_84e13336 | TERMINATED | | 113 | 3604.95 | 3556960 | 28.56 |
| APEX_BreakoutNoFrameskip-v4_84e44026 | TERMINATED | | 113 | 3608.22 | 3453440 | 12.88 |
| APEX_BreakoutNoFrameskip-v4_84e81cbe | TERMINATED | | 113 | 3600.83 | 3566880 | 32.35 |
| A2C_BreakoutNoFrameskip-v4_84e9aa48 | TERMINATED | | 350 | 3609.93 | 3094500 | 102.92 |
| A2C_BreakoutNoFrameskip-v4_84eb23fa | TERMINATED | | 350 | 3609.93 | 3000000 | 141.7 |
| A2C_BreakoutNoFrameskip-v4_84eba21c | TERMINATED | | 349 | 3601.24 | 3009500 | 87.86 |
| A2C_BreakoutNoFrameskip-v4_84eeb7c2 | TERMINATED | | 349 | 3600.57 | 3037500 | 83.63 |
| DQN_BreakoutNoFrameskip-v4_84efd378 | TERMINATED | | 32 | 3628.85 | 320000 | 11.45 |
| DQN_BreakoutNoFrameskip-v4_84f13bf0 | TERMINATED | | 31 | 3686.04 | 310000 | 10.33 |
| DQN_BreakoutNoFrameskip-v4_84f68b8c | TERMINATED | | 32 | 3686.78 | 320000 | 17.51 |
| DQN_BreakoutNoFrameskip-v4_84f7152a | TERMINATED | | 32 | 3601.57 | 320000 | 16.78 |
+----------------------------------------+------------+-------+--------+------------------+-------------+----------+
doc/dev/release_logs/0.8.1/stress_tests/application_stress_test.txt
-14
View File
@@ -1,14 +0,0 @@
== Status ==
Memory usage on this node: 37.0/480.3 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/640 CPUs, 0/8 GPUs, 0.0/2526.95 GiB heap, 0.0/128.42 GiB objects
Result logdir: /home/ubuntu/ray_results/atari-impala
Number of trials: 4 (4 TERMINATED)
+---------------------------------------------+------------+-------+-----------------------------+--------+------------------+-------------+----------+
| Trial name | status | loc | env | iter | total time (s) | timesteps | reward |
|---------------------------------------------+------------+-------+-----------------------------+--------+------------------+-------------+----------|
| IMPALA_BreakoutNoFrameskip-v4_511a1830 | TERMINATED | | BreakoutNoFrameskip-v4 | 39 | 699.024 | 3050000 | 44.21 |
| IMPALA_BeamRiderNoFrameskip-v4_511b6726 | TERMINATED | | BeamRiderNoFrameskip-v4 | 37 | 664.972 | 3083500 | 449.72 |
| IMPALA_QbertNoFrameskip-v4_511bd44a | TERMINATED | | QbertNoFrameskip-v4 | 40 | 707.975 | 3088000 | 495.464 |
| IMPALA_SpaceInvadersNoFrameskip-v4_511c4858 | TERMINATED | | SpaceInvadersNoFrameskip-v4 | 38 | 680.926 | 3004500 | 380.35 |
+---------------------------------------------+------------+-------+-----------------------------+--------+------------------+-------------+----------+
doc/dev/release_logs/0.8.1/stress_tests/test_dead_actors.txt
-4
View File
@@ -1,4 +0,0 @@
Finished in: 138.03558135032654s
Average iteration time: 1.3803531074523925s
Max iteration time: 2.4373393058776855s
Min iteration time: 0.20983195304870605s
doc/dev/release_logs/0.8.1/stress_tests/test_many_tasks.txt
-15
View File
@@ -1,15 +0,0 @@
Stage 0 results:
Total time: 20.65841770172119
Stage 1 results:
Total time: 109.76851415634155
Average iteration time: 10.976841711997986
Max iteration time: 11.290549039840698
Min iteration time: 10.660207271575928
Stage 2 results:
Total time: 476.79087471961975
Average iteration time: 95.35772457122803
Max iteration time: 98.76972222328186
Min iteration time: 90.59104466438293
Stage 3 results:
Actor creation time: 0.07228231430053711
Total time: 2383.4370431900024
doc/dev/release_logs/0.8.2/microbenchmark.txt
-18
View File
@@ -1,18 +0,0 @@
# NOTE: Make sure to run this with OMP_NUM_THREADS=64, otherwise the put gigabytes per
# seconds will be reduced. Put latency was reduced due to extra ipc call to raylet
# for ref counting.
single client get calls (Plasma Store) per second 11743.14 +- 2062.85
single client put calls (Plasma Store) per second 3133.08 +- 89.81
single client put gigabytes per second 10.33 +- 7.96
multi client put calls (Plasma Store) per second 3590.16 +- 22.04
multi client put gigabytes per second 23.38 +- 0.63
single client tasks sync per second 1263.59 +- 63.16
single client tasks async per second 13959.14 +- 393.16
multi client tasks async per second 42285.81 +- 238.55
1:1 actor calls sync per second 2159.21 +- 112.97
1:1 actor calls async per second 7048.53 +- 63.8
1:1 actor calls concurrent per second 6167.01 +- 75.67
1:n actor calls async per second 12241.67 +- 62.13
n:n actor calls async per second 41766.33 +- 672.14
n:n actor calls with arg async per second 13134.22 +- 71.68
doc/dev/release_logs/0.8.2/rllib_regression.txt
-36
View File
@@ -1,36 +0,0 @@
== Status ==
Memory usage on this node: 43.4/480.3 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/64 CPUs, 0.0/8 GPUs, 0.0/440.23 GiB heap, 0.0/12.84 GiB objects
Result logdir: /home/ubuntu/ray_results/apex
Result logdir: /home/ubuntu/ray_results/atari-a2c
Result logdir: /home/ubuntu/ray_results/atari-basic-dqn
Result logdir: /home/ubuntu/ray_results/atari-impala
Result logdir: /home/ubuntu/ray_results/atari-ppo-tf
Result logdir: /home/ubuntu/ray_results/atari-ppo-torch
Number of trials: 24 (24 TERMINATED)
Table truncated to 20 rows. 4 trials (4 TERMINATED) not shown.
+--------------------------------------+------------+-------+----------+------------------+---------+--------+
| Trial name | status | loc | reward | total time (s) | ts | iter |
|--------------------------------------+------------+-------+----------+------------------+---------+--------|
| A2C_BreakoutNoFrameskip-v4_c8ad5a48 | TERMINATED | | 139.19 | 3606.77 | 3686000 | 352 |
| A2C_BreakoutNoFrameskip-v4_c8ad1c54 | TERMINATED | | 75.56 | 3601.57 | 2932000 | 349 |
| A2C_BreakoutNoFrameskip-v4_c8acd28a | TERMINATED | | 131.97 | 3603.39 | 2928000 | 349 |
| A2C_BreakoutNoFrameskip-v4_c8ac8d16 | TERMINATED | | 105.42 | 3601.03 | 2901500 | 349 |
| DQN_BreakoutNoFrameskip-v4_c8af8a02 | TERMINATED | | 15.81 | 3665.65 | 270000 | 27 |
| DQN_BreakoutNoFrameskip-v4_c8af079e | TERMINATED | | 11.32 | 3612.1 | 270000 | 27 |
| APEX_BreakoutNoFrameskip-v4_c8ac4694 | TERMINATED | | 50.56 | 3627.89 | 5786880 | 115 |
| DQN_BreakoutNoFrameskip-v4_c8ae61ae | TERMINATED | | 7.14 | 3620.61 | 270000 | 27 |
| DQN_BreakoutNoFrameskip-v4_c8adbcea | TERMINATED | | 11.24 | 3640.35 | 270000 | 27 |
| APEX_BreakoutNoFrameskip-v4_c8abef3c | TERMINATED | | 94.5 | 3625.19 | 5820800 | 115 |
| PPO_BreakoutNoFrameskip-v4_c8ab0572 | TERMINATED | | 25.26 | 3603.23 | 1335000 | 267 |
| PPO_BreakoutNoFrameskip-v4_c8aabf36 | TERMINATED | | 18.2 | 3603.36 | 1300000 | 260 |
| APEX_BreakoutNoFrameskip-v4_c8abaa86 | TERMINATED | | 90.98 | 3627.03 | 7350400 | 116 |
| PPO_BreakoutNoFrameskip-v4_c8aa6f5e | TERMINATED | | 17.01 | 3611.01 | 1555000 | 311 |
| PPO_BreakoutNoFrameskip-v4_c8aa27e2 | TERMINATED | | 22.41 | 3609.64 | 1545000 | 309 |
| PPO_BreakoutNoFrameskip-v4_c8a9e39a | TERMINATED | | 61.25 | 3602.17 | 4475000 | 895 |
| PPO_BreakoutNoFrameskip-v4_c8a97978 | TERMINATED | | 28.19 | 3601.33 | 4415000 | 883 |
| PPO_BreakoutNoFrameskip-v4_c8a904ca | TERMINATED | | 41.3 | 3600.42 | 4515000 | 903 |
| APEX_BreakoutNoFrameskip-v4_c8ab5108 | TERMINATED | | 62.46 | 3626.37 | 5091840 | 114 |
| PPO_BreakoutNoFrameskip-v4_c8a88004 | TERMINATED | | 60.44 | 3602.52 | 3380000 | 676 |
+--------------------------------------+------------+-------+----------+------------------+---------+--------+
doc/dev/release_logs/0.8.2/stress_tests/application_stress_test.txt
-14
View File
@@ -1,14 +0,0 @@
== Status ==
Memory usage on this node: 34.6/480.3 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/640 CPUs, 0/8 GPUs, 0.0/2541.21 GiB heap, 0.0/128.42 GiB objects
Result logdir: /home/ubuntu/ray_results/atari-impala
Number of trials: 4 (4 TERMINATED)
+---------------------------------------------+------------+-------+-----------------------------+----------+------------------+----------+--------+
| Trial name | status | loc | env | reward | total time (s) | ts | iter |
|---------------------------------------------+------------+-------+-----------------------------+----------+------------------+----------+--------|
| IMPALA_BreakoutNoFrameskip-v4_2565545c | TERMINATED | | BreakoutNoFrameskip-v4 | 451.07 | 22555.3 | 30039500 | 381 |
| IMPALA_BeamRiderNoFrameskip-v4_2565e804 | TERMINATED | | BeamRiderNoFrameskip-v4 | 3124.8 | 24121.2 | 30057000 | 408 |
| IMPALA_QbertNoFrameskip-v4_256671de | TERMINATED | | QbertNoFrameskip-v4 | 8388.25 | 25163.5 | 30080000 | 453 |
| IMPALA_SpaceInvadersNoFrameskip-v4_256725ac | TERMINATED | | SpaceInvadersNoFrameskip-v4 | 780.65 | 23148.1 | 30026500 | 384 |
+---------------------------------------------+------------+-------+-----------------------------+----------+------------------+----------+--------+
doc/dev/release_logs/0.8.2/stress_tests/test_dead_actors.txt
-4
View File
@@ -1,4 +0,0 @@
Finished in: 98.49777579307556s
Average iteration time: 0.9849753308296204s
Max iteration time: 2.9459526538848877s
Min iteration time: 0.08075928688049316s
doc/dev/release_logs/0.8.2/stress_tests/test_many_tasks.txt
-15
View File
@@ -1,15 +0,0 @@
Stage 0 results:
Total time: 22.579216480255127
Stage 1 results:
Total time: 154.41431832313538
Average iteration time: 15.441423058509827
Max iteration time: 15.943994760513306
Min iteration time: 15.029884099960327
Stage 2 results:
Total time: 646.7662391662598
Average iteration time: 129.35279755592347
Max iteration time: 134.80017256736755
Min iteration time: 121.44297170639038
Stage 3 results:
Actor creation time: 0.0635519027709961
Total time: 3464.0461547374725
doc/dev/release_logs/0.8.3/microbenchmark.txt
-14
View File
@@ -1,14 +0,0 @@
single client get calls (Plasma Store) per second 9609.58 +- 1094.38
single client put calls (Plasma Store) per second 5244.46 +- 96.12
single client put gigabytes per second 13.5 +- 3.12
multi client put calls (Plasma Store) per second 11147.13 +- 25.58
multi client put gigabytes per second 19.08 +- 7.08
single client tasks sync per second 1271.4 +- 14.02
single client tasks async per second 12516.66 +- 392.73
multi client tasks async per second 36893.52 +- 1313.69
1:1 actor calls sync per second 1928.61 +- 43.71
1:1 actor calls async per second 7219.07 +- 147.61
1:1 actor calls concurrent per second 6267.15 +- 67.2
1:n actor calls async per second 9926.58 +- 143.94
n:n actor calls async per second 34545.18 +- 355.83
n:n actor calls with arg async per second 12897.18 +- 203.78
doc/dev/release_logs/0.8.3/rllib_regression.txt
-40
View File
@@ -1,40 +0,0 @@
== Status ==
Memory usage on this node: 133.8/480.3 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/64 CPUs, 0.0/8 GPUs, 0.0/323.54 GiB heap, 0.0/98.39 GiB objects
Result logdir: /home/ubuntu/ray_results/apex
Result logdir: /home/ubuntu/ray_results/atari-a2c
Result logdir: /home/ubuntu/ray_results/atari-basic-dqn
Result logdir: /home/ubuntu/ray_results/atari-impala
Result logdir: /home/ubuntu/ray_results/atari-ppo-tf
Result logdir: /home/ubuntu/ray_results/atari-ppo-torch
Number of trials: 24 (24 TERMINATED)
+-------------------------------------+------------+-------+--------+------------------+---------+----------+
| Trial name | status | loc | iter | total time (s) | ts | reward |
|-------------------------------------+------------+-------+--------+------------------+---------+----------|
| IMPALA_BreakoutNoFrameskip-v4_00000 | TERMINATED | | 293 | 3610.57 | 7244000 | 95.15 |
| IMPALA_BreakoutNoFrameskip-v4_00001 | TERMINATED | | 293 | 3610.01 | 7237500 | 233.92 |
| IMPALA_BreakoutNoFrameskip-v4_00002 | TERMINATED | | 293 | 3609.84 | 7285500 | 374.97 |
| IMPALA_BreakoutNoFrameskip-v4_00003 | TERMINATED | | 293 | 3609.78 | 7309500 | 371.48 |
| PPO_BreakoutNoFrameskip-v4_00004 | TERMINATED | | 673 | 3603.29 | 3365000 | 50.99 |
| PPO_BreakoutNoFrameskip-v4_00005 | TERMINATED | | 898 | 3602.34 | 4490000 | 56.53 |
| PPO_BreakoutNoFrameskip-v4_00006 | TERMINATED | | 903 | 3602.76 | 4515000 | 54.87 |
| PPO_BreakoutNoFrameskip-v4_00007 | TERMINATED | | 886 | 3603.85 | 4430000 | 36.06 |
| PPO_BreakoutNoFrameskip-v4_00008 | TERMINATED | | 305 | 3603.15 | 1525000 | 15.52 |
| PPO_BreakoutNoFrameskip-v4_00009 | TERMINATED | | 307 | 3607.11 | 1535000 | 19.75 |
| PPO_BreakoutNoFrameskip-v4_00010 | TERMINATED | | 257 | 3605.7 | 1285000 | 41.43 |
| PPO_BreakoutNoFrameskip-v4_00011 | TERMINATED | | 253 | 3613.21 | 1265000 | 23.07 |
| APEX_BreakoutNoFrameskip-v4_00012 | TERMINATED | | 114 | 3620.53 | 5090400 | 34.63 |
| APEX_BreakoutNoFrameskip-v4_00013 | TERMINATED | | 116 | 3619.38 | 7491040 | 78.16 |
| APEX_BreakoutNoFrameskip-v4_00014 | TERMINATED | | 115 | 3628.51 | 5769120 | 48.27 |
| APEX_BreakoutNoFrameskip-v4_00015 | TERMINATED | | 115 | 3627.06 | 5807200 | 49.18 |
| A2C_BreakoutNoFrameskip-v4_00016 | TERMINATED | | 351 | 3607.65 | 2904000 | 108.71 |
| A2C_BreakoutNoFrameskip-v4_00017 | TERMINATED | | 350 | 3608.21 | 2933000 | 63.93 |
| A2C_BreakoutNoFrameskip-v4_00018 | TERMINATED | | 351 | 3605.63 | 2898500 | 169.93 |
| A2C_BreakoutNoFrameskip-v4_00019 | TERMINATED | | 353 | 3606.64 | 3836000 | 232.4 |
| DQN_BreakoutNoFrameskip-v4_00020 | TERMINATED | | 27 | 3686.19 | 270000 | 17.53 |
| DQN_BreakoutNoFrameskip-v4_00021 | TERMINATED | | 27 | 3651.85 | 270000 | 18.34 |
| DQN_BreakoutNoFrameskip-v4_00022 | TERMINATED | | 27 | 3669.77 | 270000 | 15.45 |
| DQN_BreakoutNoFrameskip-v4_00023 | TERMINATED | | 27 | 3688.08 | 270000 | 12.25 |
+-------------------------------------+------------+-------+--------+------------------+---------+----------+
doc/dev/release_logs/0.8.3/stress_tests/application_stress_test.txt
-14
View File
@@ -1,14 +0,0 @@
== Status ==
Memory usage on this node: 23.2/480.3 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/640 CPUs, 0/8 GPUs, 0.0/1905.76 GiB heap, 0.0/566.21 GiB objects
Result logdir: /home/ubuntu/ray_results/atari-impala
Number of trials: 4 (4 TERMINATED)
+------------------------------------------+------------+-------+-----------------------------+--------+------------------+----------+----------+
| Trial name | status | loc | env | iter | total time (s) | ts | reward |
|------------------------------------------+------------+-------+-----------------------------+--------+------------------+----------+----------|
| IMPALA_BreakoutNoFrameskip-v4_00000 | TERMINATED | | BreakoutNoFrameskip-v4 | 378 | 6722.44 | 30011000 | 470.62 |
| IMPALA_BeamRiderNoFrameskip-v4_00001 | TERMINATED | | BeamRiderNoFrameskip-v4 | 362 | 6494.28 | 30026000 | 3006.44 |
| IMPALA_QbertNoFrameskip-v4_00002 | TERMINATED | | QbertNoFrameskip-v4 | 364 | 6512.25 | 30016500 | 10843.5 |
| IMPALA_SpaceInvadersNoFrameskip-v4_00003 | TERMINATED | | SpaceInvadersNoFrameskip-v4 | 380 | 6752.71 | 30014500 | 677.1 |
+------------------------------------------+------------+-------+-----------------------------+--------+------------------+----------+----------+
doc/dev/release_logs/0.8.3/stress_tests/test_dead_actors.txt
-4
View File
@@ -1,4 +0,0 @@
Finished in: 101.67017650604248s
Average iteration time: 1.0166999363899232s
Max iteration time: 2.874112129211426s
Min iteration time: 0.06804370880126953s
doc/dev/release_logs/0.8.3/stress_tests/test_many_tasks.txt
-15
View File
@@ -1,15 +0,0 @@
Stage 0 results:
Total time: 19.931869983673096
Stage 1 results:
Total time: 157.56488513946533
Average iteration time: 15.756479907035828
Max iteration time: 16.13578486442566
Min iteration time: 15.528882265090942
Stage 2 results:
Total time: 648.047709941864
Average iteration time: 129.60901088714598
Max iteration time: 135.63372230529785
Min iteration time: 119.46208763122559
Stage 3 results:
Actor creation time: 0.12419295310974121
Total time: 3563.88968873024
doc/dev/release_logs/0.8.4/microbenchmark.txt
-14
View File
@@ -1,14 +0,0 @@
single client get calls (Plasma Store) per second 9887.7 +- 1181.06
single client put calls (Plasma Store) per second 5061.57 +- 89.58
single client put gigabytes per second 13.23 +- 2.84
multi client put calls (Plasma Store) per second 10954.66 +- 107.58
multi client put gigabytes per second 17.98 +- 8.51
single client tasks sync per second 1246.16 +- 64.56
single client tasks async per second 12836.66 +- 196.27
multi client tasks async per second 31252.17 +- 9348.4
1:1 actor calls sync per second 1974.26 +- 31.31
1:1 actor calls async per second 7211.89 +- 158.3
1:1 actor calls concurrent per second 6554.62 +- 140.58
1:n actor calls async per second 9934.44 +- 103.02
n:n actor calls async per second 34703.22 +- 553.23
n:n actor calls with arg async per second 12569.31 +- 257.01
doc/dev/release_logs/0.8.4/rllib_regression.txt
-47
View File
@@ -1,47 +0,0 @@
== Status ==
Memory usage on this node: 166.1/480.3 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/64 CPUs, 0.0/8 GPUs, 0.0/323.54 GiB heap, 0.0/98.39 GiB objects
Result logdir: /home/ubuntu/ray_results/apex
Result logdir: /home/ubuntu/ray_results/atari-a2c
Result logdir: /home/ubuntu/ray_results/atari-basic-dqn
Result logdir: /home/ubuntu/ray_results/atari-impala
Result logdir: /home/ubuntu/ray_results/atari-ppo-tf
Result logdir: /home/ubuntu/ray_results/atari-ppo-torch
Number of trials: 24 (8 ERROR, 16 TERMINATED)
+-------------------------------------+------------+-------+--------+------------------+---------+-----------+
| Trial name | status | loc | iter | total time (s) | ts | reward |
|-------------------------------------+------------+-------+--------+------------------+---------+-----------|
| IMPALA_BreakoutNoFrameskip-v4_00000 | TERMINATED | | 293 | 3612.08 | 7270500 | 351.08 |
| IMPALA_BreakoutNoFrameskip-v4_00001 | TERMINATED | | 293 | 3612.11 | 7252500 | 181.3 |
| IMPALA_BreakoutNoFrameskip-v4_00002 | TERMINATED | | 293 | 3610.22 | 7283000 | 377.12 |
| IMPALA_BreakoutNoFrameskip-v4_00003 | TERMINATED | | 293 | 3612.02 | 7238000 | 1.9927 |
| PPO_BreakoutNoFrameskip-v4_00004 | TERMINATED | | 680 | 3602.41 | 3400000 | 45.13 |
| PPO_BreakoutNoFrameskip-v4_00005 | TERMINATED | | 853 | 3600.28 | 4265000 | 45.87 |
| PPO_BreakoutNoFrameskip-v4_00006 | TERMINATED | | 863 | 3603.41 | 4315000 | 40.92 |
| PPO_BreakoutNoFrameskip-v4_00007 | TERMINATED | | 859 | 3600.36 | 4295000 | 80.42 |
| PPO_BreakoutNoFrameskip-v4_00000 | TERMINATED | | 433 | 3606.17 | 2165000 | 23.65 |
| PPO_BreakoutNoFrameskip-v4_00001 | TERMINATED | | 435 | 3606.03 | 2175000 | 18.35 |
| PPO_BreakoutNoFrameskip-v4_00002 | TERMINATED | | 436 | 3608.19 | 2180000 | 31.09 |
| PPO_BreakoutNoFrameskip-v4_00003 | TERMINATED | | 434 | 3602.58 | 2170000 | 17.32 |
| APEX_BreakoutNoFrameskip-v4_00004 | TERMINATED | | 115 | 3609.98 | 6512320 | 55.76 |
| APEX_BreakoutNoFrameskip-v4_00005 | TERMINATED | | 115 | 3609.56 | 6491040 | 66.91 |
| APEX_BreakoutNoFrameskip-v4_00006 | TERMINATED | | 116 | 3607.5 | 8302720 | 91.65 |
| APEX_BreakoutNoFrameskip-v4_00007 | TERMINATED | | 116 | 3605.31 | 8500960 | 79.94 |
| A2C_BreakoutNoFrameskip-v4_00016 | TERMINATED | | 354 | 3608.77 | 4223000 | 288.79 |
| A2C_BreakoutNoFrameskip-v4_00017 | TERMINATED | | 354 | 3608.4 | 4119000 | 231.83 |
| A2C_BreakoutNoFrameskip-v4_00018 | TERMINATED | | 354 | 3605.56 | 4121500 | 169.97 |
| A2C_BreakoutNoFrameskip-v4_00019 | TERMINATED | | 354 | 3606.65 | 4169500 | 276.07 |
| DQN_BreakoutNoFrameskip-v4_00020 | TERMINATED | | 33 | 3675.38 | 330000 | 15.44 |
| DQN_BreakoutNoFrameskip-v4_00021 | TERMINATED | | 33 | 3674.96 | 330000 | 13.14 |
| DQN_BreakoutNoFrameskip-v4_00022 | TERMINATED | | 33 | 3679.57 | 330000 | 17.47 |
| DQN_BreakoutNoFrameskip-v4_00023 | TERMINATED | | 33 | 3657.31 | 330000 | 17.47 |
+-------------------------------------+------------+-------+--------+------------------+---------+-----------+
doc/dev/release_logs/0.8.4/stress_tests/application_stress_test.txt
-14
View File
@@ -1,14 +0,0 @@
== Status ==
Memory usage on this node: 24.9/480.3 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/640 CPUs, 0/8 GPUs, 0.0/1905.71 GiB heap, 0.0/566.21 GiB objects
Result logdir: /home/ubuntu/ray_results/atari-impala
Number of trials: 4 (4 TERMINATED)
+------------------------------------------+------------+-------+-----------------------------+--------+------------------+----------+----------+
| Trial name | status | loc | env | iter | total time (s) | ts | reward |
|------------------------------------------+------------+-------+-----------------------------+--------+------------------+----------+----------|
| IMPALA_BreakoutNoFrameskip-v4_00000 | TERMINATED | | BreakoutNoFrameskip-v4 | 379 | 6695.67 | 30028000 | 547.94 |
| IMPALA_BeamRiderNoFrameskip-v4_00001 | TERMINATED | | BeamRiderNoFrameskip-v4 | 382 | 6742.5 | 30068000 | 2520.02 |
| IMPALA_QbertNoFrameskip-v4_00002 | TERMINATED | | QbertNoFrameskip-v4 | 374 | 6623.08 | 30024500 | 9815 |
| IMPALA_SpaceInvadersNoFrameskip-v4_00003 | TERMINATED | | SpaceInvadersNoFrameskip-v4 | 380 | 6711.43 | 30058500 | 762.9 |
+------------------------------------------+------------+-------+-----------------------------+--------+------------------+----------+----------+
doc/dev/release_logs/0.8.4/stress_tests/test_dead_actors.txt
-4
View File
@@ -1,4 +0,0 @@
Finished in: 103.75814247131348s
Average iteration time: 1.037579414844513s
Max iteration time: 2.213489055633545s
Min iteration time: 0.09523892402648926s
doc/dev/release_logs/0.8.4/stress_tests/test_many_tasks.txt
-15
View File
@@ -1,15 +0,0 @@
Stage 0 results:
Total time: 24.571804523468018
Stage 1 results:
Total time: 155.27635407447815
Average iteration time: 15.527625465393067
Max iteration time: 16.342751502990723
Min iteration time: 15.177084684371948
Stage 2 results:
Total time: 661.6063549518585
Average iteration time: 132.3208240032196
Max iteration time: 134.2343475818634
Min iteration time: 128.90119433403015
Stage 3 results:
Actor creation time: 0.07286763191223145
Total time: 3591.7382633686066
doc/dev/release_logs/0.8.5/microbenchmark.txt
-19
View File
@@ -1,19 +0,0 @@
single client get calls (Plasma Store) per second 8214.4 +- 892.64
single client put calls (Plasma Store) per second 5356.26 +- 36.53
single client put gigabytes per second 13.46 +- 2.94
multi client put calls (Plasma Store) per second 11351.56 +- 62.33
multi client put gigabytes per second 16.44 +- 8.49
single client tasks sync per second 1311.81 +- 66.92
single client tasks async per second 15531.87 +- 220.75
multi client tasks async per second 43140.09 +- 880.94
1:1 actor calls sync per second 1916.18 +- 97.0
1:1 actor calls async per second 5963.83 +- 115.07
1:1 actor calls concurrent per second 5604.0 +- 222.31
1:n actor calls async per second 10275.4 +- 167.94
n:n actor calls async per second 34553.25 +- 196.74
n:n actor calls with arg async per second 12522.36 +- 228.16
1:1 async-actor calls sync per second 1096.88 +- 4.27
1:1 async-actor calls async per second 3407.27 +- 74.07
1:1 async-actor calls with args async per second 2406.66 +- 46.86
1:n async-actor calls async per second 9109.89 +- 121.35
n:n async-actor calls async per second 27338.51 +- 358.6
doc/dev/release_logs/0.8.5/rllib_regression.txt
-39
View File
@@ -1,39 +0,0 @@
== Status ==
Memory usage on this node: 15.5/480.3 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/64 CPUs, 0.0/8 GPUs, 0.0/323.54 GiB heap, 0.0/98.39 GiB objects
Result logdir: /home/ubuntu/ray_results/apex
Result logdir: /home/ubuntu/ray_results/atari-a2c
Result logdir: /home/ubuntu/ray_results/atari-basic-dqn
Result logdir: /home/ubuntu/ray_results/atari-impala
Result logdir: /home/ubuntu/ray_results/atari-ppo-tf
Result logdir: /home/ubuntu/ray_results/atari-ppo-torch
Number of trials: 24 (24 TERMINATED)
+-------------------------------------+------------+-------+--------+------------------+---------+----------+
| Trial name | status | loc | iter | total time (s) | ts | reward |
|-------------------------------------+------------+-------+--------+------------------+---------+----------|
| IMPALA_BreakoutNoFrameskip-v4_00000 | TERMINATED | | 300 | 3609.38 | 7596000 | 371.22 |
| IMPALA_BreakoutNoFrameskip-v4_00001 | TERMINATED | | 300 | 3604.49 | 7561500 | 361.02 |
| IMPALA_BreakoutNoFrameskip-v4_00002 | TERMINATED | | 300 | 3605.28 | 7581500 | 358.73 |
| IMPALA_BreakoutNoFrameskip-v4_00003 | TERMINATED | | 300 | 3604.35 | 7567500 | 336.82 |
| PPO_BreakoutNoFrameskip-v4_00004 | TERMINATED | | 682 | 3601.14 | 3410000 | 66.3 |
| PPO_BreakoutNoFrameskip-v4_00005 | TERMINATED | | 951 | 3602.81 | 4755000 | 36.08 |
| PPO_BreakoutNoFrameskip-v4_00006 | TERMINATED | | 951 | 3601.49 | 4755000 | 72.78 |
| PPO_BreakoutNoFrameskip-v4_00007 | TERMINATED | | 950 | 3601.34 | 4750000 | 38.93 |
| PPO_BreakoutNoFrameskip-v4_00008 | TERMINATED | | 27 | 3726.15 | 135000 | 5.77 |
| PPO_BreakoutNoFrameskip-v4_00009 | TERMINATED | | 27 | 3626.05 | 135000 | 2.16 |
| PPO_BreakoutNoFrameskip-v4_00010 | TERMINATED | | 22 | 3645.07 | 110000 | 2.84 |
| PPO_BreakoutNoFrameskip-v4_00011 | TERMINATED | | 22 | 3622.54 | 110000 | 2.32 |
| APEX_BreakoutNoFrameskip-v4_00012 | TERMINATED | | 106 | 3620.95 | 5372640 | 59.43 |
| APEX_BreakoutNoFrameskip-v4_00013 | TERMINATED | | 107 | 3617.3 | 8081280 | 97.22 |
| APEX_BreakoutNoFrameskip-v4_00014 | TERMINATED | | 88 | 3601.26 | 6635200 | 79.08 |
| APEX_BreakoutNoFrameskip-v4_00015 | TERMINATED | | 89 | 3618.23 | 6583040 | 85.28 |
| A2C_BreakoutNoFrameskip-v4_00016 | TERMINATED | | 353 | 3603.02 | 3351500 | 126.25 |
| A2C_BreakoutNoFrameskip-v4_00017 | TERMINATED | | 353 | 3607.96 | 3339000 | 125.67 |
| A2C_BreakoutNoFrameskip-v4_00018 | TERMINATED | | 351 | 3605.17 | 3306000 | 224.59 |
| A2C_BreakoutNoFrameskip-v4_00019 | TERMINATED | | 353 | 3607.46 | 3676000 | 163.83 |
| DQN_BreakoutNoFrameskip-v4_00020 | TERMINATED | | 27 | 3654.34 | 280104 | 15.51 |
| DQN_BreakoutNoFrameskip-v4_00021 | TERMINATED | | 27 | 3660.45 | 280104 | 17.15 |
| DQN_BreakoutNoFrameskip-v4_00022 | TERMINATED | | 27 | 3615 | 280104 | 15.37 |
| DQN_BreakoutNoFrameskip-v4_00023 | TERMINATED | | 28 | 3655.79 | 290108 | 16.02 |
+-------------------------------------+------------+-------+--------+------------------+---------+----------+
doc/dev/release_logs/0.8.5/stress_tests/application_stress_test.txt
-15
View File
@@ -1,15 +0,0 @@
== Status ==
Memory usage on this node: 23.4/480.3 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/640 CPUs, 0/8 GPUs, 0.0/1905.86 GiB heap, 0.0/566.21 GiB objects
Result logdir: /home/ubuntu/ray_results/atari-impala
Number of trials: 4 (4 TERMINATED)
+------------------------------------------+------------+-------+-----------------------------+--------+------------------+----------+----------+
| Trial name | status | loc | env | iter | total time (s) | ts | reward |
|------------------------------------------+------------+-------+-----------------------------+--------+------------------+----------+----------|
| IMPALA_BreakoutNoFrameskip-v4_00000 | TERMINATED | | BreakoutNoFrameskip-v4 | 360 | 5974.47 | 30014500 | 488.4 |
| IMPALA_BeamRiderNoFrameskip-v4_00001 | TERMINATED | | BeamRiderNoFrameskip-v4 | 366 | 6026.01 | 30031500 | 424.84 |
| IMPALA_QbertNoFrameskip-v4_00002 | TERMINATED | | QbertNoFrameskip-v4 | 356 | 5897.19 | 30077500 | 5230.25 |
| IMPALA_SpaceInvadersNoFrameskip-v4_00003 | TERMINATED | | SpaceInvadersNoFrameskip-v4 | 361 | 5990.74 | 30103500 | 806.043 |
+------------------------------------------+------------+-------+-----------------------------+--------+------------------+----------+----------+
doc/dev/release_logs/0.8.5/stress_tests/test_dead_actors.txt
-4
View File
@@ -1,4 +0,0 @@
Finished in: 98.98898577690125s
Average iteration time: 0.9898880553245545s
Max iteration time: 2.3596835136413574s
Min iteration time: 0.1039724349975586s
doc/dev/release_logs/0.8.5/stress_tests/test_many_tasks.txt
-15
View File
@@ -1,15 +0,0 @@
Stage 0 results:
Total time: 21.52851891517639
Stage 1 results:
Total time: 151.462096452713
Average iteration time: 15.146199059486388
Max iteration time: 16.154610872268677
Min iteration time: 14.686655044555664
Stage 2 results:
Total time: 1099.6661903858185
Average iteration time: 219.9327588558197
Max iteration time: 236.58895802497864
Min iteration time: 201.1623387336731
Stage 3 results:
Actor creation time: 0.09940552711486816
Total time: 3180.518438100815
doc/dev/release_logs/0.8.6/microbenchmark.txt
-19
View File
@@ -1,19 +0,0 @@
single client get calls (Plasma Store) per second 8746.49 +- 454.49
single client put calls (Plasma Store) per second 5449.42 +- 86.57
single client put gigabytes per second 13.59 +- 3.27
multi client put calls (Plasma Store) per second 11662.1 +- 153.21
multi client put gigabytes per second 17.26 +- 7.94
single client tasks sync per second 1272.57 +- 33.26
single client tasks async per second 14621.87 +- 200.32
multi client tasks async per second 41588.79 +- 4142.82
1:1 actor calls sync per second 1914.85 +- 20.58
1:1 actor calls async per second 6541.67 +- 127.36
1:1 actor calls concurrent per second 5663.49 +- 155.04
1:n actor calls async per second 10495.3 +- 133.15
n:n actor calls async per second 34453.12 +- 329.25
n:n actor calls with arg async per second 12500.39 +- 167.03
1:1 async-actor calls sync per second 1291.48 +- 48.9
1:1 async-actor calls async per second 3510.1 +- 48.84
1:1 async-actor calls with args async per second 2321.3 +- 70.18
1:n async-actor calls async per second 8961.23 +- 111.43
n:n async-actor calls async per second 27483.48 +- 113.91
doc/dev/release_logs/0.8.6/rllib_regression.txt
-39
View File
@@ -1,39 +0,0 @@
== Status ==
Memory usage on this node: 14.3/480.3 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/64 CPUs, 0.0/8 GPUs, 0.0/323.49 GiB heap, 0.0/98.39 GiB objects
Result logdir: /home/ubuntu/ray_results/apex
Result logdir: /home/ubuntu/ray_results/atari-a2c
Result logdir: /home/ubuntu/ray_results/atari-basic-dqn
Result logdir: /home/ubuntu/ray_results/atari-impala
Result logdir: /home/ubuntu/ray_results/atari-ppo-tf
Result logdir: /home/ubuntu/ray_results/atari-ppo-torch
Number of trials: 24 (24 TERMINATED)
+-------------------------------------------+------------+-------+--------+------------------+---------+----------+
| Trial name | status | loc | iter | total time (s) | ts | reward |
|-------------------------------------------+------------+-------+--------+------------------+---------+----------|
| IMPALA_BreakoutNoFrameskip-v4_926d2_00000 | TERMINATED | | 357 | 3601.43 | 7523500 | 315.74 |
| IMPALA_BreakoutNoFrameskip-v4_926d2_00001 | TERMINATED | | 356 | 3603.43 | 7455000 | 340.84 |
| IMPALA_BreakoutNoFrameskip-v4_926d2_00002 | TERMINATED | | 357 | 3601.72 | 7622250 | 306.88 |
| IMPALA_BreakoutNoFrameskip-v4_926d2_00003 | TERMINATED | | 357 | 3601.02 | 7561500 | 331.75 |
| PPO_BreakoutNoFrameskip-v4_926d2_00004 | TERMINATED | | 692 | 3600.28 | 3460000 | 28.44 |
| PPO_BreakoutNoFrameskip-v4_926d2_00005 | TERMINATED | | 946 | 3600.06 | 4730000 | 53.61 |
| PPO_BreakoutNoFrameskip-v4_926d2_00006 | TERMINATED | | 943 | 3602.39 | 4715000 | 37.03 |
| PPO_BreakoutNoFrameskip-v4_926d2_00007 | TERMINATED | | 944 | 3602.4 | 4720000 | 36.35 |
| PPO_BreakoutNoFrameskip-v4_926d2_00008 | TERMINATED | | 27 | 3600.14 | 135000 | 2.02 |
| PPO_BreakoutNoFrameskip-v4_926d2_00009 | TERMINATED | | 27 | 3723.31 | 135000 | 5.37 |
| PPO_BreakoutNoFrameskip-v4_926d2_00010 | TERMINATED | | 21 | 3704.86 | 105000 | 1.5 |
| PPO_BreakoutNoFrameskip-v4_926d2_00011 | TERMINATED | | 22 | 3630.76 | 110000 | 1.98 |
| APEX_BreakoutNoFrameskip-v4_926d2_00012 | TERMINATED | | 106 | 3607.17 | 5372960 | 74.44 |
| APEX_BreakoutNoFrameskip-v4_926d2_00013 | TERMINATED | | 107 | 3603.72 | 8074080 | 87.35 |
| APEX_BreakoutNoFrameskip-v4_926d2_00014 | TERMINATED | | 90 | 3633.64 | 6787680 | 59.02 |
| APEX_BreakoutNoFrameskip-v4_926d2_00015 | TERMINATED | | 90 | 3616.73 | 6784160 | 61.9 |
| A2C_BreakoutNoFrameskip-v4_926d2_00016 | TERMINATED | | 354 | 3603.13 | 3347000 | 74.53 |
| A2C_BreakoutNoFrameskip-v4_926d2_00017 | TERMINATED | | 353 | 3601.95 | 3374000 | 174.69 |
| A2C_BreakoutNoFrameskip-v4_926d2_00018 | TERMINATED | | 353 | 3601.02 | 3377000 | 149.69 |
| A2C_BreakoutNoFrameskip-v4_926d2_00019 | TERMINATED | | 353 | 3605.72 | 3732000 | 203.06 |
| DQN_BreakoutNoFrameskip-v4_926d2_00020 | TERMINATED | | 27 | 3695.93 | 280000 | 14.71 |
| DQN_BreakoutNoFrameskip-v4_926d2_00021 | TERMINATED | | 26 | 3613.54 | 270000 | 16.37 |
| DQN_BreakoutNoFrameskip-v4_926d2_00022 | TERMINATED | | 27 | 3661.86 | 280000 | 15.43 |
| DQN_BreakoutNoFrameskip-v4_926d2_00023 | TERMINATED | | 27 | 3680.85 | 280000 | 15.62 |
+-------------------------------------------+------------+-------+--------+------------------+---------+----------+
doc/dev/release_logs/0.8.6/stress_tests/application_stress_test.txt
-14
View File
@@ -1,14 +0,0 @@
== Status ==
Memory usage on this node: 19.9/480.3 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/640 CPUs, 0/8 GPUs, 0.0/1906.49 GiB heap, 0.0/566.21 GiB objects
Result logdir: /home/ubuntu/ray_results/atari-impala
Number of trials: 4 (4 TERMINATED)
+------------------------------------------------+------------+-------+-----------------------------+--------+------------------+----------+----------+
| Trial name | status | loc | env | iter | total time (s) | ts | reward |
|------------------------------------------------+------------+-------+-----------------------------+--------+------------------+----------+----------|
| IMPALA_BreakoutNoFrameskip-v4_a4e31_00000 | TERMINATED | | BreakoutNoFrameskip-v4 | 680 | 6847.72 | 30047500 | 358.37 |
| IMPALA_BeamRiderNoFrameskip-v4_a4e31_00001 | TERMINATED | | BeamRiderNoFrameskip-v4 | 694 | 6987.42 | 30001500 | 2657.04 |
| IMPALA_QbertNoFrameskip-v4_a4e31_00002 | TERMINATED | | QbertNoFrameskip-v4 | 676 | 6821.02 | 30016500 | 3606.75 |
| IMPALA_SpaceInvadersNoFrameskip-v4_a4e31_00003 | TERMINATED | | SpaceInvadersNoFrameskip-v4 | 684 | 6889.83 | 30025000 | 653.4 |
+------------------------------------------------+------------+-------+-----------------------------+--------+------------------+----------+----------+
doc/dev/release_logs/0.8.6/stress_tests/test_dead_actors.txt
-4
View File
@@ -1,4 +0,0 @@
Finished in: 457.8229465484619s
Average iteration time: 4.578227622509003s
Max iteration time: 28.17513918876648s
Min iteration time: 0.31386494636535645s
doc/dev/release_logs/0.8.6/stress_tests/test_many_tasks.txt
-16
View File
@@ -1,16 +0,0 @@
Stage 0 results:
Total time: 21.636702299118042
Stage 1 results:
Total time: 150.684504032135
Average iteration time: 15.068440675735474
Max iteration time: 16.537473440170288
Min iteration time: 14.317448616027832
Stage 2 results:
Total time: 1103.2014956474304
Average iteration time: 220.63988456726074
Max iteration time: 229.98777270317078
Min iteration time: 203.2540054321289
Stage 3 results:
Actor creation time: 0.09029483795166016
Total time: 3138.855129480362
doc/dev/release_logs/0.8.7/microbenchmark.txt
-22
View File
@@ -1,22 +0,0 @@
single client get calls per second 28723.85 +- 2122.5
single client put calls per second 28457.3 +- 199.18
multi client put calls per second 136112.46 +- 2957.42
single client get calls (Plasma Store) per second 7913.41 +- 625.96
single client put calls (Plasma Store) per second 5012.85 +- 26.89
multi client put calls (Plasma Store) per second 8767.68 +- 122.88
single client put gigabytes per second 12.41 +- 11.96
multi client put gigabytes per second 39.28 +- 2.8
single client tasks sync per second 1058.16 +- 6.87
single client tasks async per second 14787.24 +- 190.62
multi client tasks async per second 36551.97 +- 3364.28
1:1 actor calls sync per second 1498.51 +- 41.25
1:1 actor calls async per second 6701.16 +- 95.16
1:1 actor calls concurrent per second 5864.73 +- 68.68
1:n actor calls async per second 14372.61 +- 542.3
n:n actor calls async per second 38479.59 +- 920.62
n:n actor calls with arg async per second 10603.63 +- 95.77
1:1 async-actor calls sync per second 1074.72 +- 34.12
1:1 async-actor calls async per second 3755.41 +- 53.77
1:1 async-actor calls with args async per second 2340.26 +- 62.86
1:n async-actor calls async per second 13353.84 +- 446.48
n:n async-actor calls async per second 29600.63 +- 284.26
doc/dev/release_logs/0.8.7/rllib_regression.txt
-41
View File
@@ -1,41 +0,0 @@
== Status ==
Memory usage on this node: 9.5/480.3 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/64 CPUs, 0.0/8 GPUs, 0.0/325.78 GiB heap, 0.0/99.07 GiB objects
Result logdir: /home/ubuntu/ray_results/apex
Result logdir: /home/ubuntu/ray_results/atari-a2c
Result logdir: /home/ubuntu/ray_results/atari-basic-dqn
Result logdir: /home/ubuntu/ray_results/atari-impala
Result logdir: /home/ubuntu/ray_results/atari-ppo-tf
Result logdir: /home/ubuntu/ray_results/atari-ppo-torch
Number of trials: 24 (24 TERMINATED)
+-------------------------------------------+------------+-------+--------+------------------+---------+----------+
| Trial name | status | loc | iter | total time (s) | ts | reward |
|-------------------------------------------+------------+-------+--------+------------------+---------+----------|
| IMPALA_BreakoutNoFrameskip-v4_b13b3_00000 | TERMINATED | | 355 | 3601.74 | 7491750 | 381.68 |
| IMPALA_BreakoutNoFrameskip-v4_b13b3_00001 | TERMINATED | | 355 | 3603.97 | 7482000 | 367.12 |
| IMPALA_BreakoutNoFrameskip-v4_b13b3_00002 | TERMINATED | | 356 | 3607.98 | 7528500 | 375.77 |
| IMPALA_BreakoutNoFrameskip-v4_b13b3_00003 | TERMINATED | | 355 | 3605.97 | 7411000 | 328.21 |
| PPO_BreakoutNoFrameskip-v4_b13b3_00004 | TERMINATED | | 664 | 3603.23 | 3320000 | 31.47 |
| PPO_BreakoutNoFrameskip-v4_b13b3_00005 | TERMINATED | | 906 | 3603.53 | 4530000 | 41.31 |
| PPO_BreakoutNoFrameskip-v4_b13b3_00006 | TERMINATED | | 897 | 3600.83 | 4485000 | 60.32 |
| PPO_BreakoutNoFrameskip-v4_b13b3_00007 | TERMINATED | | 913 | 3600.05 | 4565000 | 51.28 |
| PPO_BreakoutNoFrameskip-v4_b13b3_00008 | TERMINATED | | 492 | 3606.93 | 2460000 | 35.75 |
| PPO_BreakoutNoFrameskip-v4_b13b3_00009 | TERMINATED | | 480 | 3600.3 | 2400000 | 13.71 |
| PPO_BreakoutNoFrameskip-v4_b13b3_00010 | TERMINATED | | 416 | 3606.47 | 2080000 | 7.49 |
| PPO_BreakoutNoFrameskip-v4_b13b3_00011 | TERMINATED | | 422 | 3602.06 | 2110000 | 13.29 |
| APEX_BreakoutNoFrameskip-v4_b13b3_00012 | TERMINATED | | 106 | 3614.12 | 5371360 | 48.41 |
| APEX_BreakoutNoFrameskip-v4_b13b3_00013 | TERMINATED | | 103 | 3618.46 | 7777760 | 82.33 |
| APEX_BreakoutNoFrameskip-v4_b13b3_00014 | TERMINATED | | 94 | 3627.11 | 6155840 | 62.79 |
| APEX_BreakoutNoFrameskip-v4_b13b3_00015 | TERMINATED | | 92 | 3615.04 | 6159520 | 63.5 |
| A2C_BreakoutNoFrameskip-v4_b13b3_00016 | TERMINATED | | 352 | 3606.65 | 3135000 | 62.67 |
| A2C_BreakoutNoFrameskip-v4_b13b3_00017 | TERMINATED | | 351 | 3605.31 | 3098000 | 124.24 |
| A2C_BreakoutNoFrameskip-v4_b13b3_00018 | TERMINATED | | 351 | 3607.44 | 3113500 | 129.7 |
| A2C_BreakoutNoFrameskip-v4_b13b3_00019 | TERMINATED | | 353 | 3601.78 | 4086000 | 271.04 |
| DQN_BreakoutNoFrameskip-v4_b13b3_00020 | TERMINATED | | 24 | 3682.43 | 250000 | 11.27 |
| DQN_BreakoutNoFrameskip-v4_b13b3_00021 | TERMINATED | | 24 | 3681.4 | 250000 | 15.9 |
| DQN_BreakoutNoFrameskip-v4_b13b3_00022 | TERMINATED | | 24 | 3677.74 | 250000 | 11.62 |
| DQN_BreakoutNoFrameskip-v4_b13b3_00023 | TERMINATED | | 24 | 3664.08 | 250000 | 7.47 |
+-------------------------------------------+------------+-------+--------+------------------+---------+----------+
doc/dev/release_logs/0.8.7/stress_tests/application_stress_test.txt
-14
View File
@@ -1,14 +0,0 @@
== Status ==
Memory usage on this node: 22.0/480.3 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/640 CPUs, 0/8 GPUs, 0.0/1906.54 GiB heap, 0.0/566.21 GiB objects (0/1.0 GPUType:V100)
Result logdir: /home/ubuntu/ray_results/atari-impala
Number of trials: 4 (4 TERMINATED)
+------------------------------------------------+------------+-------+-----------------------------+--------+------------------+----------+----------+
| Trial name | status | loc | env | iter | total time (s) | ts | reward |
|------------------------------------------------+------------+-------+-----------------------------+--------+------------------+----------+----------|
| IMPALA_BreakoutNoFrameskip-v4_b783d_00000 | TERMINATED | | BreakoutNoFrameskip-v4 | 746 | 7521.77 | 30009000 | 462.43 |
| IMPALA_BeamRiderNoFrameskip-v4_b783d_00001 | TERMINATED | | BeamRiderNoFrameskip-v4 | 761 | 7671.4 | 30024500 | 2318.14 |
| IMPALA_QbertNoFrameskip-v4_b783d_00002 | TERMINATED | | QbertNoFrameskip-v4 | 739 | 7457.03 | 30022000 | 10025.5 |
| IMPALA_SpaceInvadersNoFrameskip-v4_b783d_00003 | TERMINATED | | SpaceInvadersNoFrameskip-v4 | 766 | 7718.84 | 30037000 | 786.8 |
+------------------------------------------------+------------+-------+-----------------------------+--------+------------------+----------+----------+
doc/dev/release_logs/0.8.7/stress_tests/test_dead_actors.txt
-4
View File
@@ -1,4 +0,0 @@
Finished in: 456.6799967288971s
Average iteration time: 4.566798083782196s
Max iteration time: 10.93368911743164s
Min iteration time: 1.0819299221038818s
doc/dev/release_logs/0.8.7/stress_tests/test_many_tasks.txt
-15
View File
@@ -1,15 +0,0 @@
Stage 0 results:
Total time: 22.120722770690918
Stage 1 results:
Total time: 174.56333804130554
Average iteration time: 17.456315088272095
Max iteration time: 18.66864323616028
Min iteration time: 15.951467514038086
Stage 2 results:
Total time: 1324.5760805606842
Average iteration time: 264.91476254463197
Max iteration time: 278.47786712646484
Min iteration time: 248.30649852752686
Stage 3 results:
Actor creation time: 3.133239269256592
Total time: 2001.6567075252533
doc/dev/release_logs/1.0.0/microbenchmark.txt
-22
View File
@@ -1,22 +0,0 @@
single client get calls per second 30584.85 +- 661.06
single client put calls per second 23923.57 +- 360.12
multi client put calls per second 124576.32 +- 1847.41
single client get calls (Plasma Store) per second 7940.16 +- 456.8
single client put calls (Plasma Store) per second 4956.74 +- 31.4
multi client put calls (Plasma Store) per second 7705.71 +- 120.7
single client put gigabytes per second 13.74 +- 11.88
multi client put gigabytes per second 36.83 +- 1.89
single client tasks sync per second 977.6 +- 9.39
single client tasks async per second 14228.3 +- 195.64
multi client tasks async per second 36049.11 +- 790.33
1:1 actor calls sync per second 1454.92 +- 20.51
1:1 actor calls async per second 6612.21 +- 195.01
1:1 actor calls concurrent per second 6034.76 +- 141.42
1:n actor calls async per second 14032.19 +- 434.44
n:n actor calls async per second 36895.26 +- 642.94
n:n actor calls with arg async per second 9548.31 +- 218.19
1:1 async-actor calls sync per second 1031.86 +- 17.59
1:1 async-actor calls async per second 3675.11 +- 37.81
1:1 async-actor calls with args async per second 2268.17 +- 20.47
1:n async-actor calls async per second 11776.12 +- 641.59
n:n async-actor calls async per second 24358.79 +- 362.87
doc/dev/release_logs/1.0.1/microbenchmark.txt
-22
View File
@@ -1,22 +0,0 @@
single client get calls per second 30921.88 +- 291.82
single client put calls per second 26506.75 +- 278.09
multi client put calls per second 129520.73 +- 2304.11
single client get calls (Plasma Store) per second 7555.25 +- 109.32
single client put calls (Plasma Store) per second 4936.57 +- 31.22
multi client put calls (Plasma Store) per second 7459.17 +- 271.1
single client put gigabytes per second 11.87 +- 12.03
multi client put gigabytes per second 36.23 +- 2.17
single client tasks sync per second 1044.57 +- 16.2
single client tasks async per second 14319.28 +- 105.0
multi client tasks async per second 35140.53 +- 498.82
1:1 actor calls sync per second 1545.69 +- 25.38
1:1 actor calls async per second 6656.71 +- 268.57
1:1 actor calls concurrent per second 5851.38 +- 75.39
1:n actor calls async per second 11856.28 +- 322.15
n:n actor calls async per second 37089.65 +- 847.78
n:n actor calls with arg async per second 9919.05 +- 85.45
1:1 async-actor calls sync per second 1027.93 +- 31.65
1:1 async-actor calls async per second 3861.91 +- 96.54
1:1 async-actor calls with args async per second 2351.35 +- 48.13
1:n async-actor calls async per second 12652.4 +- 637.73
n:n async-actor calls async per second 24527.67 +- 743.48
doc/dev/release_logs/1.0.1/rllib_regression.txt
-47
View File
@@ -1,47 +0,0 @@
== Status ==
Memory usage on this node: 52.0/480.3 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/64 CPUs, 0.0/8 GPUs, 0.0/326.03 GiB heap, 0.0/99.17 GiB objects (0/1.0 accelerato
r_type:V100)
Result logdir: /home/ubuntu/ray_results/apex
Result logdir: /home/ubuntu/ray_results/atari-a2c
Result logdir: /home/ubuntu/ray_results/atari-basic-dqn
Result logdir: /home/ubuntu/ray_results/atari-impala
Result logdir: /home/ubuntu/ray_results/atari-ppo-tf
Result logdir: /home/ubuntu/ray_results/atari-ppo-torch
Number of trials: 24/24 (24 TERMINATED)
+-------------------------------------------+------------+-------+--------+------------------+---------+
----------+----------------------+----------------------+--------------------+
| Trial name | status | loc | iter | total time (s) | ts |
reward | episode_reward_max | episode_reward_min | episode_len_mean |
|-------------------------------------------+------------+-------+--------+------------------+---------+
----------+----------------------+----------------------+--------------------|
| IMPALA_BreakoutNoFrameskip-v4_4060e_00000 | TERMINATED | | 355 | 3606.4 | 9039500 |
413.62 | 802 | 244 | 11042.7 |
| IMPALA_BreakoutNoFrameskip-v4_4060e_00001 | TERMINATED | | 355 | 3605.53 | 9002500 | 393.79 | 497 | 159 | 10146.2 |
| IMPALA_BreakoutNoFrameskip-v4_4060e_00002 | TERMINATED | | 355 | 3606.64 | 9023250 | 393.07 | 438 | 89 | 9990.92 |
| IMPALA_BreakoutNoFrameskip-v4_4060e_00003 | TERMINATED | | 355 | 3606.81 | 9005250 | 374.35 | 758 | 60 | 9785.87 |
| PPO_BreakoutNoFrameskip-v4_4060e_00004 | TERMINATED | | 728 | 3601.7 | 3640000 | 63.98 | 399 | 8 | 2675.23 |
| PPO_BreakoutNoFrameskip-v4_4060e_00005 | TERMINATED | | 927 | 3604.18 | 4635000 | 34.93 | 70 | 7 | 3759.3 |
| PPO_BreakoutNoFrameskip-v4_4060e_00006 | TERMINATED | | 977 | 3602.5 | 4885000 | 43.26 | 243 | 14 | 3232.05 |
| PPO_BreakoutNoFrameskip-v4_4060e_00007 | TERMINATED | | 991 | 3602.53 | 4955000 | 51.55 | 288 | 7 | 3140.33 |
| PPO_BreakoutNoFrameskip-v4_4060e_00008 | TERMINATED | | 637 | 3603.22 | 3526735 | 32.71 | 57 | 9 | 3528.47 |
| PPO_BreakoutNoFrameskip-v4_4060e_00009 | TERMINATED | | 629 | 3603.5 | 3479261 | 66.78 | 323 | 10 | 3643.22 |
| PPO_BreakoutNoFrameskip-v4_4060e_00010 | TERMINATED | | 401 | 3600.46 | 2232016 | 33.61 | 272 | 14 | 2830.18 |
| PPO_BreakoutNoFrameskip-v4_4060e_00011 | TERMINATED | | 456 | 3602.14 | 2538206 | 11.61 | 36 | 4 | 1873.75 |
| APEX_BreakoutNoFrameskip-v4_4060e_00012 | TERMINATED | | 104 | 3622.05 | 5264160 | 26.93 | 62 | 11 | 2761.03 |
| APEX_BreakoutNoFrameskip-v4_4060e_00013 | TERMINATED | | 104 | 3610.29 | 5267200 | 37.55 | 374 | 8 | 2738.51 |
| APEX_BreakoutNoFrameskip-v4_4060e_00014 | TERMINATED | | 103 | 3629.06 | 5222880 | 18.86 | 150 | 8 | 2278.69 |
| APEX_BreakoutNoFrameskip-v4_4060e_00015 | TERMINATED | | 103 | 3601.19 | 5208640 | 24.86 | 377 | 6 | 2243.86 |
| A2C_BreakoutNoFrameskip-v4_4060e_00016 | TERMINATED | | 350 | 3608.5 | 2541000 | 95.47 | 336 | 11 | 4337.52 |
| A2C_BreakoutNoFrameskip-v4_4060e_00017 | TERMINATED | | 355 | 3603.43 | 4044000 |
144.96 | 325 | 40 | 5691.41 | | A2C_BreakoutNoFrameskip-v4_4060e_00018 | TERMINATED | | 355 | 3606.74 | 4326500 $
270.58 | 424 | 36 | 7014.73 | | A2C_BreakoutNoFrameskip-v4_4060e_00019 | TERMINATED | | 356 | 3607.42 | 4307500 |
264.4 | 412 | 38 | 7066.64 | | DQN_BreakoutNoFrameskip-v4_4060e_00020 | TERMINATED | | 33 | 3679.07 | 340000 |
15.15 | 37 | 7 | 1970.04 | | DQN_BreakoutNoFrameskip-v4_4060e_00021 | TERMINATED | | 33 | 3675.73 | 340000 |
16.44 | 40 | 6 | 2055.3 | | DQN_BreakoutNoFrameskip-v4_4060e_00022 | TERMINATED | | 33 | 3657.1 | 340000 |
18.85 | 40 | 6 | 2476.26 | | DQN_BreakoutNoFrameskip-v4_4060e_00023 | TERMINATED | | 33 | 3654.52 | 340000 |
21.07 | 47 | 8 | 2586.87 | +-------------------------------------------+------------+-------+--------+------------------+---------+
----------+----------------------+----------------------+--------------------+
2020-11-05 02:48:15,548 INFO tune.py:439 -- Total run time: 14646.45 seconds (14645.94 seconds for the t
uning loop).
doc/source/profiling.rst
+1 -1
View File
@@ -74,7 +74,7 @@ To run a set of single-node Ray microbenchmarks, use:
ray microbenchmark
You can find the microbenchmark results for Ray releases in the `GitHub release logs <https://github.com/ray-project/ray/tree/master/doc/dev/release_logs>`__.
You can find the microbenchmark results for Ray releases in the `GitHub release logs <https://github.com/ray-project/ray/tree/master/release/release_logs>`__.
References
----------