wassname/ray
1
0
Fork 0
mirror of https://github.com/wassname/ray.git synced 2026-09-12 12:51:15 +08:00
Code Issues Packages Projects Releases Wiki Activity

[sgd] Add benchmarks (#7454)

Browse Source 
* Init fp16

* fp16 and schedulers

* scheduler linking and fp16

* to fp16

* loss scaling and documentation

* more documentation

* add tests, refactor config

* moredocs

* more docs

* fix logo, add test mode, add fp16 flag

* fix tests

* fix scheduler

* fix apex

* improve safety

* fix tests

* fix tests

* remove pin memory default

* rm

* fix

* Update doc/examples/doc_code/raysgd_torch_signatures.py

* fix

* migrate changes from other PR

* ok thanks

* pass

* signatures

* lint'

* Update python/ray/experimental/sgd/pytorch/utils.py

* Apply suggestions from code review

Co-Authored-By: Edward Oakes <ed.nmi.oakes@gmail.com>

* should address most comments

* comments

* fix this ci

* first_pass

* add overrides

* override

* fixing up operators

* format

* sgd

* constants

* rm

* revert

* save

* failures

* fixes

* trainer

* run test

* operator

* code

* op

* ok done

* operator

* sgd test fixes

* ok

* trainer

* format

* Apply suggestions from code review

Co-Authored-By: Edward Oakes <ed.nmi.oakes@gmail.com>

* Update doc/source/raysgd/raysgd_pytorch.rst

* docstring

* dcgan

* doc

* commits

* nit

* testing

* revert

* Start renaming pytorch to torch

* Rename PyTorchTrainer to TorchTrainer

* Rename PyTorch runners to Torch runners

* Finish renaming API

* Rename to torch in tests

* Finish renaming docs + tests

* Run format + fix DeprecationWarning

* fix

* move tests up

* benchmarks

* rename

* remove some args

* better metrics output

* fix up the benchmark

* benchmark-yaml

* horovod-benchmark

* benchmarks

* Remove benchmark code for cleanups

* benchmark-code

* nits

* benchmark yamls

* benchmark yaml

* ok

* ok

* ok

* benchmark

* nit

* finish_bench

* makedatacreator

* relax

* metrics

* autosetsampler

* profile

* movements

* OK

* smoothen

* fix

* nitdocs

* loss

* envflag

* comments

* nit

* format

* visible

* images

* move_images

* fix

* rernder

* rrender

* rest

* multgpu

* fix

* nit

* finish

* extrra

* setup

* revert

Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Co-authored-by: Maksim Smolin <maximsmol@gmail.com>
This commit is contained in:
Richard Liaw
2020-03-11 01:09:08 -07:00
committed by GitHub
co-authored by Edward Oakes Maksim Smolin
parent 49439611f1
commit fbac256982
12 changed files with 768 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/source/raysgd/raysgd_pytorch.rst
+47
View File
@@ -525,6 +525,53 @@ You can see the `DCGAN script <https://github.com/ray-project/ray/blob/master/py
stats = trainer.train()
Benchmarks
----------
RaySGD TorchTrainer provides comparable or better performance than other existing solutions for parallel or distributed training.
**Multi-GPU (Single Node) benchmarks**:
.. code-block:: bash
# Images per second for ResNet50
# Batch size per worker = 128
# GPU Type = V100
# Run on AWS us-east-1c, p3dn.24xlarge instance.
Number DataParallel Ray (PyTorch) DataParallel Ray (PyTorch)
of GPUs + Apex + Apex
======= ============ ============= ============ ==============
1 2769.7 5143 2962.7 6172
2 5492.2 9463 5886.1 10052.8
4 10733.4 18807 11705.9 20319.5
8 21872.5 36911.8 23317.9 38642
**Multi-node benchmarks**:
.. code-block:: bash
# Images per second for ResNet50
# Batch size per worker = 128
# GPU Type = V100
# Run on AWS us-east-1c, p3dn.24xlarge instances.
Number Horovod Ray (PyTorch) Horovod Ray (PyTorch)
of GPUs + Apex + Apex
======= ======= ============= ======= ==============
1 * 8 2769.7 5143 2962.7 6172
2 * 8 5492.2 9463 5886.1 10052.8
4 * 8 10733.4 18807 11705.9 20319.5
8 * 8 21872.5 36911.8 23317.9 38642
You can see more details in the `benchmarking README <https://github.com/ray-project/ray/blob/master/python/ray/util/sgd/torch/examples/benchmarks/README.rst>`_.
DISCLAIMER: RaySGD does not provide any custom communication primitives. If you see any performance issues, you may need to file them on the PyTorch github repository.
Feature Requests
----------------