Commit Graph
215 Commits
Author SHA1 Message Date
ijrsvtandGitHub 2116fd3bca Removing Pyarrow dependency (#7146) 2020-02-17 18:00:13 -08:00
mehrdadnandGitHub 3bd82d0bcd Fix various issues/warnings that come up on Jenkins (#7147)
* Avoid warning about swap being unlimited

Currently we get the following message on Jenkins:
"Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap."

Since we're not limiting swap anyway, we might as well avoid trying to.
https://docs.docker.com/config/containers/resource_constraints/#--memory-swap-details

* Fix escaping in re.search()

* Fix escaping in _noisy_layer()

* Raise a more descriptive error when dashboard data isn't found

* Don't error on dashboard files not being found when webui isn't required

* Change dashboard error to a warning instead
2020-02-17 16:08:55 -08:00
94e2fcea2e [sgd] fp16 (apex) and scheduler support + move examples page (#7061)
* Init fp16

* fp16 and schedulers

* scheduler linking and fp16

* to fp16

* loss scaling and documentation

* more documentation

* add tests, refactor config

* moredocs

* more docs

* fix logo, add test mode, add fp16 flag

* fix tests

* fix scheduler

* fix apex

* improve safety

* fix tests

* fix tests

* remove pin memory default

* rm

* fix

* Update doc/examples/doc_code/raysgd_torch_signatures.py

* fix

* migrate changes from other PR

* ok thanks

* pass

* signatures

* lint'

* Update python/ray/experimental/sgd/pytorch/utils.py

* Apply suggestions from code review

Co-Authored-By: Edward Oakes <ed.nmi.oakes@gmail.com>

* should address most comments

* comments

* fix this ci

* fix tests'

* testmode

Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>
2020-02-16 19:04:08 -08:00
Eric LiangandGitHub b7016504e8 [rllib] Only run one set of tests unless rllib or tune dirs are changed. (#7179)
* full filter

* lint
2020-02-16 08:52:49 -08:00
Sven MikaandGitHub 2e60f0d4d8 [RLlib] Move all jenkins RLlib-tests into bazel (rllib/BUILD). (#7178)
* commit

* comment
2020-02-15 14:50:44 -08:00
Simon MoandGitHub 30de1286bd Use pip install setup.py (#7158) 2020-02-14 13:53:36 -08:00
Edward OakesandGitHub b81b93a9c0 Convert stress tests to projects (#6495) 2020-02-13 09:19:24 -08:00
Sven MikaandGitHub 5518a738b3 [RLlib] Fix erroneous use of LinearSchedule (in DDPG's exploration annealing). (#7125)
* Fix erroneous use of LinearSchedule (in DDPG's exploration annealing).
Erase schedules_obsoleted.py.

* Trigger re-test.

* Re-test.
2020-02-12 23:46:49 -08:00
Edward OakesandGitHub 275fd343fb Change CI to properly list python3.6 (#7126) 2020-02-12 11:15:46 -08:00
mehrdadnandGitHub e09f63ad65 Fix build errors and add more targets to Windows builds (#6811)
* Fix common.fbs rename (due to apache/arrow/commit/bef9a1c251397311a6415d3dc362ef419d154caa)

* Add missing COPTS

* Use socketpair(AF_INET) if boost::asio::local is unavailable (e.g. on Windows)

* Fix compile bug in service_based_gcs_client_test.cc (fix build breakage in #6686)

* Work around googletest/gmock inability to specify override to avoid -Werror,-Winconsistent-missing-override

* Fix missing override on IsPlasmaBuffer()

* Fix missing libraries for streaming

* Factor out install-toolchains.sh

* Put some Bazel flags into .bazelrc

* Fix jni_md.h missing inclusion

* Add ~/bin to PATH for Bazel

* Change echo $$(date) > $@ to date > $@

* Fix lots of unquoted paths

* Add system() call checks for Windows

Co-authored-by: GitHub Web Flow <noreply@github.com>
2020-02-11 16:49:33 -08:00
Simon MoandGitHub 0ddc389830 Fix documentation building with psutil issue (#7077) 2020-02-11 10:00:29 -08:00
Simon MoandGitHub bec92a8946 [Hotfix] Fix flake8 lint failing (#7118) 2020-02-10 19:57:21 -08:00
mehrdadnandGitHub ad1848b623 Comment iwyu.sh (#7110)
Co-authored-by: GitHub Web Flow <noreply@github.com>
2020-02-10 15:28:36 -08:00
Sven MikaandGitHub 6e1c3ea824 [RLlib] Exploration API (+EpsilonGreedy sub-class). (#6974) 2020-02-10 15:22:07 -08:00
Simon MoandGitHub f6c09ff614 Add serve stress test (#7076) 2020-02-10 09:37:39 -08:00
mehrdadnandGitHub ad4ac9aa70 Add clang-iwyu (#7081)
* Add iwyu

Co-authored-by: GitHub Web Flow <noreply@github.com>
2020-02-07 16:19:46 -08:00
ijrsvtandGitHub 0826f95e1c Including psutil & setproctitle (#7031) 2020-02-05 14:16:58 -08:00
Eric LiangandGitHub fbc545c03b [rllib] Support parallel, parameterized evaluation (#6981)
* eval api

* update

* sync eval filters

* sync fix

* docs

* update

* docs

* update

* link

* nit

* doc updates

* format
2020-02-01 22:12:12 -08:00
SangBin ChoandGitHub c9f5def56a Show lint download commands if tools not installed (#6984) 2020-01-31 10:42:09 -08:00
Ameer Haj AliandGitHub b8135da122 Adding dependencies for scikit-learn in travis (#6969)
* Revert "Revert "Support of scikit-learn with ray joblib backend (#6925)" (#6957)"

This reverts commit 86100bc119.

* adding scikit-learn to dependencies
2020-01-30 09:46:54 -08:00
Amog KamsettyandGitHub 11d90d6d0c Change files tested in Travis by changing git diff from 2-dot to 3-dot (#6960). 2020-01-30 09:26:44 -08:00
Richard LiawandGitHub 037aa2b961 [sgd] Refactor PyTorch SGD Documentation. (#6910)
* Refactor documentation and directory structurre

* update loss

* ,ore examples

* fix comments

* more code

* svgs

* formatting

* more_docs

* more writing

* comments ready

* move

* whitespace

* examples

* fix

* bold

* pytorch

* batch

* fix

* fix test

* Apply suggestions from code review

* quarantinegp

* tests/

* fix missing
2020-01-29 08:51:01 -08:00
Sven MikaandEric Liang 446cbdf2e0 [RLlib] Fix issue (bug): LSTM + non-shared vf + PPO + tuple actions (#6890)
* Add `RandomEnv` example to examples folder.
Convert warning into Error message when using an LSTM in a non-shared-vf network (after the warning, the program would crash).

* LINT.

* Fix issue #6884. LSTM + non-shared vf NN + PPO crashes when using a Tuple action space.

* LINT

* Change warning message for Model: shared_vf=False, LSTM=True cases.

* Bug fix.

* Add examples/random_env.py test to Jenkins.
2020-01-24 10:29:35 -08:00
Sven MikaandEric Liang ae9a3a2237 [RLlib] from_config util method for framework agnostic components; start moving RLlib tests into Bazel. (#6865) 2020-01-22 17:02:58 -08:00
Sven MikaandEric Liang c957ed58ed [RLlib] Implement PPO torch version. (#6826) 2020-01-20 23:06:50 -08:00
Edward OakesandGitHub b750bd7fc9 Use 2xlarge instances in long running tests (#6802) 2020-01-15 19:47:59 -06:00
mehrdadnandPhilipp Moritz 4780b52ea8 Polish clang-format and let it run in a local repo (#6793) 2020-01-15 11:17:49 -08:00
Edward OakesandGitHub fc473e6a08 Use us-west-2 for application stress tests (#6782) 2020-01-13 15:01:03 -06:00
chaokunyangandQing Wang 723fe86882 [Java] Fix building Java with maven (#6764)
* lint

* gen_maven_deps for ray_java_pkg
2020-01-11 14:26:27 +08:00
SvenandEric Liang 60d4d5e1aa Remove future imports (#6724)
* Remove all __future__ imports from RLlib.

* Remove (object) again from tf_run_builder.py::TFRunBuilder.

* Fix 2xLINT warnings.

* Fix broken appo_policy import (must be appo_tf_policy)

* Remove future imports from all other ray files (not just RLlib).

* Remove future imports from all other ray files (not just RLlib).

* Remove future import blocks that contain `unicode_literals` as well.
Revert appo_tf_policy.py to appo_policy.py (belongs to another PR).

* Add two empty lines before Schedule class.

* Put back __future__ imports into determine_tests_to_run.py. Fails otherwise on a py2/print related error.
2020-01-09 00:15:48 -08:00
Edward OakesandGitHub 5f843cd998 Clean up stress_testing_config.yaml (#6738)
* Clean up stress_testing_config.yaml

* comment
2020-01-07 17:05:07 -06:00
SvenandEric Liang f1b56fa5ee PG unify/cleanup tf vs torch and PG functionality test cases (tf + torch). (#6650)
* Unifying the code for PGTrainer/Policy wrt tf vs torch.
Adding loss function test cases for the PGAgent (confirm equivalence of tf and torch).

* Fix LINT line-len errors.

* Fix LINT errors.

* Fix `tf_pg_policy` imports (formerly: `pg_policy`).

* Rename tf_pg_... into pg_tf_... following <alg>_<framework>_... convention, where ...=policy/loss/agent/trainer.
Retire `PGAgent` class (use PGTrainer instead).

* - Move PG test into agents/pg/tests directory.
- All test cases will be located near the classes that are tested and
  then built into the Bazel/Travis test suite.

* Moved post_process_advantages into pg.py (from pg_tf_policy.py), b/c
the function is not a tf-specific one.

* Fix remaining import errors for agents/pg/...

* Fix circular dependency in pg imports.

* Add pg tests to Jenkins test suite.
2020-01-02 16:08:03 -08:00
mehrdadnandPhilipp Moritz f4b29dae9c Perform Bazel install directly in Windows CI (#6653) 2019-12-31 20:48:08 -08:00
Robert NishiharaandGitHub d2c6457832 Remove public facing references to --redis-address. (#6631) 2019-12-31 13:21:53 -08:00
Philipp MoritzandGitHub 735f282494 Use 0.9.0.dev0 as the version tag (#6630) 2019-12-30 10:14:07 -08:00
Robert NishiharaandPhilipp Moritz 96f2f8ff10 Stop testing Python 2.7 and building Python 2.7 wheels. (#6601) 2019-12-27 20:47:49 -08:00
Robert NishiharaandGitHub eb0813ea35 Re-enable UI tests for wheels. (#6602) 2019-12-26 22:34:56 -08:00
Philipp MoritzandGitHub eaee672b7f Revert "Perform Bazel install directly in Windows CI (#6529)" (#6593)
This reverts commit c5f141013b.
2019-12-24 16:39:07 -08:00
micafanandHao Chen 687de41273 [GCS] refactor the GCS Client Node Interface (#6010) 2019-12-24 20:36:37 +08:00
mehrdadnandPhilipp Moritz c5f141013b Perform Bazel install directly in Windows CI (#6529) 2019-12-22 16:14:51 -08:00
Chaokun YangandHao Chen 7bbfa85c66 [Streaming] Streaming data transfer java (#6474) 2019-12-22 10:56:05 +08:00
Simon MoandGitHub 26ec500ef9 Implement async get for direct actor call (#6339) 2019-12-18 11:50:21 -08:00
Eric LiangandGitHub 6725a61bda Release 0.8.0 test logs (#6512) 2019-12-17 15:56:50 -08:00
Eric LiangandGitHub 1a1324d2a2 Bump version from 0.8.0.dev6 -> 0.9.0.dev (#6508) 2019-12-16 23:57:42 -08:00
Edward OakesandGitHub b1e83d83d1 Print summaries for stress tests (#6498) 2019-12-16 14:14:48 -08:00
Mitchell SternandSimon Mo 1531c21dbd [Dashboard] Add remaining features from old dashboard (#6489)
* [Dashboard] Add remaining features from old dashboard

* Fix linting errors

* Set cluster uptime statistic to N/A

* Use proper singular or plural words for workers column

* Ignore .js, .jsx, .ts, .tsx files in check-git-clang-format-output.sh

* Fix bash quote issue
2019-12-16 11:21:18 -08:00
Richard LiawandGitHub 5719a05757 [sgd] Add support for multi-model multi-optimizer training (#6317) 2019-12-15 15:19:45 -08:00
Philipp MoritzandGitHub f5d10eea0b [Projects] Refactor cluster specification (#6488) 2019-12-14 22:43:06 -08:00
Yuhao YangandRichard Liaw ad4da17899 [Tune] Add example and tutorial for DCGAN (#6400) 2019-12-13 14:15:44 -08:00
Eric LiangandGitHub be5dd8eb5e Enable direct calls by default (#6367)
* wip

* add

* timeout fix

* const ref

* comments

* fix

* fix

* Move actor state into actor handle

* comments 2

* enable by default

* temp reorder

* some fixes

* add debug code

* tmp

* fix

* wip

* remove dbg

* fix compile

* fix

* fix check

* remove non direct tests

* Increment ref count before resolving value

* rename

* fix another bug

* tmp

* tmp

* Fix object pinning

* build change

* lint

* ActorManager

* tmp

* ActorManager

* fix test component failures

* Remove old code

* Remove unused

* fix

* fix

* fix resources

* fix advanced

* eric's diff

* blacklist

* blacklist

* cleanup

* annotate

* disable tests for now

* remove

* fix

* fix

* clean up verbosity

* fix test

* fix concurrency test

* Update .travis.yml

* Update .travis.yml

* Update .travis.yml

* split up analysis suite

* split up trial runner suite

* fix detached direct actors

* fix

* split up advanced tesT

* lint

* fix core worker test hang

* fix bad check fail which breaks test_cluster.py in tune

* fix some minor diffs in test_cluster

* less workers

* make less stressful

* split up test

* retry flaky tests

* remove old test flags

* fixes

* lint

* Update worker_pool.cc

* fix race

* fix

* fix bugs in node failure handling

* fix race condition

* fix bugs in node failure handling

* fix race condition

* nits

* fix test

* disable heartbeatS

* disable heartbeatS

* fix

* fix

* use worker id

* fix max fail

* debug exit

* fix merge, and apply [PATCH] fix concurrency test

* [patch] fix core worker test hang

* remove NotifyActorCreation, and return worker on completion of actor creation task

* remove actor diied callback

* Update core_worker.cc

* lint

* use task manager

* fix merge

* fix deadlock

* wip

* merge conflits

* fix

* better sysexit handling

* better sysexit handling

* better sysexit handling

* check id

* better debug

* task failed msg

* task failed msg

* retry failed tasks with delay

* retry failed tasks with delay

* clip deps

* fix

* fix core worker tests

* fix task manager test

* fix all tests

* cleanup

* set to 0 for direct tests

* dont check worker id for ownership rpc

* dont check worker id for ownership rpc

* debug messages

* add comment

* remove debug statements

* nit

* check worker id

* fix test

* owner

* fix tests
2019-12-13 13:58:04 -08:00