wassname/ray
1
0
Fork 0
mirror of https://github.com/wassname/ray.git synced 2026-07-20 12:40:20 +08:00
Code Issues Packages Projects Releases Wiki Activity
4,495 Commits 36 Branches 42 Tags
a77e5a8cbff7cd4789dd09a80f06ce65166c0a87
Commit Graph
52 Commits
Author SHA1 Message Date
Sven MikaandGitHub 499ad5fbe4 [RLlib] PyTorch version of APPO. (#8120) 
- Translate all vtrace functionality to torch and added torch to the framework_iterator-loop in all existing vtrace test cases.
- Add learning test cases for APPO torch (both w/ and w/o v-trace).
- Add quick compilation tests for APPO (tf and torch, v-trace and no v-trace).
 2020-04-23 09:11:12 +02:00
Sven MikaandGitHub d15609ba2a [RLlib] PyTorch version of ARS (Augmented Random Search). (#8106) 
This PR implements a PyTorch version of RLlib's ARS algorithm using RLlib's functional algo builder API. It also adds a regression test for ARS (torch) on CartPole.
 2020-04-21 09:47:52 +02:00
Sven MikaandGitHub 3812bfedda [RLlib] PyTorch version of ES (Evolution Strategies). (#8104) 
PyTorch version of Evolution Strategies (ES) Algo.
 2020-04-20 21:47:28 +02:00
Sven MikaandGitHub d0fab84e4d [RLlib] DDPG PyTorch version. (#7953) 
The DDPG/TD3 algorithms currently do not have a PyTorch implementation. This PR adds PyTorch support for DDPG/TD3 to RLlib.
This PR:
- Depends on the re-factor PR for DDPG (Functional Algorithm API).
- Adds learning regression tests for the PyTorch version of DDPG and a DDPG (torch)
- Updates the documentation to reflect that DDPG and TD3 now support PyTorch.

* Learning Pendulum-v0 on torch version (same config as tf). Wall time a little slower (~20% than tf).
* Fix GPU target model problem.
 2020-04-16 10:20:01 +02:00
Sven MikaandGitHub d2b5c171cb [RLlib] Add pytorch sigils to toc and add links to algo overview table. (#7950) 
* Add torch sigils to toc-tree for DQN/APEX.

* WIP.
 2020-04-09 10:40:18 -07:00
Sven MikaGitHubEric Liang
 		22ccc43670 [RLlib] DQN torch version. (#7597) 
* Fix.

* Rollback.

* WIP.

* WIP.

* WIP.

* WIP.

* WIP.

* WIP.

* WIP.

* WIP.

* Fix.

* Fix.

* Fix.

* Fix.

* Fix.

* WIP.

* WIP.

* Fix.

* Test case fixes.

* Test case fixes and LINT.

* Test case fixes and LINT.

* Rollback.

* WIP.

* WIP.

* Test case fixes.

* Fix.

* Fix.

* Fix.

* Add regression test for DQN w/ param noise.

* Fixes and LINT.

* Fixes and LINT.

* Fixes and LINT.

* Fixes and LINT.

* Fixes and LINT.

* Comment

* Regression test case.

* WIP.

* WIP.

* LINT.

* LINT.

* WIP.

* Fix.

* Fix.

* Fix.

* LINT.

* Fix (SAC does currently not support eager).

* Fix.

* WIP.

* LINT.

* Update rllib/evaluation/sampler.py

Co-Authored-By: Eric Liang <ekhliang@gmail.com>

* Update rllib/evaluation/sampler.py

Co-Authored-By: Eric Liang <ekhliang@gmail.com>

* Update rllib/utils/exploration/exploration.py

Co-Authored-By: Eric Liang <ekhliang@gmail.com>

* Update rllib/utils/exploration/exploration.py

Co-Authored-By: Eric Liang <ekhliang@gmail.com>

* WIP.

* WIP.

* Fix.

* LINT.

* LINT.

* Fix and LINT.

* WIP.

* WIP.

* WIP.

* WIP.

* Fix.

* LINT.

* Fix.

* Fix and LINT.

* Update rllib/utils/exploration/exploration.py

* Update rllib/policy/dynamic_tf_policy.py

Co-Authored-By: Eric Liang <ekhliang@gmail.com>

* Update rllib/policy/dynamic_tf_policy.py

Co-Authored-By: Eric Liang <ekhliang@gmail.com>

* Update rllib/policy/dynamic_tf_policy.py

Co-Authored-By: Eric Liang <ekhliang@gmail.com>

* Fixes.

* WIP.

* LINT.

* Fixes and LINT.

* LINT and fixes.

* LINT.

* Move action_dist back into torch extra_action_out_fn and LINT.

* Working SimpleQ learning cartpole on both torch AND tf.

* Working Rainbow learning cartpole on tf.

* Working Rainbow learning cartpole on tf.

* WIP.

* LINT.

* LINT.

* Update docs and add torch to APEX test.

* LINT.

* Fix.

* LINT.

* Fix.

* Fix.

* Fix and docstrings.

* Fix broken RLlib tests in master.

* Split BAZEL learning tests into cartpole and pendulum (reached the 60min barrier).

* Fix error_outputs option in BAZEL for RLlib regression tests.

* Fix.

* Tune param-noise tests.

* LINT.

* Fix.

* Fix.

* test

* test

* test

* Fix.

* Fix.

* WIP.

* WIP.

* WIP.

* WIP.

* LINT.

* WIP.

Co-authored-by: Eric Liang <ekhliang@gmail.com>
 2020-04-06 11:56:16 -07:00
Eric LiangandGitHub 5cebee68d6 [rllib] Add scaling guide to documentation, improve bandit docs (#7780) 
* update

* reword

* update

* ms

* multi node sgd

* reorder

* improve bandit docs

* contrib

* update

* ref

* improve refs

* fix build

* add pillow dep

* add pil

* update pil

* pillow

* remove false
 2020-03-27 22:05:43 -07:00
Saurabh GuptaandGitHub 6ddf84b019 Contextual Bandit algorithms (WIP) (#7642) 2020-03-26 13:41:16 -07:00
hubcityGitHubRichard Liaw
 		3d0a8662b3 #7246 - Fixing broken links (#7247) 
* #7246 - Fixing broken links

* Apply suggestions from code review

Co-authored-by: Richard Liaw <rliaw@berkeley.edu>
 2020-03-25 21:46:13 -07:00
Eric LiangandGitHub dd70720578 [rllib] Rename sample_batch_size => rollout_fragment_length (#7503) 
* bulk rename

* deprecation warn

* update doc

* update fig

* line length

* rename

* make pytest comptaible

* fix test

* fi sys

* rename

* wip

* fix more

* lint

* update svg

* comments

* lint

* fix use of batch steps
 2020-03-14 12:05:04 -07:00
Eric LiangandGitHub 026f6884b5 [rllib] Add Decentralized DDPPO trainer and documentation (#7088) 2020-02-10 15:28:27 -08:00
Sven MikaandGitHub 0e3960893a [RLlib] Add rainbow config hint to algo-documentation. (#7052) 2020-02-05 12:01:43 -08:00
Eric LiangandSimon Mo 6bb30c9f1b fix links (#6883) 2020-01-22 01:06:07 -08:00
Eric LiangandGitHub 14016535a5 [rllib] Add TF and Torch icons to show which are available for each algo (#6869) 2020-01-20 15:22:21 -08:00
Sven MikaandEric Liang 7659cae3ba [RLlib] Add PG torch regression test (#6828) 
* Add PG torch regression test to tuned_examples/regression_tests dir.

* Rename cartpole-pg.yaml into cartpole-pg-tf.yaml

* cartpole-pg-tf.yaml: Change cartpole-pg name of tuned_example to cartpole-pg-tf.
 2020-01-18 15:57:12 -08:00
Justin TerryandEric Liang 97bf79917c [RLlib] Update MADDPG example repo to maintained fork (#6831) 2020-01-18 13:08:27 -08:00
Michael LuoandEric Liang e5dded917c SAC site changes (#6759) 2020-01-09 18:13:42 -08:00
Michael LuoandEric Liang 1cb335487e SAC for Mujoco Environments (#6642) 2019-12-31 00:16:54 -08:00
Victor LeandEric Liang 4e24c805ee AlphaZero and Ranked reward implementation (#6385) 2019-12-07 12:08:40 -08:00
Eric LiangandGitHub 243b1b7281 [rllib] Add microbatch optimizer with A2C example (#6161) 2019-11-14 12:14:00 -08:00
Olli HuotariandRichard Liaw 0916603e61 Fixed few broken links in docs (#5477) 
* hyperband link changed

* tuned_examples link fix

* doc lstm link fix

* kubernetes example link fix
 2019-08-19 14:22:25 -07:00
Eric LiangandGitHub a1d2e17623 [rllib] Autoregressive action distributions (#5304) 2019-08-10 14:05:12 -07:00
Eric LiangandGitHub 592f313210 [rllib] Centralized critic / PPO example on TwoStepGame (#5392) 2019-08-08 14:03:28 -07:00
Eric LiangandGitHub 7d747da420 [rllib] [docs] Add some architecture diagrams (#5390) 2019-08-06 20:14:57 -07:00
Wonseok JeonandEric Liang 281829e712 MADDPG implementation in RLlib (#5348) 2019-08-06 16:22:06 -07:00
Eric LiangandGitHub 5d7afe8092 [rllib] Try moving RLlib to top level dir (#5324) 2019-08-05 23:25:49 -07:00
Eric LiangandGitHub 955154a19d Reduce Ray / RLlib startup messages (#5368) 2019-08-05 13:23:54 -07:00
Kristian HartikainenandEric Liang 13fb9fe3db [rllib] Feature/soft actor critic v2 (#5328) 
* Add base for Soft Actor-Critic

* Pick changes from old SAC branch

* Update sac.py

* First implementation of sac model

* Remove unnecessary SAC imports

* Prune unnecessary noise and exploration code

* Implement SAC model and use that in SAC policy

* runs but doesn't learn

* clear state

* fix batch size

* Add missing alpha grads and vars

* -200 by 2k timesteps

* doc

* lazy squash

* one file

* ignore tfp

* revert done
 2019-08-01 23:37:36 -07:00
Eric LiangandGitHub a62c5f40f6 [rllib] Document ModelV2 and clean up the models/ directory (#5277) 2019-07-27 02:08:16 -07:00
Eric LiangandGitHub 2dd0beb5bd [rllib] Allow access to batches prior to postprocessing (#4871) 2019-05-29 18:17:14 -07:00
Eric LiangandRichard Liaw 02583a8598 [rllib] Rename PolicyGraph => Policy, move from evaluation/ to policy/ (#4819) 
This implements some of the renames proposed in #4813
We leave behind backwards-compatibility aliases for *PolicyGraph and SampleBatch.
 2019-05-20 16:46:05 -07:00
Eric LiangandGitHub 7d5ef6d99c [rllib] Support continuous action distributions in IMPALA/APPO (#4771) 2019-05-16 22:05:07 -07:00
Sam ToyerandEric Liang 663e92ab3f [rllib] TD3/DDPG improvements and MuJoCo benchmarks (#4694) 
* [rllib] Separate optimisers for DDPG actor & crit.

* [rllib] Better names for DDPG variables & options

Config changes:

- noise_scale -> exploration_ou_noise_scale
- exploration_theta -> exploration_ou_theta
- exploration_sigma -> exploration_ou_sigma
- act_noise -> exploration_gaussian_sigma
- noise_clip -> target_noise_clip

* [rllib] Make DDPG less class-y

Used functions to replace three classes with only an __init__ method & a
handful of unrelated attributes.

* [rllib] Refactor DDPG noise

* [rllib] Unify DDPG exploration annealing

Added option "exploration_should_anneal" to enable linear annealing of
exploration noise. By default this is off, for consistency with DDPG &
TD3 papers. Also renamed "exploration_final_eps" to
"exploration_final_scale" (that name seems to have been carried over
from DQN, and doesn't really make sense here). Finally, tried to rename
"eps" to "noise_scale" wherever possible.
 2019-04-26 17:49:53 -07:00
Megan KawakamiandEric Liang 346885068c [rllib] add torch pg (#3857) 
* add torch pg

* add torch imports

* added torch pg

* working torch pg implementation

* add pg pytorch

* Update a3c.py

* Update a3c.py

* Update torch_policy_graph.py

* Update torch_policy_graph.py
 2019-02-16 19:54:14 -08:00
Michael LuoandEric Liang 1a015e420b Optimal PPO Configs (10k reward in 1 hr) + PPO grad clipping implemented (#3934) 2019-02-02 22:10:58 -08:00
Michael LuoandEric Liang 16f7ca45e4 Appo (#3779) 
* Deleted old fork, updated new ray and moved PPO-impala to APPO in ppo folder

* Deleted unneccesary vtrace.py file

* Update pong-impala.yaml

* Cleaned PPO Code

* Update pong-impala.yaml

* Update pong-impala.yaml

* wip

* new ifle

* refactor

* add vtrace off option

* revert

* support any space

* docs

* fix comment

* remove kl

* Update cartpole-appo-vtrace.yaml
 2019-01-18 13:40:26 -08:00
Jones WongandEric Liang 319c1340cb [rllib] Develop MARWIL (#3635) 
*  add marvil policy graph

*  fix typo

*  add offline optimizer and enable running marwil

*  fix loss function

*  add maintaining the moving average of advantage norm

*  use sync replay optimizer for unifying

*  remove offline optimizer and use sync replay optimizer

*  format by yapf

*  add imitation learning objective

*  fix according to eric's review

*  format by yapf

* revise

* add test data

* marwil
 2019-01-16 19:00:43 -08:00
Eric LiangandGitHub 47d36d7bd6 [rllib] Refactor pytorch custom model support (#3634) 2019-01-03 13:48:33 +08:00
Eric LiangandRichard Liaw 303883a3b6 [rllib] [rfc] add contrib module and guideline for merging (#3565) 
This adds guidelines for merging code into `rllib/contrib` vs `rllib/agents`. Also, clean up the agent import code to make registration easier.
 2018-12-20 10:44:34 -08:00
Eric LiangandGitHub db0dee573e [rllib] Q-Mix implementation (Q-Mix, VDN, IQN, and Ape-X variants) (#3548) 2018-12-18 10:40:01 -08:00
Eric LiangandGitHub f0df97db6f [rllib] example and docs on how to use parametric actions with DQN / PG algorithms (#3384) 2018-11-27 23:35:19 -08:00
Eric LiangandGitHub 8b76bab25c [rllib] docs for td3 (#3381) 
* td3 doc

* Update rllib-env.rst
 2018-11-22 13:36:47 -08:00
Eric LiangandGitHub abdc3b592e [rllib] Update multi-gpu impala numbers (#3327) 2018-11-19 20:55:27 -08:00
Eric LiangandGitHub a9e454f6fd [rllib] Include config dicts in the sphinx docs (#3064) 2018-10-16 15:55:11 -07:00
Eric LiangandGitHub 3c891c6ece [rllib] Parallel-data loading and multi-gpu support for IMPALA (#2766) 2018-10-15 11:02:50 -07:00
Eric LiangandGitHub b06c604a51 [rllib] Add some more tuned atari results to documentation (#2991) 
* dqn results ++

* add scale

* hour

* fix

* small dqn table

* update

* steps

* upd

* apex

* up

* add apex results

* tip
 2018-09-29 23:13:36 -07:00
Eric LiangandGitHub 69d1354016 [rllib] Document ARS & rainbow (#2744) 
* wip

* rainbow doc too

* e not used

* fix ppo doc

* clean list

* use same title
 2018-08-28 18:13:36 -07:00
Eric LiangandGitHub aa014af85b [rllib] Fix atari reward calculations, add LR annealing, explained var stat for A2C / impala (#2700) 
Changes needed to reproduce Atari plots in IMPALA / A2C: https://github.com/ray-project/rl-experiments
 2018-08-23 17:49:10 -07:00
Eric LiangandGitHub fbe6c59f72 [rllib] Misc fixes, A2C (#2679) 
A bunch of minor rllib fixes:

pull in latest baselines atari wrapper changes (and use deepmind wrapper by default)
move reward clipping to policy evaluator
add a2c variant of a3c
reduce vision network fc layer size to 256 units
switch to 84x84 images
doc tweaks
print timesteps in tune status
 2018-08-20 15:28:03 -07:00
Eric LiangandGitHub f7ec292360 [rllib] Support agent.get_action in multiagent (#2543) 
* support get action on policy id

* comment

* grammar fixes

* Update rllib-algorithms.rst
 2018-08-02 13:35:53 -07:00