wassname/ray
1
0
Fork 0
mirror of https://github.com/wassname/ray.git synced 2026-07-20 12:40:20 +08:00
Code Issues Packages Projects Releases Wiki Activity
2,924 Commits 36 Branches 42 Tags
d2f5b71c3bc93e5c9727d3be534ffce2f61277af
Commit Graph
73 Commits
Author SHA1 Message Date
Eric LiangandGitHub 9e328fbe6f [rllib] Add docs on how to use TF eager execution (#4927) 2019-06-07 16:42:37 -07:00
Eric LiangandGitHub 1c073e92e4 [rllib] Fix documentation on custom policies (#4910) 
* wip

* add docs

* lint

* todo sections

* fix doc
 2019-06-01 16:13:21 +08:00
Eric LiangandGitHub a45c61e19b [rllib] Update concepts docs and add "Building Policies in Torch/TensorFlow" section (#4821) 
* wip

* fix index

* fix bugs

* todo

* add imports

* note on get ph

* note on get ph

* rename to building custom algs

* add rnn state info
 2019-05-27 14:17:32 -07:00
Eric LiangandRichard Liaw 02583a8598 [rllib] Rename PolicyGraph => Policy, move from evaluation/ to policy/ (#4819) 
This implements some of the renames proposed in #4813
We leave behind backwards-compatibility aliases for *PolicyGraph and SampleBatch.
 2019-05-20 16:46:05 -07:00
Eric LiangandGitHub 6e7680bf21 [rllib] Clean up concepts documentation and policy optimizer creation (#4592) 2019-04-12 21:03:26 -07:00
Eric LiangandGitHub 37208216ae [rllib] Rename Agent to Trainer (#4556) 2019-04-07 00:36:18 -07:00
Eric LiangandGitHub 2871609296 [rllib] Report sampler performance metrics (#4427) 2019-03-27 13:24:23 -07:00
Eric LiangandGitHub 4b8b703561 [rllib] Some API cleanups and documentation improvements (#4409) 2019-03-21 21:34:22 -07:00
Eric LiangandGitHub c7f74dbdc7 [rllib] Add async remote workers (#4253) 2019-03-08 15:39:48 -08:00
Eric LiangandRobert Nishihara 78ad9c4cbb Add "ray timeline" command to auto-dump Chrome trace for the current Ray instance (#4239) 2019-03-05 16:28:00 -08:00
Eric LiangandPhilipp Moritz 3896b726dd Dynamically adjust redis memory usage (#4152) 
* f

* Update services.py
 2019-02-25 16:21:37 -08:00
Eric LiangandGitHub d9da183c7d [rllib] Custom supervised loss API (#4083) 2019-02-24 15:36:13 -08:00
Eric LiangandGitHub 152375aa8a [rllib] Add evaluation option to DQN agent (#3835) 
* add eval

* interval

* multiagent minor fix

* Update rllib.rst

* Update ddpg.py

* Update qmix.py
 2019-01-29 21:19:53 -08:00
Eric LiangandGitHub fb73cedf70 [rllib] Add examples page, add hierarchical training example, delete SC2 examples (#3815) 
* wip

* lint

* wip

* up

* wip

* update examples

* wip

* remove carla

* update

* improve envspec

* link to custom

* Update rllib-env.rst

* update

* fix

* fn

* lint

* ds

* ssd games

* desc

* fix up docs

* fix
 2019-01-29 21:06:09 -08:00
Michael LuoandEric Liang 16f7ca45e4 Appo (#3779) 
* Deleted old fork, updated new ray and moved PPO-impala to APPO in ppo folder

* Deleted unneccesary vtrace.py file

* Update pong-impala.yaml

* Cleaned PPO Code

* Update pong-impala.yaml

* Update pong-impala.yaml

* wip

* new ifle

* refactor

* add vtrace off option

* revert

* support any space

* docs

* fix comment

* remove kl

* Update cartpole-appo-vtrace.yaml
 2019-01-18 13:40:26 -08:00
Jones WongandEric Liang 319c1340cb [rllib] Develop MARWIL (#3635) 
*  add marvil policy graph

*  fix typo

*  add offline optimizer and enable running marwil

*  fix loss function

*  add maintaining the moving average of advantage norm

*  use sync replay optimizer for unifying

*  remove offline optimizer and use sync replay optimizer

*  format by yapf

*  add imitation learning objective

*  fix according to eric's review

*  format by yapf

* revise

* add test data

* marwil
 2019-01-16 19:00:43 -08:00
Eric LiangandRichard Liaw 401e656b95 [rllib] Sync filters at end of iteration not start; hierarchical docs (#3769) 2019-01-15 16:25:25 -08:00
Eric LiangandGitHub 03fe760616 [rllib] Model self loss isn't included in all algorithms (#3679) 2019-01-04 22:30:35 -08:00
Eric LiangandGitHub ca864faece [rllib] Documentation for I/O API and multi-agent support / cleanup (#3650) 2019-01-03 15:15:36 +08:00
Eric LiangandGitHub 47d36d7bd6 [rllib] Refactor pytorch custom model support (#3634) 2019-01-03 13:48:33 +08:00
Eric LiangandGitHub 9f63119a83 [rllib] Allow development without needing to compile Ray (#3623) 
* wip

* lint

* wip

* wip

* rename

* wip

* Cleaner handling of cli prompt
 2018-12-24 18:08:23 +09:00
Eric LiangandRichard Liaw 303883a3b6 [rllib] [rfc] add contrib module and guideline for merging (#3565) 
This adds guidelines for merging code into `rllib/contrib` vs `rllib/agents`. Also, clean up the agent import code to make registration easier.
 2018-12-20 10:44:34 -08:00
Eric LiangandGitHub db0dee573e [rllib] Q-Mix implementation (Q-Mix, VDN, IQN, and Ape-X variants) (#3548) 2018-12-18 10:40:01 -08:00
Eric LiangandGitHub f0df97db6f [rllib] example and docs on how to use parametric actions with DQN / PG algorithms (#3384) 2018-11-27 23:35:19 -08:00
Eric LiangandRobert Nishihara 0d56fc10cc Move setproctitle to ray[debug] package (#3415) 2018-11-27 09:50:59 -08:00
Eric LiangandGitHub 8b76bab25c [rllib] docs for td3 (#3381) 
* td3 doc

* Update rllib-env.rst
 2018-11-22 13:36:47 -08:00
Eric LiangandGitHub 706dc1d473 [rllib] Add test for multi-agent support and fix IMPALA multi-agent (#3289) 
IMPALA support for multiagent was broken since IMPALA has a requirement that batch sizes be of a certain length. However multi-agent envs can create variable-length batches.

Fix this by adding zero-padding as needed (similar to the RNN case).
 2018-11-14 14:14:07 -08:00
Eric LiangandGitHub bd0dbde149 [rllib] Rename ServingEnv => ExternalEnv (#3302) 2018-11-12 16:31:27 -08:00
Eric LiangandRichard Liaw 9dd3eedbac [rllib] rollout.py should reduce num workers (#3263) 
## What do these changes do?

Don't create an excessive amount of workers for rollout.py, and also fix up the env wrapping to be consistent with the internal agent wrapper.

## Related issue number

Closes #3260.
 2018-11-09 12:29:16 -08:00
Eric LiangandGitHub 369cb833fe [rllib] Implement custom metrics (#3144) 2018-11-03 18:48:32 -07:00
Eric LiangandRobert Nishihara 9a0f0db070 Add ray stack tool for debugging (#3213) 2018-11-03 13:13:02 -07:00
Eric LiangandGitHub a9e454f6fd [rllib] Include config dicts in the sphinx docs (#3064) 2018-10-16 15:55:11 -07:00
Eric LiangandGitHub b45bed4bce [rllib] Propagate model options correctly in ARS / ES, to action dist of PPO (#2974) 
* fix

* fix

* fix it

* propagate conf to action dist

* move carla example too

* rr

* Update policies.py

* wip

* lint
 2018-10-01 12:49:39 -07:00
Eric LiangandGitHub 69d1354016 [rllib] Document ARS & rainbow (#2744) 
* wip

* rainbow doc too

* e not used

* fix ppo doc

* clean list

* use same title
 2018-08-28 18:13:36 -07:00
Eric LiangandGitHub fbe6c59f72 [rllib] Misc fixes, A2C (#2679) 
A bunch of minor rllib fixes:

pull in latest baselines atari wrapper changes (and use deepmind wrapper by default)
move reward clipping to policy evaluator
add a2c variant of a3c
reduce vision network fc layer size to 256 units
switch to 84x84 images
doc tweaks
print timesteps in tune status
 2018-08-20 15:28:03 -07:00
Eric LiangandGitHub 5f430da180 [rllib] Provide internal access to episode state in compute_actions() and allow returning extra batches (#2559) 
The goal of this PR is to allow custom policies to perform model-based rollouts. In the multi-agent setting, this requires access to not only policies of other agents, but also their current observations.
Also, you might want to return the model-based trajectories as part of the rollout for efficiency.

  compute_actions() now takes a new keyword arg episodes
  pull out internal episode class into a top-level file
  add function to return extra trajectories from an episode that will be appended to the sample batch
  documentation
 2018-08-16 14:37:21 -07:00
Eric LiangandGitHub 079c4e482a ray exec and ray attach commands (#2560) 
ray exec CLUSTER CMD [--screen] [--start] [--stop]
ray attach CLUSTER [--start]

Example:
ray exec sgd.yaml 'source activate tensorflow_p27 && cd ~/ray/python/ray/rllib && ./train.py --run=PPO --env=CartPole-v0' --screen --start --stop

This will in one command create a cluster and run the command on it in a screen session. The screen can later be attached to via ray attach. After the command finishes, the cluster workers will be terminated and the head node stopped.
 2018-08-15 14:31:50 -07:00
Eric LiangandGitHub 9ea57c2a93 [rllib] Basic IMPALA implementation (using deepmind's reference vtrace.py) (#2504) 
Rename AsyncSamplesOptimizer -> AsyncReplayOptimizer
  Add AsyncSamplesOptimizer that implements the IMPALA architecture
  integrate V-trace with a3c policy graph
  audit V-trace integration
  benchmark compare vs A3C and with V-trace on/off
PongNoFrameskip-v4 on IMPALA scaling from 16 to 128 workers, solving Pong in <10 min. For reference, solving this env takes ~40 minutes for Ape-X and several hours for A3C.
 2018-08-01 20:53:53 -07:00
Sergey KolesnikovandEric Liang 05490b8cb9 [rllib] dqn/ddpg policy customization (#2445) 
* dqn policy update - more customization

* docs for custom DQN graph

* Update rllib-training.rst

* Update rllib-models.rst

* Update rllib.rst

* Update rllib-training.rst

* Update rllib-concepts.rst

* yapf codestyle
 2018-07-22 14:47:14 -07:00
Eric LiangandGitHub 4ef9d15315 [rllib] Add concepts section of docs (#2373) 
This fills in the rllib concepts documentation.
 2018-07-08 18:46:52 -07:00
Eric LiangandGitHub 8aa56c12e6 [rllib] Document "v2" APIs (#2316) 
* re

* wip

* wip

* a3c working

* torch support

* pg works

* lint

* rm v2

* consumer id

* clean up pg

* clean up more

* fix python 2.7

* tf session management

* docs

* dqn wip

* fix compile

* dqn

* apex runs

* up

* impotrs

* ddpg

* quotes

* fix tests

* fix last r

* fix tests

* lint

* pass checkpoint restore

* kwar

* nits

* policy graph

* fix yapf

* com

* class

* pyt

* vectorization

* update

* test cpe

* unit test

* fix ddpg2

* changes

* wip

* args

* faster test

* common

* fix

* add alg option

* batch mode and policy serving

* multi serving test

* todo

* wip

* serving test

* doc async env

* num envs

* comments

* thread

* remove init hook

* update

* fix ppo

* comments1

* fix

* updates

* add jenkins tests

* fix

* fix pytorch

* fix

* fixes

* fix a3c policy

* fix squeeze

* fix trunc on apex

* fix squeezing for real

* update

* remove horizon test for now

* multiagent wip

* update

* fix race condition

* fix ma

* t

* doc

* st

* wip

* example

* wip

* working

* cartpole

* wip

* batch wip

* fix bug

* make other_batches None default

* working

* debug

* nit

* warn

* comments

* fix ppo

* fix obs filter

* update

* wip

* tf

* update

* fix

* cleanup

* cleanup

* spacing

* model

* fix

* dqn

* fix ddpg

* doc

* keep names

* update

* fix

* com

* docs

* clarify model outputs

* Update torch_policy_graph.py

* fix obs filter

* pass thru worker index

* fix

* rename

* vlad torch comments

* fix log action

* debug name

* fix lstm

* remove unused ddpg net

* remove conv net

* revert lstm

* wip

* wip

* cast

* wip

* works

* fix a3c

* works

* lstm util test

* doc

* clean up

* update

* fix lstm check

* move to end

* fix sphinx

* fix cmd

* remove bad doc

* envs

* vec

* doc prep

* models

* rl

* alg

* up

* clarify

* copy

* async sa

* fix

* comments

* fix a3c conf

* tune lstm

* fix reshape

* fix

* back to 16

* tuned a3c update

* update

* tuned

* optional

* merge

* wip

* fix up

* move pg class

* rename env

* wip

* update

* tip

* alg

* readme

* fix catalog

* readme

* doc

* context

* remove prep

* comma

* add env

* link to paper

* paper

* update

* rnn

* update

* wip

* clean up ev creation

* fix

* fix

* fix

* fix lint

* up

* no comma

* ma

* Update run_multi_node_tests.sh

* fix

* sphinx is stupid

* sphinx is stupid

* clarify torch graph

* no horizon

* fix config

* sb

* Update test_optimizers.py
 2018-07-01 00:05:08 -07:00
Richard LiawandGitHub f19decb848 [docs] Update RLlib install to not include Tensorflow (#2178) 2018-06-10 10:29:12 -07:00
andrewztanandRichard Liaw 1475600c81 [rllib] Merge DDPG and DDPG2 implementations (#2202) 
* removed ddpg2

* removed ddpg2 from codebase

* added tests used in ddpg vs ddpg2 comparison

* added notes about training timesteps to yaml files

* removed ddpg2 yaml files

* removed unnecessary configs from yaml files

* removed unnecessary configs from yaml files

* moved pendulum, mountaincarcontinuous, and halfcheetah tests to tuned_examples

* moved pendulum, mountaincarcontinuous, and halfcheetah tests to tuned_examples

* added more configuration details to yaml files

* removed random starts from halfcheetah
 2018-06-09 16:46:23 -07:00
Eric LiangandGitHub f37e2e5d2f [rllib] [doc] Broken link in ddpg doc 2018-05-20 00:10:59 -07:00
Eric LiangandPhilipp Moritz b55f4a7f04 [rllib] Fix broken link in docs (#1967) 
* Update README.rst

* Update rllib.rst
 2018-04-30 16:02:48 -07:00
Eric LiangandGitHub 47bc4c3009 [rllib] Add DDPG documentation, rename DDPG2 <=> DDPG (#1946) 
* updates

* updates

* updates

* updates

* updates

* updates

* Update rllib.rst

* Update policy-optimizers.rst
 2018-04-30 00:18:15 -07:00
Eric LiangandRichard Liaw 7ab890f4a1 [tune] [rllib] Automatically determine RLlib resources and add queueing mechanism for autoscaling (#1848) 2018-04-16 16:58:15 -07:00
Eric LiangandPhilipp Moritz 72595cca0d [tune] Change tune resource request syntax to be less confusing (#1764) 
* update

* update examples

* Wed Mar 21 15:19:56 PDT 2018

* Wed Mar 21 15:21:32 PDT 2018

* Update train_a3c.py

* Update train.py

* fix resources accounting
 2018-03-23 06:25:01 -07:00
Eric LiangandGitHub e3685fca5e [rllib] remove redundant docs (#1728) 
* wip

* more work

* fix apex

* docs

* apex doc

* pool comment

* clean up

* make wrap stack pluggable

* Mon Mar 12 21:45:50 PDT 2018

* clean up comment

* table

* Mon Mar 12 22:51:57 PDT 2018

* Mon Mar 12 22:53:05 PDT 2018

* Mon Mar 12 22:55:03 PDT 2018

* Mon Mar 12 22:56:18 PDT 2018

* Mon Mar 12 22:59:54 PDT 2018

* Update apex_optimizer.py

* Update index.rst

* Update README.rst

* Update README.rst

* comments

* Wed Mar 14 19:01:02 PDT 2018

* Fri Mar 16 15:44:27 PDT 2018
 2018-03-17 14:45:04 -07:00
Eric LiangandGitHub 882a649f0c [rllib] [docs] Cleanup RLlib API and make docs consistent with upcoming blog post (#1708) 
* wip

* more work

* fix apex

* docs

* apex doc

* pool comment

* clean up

* make wrap stack pluggable

* Mon Mar 12 21:45:50 PDT 2018

* clean up comment

* table

* Mon Mar 12 22:51:57 PDT 2018

* Mon Mar 12 22:53:05 PDT 2018

* Mon Mar 12 22:55:03 PDT 2018

* Mon Mar 12 22:56:18 PDT 2018

* Mon Mar 12 22:59:54 PDT 2018

* Update apex_optimizer.py

* Update index.rst

* Update README.rst

* Update README.rst

* comments

* Wed Mar 14 19:01:02 PDT 2018
 2018-03-15 15:57:31 -07:00