914a433e3f
[tune] Split Search from Scheduling ( #2452 )
...
Introduces SearchAlgorithm concept, separate from schedulers in Tune. Moves HyperOpt under this concept.
2018-08-04 21:27:39 -07:00
f7ec292360
[rllib] Support agent.get_action in multiagent ( #2543 )
...
* support get action on policy id
* comment
* grammar fixes
* Update rllib-algorithms.rst
2018-08-02 13:35:53 -07:00
9ea57c2a93
[rllib] Basic IMPALA implementation (using deepmind's reference vtrace.py) ( #2504 )
...
Rename AsyncSamplesOptimizer -> AsyncReplayOptimizer
Add AsyncSamplesOptimizer that implements the IMPALA architecture
integrate V-trace with a3c policy graph
audit V-trace integration
benchmark compare vs A3C and with V-trace on/off
PongNoFrameskip-v4 on IMPALA scaling from 16 to 128 workers, solving Pong in <10 min. For reference, solving this env takes ~40 minutes for Ape-X and several hours for A3C.
2018-08-01 20:53:53 -07:00
a630e332f3
[rllib] Don't use get_gpu_ids() in ppo
...
This lets the num_gpus config work properly even when not using tune, since the gpu ids won't be set by ray in that case.
2018-08-01 16:25:11 -07:00
d9a36c4e39
[rllib] Document auto-concat in a3c ( #2533 )
...
* docs
* update hyperparm docs
2018-08-01 15:11:30 -07:00
38d00986a5
[rllib] Cleanups: deep merge configs properly; enforce min iter time on APEX ( #2500 )
...
The dict merge prevents crashes when tune is trying to get resource requests for agents and you override a config subkey. The min iter time prevents iterations from getting too small, incurring high overhead. This is easy to run into on Ape-X since throughput can get very high.
2018-07-30 13:25:35 -07:00
24649726dc
[rllib] Use batch.count in async samples optimizer ( #2488 )
...
Using the actual batch size reduces the risk of mis-accounting. Here, we under-counted samples since in truncate_episodes mode we were doubling the batch size by accident in policy_evaluator.
2018-07-27 16:44:21 -07:00
05490b8cb9
[rllib] dqn/ddpg policy customization ( #2445 )
...
* dqn policy update - more customization
* docs for custom DQN graph
* Update rllib-training.rst
* Update rllib-models.rst
* Update rllib.rst
* Update rllib-training.rst
* Update rllib-concepts.rst
* yapf codestyle
2018-07-22 14:47:14 -07:00
68660453e4
[rllib] Better support and add two-trainer example for multiagent ( #2443 )
...
This adds a simple DQN+PPO example for multi-agent. We don't do anything fancy here, just syncing weights between two separate trainers. This potentially is wasting some compute, but is very simple to set up.
It might be nice to share experience collection between the top-level trainers in the future.
2018-07-22 05:09:25 -07:00
8e75d150f7
[rllib] Apex crash when compress_observations: False ( #2426 )
...
We shouldn't try to decompress uncompressed data.
Also, fix resource requests for ddpg + GPU.
2018-07-19 15:58:09 -07:00
d01dc9e22d
[rllib] format with yapf ( #2427 )
...
* initial yapf
* manual fix yapf bugs
2018-07-19 15:30:36 -07:00
0cecf6b79c
[rllib] Cleanup RNN support and make it work with multi-GPU optimizer ( #2394 )
...
Cleanup: TFPolicyGraph now automatically adds loss input entries for state_in_*, so that graph sub-classes don't need to worry about it.
Multi-GPU support:
Allow setting up model tower replicas with existing state input tensors
Truncate the per-device minibatch slices so that they are always a multiple of max_seq_len.
2018-07-17 06:55:46 +02:00
62f84d2f07
[rllib] Restore TF soft placement config to fix multi-GPU optimizer ( #2395 )
2018-07-13 10:34:37 +02:00
b316afeb43
[rllib] Add debug info back to PPO and fix optimizer compatibility ( #2366 )
2018-07-12 19:22:46 +02:00
0048e77093
[rllib] RLlib CLI ( #2375 )
2018-07-12 19:12:04 +02:00
4d7da9f668
[rllib] Remove "Common", cleanup some code ( #2348 )
2018-07-08 13:03:53 -07:00
d24f19fd1e
[rllib] Fix stats collection and some docs bugs since the refactoring ( #2361 )
...
* fix
* fix pbt example
* fix
* fix
* single thread by default
* vec
* fix
* fix
2018-07-07 13:29:20 -07:00
e32aed8717
[rllib] more user-friendly Optimizer signature + compute_apply ( #2335 )
...
* Move signature of optimizers
* fix
* expose compute_apply for policy_graphs
* dictionaries and such
* test for multiagent
2018-07-07 12:08:49 -07:00
8aa56c12e6
[rllib] Document "v2" APIs ( #2316 )
...
* re
* wip
* wip
* a3c working
* torch support
* pg works
* lint
* rm v2
* consumer id
* clean up pg
* clean up more
* fix python 2.7
* tf session management
* docs
* dqn wip
* fix compile
* dqn
* apex runs
* up
* impotrs
* ddpg
* quotes
* fix tests
* fix last r
* fix tests
* lint
* pass checkpoint restore
* kwar
* nits
* policy graph
* fix yapf
* com
* class
* pyt
* vectorization
* update
* test cpe
* unit test
* fix ddpg2
* changes
* wip
* args
* faster test
* common
* fix
* add alg option
* batch mode and policy serving
* multi serving test
* todo
* wip
* serving test
* doc async env
* num envs
* comments
* thread
* remove init hook
* update
* fix ppo
* comments1
* fix
* updates
* add jenkins tests
* fix
* fix pytorch
* fix
* fixes
* fix a3c policy
* fix squeeze
* fix trunc on apex
* fix squeezing for real
* update
* remove horizon test for now
* multiagent wip
* update
* fix race condition
* fix ma
* t
* doc
* st
* wip
* example
* wip
* working
* cartpole
* wip
* batch wip
* fix bug
* make other_batches None default
* working
* debug
* nit
* warn
* comments
* fix ppo
* fix obs filter
* update
* wip
* tf
* update
* fix
* cleanup
* cleanup
* spacing
* model
* fix
* dqn
* fix ddpg
* doc
* keep names
* update
* fix
* com
* docs
* clarify model outputs
* Update torch_policy_graph.py
* fix obs filter
* pass thru worker index
* fix
* rename
* vlad torch comments
* fix log action
* debug name
* fix lstm
* remove unused ddpg net
* remove conv net
* revert lstm
* wip
* wip
* cast
* wip
* works
* fix a3c
* works
* lstm util test
* doc
* clean up
* update
* fix lstm check
* move to end
* fix sphinx
* fix cmd
* remove bad doc
* envs
* vec
* doc prep
* models
* rl
* alg
* up
* clarify
* copy
* async sa
* fix
* comments
* fix a3c conf
* tune lstm
* fix reshape
* fix
* back to 16
* tuned a3c update
* update
* tuned
* optional
* merge
* wip
* fix up
* move pg class
* rename env
* wip
* update
* tip
* alg
* readme
* fix catalog
* readme
* doc
* context
* remove prep
* comma
* add env
* link to paper
* paper
* update
* rnn
* update
* wip
* clean up ev creation
* fix
* fix
* fix
* fix lint
* up
* no comma
* ma
* Update run_multi_node_tests.sh
* fix
* sphinx is stupid
* sphinx is stupid
* clarify torch graph
* no horizon
* fix config
* sb
* Update test_optimizers.py
2018-07-01 00:05:08 -07:00
Richard Liaw