This refactors the RLlib sampler to support multi-agent environments. The main changes were:
AsyncVectorEnv now produces dicts of env_id -> agent_id -> value rather than env_id -> value. This lets it model both vectorized and multi-agent envs (or both).
The sampler class operates over the above nested dict structure for all envs. Single agent envs just return a dict with one agent_id=single_agent.
When sample() is called on a policy evaluator, in the single agent case we return a SampleBatch, otherwise we return a MultiAgentBatch (which is a list of sample batches per policy).
Left for another PR:
Exposing multi-agent in the public interfaces.
Optimizations such as evaluating multiple policies in one TF run.
## What do these changes do?
**Vectorized envs**: Users can either implement `VectorEnv`, or alternatively set `num_envs=N` to auto-vectorize gym envs (this vectorizes just the action computation part).
```
# CartPole-v0 on single core with 64x64 MLP:
# vector_width=1:
Actions per second 2720.1284458322966
# vector_width=8:
Actions per second 13773.035334888269
# vector_width=64:
Actions per second 37903.20472563333
```
**Async envs**: The more general form of `VectorEnv` is `AsyncVectorEnv`, which allows agents to execute out of lockstep. We use this as an adapter to support `ServingEnv`. Since we can convert any other form of env to `AsyncVectorEnv`, utils.sampler has been rewritten to run against this interface.
**Policy serving**: This provides an env which is not stepped. Rather, the env executes in its own thread, querying the policy for actions via `self.get_action(obs)`, and reporting results via `self.log_returns(rewards)`. We also support logging of off-policy actions via `self.log_action(obs, action)`. This is a more convenient API for some use cases, and also provides parallelizable support for policy serving (for example, if you start a HTTP server in the env) and ingest of offline logs (if the env reads from serving logs).
Any of these types of envs can be passed to RLlib agents. RLlib handles conversions internally in CommonPolicyEvaluator, for example:
```
gym.Env => rllib.VectorEnv => rllib.AsyncVectorEnv
rllib.ServingEnv => rllib.AsyncVectorEnv
```
* working multi action distribution and multiagent model
* currently working but the splits arent done in the right place
* added shared models
* added categorical support and mountain car example
* now compatible with generalized advantage estimation
* working multiagent code with discrete and continuous example
* moved reshaper to utils
* code review changes made, ppo action placeholder moved to model catalog, all multiagent code moved out of fcnet
* added examples in
* added PEP8 compliance
* examples are mostly pep8 compliant
* removed all flake errors
* added examples to jenkins tests
* fixed custom options bug
* added lines to let docker file find multiagent tests
* shortened example run length
* corrected nits
* fixed flake errors
* wip
* Sat Dec 30 15:07:28 PST 2017
* log video
* video doesn't work well
* scenario integration
* Sat Dec 30 17:30:22 PST 2017
* Sat Dec 30 17:31:05 PST 2017
* Sat Dec 30 17:31:32 PST 2017
* Sat Dec 30 17:32:16 PST 2017
* Sat Dec 30 17:34:11 PST 2017
* Sat Dec 30 17:34:50 PST 2017
* Sat Dec 30 17:35:34 PST 2017
* Sat Dec 30 17:38:49 PST 2017
* Sat Dec 30 17:40:39 PST 2017
* Sat Dec 30 17:43:00 PST 2017
* Sat Dec 30 17:43:04 PST 2017
* Sat Dec 30 17:45:56 PST 2017
* Sat Dec 30 17:46:26 PST 2017
* Sat Dec 30 17:47:02 PST 2017
* Sat Dec 30 17:51:53 PST 2017
* Sat Dec 30 17:52:54 PST 2017
* Sat Dec 30 17:56:43 PST 2017
* Sat Dec 30 18:27:07 PST 2017
* Sat Dec 30 18:27:52 PST 2017
* fix train
* Sat Dec 30 18:41:51 PST 2017
* Sat Dec 30 18:54:11 PST 2017
* Sat Dec 30 18:56:22 PST 2017
* Sat Dec 30 19:05:04 PST 2017
* Sat Dec 30 19:05:23 PST 2017
* Sat Dec 30 19:11:53 PST 2017
* Sat Dec 30 19:14:31 PST 2017
* Sat Dec 30 19:16:20 PST 2017
* Sat Dec 30 19:18:05 PST 2017
* Sat Dec 30 19:18:45 PST 2017
* Sat Dec 30 19:22:44 PST 2017
* Sat Dec 30 19:24:41 PST 2017
* Sat Dec 30 19:26:57 PST 2017
* Sat Dec 30 19:40:37 PST 2017
* wip models
* reward bonus
* test prep
* Sun Dec 31 18:45:25 PST 2017
* Sun Dec 31 18:58:28 PST 2017
* Sun Dec 31 18:59:34 PST 2017
* Sun Dec 31 19:03:33 PST 2017
* Sun Dec 31 19:05:05 PST 2017
* Sun Dec 31 19:09:25 PST 2017
* fix train
* kill
* add tuple preprocessor
* Sun Dec 31 20:38:33 PST 2017
* Sun Dec 31 22:51:24 PST 2017
* Sun Dec 31 23:14:13 PST 2017
* Sun Dec 31 23:16:04 PST 2017
* Mon Jan 1 00:08:35 PST 2018
* Mon Jan 1 00:10:48 PST 2018
* Mon Jan 1 01:08:31 PST 2018
* Mon Jan 1 14:45:44 PST 2018
* Mon Jan 1 14:54:56 PST 2018
* Mon Jan 1 17:29:29 PST 2018
* switch to euclidean dists
* Mon Jan 1 17:39:27 PST 2018
* Mon Jan 1 17:41:47 PST 2018
* Mon Jan 1 17:44:18 PST 2018
* Mon Jan 1 17:47:09 PST 2018
* Mon Jan 1 20:31:02 PST 2018
* Mon Jan 1 20:39:33 PST 2018
* Mon Jan 1 20:40:55 PST 2018
* Mon Jan 1 20:55:06 PST 2018
* Mon Jan 1 21:05:52 PST 2018
* fix env path
* merge richards fix
* fix hash
* Mon Jan 1 22:04:00 PST 2018
* Mon Jan 1 22:25:29 PST 2018
* Mon Jan 1 22:30:42 PST 2018
* simplified reward function
* add framestack
* add env configs
* simplify speed reward
* Tue Jan 2 17:36:15 PST 2018
* Tue Jan 2 17:49:16 PST 2018
* Tue Jan 2 18:10:38 PST 2018
* add lane keeping simple mode
* Tue Jan 2 20:25:26 PST 2018
* Tue Jan 2 20:30:30 PST 2018
* Tue Jan 2 20:33:26 PST 2018
* Tue Jan 2 20:41:42 PST 2018
* ppo lane keep
* simplify discrete actions
* Tue Jan 2 21:41:05 PST 2018
* Tue Jan 2 21:49:03 PST 2018
* Tue Jan 2 22:12:23 PST 2018
* Tue Jan 2 22:14:42 PST 2018
* Tue Jan 2 22:20:59 PST 2018
* Tue Jan 2 22:23:43 PST 2018
* Tue Jan 2 22:26:27 PST 2018
* Tue Jan 2 22:27:20 PST 2018
* Tue Jan 2 22:44:00 PST 2018
* Tue Jan 2 22:57:58 PST 2018
* Tue Jan 2 23:08:51 PST 2018
* Tue Jan 2 23:11:32 PST 2018
* update dqn reward
* Thu Jan 4 12:29:40 PST 2018
* Thu Jan 4 12:30:26 PST 2018
* Update train_dqn.py
* fix
Eric Liang