[rllib] Parallelize sample collection and gradient computation in DQN (#746)

* wip

* works with cartpole

* lint

* fix pg

* comment

* action dist rename

* preprocessor

* fix test

* typo

* fix the action[0] nonsense

* revert

* satisfy the lint

* wip

* wip

* works with cartpole

* lint

* fix pg

* comment

* action dist rename

* preprocessor

* fix test

* typo

* fix the action[0] nonsense

* revert

* satisfy the lint

* Minor indentation changes.

* fix merge

* add humanoid

* initial dqn refactor

* remove tfutil

* fix calls

* fix tf errors 1

* closer

* runs now

* lint

* tensorboard graph

* fix linting

* more 4 space

* fix

* fix linT

* more lint

* oops

* es parity

* remove example.py

* fix training bug

* add cartpole demo

* try fixing cartpole

* allow model options, configure cartpole

* debug

* simplify

* no dueling

* avoid out of file handles

* Test dqn in jenkins.

* Minor formatting.

* lint

* fix py3

* fix issue

* remove chekcpoint

* revert

* Fixit

* sanity check configs

* update cuda

* fix

* parallel gradient computation

* update

* upd

* bug

* upd

* always record training stats

* fix

* comments

* revert assert

* add gpu mask

* fofset

* a tie

* Merge

* fix

* fix

* fix examples

* A3C -> DQN

* fix dqn test

* remove submodule

* fix linting

test/jenkins_tests/run_multi_node_tests.sh

@@ -96,7 +96,7 @@ docker run --shm-size=10G --memory=10G $DOCKER_SHA \
     --env PongNoFrameskip-v4 \
     --alg DQN \
     --num-iterations 2 \
-    --config '{"lr": 1e-4, "schedule_max_timesteps": 2000000, "buffer_size": 10000, "exploration_fraction": 0.1, "exploration_final_eps": 0.01, "train_freq": 4, "learning_starts": 10000, "target_network_update_freq": 1000, "gamma": 0.99, "prioritized_replay": true}'
+    --config '{"lr": 1e-4, "schedule_max_timesteps": 2000000, "buffer_size": 10000, "exploration_fraction": 0.1, "exploration_final_eps": 0.01, "sample_batch_size": 4, "learning_starts": 10000, "target_network_update_freq": 1000, "gamma": 0.99, "prioritized_replay": true}'
 
 docker run --shm-size=10G --memory=10G $DOCKER_SHA \
     python /ray/python/ray/rllib/train.py \