[rllib] Also refactor DQN to use shared RLlib models (#730)

* wip

* works with cartpole

* lint

* fix pg

* comment

* action dist rename

* preprocessor

* fix test

* typo

* fix the action[0] nonsense

* revert

* satisfy the lint

* wip

* works with cartpole

* lint

* fix pg

* comment

* action dist rename

* preprocessor

* fix test

* typo

* fix the action[0] nonsense

* revert

* satisfy the lint

* Minor indentation changes.

* fix merge

* add humanoid

* initial dqn refactor

* remove tfutil

* fix calls

* fix tf errors 1

* closer

* runs now

* lint

* tensorboard graph

* fix linting

* more 4 space

* fix

* fix linT

* more lint

* oops

* es parity

* remove example.py

* fix training bug

* add cartpole demo

* try fixing cartpole

* allow model options, configure cartpole

* debug

* simplify

* no dueling

* avoid out of file handles

* Test dqn in jenkins.

* Minor formatting.

* fix issue

* fix another

* Fix problem in which we log to a directory that hasn't been created.

python/ray/rllib/models/visionnet.py

@@ -11,7 +11,7 @@ from ray.rllib.models.model import Model
 class VisionNetwork(Model):
     """Generic vision network."""
 
-    def _init(self, inputs, num_outputs):
+    def _init(self, inputs, num_outputs, options):
         with tf.name_scope("vision_net"):
             conv1 = slim.conv2d(inputs, 16, [8, 8], 4, scope="conv1")
             conv2 = slim.conv2d(conv1, 32, [4, 4], 2, scope="conv2")
@@ -19,4 +19,4 @@ class VisionNetwork(Model):
                 conv2, 512, [10, 10], padding="VALID", scope="fc1")
             fc2 = slim.conv2d(fc1, num_outputs, [1, 1], activation_fn=None,
                               normalizer_fn=None, scope="fc2")
-            return tf.squeeze(fc2, [1, 2])
+            return tf.squeeze(fc2, [1, 2]), tf.squeeze(fc1, [1, 2])