[rllib] General RNN support (#2299)

* wip

* cls

* re

* wip

* wip

* a3c working

* torch support

* pg works

* lint

* rm v2

* consumer id

* clean up pg

* clean up more

* fix python 2.7

* tf session management

* docs

* dqn wip

* fix compile

* dqn

* apex runs

* up

* impotrs

* ddpg

* quotes

* fix tests

* fix last r

* fix tests

* lint

* pass checkpoint restore

* kwar

* nits

* policy graph

* fix yapf

* com

* class

* pyt

* vectorization

* update

* test cpe

* unit test

* fix ddpg2

* changes

* wip

* args

* faster test

* common

* fix

* add alg option

* batch mode and policy serving

* multi serving test

* todo

* wip

* serving test

* doc async env

* num envs

* comments

* thread

* remove init hook

* update

* fix ppo

* comments1

* fix

* updates

* add jenkins tests

* fix

* fix pytorch

* fix

* fixes

* fix a3c policy

* fix squeeze

* fix trunc on apex

* fix squeezing for real

* update

* remove horizon test for now

* multiagent wip

* update

* fix race condition

* fix ma

* t

* doc

* st

* wip

* example

* wip

* working

* cartpole

* wip

* batch wip

* fix bug

* make other_batches None default

* working

* debug

* nit

* warn

* comments

* fix ppo

* fix obs filter

* update

* wip

* tf

* update

* fix

* cleanup

* cleanup

* spacing

* model

* fix

* dqn

* fix ddpg

* doc

* keep names

* update

* fix

* com

* docs

* clarify model outputs

* Update torch_policy_graph.py

* fix obs filter

* pass thru worker index

* fix

* rename

* vlad torch comments

* fix log action

* debug name

* fix lstm

* remove unused ddpg net

* remove conv net

* revert lstm

* wip

* wip

* cast

* wip

* works

* fix a3c

* works

* lstm util test

* doc

* clean up

* update

* fix lstm check

* move to end

* fix sphinx

* fix cmd

* remove bad doc

* clarify

* copy

* async sa

* fix

* comments

* fix a3c conf

* tune lstm

* fix reshape

* fix

* back to 16

* tuned a3c update

* update

* tuned

* optional

* fix catalog

* remove prep

python/ray/rllib/models/model.py

@@ -15,6 +15,19 @@ class Model(object):
     The last layer of the network can also be retrieved if the algorithm
     needs to further post-processing (e.g. Actor and Critic networks in A3C).
 
+    Attributes:
+        inputs (Tensor): The input placeholder for this model, of shape
+            [BATCH_SIZE, ...].
+        outputs (Tensor): The output vector of this model, of shape
+            [BATCH_SIZE, num_outputs].
+        last_layer (Tensor): The network layer right before the model output,
+            of shape [BATCH_SIZE, N].
+        state_init (list): List of initial recurrent state tensors (if any).
+        state_in (list): List of input recurrent state tensors (if any).
+        state_out (list): List of output recurrent state tensors (if any).
+        seq_lens (Tensor): The tensor input for RNN sequence lengths. This
+            defaults to a Tensor of [1] * len(batch) in the non-RNN case.
+
     If `options["free_log_std"]` is True, the last half of the
     output layer will be free variables that are not dependent on
     inputs. This is often used if the output of the network is used
@@ -22,25 +35,24 @@ class Model(object):
     first half of the parameters can be interpreted as a location
     parameter (like a mean) and the second half can be interpreted as
     a scale parameter (like a standard deviation).
-
-    Attributes:
-        inputs (Tensor): The input placeholder for this model.
-        outputs (Tensor): The output vector of this model.
-        last_layer (Tensor): The network layer right before the model output.
-        state_init (list): List of initial recurrent state tensors (if any).
-        state_in (list): List of input recurrent state tensors (if any).
-        state_out (list): List of output recurrent state tensors (if any).
     """
 
     def __init__(self, inputs, num_outputs, options):
+        self.inputs = inputs
+
+        # Default attribute values for the non-RNN case
         self.state_init = []
         self.state_in = []
         self.state_out = []
-        self.inputs = inputs
+        self.seq_lens = tf.placeholder_with_default(
+            tf.ones(  # reshape needed for older tf versions
+                tf.reshape(tf.shape(inputs)[0], [1]), dtype=tf.int32),
+            [None], name="seq_lens")
+
         if options.get("free_log_std", False):
             assert num_outputs % 2 == 0
             num_outputs = num_outputs // 2
-        self.outputs, self.last_layer = self._init(
+        self.outputs, self.last_layer = self._build_layers(
             inputs, num_outputs, options)
         if options.get("free_log_std", False):
             log_std = tf.get_variable(name="log_std", shape=[num_outputs],
@@ -48,6 +60,6 @@ class Model(object):
             self.outputs = tf.concat(
                 [self.outputs, 0.0 * self.outputs + log_std], 1)
 
-    def _init(self):
+    def _build_layers(self):
         """Builds and returns the output and last layer of the network."""
         raise NotImplementedError