[RLlib] DQN torch version. (#7597)

* Fix. * Rollback. * WIP. * WIP. * WIP. * WIP. * WIP. * WIP. * WIP. * WIP. * Fix. * Fix. * Fix. * Fix. * Fix. * WIP. * WIP. * Fix. * Test case fixes. * Test case fixes and LINT. * Test case fixes and LINT. * Rollback. * WIP. * WIP. * Test case fixes. * Fix. * Fix. * Fix. * Add regression test for DQN w/ param noise. * Fixes and LINT. * Fixes and LINT. * Fixes and LINT. * Fixes and LINT. * Fixes and LINT. * Comment * Regression test case. * WIP. * WIP. * LINT. * LINT. * WIP. * Fix. * Fix. * Fix. * LINT. * Fix (SAC does currently not support eager). * Fix. * WIP. * LINT. * Update rllib/evaluation/sampler.py Co-Authored-By: Eric Liang <ekhliang@gmail.com> * Update rllib/evaluation/sampler.py Co-Authored-By: Eric Liang <ekhliang@gmail.com> * Update rllib/utils/exploration/exploration.py Co-Authored-By: Eric Liang <ekhliang@gmail.com> * Update rllib/utils/exploration/exploration.py Co-Authored-By: Eric Liang <ekhliang@gmail.com> * WIP. * WIP. * Fix. * LINT. * LINT. * Fix and LINT. * WIP. * WIP. * WIP. * WIP. * Fix. * LINT. * Fix. * Fix and LINT. * Update rllib/utils/exploration/exploration.py * Update rllib/policy/dynamic_tf_policy.py Co-Authored-By: Eric Liang <ekhliang@gmail.com> * Update rllib/policy/dynamic_tf_policy.py Co-Authored-By: Eric Liang <ekhliang@gmail.com> * Update rllib/policy/dynamic_tf_policy.py Co-Authored-By: Eric Liang <ekhliang@gmail.com> * Fixes. * WIP. * LINT. * Fixes and LINT. * LINT and fixes. * LINT. * Move action_dist back into torch extra_action_out_fn and LINT. * Working SimpleQ learning cartpole on both torch AND tf. * Working Rainbow learning cartpole on tf. * Working Rainbow learning cartpole on tf. * WIP. * LINT. * LINT. * Update docs and add torch to APEX test. * LINT. * Fix. * LINT. * Fix. * Fix. * Fix and docstrings. * Fix broken RLlib tests in master. * Split BAZEL learning tests into cartpole and pendulum (reached the 60min barrier). * Fix error_outputs option in BAZEL for RLlib regression tests. * Fix. * Tune param-noise tests. * LINT. * Fix. * Fix. * test * test * test * Fix. * Fix. * WIP. * WIP. * WIP. * WIP. * LINT. * WIP. Co-authored-by: Eric Liang <ekhliang@gmail.com>
2026-09-12 12:51:15 +08:00 · 2020-04-06 11:56:16 -07:00
parent f63b4c1110
commit 22ccc43670
56 changed files with 1292 additions and 277 deletions
@@ -83,6 +83,10 @@ def one_hot(x, depth=0, on_value=1, off_value=0):
    Returns:
        np.ndarray: The one-hot encoded equivalent of the input array.
    """
+    # Handle torch arrays properly.
+    if torch and isinstance(x, torch.Tensor):
+        x = x.numpy()
+
    # Handle bool arrays correctly.
    if x.dtype == np.bool_:
        x = x.astype(np.int)
@@ -114,7 +118,7 @@ def one_hot(x, depth=0, on_value=1, off_value=0):
    return out


-def fc(x, weights, biases=None):
+def fc(x, weights, biases=None, framework=None):
    """
    Calculates the outputs of a fully-connected (dense) layer given
    weights/biases and an input.
@@ -123,23 +127,29 @@ def fc(x, weights, biases=None):
        x (np.ndarray): The input to the dense layer.
        weights (np.ndarray): The weights matrix.
        biases (Optional[np.ndarray]): The biases vector. All 0s if None.
+        framework (Optional[str]): An optional framework hint (to figure out,
+            e.g. whether to transpose torch weight matrices).

    Returns:
        The dense layer's output.
    """

+    def map_(data, transpose=False):
+        if torch:
+            if isinstance(data, torch.Tensor):
+                data = data.cpu().detach().numpy()
+        if tf and tf.executing_eagerly():
+            if isinstance(data, tf.Variable):
+                data = data.numpy()
+        if transpose:
+            data = np.transpose(data)
+        return data
+
+    x = map_(x)
    # Torch stores matrices in transpose (faster for backprop).
-    if torch:  # and isinstance(weights, torch.Tensor):
-        x = x.detach().numpy() if isinstance(x, torch.Tensor) else x
-        weights = np.transpose(weights.detach().numpy()) if \
-            isinstance(weights, torch.Tensor) else weights
-        biases = biases.detach().numpy() if \
-            isinstance(biases, torch.Tensor) else biases
-    if tf and tf.executing_eagerly():
-        x = x.numpy() if isinstance(x, tf.Variable) else x
-        weights = weights.numpy() if isinstance(weights, tf.Variable) else \
-            weights
-        biases = biases.numpy() if isinstance(biases, tf.Variable) else biases
+    weights = map_(weights, transpose=framework == "torch")
+    biases = map_(biases)
+
    return np.matmul(x, weights) + (0.0 if biases is None else biases)


@@ -216,3 +226,10 @@ def lstm(x,
            unrolled_outputs[:, t, :] = h_states

    return unrolled_outputs, (c_states, h_states)
+
+
+def huber_loss(x, delta=1.0):
+    """Reference: https://en.wikipedia.org/wiki/Huber_loss"""
+    return np.where(
+        np.abs(x) < delta,
+        np.power(x, 2.0) * 0.5, delta * (np.abs(x) - 0.5 * delta))