add papers (ongoing)

2026-09-11 12:31:07 +08:00 · 2019-06-13 22:35:20 +02:00
parent 80c372da11
commit 44058291a9
9 changed files with 326 additions and 0 deletions
@@ -0,0 +1,45 @@
+## Implemented papers
+
+This folder contains subfolders where in each one we try to reproduce the experiments and results obtained in the 
+literature using the *PyRoboLearn* framework.
+
+Each subfolder contains 
+- a `README` file that:
+    - summarizes the paper and provides the BibTex reference with a link to the original paper
+    - states the possible differences between their experiment setup and ours and the obtained results
+    - provides the current status (if it has been implemented yet, if the reproduction was successful or not), and 
+    describes what needs to be improved.
+    - describes which variables the user can play with
+- an environment (similar to gym environments) in the `env.py` file (that initializes the world, robots, rewards, 
+states, actions, and so on) which allows to be used in other codes.
+- a main file (`main.py`) that initializes the learning task (environment, policy, and so on), and provides an 
+example on how to use the environment defined in `env.py`.
+- a possible `figures` directory containing figures; this can include a picture of the environment, training plots, 
+etc.
+- a possible `meshes` directory containing meshes for the different objects that are present in the environment.
+
+
+### Structure
+
+The name of each folder is `<author_last_name><year><first_non_stop_word_in_title>`, this is the same as returned by 
+[Google Scholar](https://scholar.google.com/), when clicking on the `BibTex` button:
+
+```
+@article{<1st author's last name><year><1st non stop word in title>,
+  title={<title>},
+  author={<1st author's last name>, 1st author's first name and ...},
+  journal={...},
+  year={<year>}
+}
+```
+
+*Note*: Stop words are commonly used words such as 'the', 'an', 'a', 'in', and others that search engines usually 
+ignore.
+
+
+### Further notes
+
+- It might be that in the future, this folder will be moved to its own repository.
+- I will often focus more on implementing the environments than training the various policies using the different 
+algorithms
+- Any comments, help, pull requests are appreciated. 
@@ -0,0 +1,13 @@
+# 
+# Wavefront material file
+# Created with Kaydara FBX
+# 
+
+newmtl Mat_3
+map_Kd cube.png
+Ka 0.000000 0.000000 0.000000
+Kd 0.800000 0.800000 0.800000
+Ks 0.200000 0.200000 0.200000
+illum 2
+Ns 50.000000
+Tr 1.000000
@@ -0,0 +1,76 @@
+# 
+# Wavefront OBJ file
+# Created with Kaydara FBX
+# Created by Melissa Britelle, 2019
+#
+mtllib cube.mtl
+
+g Cube_1
+v -0.5000000 -0.5000000 0.5000000
+v -0.5000000 0.5000000 0.5000000
+v 0.5000000 -0.5000000 0.5000000
+v 0.5000000 0.5000000 0.5000000
+v 0.5000000 -0.5000000 -0.5000000
+v 0.5000000 0.5000000 -0.5000000
+v -0.5000000 -0.5000000 -0.5000000
+v -0.5000000 0.5000000 -0.5000000
+
+vt 0.250000 0.625000
+vt 0.500000 0.375000
+vt 0.500000 0.625000
+vt 0.500000 0.375000
+vt 0.250000 0.625000
+vt 0.250000 0.375000
+vt 0.500000 0.625000
+vt 0.500000 0.375000
+vt 0.750000 0.375000
+vt 0.500000 0.625000
+vt 0.750000 0.375000
+vt 0.750000 0.625000
+vt 0.750000 0.375000
+vt 1.000000 0.375000
+vt 0.750000 0.625000
+vt 1.000000 0.375000
+vt 1.000000 0.625000
+vt 0.750000 0.625000
+vt 0.250000 0.625000
+vt 0.000000 0.625000
+vt 0.250000 0.375000
+vt 0.250000 0.375000
+vt 0.000000 0.625000
+vt 0.000000 0.375000
+vt 0.250000 0.625000
+vt 0.500000 0.625000
+vt 0.250000 0.875000
+vt 0.500000 0.625000
+vt 0.500000 0.875000
+vt 0.250000 0.875000
+vt 0.250000 0.375000
+vt 0.500000 0.125000
+vt 0.500000 0.375000
+vt 0.500000 0.125000
+vt 0.250000 0.375000
+vt 0.250000 0.125000
+
+vn 0.000000 0.000000 1.000000
+vn 0.000000 0.000000 1.000000
+vn 0.000000 0.000000 1.000000
+vn 0.000000 0.000000 1.000000
+vn 0.000000 0.000000 1.000000
+vn 0.000000 0.000000 1.000000
+vn 1.000000 0.000000 0.000000
+vn 1.000000 0.000000 0.000000
+
+usemtl Mat_3
+f 2/1/2 3/2/3 4/3/4
+f 3/4/3 2/5/2 1/6/1
+f 4/7/4 3/8/3 5/9/5
+f 4/10/4 5/11/5 6/12/6
+f 5/13/5 7/14/7 6/15/6
+f 7/16/7 8/17/8 6/18/6
+f 2/19/2 8/20/8 1/21/1
+f 1/22/1 8/23/8 7/24/7
+f 2/25/2 4/26/4 8/27/8
+f 4/28/4 6/29/6 8/30/8
+f 1/31/1 5/32/5 3/33/3
+f 5/34/5 1/35/1 7/36/7
@@ -0,0 +1,80 @@
+### Learning Dexterous In-Hand Manipulation
+
+Status: ongoing
+
+Abstract (from [1]): "We use reinforcement learning (RL) to learn dexterous in-hand manipulation policies which can 
+perform vision-based object reorientation on a physical Shadow Dexterous Hand. The training is performed in a simulated 
+environment in which we randomize many of the physical properties of the system like friction coefficients and an 
+object's appearance. Our policies transfer to the physical robot despite being trained entirely in simulation. Our 
+method does not rely on any human demonstrations, but many behaviors found in human manipulation emerge naturally, 
+including finger gaiting, multi-finger coordination, and the controlled use of gravity. Our results were obtained 
+using the same distributed RL system that was used to train OpenAI Five. We also include a video of our results: 
+[this url](https://www.youtube.com/watch?v=jwSbzNHGflM)."
+
+
+#### Implementation details
+
+From the paper [1]:
+
+* simulator: Mujoco [2]
+* robot: ShadowHand [3]
+* states:
+    * for value network: fingertip positions (5*3D), object position (3D), object orientation (4D=quaternion), 
+        target orientation (4D=quaternion), relative target orientation (4D=quaternion), hand joint angles (24D), 
+        hand joint velocities (24D), object velocity (3D), object angular velocity (3D)
+    * for policy: fingertip positions (5*3D), object position (3D), relative target orientation (4D=quaternion)
+* actions: desired joint angles of the hand relative to the current ones. The actions are discretized into 11 bins. 
+* policy: fully-connected neural network composed of a normalization layer, dense ReLU (1024), LSTM (512)
+* value: fully-connected neural network composed of a normalization layer, dense ReLU (1024), LSTM (512)
+* reward function:
+    * `r_t = d_t - d_{t+1}`, where `d_t` and `d_{t+1}` are the rotation angles between the desired and current object 
+    orientations before and after the transition, respectively.
+    * 5 if the goal is achieved
+    * -20 if the object drop
+* termination condition (goal): 
+* exploration: in the action space using a categorical distribution with 11 bins for each action coordinate
+* RL algorithm: PPO
+    * clip parameter=0.2, entropy regularization coefficient=0.01, GAE(discount factor=0.998, lambda=0.95)), Adam 
+    optimizer with learning rate=3e-4.
+
+Compared to [1], we use the PyBullet simulator [4] and not Mujoco (because of its license), and we do not use a pose 
+estimation network (i.e. the convolutional neural network).
+
+
+#### Results
+
+<p align="center">
+    <img src="./figures/hand_cube.png" alt="a ShadowHand holding a cube" width="400" height="300"/>
+</p>
+
+TODO: figures + plots + link to video
+
+
+#### References
+
+- [1] ["Learning Dexterous In-Hand Manipulation"](https://arxiv.org/abs/1808.00177), OpenAI et al., 2018
+```
+@article{openai2018learning,
+  title={Learning dexterous in-hand manipulation},
+  author={OpenAI and Andrychowicz, Marcin and Baker, Bowen and Chociej, Maciek and Jozefowicz, Rafal and McGrew, Bob and Pachocki, Jakub and Petron, Arthur and Plappert, Matthias and Powell, Glenn and Ray, Alex and others},
+  journal={arXiv preprint arXiv:1808.00177},
+  year={2018}
+}
+```
+- [2] ["Mujoco: A physics engine for model-based control"](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.296.6848&rep=rep1&type=pdf), 
+    Todorov et al., 2012
+- [3] [ShadowHand](https://www.shadowrobot.com/products/dexterous-hand/) 
+    ([URDF](https://github.com/shadow-robot/sr_common))
+- [4] ["PyBullet, a Python module for physics simulation for games, robotics and machine learning"](https://github.com/bulletphysics/bullet3), 
+    Coumans et al., 2017
+
+
+The 3D cube (the OBJ file as well as the texture) used for this experiment was modeled by Melissa Britelle based on 
+the cube showed in the paper [1].
+
+
+#### What's next?
+
+Next possible interesting tasks:
+- Use various tools (see )
+- juggle different objects (balls, clubs, different objects)
@@ -0,0 +1,84 @@
+#!/usr/bin/env python
+"""
+Provide the environment (world, states, actions, rewards, terminal conditions, etc) defined in the paper [1] using the
+PyRoboLearn framework.
+
+Reference:
+    [1] "Learning Dexterous In-Hand Manipulation", OpenAI et al., 2018 (https://arxiv.org/abs/1808.00177)
+"""
+
+# TODO: end the implementation
+
+# import PyRoboLearn
+import pyrobolearn as prl
+
+__author__ = "Brian Delhaisse"
+__copyright__ = "Copyright 2018, PyRoboLearn"
+__credits__ = ["OpenAI et al."]
+__license__ = "MIT"
+__version__ = "1.0.0"
+__maintainer__ = "Brian Delhaisse"
+__email__ = "briandelhaisse@gmail.com"
+__status__ = "Development"
+
+
+# define environment
+class Openai2018LearningEnv(prl.envs.Env):
+    r"""Openai2018Learning Environment
+
+    Create the environment (world, states, actions, rewards, terminal conditions, etc) defined in the paper [1].
+
+    Reference:
+    [1] "Learning Dexterous In-Hand Manipulation", OpenAI et al., 2018 (https://arxiv.org/abs/1808.00177)
+    """
+
+    def __init__(self, simulator=None):
+        r"""
+        Initialize the environment.
+
+        Args:
+            simulator (prl.simulators.Simulator): simulator instance or None. If None, it will use PyBullet by default.
+        """
+        # create simulator if None
+        if simulator is None:
+            simulator = prl.simulators.Bullet()
+        if not isinstance(simulator, prl.simulators.Simulator):
+            raise TypeError("Expecting the given simulator to be None or an instance of `prl.simulators.Simulator`, "
+                            "instead got: {}".format(type(simulator)))
+
+        # create world
+        world = prl.worlds.BasicWorld(simulator)
+
+        # load hand
+        robot = world.load_robot('shadowhand', position=(-0.2, 0, 0.5), orientation=(-0.5, 0.5, -0.5, 0.5), left=False)
+        robot.print_info()
+
+        # load cube in hand
+        cube = world.load_mesh('../meshes/cube.obj', position=[0.1, 0, 0.57], scale=(.05, .05, .05), flags=0)
+        cube = prl.robots.Body(simulator, body_id=cube)
+
+        # create state: fingertip positions (5*3D), object position (3D), object orientation (4D=quaternion),
+        # target orientation (4D=quaternion), relative target orientation (4D=quaternion), hand joint angles (24D),
+        # hand joint velocities (24D), object velocity (3D), object angular velocity (3D)
+        state = prl.states.PositionState(cube) + prl.states.OrientationState(cube) + \
+                prl.states.JointPositionState(robot)
+
+        # create action
+
+        # create reward
+
+        super(Openai2018LearningEnv, self).__init__(world, state)
+
+
+# Test environment
+if __name__ == '__main__':
+    import time
+    from itertools import count
+
+    # create environment
+    env = Openai2018LearningEnv()
+
+    # run environment
+    for t in count():
+        env.step()
+        time.sleep(1./240)
@@ -0,0 +1,28 @@
+#!/usr/bin/env python
+"""
+This sample of code implements the "Learning Dexterous In-Hand Manipulation" paper [1] using the PyRoboLearn framework.
+
+We use the same factors as described in the paper; i.e. the same robotic platform, same states and actions,
+same policies, same rewards (and coefficients), same learning algorithm with same hyperparameters, and so on.
+
+Reference:
+    [1] "Learning Dexterous In-Hand Manipulation", OpenAI et al., 2019 (https://arxiv.org/abs/1808.00177)
+"""
+
+import time
+from itertools import count
+import pyrobolearn as prl
+from env import Openai2018LearningEnv
+
+
+# create environment (create world, states, rewards, actions)
+environment = Openai2018LearningEnv()
+
+# run environment
+for t in count():
+    environment.step()
+    time.sleep(1./240)
+
+# create policy
+
+# create algo