From 81a9d861b67fe56ce1770e15331e001a1646ed5d Mon Sep 17 00:00:00 2001 From: Seungjae Ryan Lee Date: Mon, 25 Mar 2019 18:53:36 +0900 Subject: [PATCH] Improve README design and content (#30) * Add logo to README * Add status shields to README * Align status shields to center * Add elevator pitch line to README * Add Algorithms section to README * Add How to Cite section * Remove Repository Structure section * Add basic Installation section --- README.md | 71 +++++++++++++++++++++++++++++++++++-------------------- 1 file changed, 45 insertions(+), 26 deletions(-) diff --git a/README.md b/README.md index a8e3e3d..35e1a52 100644 --- a/README.md +++ b/README.md @@ -1,34 +1,53 @@ -# Research Repository Structure +

+ KAIR +

-We follow the `.git` strategy specified [here](https://answers.ros.org/question/257855/git-strategy-for-catkin-and-package-folders/), creating packages into separate repositories. -These are repositories checked to set this structure: -- [cadrl_ros](https://github.com/mfe7/cadrl_ros) -- [ros_best_practices](https://github.com/leggedrobotics/ros_best_practices) +

+ + Build Status + + + Google Docstring style + + + Pre-commit enabled + +

-## TODO +KAIR algorithm is a research repository with state of the art reinforcement learning algorithms for robot control tasks. It allows the researchers to experiment with novel ideas with minimal code changes. -1. Add [medipixel/reinforcement_learning_examples](https://github.com/medipixel/reinforcement_learning_examples) to `/scripts/`. -2. `launch` and `urdf` contains code for Sawyer robot. We plan to use OpenManipulator. -3. `package.xml` should be updated appropriately. -4. Get repository verified by a ROS expert. +## Algorithms -## Repository Structure +The [scripts](/scripts) folder contains implementations of a curated list of RL algorithms verified in MuJoCo environment. -``` -+ launch/ - -+ msg/ - These are message descriptions for ROS. -+ scripts/ - This directory contains RL algorithms. -+ urdf/ - This package contains a C++ parser for the Unified Robot Description Format (URDF), -- .flake8 - This file specifies what rules should be enforced via flake8. -- .gitignore - This file specifies which folders and files to ignore in Git. -- CMakeLists.txt - This file specifies behavior of CMake. -- package.xml - This file specifies the ROS package. -- README.md - You are here +- Twin Delayed Deep Deterministic Policy Gradient (TD3) + - TD3 (Fujimoto et al., 2018) is an extension of DDPG (Lillicrap et al., 2015), a deterministic policy gradient algorithm that uses deep neural networks for function approximation. Inspired by Deep Q-Networks (Mnih et al., 2015), DDPG uses experience replay and target network to improve stability. TD3 further improves DDPG by adding clipped double Q-learning (Van Hasselt, 2010) to mitigate overestimation bias (Thrun & Schwartz, 1993) and delaying policy updates to address variance. + - [Example Script on LunarLander](/scripts/examples/lunarlander_continuous_v2/td3.py) + - [ArXiv Preprint](https://arxiv.org/abs/1802.09477) + +- (Twin) Soft Actor Critic (SAC) + - SAC (Haarnoja et al., 2018a) incorporates maximum entropy reinforcment learning, where the agent's goal is to maximize expected reward and entropy concurrently. Combined with TD3, SAC achieves state of the art performance in various continuous control tasks. SAC has been extended to allow automatically tuning of the temperature parameter (Haarnoja et al., 2018b), which determines the importance of entropy against the expected reward. + - [Example Script on LunarLander](/scripts/examples/lunarlander_continuous_v2/sac.py) + - [ArXiv Preprint](https://arxiv.org/abs/1801.01290) (Original SAC) + - [ArXiv Preprint](https://arxiv.org/abs/1812.05905) (SAC with autotuned temperature) + + - TD3 from Demonstrations, SAC from Demonstrations (TD3fD, SACfD) + - DDPGfD (Vecerik et al., 2017) is an imitation learning algorithm that infuses demonstration data into experience replay. DDPGfD also improved DDPG by (1) using prioritized experience replay (Schaul et al., 2015), (2) adding n-step returns, (3) learning multiple times per environment step, and (4) adding L2 regularizers to actor and critic losses. We incorporated these improvements to TD3 and SAC and found that it dramatically improves their performance. + - [Example Script of TD3fD on LunarLander](/scripts/examples/lunarlander_continuous_v2/td3fd.py) + - [Example Script of SACfD on LunarLander](/scripts/examples/lunarlander_continuous_v2/sacfd.py) + - [ArXiv Preprint](https://arxiv.org/abs/1707.08817) + +## Installation + +To use the algorithms, first use the [requirements.txt](/scripts/requirements.txt) file to install appropriate Python packages from PyPI. + +```bash +cd scripts +pip install -r requirements.txt ``` -- **roslaunch/** [http://wiki.ros.org/roslaunch/XML](http://wiki.ros.org/roslaunch/XML). -- **msg/** [http://wiki.ros.org/msg](http://wiki.ros.org/msg) -- **urdf/** [http://wiki.ros.org/urdf](http://wiki.ros.org/urdf) -- **CMakeLists.txt** [http://wiki.ros.org/catkin/CMakeLists.txt](http://wiki.ros.org/catkin/CMakeLists.txt) -- **package.xml** [http://wiki.ros.org/catkin/package.xml](http://wiki.ros.org/catkin/package.xml) + +## How to Cite + +We are currently writing a white paper to summarize the results. We will add a BibTeX entry below once the paper is finalized.