diff --git a/README.md b/README.md index 4a445af..c42f03a 100644 --- a/README.md +++ b/README.md @@ -7,35 +7,34 @@ Implemented algorithms: * Deep Q-Learning (DQN) * Double DQN * Dueling DQN +* Async Advantage Actor Critic (A3C) * Async One-Step Q-Learning * Async One-Step Sarsa * Async N-Step Q-Learning -* Async Advantage Actor Critic (A3C) # Curves > Curves for CartPole is trivial so I didn't place it here. -## Deep Q-Learning (DQN) -![alt text](https://raw.githubusercontent.com/ShangtongZhang/DeepRL/master/images/DQN-BreakoutNoFrameskip-v3-Train.png) -![alt_text](https://raw.githubusercontent.com/ShangtongZhang/DeepRL/master/images/DQN-BreakoutNoFrameskip-v3-Test.png) +## DQN, Double DQN, Dueling DQN +![Loading...](https://raw.githubusercontent.com/ShangtongZhang/DeepRL/master/images/DQN-breakout.png) +![Loading...](https://raw.githubusercontent.com/ShangtongZhang/DeepRL/master/images/DQN-Pong.png) -The network and parameters here are exactly same as the DeepMind Nature paper. -Training curve is smoothed by window of size 100. Test is triggered every 1000 episodes. -In total it took about 16M frames. Training time is 4 days and 10 hours in a server with -Xeon E5-2620 v3 and Titan X. +The network and parameters here are exactly same as the [DeepMind Nature paper](https://www.nature.com/nature/journal/v518/n7540/full/nature14236.html). +Training curve is smoothed by a window of size 100. All the models are trained in a server with +Xeon E5-2620 v3 and Titan X. For Breakout, test is triggered every 1000 episodes with 50 repetitions. +In total, 16M frames cost about 4 days and 10 hours. For Pong, test is triggered +every 10 episodes with no repetition. In total, 4M frames cost about 12 hours. -## Asynchronous Advantage Actor Critic (A3C) +## A3C -![alt text](https://raw.githubusercontent.com/ShangtongZhang/DeepRL/master/images/A3C-PongNoFrameskip-v3.png) +![Loading...](https://raw.githubusercontent.com/ShangtongZhang/DeepRL/master/images/A3C-PongNoFrameskip-v3.png) The network I used here is same as the network in DQN except the activation function is **Elu** rather than Relu. The optimizer is **Adam** with non-shared parameters. To my best knowledge, this network architecture is not the most suitable for A3C. -If you use a 42 * 42 input, add a LSTM layer, you will get much much much better training speed +If you use a 42 * 42 input, add a LSTM layer at last, you will get **much much much** better training speed than this. [GAE](http://www.breloff.com/DeepRL-OnlineGAE/) can also improve performance. -Another important thing is I didn't use lock for syncing up networks. Although I think there -should be a lock, locking can hurt the performance heavily (about 50%). -The first 15 million frames took about 5 hours (16 processes) in a server with two Xeon E5-2620 v3. +The first 15M frames took about 5 hours (16 processes) in a server with two Xeon E5-2620 v3. This is the test curve. Test is triggered in a separate deterministic test process every 50K frames. # Dependency @@ -52,5 +51,7 @@ Detailed usage and all training details can be found in ```main.py``` * [Asynchronous Methods for Deep Reinforcement Learning](https://arxiv.org/abs/1602.01783) * [Deep Reinforcement Learning with Double Q-learning](https://arxiv.org/abs/1509.06461) * [Dueling Network Architectures for Deep Reinforcement Learning](https://arxiv.org/abs/1511.06581) +* [Playing Atari with Deep Reinforcement Learning](https://arxiv.org/abs/1312.5602) +* [HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent](https://arxiv.org/abs/1106.5730) * [transedward/pytorch-dqn](https://github.com/transedward/pytorch-dqn) * [ikostrikov/pytorch-a3c](https://github.com/ikostrikov/pytorch-a3c)