diff --git a/README.md b/README.md index 7d523cb..c8de84a 100644 --- a/README.md +++ b/README.md @@ -1,87 +1,39 @@ # DeepRL -Highly modularized implementation of popular deep RL algorithms by PyTorch. My principal here is to -reuse as much components as I can through different algorithms, use as less tricks as I can and switch -easily between classical control tasks like CartPole and Atari games with raw pixel inputs. +Highly modularized implementation of popular deep RL algorithms by PyTorch. My principal here is to reuse as much components as possible through different algorithms and switch easily between classical control tasks like CartPole and Atari games with raw pixel inputs. Implemented algorithms: -* Deep Q-Learning (DQN) -* Double DQN -* Dueling DQN -* (Async) Advantage Actor Critic (A3C / A2C) +* (Double/Dueling) Deep Q-Learning (DQN) +* Categorical DQN (C51, Distributional DQN with KL Distance) +* Quantile Regression DQN (Distributional DQN with Wasserstein Distance) +* Synchronous Advantage Actor Critic (A2C) +* Synchronous N-Step Q-Learning +* Deep Deterministic Policy Gradient (DDPG) +* Synchronous Proximal Policy Optimization (PPO) +* Action Conditional Video Prediction + +Following asynchronous algorithms are removed now in this repo but can be found in [previous release ](https://github.com/ShangtongZhang/DeepRL/releases/tag/v0.1) +* Async Advantage Actor Critic (A3C) * Async One-Step Q-Learning * Async One-Step Sarsa * Async N-Step Q-Learning * Continuous A3C * Distributed Deep Deterministic Policy Gradient (Distributed DDPG, aka D3PG) * Parallelized Proximal Policy Optimization (P3O, similar to DPPO) -* Action Conditional Video Prediction -* Categorical DQN (C51, Distributional DQN with KL Distance) -* Quantile Regression DQN (Distributional DQN with Wasserstein Distance) -* N-Step DQN (similar to A2C) # Curves -> Curves for CartPole are trivial so I didn't place it here. There isn't any fixed random seed. -## DQN, Double DQN, Dueling DQN +> Curves for CartPole are trivial so I didn't place it here. And there isn't any fixed random seed. The curves are generated in the same manner as OpenAI baselines (one run and smoothed by recent 100 episodes) +## DQN ![Loading...](https://raw.githubusercontent.com/ShangtongZhang/DeepRL/master/images/DQN-breakout.png) -![Loading...](https://raw.githubusercontent.com/ShangtongZhang/DeepRL/master/images/DQN-Pong.png) - -The network and parameters here are exactly same as the [DeepMind Nature paper](https://www.nature.com/nature/journal/v518/n7540/full/nature14236.html). -Training curve is smoothed by a window of size 100. All the models are trained in a server with -Xeon E5-2620 v3 and Titan X. For Breakout, test is triggered every 1000 episodes with 50 repetitions. -In total, 16M frames cost about 4 days and 10 hours. For Pong, test is triggered -every 10 episodes with no repetition. In total, 4M frames cost about 18 hours. - -I referred this [repo](https://github.com/transedward/pytorch-dqn). - -## Discrete A3C +## A2C ![Loading...](https://raw.githubusercontent.com/ShangtongZhang/DeepRL/master/images/A3C-Pong.png) -![Loading...](https://raw.githubusercontent.com/ShangtongZhang/DeepRL/master/images/Async-Pong.png) - -The network I used here is a smaller network with only 42 * 42 input, alougth the network for DQN can also work here, -it's quite slow. - -Training of A3C took about 2 hours (16 processes) in a server with two Xeon E5-2620 v3. While other async methods took about 1 day. -Those value based async methods do work but I don't know how to make them stable. -This is the test curve. Test is triggered in a separate deterministic test process every 50K frames. - -I referred this [repo](https://github.com/ikostrikov/pytorch-a3c) for the parallelization. - -## Continuous A3C -![Loading...](https://raw.githubusercontent.com/ShangtongZhang/DeepRL/master/images/Continuous-A3C.png) - -For continuous A3C and DPPO, I use fixed unit variance rather than a separate head, so entropy weight is simply set to 0. -Of course you can also use another head to output variance. In that case, a good practice is to bound your mean while leave -variance unbounded, which is also included in the implementation. - -## D3PG +## DDPG ![Loading...](https://raw.githubusercontent.com/ShangtongZhang/DeepRL/master/images/DDPG.png) -Extra caution is necessary when computing gradients. The [repo](https://github.com/ghliu/pytorch-ddpg) I referred -for DDPG is wrong in computing the deterministic gradients at least at this [commit](https://github.com/ghliu/pytorch-ddpg/tree/ffea335ee53f2ff90b6d7eaf9d0cee705270c0f1). -Theoretically I believe that implementation should work, but in practice it doesn't work. Even this is PyTorch you need to manually deal with gradients in this case. -DDPG is not very stable. - -Setting the number of workers to 1 will reduce the implementation to exact DDPG. I have to adopt the most straightforward distribution method, as -P3O and A3C style distribution doesn't work for DDPG. The figures were done with 6 workers. - - -## P3O - +## PPO ![Loading...](https://raw.githubusercontent.com/ShangtongZhang/DeepRL/master/images/P3O.png) -The difference between my implementation and [DeepMind's DPPO](https://arxiv.org/abs/1707.02286) is: -1. PPO stands for different algorithms. -2. I use a much simpler A3C-like synchronization protocol. - -The body of PPO is based on this [repo](https://github.com/alexis-jacq/Pytorch-DPPO). -However that implementation has two critical bugs at least at this [commit](https://github.com/ghliu/pytorch-ddpg/tree/ffea335ee53f2ff90b6d7eaf9d0cee705270c0f1). -Its computation of the clipped loss is correct with one-dimensional action by accident, -but is wrong with high-dimensional action. And its computation of entropy is wrong in any case. - -I use 8 threads and a two tanh hidden layer network, each hidden layer has 64 hidden units. - ## Action Conditional Video Prediction ![Loading...](https://raw.githubusercontent.com/ShangtongZhang/DeepRL/master/images/ACVP.png) @@ -93,22 +45,19 @@ Prediction is sampled after 110K iterations and I only implemented one-step trai ## Categorical DQN ![Loading...](https://raw.githubusercontent.com/ShangtongZhang/DeepRL/master/images/CategoricalDQN.png) -A deterministic test episode is triggered every 10 episodes. 2.5M steps and 14 hours in total. ## Quantile Regression DQN ![Loading...](https://raw.githubusercontent.com/ShangtongZhang/DeepRL/master/images/QRDQN.png) -A deterministic test episode is triggered every 10 episodes. 2M steps and 8 hours in total. -## A2C & N-Step DQN +## N-Step Q-Learning ![Loading...](https://raw.githubusercontent.com/ShangtongZhang/DeepRL/master/images/A2C-NStepQ.png) -Online training progression of a single run. Entropy regularization is used for A2C, resulting in the variance in the curve. # Dependency > Tested in macOS 10.12 and CentO/S 6.8 -* Open AI gym +* OpenAI gym * [Roboschool](https://github.com/openai/roboschool) (Optional) * PyTorch v0.3.0 -* Python 2.7 +* Python 2.7 / 3.6 * [TensorboardX](https://github.com/lanpa/tensorboard-pytorch) diff --git a/agent/A2C_agent.py b/agent/A2C_agent.py index 45a0652..3b69cc3 100644 --- a/agent/A2C_agent.py +++ b/agent/A2C_agent.py @@ -31,8 +31,7 @@ class A2CAgent(BaseAgent): rollout = [] states = self.states for i in range(config.rollout_length): - states = config.state_normalizer(states) - prob, log_prob, value = self.network.predict(states) + prob, log_prob, value = self.network.predict(config.state_normalizer(states)) actions = [self.policy.sample(p) for p in prob.data.cpu().numpy()] next_states, rewards, terminals, _ = self.task.step(actions) self.episode_rewards += rewards diff --git a/agent/CategoricalDQN_agent.py b/agent/CategoricalDQN_agent.py index 34e9922..c8d9ccd 100644 --- a/agent/CategoricalDQN_agent.py +++ b/agent/CategoricalDQN_agent.py @@ -95,6 +95,7 @@ class CategoricalDQNAgent(BaseAgent): # self.config.logger.scalar_summary('loss', loss.data.cpu().numpy().flatten(), self.total_steps) self.optimizer.zero_grad() loss.backward() + nn.utils.clip_grad_norm(self.network.parameters(), self.config.gradient_clip) self.optimizer.step() if not deterministic and self.total_steps % self.config.target_network_update_freq == 0: self.target_network.load_state_dict(self.network.state_dict()) diff --git a/component/bench.py b/component/bench.py index 59e68fe..75cc99f 100644 --- a/component/bench.py +++ b/component/bench.py @@ -1,7 +1,5 @@ # from https://raw.githubusercontent.com/openai/baselines/master/baselines/bench/monitor.py -__all__ = ['Monitor', 'get_monitor_files', 'load_results'] - import gym from gym.core import Wrapper import time @@ -100,7 +98,7 @@ class LoadMonitorResultsError(Exception): def get_monitor_files(dir): return glob(osp.join(dir, "*" + Monitor.EXT)) -def load_results(dir): +def load_monitor_log(dir): import pandas monitor_files = ( glob(osp.join(dir, "*monitor.json")) + diff --git a/images/A2C-NStepQ.png b/images/A2C-NStepQ.png deleted file mode 100644 index d5e2813..0000000 Binary files a/images/A2C-NStepQ.png and /dev/null differ diff --git a/images/A3C-Pong.png b/images/A3C-Pong.png deleted file mode 100644 index 067b577..0000000 Binary files a/images/A3C-Pong.png and /dev/null differ diff --git a/images/ACVP.png b/images/ACVP.png deleted file mode 100644 index 8ce8312..0000000 Binary files a/images/ACVP.png and /dev/null differ diff --git a/images/Async-Pong.png b/images/Async-Pong.png deleted file mode 100644 index 3398eb4..0000000 Binary files a/images/Async-Pong.png and /dev/null differ diff --git a/images/CategoricalDQN.png b/images/CategoricalDQN.png deleted file mode 100644 index 3457d5f..0000000 Binary files a/images/CategoricalDQN.png and /dev/null differ diff --git a/images/Continuous-A3C.png b/images/Continuous-A3C.png deleted file mode 100644 index b5bc31d..0000000 Binary files a/images/Continuous-A3C.png and /dev/null differ diff --git a/images/DDPG.png b/images/DDPG.png deleted file mode 100644 index 164511c..0000000 Binary files a/images/DDPG.png and /dev/null differ diff --git a/images/DQN-Pong.png b/images/DQN-Pong.png deleted file mode 100644 index 1474962..0000000 Binary files a/images/DQN-Pong.png and /dev/null differ diff --git a/images/DQN-breakout.png b/images/DQN-breakout.png deleted file mode 100644 index 707f433..0000000 Binary files a/images/DQN-breakout.png and /dev/null differ diff --git a/images/P3O.png b/images/P3O.png deleted file mode 100644 index 7d21f7d..0000000 Binary files a/images/P3O.png and /dev/null differ diff --git a/images/QRDQN.png b/images/QRDQN.png deleted file mode 100644 index 27e05ee..0000000 Binary files a/images/QRDQN.png and /dev/null differ diff --git a/main.py b/main.py index 87508ea..6c69cb1 100644 --- a/main.py +++ b/main.py @@ -105,7 +105,7 @@ def dqn_pixel_atari(name): config.network_fn = lambda state_dim, action_dim: ConvNet(config.history_length, action_dim, gpu=0) # config.network_fn = lambda state_dim, action_dim: DuelingConvNet(config.history_length, action_dim) config.policy_fn = lambda: GreedyPolicy(epsilon=1.0, final_step=1000000, min_epsilon=0.1) - config.replay_fn = lambda: Replay(memory_size=1000000, batch_size=32, dtype=np.uint8) + config.replay_fn = lambda: Replay(memory_size=100000, batch_size=32, dtype=np.uint8) config.state_normalizer = ImageNormalizer() config.reward_normalizer = SignNormalizer() config.discount = 0.99 @@ -119,7 +119,7 @@ def dqn_pixel_atari(name): def a2c_pixel_atari(name): config = Config() config.history_length = 4 - config.num_workers = 5 + config.num_workers = 16 task_fn = lambda log_dir: PixelAtari(name, frame_skip=4, history_length=config.history_length, log_dir=log_dir) config.task_fn = lambda: ParallelizedTask(task_fn, config.num_workers, log_dir=get_default_log_dir(a2c_pixel_atari.__name__)) config.optimizer_fn = lambda params: torch.optim.RMSprop(params, lr=0.0007) @@ -143,9 +143,10 @@ def categorical_dqn_pixel_atari(name): config.task_fn = lambda: PixelAtari(name, frame_skip=4, history_length=config.history_length, log_dir=get_default_log_dir(categorical_dqn_pixel_atari.__name__)) config.optimizer_fn = lambda params: torch.optim.Adam(params, lr=0.00025, eps=0.01 / 32) - config.network_fn = lambda state_dim, action_dim: CategoricalConvNet(config.history_length, action_dim, config.categorical_n_atoms, gpu=0) + config.network_fn = lambda state_dim, action_dim: \ + CategoricalConvNet(config.history_length, action_dim, config.categorical_n_atoms, gpu=1) config.policy_fn = lambda: GreedyPolicy(epsilon=1.0, final_step=1000000, min_epsilon=0.1) - config.replay_fn = lambda: Replay(memory_size=1000000, batch_size=32, dtype=np.uint8) + config.replay_fn = lambda: Replay(memory_size=100000, batch_size=32, dtype=np.uint8) config.discount = 0.99 config.state_normalizer = ImageNormalizer() config.reward_normalizer = SignNormalizer() @@ -164,9 +165,10 @@ def quantile_regression_dqn_pixel_atari(name): config.task_fn = lambda: PixelAtari(name, frame_skip=4, history_length=config.history_length, log_dir=get_default_log_dir(quantile_regression_dqn_pixel_atari.__name__)) config.optimizer_fn = lambda params: torch.optim.Adam(params, lr=0.00005, eps=0.01 / 32) - config.network_fn = lambda state_dim, action_dim: QuantileConvNet(config.history_length, action_dim, config.num_quantiles, gpu=0) + config.network_fn = lambda state_dim, action_dim: \ + QuantileConvNet(config.history_length, action_dim, config.num_quantiles, gpu=2) config.policy_fn = lambda: GreedyPolicy(epsilon=1.0, final_step=1000000, min_epsilon=0.01) - config.replay_fn = lambda: Replay(memory_size=1000000, batch_size=32, dtype=np.uint8) + config.replay_fn = lambda: Replay(memory_size=100000, batch_size=32, dtype=np.uint8) config.state_normalizer = ImageNormalizer() config.reward_normalizer = SignNormalizer() config.discount = 0.99 @@ -262,6 +264,16 @@ def ddpg_continuous(): config.logger = Logger('./log', logger) run_episodes(DDPGAgent(config)) +def plot(): + import matplotlib.pyplot as plt + plotter = Plotter() + name = 'to_plot/a2c_pixel_atari-180407-92711' + # name = 'to_plot/dqn_pixel_atari-180407-01414' + # name = 'to_plot/quantile_regression_dqn_pixel_atari-180407-01604' + # name = 'to_plot/categorical_dqn_pixel_atari-180407-01537' + plotter.plot_results([name]) + plt.show() + if __name__ == '__main__': mkdir('data') mkdir('data/video') @@ -288,3 +300,5 @@ if __name__ == '__main__': # acvp.train('PongNoFrameskip-v4') + plot() + diff --git a/utils/__init__.py b/utils/__init__.py index e71c201..0b89911 100644 --- a/utils/__init__.py +++ b/utils/__init__.py @@ -2,6 +2,7 @@ from .config import * from .normalizer import * from .misc import * from .tf_logger import Logger +from .plot import Plotter import logging logging.basicConfig(format='%(asctime)s - %(name)s - %(levelname)s: %(message)s') logger = logging.getLogger('MAIN') diff --git a/utils/misc.py b/utils/misc.py index 7690762..2ce5ae0 100644 --- a/utils/misc.py +++ b/utils/misc.py @@ -70,7 +70,7 @@ def run_iterations(agent): return steps, rewards def get_time_str(): - return datetime.datetime.now().strftime("%y%m%d-%-H%M%S") + return datetime.datetime.now().strftime("%y%m%d-%H%M%S") def get_default_log_dir(name): return './log/%s-%s' % (name, get_time_str()) diff --git a/utils/plot.py b/utils/plot.py index 0d875e2..ae22978 100644 --- a/utils/plot.py +++ b/utils/plot.py @@ -1,5 +1,5 @@ import numpy as np -from component import load_results +import component class Plotter: COLORS = ['blue', 'green', 'red', 'cyan', 'magenta', 'yellow', 'black', 'purple', 'pink', @@ -40,7 +40,7 @@ class Plotter: def load_results(self, dirs, max_timesteps=1e8, x_axis=X_TIMESTEPS, episode_window=100): tslist = [] for dir in dirs: - ts = load_results(dir) + ts = component.load_monitor_log(dir) ts = ts[ts.l.cumsum() <= max_timesteps] tslist.append(ts) xy_list = [self.ts2xy(ts, x_axis) for ts in tslist] @@ -49,9 +49,10 @@ class Plotter: def plot_results(self, dirs, max_timesteps=1e8, x_axis=X_TIMESTEPS, episode_window=100): import matplotlib.pyplot as plt + plt.ticklabel_format(axis='x', style='sci', scilimits=(1, 1)) xy_list = self.load_results(dirs, max_timesteps, x_axis, episode_window) - for (i, (x, y, y_mean)) in enumerate(xy_list): + for (i, (x, y, smoothed)) in enumerate(xy_list): color = Plotter.COLORS[i] - plt.plot(x, y_mean, color=color) + plt.plot(smoothed[0], smoothed[1], color=color) plt.xlabel(x_axis) plt.ylabel("Episode Rewards")