first commit

CODEOWNERS

@@ -0,0 +1,2 @@
+# Comment line immediately above ownership line is reserved for related other information. Please be careful while editing.
+#ECCN:Open Source

CODE_OF_CONDUCT.md

@@ -0,0 +1,105 @@
+# Salesforce Open Source Community Code of Conduct
+
+## About the Code of Conduct
+
+Equality is a core value at Salesforce. We believe a diverse and inclusive
+community fosters innovation and creativity, and are committed to building a
+culture where everyone feels included.
+
+Salesforce open-source projects are committed to providing a friendly, safe, and
+welcoming environment for all, regardless of gender identity and expression,
+sexual orientation, disability, physical appearance, body size, ethnicity, nationality, 
+race, age, religion, level of experience, education, socioeconomic status, or 
+other similar personal characteristics.
+
+The goal of this code of conduct is to specify a baseline standard of behavior so
+that people with different social values and communication styles can work
+together effectively, productively, and respectfully in our open source community.
+It also establishes a mechanism for reporting issues and resolving conflicts.
+
+All questions and reports of abusive, harassing, or otherwise unacceptable behavior
+in a Salesforce open-source project may be reported by contacting the Salesforce
+Open Source Conduct Committee at ossconduct@salesforce.com.
+
+## Our Pledge
+
+In the interest of fostering an open and welcoming environment, we as
+contributors and maintainers pledge to making participation in our project and
+our community a harassment-free experience for everyone, regardless of gender 
+identity and expression, sexual orientation, disability, physical appearance, 
+body size, ethnicity, nationality, race, age, religion, level of experience, education, 
+socioeconomic status, or other similar personal characteristics.
+
+## Our Standards
+
+Examples of behavior that contributes to creating a positive environment
+include:
+
+* Using welcoming and inclusive language
+* Being respectful of differing viewpoints and experiences
+* Gracefully accepting constructive criticism
+* Focusing on what is best for the community
+* Showing empathy toward other community members
+
+Examples of unacceptable behavior by participants include:
+
+* The use of sexualized language or imagery and unwelcome sexual attention or
+advances
+* Personal attacks, insulting/derogatory comments, or trolling
+* Public or private harassment
+* Publishing, or threatening to publish, others' private information—such as
+a physical or electronic address—without explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+professional setting
+* Advocating for or encouraging any of the above behaviors
+
+## Our Responsibilities
+
+Project maintainers are responsible for clarifying the standards of acceptable
+behavior and are expected to take appropriate and fair corrective action in
+response to any instances of unacceptable behavior.
+
+Project maintainers have the right and responsibility to remove, edit, or
+reject comments, commits, code, wiki edits, issues, and other contributions
+that are not aligned with this Code of Conduct, or to ban temporarily or
+permanently any contributor for other behaviors that they deem inappropriate,
+threatening, offensive, or harmful.
+
+## Scope
+
+This Code of Conduct applies both within project spaces and in public spaces
+when an individual is representing the project or its community. Examples of
+representing a project or community include using an official project email
+address, posting via an official social media account, or acting as an appointed
+representative at an online or offline event. Representation of a project may be
+further defined and clarified by project maintainers.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported by contacting the Salesforce Open Source Conduct Committee 
+at ossconduct@salesforce.com. All complaints will be reviewed and investigated 
+and will result in a response that is deemed necessary and appropriate to the 
+circumstances. The committee is obligated to maintain confidentiality with 
+regard to the reporter of an incident. Further details of specific enforcement 
+policies may be posted separately.
+
+Project maintainers who do not follow or enforce the Code of Conduct in good
+faith may face temporary or permanent repercussions as determined by other
+members of the project's leadership and the Salesforce Open Source Conduct 
+Committee.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][contributor-covenant-home],
+version 1.4, available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html. 
+It includes adaptions and additions from [Go Community Code of Conduct][golang-coc], 
+[CNCF Code of Conduct][cncf-coc], and [Microsoft Open Source Code of Conduct][microsoft-coc].
+
+This Code of Conduct is licensed under the [Creative Commons Attribution 3.0 License][cc-by-3-us].
+
+[contributor-covenant-home]: https://www.contributor-covenant.org (https://www.contributor-covenant.org/)
+[golang-coc]: https://golang.org/conduct
+[cncf-coc]: https://github.com/cncf/foundation/blob/master/code-of-conduct.md
+[microsoft-coc]: https://opensource.microsoft.com/codeofconduct/
+[cc-by-3-us]: https://creativecommons.org/licenses/by/3.0/us/

LICENSE.txt

@@ -0,0 +1,12 @@
+Copyright (c) 2022, Salesforce.com, Inc.
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
+
+* Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
+
+* Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
+
+* Neither the name of Salesforce.com nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Makefile

@@ -0,0 +1,27 @@
+ROOT := $(shell dirname $(realpath $(firstword ${MAKEFILE_LIST})))
+
+build: .require-config
+	python -m experiments.forecast --config_path=${config} build_experiment
+
+build-all: .require-path
+	for config in $(shell ls ${path}/*.gin); do \
+      make build config=$$config; \
+    done
+
+run: .require-command
+	bash -c "`cat ${ROOT}/${command}`"
+
+.require-config:
+ifndef config
+	$(error config is required)
+endif
+
+.require-command:
+ifndef command
+	$(error command is required)
+endif
+
+.require-path:
+ifndef path
+	$(error path is required)
+endif

README.md

@@ -0,0 +1,87 @@
+# DeepTIMe: Deep Time-Index Meta-Learning for Non-stationary Forecasting
+
+<p align="center">
+<img src=".\pics\deeptime.png" width = "700" alt="" align=center />
+<br><br>
+<b>Figure 1.</b> Overall approach of DeepTIMe.
+</p>
+
+Official PyTorch code repository for the DeepTIMe paper.
+
+## Requirements
+
+Dependencies for this project can be installed by:
+
+```bash
+pip install -r requirements.txt
+```
+
+## Quick Start
+
+### Data
+
+To get started, you will need to download the datasets as described in our paper:
+
+* Pre-processed datasets can be downloaded from the following
+  links, [Tsinghua Cloud](https://cloud.tsinghua.edu.cn/d/e1ccfff39ad541908bae/)
+  or [Google Drive](https://drive.google.com/drive/folders/1ZOYpTUa82_jCcxIdTmyr0LXQfvaM9vIy?usp=sharing), as obtained
+  from [Autoformer's](https://github.com/thuml/Autoformer) GitHub repository.
+* Place the downloaded datasets into the `storage/datasets/` folder, e.g. `storage/datasets/ETT-small/ETTm2.csv`.
+
+### Reproducing Experiment Results
+
+We provide some scripts to quickly reproduce the results reported in our paper. There are two options, to run the full
+hyperparameter search, or to directly run the experiments with hyperparameters provided in the configuration files.
+
+__Option A__: Run the full hyperparameter search.
+
+1. Run the following command to generate the experiments: `make build-all path=experiments/configs/hp_search`.
+2. Run the following script to perform training and evaluation: `./run_hp_search.sh` (you may need to
+   run `chmod u+x run_hp_search.sh` first).
+
+__Option B__: Directly run the experiments with hyperparameters provided in the configuration files.
+
+1. Run the following command to generate the experiments: `make build-all path=experiments/configs`.
+2. Run the following script to perform training and evaluation: `./run.sh` (you may need to run `chmod u+x run.sh`
+   first).
+
+Finally, results can be viewed on tensorboard by running `tensorboard --logdir storage/experiments/`, or in
+the `storage/experiments/experiment_name/metrics.npy` file.
+
+## Detailed Usage
+
+Further details of the code repository can be found here. The codebase is structured to generate experiments from
+a `.gin` configuration file based on the `build.variables_dict` argument.
+
+1. First, build the experiment from a config file. We provide 2 ways to build an experiment.
+    1. Build a single config file:
+       ```
+       make build config=experiments/configs/folder_name/file_name.gin
+       ```
+    2. Build a group of config files:
+       ```bash
+       make build-all path=experiments/configs/folder_name
+       ```
+2. Next, run the experiment using the following command
+    ```bash 
+    python -m experiments.forecast --config_path=storage/experiments/experiment_name/config.gin run
+   ```
+   Alternatively, the first step generates a command file found in `storage/experiments/experiment_name/command`, which
+   you can use by the following command,
+   ```bash
+   make run command=storage/experiments/experiment_name/command
+   ```
+3. Finally, you can observe the results on tensorboard
+   ```bash
+   tensorboard --logdir storage/experiments/
+   ``` 
+   or view the `storage/experiments/deeptime/experiment_name/metrics.npy` file.
+
+## Acknowledgements
+
+The implementation of DeepTIMe relies on resources from the following codebases and repositories, we thank the original
+authors for open-sourcing their work.
+
+* https://github.com/ElementAI/N-BEATS
+* https://github.com/zhouhaoyi/Informer2020
+* https://github.com/thuml/Autoformer

SECURITY.md

@@ -0,0 +1,7 @@
+## Security
+
+Please report any security issue to [security@salesforce.com](mailto:security@salesforce.com)
+as soon as it is discovered. This library limits its runtime dependencies in
+order to reduce the total cost of ownership as much as can be, but all consumers
+should remain vigilant and have their security stakeholders review all third-party
+products (3PP) like this one and their dependencies.

data/datasets.py

@@ -0,0 +1,150 @@
+import os
+from os.path import join
+from typing import Optional, List, Tuple
+
+import gin
+import numpy as np
+import pandas as pd
+from sklearn.preprocessing import StandardScaler
+from torch.utils.data import Dataset, DataLoader
+
+from utils.time_features import get_time_features
+
+
+@gin.configurable()
+class ForecastDataset(Dataset):
+    def __init__(self,
+                 flag: str,
+                 horizon_len: int,
+                 scale: bool,
+                 cross_learn: bool,
+                 data_path: str,
+                 root_path: Optional[str] = 'storage/datasets',
+                 features: Optional[str] = 'S',
+                 target: Optional[str] = 'OT',
+                 lookback_len: Optional[int] = None,
+                 lookback_aux_len: Optional[int] = 0,
+                 lookback_mult: Optional[float] = None,
+                 time_features: Optional = [],
+                 normalise_time_features: Optional = True,):
+        """
+        :param flag: train/val/test flag
+        :param horizon_len: number of time steps in forecast horizon
+        :param scale: performs standard scaling
+        :param data_path: relative (to root_path) path to data file (.csv)
+        :param cross_learn: treats multivariate time series as multiple univar time series
+        :param root_path: path to datasets folder
+        :param features: multivar (M) or univar (S) forecasting
+        :param target: name of target variable for univar forecasting (features=S)
+        :param lookback_len: number of time steps in lookback window
+        :param lookback_aux_len: number of time steps to append to y from the lookback window
+        (for models with decoders which requires initialisation)
+        :param lookback_mult: multiplier to decide lookback window length
+        """
+        assert flag in ('train', 'val', 'test'), \
+            f"flag should be one of (train, val, test)"
+        assert features in ('M', 'S'), \
+            f"features should be one of (M: multivar, S: univar)"
+        assert (lookback_len is not None) ^ (lookback_mult is not None), \
+            f"only 'lookback_len' xor 'lookback_mult' should be specified"
+
+        self.flag = flag
+        self.lookback_len = lookback_len or int(horizon_len * lookback_mult)
+        self.lookback_aux_len = lookback_aux_len
+        self.horizon_len = horizon_len
+        self.scale = scale
+        self.cross_learn = cross_learn
+        self.data_path = data_path
+        self.root_path = root_path
+        self.features = features
+        self.target = target
+        self.time_features = time_features
+        self.normalise_time_features = normalise_time_features
+
+        self.n_dims = None
+        self.scaler = None
+        self.data_x = None
+        self.data_y = None
+        self.timestamps = None
+        self.n_time = None
+        self.n_time_samples = None
+        self.load_data()
+
+    def load_data(self):
+        df_raw = pd.read_csv(join(self.root_path, self.data_path))  # (n_obs, date + n_feats)
+        cols = list(df_raw.columns)
+        cols.remove('date')
+        cols.remove(self.target)
+        df_raw = df_raw[['date'] + cols + [self.target]]
+        border1s, border2s, border1, border2 = self.get_borders(df_raw)
+
+        if self.features == 'M':
+            df_data = df_raw[cols + [self.target]]
+            self.n_dims = len(cols + [self.target])
+        elif self.features == 'S':
+            df_data = df_raw[[self.target]]
+            self.n_dims = 1
+        else:
+            raise ValueError
+
+        self.scaler = StandardScaler()
+        if self.scale:
+            train_data = df_data[border1s[0]:border2s[0]]
+            self.scaler.fit(train_data.values)
+            data = self.scaler.transform(df_data.values)
+        else:
+            data = df_data.values
+
+        self.data_x = data[border1:border2]
+        self.data_y = data[border1:border2]
+        self.timestamps = get_time_features(pd.to_datetime(df_raw.date[border1:border2].values),
+                                            normalise=self.normalise_time_features,
+                                            features=self.time_features)
+        self.n_time = len(self.data_x)
+        self.n_time_samples = self.n_time - self.lookback_len - self.horizon_len + 1
+
+    def get_borders(self, df_raw: pd.DataFrame) -> Tuple[List[int], List[int], List[int], List[int]]:
+        set_type = {'train': 0, 'val': 1, 'test': 2}[self.flag]
+        if self.data_path.startswith('ETT-small/ETTh'):
+            border1s = [0, 12 * 30 * 24 - self.lookback_len, 12 * 30 * 24 + 4 * 30 * 24 - self.lookback_len]
+            border2s = [12 * 30 * 24, 12 * 30 * 24 + 4 * 30 * 24, 12 * 30 * 24 + 8 * 30 * 24]
+        elif self.data_path.startswith('ETT-small/ETTm'):
+            border1s = [0, 12 * 30 * 24 * 4 - self.lookback_len, 12 * 30 * 24 * 4 + 4 * 30 * 24 * 4 - self.lookback_len]
+            border2s = [12 * 30 * 24 * 4, 12 * 30 * 24 * 4 + 4 * 30 * 24 * 4, 12 * 30 * 24 * 4 + 8 * 30 * 24 * 4]
+        else:
+            num_train = int(len(df_raw) * 0.7)
+            num_test = int(len(df_raw) * 0.2)
+            num_val = len(df_raw) - num_train - num_test
+            border1s = [0, num_train - self.lookback_len, len(df_raw) - num_test - self.lookback_len]
+            border2s = [num_train, num_train + num_val, len(df_raw)]
+        border1 = border1s[set_type]
+        border2 = border2s[set_type]
+        return border1s, border2s, border1, border2
+
+    def __len__(self):
+        if self.cross_learn:
+            return self.n_time_samples * self.n_dims
+        return self.n_time_samples
+
+    def __getitem__(self, idx: int) -> Tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray]:
+        if self.cross_learn:
+            dim_idx = idx // self.n_time_samples
+            dim_slice = slice(dim_idx, dim_idx + 1)
+            idx = idx % self.n_time_samples
+        else:
+            dim_slice = slice(None)
+
+        x_start = idx
+        x_end = x_start + self.lookback_len
+        y_start = x_end - self.lookback_aux_len
+        y_end = y_start + self.lookback_aux_len + self.horizon_len
+
+        x = self.data_x[x_start:x_end, dim_slice]
+        y = self.data_y[y_start:y_end, dim_slice]
+        x_time = self.timestamps[x_start:x_end]
+        y_time = self.timestamps[y_start:y_end]
+
+        return x, y, x_time, y_time
+
+    def inverse_transform(self, data):
+        return self.scaler.inverse_transform(data)

experiments/base.py

@@ -0,0 +1,105 @@
+import os
+import logging
+import time
+import random
+from abc import ABC, abstractmethod
+from pathlib import Path
+from shutil import copy
+from typing import Dict, List, Union, Optional
+from itertools import product
+
+from tqdm import tqdm
+import gin
+
+
+EXPERIMENTS_PATH = 'storage/experiments'
+SearchSpace = List[Union[str, int, float]]
+
+
+class Experiment(ABC):
+
+    def __init__(self, config_path: str):
+        self.config_path = config_path
+        self.root = Path(config_path).parent
+        gin.parse_config_file(self.config_path)
+
+    @gin.configurable()
+    def build(self,
+              experiment_name: str,
+              module: str,
+              repeat: int,
+              variables_dict: Dict[str, SearchSpace]):
+        """
+        Builds an experiment, which consists of a list of instances.
+        Can be used for hyperparam optimization, or training an ensemble.
+        :param experiment_name: Name of experiment.
+        :param module: Name of the file to run.
+        :param repeat: Number of repeated instances per hyperparam setting.
+        :param variables_dict: Dictionary containing hyperparams to test.
+        """
+        # create experiment instance(s)
+        logging.info('Creating experiment instances ...')
+        experiment_path = os.path.join(EXPERIMENTS_PATH, experiment_name)
+        variables_dict['repeat'] = list(range(repeat))
+        variable_names, variables = zip(*variables_dict.items())
+        for instance_values in tqdm(product(*variables)):
+            instance_variables = dict(zip(variable_names, instance_values))
+            instance_name = ','.join(['%s=%.4g' % (name.split('.')[-1], value)
+                                      if isinstance(value, float)
+                                      else '%s=%s' % (name.split('.')[-1], str(value).replace(' ', '_'))
+                                      for name, value in instance_variables.items()])
+            instance_path = os.path.join(experiment_path, instance_name)
+            Path(instance_path).mkdir(parents=True, exist_ok=False)
+
+            # write parameters
+            instance_config_path = os.path.join(instance_path, 'config.gin')
+            copy(self.config_path, instance_config_path)
+            with open(instance_config_path, 'a') as cfg:
+                for name, value in instance_variables.items():
+                    value = f"'{value}'" if isinstance(value, str) else str(value)
+                    cfg.write(f'{name} = {value}\n')
+
+            # write command file
+            command_file = os.path.join(instance_path, 'command')
+            with open(command_file, 'w') as cmd:
+                cmd.write(f'python -m {module} '
+                          f'--config_path={instance_config_path} '
+                          f'run >> {instance_path}/instance.log 2>&1')
+
+    @abstractmethod
+    def instance(self):
+        """
+        Instance logic method must be implemented with @gin.configurable()
+        """
+        ...
+
+    @gin.configurable()
+    def run(self, timer: Optional[int] = 0):
+        """
+        Run instance logic.
+        """
+        time.sleep(random.uniform(0, timer))
+        running_flag = os.path.join(self.root, '_RUNNING')
+        success_flag = os.path.join(self.root, '_SUCCESS')
+        if os.path.isfile(success_flag) or os.path.isfile(running_flag):
+            return
+        elif not os.path.isfile(running_flag):
+            Path(running_flag).touch()
+
+        try:
+            self.instance()
+        except Exception as e:
+            Path(running_flag).unlink()
+            raise e
+        except KeyboardInterrupt:
+            Path(running_flag).unlink()
+            raise Exception('KeyboardInterrupt')
+
+        # mark experiment as finished.
+        Path(running_flag).unlink()
+        Path(success_flag).touch()
+
+    def build_experiment(self):
+        if EXPERIMENTS_PATH in str(self.root):
+            raise Exception('Cannot build ensemble from ensemble member configuration.')
+        self.build()

experiments/configs/ECL/192M.gin

@@ -0,0 +1,37 @@
+build.experiment_name = 'ECL/192M'
+build.module = 'experiments.forecast'
+build.repeat = 1
+build.variables_dict = {
+}
+
+instance.model_type = 'deeptime'
+instance.save_vals = False
+
+get_optimizer.lr = 1e-3
+get_optimizer.lambda_lr = 1.
+get_optimizer.weight_decay = 0.
+
+get_scheduler.warmup_epochs = 5
+
+get_data.batch_size = 256
+
+train.loss_name = 'mse'
+train.epochs = 50
+train.clip = 10.
+
+Checkpoint.patience = 7
+
+deeptime.layer_size = 256
+deeptime.inr_layers = 5
+deeptime.n_fourier_feats = 4096
+deeptime.scales = [0.01, 0.1, 1, 5, 10, 20, 50, 100]
+
+ForecastDataset.data_path = 'electricity/electricity.csv'
+ForecastDataset.target = 'OT'
+ForecastDataset.scale = True
+ForecastDataset.cross_learn = False
+ForecastDataset.time_features = []
+ForecastDataset.normalise_time_features = True
+ForecastDataset.features = 'M'
+ForecastDataset.horizon_len = 192
+ForecastDataset.lookback_mult = 5

experiments/configs/ECL/336M.gin

@@ -0,0 +1,37 @@
+build.experiment_name = 'ECL/336M'
+build.module = 'experiments.forecast'
+build.repeat = 1
+build.variables_dict = {
+}
+
+instance.model_type = 'deeptime'
+instance.save_vals = False
+
+get_optimizer.lr = 1e-3
+get_optimizer.lambda_lr = 1.
+get_optimizer.weight_decay = 0.
+
+get_scheduler.warmup_epochs = 5
+
+get_data.batch_size = 256
+
+train.loss_name = 'mse'
+train.epochs = 50
+train.clip = 10.
+
+Checkpoint.patience = 7
+
+deeptime.layer_size = 256
+deeptime.inr_layers = 5
+deeptime.n_fourier_feats = 4096
+deeptime.scales = [0.01, 0.1, 1, 5, 10, 20, 50, 100]
+
+ForecastDataset.data_path = 'electricity/electricity.csv'
+ForecastDataset.target = 'OT'
+ForecastDataset.scale = True
+ForecastDataset.cross_learn = False
+ForecastDataset.time_features = []
+ForecastDataset.normalise_time_features = True
+ForecastDataset.features = 'M'
+ForecastDataset.horizon_len = 336
+ForecastDataset.lookback_mult = 3

experiments/configs/ECL/720M.gin

@@ -0,0 +1,37 @@
+build.experiment_name = 'ECL/720M'
+build.module = 'experiments.forecast'
+build.repeat = 1
+build.variables_dict = {
+}
+
+instance.model_type = 'deeptime'
+instance.save_vals = False
+
+get_optimizer.lr = 1e-3
+get_optimizer.lambda_lr = 1.
+get_optimizer.weight_decay = 0.
+
+get_scheduler.warmup_epochs = 5
+
+get_data.batch_size = 256
+
+train.loss_name = 'mse'
+train.epochs = 50
+train.clip = 10.
+
+Checkpoint.patience = 7
+
+deeptime.layer_size = 256
+deeptime.inr_layers = 5
+deeptime.n_fourier_feats = 4096
+deeptime.scales = [0.01, 0.1, 1, 5, 10, 20, 50, 100]
+
+ForecastDataset.data_path = 'electricity/electricity.csv'
+ForecastDataset.target = 'OT'
+ForecastDataset.scale = True
+ForecastDataset.cross_learn = False
+ForecastDataset.time_features = []
+ForecastDataset.normalise_time_features = True
+ForecastDataset.features = 'M'
+ForecastDataset.horizon_len = 720
+ForecastDataset.lookback_mult = 1

experiments/configs/ECL/96M.gin

@@ -0,0 +1,37 @@
+build.experiment_name = 'ECL/96M'
+build.module = 'experiments.forecast'
+build.repeat = 1
+build.variables_dict = {
+}
+
+instance.model_type = 'deeptime'
+instance.save_vals = False
+
+get_optimizer.lr = 1e-3
+get_optimizer.lambda_lr = 1.
+get_optimizer.weight_decay = 0.
+
+get_scheduler.warmup_epochs = 5
+
+get_data.batch_size = 256
+
+train.loss_name = 'mse'
+train.epochs = 50
+train.clip = 10.
+
+Checkpoint.patience = 7
+
+deeptime.layer_size = 256
+deeptime.inr_layers = 5
+deeptime.n_fourier_feats = 4096
+deeptime.scales = [0.01, 0.1, 1, 5, 10, 20, 50, 100]
+
+ForecastDataset.data_path = 'electricity/electricity.csv'
+ForecastDataset.target = 'OT'
+ForecastDataset.scale = True
+ForecastDataset.cross_learn = False
+ForecastDataset.time_features = []
+ForecastDataset.normalise_time_features = True
+ForecastDataset.features = 'M'
+ForecastDataset.horizon_len = 96
+ForecastDataset.lookback_mult = 9

experiments/configs/ETTm2/192M.gin

@@ -0,0 +1,37 @@
+build.experiment_name = 'ETTm2/192M'
+build.module = 'experiments.forecast'
+build.repeat = 1
+build.variables_dict = {
+}
+
+instance.model_type = 'deeptime'
+instance.save_vals = False
+
+get_optimizer.lr = 1e-3
+get_optimizer.lambda_lr = 1.
+get_optimizer.weight_decay = 0.
+
+get_scheduler.warmup_epochs = 5
+
+get_data.batch_size = 256
+
+train.loss_name = 'mse'
+train.epochs = 50
+train.clip = 10.
+
+Checkpoint.patience = 7
+
+deeptime.layer_size = 256
+deeptime.inr_layers = 5
+deeptime.n_fourier_feats = 4096
+deeptime.scales = [0.01, 0.1, 1, 5, 10, 20, 50, 100]
+
+ForecastDataset.data_path = 'ETT-small/ETTm2.csv'
+ForecastDataset.target = 'OT'
+ForecastDataset.scale = True
+ForecastDataset.cross_learn = False
+ForecastDataset.time_features = []
+ForecastDataset.normalise_time_features = True
+ForecastDataset.features = 'M'
+ForecastDataset.horizon_len = 192
+ForecastDataset.lookback_mult = 5

experiments/configs/ETTm2/192S.gin

@@ -0,0 +1,37 @@
+build.experiment_name = 'ETTm2/192S'
+build.module = 'experiments.forecast'
+build.repeat = 1
+build.variables_dict = {
+}
+
+instance.model_type = 'deeptime'
+instance.save_vals = False
+
+get_optimizer.lr = 1e-3
+get_optimizer.lambda_lr = 1.
+get_optimizer.weight_decay = 0.
+
+get_scheduler.warmup_epochs = 5
+
+get_data.batch_size = 256
+
+train.loss_name = 'mse'
+train.epochs = 50
+train.clip = 10.
+
+Checkpoint.patience = 7
+
+deeptime.layer_size = 256
+deeptime.inr_layers = 5
+deeptime.n_fourier_feats = 4096
+deeptime.scales = [0.01, 0.1, 1, 5, 10, 20, 50, 100]
+
+ForecastDataset.data_path = 'ETT-small/ETTm2.csv'
+ForecastDataset.target = 'OT'
+ForecastDataset.scale = True
+ForecastDataset.cross_learn = False
+ForecastDataset.time_features = []
+ForecastDataset.normalise_time_features = True
+ForecastDataset.features = 'M'
+ForecastDataset.horizon_len = 192
+ForecastDataset.lookback_mult = 3

experiments/configs/ETTm2/336M.gin

@@ -0,0 +1,37 @@
+build.experiment_name = 'ETTm2/336M'
+build.module = 'experiments.forecast'
+build.repeat = 1
+build.variables_dict = {
+}
+
+instance.model_type = 'deeptime'
+instance.save_vals = False
+
+get_optimizer.lr = 1e-3
+get_optimizer.lambda_lr = 1.
+get_optimizer.weight_decay = 0.
+
+get_scheduler.warmup_epochs = 5
+
+get_data.batch_size = 256
+
+train.loss_name = 'mse'
+train.epochs = 50
+train.clip = 10.
+
+Checkpoint.patience = 7
+
+deeptime.layer_size = 256
+deeptime.inr_layers = 5
+deeptime.n_fourier_feats = 4096
+deeptime.scales = [0.01, 0.1, 1, 5, 10, 20, 50, 100]
+
+ForecastDataset.data_path = 'ETT-small/ETTm2.csv'
+ForecastDataset.target = 'OT'
+ForecastDataset.scale = True
+ForecastDataset.cross_learn = False
+ForecastDataset.time_features = []
+ForecastDataset.normalise_time_features = True
+ForecastDataset.features = 'M'
+ForecastDataset.horizon_len = 336
+ForecastDataset.lookback_mult = 3

experiments/configs/ETTm2/336S.gin

@@ -0,0 +1,37 @@
+build.experiment_name = 'ETTm2/336S'
+build.module = 'experiments.forecast'
+build.repeat = 1
+build.variables_dict = {
+}
+
+instance.model_type = 'deeptime'
+instance.save_vals = False
+
+get_optimizer.lr = 1e-3
+get_optimizer.lambda_lr = 1.
+get_optimizer.weight_decay = 0.
+
+get_scheduler.warmup_epochs = 5
+
+get_data.batch_size = 256
+
+train.loss_name = 'mse'
+train.epochs = 50
+train.clip = 10.
+
+Checkpoint.patience = 7
+
+deeptime.layer_size = 256
+deeptime.inr_layers = 5
+deeptime.n_fourier_feats = 4096
+deeptime.scales = [0.01, 0.1, 1, 5, 10, 20, 50, 100]
+
+ForecastDataset.data_path = 'ETT-small/ETTm2.csv'
+ForecastDataset.target = 'OT'
+ForecastDataset.scale = True
+ForecastDataset.cross_learn = False
+ForecastDataset.time_features = []
+ForecastDataset.normalise_time_features = True
+ForecastDataset.features = 'S'
+ForecastDataset.horizon_len = 336
+ForecastDataset.lookback_mult = 7

experiments/configs/ETTm2/720M.gin

@@ -0,0 +1,37 @@
+build.experiment_name = 'ETTm2/720M'
+build.module = 'experiments.forecast'
+build.repeat = 1
+build.variables_dict = {
+}
+
+instance.model_type = 'deeptime'
+instance.save_vals = False
+
+get_optimizer.lr = 1e-3
+get_optimizer.lambda_lr = 1.
+get_optimizer.weight_decay = 0.
+
+get_scheduler.warmup_epochs = 5
+
+get_data.batch_size = 256
+
+train.loss_name = 'mse'
+train.epochs = 50
+train.clip = 10.
+
+Checkpoint.patience = 7
+
+deeptime.layer_size = 256
+deeptime.inr_layers = 5
+deeptime.n_fourier_feats = 4096
+deeptime.scales = [0.01, 0.1, 1, 5, 10, 20, 50, 100]
+
+ForecastDataset.data_path = 'ETT-small/ETTm2.csv'
+ForecastDataset.target = 'OT'
+ForecastDataset.scale = True
+ForecastDataset.cross_learn = False
+ForecastDataset.time_features = []
+ForecastDataset.normalise_time_features = True
+ForecastDataset.features = 'M'
+ForecastDataset.horizon_len = 720
+ForecastDataset.lookback_mult = 1

experiments/configs/ETTm2/720S.gin

@@ -0,0 +1,37 @@
+build.experiment_name = 'ETTm2/720S'
+build.module = 'experiments.forecast'
+build.repeat = 1
+build.variables_dict = {
+}
+
+instance.model_type = 'deeptime'
+instance.save_vals = False
+
+get_optimizer.lr = 1e-3
+get_optimizer.lambda_lr = 1.
+get_optimizer.weight_decay = 0.
+
+get_scheduler.warmup_epochs = 5
+
+get_data.batch_size = 256
+
+train.loss_name = 'mse'
+train.epochs = 50
+train.clip = 10.
+
+Checkpoint.patience = 7
+
+deeptime.layer_size = 256
+deeptime.inr_layers = 5
+deeptime.n_fourier_feats = 4096
+deeptime.scales = [0.01, 0.1, 1, 5, 10, 20, 50, 100]
+
+ForecastDataset.data_path = 'ETT-small/ETTm2.csv'
+ForecastDataset.target = 'OT'
+ForecastDataset.scale = True
+ForecastDataset.cross_learn = False
+ForecastDataset.time_features = []
+ForecastDataset.normalise_time_features = True
+ForecastDataset.features = 'S'
+ForecastDataset.horizon_len = 720
+ForecastDataset.lookback_mult = 1

experiments/configs/ETTm2/96M.gin

@@ -0,0 +1,37 @@
+build.experiment_name = 'ETTm2/96M'
+build.module = 'experiments.forecast'
+build.repeat = 1
+build.variables_dict = {
+}
+
+instance.model_type = 'deeptime'
+instance.save_vals = False
+
+get_optimizer.lr = 1e-3
+get_optimizer.lambda_lr = 1.
+get_optimizer.weight_decay = 0.
+
+get_scheduler.warmup_epochs = 5
+
+get_data.batch_size = 256
+
+train.loss_name = 'mse'
+train.epochs = 50
+train.clip = 10.
+
+Checkpoint.patience = 7
+
+deeptime.layer_size = 256
+deeptime.inr_layers = 5
+deeptime.n_fourier_feats = 4096
+deeptime.scales = [0.01, 0.1, 1, 5, 10, 20, 50, 100]
+
+ForecastDataset.data_path = 'ETT-small/ETTm2.csv'
+ForecastDataset.target = 'OT'
+ForecastDataset.scale = True
+ForecastDataset.cross_learn = False
+ForecastDataset.time_features = []
+ForecastDataset.normalise_time_features = True
+ForecastDataset.features = 'M'
+ForecastDataset.horizon_len = 96
+ForecastDataset.lookback_mult = 7

experiments/configs/ETTm2/96S.gin

@@ -0,0 +1,37 @@
+build.experiment_name = 'ETTm2/96S'
+build.module = 'experiments.forecast'
+build.repeat = 1
+build.variables_dict = {
+}
+
+instance.model_type = 'deeptime'
+instance.save_vals = False
+
+get_optimizer.lr = 1e-3
+get_optimizer.lambda_lr = 1.
+get_optimizer.weight_decay = 0.
+
+get_scheduler.warmup_epochs = 5
+
+get_data.batch_size = 256
+
+train.loss_name = 'mse'
+train.epochs = 50
+train.clip = 10.
+
+Checkpoint.patience = 7
+
+deeptime.layer_size = 256
+deeptime.inr_layers = 5
+deeptime.n_fourier_feats = 4096
+deeptime.scales = [0.01, 0.1, 1, 5, 10, 20, 50, 100]
+
+ForecastDataset.data_path = 'ETT-small/ETTm2.csv'
+ForecastDataset.target = 'OT'
+ForecastDataset.scale = True
+ForecastDataset.cross_learn = False
+ForecastDataset.time_features = []
+ForecastDataset.normalise_time_features = True
+ForecastDataset.features = 'S'
+ForecastDataset.horizon_len = 96
+ForecastDataset.lookback_mult = 5

experiments/configs/Exchange/192M.gin

@@ -0,0 +1,37 @@
+build.experiment_name = 'Exchange/192M'
+build.module = 'experiments.forecast'
+build.repeat = 1
+build.variables_dict = {
+}
+
+instance.model_type = 'deeptime'
+instance.save_vals = False
+
+get_optimizer.lr = 1e-3
+get_optimizer.lambda_lr = 1.
+get_optimizer.weight_decay = 0.
+
+get_scheduler.warmup_epochs = 5
+
+get_data.batch_size = 256
+
+train.loss_name = 'mse'
+train.epochs = 50
+train.clip = 10.
+
+Checkpoint.patience = 7
+
+deeptime.layer_size = 256
+deeptime.inr_layers = 5
+deeptime.n_fourier_feats = 4096
+deeptime.scales = [0.01, 0.1, 1, 5, 10, 20, 50, 100]
+
+ForecastDataset.data_path = 'exchange_rate/exchange_rate.csv'
+ForecastDataset.target = 'OT'
+ForecastDataset.scale = True
+ForecastDataset.cross_learn = False
+ForecastDataset.time_features = []
+ForecastDataset.normalise_time_features = True
+ForecastDataset.features = 'M'
+ForecastDataset.horizon_len = 192
+ForecastDataset.lookback_mult = 5

experiments/configs/Exchange/192S.gin

@@ -0,0 +1,37 @@
+build.experiment_name = 'Exchange/192S'
+build.module = 'experiments.forecast'
+build.repeat = 1
+build.variables_dict = {
+}
+
+instance.model_type = 'deeptime'
+instance.save_vals = False
+
+get_optimizer.lr = 1e-3
+get_optimizer.lambda_lr = 1.
+get_optimizer.weight_decay = 0.
+
+get_scheduler.warmup_epochs = 5
+
+get_data.batch_size = 256
+
+train.loss_name = 'mse'
+train.epochs = 50
+train.clip = 10.
+
+Checkpoint.patience = 7
+
+deeptime.layer_size = 256
+deeptime.inr_layers = 5
+deeptime.n_fourier_feats = 4096
+deeptime.scales = [0.01, 0.1, 1, 5, 10, 20, 50, 100]
+
+ForecastDataset.data_path = 'exchange_rate/exchange_rate.csv'
+ForecastDataset.target = 'OT'
+ForecastDataset.scale = True
+ForecastDataset.cross_learn = False
+ForecastDataset.time_features = []
+ForecastDataset.normalise_time_features = True
+ForecastDataset.features = 'S'
+ForecastDataset.horizon_len = 192
+ForecastDataset.lookback_mult = 1

experiments/configs/Exchange/336M.gin

@@ -0,0 +1,37 @@
+build.experiment_name = 'Exchange/336M'
+build.module = 'experiments.forecast'
+build.repeat = 1
+build.variables_dict = {
+}
+
+instance.model_type = 'deeptime'
+instance.save_vals = False
+
+get_optimizer.lr = 1e-3
+get_optimizer.lambda_lr = 1.
+get_optimizer.weight_decay = 0.
+
+get_scheduler.warmup_epochs = 5
+
+get_data.batch_size = 256
+
+train.loss_name = 'mse'
+train.epochs = 50
+train.clip = 10.
+
+Checkpoint.patience = 7
+
+deeptime.layer_size = 256
+deeptime.inr_layers = 5
+deeptime.n_fourier_feats = 4096
+deeptime.scales = [0.01, 0.1, 1, 5, 10, 20, 50, 100]
+
+ForecastDataset.data_path = 'exchange_rate/exchange_rate.csv'
+ForecastDataset.target = 'OT'
+ForecastDataset.scale = True
+ForecastDataset.cross_learn = False
+ForecastDataset.time_features = []
+ForecastDataset.normalise_time_features = True
+ForecastDataset.features = 'M'
+ForecastDataset.horizon_len = 336
+ForecastDataset.lookback_mult = 7

experiments/configs/Exchange/336S.gin

@@ -0,0 +1,37 @@
+build.experiment_name = 'Exchange/336S'
+build.module = 'experiments.forecast'
+build.repeat = 1
+build.variables_dict = {
+}
+
+instance.model_type = 'deeptime'
+instance.save_vals = False
+
+get_optimizer.lr = 1e-3
+get_optimizer.lambda_lr = 1.
+get_optimizer.weight_decay = 0.
+
+get_scheduler.warmup_epochs = 5
+
+get_data.batch_size = 256
+
+train.loss_name = 'mse'
+train.epochs = 50
+train.clip = 10.
+
+Checkpoint.patience = 7
+
+deeptime.layer_size = 256
+deeptime.inr_layers = 5
+deeptime.n_fourier_feats = 4096
+deeptime.scales = [0.01, 0.1, 1, 5, 10, 20, 50, 100]
+
+ForecastDataset.data_path = 'exchange_rate/exchange_rate.csv'
+ForecastDataset.target = 'OT'
+ForecastDataset.scale = True
+ForecastDataset.cross_learn = False
+ForecastDataset.time_features = []
+ForecastDataset.normalise_time_features = True
+ForecastDataset.features = 'S'
+ForecastDataset.horizon_len = 336
+ForecastDataset.lookback_mult = 7

experiments/configs/Exchange/720M.gin

@@ -0,0 +1,37 @@
+build.experiment_name = 'Exchange/720M'
+build.module = 'experiments.forecast'
+build.repeat = 1
+build.variables_dict = {
+}
+
+instance.model_type = 'deeptime'
+instance.save_vals = False
+
+get_optimizer.lr = 1e-3
+get_optimizer.lambda_lr = 1.
+get_optimizer.weight_decay = 0.
+
+get_scheduler.warmup_epochs = 5
+
+get_data.batch_size = 256
+
+train.loss_name = 'mse'
+train.epochs = 50
+train.clip = 10.
+
+Checkpoint.patience = 7
+
+deeptime.layer_size = 256
+deeptime.inr_layers = 5
+deeptime.n_fourier_feats = 4096
+deeptime.scales = [0.01, 0.1, 1, 5, 10, 20, 50, 100]
+
+ForecastDataset.data_path = 'exchange_rate/exchange_rate.csv'
+ForecastDataset.target = 'OT'
+ForecastDataset.scale = True
+ForecastDataset.cross_learn = False
+ForecastDataset.time_features = []
+ForecastDataset.normalise_time_features = True
+ForecastDataset.features = 'M'
+ForecastDataset.horizon_len = 720
+ForecastDataset.lookback_mult = 3

experiments/configs/Exchange/720S.gin

@@ -0,0 +1,37 @@
+build.experiment_name = 'Exchange/720S'
+build.module = 'experiments.forecast'
+build.repeat = 1
+build.variables_dict = {
+}
+
+instance.model_type = 'deeptime'
+instance.save_vals = False
+
+get_optimizer.lr = 1e-3
+get_optimizer.lambda_lr = 1.
+get_optimizer.weight_decay = 0.
+
+get_scheduler.warmup_epochs = 5
+
+get_data.batch_size = 256
+
+train.loss_name = 'mse'
+train.epochs = 50
+train.clip = 10.
+
+Checkpoint.patience = 7
+
+deeptime.layer_size = 256
+deeptime.inr_layers = 5
+deeptime.n_fourier_feats = 4096
+deeptime.scales = [0.01, 0.1, 1, 5, 10, 20, 50, 100]
+
+ForecastDataset.data_path = 'exchange_rate/exchange_rate.csv'
+ForecastDataset.target = 'OT'
+ForecastDataset.scale = True
+ForecastDataset.cross_learn = False
+ForecastDataset.time_features = []
+ForecastDataset.normalise_time_features = True
+ForecastDataset.features = 'S'
+ForecastDataset.horizon_len = 720
+ForecastDataset.lookback_mult = 3

experiments/configs/Exchange/96M.gin

@@ -0,0 +1,37 @@
+build.experiment_name = 'Exchange/96M'
+build.module = 'experiments.forecast'
+build.repeat = 1
+build.variables_dict = {
+}
+
+instance.model_type = 'deeptime'
+instance.save_vals = False
+
+get_optimizer.lr = 1e-3
+get_optimizer.lambda_lr = 1.
+get_optimizer.weight_decay = 0.
+
+get_scheduler.warmup_epochs = 5
+
+get_data.batch_size = 256
+
+train.loss_name = 'mse'
+train.epochs = 50
+train.clip = 10.
+
+Checkpoint.patience = 7
+
+deeptime.layer_size = 256
+deeptime.inr_layers = 5
+deeptime.n_fourier_feats = 4096
+deeptime.scales = [0.01, 0.1, 1, 5, 10, 20, 50, 100]
+
+ForecastDataset.data_path = 'exchange_rate/exchange_rate.csv'
+ForecastDataset.target = 'OT'
+ForecastDataset.scale = True
+ForecastDataset.cross_learn = False
+ForecastDataset.time_features = []
+ForecastDataset.normalise_time_features = True
+ForecastDataset.features = 'M'
+ForecastDataset.horizon_len = 96
+ForecastDataset.lookback_mult = 1

experiments/configs/Exchange/96S.gin

@@ -0,0 +1,37 @@
+build.experiment_name = 'Exchange/96S'
+build.module = 'experiments.forecast'
+build.repeat = 1
+build.variables_dict = {
+}
+
+instance.model_type = 'deeptime'
+instance.save_vals = False
+
+get_optimizer.lr = 1e-3
+get_optimizer.lambda_lr = 1.
+get_optimizer.weight_decay = 0.
+
+get_scheduler.warmup_epochs = 5
+
+get_data.batch_size = 256
+
+train.loss_name = 'mse'
+train.epochs = 50
+train.clip = 10.
+
+Checkpoint.patience = 7
+
+deeptime.layer_size = 256
+deeptime.inr_layers = 5
+deeptime.n_fourier_feats = 4096
+deeptime.scales = [0.01, 0.1, 1, 5, 10, 20, 50, 100]
+
+ForecastDataset.data_path = 'exchange_rate/exchange_rate.csv'
+ForecastDataset.target = 'OT'
+ForecastDataset.scale = True
+ForecastDataset.cross_learn = False
+ForecastDataset.time_features = []
+ForecastDataset.normalise_time_features = True
+ForecastDataset.features = 'S'
+ForecastDataset.horizon_len = 96
+ForecastDataset.lookback_mult = 3

experiments/configs/ILI/192M.gin

@@ -0,0 +1,37 @@
+build.experiment_name = 'ILI/192M'
+build.module = 'experiments.forecast'
+build.repeat = 1
+build.variables_dict = {
+}
+
+instance.model_type = 'deeptime'
+instance.save_vals = False
+
+get_optimizer.lr = 1e-3
+get_optimizer.lambda_lr = 1.
+get_optimizer.weight_decay = 0.
+
+get_scheduler.warmup_epochs = 5
+
+get_data.batch_size = 256
+
+train.loss_name = 'mse'
+train.epochs = 50
+train.clip = 10.
+
+Checkpoint.patience = 7
+
+deeptime.layer_size = 256
+deeptime.inr_layers = 5
+deeptime.n_fourier_feats = 4096
+deeptime.scales = [0.01, 0.1, 1, 5, 10, 20, 50, 100]
+
+ForecastDataset.data_path = 'illness/national_illness.csv'
+ForecastDataset.target = 'OT'
+ForecastDataset.scale = True
+ForecastDataset.cross_learn = False
+ForecastDataset.time_features = []
+ForecastDataset.normalise_time_features = True
+ForecastDataset.features = 'M'
+ForecastDataset.horizon_len = 192
+ForecastDataset.lookback_mult = 7

experiments/configs/ILI/336M.gin

@@ -0,0 +1,37 @@
+build.experiment_name = 'ILI/336M'
+build.module = 'experiments.forecast'
+build.repeat = 1
+build.variables_dict = {
+}
+
+instance.model_type = 'deeptime'
+instance.save_vals = False
+
+get_optimizer.lr = 1e-3
+get_optimizer.lambda_lr = 1.
+get_optimizer.weight_decay = 0.
+
+get_scheduler.warmup_epochs = 5
+
+get_data.batch_size = 256
+
+train.loss_name = 'mse'
+train.epochs = 50
+train.clip = 10.
+
+Checkpoint.patience = 7
+
+deeptime.layer_size = 256
+deeptime.inr_layers = 5
+deeptime.n_fourier_feats = 4096
+deeptime.scales = [0.01, 0.1, 1, 5, 10, 20, 50, 100]
+
+ForecastDataset.data_path = 'illness/national_illness.csv'
+ForecastDataset.target = 'OT'
+ForecastDataset.scale = True
+ForecastDataset.cross_learn = False
+ForecastDataset.time_features = []
+ForecastDataset.normalise_time_features = True
+ForecastDataset.features = 'M'
+ForecastDataset.horizon_len = 336
+ForecastDataset.lookback_mult = 5

experiments/configs/ILI/720M.gin

@@ -0,0 +1,37 @@
+build.experiment_name = 'ILI/720M'
+build.module = 'experiments.forecast'
+build.repeat = 1
+build.variables_dict = {
+}
+
+instance.model_type = 'deeptime'
+instance.save_vals = False
+
+get_optimizer.lr = 1e-3
+get_optimizer.lambda_lr = 1.
+get_optimizer.weight_decay = 0.
+
+get_scheduler.warmup_epochs = 5
+
+get_data.batch_size = 256
+
+train.loss_name = 'mse'
+train.epochs = 50
+train.clip = 10.
+
+Checkpoint.patience = 7
+
+deeptime.layer_size = 256
+deeptime.inr_layers = 5
+deeptime.n_fourier_feats = 4096
+deeptime.scales = [0.01, 0.1, 1, 5, 10, 20, 50, 100]
+
+ForecastDataset.data_path = 'illness/national_illness.csv'
+ForecastDataset.target = 'OT'
+ForecastDataset.scale = True
+ForecastDataset.cross_learn = False
+ForecastDataset.time_features = []
+ForecastDataset.normalise_time_features = True
+ForecastDataset.features = 'M'
+ForecastDataset.horizon_len = 720
+ForecastDataset.lookback_mult = 3

experiments/configs/ILI/96M.gin

@@ -0,0 +1,37 @@
+build.experiment_name = 'ILI/96M'
+build.module = 'experiments.forecast'
+build.repeat = 1
+build.variables_dict = {
+}
+
+instance.model_type = 'deeptime'
+instance.save_vals = False
+
+get_optimizer.lr = 1e-3
+get_optimizer.lambda_lr = 1.
+get_optimizer.weight_decay = 0.
+
+get_scheduler.warmup_epochs = 5
+
+get_data.batch_size = 256
+
+train.loss_name = 'mse'
+train.epochs = 50
+train.clip = 10.
+
+Checkpoint.patience = 7
+
+deeptime.layer_size = 256
+deeptime.inr_layers = 5
+deeptime.n_fourier_feats = 4096
+deeptime.scales = [0.01, 0.1, 1, 5, 10, 20, 50, 100]
+
+ForecastDataset.data_path = 'illness/national_illness.csv'
+ForecastDataset.target = 'OT'
+ForecastDataset.scale = True
+ForecastDataset.cross_learn = False
+ForecastDataset.time_features = []
+ForecastDataset.normalise_time_features = True
+ForecastDataset.features = 'M'
+ForecastDataset.horizon_len = 96
+ForecastDataset.lookback_mult = 9

experiments/configs/Traffic/192M.gin

@@ -0,0 +1,37 @@
+build.experiment_name = 'Traffic/192M'
+build.module = 'experiments.forecast'
+build.repeat = 1
+build.variables_dict = {
+}
+
+instance.model_type = 'deeptime'
+instance.save_vals = False
+
+get_optimizer.lr = 1e-3
+get_optimizer.lambda_lr = 1.
+get_optimizer.weight_decay = 0.
+
+get_scheduler.warmup_epochs = 5
+
+get_data.batch_size = 256
+
+train.loss_name = 'mse'
+train.epochs = 50
+train.clip = 10.
+
+Checkpoint.patience = 7
+
+deeptime.layer_size = 256
+deeptime.inr_layers = 5
+deeptime.n_fourier_feats = 4096
+deeptime.scales = [0.01, 0.1, 1, 5, 10, 20, 50, 100]
+
+ForecastDataset.data_path = 'traffic/traffic.csv'
+ForecastDataset.target = 'OT'
+ForecastDataset.scale = True
+ForecastDataset.cross_learn = False
+ForecastDataset.time_features = []
+ForecastDataset.normalise_time_features = True
+ForecastDataset.features = 'M'
+ForecastDataset.horizon_len = 192
+ForecastDataset.lookback_mult = 5

experiments/configs/Traffic/336M.gin

@@ -0,0 +1,37 @@
+build.experiment_name = 'Traffic/336M'
+build.module = 'experiments.forecast'
+build.repeat = 1
+build.variables_dict = {
+}
+
+instance.model_type = 'deeptime'
+instance.save_vals = False
+
+get_optimizer.lr = 1e-3
+get_optimizer.lambda_lr = 1.
+get_optimizer.weight_decay = 0.
+
+get_scheduler.warmup_epochs = 5
+
+get_data.batch_size = 256
+
+train.loss_name = 'mse'
+train.epochs = 50
+train.clip = 10.
+
+Checkpoint.patience = 7
+
+deeptime.layer_size = 256
+deeptime.inr_layers = 5
+deeptime.n_fourier_feats = 4096
+deeptime.scales = [0.01, 0.1, 1, 5, 10, 20, 50, 100]
+
+ForecastDataset.data_path = 'traffic/traffic.csv'
+ForecastDataset.target = 'OT'
+ForecastDataset.scale = True
+ForecastDataset.cross_learn = False
+ForecastDataset.time_features = []
+ForecastDataset.normalise_time_features = True
+ForecastDataset.features = 'M'
+ForecastDataset.horizon_len = 336
+ForecastDataset.lookback_mult = 5

experiments/configs/Traffic/720M.gin

@@ -0,0 +1,37 @@
+build.experiment_name = 'Traffic/720M'
+build.module = 'experiments.forecast'
+build.repeat = 1
+build.variables_dict = {
+}
+
+instance.model_type = 'deeptime'
+instance.save_vals = False
+
+get_optimizer.lr = 1e-3
+get_optimizer.lambda_lr = 1.
+get_optimizer.weight_decay = 0.
+
+get_scheduler.warmup_epochs = 5
+
+get_data.batch_size = 256
+
+train.loss_name = 'mse'
+train.epochs = 50
+train.clip = 10.
+
+Checkpoint.patience = 7
+
+deeptime.layer_size = 256
+deeptime.inr_layers = 5
+deeptime.n_fourier_feats = 4096
+deeptime.scales = [0.01, 0.1, 1, 5, 10, 20, 50, 100]
+
+ForecastDataset.data_path = 'traffic/traffic.csv'
+ForecastDataset.target = 'OT'
+ForecastDataset.scale = True
+ForecastDataset.cross_learn = False
+ForecastDataset.time_features = []
+ForecastDataset.normalise_time_features = True
+ForecastDataset.features = 'M'
+ForecastDataset.horizon_len = 720
+ForecastDataset.lookback_mult = 3

experiments/configs/Traffic/96M.gin

@@ -0,0 +1,37 @@
+build.experiment_name = 'Traffic/96M'
+build.module = 'experiments.forecast'
+build.repeat = 1
+build.variables_dict = {
+}
+
+instance.model_type = 'deeptime'
+instance.save_vals = False
+
+get_optimizer.lr = 1e-3
+get_optimizer.lambda_lr = 1.
+get_optimizer.weight_decay = 0.
+
+get_scheduler.warmup_epochs = 5
+
+get_data.batch_size = 256
+
+train.loss_name = 'mse'
+train.epochs = 50
+train.clip = 10.
+
+Checkpoint.patience = 7
+
+deeptime.layer_size = 256
+deeptime.inr_layers = 5
+deeptime.n_fourier_feats = 4096
+deeptime.scales = [0.01, 0.1, 1, 5, 10, 20, 50, 100]
+
+ForecastDataset.data_path = 'traffic/traffic.csv'
+ForecastDataset.target = 'OT'
+ForecastDataset.scale = True
+ForecastDataset.cross_learn = False
+ForecastDataset.time_features = []
+ForecastDataset.normalise_time_features = True
+ForecastDataset.features = 'M'
+ForecastDataset.horizon_len = 96
+ForecastDataset.lookback_mult = 9

experiments/configs/Weather/192M.gin

@@ -0,0 +1,37 @@
+build.experiment_name = 'Weather/192M'
+build.module = 'experiments.forecast'
+build.repeat = 1
+build.variables_dict = {
+}
+
+instance.model_type = 'deeptime'
+instance.save_vals = False
+
+get_optimizer.lr = 1e-3
+get_optimizer.lambda_lr = 1.
+get_optimizer.weight_decay = 0.
+
+get_scheduler.warmup_epochs = 5
+
+get_data.batch_size = 256
+
+train.loss_name = 'mse'
+train.epochs = 50
+train.clip = 10.
+
+Checkpoint.patience = 7
+
+deeptime.layer_size = 256
+deeptime.inr_layers = 5
+deeptime.n_fourier_feats = 4096
+deeptime.scales = [0.01, 0.1, 1, 5, 10, 20, 50, 100]
+
+ForecastDataset.data_path = 'weather/weather.csv'
+ForecastDataset.target = 'OT'
+ForecastDataset.scale = True
+ForecastDataset.cross_learn = False
+ForecastDataset.time_features = []
+ForecastDataset.normalise_time_features = True
+ForecastDataset.features = 'M'
+ForecastDataset.horizon_len = 192
+ForecastDataset.lookback_mult = 7

experiments/configs/Weather/336M.gin

@@ -0,0 +1,37 @@
+build.experiment_name = 'Weather/336M'
+build.module = 'experiments.forecast'
+build.repeat = 1
+build.variables_dict = {
+}
+
+instance.model_type = 'deeptime'
+instance.save_vals = False
+
+get_optimizer.lr = 1e-3
+get_optimizer.lambda_lr = 1.
+get_optimizer.weight_decay = 0.
+
+get_scheduler.warmup_epochs = 5
+
+get_data.batch_size = 256
+
+train.loss_name = 'mse'
+train.epochs = 50
+train.clip = 10.
+
+Checkpoint.patience = 7
+
+deeptime.layer_size = 256
+deeptime.inr_layers = 5
+deeptime.n_fourier_feats = 4096
+deeptime.scales = [0.01, 0.1, 1, 5, 10, 20, 50, 100]
+
+ForecastDataset.data_path = 'weather/weather.csv'
+ForecastDataset.target = 'OT'
+ForecastDataset.scale = True
+ForecastDataset.cross_learn = False
+ForecastDataset.time_features = []
+ForecastDataset.normalise_time_features = True
+ForecastDataset.features = 'M'
+ForecastDataset.horizon_len = 192
+ForecastDataset.lookback_mult = 3

experiments/configs/Weather/720M.gin

@@ -0,0 +1,37 @@
+build.experiment_name = 'Weather/720M'
+build.module = 'experiments.forecast'
+build.repeat = 1
+build.variables_dict = {
+}
+
+instance.model_type = 'deeptime'
+instance.save_vals = False
+
+get_optimizer.lr = 1e-3
+get_optimizer.lambda_lr = 1.
+get_optimizer.weight_decay = 0.
+
+get_scheduler.warmup_epochs = 5
+
+get_data.batch_size = 256
+
+train.loss_name = 'mse'
+train.epochs = 50
+train.clip = 10.
+
+Checkpoint.patience = 7
+
+deeptime.layer_size = 256
+deeptime.inr_layers = 5
+deeptime.n_fourier_feats = 4096
+deeptime.scales = [0.01, 0.1, 1, 5, 10, 20, 50, 100]
+
+ForecastDataset.data_path = 'weather/weather.csv'
+ForecastDataset.target = 'OT'
+ForecastDataset.scale = True
+ForecastDataset.cross_learn = False
+ForecastDataset.time_features = []
+ForecastDataset.normalise_time_features = True
+ForecastDataset.features = 'M'
+ForecastDataset.horizon_len = 720
+ForecastDataset.lookback_mult = 5

experiments/configs/Weather/96M.gin

@@ -0,0 +1,37 @@
+build.experiment_name = 'Weather/96M'
+build.module = 'experiments.forecast'
+build.repeat = 1
+build.variables_dict = {
+}
+
+instance.model_type = 'deeptime'
+instance.save_vals = False
+
+get_optimizer.lr = 1e-3
+get_optimizer.lambda_lr = 1.
+get_optimizer.weight_decay = 0.
+
+get_scheduler.warmup_epochs = 5
+
+get_data.batch_size = 256
+
+train.loss_name = 'mse'
+train.epochs = 50
+train.clip = 10.
+
+Checkpoint.patience = 7
+
+deeptime.layer_size = 256
+deeptime.inr_layers = 5
+deeptime.n_fourier_feats = 4096
+deeptime.scales = [0.01, 0.1, 1, 5, 10, 20, 50, 100]
+
+ForecastDataset.data_path = 'weather/weather.csv'
+ForecastDataset.target = 'OT'
+ForecastDataset.scale = True
+ForecastDataset.cross_learn = False
+ForecastDataset.time_features = []
+ForecastDataset.normalise_time_features = True
+ForecastDataset.features = 'M'
+ForecastDataset.horizon_len = 96
+ForecastDataset.lookback_mult = 9

experiments/configs/hp_search/ECL.gin

@@ -0,0 +1,37 @@
+build.experiment_name = 'hp_search/ECL'
+build.module = 'experiments.forecast'
+build.repeat = 3
+build.variables_dict = {
+    'ForecastDataset.lookback_mult': [1, 3, 5, 7, 9],
+    'ForecastDataset.horizon_len': [96, 192, 336, 720],
+    'ForecastDataset.features': ['M'],
+}
+
+instance.model_type = 'deeptime'
+instance.save_vals = False
+
+get_optimizer.lr = 1e-3
+get_optimizer.lambda_lr = 1.
+get_optimizer.weight_decay = 0.
+
+get_scheduler.warmup_epochs = 5
+
+get_data.batch_size = 256
+
+train.loss_name = 'mse'
+train.epochs = 50
+train.clip = 10.
+
+Checkpoint.patience = 7
+
+deeptime.layer_size = 256
+deeptime.inr_layers = 5
+deeptime.n_fourier_feats = 4096
+deeptime.scales = [0.01, 0.1, 1, 5, 10, 20, 50, 100]
+
+ForecastDataset.data_path = 'electricity/electricity.csv'
+ForecastDataset.target = 'OT'
+ForecastDataset.scale = True
+ForecastDataset.cross_learn = False
+ForecastDataset.time_features = []
+ForecastDataset.normalise_time_features = True

experiments/configs/hp_search/ETTm2.gin

@@ -0,0 +1,37 @@
+build.experiment_name = 'hp_search/ETTm2'
+build.module = 'experiments.forecast'
+build.repeat = 3
+build.variables_dict = {
+    'ForecastDataset.lookback_mult': [1, 3, 5, 7, 9],
+    'ForecastDataset.horizon_len': [96, 192, 336, 720],
+    'ForecastDataset.features': ['M', 'S'],
+}
+
+instance.model_type = 'deeptime'
+instance.save_vals = False
+
+get_optimizer.lr = 1e-3
+get_optimizer.lambda_lr = 1.
+get_optimizer.weight_decay = 0.
+
+get_scheduler.warmup_epochs = 5
+
+get_data.batch_size = 256
+
+train.loss_name = 'mse'
+train.epochs = 50
+train.clip = 10.
+
+Checkpoint.patience = 7
+
+deeptime.layer_size = 256
+deeptime.inr_layers = 5
+deeptime.n_fourier_feats = 4096
+deeptime.scales = [0.01, 0.1, 1, 5, 10, 20, 50, 100]
+
+ForecastDataset.data_path = 'ETT-small/ETTm2.csv'
+ForecastDataset.target = 'OT'
+ForecastDataset.scale = True
+ForecastDataset.cross_learn = False
+ForecastDataset.time_features = []
+ForecastDataset.normalise_time_features = True

experiments/configs/hp_search/Exchange.gin

@@ -0,0 +1,37 @@
+build.experiment_name = 'hp_search/Exchange'
+build.module = 'experiments.forecast'
+build.repeat = 3
+build.variables_dict = {
+    'ForecastDataset.lookback_mult': [1, 3, 5, 7, 9],
+    'ForecastDataset.horizon_len': [96, 192, 336, 720],
+    'ForecastDataset.features': ['M', 'S'],
+}
+
+instance.model_type = 'deeptime'
+instance.save_vals = False
+
+get_optimizer.lr = 1e-3
+get_optimizer.lambda_lr = 1.
+get_optimizer.weight_decay = 0.
+
+get_scheduler.warmup_epochs = 5
+
+get_data.batch_size = 256
+
+train.loss_name = 'mse'
+train.epochs = 50
+train.clip = 10.
+
+Checkpoint.patience = 7
+
+deeptime.layer_size = 256
+deeptime.inr_layers = 5
+deeptime.n_fourier_feats = 4096
+deeptime.scales = [0.01, 0.1, 1, 5, 10, 20, 50, 100]
+
+ForecastDataset.data_path = 'exchange_rate/exchange_rate.csv'
+ForecastDataset.target = 'OT'
+ForecastDataset.scale = True
+ForecastDataset.cross_learn = False
+ForecastDataset.time_features = []
+ForecastDataset.normalise_time_features = True

experiments/configs/hp_search/ILI.gin

@@ -0,0 +1,37 @@
+build.experiment_name = 'hp_search/ILI'
+build.module = 'experiments.forecast'
+build.repeat = 3
+build.variables_dict = {
+    'ForecastDataset.lookback_mult': [1, 3, 5, 7, 9],
+    'ForecastDataset.horizon_len': [24, 36, 48, 60],
+    'ForecastDataset.features': ['M'],
+}
+
+instance.model_type = 'deeptime'
+instance.save_vals = False
+
+get_optimizer.lr = 1e-3
+get_optimizer.lambda_lr = 1.
+get_optimizer.weight_decay = 0.
+
+get_scheduler.warmup_epochs = 5
+
+get_data.batch_size = 256
+
+train.loss_name = 'mse'
+train.epochs = 50
+train.clip = 10.
+
+Checkpoint.patience = 7
+
+deeptime.layer_size = 256
+deeptime.inr_layers = 5
+deeptime.n_fourier_feats = 4096
+deeptime.scales = [0.01, 0.1, 1, 5, 10, 20, 50, 100]
+
+ForecastDataset.data_path = 'illness/national_illness.csv'
+ForecastDataset.target = 'OT'
+ForecastDataset.scale = True
+ForecastDataset.cross_learn = False
+ForecastDataset.time_features = []
+ForecastDataset.normalise_time_features = True

experiments/configs/hp_search/Traffic.gin

@@ -0,0 +1,37 @@
+build.experiment_name = 'hp_search/Traffic'
+build.module = 'experiments.forecast'
+build.repeat = 3
+build.variables_dict = {
+    'ForecastDataset.lookback_mult': [1, 3, 5, 7, 9],
+    'ForecastDataset.horizon_len': [96, 192, 336, 720],
+    'ForecastDataset.features': ['M'],
+}
+
+instance.model_type = 'deeptime'
+instance.save_vals = False
+
+get_optimizer.lr = 1e-3
+get_optimizer.lambda_lr = 1.
+get_optimizer.weight_decay = 0.
+
+get_scheduler.warmup_epochs = 5
+
+get_data.batch_size = 256
+
+train.loss_name = 'mse'
+train.epochs = 50
+train.clip = 10.
+
+Checkpoint.patience = 7
+
+deeptime.layer_size = 256
+deeptime.inr_layers = 5
+deeptime.n_fourier_feats = 4096
+deeptime.scales = [0.01, 0.1, 1, 5, 10, 20, 50, 100]
+
+ForecastDataset.data_path = 'traffic/traffic.csv'
+ForecastDataset.target = 'OT'
+ForecastDataset.scale = True
+ForecastDataset.cross_learn = False
+ForecastDataset.time_features = []
+ForecastDataset.normalise_time_features = True

experiments/configs/hp_search/Weather.gin

@@ -0,0 +1,37 @@
+build.experiment_name = 'hp_search/Weather'
+build.module = 'experiments.forecast'
+build.repeat = 3
+build.variables_dict = {
+    'ForecastDataset.lookback_mult': [1, 3, 5, 7, 9],
+    'ForecastDataset.horizon_len': [96, 192, 336, 720],
+    'ForecastDataset.features': ['M'],
+}
+
+instance.model_type = 'deeptime'
+instance.save_vals = False
+
+get_optimizer.lr = 1e-3
+get_optimizer.lambda_lr = 1.
+get_optimizer.weight_decay = 0.
+
+get_scheduler.warmup_epochs = 5
+
+get_data.batch_size = 256
+
+train.loss_name = 'mse'
+train.epochs = 50
+train.clip = 10.
+
+Checkpoint.patience = 7
+
+deeptime.layer_size = 256
+deeptime.inr_layers = 5
+deeptime.n_fourier_feats = 4096
+deeptime.scales = [0.01, 0.1, 1, 5, 10, 20, 50, 100]
+
+ForecastDataset.data_path = 'weather/weather.csv'
+ForecastDataset.target = 'OT'
+ForecastDataset.scale = True
+ForecastDataset.cross_learn = False
+ForecastDataset.time_features = []
+ForecastDataset.normalise_time_features = True

experiments/forecast.py

@@ -0,0 +1,226 @@
+import os
+from os.path import join
+import math
+import logging
+from typing import Callable, Optional, Union, Dict, Tuple
+
+import gin
+from fire import Fire
+import numpy as np
+import torch
+from torch.utils.data import DataLoader
+from torch import optim
+from torch import nn
+
+from experiments.base import Experiment
+from data.datasets import ForecastDataset
+from models import get_model
+from utils.checkpoint import Checkpoint
+from utils.ops import default_device, to_tensor
+from utils.losses import get_loss_fn
+from utils.metrics import calc_metrics
+
+
+class ForecastExperiment(Experiment):
+    @gin.configurable()
+    def instance(self,
+                 model_type: str,
+                 save_vals: Optional[bool] = True,):
+        # load datasets, model, checkpointer
+        train_set, train_loader = get_data(flag='train')
+        val_set, val_loader = get_data(flag='val')
+        test_set, test_loader = get_data(flag='test')
+
+        model = get_model(model_type,
+                          dim_size=train_set.data_x.shape[1],
+                          datetime_feats=train_set.timestamps.shape[-1]).to(default_device())
+        checkpoint = Checkpoint(self.root)
+
+        # train forecasting task
+        model = train(model, checkpoint, train_loader, val_loader, test_loader)
+
+        # testing
+        val_metrics = validate(model, loader=val_loader, report_metrics=True)
+        test_metrics = validate(model, loader=test_loader, report_metrics=True,
+                                save_path=self.root if save_vals else None)
+        np.save(join(self.root, 'metrics.npy'), {'val': val_metrics, 'test': test_metrics})
+
+        val_metrics = {f'ValMetric/{k}': v for k, v in val_metrics.items()}
+        test_metrics = {f'TestMetric/{k}': v for k, v in test_metrics.items()}
+        checkpoint.close({**val_metrics, **test_metrics})
+
+
+@gin.configurable()
+def get_optimizer(model: nn.Module,
+                  lr: Optional[float] = 1e-3,
+                  lambda_lr: Optional[float] = 1.,
+                  weight_decay: Optional[float] = 1e-2) -> optim.Optimizer:
+    group1 = []  # lambda
+    group2 = []  # no decay
+    group3 = []  # decay
+    no_decay_list = ('bias', 'norm',)
+    for param_name, param in model.named_parameters():
+        if '_lambda' in param_name:
+            group1.append(param)
+        elif any([mod in param_name for mod in no_decay_list]):
+            group2.append(param)
+        else:
+            group3.append(param)
+    optimizer = optim.Adam([
+        {'params': group1, 'weight_decay': 0, 'lr': lambda_lr, 'scheduler': 'cosine_annealing'},
+        {'params': group2, 'weight_decay': 0, 'scheduler': 'cosine_annealing_with_linear_warmup'},
+        {'params': group3, 'scheduler': 'cosine_annealing_with_linear_warmup'}
+    ], lr=lr, weight_decay=weight_decay)
+    return optimizer
+
+
+@gin.configurable()
+def get_scheduler(optimizer: optim.Optimizer,
+                  T_max: int,
+                  warmup_epochs: int,
+                  eta_min: Optional[float] = 0.) -> optim.lr_scheduler.LambdaLR:
+    scheduler_fns = []
+    for param_group in optimizer.param_groups:
+        scheduler = param_group['scheduler']
+        if scheduler == 'none':
+            fn = lambda T_cur: 1
+        elif scheduler == 'cosine_annealing':
+            lr = eta_max = param_group['lr']
+            fn = lambda T_cur: (eta_min + 0.5 * (eta_max - eta_min) * (
+                        1.0 + math.cos((T_cur - warmup_epochs) / (T_max - warmup_epochs) * math.pi))) / lr
+        elif scheduler == 'cosine_annealing_with_linear_warmup':
+            lr = eta_max = param_group['lr']
+            # https://blog.csdn.net/qq_36560894/article/details/114004799
+            fn = lambda T_cur: T_cur / warmup_epochs if T_cur < warmup_epochs else (eta_min + 0.5 * (
+                        eta_max - eta_min) * (1.0 + math.cos(
+                (T_cur - warmup_epochs) / (T_max - warmup_epochs) * math.pi))) / lr
+        else:
+            raise ValueError(f'No such scheduler, {scheduler}')
+        scheduler_fns.append(fn)
+    scheduler = optim.lr_scheduler.LambdaLR(optimizer, lr_lambda=scheduler_fns)
+    return scheduler
+
+
+@gin.configurable()
+def get_data(flag: bool,
+             batch_size: int) -> Tuple[ForecastDataset, DataLoader]:
+    if flag in ('val', 'test'):
+        shuffle = False
+        drop_last = False
+    elif flag == 'train':
+        shuffle = True
+        drop_last = True
+    else:
+        raise ValueError(f'no such flag {flag}')
+    dataset = ForecastDataset(flag)
+    data_loader = DataLoader(dataset,
+                             batch_size=batch_size,
+                             shuffle=shuffle,
+                             drop_last=drop_last)
+    return dataset, data_loader
+
+
+@gin.configurable()
+def train(model: nn.Module,
+          checkpoint: Checkpoint,
+          train_loader: DataLoader,
+          val_loader: DataLoader,
+          test_loader: DataLoader,
+          loss_name: str,
+          epochs: int,
+          clip: float) -> nn.Module:
+
+    optimizer = get_optimizer(model)
+    scheduler = get_scheduler(optimizer=optimizer, T_max=epochs)
+    training_loss_fn = get_loss_fn(loss_name)
+
+    for epoch in range(epochs):
+        train_loss = []
+        model.train()
+        for it, data in enumerate(train_loader):
+            optimizer.zero_grad()
+
+            x, y, x_time, y_time = map(to_tensor, data)
+            forecast = model(x, x_time, y_time)
+
+            if isinstance(forecast, tuple):
+                # for models which require reconstruction + forecast loss
+                loss = training_loss_fn(forecast[0], x) + \
+                       training_loss_fn(forecast[1], y)
+            else:
+                loss = training_loss_fn(forecast, y)
+            loss.backward()
+            nn.utils.clip_grad_norm_(model.parameters(), clip)
+            optimizer.step()
+
+            train_loss.append(loss.item())
+            if (it + 1) % 100 == 0:
+                logging.info(f"epochs: {epoch + 1}, iters: {it + 1} | training loss: {loss.item():.2f}")
+        scheduler.step()
+
+        train_loss = np.average(train_loss)
+        val_loss = validate(model, loader=val_loader, loss_fn=training_loss_fn)
+        test_loss = validate(model, loader=test_loader, loss_fn=training_loss_fn)
+
+        scalars = {'Loss/Train': train_loss,
+                   'Loss/Val': val_loss,
+                   'Loss/Test': test_loss}
+        checkpoint(epoch + 1, model, scalars=scalars)
+
+        if checkpoint.early_stop:
+            logging.info("Early stopping")
+            break
+
+    if epochs > 0:
+        model.load_state_dict(torch.load(checkpoint.model_path))
+    return model
+
+
+@torch.no_grad()
+def validate(model: nn.Module,
+             loader: DataLoader,
+             loss_fn: Optional[Callable] = None,
+             report_metrics: Optional[bool] = False,
+             save_path: Optional[str] = None) -> Union[Dict[str, float], float]:
+    model.eval()
+    preds = []
+    trues = []
+    inps = []
+    total_loss = []
+    for it, data in enumerate(loader):
+        x, y, x_time, y_time = map(to_tensor, data)
+
+        if x.shape[0] == 1:
+            # skip final batch if batch_size == 1
+            # due to bug in torch.linalg.solve which raises error when batch_size == 1
+            continue
+
+        forecast = model(x, x_time, y_time)
+
+        if report_metrics:
+            preds.append(forecast)
+            trues.append(y)
+            if save_path is not None:
+                inps.append(x)
+        else:
+            loss = loss_fn(forecast, y, reduction='none')
+            total_loss.append(loss)
+
+    if report_metrics:
+        preds = torch.cat(preds, dim=0).detach().cpu().numpy()
+        trues = torch.cat(trues, dim=0).detach().cpu().numpy()
+        if save_path is not None:
+            inps = torch.cat(inps, dim=0).detach().cpu().numpy()
+            np.save(join(save_path, 'inps.npy'), inps)
+            np.save(join(save_path, 'preds.npy'), preds)
+            np.save(join(save_path, 'trues.npy'), trues)
+        metrics = calc_metrics(preds, trues)
+        return metrics
+
+    total_loss = torch.cat(total_loss, dim=0).cpu()
+    return np.average(total_loss)
+
+
+if __name__ == '__main__':
+    logging.root.setLevel(logging.INFO)
+    Fire(ForecastExperiment)

models/DeepTIMe.py

@@ -0,0 +1,55 @@
+from typing import Optional
+
+import gin
+import torch
+import torch.nn as nn
+from torch import Tensor
+from einops import rearrange, repeat, reduce
+
+from models.modules.inr import INR
+from models.modules.regressors import RidgeRegressor
+
+
+@gin.configurable()
+def deeptime(datetime_feats: int, layer_size: int, inr_layers: int, n_fourier_feats: int, scales: float):
+    return DeepTIMe(datetime_feats, layer_size, inr_layers, n_fourier_feats, scales)
+
+
+class DeepTIMe(nn.Module):
+    def __init__(self, datetime_feats: int, layer_size: int, inr_layers: int, n_fourier_feats: int, scales: float):
+        super().__init__()
+        self.inr = INR(in_feats=datetime_feats + 1, layers=inr_layers, layer_size=layer_size,
+                       n_fourier_feats=n_fourier_feats, scales=scales)
+        self.adaptive_weights = RidgeRegressor()
+
+        self.datetime_feats = datetime_feats
+        self.inr_layers = inr_layers
+        self.layer_size = layer_size
+        self.n_fourier_feats = n_fourier_feats
+        self.scales = scales
+
+    def forward(self, x: Tensor, x_time: Tensor, y_time: Tensor) -> Tensor:
+        tgt_horizon_len = y_time.shape[1]
+        batch_size, lookback_len, _ = x.shape
+        coords = self.get_coords(lookback_len, tgt_horizon_len).to(x.device)
+
+        if y_time.shape[-1] != 0:
+            time = torch.cat([x_time, y_time], dim=1)
+            coords = repeat(coords, '1 t 1 -> b t 1', b=time.shape[0])
+            coords = torch.cat([coords, time], dim=-1)
+            time_reprs = self.inr(coords)
+        else:
+            time_reprs = repeat(self.inr(coords), '1 t d -> b t d', b=batch_size)
+
+        lookback_reprs = time_reprs[:, :-tgt_horizon_len]
+        horizon_reprs = time_reprs[:, -tgt_horizon_len:]
+        w, b = self.adaptive_weights(lookback_reprs, x)
+        preds = self.forecast(horizon_reprs, w, b)
+        return preds
+
+    def forecast(self, inp: Tensor, w: Tensor, b: Tensor) -> Tensor:
+        return torch.einsum('... d o, ... t d -> ... t o', [w, inp]) + b
+
+    def get_coords(self, lookback_len: int, horizon_len: int) -> Tensor:
+        coords = torch.linspace(0, 1, lookback_len + horizon_len)
+        return rearrange(coords, 't -> 1 t 1')

models/__init__.py

@@ -0,0 +1,13 @@
+from typing import Union
+
+import torch
+
+from .DeepTIMe import deeptime
+
+
+def get_model(model_type: str, **kwargs: Union[int, float]) -> torch.nn.Module:
+    if model_type == 'deeptime':
+        model = deeptime(datetime_feats=kwargs['datetime_feats'])
+    else:
+        raise ValueError(f"Unknown model type {model_type}")
+    return model

models/modules/feature_transforms.py

@@ -0,0 +1,36 @@
+from typing import List, Optional
+
+import math
+import torch
+from torch import nn
+from torch import Tensor
+
+
+class GaussianFourierFeatureTransform(nn.Module):
+    """
+    https://github.com/ndahlquist/pytorch-fourier-feature-networks
+    Given an input of size [..., time, dim], returns a tensor of size [..., n_fourier_feats, time].
+    """
+    def __init__(self, input_dim: int, n_fourier_feats: int, scales: List[int]):
+        super().__init__()
+        self.input_dim = input_dim
+        self.n_fourier_feats = n_fourier_feats
+        self.scales = scales
+
+        n_scale_feats = n_fourier_feats // (2 * len(scales))
+        assert n_scale_feats * 2 * len(scales) == n_fourier_feats, \
+            f"n_fourier_feats: {n_fourier_feats} must be divisible by 2 * len(scales) = {2 * len(scales)}"
+        B_size = (input_dim, n_scale_feats)
+        B = torch.cat([torch.randn(B_size) * scale for scale in scales], dim=1)
+        self.register_buffer('B', B)
+
+    def forward(self, x: Tensor) -> Tensor:
+        assert x.dim() >= 2, f"Expected 2 or more dimensional input (got {x.dim()}D input)"
+        time, dim = x.shape[-2], x.shape[-1]
+
+        assert dim == self.input_dim, \
+            f"Expected input to have {self.input_dim} channels (got {dim} channels)"
+
+        x = torch.einsum('... t n, n d -> ... t d', [x, self.B])
+        x = 2 * math.pi * x
+        return torch.cat([torch.sin(x), torch.cos(x)], dim=-1)

models/modules/inr.py

@@ -0,0 +1,42 @@
+from typing import Optional
+
+import torch
+import torch.nn as nn
+from torch import Tensor
+
+from models.modules.feature_transforms import GaussianFourierFeatureTransform
+
+
+class INRLayer(nn.Module):
+    def __init__(self, input_size: int, output_size: int,
+                 dropout: Optional[float] = 0.1):
+        super().__init__()
+        self.input_size = input_size
+        self.output_size = output_size
+        self.linear = nn.Linear(input_size, output_size)
+        self.dropout = nn.Dropout(dropout)
+        self.norm = nn.LayerNorm(output_size)
+
+    def forward(self, x: Tensor) -> Tensor:
+        out = self._layer(x)
+        return self.norm(out)
+
+    def _layer(self, x: Tensor) -> Tensor:
+        return self.dropout(torch.relu(self.linear(x)))
+
+
+class INR(nn.Module):
+    def __init__(self, in_feats: int, layers: int, layer_size: int, n_fourier_feats: int, scales: float,
+                 dropout: Optional[float] = 0.1):
+        super().__init__()
+        self.features = nn.Linear(in_feats, layer_size) if n_fourier_feats == 0 \
+            else GaussianFourierFeatureTransform(in_feats, n_fourier_feats, scales)
+        in_size = layer_size if n_fourier_feats == 0 \
+            else n_fourier_feats
+        layers = [INRLayer(in_size, layer_size, dropout=dropout)] + \
+                 [INRLayer(layer_size, layer_size, dropout=dropout) for _ in range(layers - 1)]
+        self.layers = nn.Sequential(*layers)
+
+    def forward(self, x: Tensor) -> Tensor:
+        x = self.features(x)
+        return self.layers(x)

models/modules/regressors.py

@@ -0,0 +1,40 @@
+from typing import Optional
+
+import torch
+from torch import Tensor
+from torch import nn
+import torch.nn.functional as F
+
+
+class RidgeRegressor(nn.Module):
+    def __init__(self, lambda_init: Optional[float] =0.):
+        super().__init__()
+        self._lambda = nn.Parameter(torch.as_tensor(lambda_init, dtype=torch.float))
+
+    def forward(self, reprs: Tensor, x: Tensor, reg_coeff: Optional[float] = None) -> Tensor:
+        if reg_coeff is None:
+            reg_coeff = self.reg_coeff()
+        w, b = self.get_weights(reprs, x, reg_coeff)
+        return w, b
+
+    def get_weights(self, X: Tensor, Y: Tensor, reg_coeff: float) -> Tensor:
+        batch_size, n_samples, n_dim = X.shape
+        ones = torch.ones(batch_size, n_samples, 1, device=X.device)
+        X = torch.concat([X, ones], dim=-1)
+
+        if n_samples >= n_dim:
+            # standard
+            A = torch.bmm(X.mT, X)
+            A.diagonal(dim1=-2, dim2=-1).add_(reg_coeff)
+            B = torch.bmm(X.mT, Y)
+            weights = torch.linalg.solve(A, B)
+        else:
+            # Woodbury
+            A = torch.bmm(X, X.mT)
+            A.diagonal(dim1=-2, dim2=-1).add_(reg_coeff)
+            weights = torch.bmm(X.mT, torch.linalg.solve(A, Y))
+
+        return weights[:, :-1], weights[:, -1:]
+
+    def reg_coeff(self) -> Tensor:
+        return F.softplus(self._lambda)

requirements.txt

@@ -0,0 +1,13 @@
+fire==0.4.0
+functorch==0.1.1
+gin-config==0.5.0
+matplotlib==3.5.1
+numpy ==1.19.2
+pandas==1.4.2
+scikit-learn==1.0.2
+scipy==1.7.3
+tensorboard==2.8.0
+torch==1.11.0
+TorchOpt==0.4.1
+tqdm==4.62.3
+einops==0.4.1

run.sh

@@ -0,0 +1,6 @@
+for dataset in ECL ETTm2 Exchange ILI Traffic Weather; do
+  for instance in `/bin/ls -d storage/experiments/$dataset/*/*`; do
+      echo $instance
+      make run command=${instance}/command
+  done
+done

run_hp_search.sh

@@ -0,0 +1,4 @@
+for instance in `/bin/ls -d storage/experiments/hp_search/*/*`; do
+    echo $instance
+    make run command=${instance}/command
+done

storage/datasets/.gitignore

@@ -0,0 +1,4 @@
+# Ignore everything in this directory
+*
+# Except this file
+!.gitignore

storage/experiments/.gitignore

@@ -0,0 +1,4 @@
+# Ignore everything in this directory
+*
+# Except this file
+!.gitignore

utils/checkpoint.py

@@ -0,0 +1,59 @@
+from typing import Optional, Dict
+import logging
+from os.path import join
+
+import gin
+import torch
+from torch.utils.tensorboard import SummaryWriter
+
+
+@gin.configurable()
+class Checkpoint:
+    def __init__(self,
+                 checkpoint_dir: str,
+                 patience: Optional[int] = 7,
+                 delta: Optional[float] = 0.):
+        self.checkpoint_dir = checkpoint_dir
+        self.model_path = join(checkpoint_dir, 'model.pth')
+
+        # early stopping
+        self.patience = patience
+        self.counter = 0
+        self.best_loss = float('inf')
+        self.early_stop = False
+        self.delta = delta
+
+        # logging
+        self.summary_writer = SummaryWriter(log_dir=checkpoint_dir)
+
+    def __call__(self,
+                 epoch: int,
+                 model: torch.nn.Module,
+                 scalars: Optional[Dict[str, float]] = None):
+        for name, value in scalars.items():
+            # logging
+            self.summary_writer.add_scalar(name, value, epoch)
+
+            # early stopping
+            if name == 'Loss/Val':
+                val_loss = value
+                if val_loss <= self.best_loss + self.delta:
+                    logging.info(
+                        f"Validation loss decreased ({self.best_loss:.3f} --> {val_loss:.3f}). Saving model ...")
+                    torch.save(model.state_dict(), self.model_path)
+                    self.best_loss = val_loss
+                    self.counter = 0
+                else:
+                    self.counter += 1
+                    logging.info(f"Validation loss increased ({self.best_loss:.3f} --> {val_loss:.3f}). "
+                                 f"Early stopping counter: {self.counter} out of {self.patience}")
+                    if self.counter >= self.patience >= 0:
+                        self.early_stop = True
+
+        self.summary_writer.flush()
+
+    def close(self, scores: Optional[Dict[str, float]] = None):
+        if scores is not None:
+            for name, value in scores.items():
+                self.summary_writer.add_scalar(name, value)
+        self.summary_writer.close()

utils/losses.py

@@ -0,0 +1,15 @@
+from typing import Optional, Callable
+from functools import partial
+
+import torch
+import torch.nn.functional as F
+from torch import Tensor
+
+
+def get_loss_fn(loss_name: str,
+                delta: Optional[float] = 1.0,
+                beta: Optional[float] = 1.0) -> Callable:
+    return {'mse': F.mse_loss,
+            'mae': F.l1_loss,
+            'huber': partial(F.huber_loss, delta=delta),
+            'smooth_l1': partial(F.smooth_l1_loss, beta=beta)}[loss_name]

utils/metrics.py

@@ -0,0 +1,39 @@
+import numpy as np
+
+
+def rse(pred, true):
+    return np.sqrt(np.sum((true - pred) ** 2)) / np.sqrt(np.sum((true - true.mean()) ** 2))
+
+
+def corr(pred, true):
+    u = ((true - true.mean(0)) * (pred - pred.mean(0))).sum(0)
+    d = np.sqrt(((true - true.mean(0)) ** 2 * (pred - pred.mean(0)) ** 2).sum(0))
+    return (u / d).mean(-1)
+
+
+def mae(pred, true):
+    return np.mean(np.abs(pred - true))
+
+
+def mse(pred, true):
+    return np.mean((pred - true) ** 2)
+
+
+def rmse(pred, true):
+    return np.sqrt(mse(pred, true))
+
+
+def mape(pred, true):
+    return np.mean(np.abs((pred - true) / true))
+
+
+def mspe(pred, true):
+    return np.mean(np.square((pred - true) / true))
+
+
+def calc_metrics(pred, true):
+    return {'mae': mae(pred, true),
+            'mse': mse(pred, true),
+            'rmse': rmse(pred, true),
+            'mape': mape(pred, true),
+            'mspe': mspe(pred, true)}

utils/ops.py

@@ -0,0 +1,54 @@
+from typing import Optional, Tuple
+
+import numpy as np
+import torch
+from torch import Tensor
+from einops import reduce
+
+
+def default_device() -> torch.device:
+    """
+    PyTorch default device is GPU when available, CPU otherwise.
+    :return: Default device.
+    """
+    return torch.device('cuda' if torch.cuda.is_available() else 'cpu')
+
+
+def to_tensor(array: np.ndarray, to_default_device: Optional[bool] = True) -> Tensor:
+    """
+    Convert numpy array to tensor on default device.
+    :param array: Numpy array to convert.
+    :param to_default_device Place tensor on default device or not.
+    :return: PyTorch tensor, optionally on default device.
+    """
+    if to_default_device:
+        return torch.as_tensor(array, dtype=torch.float32).to(default_device())
+
+
+def divide_no_nan(a, b):
+    """
+    a/b where the resulted NaN or Inf are replaced by 0.
+    """
+    mask = b == .0
+    b[mask] = 1.
+    result = a / b
+    result[mask] = .0
+    result[result != result] = .0
+    result[result == np.inf] = .0
+    return result
+
+
+def scale(x: Tensor,
+          scaling_factor: Optional[Tensor] = None) -> Tuple[Tensor, Tensor]:
+    if scaling_factor is not None:
+        x = x / scaling_factor
+        return x, scaling_factor
+
+    scaling_factor = reduce(torch.abs(x).data, 'b t d -> b 1 d', 'mean')
+    scaling_factor[scaling_factor == 0.0] = 1.0
+    x = x / scaling_factor
+    return x, scaling_factor
+
+
+def descale(forecast: Tensor, scaling_factor: Tensor) -> Tensor:
+    return forecast * scaling_factor

utils/time_features.py

@@ -0,0 +1,182 @@
+from abc import ABC, abstractmethod
+from typing import Optional, List, Union
+
+import numpy as np
+import pandas as pd
+
+
+class TimeFeature(ABC):
+    """Abstract class for time features"""
+    def __init__(self, normalise: bool, a: float, b: float):
+        self.normalise = normalise
+        self.a = a
+        self.b = b
+
+    @abstractmethod
+    def __call__(self, idx: pd.DatetimeIndex) -> np.ndarray:
+        ...
+
+    @property
+    @abstractmethod
+    def _max_val(self) -> float:
+        ...
+
+    @property
+    def max_val(self) -> float:
+        return self._max_val if self.normalise else 1.0
+
+    def scale(self, val: np.ndarray) -> np.ndarray:
+        return val * (self.b - self.a) + self.a
+
+    def process(self, val: np.ndarray) -> np.ndarray:
+        features = self.scale(val / self.max_val)
+        if self.normalise:
+            return features
+        return features.astype(int)
+
+    def __repr__(self) -> str:
+        return f"{self.__class__.__name__}(normalise={self.normalise}, a={self.a}, b={self.b})"
+
+
+class SecondOfMinute(TimeFeature):
+    """Second of minute, unnormalised: [0, 59]"""
+    def __call__(self, idx: pd.DatetimeIndex) -> np.ndarray:
+        return self.process(idx.second)
+
+    @property
+    def _max_val(self):
+        return 59.0
+
+
+class MinuteOfHour(TimeFeature):
+    """Minute of hour, unnormalised: [0, 59]"""
+    def __call__(self, idx: pd.DatetimeIndex) -> np.ndarray:
+        return self.process(idx.minute)
+
+    @property
+    def _max_val(self):
+        return 59.0
+
+
+class HourOfDay(TimeFeature):
+    """Hour of day, unnormalised: [0, 23]"""
+    def __call__(self, idx: pd.DatetimeIndex) -> np.ndarray:
+        return self.process(idx.hour)
+
+    @property
+    def _max_val(self):
+        return 23.0
+
+
+class DayOfWeek(TimeFeature):
+    """Hour of day, unnormalised: [0, 6]"""
+    def __call__(self, idx: pd.DatetimeIndex) -> np.ndarray:
+        return self.process(idx.dayofweek)
+
+    @property
+    def _max_val(self):
+        return 6.0
+
+
+class DayOfMonth(TimeFeature):
+    """Day of month, unnormalised: [0, 30]"""
+    def __call__(self, idx: pd.DatetimeIndex) -> np.ndarray:
+        return self.process(idx.day - 1)
+
+    @property
+    def _max_val(self):
+        return 30.0
+
+
+class DayOfYear(TimeFeature):
+    """Day of year, unnormalised: [0, 365]"""
+    def __call__(self, idx: pd.DatetimeIndex) -> np.ndarray:
+        return self.process(idx.dayofyear - 1)
+
+    @property
+    def _max_val(self):
+        return 365.0
+
+
+class WeekOfYear(TimeFeature):
+    """Week of year, unnormalised: [0, 52]"""
+    def __call__(self, idx: pd.DatetimeIndex) -> np.ndarray:
+        return self.process(pd.Index(idx.isocalendar().week, dtype=int) - 1)
+
+    @property
+    def _max_val(self):
+        return 52.0
+
+class MonthOfYear(TimeFeature):
+    """Month of year, unnormalised: [0, 11]"""
+    def __call__(self, idx: pd.DatetimeIndex) -> np.ndarray:
+        return self.process(idx.month - 1)
+
+    @property
+    def _max_val(self):
+        return 11.0
+
+
+class QuarterOfYear(TimeFeature):
+    """Quarter of year, unnormalised: [0, 3]"""
+    def __call__(self, idx: pd.DatetimeIndex) -> np.ndarray:
+        return self.process(idx.quarter - 1)
+
+    @property
+    def _max_val(self):
+        return 3.0
+
+
+str_to_feat = {
+    # dictionary mapping name to TimeFeature function
+    'SecondOfMinute': SecondOfMinute,
+    'MinuteOfHour': MinuteOfHour,
+    'HourOfDay': HourOfDay,
+    'DayOfWeek': DayOfWeek,
+    'DayOfMonth': DayOfMonth,
+    'DayOfYear': DayOfYear,
+    'WeekOfYear': WeekOfYear,
+    'MonthOfYear': MonthOfYear,
+    'QuarterOfYear': QuarterOfYear,
+}
+
+
+freq_to_feats = {
+    # dictionary mapping frequency to list of TimeFeature functions
+    'q': [QuarterOfYear],
+    'm': [QuarterOfYear, MonthOfYear],
+    'w': [QuarterOfYear, MonthOfYear, WeekOfYear],
+    'd': [QuarterOfYear, MonthOfYear, WeekOfYear, DayOfYear, DayOfMonth, DayOfWeek],
+    'h': [QuarterOfYear, MonthOfYear, WeekOfYear, DayOfYear, DayOfMonth, DayOfWeek, HourOfDay],
+    't': [QuarterOfYear, MonthOfYear, WeekOfYear, DayOfYear, DayOfMonth, DayOfWeek, HourOfDay, MinuteOfHour],
+    's': [QuarterOfYear, MonthOfYear, WeekOfYear, DayOfYear, DayOfMonth, DayOfWeek, HourOfDay, MinuteOfHour, SecondOfMinute],
+}
+
+
+def get_time_features(dates: pd.DatetimeIndex, normalise: bool, a: Optional[float] = 0., b: Optional[float] = 1.,
+                      features: Optional[Union[str, List[str]]] = None) -> np.ndarray:
+    """
+    Returns a numpy array of date/time features based on either frequency or directly specifying a list of features.
+    :param dates: DatetimeIndex object of shape (time,)
+    :param normalise: Whether to normalise feature between [a, b]. If not, return as an int in the original feature range.
+    :param a: Lower bound of feature
+    :param b: Upper bound of feature
+    :param features: Frequency string used to obtain list of TimeFeatures, or directly a list of names of TimeFeatures
+    :return: np array of date/time features of shape (time, n_feats)
+    """
+    if isinstance(features, list):
+        assert all([feat in str_to_feat.keys() for feat in features]), \
+            f"items in list should be one of {[*str_to_feat.keys()]}"
+        features = [str_to_feat[feat] for feat in features]
+    elif isinstance(features, str):
+        assert features in freq_to_feats.keys(), \
+            f"features should be one of {[*freq_to_feats.keys()]}"
+        features = freq_to_feats[features]
+    else:
+        raise ValueError(f"features should be a list or str, not a {type(features)}")
+
+    features = [feat(normalise, a, b)(dates) for feat in features]
+
+    if len(features) == 0:
+        return np.empty((dates.shape[0], 0))
+    return np.stack(features, axis=1)