wassname/Volt
1
0
Fork 0
mirror of https://github.com/wassname/Volt.git synced 2026-09-09 11:16:09 +08:00
Code Issues Packages Projects Releases Wiki Activity

first commit

Browse Source
This commit is contained in:
g-benton
2022-06-14 16:18:08 -07:00
parent f293a6c489
commit a62879ada8
160 changed files with 1847 additions and 792701 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.ipynb_checkpoints/Example-checkpoint.ipynb
+460
View File
File diff suppressed because one or more lines are too long
.ipynb_checkpoints/README-checkpoint.md
+27
View File
@@ -0,0 +1,27 @@
# Volt
Public Implementation of
*Volatility Based Kernels and Moving Average Means for Accurate Forecasting with Gaussian Processes* [link]
by [Gregory Benton](https://g-benton.github.io/), [Wesley Maddox](https://wjmaddox.github.io), and [Andrew Gordon Wilson](https://cims.nyu.edu/~andrewgw/).
Please cite our work if you find it useful:
```
@inproceedings{benton2022volatility,
title={olatility Based Kernels and Moving Average Means for Accurate Forecasting with Gaussian Processes},
author={Benton, Gregory and Maddox, Wesley and Wilson, Andrew Gordon Gordon},
booktitle={International Conference on Machine Learning},
year={2022},
organization={PMLR}
}
```
![Overview of the Volt modeling pipeline](./figs/ret-vol-px.jpg)
## Explanatory Notebook
To see an overview of how to use Volt with synthetically generated code, see the `Example` notebook which walks through how the code is organized step by step.
## Experiments
The two core experimental settings from the paper involve modeling historical wind speeds and stock prices. The code to run these experiments with example commands is in the `experiments` folder.
.ipynb_checkpoints/setup-checkpoint.py
+26
View File
@@ -0,0 +1,26 @@
from setuptools import setup
import os
import sys
setup(
name='voltron',
version='alpha',
description=('Voltron Repo'),
author='Greg Benton',
author_email='greg.w.benton@gmail.com',
url='https://github.com/g-benton/voltron',
license='Apache-2.0',
packages=['voltron'],
install_requires=[
'matplotlib>=3.0.3',
'setuptools>=41.0.0',
'torch>=1.11.0',
'numpy>=1.16.2',
'gpytorch>=1.0.1',
],
include_package_data=True,
classifiers=[
'Development Status :: 0',
'Intended Audience :: Science/Research',
'Programming Language :: Python :: 3.7'],
)
Example.ipynb
+460
View File
File diff suppressed because one or more lines are too long
README.md
+26 -1
View File
@@ -1,2 +1,27 @@
# Volt
Public Implementation of Volatility Based Kernels and Moving Average Means for Accurate Forecasting with Gaussian Processes
Public Implementation of
*Volatility Based Kernels and Moving Average Means for Accurate Forecasting with Gaussian Processes* [link]
by [Gregory Benton](https://g-benton.github.io/), [Wesley Maddox](https://wjmaddox.github.io), and [Andrew Gordon Wilson](https://cims.nyu.edu/~andrewgw/).
Please cite our work if you find it useful:
```
@inproceedings{benton2022volatility,
title={olatility Based Kernels and Moving Average Means for Accurate Forecasting with Gaussian Processes},
author={Benton, Gregory and Maddox, Wesley and Wilson, Andrew Gordon Gordon},
booktitle={International Conference on Machine Learning},
year={2022},
organization={PMLR}
}
```
![Overview of the Volt modeling pipeline](./figs/ret-vol-px.jpg)
## Explanatory Notebook
To see an overview of how to use Volt with synthetically generated code, see the `Example` notebook which walks through how the code is organized step by step.
## Experiments
The two core experimental settings from the paper involve modeling historical wind speeds and stock prices. The code to run these experiments with example commands is in the `experiments` folder.
experiments/SPY-options/SABR_calibration.ipynb
-637
View File
@@ -1,637 +0,0 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": 1,
"id": "0d67e226",
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"import seaborn as sns\n",
"import matplotlib.pyplot as plt\n",
"from matplotlib.lines import Line2D\n",
"from matplotlib.patches import Patch\n",
"import torch\n",
"import pandas as pd\n",
"import copy\n",
"from voltron.option_utils import GetTradingDays, GetTrainingData, Pricer, FindLastTradingDays\n",
"from scipy.optimize import minimize"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "8ad5b4c2",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/Users/gregorybenton/miniconda3/lib/python3.8/site-packages/IPython/core/interactiveshell.py:3169: DtypeWarning: Columns (19) have mixed types.Specify dtype option on import or set low_memory=False.\n",
" has_raised = await self.run_ast_nodes(code_ast.body, cell_name,\n"
]
}
],
"source": [
"SPY = pd.read_csv(\"./data/SPY_prices.csv\")\n",
"SPY['Date'] = pd.to_datetime(SPY['Date'])\n",
"\n",
"years = np.arange(2009, 2018)\n",
"opts = pd.DataFrame()\n",
"for year in years:\n",
" dat = pd.read_csv(\"./data/SPY_\" + str(year) + \".csv\")\n",
" dat = dat[dat.type == \"call\"]\n",
" quotedate = dat.quotedate.unique()[0]\n",
" dat = dat[dat.quotedate == quotedate] \n",
" opts = pd.concat((opts, dat), ignore_index=True)\n",
"\n",
"# exps = [] \n",
"# for idx, row in opts.iterrows():\n",
"# eday = row.expiration\n",
"# exps.append(SPY[SPY.Date == FindLastTradingDays(SPY, [pd.Timestamp(eday)])[0]].Close.item())\n",
"# opts['exp_price'] = exps"
]
},
{
"cell_type": "markdown",
"id": "f361417a",
"metadata": {},
"source": [
"## Torch Attempt"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "06969ba7",
"metadata": {},
"outputs": [],
"source": [
"ivol = torch.tensor(opts.impliedvol.to_numpy())\n",
"Fs = torch.tensor(opts.underlying_last.to_numpy())\n",
"Ks = torch.tensor(opts.strike.to_numpy())\n",
"qdays = pd.to_datetime(opts.quotedate).dt.date.to_numpy()\n",
"edays = pd.to_datetime(opts.expiration).dt.date.to_numpy()\n",
"Ts = torch.tensor(([np.busday_count(qd, ed)/252. for qd, ed in zip(qdays, edays)]))"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "465286fb",
"metadata": {},
"outputs": [],
"source": [
"def BlackVol(pars, K, f, T):\n",
" alpha = torch.exp(pars[0][0])\n",
" rho = 2 * torch.sigmoid(pars[0][1]) - 1.\n",
" v = torch.exp(pars[0][2])\n",
" beta = 1.\n",
" num = 1 + (alpha**2 * (1-beta)**2/(24 * (f*K)**(1-beta)) + 0.25 * rho*beta*v*alpha/((f*K)**(0.5*(1-beta))) + v**2*(2-3*rho**2)/24)*T\n",
" num*= alpha\n",
" \n",
" denom = (f*K)**(0.5*(1-beta)) * (1 + (1-beta)**2/24 * torch.log(f/K)**2 + (1-beta)**4/1920 * torch.log(f/K)**4)\n",
" \n",
" z = v/alpha * (f*K)**(0.5*(1-beta)) * np.log(f/K)\n",
" xi_z = torch.log((torch.sqrt(1 - 2 * rho * z + z**2) + z - rho)/(1-rho))\n",
" \n",
" return num/denom * z/xi_z"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "1ebbc187",
"metadata": {},
"outputs": [],
"source": [
"def MinVol(pars):\n",
" return torch.mean((ivol - BlackVol(pars, Ks, Fs, Ts)).pow(2))"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "57bc7a65",
"metadata": {},
"outputs": [],
"source": [
"pars = [torch.tensor([-1., -5., -3.], requires_grad=True)]\n",
"opt = torch.optim.SGD(pars, lr=0.1)"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "ab9c2a60",
"metadata": {},
"outputs": [],
"source": [
"iters = 1000\n",
"stored_pars = torch.zeros(iters, 3)\n",
"losses = []\n",
"for e in range(iters):\n",
" stored_pars[e, :] = pars[0]\n",
" loss = MinVol(pars)\n",
" opt.zero_grad()\n",
" loss.backward()\n",
" losses.append(loss.item())\n",
" opt.step() "
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "6f6aeab5",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[<matplotlib.lines.Line2D at 0x7fc4a7df89d0>,\n",
" <matplotlib.lines.Line2D at 0x7fc4a7df8a00>,\n",
" <matplotlib.lines.Line2D at 0x7fc4a7df8b80>]"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"plt.plot(stored_pars.detach())"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "271add47",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[<matplotlib.lines.Line2D at 0x7fc4a81a7700>]"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"plt.plot(losses)"
]
},
{
"cell_type": "markdown",
"id": "93a71a4d",
"metadata": {},
"source": [
"# Running SABR"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "0bb6a328",
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"import torch\n",
"import pandas as pd\n",
"import copy\n",
"from voltron.option_utils import GetTradingDays, GetTrainingData, Pricer, FindLastTradingDays\n",
"from scipy.optimize import minimize\n",
"\n",
"def BlackVol(pars, K, f, T):\n",
" alpha = torch.exp(pars[0][0]) ## v0\n",
" rho = 2 * torch.sigmoid(pars[0][1]) - 1. ##rho\n",
" v = torch.exp(pars[0][2]) ## \"sigma\" \n",
" beta = 1.\n",
" num = 1 + (alpha**2 * (1-beta)**2/(24 * (f*K)**(1-beta)) +\\\n",
" 0.25 * rho*beta*v*alpha/((f*K)**(0.5*(1-beta))) +\\\n",
" v**2*(2-3*rho**2)/24)*T\n",
" num*= alpha\n",
" \n",
" denom = (f*K)**(0.5*(1-beta)) * (1 + (1-beta)**2/24 * torch.log(f/K)**2 +\\\n",
" (1-beta)**4/1920 * torch.log(f/K)**4)\n",
" \n",
" z = v/alpha * (f*K)**(0.5*(1-beta)) * np.log(f/K)\n",
" xi_z = torch.log((torch.sqrt(1 - 2 * rho * z + z**2) + z - rho)/(1-rho))\n",
" \n",
" return num/denom * z/xi_z\n",
"\n",
"def MinVol(pars, Ks, Fs, Ts, ivol):\n",
" return torch.mean((ivol - BlackVol(pars, Ks, Fs, Ts)).pow(2))\n",
"\n",
"def Calibrate(Fs, Ks, Ts, ivol, iters=1000):\n",
" pars = [torch.tensor([-1., -5., -3.], requires_grad=True)]\n",
" opt = torch.optim.SGD(pars, lr=0.1)\n",
" stored_pars = torch.zeros(iters, 3)\n",
" losses = []\n",
" for e in range(iters):\n",
" stored_pars[e, :] = pars[0]\n",
" loss = MinVol(pars, Ks, Fs, Ts, ivol)\n",
" opt.zero_grad()\n",
" loss.backward()\n",
" losses.append(loss.item())\n",
" opt.step() \n",
" \n",
" return pars[0].detach().numpy()\n",
"\n",
"def SABRSim(Np, Nt, S0, V0, sigma, rho, dt=1./252.):\n",
" dW = np.random.randn(Nt+1, Np) * np.sqrt(dt)\n",
" dZ = rho * dW + np.sqrt(1-rho**2) * np.random.randn(Nt+1, Np) * np.sqrt(dt)\n",
" \n",
" S = np.zeros((Nt+1, Np))\n",
" S[0] = S0\n",
" V = np.zeros((Nt+1, Np))\n",
" V[0] = V0\n",
" \n",
" for t in range(Nt):\n",
" S[t+1] = S[t] + V[t]*S[t]*dW[t]\n",
" V[t+1] = V[t] + sigma*V[t]*dZ[t]\n",
" \n",
" return S[1:]"
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "67d9cf31",
"metadata": {},
"outputs": [],
"source": [
"logger = []\n",
"full_logger = []\n",
"SPY = pd.read_csv(\"./data/SPY_prices.csv\")\n",
"SPY['Date'] = pd.to_datetime(SPY['Date'])\n",
"Np = 10000\n",
"ntrain = 252\n",
"year = 2012\n",
"options = pd.read_csv(\"./data/SPY_\" + str(year) + \".csv\")\n",
"options.expiration = pd.to_datetime(options.expiration)\n",
"options.quotedate = pd.to_datetime(options.quotedate)\n",
"qday = options.quotedate.unique()[0]\n",
"quote_price = SPY[SPY['Date']==qday].Close.item()\n",
"options = options[(options.quotedate == qday) & (options.type=='call')]\n",
"edays = options.expiration.sort_values().unique()\n",
"testdays = (edays - qday)/np.timedelta64(1, \"D\")\n",
"edays = edays[(testdays > 100) & (testdays < 365)]\n",
"lastdays = FindLastTradingDays(SPY, edays)\n",
"ntests = np.array([GetTradingDays(SPY, qday, \n",
" pd.Timestamp(ld)) for ld in lastdays])\n",
"fulltest = ntests[-1]\n",
"train_y = torch.FloatTensor(GetTrainingData(SPY, qday, ntrain).to_numpy())\n",
"test_y = torch.FloatTensor(GetTrainingData(SPY, \n",
" pd.Timestamp(lastdays[-1]),\n",
" fulltest).to_numpy())\n",
"full_x = torch.arange(ntrain+fulltest).type(torch.FloatTensor)\n",
"full_x = full_x/252.\n",
"train_x = full_x[:ntrain]\n",
"test_x = full_x[ntrain:]\n",
"\n",
"## extract data for calibration ##\n",
"ivol = torch.tensor(options.impliedvol.to_numpy())\n",
"Fs = torch.tensor(options.underlying_last.to_numpy())\n",
"Ks = torch.tensor(options.strike.to_numpy())\n",
"starts = options.quotedate.dt.date.to_numpy()\n",
"ends = options.expiration.dt.date.to_numpy()\n",
"Ts = torch.tensor(([np.busday_count(qd, ed)/252. for qd, ed in zip(starts, ends)]))\n",
"\n",
"pars = Calibrate(Fs, Ks, Ts, ivol)\n",
"v0 = np.exp(pars[0])\n",
"1/(1 + np.exp(-pars[1]))\n",
"rho = (2/(1 + np.exp(-pars[1])) - 1.)\n",
"sigma = np.exp(pars[2])\n",
"px_paths = SABRSim(Np, fulltest, quote_price, v0, sigma, rho)\n",
"px_samples = torch.tensor(px_paths[ntests-1])"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "abc47786",
"metadata": {},
"outputs": [],
"source": [
"option_output = Pricer(torch.tensor(px_paths), options, edays, test_y[ntests-1],\n",
" quote_price)"
]
},
{
"cell_type": "code",
"execution_count": 13,
"id": "9bb8972b",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Expiry</th>\n",
" <th>Strike</th>\n",
" <th>Bid</th>\n",
" <th>Ask</th>\n",
" <th>Voltron</th>\n",
" <th>Return</th>\n",
" <th>ExpClose</th>\n",
" <th>QuoteClose</th>\n",
" <th>Year</th>\n",
" <th>Sample_Percentile</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>2013-03-16</td>\n",
" <td>20.0</td>\n",
" <td>121.39</td>\n",
" <td>121.61</td>\n",
" <td>126.884545</td>\n",
" <td>136.729996</td>\n",
" <td>156.729996</td>\n",
" <td>141.449997</td>\n",
" <td>2013</td>\n",
" <td>0.781553</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>2013-03-16</td>\n",
" <td>25.0</td>\n",
" <td>116.39</td>\n",
" <td>116.61</td>\n",
" <td>121.884545</td>\n",
" <td>131.729996</td>\n",
" <td>156.729996</td>\n",
" <td>141.449997</td>\n",
" <td>2013</td>\n",
" <td>0.781553</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>2013-03-16</td>\n",
" <td>30.0</td>\n",
" <td>111.39</td>\n",
" <td>111.61</td>\n",
" <td>116.884545</td>\n",
" <td>126.729996</td>\n",
" <td>156.729996</td>\n",
" <td>141.449997</td>\n",
" <td>2013</td>\n",
" <td>0.781553</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>2013-03-16</td>\n",
" <td>35.0</td>\n",
" <td>106.39</td>\n",
" <td>106.61</td>\n",
" <td>111.884545</td>\n",
" <td>121.729996</td>\n",
" <td>156.729996</td>\n",
" <td>141.449997</td>\n",
" <td>2013</td>\n",
" <td>0.781553</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>2013-03-16</td>\n",
" <td>40.0</td>\n",
" <td>101.39</td>\n",
" <td>101.61</td>\n",
" <td>106.884545</td>\n",
" <td>116.729996</td>\n",
" <td>156.729996</td>\n",
" <td>141.449997</td>\n",
" <td>2013</td>\n",
" <td>0.781553</td>\n",
" </tr>\n",
" <tr>\n",
" <th>...</th>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>651</th>\n",
" <td>2013-09-30</td>\n",
" <td>169.0</td>\n",
" <td>0.33</td>\n",
" <td>0.42</td>\n",
" <td>0.000000</td>\n",
" <td>0.690002</td>\n",
" <td>169.690002</td>\n",
" <td>141.449997</td>\n",
" <td>2013</td>\n",
" <td>1.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>652</th>\n",
" <td>2013-09-30</td>\n",
" <td>170.0</td>\n",
" <td>0.28</td>\n",
" <td>0.37</td>\n",
" <td>0.000000</td>\n",
" <td>0.000000</td>\n",
" <td>169.690002</td>\n",
" <td>141.449997</td>\n",
" <td>2013</td>\n",
" <td>1.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>653</th>\n",
" <td>2013-09-30</td>\n",
" <td>175.0</td>\n",
" <td>0.14</td>\n",
" <td>0.21</td>\n",
" <td>0.000000</td>\n",
" <td>0.000000</td>\n",
" <td>169.690002</td>\n",
" <td>141.449997</td>\n",
" <td>2013</td>\n",
" <td>1.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>654</th>\n",
" <td>2013-09-30</td>\n",
" <td>180.0</td>\n",
" <td>0.08</td>\n",
" <td>0.12</td>\n",
" <td>0.000000</td>\n",
" <td>0.000000</td>\n",
" <td>169.690002</td>\n",
" <td>141.449997</td>\n",
" <td>2013</td>\n",
" <td>1.000000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>655</th>\n",
" <td>2013-09-30</td>\n",
" <td>185.0</td>\n",
" <td>0.04</td>\n",
" <td>0.08</td>\n",
" <td>0.000000</td>\n",
" <td>0.000000</td>\n",
" <td>169.690002</td>\n",
" <td>141.449997</td>\n",
" <td>2013</td>\n",
" <td>1.000000</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>656 rows × 10 columns</p>\n",
"</div>"
],
"text/plain": [
" Expiry Strike Bid Ask Voltron Return ExpClose \\\n",
"0 2013-03-16 20.0 121.39 121.61 126.884545 136.729996 156.729996 \n",
"1 2013-03-16 25.0 116.39 116.61 121.884545 131.729996 156.729996 \n",
"2 2013-03-16 30.0 111.39 111.61 116.884545 126.729996 156.729996 \n",
"3 2013-03-16 35.0 106.39 106.61 111.884545 121.729996 156.729996 \n",
"4 2013-03-16 40.0 101.39 101.61 106.884545 116.729996 156.729996 \n",
".. ... ... ... ... ... ... ... \n",
"651 2013-09-30 169.0 0.33 0.42 0.000000 0.690002 169.690002 \n",
"652 2013-09-30 170.0 0.28 0.37 0.000000 0.000000 169.690002 \n",
"653 2013-09-30 175.0 0.14 0.21 0.000000 0.000000 169.690002 \n",
"654 2013-09-30 180.0 0.08 0.12 0.000000 0.000000 169.690002 \n",
"655 2013-09-30 185.0 0.04 0.08 0.000000 0.000000 169.690002 \n",
"\n",
" QuoteClose Year Sample_Percentile \n",
"0 141.449997 2013 0.781553 \n",
"1 141.449997 2013 0.781553 \n",
"2 141.449997 2013 0.781553 \n",
"3 141.449997 2013 0.781553 \n",
"4 141.449997 2013 0.781553 \n",
".. ... ... ... \n",
"651 141.449997 2013 1.000000 \n",
"652 141.449997 2013 1.000000 \n",
"653 141.449997 2013 1.000000 \n",
"654 141.449997 2013 1.000000 \n",
"655 141.449997 2013 1.000000 \n",
"\n",
"[656 rows x 10 columns]"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"option_output"
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "51771e55",
"metadata": {},
"outputs": [],
"source": [
"plt.plot(train_x, train_y)\n",
"plt.plot(test_x, test_y)\n",
"plt.plot(test_x, px_paths[:, :20], c='gray', alpha=0.5);"
]
},
{
"cell_type": "code",
"execution_count": 15,
"id": "a9503b79",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(206, 20)"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"px_paths[:, :20].shape"
]
},
{
"cell_type": "code",
"execution_count": 16,
"id": "13d5c888",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(206, 10000)"
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"px_paths.shape"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "5867b14f",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.3"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
experiments/SPY-options/basic_model_explore.ipynb
-438
View File
File diff suppressed because one or more lines are too long
experiments/SPY-options/basic_runner.py
-97
View File
@@ -1,97 +0,0 @@
import numpy as np
import datetime as dt
import matplotlib.pyplot as plt
import seaborn as sns
import torch
import gpytorch
import os
# import robin_stocks.robinhood as r
import pickle5 as pickle
import pandas as pd
import argparse
import sys
sys.path.append("../")
from voltron.likelihoods import VolatilityGaussianLikelihood
from voltron.models import SingleTaskVariationalGP as SingleTaskCopulaProcessModel
from voltron.kernels import BMKernel, VolatilityKernel
from voltron.models import BMGP, VoltronGP
from gpytorch.kernels import ScaleKernel, RBFKernel, MaternKernel
from voltron.option_utils import GetTradingDays, GetTrainingData, Pricer, FindLastTradingDays
from voltron.train_utils import TrainBasicModel
def main(args):
years = [yr for yr in range(2006, 2018)]
logger = []
full_logger = []
SPY = pd.read_csv("./data/SPY_prices.csv")
SPY['Date'] = pd.to_datetime(SPY['Date'])
ntrain = 252
nvol = 100
npx = 100
for year in years:
options = pd.read_csv("./data/SPY_" + str(year) + ".csv")
options.expiration = pd.to_datetime(options.expiration)
options.quotedate = pd.to_datetime(options.quotedate)
qday = options.quotedate.unique()[0]
quote_price = SPY[SPY['Date']==qday].Close.item()
options = options[(options.quotedate == qday) & (options.type=='call')]
edays = options.expiration.sort_values().unique()
testdays = (edays - qday)/np.timedelta64(1, "D")
edays = edays[(testdays > 100) & (testdays < 365)]
lastdays = FindLastTradingDays(SPY, edays)
ntests = np.array([GetTradingDays(SPY, qday, pd.Timestamp(ld)) for ld in lastdays])
fulltest = ntests[-1]
train_y = torch.FloatTensor(GetTrainingData(SPY, qday, ntrain).to_numpy())
test_y = torch.FloatTensor(GetTrainingData(SPY,
pd.Timestamp(lastdays[-1]),
fulltest).to_numpy())
full_x = torch.arange(ntrain+fulltest).type(torch.FloatTensor)
full_x = full_x/252.
train_x = full_x[:ntrain]
test_x = full_x[ntrain:]
dmod, dlh = TrainBasicModel(train_x, train_y, train_iters=500, model_type=args.model,
mean_func=args.mean_func)
## figure out how to price options sanely ##
nvol = 100
npx = 100
px_samples = torch.zeros(npx*nvol, len(edays))
px_paths = torch.zeros(npx*nvol, fulltest)
dmod.eval();
for vidx in range(nvol):
px_pred = dlh(dmod(test_x)).sample(torch.Size((npx,))).exp()
px_paths[vidx*npx:(vidx*npx + npx), :] = px_pred.detach()
px_samples[vidx*npx:(vidx*npx+npx), :] = px_pred[:, ntests-1].detach()
option_output = Pricer(px_samples, options, edays, test_y[ntests-1],
quote_price)
option_output.to_pickle("./output/" + args.model + "_options" + str(year) + ".pkl")
print(str(year), "Done")
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument(
"--mean_func",
type=str,
default="loglinear",
)
parser.add_argument(
"--model",
type=str,
default="matern",
)
args = parser.parse_args()
main(args)
experiments/SPY-options/check_outputs.ipynb
-536
View File
File diff suppressed because one or more lines are too long
experiments/SPY-options/custom_plotter.ipynb
-835
View File
File diff suppressed because one or more lines are too long
experiments/SPY-options/data/.DS_Store
Vendored
BIN
View File
Binary file not shown.
experiments/SPY-options/data/SPY_2006.csv
-19177
View File
File diff suppressed because it is too large Load Diff
experiments/SPY-options/data/SPY_2007.csv
-37211
View File
File diff suppressed because it is too large Load Diff
experiments/SPY-options/data/SPY_2008.csv
-57398
View File
File diff suppressed because it is too large Load Diff
experiments/SPY-options/data/SPY_2009.csv
-46609
View File
File diff suppressed because it is too large Load Diff
experiments/SPY-options/data/SPY_2010.csv
-49639
View File
File diff suppressed because it is too large Load Diff
experiments/SPY-options/data/SPY_2011.csv
-49923
View File
File diff suppressed because it is too large Load Diff
experiments/SPY-options/data/SPY_2012.csv
-60594
View File
File diff suppressed because it is too large Load Diff
experiments/SPY-options/data/SPY_2013.csv
-61521
View File
File diff suppressed because it is too large Load Diff
experiments/SPY-options/data/SPY_2014.csv
-77877
View File
File diff suppressed because it is too large Load Diff
experiments/SPY-options/data/SPY_2015.csv
-85353
View File
File diff suppressed because it is too large Load Diff
experiments/SPY-options/data/SPY_2016.csv
-100815
View File
File diff suppressed because it is too large Load Diff
experiments/SPY-options/data/SPY_2017.csv
-90595
View File
File diff suppressed because it is too large Load Diff
experiments/SPY-options/data/SPY_prices.csv
-7215
View File
File diff suppressed because it is too large Load Diff
experiments/SPY-options/data_explore.ipynb
-551
View File
File diff suppressed because one or more lines are too long
experiments/SPY-options/option_runner.py
-97
View File
@@ -1,97 +0,0 @@
import numpy as np
import datetime as dt
import matplotlib.pyplot as plt
import seaborn as sns
import torch
import gpytorch
import os
# import robin_stocks.robinhood as r
import pickle5 as pickle
import pandas as pd
sns.set_style("whitegrid")
sns.set_palette("bright")
sns.set(font_scale=2.0)
import sys
sys.path.append("../")
from voltron.likelihoods import VolatilityGaussianLikelihood
from voltron.models import SingleTaskVariationalGP as SingleTaskCopulaProcessModel
from voltron.kernels import BMKernel, VolatilityKernel
from voltron.models import BMGP, VoltronGP
from gpytorch.kernels import ScaleKernel, RBFKernel, MaternKernel
from voltron.option_utils import GetTradingDays, GetTrainingData, Pricer, FindLastTradingDays
from voltron.train_utils import LearnGPCV, TrainVolModel, TrainDataModel
def main():
years = [yr for yr in range(2006, 2018)]
logger = []
full_logger = []
SPY = pd.read_csv("./data/SPY_prices.csv")
SPY['Date'] = pd.to_datetime(SPY['Date'])
ntrain = 375
nvol = 100
npx = 100
for year in years:
options = pd.read_csv("./data/SPY_" + str(year) + ".csv")
options.expiration = pd.to_datetime(options.expiration)
options.quotedate = pd.to_datetime(options.quotedate)
qday = options.quotedate.unique()[0]
quote_price = SPY[SPY['Date']==qday].Close.item()
options = options[(options.quotedate == qday) & (options.type=='call')]
edays = options.expiration.sort_values().unique()
testdays = (edays - qday)/np.timedelta64(1, "D")
edays = edays[(testdays > 100) & (testdays < 365)]
lastdays = FindLastTradingDays(SPY, edays)
ntests = np.array([GetTradingDays(SPY, qday, pd.Timestamp(ld)) for ld in lastdays])
fulltest = ntests[-1]
train_y = torch.FloatTensor(GetTrainingData(SPY, qday, ntrain).to_numpy())
test_y = torch.FloatTensor(GetTrainingData(SPY,
pd.Timestamp(lastdays[-1]),
fulltest).to_numpy())
full_x = torch.arange(ntrain+fulltest).type(torch.FloatTensor)
full_x = full_x/252.
train_x = full_x[:ntrain]
dt = train_x[1]-train_x[0]
test_x = full_x[ntrain:]
## learn vol with GPCV ##
vol = LearnGPCV(train_x, train_y, train_iters=750)/(dt**0.5)
## train vol GP ##
vmod, vlh = TrainVolModel(train_x, vol, train_iters=750)
## train data gp ##
dmod, dlh = TrainDataModel(train_x, train_y, vmod, vlh, vol,
printing=False, train_iters=750)
## figure out how to price options sanely ##
px_samples = torch.zeros(npx*nvol, len(edays))
px_paths = torch.zeros(npx*nvol, fulltest)
vol_paths = torch.zeros(nvol, fulltest)
dmod.vol_model.eval();
dmod.eval();
for vidx in range(nvol):
# print(vidx)
vol_pred = dmod.vol_model(test_x).sample().exp()
vol_paths[vidx, :] = vol_pred.detach()
px_pred = dmod.GeneratePrediction(test_x, vol_pred, npx).exp()
px_paths[vidx*npx:(vidx*npx + npx), :] = px_pred.detach().T
px_samples[vidx*npx:(vidx*npx+npx), :] = px_pred[ntests-1].detach().T
option_output = Pricer(px_samples, options, edays, test_y[ntests-1],
quote_price)
option_output.to_pickle("./output/options" + str(year) + ".pkl")
print(str(year), "Done")
if __name__ == "__main__":
main()
experiments/SPY-options/sabr_runner.py
-110
View File
@@ -1,110 +0,0 @@
import numpy as np
import torch
import pandas as pd
import copy
from voltron.option_utils import GetTradingDays, GetTrainingData, Pricer, FindLastTradingDays
from scipy.optimize import minimize
def BlackVol(pars, K, f, T):
alpha = torch.exp(pars[0][0]) ## v0
rho = 2 * torch.sigmoid(pars[0][1]) - 1. ##rho
v = torch.exp(pars[0][2]) ## "sigma"
beta = 1.
num = 1 + (alpha**2 * (1-beta)**2/(24 * (f*K)**(1-beta)) +\
0.25 * rho*beta*v*alpha/((f*K)**(0.5*(1-beta))) +\
v**2*(2-3*rho**2)/24)*T
num*= alpha
denom = (f*K)**(0.5*(1-beta)) * (1 + (1-beta)**2/24 * torch.log(f/K)**2 +\
(1-beta)**4/1920 * torch.log(f/K)**4)
z = v/alpha * (f*K)**(0.5*(1-beta)) * np.log(f/K)
xi_z = torch.log((torch.sqrt(1 - 2 * rho * z + z**2) + z - rho)/(1-rho))
return num/denom * z/xi_z
def MinVol(pars, Ks, Fs, Ts, ivol):
return torch.mean((ivol - BlackVol(pars, Ks, Fs, Ts)).pow(2))
def Calibrate(Fs, Ks, Ts, ivol, iters=1000):
pars = [torch.tensor([-1., -5., -3.], requires_grad=True)]
opt = torch.optim.SGD(pars, lr=0.1)
stored_pars = torch.zeros(iters, 3)
losses = []
for e in range(iters):
stored_pars[e, :] = pars[0]
loss = MinVol(pars, Ks, Fs, Ts, ivol)
opt.zero_grad()
loss.backward()
losses.append(loss.item())
opt.step()
return pars[0].detach().numpy()
def SABRSim(Np, Nt, S0, V0, sigma, rho, dt=1./252.):
dW = np.random.randn(Nt+1, Np) * np.sqrt(dt)
dZ = rho * dW + np.sqrt(1-rho**2) * np.random.randn(Nt+1, Np) * np.sqrt(dt)
S = np.zeros((Nt+1, Np))
S[0] = S0
V = np.zeros((Nt+1, Np))
V[0] = V0
for t in range(Nt):
S[t+1] = S[t] + V[t]*S[t]*dW[t]
V[t+1] = V[t] + sigma*V[t]*dZ[t]
return S[1:]
def main():
years = [yr for yr in range(2006, 2018)]
logger = []
full_logger = []
SPY = pd.read_csv("./data/SPY_prices.csv")
SPY['Date'] = pd.to_datetime(SPY['Date'])
Np = 10000
for year in years:
options = pd.read_csv("./data/SPY_" + str(year) + ".csv")
options.expiration = pd.to_datetime(options.expiration)
options.quotedate = pd.to_datetime(options.quotedate)
qday = options.quotedate.unique()[0]
quote_price = SPY[SPY['Date']==qday].Close.item()
options = options[(options.quotedate == qday) & (options.type=='call')]
edays = options.expiration.sort_values().unique()
testdays = (edays - qday)/np.timedelta64(1, "D")
edays = edays[(testdays > 100) & (testdays < 365)]
lastdays = FindLastTradingDays(SPY, edays)
ntests = np.array([GetTradingDays(SPY, qday,
pd.Timestamp(ld)) for ld in lastdays])
fulltest = ntests[-1]
test_y = torch.FloatTensor(GetTrainingData(SPY,
pd.Timestamp(lastdays[-1]),
fulltest).to_numpy())
## extract data for calibration ##
ivol = torch.tensor(options.impliedvol.to_numpy())
Fs = torch.tensor(options.underlying_last.to_numpy())
Ks = torch.tensor(options.strike.to_numpy())
starts = options.quotedate.dt.date.to_numpy()
ends = options.expiration.dt.date.to_numpy()
Ts = torch.tensor(([np.busday_count(qd, ed)/252. for qd, ed in zip(starts, ends)]))
pars = Calibrate(Fs, Ks, Ts, ivol)
v0 = np.exp(pars[0])
1/(1 + np.exp(-pars[1]))
rho = (2/(1 + np.exp(-pars[1])) - 1.)
sigma = np.exp(pars[2])
px_paths = SABRSim(Np, fulltest, quote_price, v0, sigma, rho)
# px_samples = torch.tensor(px_paths[ntests-1])
option_output = Pricer(torch.tensor(px_paths), options, edays, test_y[ntests-1],
quote_price)
option_output.to_pickle("./output/sabr" + str(year) + ".pkl")
print(str(year), "Done")
if __name__ == "__main__":
main()
experiments/calibration/OutputTest.ipynb
-142
View File
File diff suppressed because one or more lines are too long
experiments/calibration/calib_grabber.ipynb
-588
View File
@@ -1,588 +0,0 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": 2,
"id": "8d4521d7-e61a-4820-b704-4908e9210ee7",
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"import matplotlib.pyplot as plt\n",
"from matplotlib.lines import Line2D\n",
"import pandas as pd\n",
"import torch\n",
"import gpytorch\n",
"from gpytorch.means import Mean\n",
"import seaborn as sns\n",
"import time\n",
"import copy\n",
"import sys\n",
"import os\n",
"\n",
"sns.set_style('white')\n",
"# style.use('whitegrid')\n",
"palette = [\"#1b4079\", \"#C6DDF0\", \"#048A81\", \"#B9E28C\", \"#8C2155\", \"#AF7595\", \"#E6480F\", \"#FA9500\"]\n",
"sns.set(palette = palette, font_scale=2.0, style=\"white\", rc={\"lines.linewidth\": 2.0})"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "3744a8e8-cb18-43ed-81c1-54aed694e292",
"metadata": {},
"outputs": [],
"source": [
"def LoadDat(dat, sym, start_idx, ntrain, ntest):\n",
" px = torch.FloatTensor(dat[dat.symbol == sym].close_price.to_numpy())\n",
" train_y = px[start_idx:ntrain+start_idx].squeeze()\n",
" test_y = px[start_idx + ntrain:start_idx + ntrain+ntest].squeeze()\n",
" return train_y, test_y\n",
"\n",
"def LoadSims(SPDR, sym, kernel, mean, k=100):\n",
" fpath = \"./saved-outputs/\" + SPDR + \"/\"\n",
" fname = sym + \"_\" + kernel + \"_\" + mean + str(k) + \".pt\"\n",
" \n",
" return torch.load(fpath + fname)"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "19d2a38f-0bb2-427f-bd7a-f9423b60e448",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"['XOM' 'CVX' 'EOG' 'COP' 'SLB']\n"
]
}
],
"source": [
"hypers = torch.load(\"./saved-outputs/metadata.pt\")\n",
"ntrain = hypers['ntrain']\n",
"ntest = hypers['ntest']\n",
"start_idxs = hypers['start_idxs']\n",
"\n",
"SPDR = \"XLE\"\n",
"dat = pd.read_pickle(dpath + SPDR + \".pkl\")\n",
"syms = dat.symbol.unique()\n",
"print(syms)\n",
"\n",
"train_x = torch.arange(ntrain) * 1./252\n",
"test_x = torch.arange(ntest) * 1./252 + train_x[-1] + train_x[1]\n",
"percentiles = np.linspace(0.05, 0.95, 19)"
]
},
{
"cell_type": "markdown",
"id": "4257adde-7106-429c-b418-973dc0345424",
"metadata": {},
"source": [
"## Examples"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "6ccf46b5-183d-400f-ace6-200a3d9d7844",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"['AAPL' 'MSFT' 'NVDA' 'V' 'PYPL']\n"
]
}
],
"source": [
"hypers = torch.load(\"./saved-outputs/metadata.pt\")\n",
"ntrain = hypers['ntrain']\n",
"ntest = hypers['ntest']\n",
"start_idxs = hypers['start_idxs']\n",
"\n",
"SPDR = \"XLK\"\n",
"dat = pd.read_pickle(dpath + SPDR + \".pkl\")\n",
"syms = dat.symbol.unique()\n",
"print(syms)\n",
"train_x = torch.arange(ntrain) * 1./252\n",
"test_x = torch.arange(ntest) * 1./252 + train_x[-1] + train_x[1]"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "9704d083-ce47-4c5e-a5e6-029fa1effbd7",
"metadata": {},
"outputs": [],
"source": [
"def ECDF(sample_pxs, true_px): \n",
" return (torch.sum(sample_pxs < true_px, 0)/sample_pxs.shape[0])\n",
" \n",
"def Calibration(pcts, percentile=0.95):\n",
" in_band = np.where((pcts < percentile))[0].shape[0]\n",
" return in_band/pcts.shape[0]\n",
"\n",
"def GetCalibration(kernel, mean, k=100, horizon=np.arange(75,100), logger=[], exp=True):\n",
" pcts = torch.zeros(len(syms), len(start_idxs), horizon.shape[0])\n",
" for sym_idx, sym in enumerate(syms):\n",
" \n",
" fpath = \"./saved-outputs/\" + SPDR + \"/\"\n",
" fname = sym + \"_\" + kernel + \"_\" + mean + str(k) + \".pt\"\n",
" if os.path.exists(fpath + fname):\n",
" for idx, start_idx in enumerate(start_idxs):\n",
" train_y, test_y = LoadDat(dat, sym, start_idx, ntrain, ntest)\n",
" preds = LoadSims(SPDR, sym, kernel, mean, k=k)[idx, :, horizon]\n",
" if exp:\n",
" preds = preds.exp()\n",
" \n",
" \n",
" pcts = pcts.flatten()\n",
" percentiles = np.linspace(0.05, 0.95, 19)\n",
" log_name = kernel\n",
" for pct in percentiles:\n",
" clb = Calibration(pcts, pct)\n",
" logger.append([clb, np.round(pct, 2), log_name, mean, k])\n",
" \n",
" return logger"
]
},
{
"cell_type": "code",
"execution_count": 20,
"id": "4ff19402-e7d7-49c0-890c-5b136265f3ff",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"['AMT' 'PLD' 'CCI' 'EQIX' 'PSA']\n"
]
}
],
"source": [
"SPDR = \"XLRE\"\n",
"dat = pd.read_pickle(dpath + SPDR + \".pkl\")\n",
"syms = dat.symbol.unique()\n",
"print(syms)\n",
"horizon = np.arange(75, 100)\n",
"train_x = torch.arange(ntrain) * 1./252\n",
"test_x = torch.arange(ntest) * 1./252 + train_x[-1] + train_x[1]\n",
"\n",
"logger = []\n",
"logger = GetCalibration('matern', 'ewma', 100, logger=logger, horizon=horizon)\n",
"logger = GetCalibration('matern', 'ewma', 200, logger=logger, horizon=horizon)\n",
"logger = GetCalibration('matern', 'ewma', 400, logger=logger, horizon=horizon)\n",
"logger = GetCalibration('matern', 'dewma', 100, logger=logger, horizon=horizon)\n",
"logger = GetCalibration('matern', 'dewma', 200, logger=logger, horizon=horizon)\n",
"logger = GetCalibration('matern', 'dewma', 400, logger=logger, horizon=horizon)\n",
"logger = GetCalibration('matern', 'tewma', 100, logger=logger, horizon=horizon)\n",
"logger = GetCalibration('matern', 'tewma', 200, logger=logger, horizon=horizon)\n",
"logger = GetCalibration('matern', 'tewma', 400, logger=logger, horizon=horizon)\n",
"logger = GetCalibration('matern', 'constant', 100, logger=logger, horizon=horizon, exp=False)\n",
"xlre_df = pd.DataFrame(logger)\n",
"xlre_df.columns = [\"Calibration\", \"Percentile\", \"Type\", 'Mean', \"k\"]"
]
},
{
"cell_type": "code",
"execution_count": 22,
"id": "49f482b5-af3b-4b77-b04d-82716d5b7afb",
"metadata": {},
"outputs": [],
"source": [
"pd.to_pickle(xlre_df, \"./new_matern_calib.pkl\")"
]
},
{
"cell_type": "markdown",
"id": "7b846b87-3132-42e1-9bff-f9d117176543",
"metadata": {},
"source": [
"## Get NLL"
]
},
{
"cell_type": "code",
"execution_count": 94,
"id": "63958d47-67f8-4298-94ab-3ff353b4f9ff",
"metadata": {},
"outputs": [],
"source": [
"def GetNLL(kernel, mean, k=100, horizon=np.arange(75,100), logger=[], exp=True):\n",
" N = 0\n",
" nll = 0.\n",
" for spdr_idx, spdr in enumerate(SPDRS):\n",
" dat = pd.read_pickle(dpath + spdr + \".pkl\")\n",
" syms = dat.symbol.unique()\n",
" for sym_idx, sym in enumerate(syms):\n",
" fpath = \"./saved-outputs/\" + spdr + \"/\"\n",
" fname = sym + \"_\" + kernel + \"_\" + mean + str(k) + \".pt\"\n",
" if os.path.exists(fpath + fname):\n",
" for idx, start_idx in enumerate(start_idxs):\n",
" train_y, test_y = LoadDat(dat, sym, start_idx, ntrain, ntest)\n",
" test_y = test_y[horizon]\n",
" preds = LoadSims(spdr, sym, kernel, mean, k=k)[idx, :, horizon]\n",
" if exp:\n",
" preds = preds.exp() \n",
" try:\n",
" nll -= torch.distributions.Normal(preds.mean(0), preds.std(0)).log_prob(test_y).sum()\n",
" N += test_y.numel()\n",
" except:\n",
" pass\n",
" # print(\"Failed:\", spdr, sym, idx)\n",
"\n",
" if N >= 0:\n",
" logger.append([nll.item(), N, kernel, mean, k])\n",
" return logger"
]
},
{
"cell_type": "code",
"execution_count": 105,
"id": "8bc987c8-982a-4f1c-bd40-5d444e58e7e0",
"metadata": {},
"outputs": [],
"source": [
"SPDRS = [\"XLRE\", \"XLY\", \"XLF\", \"XLE\", \"XLK\"]\n",
"horizon = np.arange(75, 100)\n",
"train_x = torch.arange(ntrain) * 1./252\n",
"test_x = torch.arange(ntest) * 1./252 + train_x[-1] + train_x[1]\n",
"\n",
"logger = []\n",
"logger = GetNLL('matern', 'ewma', 100, logger=logger, horizon=horizon)\n",
"logger = GetNLL('matern', 'ewma', 200, logger=logger, horizon=horizon)\n",
"logger = GetNLL('matern', 'ewma', 400, logger=logger, horizon=horizon)\n",
"logger = GetNLL('matern', 'dewma', 100, logger=logger, horizon=horizon)\n",
"logger = GetNLL('matern', 'dewma', 200, logger=logger, horizon=horizon)\n",
"logger = GetNLL('matern', 'dewma', 400, logger=logger, horizon=horizon)\n",
"logger = GetNLL('matern', 'tewma', 100, logger=logger, horizon=horizon)\n",
"logger = GetNLL('matern', 'tewma', 200, logger=logger, horizon=horizon)\n",
"logger = GetNLL('matern', 'tewma', 400, logger=logger, horizon=horizon)\n",
"logger = GetNLL('matern', 'constant', 100, logger=logger, horizon=horizon, exp=False)"
]
},
{
"cell_type": "code",
"execution_count": 106,
"id": "dee5af9f-d8d1-4576-bc42-eba43d846e8e",
"metadata": {},
"outputs": [],
"source": [
"df = pd.DataFrame(logger)\n",
"df.columns = [\"NLL\", \"N\", \"Kernel\", \"Mean\", \"K\"]"
]
},
{
"cell_type": "code",
"execution_count": 107,
"id": "84d27556-5456-433b-945a-8107613c7d0f",
"metadata": {},
"outputs": [],
"source": [
"df['Mean_NLL'] = df[\"NLL\"]/df['N']"
]
},
{
"cell_type": "code",
"execution_count": 108,
"id": "0d562366-4268-4bd2-a4ff-2e62797cbdb7",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>NLL</th>\n",
" <th>N</th>\n",
" <th>Kernel</th>\n",
" <th>Mean</th>\n",
" <th>K</th>\n",
" <th>Mean_NLL</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>6.726755e+04</td>\n",
" <td>4350</td>\n",
" <td>matern</td>\n",
" <td>ewma</td>\n",
" <td>100</td>\n",
" <td>15.463804</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>5.704342e+04</td>\n",
" <td>4200</td>\n",
" <td>matern</td>\n",
" <td>ewma</td>\n",
" <td>200</td>\n",
" <td>13.581766</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>6.180194e+04</td>\n",
" <td>6300</td>\n",
" <td>matern</td>\n",
" <td>ewma</td>\n",
" <td>400</td>\n",
" <td>9.809832</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>5.138177e+05</td>\n",
" <td>4800</td>\n",
" <td>matern</td>\n",
" <td>dewma</td>\n",
" <td>100</td>\n",
" <td>107.045358</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>1.109547e+05</td>\n",
" <td>4200</td>\n",
" <td>matern</td>\n",
" <td>dewma</td>\n",
" <td>200</td>\n",
" <td>26.417781</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>7.489109e+04</td>\n",
" <td>6450</td>\n",
" <td>matern</td>\n",
" <td>dewma</td>\n",
" <td>400</td>\n",
" <td>11.611021</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>3.585653e+06</td>\n",
" <td>5250</td>\n",
" <td>matern</td>\n",
" <td>tewma</td>\n",
" <td>100</td>\n",
" <td>682.981476</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>1.216625e+05</td>\n",
" <td>3900</td>\n",
" <td>matern</td>\n",
" <td>tewma</td>\n",
" <td>200</td>\n",
" <td>31.195507</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>7.496420e+04</td>\n",
" <td>6450</td>\n",
" <td>matern</td>\n",
" <td>tewma</td>\n",
" <td>400</td>\n",
" <td>11.622357</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>5.738497e+04</td>\n",
" <td>7500</td>\n",
" <td>matern</td>\n",
" <td>constant</td>\n",
" <td>100</td>\n",
" <td>7.651330</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" NLL N Kernel Mean K Mean_NLL\n",
"0 6.726755e+04 4350 matern ewma 100 15.463804\n",
"1 5.704342e+04 4200 matern ewma 200 13.581766\n",
"2 6.180194e+04 6300 matern ewma 400 9.809832\n",
"3 5.138177e+05 4800 matern dewma 100 107.045358\n",
"4 1.109547e+05 4200 matern dewma 200 26.417781\n",
"5 7.489109e+04 6450 matern dewma 400 11.611021\n",
"6 3.585653e+06 5250 matern tewma 100 682.981476\n",
"7 1.216625e+05 3900 matern tewma 200 31.195507\n",
"8 7.496420e+04 6450 matern tewma 400 11.622357\n",
"9 5.738497e+04 7500 matern constant 100 7.651330"
]
},
"execution_count": 108,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df"
]
},
{
"cell_type": "code",
"execution_count": 110,
"id": "733dcdd7-bc8a-4530-9ac5-eaa2e5f86085",
"metadata": {},
"outputs": [],
"source": [
"pd.to_pickle(df, \"./matern_nll.pkl\")"
]
},
{
"cell_type": "code",
"execution_count": 56,
"id": "330cbdba-bf55-4bf6-814a-09d10a53fd04",
"metadata": {},
"outputs": [],
"source": [
"mean = torch.tensor([1., 5.])\n",
"std = torch.tensor([1., 1.])\n",
"nrml = torch.distributions.Normal(mean, std)"
]
},
{
"cell_type": "code",
"execution_count": 57,
"id": "1582602c-8015-476a-b01a-a05d8aaf4a4a",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"tensor([0.3989, 0.3989])"
]
},
"execution_count": 57,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"nrml.log_prob(torch.tensor([1., 5.])).exp()"
]
},
{
"cell_type": "code",
"execution_count": 68,
"id": "0f47d060-7f64-4537-b18f-7c5772df4bea",
"metadata": {},
"outputs": [],
"source": [
"preds, y = GetNLL('matern', 'tewma', 400, logger=logger, horizon=horizon)"
]
},
{
"cell_type": "code",
"execution_count": 69,
"id": "e2403da1-c3b7-4d9d-8ae2-c00babfcfdcd",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"torch.Size([1000, 25])"
]
},
"execution_count": 69,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"preds.shape"
]
},
{
"cell_type": "code",
"execution_count": 73,
"id": "ff68035a-39a5-44e1-bf6a-ecd9b0837bc9",
"metadata": {},
"outputs": [],
"source": [
"nll = 0."
]
},
{
"cell_type": "code",
"execution_count": 78,
"id": "012b1b96-1afb-4f0a-8ab8-f96e92beacaf",
"metadata": {},
"outputs": [],
"source": [
"nll -= torch.distributions.Normal(preds.mean(0), preds.std(0)).log_prob(y).sum()"
]
},
{
"cell_type": "code",
"execution_count": 79,
"id": "5a9241b1-a686-40fe-a378-cc219333f9bb",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"tensor(242.3718)"
]
},
"execution_count": 79,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"nll"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "5cc3b61d-a2eb-4a33-9dfa-1049c9992345",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.8"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
experiments/calibration/lstm_scratch.ipynb
-408
View File
File diff suppressed because one or more lines are too long
experiments/calibration/nll.ipynb
-854
View File
@@ -1,854 +0,0 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": 2,
"id": "92e5b691-e01a-438a-b2b9-543c5fa3f8b2",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Warning no robinhood utils.\n"
]
}
],
"source": [
"import pickle as pkl\n",
"import pandas as pd\n",
"import numpy as np\n",
"import torch\n",
"import matplotlib.pyplot as plt\n",
"import seaborn as sns\n",
"import os\n",
"from voltron.data import make_ticker_list, GetStockHistory\n",
"\n",
"sns.set_style('white')\n",
"palette = [\"#1b4079\", \"#C6DDF0\", \"#048A81\", \"#B9E28C\", \"#8C2155\", \"#AF7595\", \"#E6480F\", \"#FA9500\"]\n",
"sns.set(palette = palette, font_scale=2.0, style=\"white\", rc={\"lines.linewidth\": 4.0})"
]
},
{
"cell_type": "code",
"execution_count": 30,
"id": "1f59e48e-eb80-48b2-8ce5-91596ad136f2",
"metadata": {},
"outputs": [],
"source": [
"def ECDF(sample_pxs, true_px): \n",
" return (torch.sum(sample_pxs < true_px, 0)/sample_pxs.shape[0])\n",
" \n",
"def Calibration(pcts, percentile=0.95):\n",
" in_band = np.where((pcts < percentile))[0].shape[0]\n",
" return in_band/pcts.shape[0]\n",
"\n",
"def GetNLL(model, mean='ewma', k=100, horizon=np.arange(75,100), \n",
" logger=[], exp=True, fdir=\"./saved-outputs/\"):\n",
" \n",
"\n",
" ntrain = 400\n",
" n_test_times = 20\n",
" ntest = 100\n",
" nll = 0.\n",
" N = 0\n",
" nlls = torch.tensor([])\n",
" for tckr in ticker_list:\n",
" data = None\n",
" try:\n",
" data = GetStockHistory(tckr, history=1000, end_date=end_date)\n",
" except:\n",
" print(\"failed\", tckr)\n",
" \n",
" if data is not None:\n",
"\n",
" for idx, date in enumerate(data.index):\n",
" fpath = fdir + tckr + \"/\"\n",
" fname = model + \"_\"\n",
" if model in ['volt', 'matern', 'sm']:\n",
" fname += mean + str(k) + \"_\"\n",
"\n",
" fname += str(date.date()) + \".pt\"\n",
"# print(fpath + fname)\n",
" if os.path.exists(fpath + fname): \n",
" preds = torch.load(fpath + fname) \n",
" if isinstance(preds, tuple):\n",
" preds = preds[0]\n",
" if preds.shape[-1] == 100:\n",
" preds = preds[:, horizon]\n",
"\n",
" test_y = torch.tensor(data.iloc[idx:idx+100].Close.to_numpy())\n",
" if test_y.shape[0] == 100:\n",
" if exp:\n",
" preds = preds.exp()\n",
"\n",
" try:\n",
" curr = torch.distributions.Normal(preds.mean(0), preds.std(0)).log_prob(test_y[horizon])\n",
" if curr.mean().abs() < 500:\n",
" nlls = torch.cat((curr, nlls))\n",
" except:\n",
" pass\n",
"\n",
"\n",
" if nlls.numel() > 0:\n",
" logger.append([-nlls.sum().item(), -nlls.mean().item(), nlls.std().item(), model, mean, k])\n",
" \n",
" return logger"
]
},
{
"cell_type": "code",
"execution_count": 25,
"id": "5fe6b4d6-b49e-4e06-8874-9be43b4ee4bd",
"metadata": {},
"outputs": [],
"source": [
"data_path = \"../../voltron/data/\"\n",
"ticker_list = make_ticker_list(data_path + \"nasdaq100.txt\")"
]
},
{
"cell_type": "code",
"execution_count": 27,
"id": "d9ed9129-be2d-47e8-9866-5a708c27d225",
"metadata": {
"collapsed": true,
"jupyter": {
"outputs_hidden": true
},
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"1 Failed download:\n",
"- ALXN: No data found, symbol may be delisted\n",
"failed ALXN\n",
"failed CA\n",
"\n",
"1 Failed download:\n",
"- CELG: No data found, symbol may be delisted\n",
"failed CELG\n",
"\n",
"1 Failed download:\n",
"- CTRP: No data found, symbol may be delisted\n",
"failed CTRP\n",
"failed ESRX\n",
"\n",
"1 Failed download:\n",
"- LVNTA: No data found for this date range, symbol may be delisted\n",
"failed LVNTA\n",
"\n",
"1 Failed download:\n",
"- QVCA: No data found for this date range, symbol may be delisted\n",
"failed QVCA\n",
"\n",
"1 Failed download:\n",
"- LMCA: No data found for this date range, symbol may be delisted\n",
"failed LMCA\n",
"\n",
"1 Failed download:\n",
"- LMCK: No data found, symbol may be delisted\n",
"failed LMCK\n",
"\n",
"1 Failed download:\n",
"- LLTC: No data found for this date range, symbol may be delisted\n",
"failed LLTC\n",
"\n",
"1 Failed download:\n",
"- MXIM: No data found, symbol may be delisted\n",
"failed MXIM\n",
"\n",
"1 Failed download:\n",
"- MYL: No data found, symbol may be delisted\n",
"failed MYL\n",
"\n",
"1 Failed download:\n",
"- SYMC: No data found, symbol may be delisted\n",
"failed SYMC\n",
"\n",
"1 Failed download:\n",
"- PCLN: No data found for this date range, symbol may be delisted\n",
"failed PCLN\n",
"\n",
"1 Failed download:\n",
"- VIAB: No data found, symbol may be delisted\n",
"failed VIAB\n",
"\n",
"1 Failed download:\n",
"- WFM: No data found for this date range, symbol may be delisted\n",
"failed WFM\n",
"\n",
"1 Failed download:\n",
"- YHOO: No data found for this date range, symbol may be delisted\n",
"failed YHOO\n"
]
}
],
"source": [
"log = []\n",
"end_date = \"2022-01-20\"\n",
"for k in [400]:\n",
" for mean in ['ewma']:\n",
" log = GetNLL('volt', mean=mean, k=k, horizon=np.arange(75,100), \n",
" logger=log, exp=True, fdir=\"../trading/saved-outputs/\")\n",
" \n",
"# log = GetNLL('matern', mean='constant', k=100, horizon=np.arange(75,100), \n",
"# logger=log, exp=True)"
]
},
{
"cell_type": "code",
"execution_count": 31,
"id": "c00697ab-5884-4c70-9843-1bb49a52716f",
"metadata": {
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"1 Failed download:\n",
"- ALXN: No data found, symbol may be delisted\n",
"failed ALXN\n",
"failed CA\n",
"\n",
"1 Failed download:\n",
"- CELG: No data found, symbol may be delisted\n",
"failed CELG\n",
"\n",
"1 Failed download:\n",
"- CTRP: No data found, symbol may be delisted\n",
"failed CTRP\n",
"failed ESRX\n",
"\n",
"1 Failed download:\n",
"- LVNTA: No data found for this date range, symbol may be delisted\n",
"failed LVNTA\n",
"\n",
"1 Failed download:\n",
"- QVCA: No data found for this date range, symbol may be delisted\n",
"failed QVCA\n",
"\n",
"1 Failed download:\n",
"- LMCA: No data found for this date range, symbol may be delisted\n",
"failed LMCA\n",
"\n",
"1 Failed download:\n",
"- LMCK: No data found, symbol may be delisted\n",
"failed LMCK\n",
"\n",
"1 Failed download:\n",
"- LLTC: No data found for this date range, symbol may be delisted\n",
"failed LLTC\n",
"\n",
"1 Failed download:\n",
"- MXIM: No data found, symbol may be delisted\n",
"failed MXIM\n",
"\n",
"1 Failed download:\n",
"- MYL: No data found, symbol may be delisted\n",
"failed MYL\n",
"\n",
"1 Failed download:\n",
"- SYMC: No data found, symbol may be delisted\n",
"failed SYMC\n",
"\n",
"1 Failed download:\n",
"- PCLN: No data found for this date range, symbol may be delisted\n",
"failed PCLN\n",
"\n",
"1 Failed download:\n",
"- VIAB: No data found, symbol may be delisted\n",
"failed VIAB\n",
"\n",
"1 Failed download:\n",
"- WFM: No data found for this date range, symbol may be delisted\n",
"failed WFM\n",
"\n",
"1 Failed download:\n",
"- YHOO: No data found for this date range, symbol may be delisted\n",
"failed YHOO\n",
"\n",
"1 Failed download:\n",
"- ALXN: No data found, symbol may be delisted\n",
"failed ALXN\n",
"failed CA\n",
"\n",
"1 Failed download:\n",
"- CELG: No data found, symbol may be delisted\n",
"failed CELG\n",
"\n",
"1 Failed download:\n",
"- CTRP: No data found, symbol may be delisted\n",
"failed CTRP\n",
"failed ESRX\n",
"\n",
"1 Failed download:\n",
"- LVNTA: No data found for this date range, symbol may be delisted\n",
"failed LVNTA\n",
"\n",
"1 Failed download:\n",
"- QVCA: No data found for this date range, symbol may be delisted\n",
"failed QVCA\n",
"\n",
"1 Failed download:\n",
"- LMCA: No data found for this date range, symbol may be delisted\n",
"failed LMCA\n",
"\n",
"1 Failed download:\n",
"- LMCK: No data found, symbol may be delisted\n",
"failed LMCK\n",
"\n",
"1 Failed download:\n",
"- LLTC: No data found for this date range, symbol may be delisted\n",
"failed LLTC\n",
"\n",
"1 Failed download:\n",
"- MXIM: No data found, symbol may be delisted\n",
"failed MXIM\n",
"\n",
"1 Failed download:\n",
"- MYL: No data found, symbol may be delisted\n",
"failed MYL\n",
"\n",
"1 Failed download:\n",
"- SYMC: No data found, symbol may be delisted\n",
"failed SYMC\n",
"\n",
"1 Failed download:\n",
"- PCLN: No data found for this date range, symbol may be delisted\n",
"failed PCLN\n",
"\n",
"1 Failed download:\n",
"- VIAB: No data found, symbol may be delisted\n",
"failed VIAB\n",
"\n",
"1 Failed download:\n",
"- WFM: No data found for this date range, symbol may be delisted\n",
"failed WFM\n",
"\n",
"1 Failed download:\n",
"- YHOO: No data found for this date range, symbol may be delisted\n",
"failed YHOO\n"
]
}
],
"source": [
"log = []\n",
"end_date = \"2022-01-20\"\n",
"log = GetNLL('sm', mean='constant', k=100, horizon=np.arange(75,100), \n",
" logger=[], exp=True)\n",
"log = GetNLL('sm', mean='ewma', k=400, horizon=np.arange(75,100), \n",
" logger=log, exp=True)\n"
]
},
{
"cell_type": "code",
"execution_count": 32,
"id": "89054cf9-fb1c-45a1-aa8b-8884713f4c07",
"metadata": {},
"outputs": [],
"source": [
"df = pd.DataFrame(log)\n",
"df.columns = [\"Mean_NLL\", \"NLL\", \"Std_NLL\", \"Model\", \"Mean\", \"k\"]"
]
},
{
"cell_type": "code",
"execution_count": 33,
"id": "95915269-4bbf-4949-a944-37e076db8889",
"metadata": {},
"outputs": [],
"source": [
"pd.to_pickle(df, \"./sm_nll.pkl\")"
]
},
{
"cell_type": "code",
"execution_count": 34,
"id": "e2a5b10f-61a9-4876-a53c-385380955e28",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Mean_NLL</th>\n",
" <th>NLL</th>\n",
" <th>Std_NLL</th>\n",
" <th>Model</th>\n",
" <th>Mean</th>\n",
" <th>k</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>3.201143e+06</td>\n",
" <td>80.430728</td>\n",
" <td>113.825740</td>\n",
" <td>sm</td>\n",
" <td>constant</td>\n",
" <td>100</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>1.134694e+06</td>\n",
" <td>147.842929</td>\n",
" <td>161.222031</td>\n",
" <td>sm</td>\n",
" <td>ewma</td>\n",
" <td>400</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" Mean_NLL NLL Std_NLL Model Mean k\n",
"0 3.201143e+06 80.430728 113.825740 sm constant 100\n",
"1 1.134694e+06 147.842929 161.222031 sm ewma 400"
]
},
"execution_count": 34,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df"
]
},
{
"cell_type": "markdown",
"id": "eb9db52e-d56a-47a8-86f5-05e43437c635",
"metadata": {},
"source": [
"## NLL Plotter"
]
},
{
"cell_type": "code",
"execution_count": 48,
"id": "38b65cc2-ed27-41a6-ae83-5396a40b2e0d",
"metadata": {},
"outputs": [],
"source": [
"nll = pd.read_pickle(\"./volt_matern_const_nll.pkl\")\n",
"nll = pd.concat((nll, pd.read_pickle(\"volt_const_nll.pkl\")))\n",
"mat_nll = pd.read_pickle(\"matern_nll.pkl\")\n",
"mat_nll[mat_nll[\"Mean\"] != 'constant']\n",
"mat_nll.columns = nll.columns\n",
"nll = pd.concat((mat_nll, nll))"
]
},
{
"cell_type": "code",
"execution_count": 49,
"id": "7952ac2d-0036-4ce4-bcd9-f7b1cfe1ca68",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Mean_NLL</th>\n",
" <th>NLL</th>\n",
" <th>Std_NLL</th>\n",
" <th>Model</th>\n",
" <th>Mean</th>\n",
" <th>k</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>6.726756e+04</td>\n",
" <td>15.463807</td>\n",
" <td>82.817116</td>\n",
" <td>matern</td>\n",
" <td>ewma</td>\n",
" <td>100</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>5.704343e+04</td>\n",
" <td>13.581769</td>\n",
" <td>62.178253</td>\n",
" <td>matern</td>\n",
" <td>ewma</td>\n",
" <td>200</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>6.180195e+04</td>\n",
" <td>9.809834</td>\n",
" <td>21.427580</td>\n",
" <td>matern</td>\n",
" <td>ewma</td>\n",
" <td>400</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>5.138178e+05</td>\n",
" <td>107.045380</td>\n",
" <td>558.429443</td>\n",
" <td>matern</td>\n",
" <td>dewma</td>\n",
" <td>100</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>1.109547e+05</td>\n",
" <td>26.417788</td>\n",
" <td>160.354996</td>\n",
" <td>matern</td>\n",
" <td>dewma</td>\n",
" <td>200</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>7.489107e+04</td>\n",
" <td>11.611018</td>\n",
" <td>42.626156</td>\n",
" <td>matern</td>\n",
" <td>dewma</td>\n",
" <td>400</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>3.585652e+06</td>\n",
" <td>682.981384</td>\n",
" <td>2605.378174</td>\n",
" <td>matern</td>\n",
" <td>tewma</td>\n",
" <td>100</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>1.216624e+05</td>\n",
" <td>31.195498</td>\n",
" <td>169.649368</td>\n",
" <td>matern</td>\n",
" <td>tewma</td>\n",
" <td>200</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>7.496416e+04</td>\n",
" <td>11.622351</td>\n",
" <td>42.427406</td>\n",
" <td>matern</td>\n",
" <td>tewma</td>\n",
" <td>400</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>4.061322e+03</td>\n",
" <td>7.735851</td>\n",
" <td>4.734124</td>\n",
" <td>matern</td>\n",
" <td>constant</td>\n",
" <td>100</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>1.173272e+03</td>\n",
" <td>4.693086</td>\n",
" <td>0.389815</td>\n",
" <td>volt</td>\n",
" <td>constant</td>\n",
" <td>100</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" Mean_NLL NLL Std_NLL Model Mean k\n",
"0 6.726756e+04 15.463807 82.817116 matern ewma 100\n",
"1 5.704343e+04 13.581769 62.178253 matern ewma 200\n",
"2 6.180195e+04 9.809834 21.427580 matern ewma 400\n",
"3 5.138178e+05 107.045380 558.429443 matern dewma 100\n",
"4 1.109547e+05 26.417788 160.354996 matern dewma 200\n",
"5 7.489107e+04 11.611018 42.626156 matern dewma 400\n",
"6 3.585652e+06 682.981384 2605.378174 matern tewma 100\n",
"7 1.216624e+05 31.195498 169.649368 matern tewma 200\n",
"8 7.496416e+04 11.622351 42.427406 matern tewma 400\n",
"0 4.061322e+03 7.735851 4.734124 matern constant 100\n",
"0 1.173272e+03 4.693086 0.389815 volt constant 100"
]
},
"execution_count": 49,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"nll"
]
},
{
"cell_type": "code",
"execution_count": 52,
"id": "76d6abf1-8385-4450-9c0e-4e92f897bff8",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/home/greg_b/miniconda3/envs/rpp/lib/python3.8/site-packages/pandas/core/indexing.py:1720: SettingWithCopyWarning: \n",
"A value is trying to be set on a copy of a slice from a DataFrame.\n",
"Try using .loc[row_indexer,col_indexer] = value instead\n",
"\n",
"See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
" self._setitem_single_column(loc, value, pi)\n"
]
}
],
"source": [
"temp_df = nll[(nll['Mean'].isin(['constant', 'ewma']))]\n",
"temp_df.loc[(temp_df['Mean']=='constant') & (temp_df['Model']=='volt'), 'k'] = 400\n",
"temp_df.loc[(temp_df['Mean']=='constant') & (temp_df['Model']=='matern'), 'k'] = 400\n",
"\n",
"temp_df = temp_df[temp_df['k']==400]"
]
},
{
"cell_type": "code",
"execution_count": 53,
"id": "6582a178-a65f-4f24-a20e-8b8b17e56e5a",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Mean_NLL</th>\n",
" <th>NLL</th>\n",
" <th>Std_NLL</th>\n",
" <th>Model</th>\n",
" <th>Mean</th>\n",
" <th>k</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>61801.953125</td>\n",
" <td>9.809834</td>\n",
" <td>21.427580</td>\n",
" <td>matern</td>\n",
" <td>ewma</td>\n",
" <td>400</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>4061.321999</td>\n",
" <td>7.735851</td>\n",
" <td>4.734124</td>\n",
" <td>matern</td>\n",
" <td>constant</td>\n",
" <td>400</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>1173.271529</td>\n",
" <td>4.693086</td>\n",
" <td>0.389815</td>\n",
" <td>volt</td>\n",
" <td>constant</td>\n",
" <td>400</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" Mean_NLL NLL Std_NLL Model Mean k\n",
"2 61801.953125 9.809834 21.427580 matern ewma 400\n",
"0 4061.321999 7.735851 4.734124 matern constant 400\n",
"0 1173.271529 4.693086 0.389815 volt constant 400"
]
},
"execution_count": 53,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"temp_df"
]
},
{
"cell_type": "code",
"execution_count": 51,
"id": "20d9682b-5e98-4793-8a65-42532cc476d9",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Mean_NLL</th>\n",
" <th>NLL</th>\n",
" <th>Std_NLL</th>\n",
" <th>Model</th>\n",
" <th>Mean</th>\n",
" <th>k</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>61801.953125</td>\n",
" <td>9.809834</td>\n",
" <td>21.427580</td>\n",
" <td>matern</td>\n",
" <td>ewma</td>\n",
" <td>400</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>4061.321999</td>\n",
" <td>7.735851</td>\n",
" <td>4.734124</td>\n",
" <td>matern</td>\n",
" <td>constant</td>\n",
" <td>400</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>1173.271529</td>\n",
" <td>4.693086</td>\n",
" <td>0.389815</td>\n",
" <td>volt</td>\n",
" <td>constant</td>\n",
" <td>400</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" Mean_NLL NLL Std_NLL Model Mean k\n",
"2 61801.953125 9.809834 21.427580 matern ewma 400\n",
"0 4061.321999 7.735851 4.734124 matern constant 400\n",
"0 1173.271529 4.693086 0.389815 volt constant 400"
]
},
"execution_count": 51,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"temp_df"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "2cb0ed8b-e4d7-4b78-b676-96d7123ad643",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.8"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
experiments/calibration/plotter.ipynb
-339
View File
File diff suppressed because one or more lines are too long
experiments/calibration/scratch.ipynb
-722
View File
File diff suppressed because one or more lines are too long
experiments/exchange-rate/exchange.ipynb
-494
View File
File diff suppressed because one or more lines are too long
experiments/exchange-rate/exchange_vol_toy.ipynb
-221
View File
File diff suppressed because one or more lines are too long
experiments/mt-SPY/.DS_Store
Vendored
BIN
View File
Binary file not shown.
experiments/mt-SPY/finance_plotting.ipynb
-271
View File
File diff suppressed because one or more lines are too long
experiments/mt-SPY/finance_plotting_v2.ipynb
-565
View File
File diff suppressed because one or more lines are too long
experiments/mt-SPY/finance_sector_mt.ipynb
-866
View File
File diff suppressed because one or more lines are too long
experiments/mt-SPY/first_pass_svgp_multitask.ipynb
-1177
View File
File diff suppressed because one or more lines are too long
experiments/mt-SPY/joint_sabr_retry.ipynb
-1177
View File
File diff suppressed because one or more lines are too long
experiments/mt-SPY/mt_quantile_plotting.ipynb
-350
View File
File diff suppressed because one or more lines are too long
experiments/mt-SPY/single_task_fitting.ipynb
-805
View File
File diff suppressed because one or more lines are too long
experiments/mt-SPY/single_task_fitting_finance.ipynb
-4884
View File
File diff suppressed because it is too large Load Diff
experiments/mt-SPY/svgp_mt_clustering.ipynb
-1547
View File
File diff suppressed because one or more lines are too long
experiments/px-forecasting/Trainers.py
-232
View File
@@ -1,232 +0,0 @@
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import torch
import gpytorch
# from voltron.robinhood_utils import GetStockData
import os
# import robin_stocks.robinhood as r
import pickle5 as pickle
sns.set_style("whitegrid")
sns.set_palette("bright")
sns.set(font_scale=2.0)
sns.set_style('whitegrid')
import sys
sys.path.append("../")
from voltron.likelihoods import VolatilityGaussianLikelihood
from voltron.models import SingleTaskVariationalGP as SingleTaskCopulaProcessModel
from voltron.kernels import BMKernel, VolatilityKernel
from voltron.models import BMGP, VoltronGP, MaternGP, SMGP
from voltron.means import LogLinearMean
from gpytorch.kernels import ScaleKernel, RBFKernel, MaternKernel
def get_and_fit_gpcv(x, log_returns, printing=False):
train_x = x[:-1]
dt = train_x[1]-train_x[0]
# prepare model
likelihood = VolatilityGaussianLikelihood(param="exp")
# likelihood.raw_a.data -= 6.
covar_module = BMKernel()
model = SingleTaskCopulaProcessModel(
init_points=train_x.view(-1,1),
likelihood=likelihood,
use_piv_chol_init=False,
mean_module = gpytorch.means.ConstantMean(),
covar_module=covar_module,
learn_inducing_locations=False
)
model.mean_module.constant.data -= 4.
# model.initialize_variational_parameters(likelihood, train_x, y=log_returns)
import os
smoke_test = ('CI' in os.environ)
training_iterations = 2 if smoke_test else 500
# Find optimal model hyperparameters
model.train()
likelihood.train()
# Use the adam optimizer
# likelihood parameters should be taken acct of in the model
optimizer = torch.optim.Adam([
{"params": model.parameters()},
# {"params": likelihood.parameters(), "lr": 0.1}
], lr=0.01)
# "Loss" for GPs - the marginal log likelihood
# num_data refers to the number of training datapoints
mll = gpytorch.mlls.VariationalELBO(likelihood, model, log_returns.numel())
old_loss = 10000.
print_every = 50
for i in range(training_iterations):
# Zero backpropped gradients from previous iteration
optimizer.zero_grad()
# Get predictive output
with gpytorch.settings.num_gauss_hermite_locs(75):
output = model(train_x)
# Calc loss and backprop gradients
loss = -mll(output, log_returns)
loss.backward()
if printing:
if i % print_every == 0:
print('Iter %d/%d - Loss: %.3f' % (i + 1,
training_iterations,
loss.item()))
optimizer.step()
if old_loss <= loss and i > 100:
if printing:
print(old_loss, loss)
break
else:
old_loss = loss.item()
model.eval();
likelihood.eval();
predictive = model(x)
pred_scale = likelihood(predictive).scale.mean(0).detach()
samples = likelihood(predictive).scale.detach()
# plt.plot(x, pred_scale, linewidth = 4)
# plt.plot(x, samples.t(), color = "gray", alpha = 0.3)
# # plt.ylim((0, 0.25))
# plt.show()
# return scaled volatility prediction
return pred_scale / dt**0.5
def get_and_fit_vol_model(train_x, est_vol):
vol_lh = gpytorch.likelihoods.GaussianLikelihood()
vol_lh.noise.data = torch.tensor([1e-6])
vol_model = BMGP(train_x, est_vol.log(), vol_lh)
optimizer = torch.optim.Adam([
{'params': vol_model.parameters()}, # Includes GaussianLikelihood parameters
], lr=0.01)
# "Loss" for GPs - the marginal log likelihood
mll = gpytorch.mlls.ExactMarginalLogLikelihood(vol_lh, vol_model)
old_loss = 10000
for i in range(500):
# Zero gradients from previous iteration
optimizer.zero_grad()
# Output from model
output = vol_model(train_x)
# Calc loss and backprop gradients
loss = -mll(output, est_vol.log())
loss.backward()
if i % 50 == 0:
print(loss.item())
optimizer.step()
# if old_loss <= loss:
# break
# else:
# old_loss = loss.item()
return vol_model
def get_and_fit_data_model(train_x, train_y, pred_vol, vol_model):
voltron_lh = gpytorch.likelihoods.GaussianLikelihood()
voltron = VoltronGP(train_x, train_y.log(), voltron_lh, pred_vol)
# voltron.mean_module = gpytorch.means.LinearMean(1)
voltron.mean_module = LogLinearMean(1)
voltron.mean_module.initialize_from_data(train_x, train_y.log())
voltron.likelihood.raw_noise.data = torch.tensor([1e-6])
voltron.vol_lh = vol_model.likelihood
voltron.vol_model = vol_model
grad_flags = [False, True, True, True, False, False, False]
for idx, p in enumerate(voltron.parameters()):
p.requires_grad = grad_flags[idx]
voltron.train();
voltron_lh.train();
voltron.vol_lh.train();
voltron.vol_model.train();
# Use the adam optimizer
optimizer = torch.optim.Adam([
{'params': voltron.parameters()}, # Includes GaussianLikelihood parameters
], lr=0.1)
# "Loss" for GPs - the marginal log likelihood
mll = gpytorch.mlls.ExactMarginalLogLikelihood(voltron_lh, voltron)
for i in range(500):
# Zero gradients from previous iteration
optimizer.zero_grad()
# Output from model
output = voltron(train_x)
# Calc loss and backprop gradients
loss = -mll(output, train_y.log())
loss.backward()
# print(loss.item())
optimizer.step()
return voltron
def predict_prices(test_x, voltron, nvol=10, npx=10):
ntest = test_x.shape[0]
vol_paths = torch.zeros(nvol, ntest)
px_paths = torch.zeros(npx*nvol, ntest)
voltron.vol_model.eval();
voltron.eval();
for vidx in range(nvol):
vol_pred = voltron.vol_model(test_x).sample().exp()
vol_paths[vidx, :] = vol_pred.detach()
px_pred = voltron.GeneratePrediction(test_x, vol_pred, npx).exp()
px_paths[vidx*npx:(vidx*npx+npx), :] = px_pred.detach().T
return px_paths
def get_and_fit_basic_model(train_x, train_y, cov="matern", mean="loglinear"):
voltron_lh = gpytorch.likelihoods.GaussianLikelihood()
# voltron = VoltronGP(train_x, train_y.log(), voltron_lh, pred_vol)
if cov == "matern":
model = MaternGP(train_x,
train_y.log(), likelihood=voltron_lh)
else:
model = SMGP(train_x,
train_y.log(), likelihood=voltron_lh)
if mean == "loglinear":
model.mean_module = LogLinearMean(1)
model.mean_module.initialize_from_data(train_x, train_y.log())
else:
model.mean_module = gpytorch.means.ConstantMean()
model.likelihood.raw_noise.data = torch.tensor([1e-6])
# Use the adam optimizer
optimizer = torch.optim.Adam([
{'params': model.parameters()}, # Includes GaussianLikelihood parameters
], lr=0.1)
# "Loss" for GPs - the marginal log likelihood
mll = gpytorch.mlls.ExactMarginalLogLikelihood(voltron_lh, model)
for i in range(500):
# Zero gradients from previous iteration
optimizer.zero_grad()
# Output from model
output = model(train_x)
# Calc loss and backprop gradients
loss = -mll(output, train_y.log())
loss.backward()
# print(loss.item())
optimizer.step()
return model, voltron_lh
def predict_basic_prices(test_x, voltron, voltron_lh, npath=1000):
ntest = test_x.shape[0]
voltron.eval();
mod = voltron_lh(voltron(test_x))
px_paths = mod.sample(torch.Size(((npath),))).exp().squeeze(-1)
return px_paths
experiments/px-forecasting/basic_price_prediction.py
-71
View File
@@ -1,71 +0,0 @@
import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np
import torch
import pandas as pd
import gpytorch
from torch.nn.functional import softplus
from voltron.kernels import BMKernel, VolatilityKernel
from voltron.models import BMGP, VoltronGP
import argparse
from torch.distributions import Beta
from scipy.special import betainc
from Trainers import *
def main(args):
full_data = pd.read_pickle("../../spdr-data/" + args.SPDR + ".pkl")
tckrs = full_data.symbol.unique()
for tckr in tckrs:
data = full_data[full_data["symbol"] == tckr]
ts = torch.linspace(0, data.shape[0]/252., data.shape[0])
# train_x = ts[:ntrain]
# test_x = ts[ntrain:(ntrain+ntest)]
y = torch.FloatTensor(data['close_price'].to_numpy())
log_returns = torch.log(y[1:]) - torch.log(y[:-1])
dt = ts[1] - ts[0]
eval_times = list(range(100, ts.shape[0], 100)) #+ [ts.shape[0]]
prob_of_increases = []
for i, time in enumerate(eval_times):
print("now running time: ", time)
with gpytorch.settings.max_cholesky_size(2000):
data_model, data_lh = get_and_fit_basic_model(ts[:time], y[:time],
cov=args.kernel, mean=args.mean)
end_ind = -1 if i + 1 >= len(eval_times) else eval_times[i+1]
paths = predict_basic_prices(ts[time:end_ind], data_model,
data_lh).detach()
# now we predict the probability of increase at time i + 1
prob_of_increase = (paths[..., -1] > y[time]).sum() / paths.shape[-2]
print("prob of stock increase: ", prob_of_increase.detach())
prob_of_increases.append(prob_of_increase.detach())
torch.save(obj=prob_of_increases, f="./outputs/" + args.kernel + "_" + tckr + ".pt")
print(tckr, "Done")
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument(
"--SPDR",
type=str,
default="XLE",
)
parser.add_argument(
"--kernel",
type=str,
default="matern",
)
parser.add_argument(
"--mean",
type=str,
default="loglinear",
)
args = parser.parse_args()
main(args)
experiments/px-forecasting/data/matern_v.pt
BIN
View File
Binary file not shown.
experiments/px-forecasting/data/specmix_v.pt
BIN
View File
Binary file not shown.
experiments/px-forecasting/data/voltron_v.pt
BIN
View File
Binary file not shown.
experiments/px-forecasting/first_pass_svgp_predictions_working.ipynb
-591
View File
File diff suppressed because one or more lines are too long
experiments/px-forecasting/latex-gen.ipynb
-270
View File
@@ -1,270 +0,0 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": 15,
"id": "ddc44d8f",
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"import glob"
]
},
{
"cell_type": "code",
"execution_count": 29,
"id": "79dbbbf6",
"metadata": {},
"outputs": [],
"source": [
"start = r\"\"\"\n",
"\\begin{figure}\n",
" \\centering\n",
" \\includegraphics[width=\\linewidth]{figs/\"\"\"\n",
"\n",
"mid = r\"\"\"}\n",
" \\caption{Trading Strategy for \"\"\"\n",
"end = \"\"\".}\n",
"\\end{figure}\n",
"\"\"\""
]
},
{
"cell_type": "code",
"execution_count": 26,
"id": "ef42d2c4",
"metadata": {},
"outputs": [],
"source": [
"files = glob.glob(\"./trading*\")"
]
},
{
"cell_type": "code",
"execution_count": 27,
"id": "40d9639b",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'WFC'"
]
},
"execution_count": 27,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"files[0].split(\"_\")[-1][:-4]"
]
},
{
"cell_type": "code",
"execution_count": 30,
"id": "c802c9e3",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"\\begin{figure}\n",
" \\centering\n",
" \\includegraphics[width=\\linewidth]{figs/./trading_strategy_WFC.pdf}\n",
" \\caption{Trading Strategy for WFC.}\n",
"\\end{figure}\n",
"\n"
]
}
],
"source": [
"print(start + files[0] + mid + files[0].split(\"_\")[-1][:-4] + end)"
]
},
{
"cell_type": "code",
"execution_count": 37,
"id": "b8a6a5fa",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"\\begin{figure}\n",
" \\centering\n",
" \\includegraphics[width=\\linewidth]{figs/trading_strategy_WFC.pdf}\n",
" \\caption{Trading Strategy for WFC.}\n",
"\\end{figure}\n",
"\n",
"\n",
"\\begin{figure}\n",
" \\centering\n",
" \\includegraphics[width=\\linewidth]{figs/trading_strategy_PLD.pdf}\n",
" \\caption{Trading Strategy for PLD.}\n",
"\\end{figure}\n",
"\n",
"\n",
"\\begin{figure}\n",
" \\centering\n",
" \\includegraphics[width=\\linewidth]{figs/trading_strategy_EOG.pdf}\n",
" \\caption{Trading Strategy for EOG.}\n",
"\\end{figure}\n",
"\n",
"\n",
"\\begin{figure}\n",
" \\centering\n",
" \\includegraphics[width=\\linewidth]{figs/trading_strategy_COP.pdf}\n",
" \\caption{Trading Strategy for COP.}\n",
"\\end{figure}\n",
"\n",
"\n",
"\\begin{figure}\n",
" \\centering\n",
" \\includegraphics[width=\\linewidth]{figs/trading_strategy_JPM.pdf}\n",
" \\caption{Trading Strategy for JPM.}\n",
"\\end{figure}\n",
"\n",
"\n",
"\\begin{figure}\n",
" \\centering\n",
" \\includegraphics[width=\\linewidth]{figs/trading_strategy_BRK.B.pdf}\n",
" \\caption{Trading Strategy for BRK.B.}\n",
"\\end{figure}\n",
"\n",
"\n",
"\\begin{figure}\n",
" \\centering\n",
" \\includegraphics[width=\\linewidth]{figs/trading_strategy_BLK.pdf}\n",
" \\caption{Trading Strategy for BLK.}\n",
"\\end{figure}\n",
"\n",
"\n",
"\\begin{figure}\n",
" \\centering\n",
" \\includegraphics[width=\\linewidth]{figs/trading_strategy_C.pdf}\n",
" \\caption{Trading Strategy for C.}\n",
"\\end{figure}\n",
"\n",
"\n",
"\\begin{figure}\n",
" \\centering\n",
" \\includegraphics[width=\\linewidth]{figs/trading_strategy_SLB.pdf}\n",
" \\caption{Trading Strategy for SLB.}\n",
"\\end{figure}\n",
"\n",
"\n",
"\\begin{figure}\n",
" \\centering\n",
" \\includegraphics[width=\\linewidth]{figs/trading_strategy_BAC.pdf}\n",
" \\caption{Trading Strategy for BAC.}\n",
"\\end{figure}\n",
"\n",
"\n",
"\\begin{figure}\n",
" \\centering\n",
" \\includegraphics[width=\\linewidth]{figs/trading_strategy_MS.pdf}\n",
" \\caption{Trading Strategy for MS.}\n",
"\\end{figure}\n",
"\n",
"\n",
"\\begin{figure}\n",
" \\centering\n",
" \\includegraphics[width=\\linewidth]{figs/trading_strategy_SCHW.pdf}\n",
" \\caption{Trading Strategy for SCHW.}\n",
"\\end{figure}\n",
"\n",
"\n",
"\\begin{figure}\n",
" \\centering\n",
" \\includegraphics[width=\\linewidth]{figs/trading_strategy_GS.pdf}\n",
" \\caption{Trading Strategy for GS.}\n",
"\\end{figure}\n",
"\n",
"\n",
"\\begin{figure}\n",
" \\centering\n",
" \\includegraphics[width=\\linewidth]{figs/trading_strategy_CCI.pdf}\n",
" \\caption{Trading Strategy for CCI.}\n",
"\\end{figure}\n",
"\n",
"\n",
"\\begin{figure}\n",
" \\centering\n",
" \\includegraphics[width=\\linewidth]{figs/trading_strategy_AMT.pdf}\n",
" \\caption{Trading Strategy for AMT.}\n",
"\\end{figure}\n",
"\n",
"\n",
"\\begin{figure}\n",
" \\centering\n",
" \\includegraphics[width=\\linewidth]{figs/trading_strategy_XOM.pdf}\n",
" \\caption{Trading Strategy for XOM.}\n",
"\\end{figure}\n",
"\n",
"\n",
"\\begin{figure}\n",
" \\centering\n",
" \\includegraphics[width=\\linewidth]{figs/trading_strategy_CVX.pdf}\n",
" \\caption{Trading Strategy for CVX.}\n",
"\\end{figure}\n",
"\n",
"\n",
"\\begin{figure}\n",
" \\centering\n",
" \\includegraphics[width=\\linewidth]{figs/trading_strategy_EQIX.pdf}\n",
" \\caption{Trading Strategy for EQIX.}\n",
"\\end{figure}\n",
"\n",
"\n",
"\\begin{figure}\n",
" \\centering\n",
" \\includegraphics[width=\\linewidth]{figs/trading_strategy_AXP.pdf}\n",
" \\caption{Trading Strategy for AXP.}\n",
"\\end{figure}\n",
"\n"
]
}
],
"source": [
"for stck in files:\n",
" tckr = stck.split(\"_\")[-1][:-4]\n",
" print(start + stck[2:] + mid + tckr + end)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c9e72667",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.3"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
experiments/px-forecasting/output_plotter.ipynb
-968
View File
File diff suppressed because one or more lines are too long
experiments/px-forecasting/price_prediction.ipynb
-408
View File
File diff suppressed because one or more lines are too long
experiments/px-forecasting/price_prediction.py
-61
View File
@@ -1,61 +0,0 @@
import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np
import torch
import pandas as pd
import gpytorch
from torch.nn.functional import softplus
from voltron.kernels import BMKernel, VolatilityKernel
from voltron.models import BMGP, VoltronGP
import argparse
from torch.distributions import Beta
from scipy.special import betainc
from Trainers import *
def main(args):
full_data = pd.read_pickle("../../spdr-data/" + args.SPDR + ".pkl")
tckrs = full_data.symbol.unique()
for tckr in tckrs:
data = full_data[full_data["symbol"] == tckr]
ts = torch.linspace(0, data.shape[0]/252., data.shape[0])
# train_x = ts[:ntrain]
# test_x = ts[ntrain:(ntrain+ntest)]
y = torch.FloatTensor(data['close_price'].to_numpy())
log_returns = torch.log(y[1:]) - torch.log(y[:-1])
dt = ts[1] - ts[0]
eval_times = list(range(100, ts.shape[0], 100)) #+ [ts.shape[0]]
prob_of_increases = []
for i, time in enumerate(eval_times):
print("now running time: ", time)
with gpytorch.settings.max_cholesky_size(2000):
pred_vol = get_and_fit_gpcv(ts[:time], log_returns[:(time - 1)])
vol_model = get_and_fit_vol_model(ts[:time], pred_vol)
data_model = get_and_fit_data_model(ts[:time], y[:time],
pred_vol, vol_model)
end_ind = -1 if i + 1 >= len(eval_times) else eval_times[i+1]
paths = predict_prices(ts[time:end_ind], data_model).detach()
# now we predict the probability of increase at time i + 1
prob_of_increase = (paths[..., -1] > y[time]).sum() / paths.shape[-2]
# print("prob of stock increase: ", prob_of_increase.detach())
prob_of_increases.append(prob_of_increase.detach())
torch.save(obj=prob_of_increases, f="./outputs/voltron_" + tckr + ".pt")
print(tckr, "Done")
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument(
"--SPDR",
type=str,
default="XLE",
)
args = parser.parse_args()
main(args)
experiments/px-forecasting/price_strat_opt_sectors.ipynb
-625
View File
File diff suppressed because one or more lines are too long
experiments/px-forecasting/price_strategy_optimization.ipynb
-891
View File
File diff suppressed because one or more lines are too long
experiments/px-forecasting/svgp_price_strategy.ipynb
-1329
View File
File diff suppressed because one or more lines are too long
experiments/px-forecasting/svgp_price_strategy_matern.ipynb
-601
View File
File diff suppressed because one or more lines are too long
experiments/stocks/.DS_Store
Vendored
BIN
View File
Binary file not shown.
experiments/trading/TickerSingleDayGenerator.py → experiments/stocks/.ipynb_checkpoints/ForecastGenerator-checkpoint.py
+37 -39
View File
@@ -7,66 +7,64 @@ import datetime
import warnings
from voltron.data import make_ticker_list, GetStockHistory
from GenerateMultiMeanPreds import GenerateStockPredictions, GenerateBasicPredictions, GenerateOneDayPredictions
import sys
from GenerateMultiMeanPreds import GenerateStockPredictions, GenerateBasicPredictions
from gpytorch.utils.warnings import NumericalWarning
warnings.simplefilter("ignore", NumericalWarning)
warnings.simplefilter("ignore", UserWarning)
def main(args):
tckr = args.ticker
## download data ##
ticker_file = args.ticker_fname + ".txt"
tckr_list = make_ticker_list(ticker_file)
if args.end_date.lower() == "none":
end_date = datetime.date.today()
else:
end_date = datetime.datetime.strptime(args.end_date, "%Y-%m-%d").date()
dat = GetStockHistory(tckr, history=args.ntrain + args.lookback,
end_date=str(end_date))
end_date = datetime.datetime.strptime(args.end_date, "%Y-%m-%d")
## pick a day to generate forecasts ##
end_idxs = torch.arange(args.ntrain, dat.shape[0],
int((dat.shape[0]-args.ntrain)/args.ntimes))
last_day = end_idxs[args.test_idx]
date = str(dat.index[last_day.item()].date())
print(date, tckr)
train_y = torch.FloatTensor(dat.Close[last_day.item()-args.ntrain:last_day.item()].to_numpy())
GenerateOneDayPredictions(tckr, train_y, date,
forecast_horizon=args.forecast_horizon,
train_iters=args.train_iters,
nsample=args.nsample,
ntrain=400, save=args.save, mean=args.mean)
for tckr in tckr_list:
# try:
data = GetStockHistory(tckr, history=args.ntrain + args.lookback)
if args.kernel.lower() == 'volt':
GenerateStockPredictions(tckr, data, forecast_horizon=args.forecast_horizon,
train_iters=args.train_iters,
nsample=args.nsample, mean=args.mean,
ntrain=args.ntrain, save=args.save,
ntimes=args.ntimes)
else:
GenerateBasicPredictions(tckr, data, forecast_horizon=args.forecast_horizon,
kernel_name=args.kernel, mean_name=args.mean,
k=args.k, train_iters=args.train_iters,
nsample=args.nsample, ntimes=args.ntimes,
ntrain=args.ntrain, save=args.save)
# except:
# print("FAILED ", tckr)
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument(
"--ticker_fname",
type=str,
default='test_tickers',
)
parser.add_argument(
"--ntrain",
type=int,
default=400,
)
parser.add_argument(
"--ntimes",
type=int,
default=25,
)
parser.add_argument(
"--test_idx",
type=int,
default=0,
)
parser.add_argument(
"--forecast_horizon",
type=int,
default=100,
)
parser.add_argument(
"--ticker",
type=str,
default='ADBE',
)
parser.add_argument(
"--ntrain",
type=int,
default=400,
)
parser.add_argument(
'--kernel',
type=str,
experiments/stocks/.ipynb_checkpoints/Forecaster-checkpoint.py
-152
View File
@@ -1,152 +0,0 @@
import numpy as np
import torch
import pandas as pd
import gpytorch
import argparse
import datetime
import warnings
from voltron.data import make_ticker_list, GetStockHistory
import sys
sys.path.append("../trading/")
from GenerateMultiMeanPreds import GenerateStockPredictions, GenerateBasicPredictions
from gpytorch.utils.warnings import NumericalWarning
warnings.simplefilter("ignore", NumericalWarning)
def main(args):
if args.end_date is None:
end_date = datetime.date.today()
else:
end_date = datetime.datetime.strptime(args.end_date, "%Y-%m-%d")
data = GetStockHistory(args.ticker, history=args.ntrain + args.lookback)
ntest = args.forecast_horizon
ntrain = args.ntrain
n_test_times = args.n_test_times
ntime = data.shape[0]
test_idxs = torch.arange(ntrain, ntime-ntest,
int((ntime-ntest-ntrain)/n_test_times))
train_x = torch.arange(ntrain) * dt
test_x = torch.arange(ntest) * dt + train_x[-1] + dt
if torch.cuda.is_available():
train_x = train_x.cuda()
test_x = test_x.cuda()
####################
## setup filename ##
####################
savepath = "./saved-outputs/" + args.ticker + "/"
if not os.path.exists(savepath):
os.mkdir(savepath)
if args.model.lower() == 'lstm':
modelname = "lstm"
else:
if args.model.lower() == 'gp':
modelname = "gp_" + args.kernel + "_"
elif args.model.lower() == 'volt':
modelname = "volt_"
if args.mean.lower() == 'constant':
modelname += 'constant' + "_"
elif args.mean.lower() in ['ewma', 'dewma', 'tewma']:
modelname += args.mean + args.k + "_"
###############
## Main Loop ##
###############
for last_day in test_idxs:
date = str(data.index[last_day.item()].date())
train_y = data.Close[last_day.item()-ntrain:last_day.item()].to_numpy()
train_y = torch.FloatTensor(train_y).to(train_x.device)
if args.model.lower() == 'lstm':
model = LSTM(train_x, train_y, 10, 128, 1)
model.Train(args.train_iters)
elif args.model.lower() == 'gp':
model = BasicGP(train_x, train_y, kernel=args.kernel,
mean=args.mean, k=args.k)
model.Train(args.train_iters)
elif args.model.lower() == 'volt':
model = Volt(train_x, train_y, mean=args.mean, k=args.k)
model.Train(gpcv_iters=args.train_iters,
vol_mod_iters=args.train_iters,
data_mod_iters=args.train_iters)
else:
print("ERROR: Model not found")
samples = model.Forecast(test_x).squeeze()
torch.save(samples, savepath + modelname + date + ".pt")
torch.cuda.empty_cache()
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument(
"--ticker",
type=str,
default='F',
)
parser.add_argument(
"--ntrain",
type=int,
default=400,
)
parser.add_argument(
"--n_test_times",
type=int,
default=25,
)
parser.add_argument(
"--forecast_horizon",
type=int,
default=100,
)
parser.add_argument(
'--kernel',
type=str,
default="matern",
)
parser.add_argument(
'--model',
type=str,
default="volt",
)
parser.add_argument(
'--mean',
type=str,
default="ewma",
)
parser.add_argument(
"--nsample",
type=int,
default=1000,
)
parser.add_argument(
"--train_iters",
type=int,
default=500,
)
parser.add_argument(
"--end_date",
default=None,
)
parser.add_argument(
"--lookback",
type=int,
default=500,
)
parser.add_argument(
"--k",
type=int,
default=100,
)
args = parser.parse_args()
main(args)
experiments/stocks/.ipynb_checkpoints/GenerateMultiMeanPreds-checkpoint.py
+298
View File
@@ -0,0 +1,298 @@
import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np
import torch
import pandas as pd
import os
import gpytorch
import argparse
import datetime
from botorch.models import SingleTaskGP
from botorch.optim.fit import fit_gpytorch_torch
from gpytorch.likelihoods import GaussianLikelihood
from gpytorch.mlls import ExactMarginalLogLikelihood
from gpytorch.means import ConstantMean, LinearMean
from gpytorch.kernels import SpectralMixtureKernel, MaternKernel, RBFKernel, ScaleKernel
from voltron.means import EWMAMean, DEWMAMean, TEWMAMean
from voltron.train_utils import LearnGPCV, TrainVolModel, TrainVoltMagpieModel, TrainBasicModel
from voltron.models import VoltMagpie
from voltron.means import LogLinearMean
from voltron.rollout_utils import GeneratePrediction, Rollouts
from voltron.data import make_ticker_list, DataGetter, GetStockHistory
def GenerateGPCVPredictions(ticker, dat,
forecast_horizon=20, ntimes=25,
train_iters=400, nsample=1000,
ntrain=400):
end_idxs = torch.arange(ntrain, dat.shape[0],
int((dat.shape[0]-ntrain)/ntimes))
ntest = forecast_horizon
dt = 1./252
savepath = "./saved-outputs/" + ticker + "/"
if not os.path.exists(savepath):
os.mkdir(savepath)
for last_day in end_idxs:
date = str(dat.index[last_day.item()].date())
print(date, ticker)
train_y = torch.FloatTensor(dat.Close[last_day.item()-ntrain:last_day.item()].to_numpy())
train_x = torch.arange(train_y.shape[0]-1) * dt
test_x = torch.arange(ntest) * dt + train_x[-1] + train_x[1]
# try:
use_cuda = torch.cuda.is_available()
if use_cuda:
train_x = train_x.cuda()
test_x = test_x.cuda()
train_y = train_y.cuda()
model, likelihood = LearnGPCV(train_x, train_y,
train_iters=train_iters, printing=False, return_model=True)
preds = likelihood(model(test_x),
return_gaussian=False).sample(torch.Size((nsample,)))
preds = preds.cumsum(-1).squeeze()
preds = preds.view(-1, preds.shape[-1])
save_samples = preds.view(-1, preds.shape[-1]) * (dt ** 0.5) + train_y[-1].log()
torch.save(save_samples, savepath + "gpcv_" + date + ".pt")
return
def GenerateStockPredictions(ticker, dat,
forecast_horizon=20,
train_iters=400, nsample=1000,
ntrain=400, mean='ewma', kernel='volt',
save=False, k=300, ntimes=-1):
if ntimes == -1:
end_idxs = torch.arange(ntrain, dat.shape[0])
else:
end_idxs = torch.arange(ntrain, dat.shape[0],
int((dat.shape[0]-ntrain)/ntimes))
ntest = forecast_horizon
dt = 1./252
model_name = kernel + "_" + mean + str(k) + "_"
par_dir = "./saved-outputs/"
if not os.path.exists(par_dir):
os.mkdir(par_dir)
savepath = par_dir + ticker + "/"
if not os.path.exists(savepath):
os.mkdir(savepath)
for last_day in end_idxs:
date = str(dat.index[last_day.item()].date())
# try:
train_y = torch.FloatTensor(dat.Close[last_day.item()-ntrain:last_day.item()].to_numpy())
train_x = torch.arange(train_y.shape[0]-1) * dt
test_x = torch.arange(ntest) * dt + train_x[-1] + train_x[1]
# try:
use_cuda = torch.cuda.is_available()
if use_cuda:
train_x = train_x.cuda()
test_x = test_x.cuda()
train_y = train_y.cuda()
# print("Producing " + ticker + " Forecasts.....")
if kernel == "volt":
vol = LearnGPCV(train_x, train_y, train_iters=train_iters,
printing=False)
vmod, vlh = TrainVolModel(train_x, vol,
train_iters=train_iters, printing=False)
voltron, lh = TrainVoltMagpieModel(train_x, train_y[1:],
vmod, vlh, vol,
printing=False,
train_iters=train_iters,
k=k, mean_func=mean)
vmod.eval();
if mean in ['ewma', 'dewma', 'tewma']:
save_samples = Rollouts(train_x, train_y, test_x, voltron,
nsample=nsample)
else: ## VOLT + STANDARD MEAN
voltron.vol_model.eval()
predvol = voltron.vol_model(test_x).sample(torch.Size((nsample, ))).exp()
save_samples = GeneratePrediction(train_x, train_y, test_x,
predvol, voltron).detach()
del predvol
del voltron, lh, vmod, vlh, vol
torch.cuda.empty_cache()
if save:
if not os.path.exists(savepath):
os.mkdir(savepath)
torch.save(save_samples, savepath + model_name + date + ".pt")
# except:
# nans = torch.ones(nsample, ntest) * torch.nan
# if save:
# if not os.path.exists(savepath):
# os.mkdir(savepath)
# torch.save(nans, savepath + model_name + date + ".pt")
return dat, save_samples
def GenerateOneDayPredictions(ticker, train_y, date,
forecast_horizon=20,
train_iters=400, nsample=1000,
ntrain=400, save=False, mean=None):
ntest = forecast_horizon
dt = 1./252
par_dir = "./saved-outputs/"
if not os.path.exists(par_dir):
os.mkdir(par_dir)
savepath = par_dir + ticker + "/"
if not os.path.exists(savepath):
os.mkdir(savepath)
train_x = torch.arange(train_y.shape[0]-1) * dt
test_x = torch.arange(ntest) * dt + train_x[-1] + train_x[1]
use_cuda = torch.cuda.is_available()
if use_cuda:
train_x = train_x.cuda()
test_x = test_x.cuda()
train_y = train_y.cuda()
vol = LearnGPCV(train_x, train_y, train_iters=train_iters,
printing=False)
vmod, vlh = TrainVolModel(train_x, vol,
train_iters=train_iters, printing=False)
if mean=='constant':
voltron, lh = TrainVoltMagpieModel(train_x, train_y[1:],
vmod, vlh, vol,
printing=False,
train_iters=200,
mean_func='constant')
vmod.eval();
voltron.eval();
save_samples = Rollouts(train_x, train_y, test_x, voltron,
nsample=nsample)
if save:
model_name = "volt_" + mean + "_"
torch.save(save_samples, savepath + model_name + date + ".pt")
else:
for mean in ['ewma', 'dewma', 'tewma']:
for k in [25, 50, 100, 200, 300, 400]:
try:
voltron, lh = TrainVoltMagpieModel(train_x, train_y[1:],
vmod, vlh, vol,
printing=False,
train_iters=0,
k=k, mean_func=mean)
vmod.eval();
voltron.eval();
save_samples = Rollouts(train_x, train_y, test_x, voltron,
nsample=nsample)
except:
print("Failed: ", ticker, mean, k)
if save:
save_samples = torch.ones(nsample, ntest) * torch.nan
if save:
model_name = "volt_" + mean + str(k) + "_"
torch.save(save_samples, savepath + model_name + date + ".pt")
del voltron, lh, vmod, vlh, vol, save_samples
torch.cuda.empty_cache()
return
def GenerateBasicPredictions(ticker, dat, kernel_name, mean_name='ewma', k=400,
forecast_horizon=100,
train_iters=600, nsample=1000,
ntrain=400, save=False, ntimes=-1):
if ntimes == -1:
end_idxs = torch.arange(ntrain, dat.shape[0])
else:
end_idxs = torch.arange(ntrain, dat.shape[0],
int((dat.shape[0]-ntrain)/ntimes))
ntest = forecast_horizon
dt = 1./252
par_dir = "./saved-outputs/"
if not os.path.exists(par_dir):
os.mkdir(par_dir)
savepath = par_dir + ticker + "/"
if not os.path.exists(savepath):
os.mkdir(savepath)
for last_day in end_idxs:
date = str(dat.index[last_day.item()].date())
train_y = torch.FloatTensor(dat.Close[last_day.item()-ntrain:last_day.item()].to_numpy())
train_x = torch.arange(train_y.shape[0]-1) * dt
test_x = torch.arange(ntest) * dt + train_x[-1] + train_x[1]
# try:
use_cuda = torch.cuda.is_available()
if use_cuda:
train_x = train_x.cuda()
test_x = test_x.cuda()
train_y = train_y.cuda()
# print("Producing " + ticker + " Forecasts.....")
kernel_possibilities = {"sm": SpectralMixtureKernel,
"matern": MaternKernel,
"rbf": RBFKernel}
kernel = kernel_possibilities[kernel_name.lower()]
if kernel_name.lower() != 'sm':
kernel = ScaleKernel(kernel())
else:
kernel = kernel(num_mixtures=15)
kernel.initialize_from_data_empspect(train_x, train_y.log())
train_y = train_y[1:]
model = SingleTaskGP(
train_x.view(-1,1),
train_y.log().reshape(-1, 1),
covar_module=kernel,
likelihood=GaussianLikelihood()
)
mean_name = mean_name.lower()
if mean_name == "loglinear":
model.mean_module = LogLinearMean(1)
model.mean_module.initialize_from_data(train_x, train_y.log())
elif mean_name == 'linear':
model.mean_module = LinearMean(1)
elif mean_name == "constant":
model.mean_module = ConstantMean()
elif mean_name == "ewma":
model.mean_module = EWMAMean(train_x, train_y.log(), k=k).to(train_x.device)
elif mean_name == "dewma":
model.mean_module = DEWMAMean(train_x, train_y.log(), k=k).to(train_x.device)
elif mean_name == "tewma":
model.mean_module = TEWMAMean(train_x, train_y.log(), k=k).to(train_x.device)
if use_cuda:
model = model.to(train_x.device)
print("Fitting Model", ticker)
mll = ExactMarginalLogLikelihood(model.likelihood, model)
fit_gpytorch_torch(mll, options={'maxiter':train_iters, 'disp':False})
if mean_name in ["loglinear", "constant", 'linear']:
save_samples = model.posterior(test_x).sample(torch.Size((nsample,
))).squeeze(-1).cpu().detach()
else:
save_samples = Rollouts(
train_x, train_y, test_x, model, nsample=nsample, method = "nonvol"
).cpu().detach()
model_name = kernel_name + "_" + mean_name + str(k) + "_"
torch.save(save_samples, savepath + model_name + date + ".pt")
model.train()
torch.cuda.empty_cache()
del model
return dat, save_samples
experiments/calibration/LSTMGenerator.py → experiments/stocks/.ipynb_checkpoints/LSTMGenerator-checkpoint.py
View File
experiments/stocks/.ipynb_checkpoints/OutputTest-checkpoint.ipynb
-6
View File
@@ -1,6 +0,0 @@
{
"cells": [],
"metadata": {},
"nbformat": 4,
"nbformat_minor": 5
}
experiments/stocks/.ipynb_checkpoints/example_tickers-checkpoint.txt
+3
View File
@@ -0,0 +1,3 @@
AAPL
MSFT
GOOG
experiments/stocks/.ipynb_checkpoints/lstm_scratch-checkpoint.ipynb
-408
View File
File diff suppressed because one or more lines are too long
experiments/stocks/.ipynb_checkpoints/plotter-checkpoint.ipynb
-454
View File
File diff suppressed because one or more lines are too long
experiments/calibration/ForecastGenerator.py → experiments/stocks/ForecastGenerator.py
+18 -19
View File
@@ -8,7 +8,6 @@ import warnings
from voltron.data import make_ticker_list, GetStockHistory
import sys
sys.path.append("../trading/")
from GenerateMultiMeanPreds import GenerateStockPredictions, GenerateBasicPredictions
from gpytorch.utils.warnings import NumericalWarning
warnings.simplefilter("ignore", NumericalWarning)
@@ -16,33 +15,33 @@ warnings.simplefilter("ignore", NumericalWarning)
def main(args):
data_path = "../../voltron/data/"
ticker_file = args.ticker_fname + ".txt"
tckr_list = make_ticker_list(data_path + ticker_file)
tckr_list = make_ticker_list(ticker_file)
if args.end_date.lower() == "none":
end_date = datetime.date.today()
else:
end_date = datetime.datetime.strptime(args.end_date, "%Y-%m-%d")
end_date = datetime.datetime.strptime(args.end_date, "%Y-%m-%d")
for tckr in tckr_list:
try:
data = GetStockHistory(tckr, history=args.ntrain + args.lookback)
if args.kernel.lower() == 'volt':
GenerateStockPredictions(tckr, data, forecast_horizon=args.forecast_horizon,
train_iters=args.train_iters,
nsample=args.nsample, mean_name=args.mean,
ntrain=args.ntrain, save=args.save)
else:
GenerateBasicPredictions(tckr, data, forecast_horizon=args.forecast_horizon,
kernel_name=args.kernel, mean_name=args.mean,
k=args.k, train_iters=args.train_iters,
nsample=args.nsample, ntimes=args.ntimes,
ntrain=args.ntrain, save=args.save)
# try:
data = GetStockHistory(tckr, history=args.ntrain + args.lookback)
if args.kernel.lower() == 'volt':
GenerateStockPredictions(tckr, data, forecast_horizon=args.forecast_horizon,
train_iters=args.train_iters,
nsample=args.nsample, mean=args.mean,
ntrain=args.ntrain, save=args.save,
ntimes=args.ntimes)
else:
GenerateBasicPredictions(tckr, data, forecast_horizon=args.forecast_horizon,
kernel_name=args.kernel, mean_name=args.mean,
k=args.k, train_iters=args.train_iters,
nsample=args.nsample, ntimes=args.ntimes,
ntrain=args.ntrain, save=args.save)
except:
print("FAILED ", tckr)
# except:
# print("FAILED ", tckr)
if __name__ == "__main__":
parser = argparse.ArgumentParser()
experiments/stocks/Forecaster.py
-152
View File
@@ -1,152 +0,0 @@
import numpy as np
import torch
import pandas as pd
import gpytorch
import argparse
import datetime
import warnings
from voltron.data import make_ticker_list, GetStockHistory
import sys
sys.path.append("../trading/")
from GenerateMultiMeanPreds import GenerateStockPredictions, GenerateBasicPredictions
from gpytorch.utils.warnings import NumericalWarning
warnings.simplefilter("ignore", NumericalWarning)
def main(args):
if args.end_date is None:
end_date = datetime.date.today()
else:
end_date = datetime.datetime.strptime(args.end_date, "%Y-%m-%d")
data = GetStockHistory(args.ticker, history=args.ntrain + args.lookback)
ntest = args.forecast_horizon
ntrain = args.ntrain
n_test_times = args.n_test_times
ntime = data.shape[0]
test_idxs = torch.arange(ntrain, ntime-ntest,
int((ntime-ntest-ntrain)/n_test_times))
train_x = torch.arange(ntrain) * dt
test_x = torch.arange(ntest) * dt + train_x[-1] + dt
if torch.cuda.is_available():
train_x = train_x.cuda()
test_x = test_x.cuda()
####################
## setup filename ##
####################
savepath = "./saved-outputs/" + args.ticker + "/"
if not os.path.exists(savepath):
os.mkdir(savepath)
if args.model.lower() == 'lstm':
modelname = "lstm"
else:
if args.model.lower() == 'gp':
modelname = "gp_" + args.kernel + "_"
elif args.model.lower() == 'volt':
modelname = "volt_"
if args.mean.lower() == 'constant':
modelname += 'constant' + "_"
elif args.mean.lower() in ['ewma', 'dewma', 'tewma']:
modelname += args.mean + args.k + "_"
###############
## Main Loop ##
###############
for last_day in test_idxs:
date = str(data.index[last_day.item()].date())
train_y = data.Close[last_day.item()-ntrain:last_day.item()].to_numpy()
train_y = torch.FloatTensor(train_y).to(train_x.device)
if args.model.lower() == 'lstm':
model = LSTM(train_x, train_y, 10, 128, 1)
model.Train(args.train_iters)
elif args.model.lower() == 'gp':
model = BasicGP(train_x, train_y, kernel=args.kernel,
mean=args.mean, k=args.k)
model.Train(args.train_iters)
elif args.model.lower() == 'volt':
model = Volt(train_x, train_y, mean=args.mean, k=args.k)
model.Train(gpcv_iters=args.train_iters,
vol_mod_iters=args.train_iters,
data_mod_iters=args.train_iters)
else:
print("ERROR: Model not found")
samples = model.Forecast(test_x).squeeze()
torch.save(samples, savepath + modelname + date + ".pt")
torch.cuda.empty_cache()
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument(
"--ticker",
type=str,
default='F',
)
parser.add_argument(
"--ntrain",
type=int,
default=400,
)
parser.add_argument(
"--n_test_times",
type=int,
default=25,
)
parser.add_argument(
"--forecast_horizon",
type=int,
default=100,
)
parser.add_argument(
'--kernel',
type=str,
default="matern",
)
parser.add_argument(
'--model',
type=str,
default="volt",
)
parser.add_argument(
'--mean',
type=str,
default="ewma",
)
parser.add_argument(
"--nsample",
type=int,
default=1000,
)
parser.add_argument(
"--train_iters",
type=int,
default=500,
)
parser.add_argument(
"--end_date",
default=None,
)
parser.add_argument(
"--lookback",
type=int,
default=500,
)
parser.add_argument(
"--k",
type=int,
default=100,
)
args = parser.parse_args()
main(args)
experiments/trading/GenerateMultiMeanPreds.py → experiments/stocks/GenerateMultiMeanPreds.py
+58 -66
View File
@@ -63,20 +63,28 @@ def GenerateGPCVPredictions(ticker, dat,
def GenerateStockPredictions(ticker, dat,
forecast_horizon=20,
train_iters=400, nsample=1000,
ntrain=400, save=False, ntimes=1,
vol_kernel='bm', gpcv_preds=False):
ntrain=400, mean='ewma', kernel='volt',
save=False, k=300, ntimes=-1):
end_idxs = torch.arange(ntrain, dat.shape[0],
int((dat.shape[0]-ntrain)/ntimes))
if ntimes == -1:
end_idxs = torch.arange(ntrain, dat.shape[0])
else:
end_idxs = torch.arange(ntrain, dat.shape[0],
int((dat.shape[0]-ntrain)/ntimes))
ntest = forecast_horizon
dt = 1./252
savepath = "./saved-outputs/" + ticker + "/"
model_name = kernel + "_" + mean + str(k) + "_"
par_dir = "./saved-outputs/"
if not os.path.exists(par_dir):
os.mkdir(par_dir)
savepath = par_dir + ticker + "/"
if not os.path.exists(savepath):
os.mkdir(savepath)
for last_day in end_idxs:
date = str(dat.index[last_day.item()].date())
print(date, ticker)
# try:
train_y = torch.FloatTensor(dat.Close[last_day.item()-ntrain:last_day.item()].to_numpy())
train_x = torch.arange(train_y.shape[0]-1) * dt
test_x = torch.arange(ntest) * dt + train_x[-1] + train_x[1]
@@ -88,64 +96,46 @@ def GenerateStockPredictions(ticker, dat,
train_y = train_y.cuda()
# print("Producing " + ticker + " Forecasts.....")
vol = LearnGPCV(train_x, train_y, train_iters=train_iters,
printing=False, kernel=vol_kernel)
vmod, vlh = TrainVolModel(train_x, vol,
train_iters=train_iters, printing=False)
# for mean in ['ewma']:#, 'dewma', 'tewma']:
# for k in [25, 50, 100, 200, 300, 400]:
# try:
# voltron, lh = TrainVoltMagpieModel(train_x, train_y[1:],
# vmod, vlh, vol,
# printing=False,
# train_iters=0,
# k=k, mean_func=mean)
# vmod.eval();
# voltron.eval();
# save_samples = Rollouts(train_x, train_y, test_x, voltron,
# nsample=nsample)
# except:
# print("Failed: ", ticker, mean, k)
# if save:
# save_samples = torch.ones(nsample, ntest) * torch.nan
if kernel == "volt":
vol = LearnGPCV(train_x, train_y, train_iters=train_iters,
printing=False)
vmod, vlh = TrainVolModel(train_x, vol,
train_iters=train_iters, printing=False)
voltron, lh = TrainVoltMagpieModel(train_x, train_y[1:],
vmod, vlh, vol,
printing=False,
train_iters=train_iters,
k=k, mean_func=mean)
vmod.eval();
if mean in ['ewma', 'dewma', 'tewma']:
save_samples = Rollouts(train_x, train_y, test_x, voltron,
nsample=nsample)
# if save:
# model_name = "volt_"
# if vol_kernel == 'ou':
# model_name = 'vhgp_'
# model_name += mean + str(k) + "_"
# torch.save(save_samples, savepath + model_name + date + ".pt")
# del voltron, lh, vmod, vlh, vol
# torch.cuda.empty_cache()
###################
## CONSTANT MEAN ##
###################
# try:
voltron, lh = TrainVoltMagpieModel(train_x, train_y[1:],
vmod, vlh, vol,
printing=False,
train_iters=0,
k=100, mean_func='constant')
vmod.eval();
voltron.eval();
save_samples = Rollouts(train_x, train_y, test_x, voltron,
nsample=nsample)
# except:
# print("Failed: ", ticker, mean, k)
# if save:
# save_samples = torch.ones(nsample, ntest) * torch.nan
else: ## VOLT + STANDARD MEAN
voltron.vol_model.eval()
predvol = voltron.vol_model(test_x).sample(torch.Size((nsample, ))).exp()
save_samples = GeneratePrediction(train_x, train_y, test_x,
predvol, voltron).detach()
del predvol
del voltron, lh, vmod, vlh, vol
torch.cuda.empty_cache()
if save:
model_name = "volt_"
if vol_kernel == 'ou':
model_name = 'vhgp_'
model_name += "constant_"
if not os.path.exists(savepath):
os.mkdir(savepath)
torch.save(save_samples, savepath + model_name + date + ".pt")
del voltron, lh, vmod, vlh, vol
torch.cuda.empty_cache()
return
# except:
# nans = torch.ones(nsample, ntest) * torch.nan
# if save:
# if not os.path.exists(savepath):
# os.mkdir(savepath)
# torch.save(nans, savepath + model_name + date + ".pt")
return dat, save_samples
def GenerateOneDayPredictions(ticker, train_y, date,
@@ -155,8 +145,10 @@ def GenerateOneDayPredictions(ticker, train_y, date,
ntest = forecast_horizon
dt = 1./252
savepath = "./saved-outputs/" + ticker + "/"
par_dir = "./saved-outputs/"
if not os.path.exists(par_dir):
os.mkdir(par_dir)
savepath = par_dir + ticker + "/"
if not os.path.exists(savepath):
os.mkdir(savepath)
@@ -211,8 +203,6 @@ def GenerateOneDayPredictions(ticker, train_y, date,
del voltron, lh, vmod, vlh, vol, save_samples
torch.cuda.empty_cache()
return
@@ -229,8 +219,10 @@ def GenerateBasicPredictions(ticker, dat, kernel_name, mean_name='ewma', k=400,
int((dat.shape[0]-ntrain)/ntimes))
ntest = forecast_horizon
dt = 1./252
savepath = "./saved-outputs/" + ticker + "/"
par_dir = "./saved-outputs/"
if not os.path.exists(par_dir):
os.mkdir(par_dir)
savepath = par_dir + ticker + "/"
if not os.path.exists(savepath):
os.mkdir(savepath)
experiments/stocks/LSTMGenerator.py
+133
View File
@@ -0,0 +1,133 @@
import numpy as np
import torch
import pandas as pd
import gpytorch
import argparse
import datetime
import warnings
import os
from voltron.data import make_ticker_list, GetStockHistory
from LSTMUtils import SequenceDataset, LSTM, TrainLSTM, LSTMRollouts, NLL
from torch.utils.data import DataLoader
def main(args):
data_path = "../../voltron/data/"
ticker_file = args.ticker_fname + ".txt"
tckr_list = make_ticker_list(data_path + ticker_file)
# tckr_list = ['TSLA']
use_cuda = False
if torch.cuda.is_available():
use_cuda = True
ntest = args.forecast_horizon
ntrain = args.ntrain
seq_len = args.seq_length
if args.end_date.lower() == "none":
end_date = datetime.date.today()
else:
end_date = datetime.datetime.strptime(args.end_date, "%Y-%m-%d")
for tckr in tckr_list:
try:
data = GetStockHistory(tckr, history= ntrain + args.lookback)
end_idxs = torch.arange(args.ntrain, data.shape[0],
int((data.shape[0]-args.ntrain)/args.ntimes))
savepath = "./saved-outputs/" + tckr + "/"
if not os.path.exists(savepath):
os.mkdir(savepath)
for last_day in end_idxs:
date = str(data.index[last_day.item()].date())
raw_y = data.Close[last_day.item()-ntrain:last_day.item()].to_numpy()
raw_y = torch.FloatTensor(raw_y).log()
train_y = (raw_y - raw_y.mean())/raw_y.std()
## make trainloader ##
dset = SequenceDataset(train_y, seq_len)
trainloader = DataLoader(dset, batch_size=args.batch_size, shuffle=True)
model = LSTM(2, seq_len, 128, 1)
if use_cuda:
model = model.cuda()
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)
TrainLSTM(trainloader, model, NLL, optimizer, epochs=args.train_epochs,
printing=True, use_cuda=use_cuda)
rollouts = LSTMRollouts(model, args.nsample, ntest,
dset, use_cuda).cpu()
rollouts = rollouts * raw_y.std() + raw_y.mean()
torch.save(rollouts, savepath + "lstm_" + date + ".pt")
del model
except:
print("FAILED ", tckr)
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument(
"--ntimes",
type=int,
default=25,
)
parser.add_argument(
"--forecast_horizon",
type=int,
default=20,
)
parser.add_argument(
"--seq_length",
type=int,
default=25,
)
parser.add_argument(
"--ticker_fname",
type=str,
default='test_tickers',
)
parser.add_argument(
"--ntrain",
type=int,
default=400,
)
parser.add_argument(
"--batch_size",
type=int,
default=128,
)
parser.add_argument(
"--nsample",
type=int,
default=1000,
)
parser.add_argument(
"--printing",
type=bool,
default=False
)
parser.add_argument(
"--train_epochs",
type=int,
default=200,
)
parser.add_argument(
"--end_date",
default="none",
)
parser.add_argument(
"--lookback",
type=int,
default=500,
)
parser.add_argument(
"--save",
type=bool,
default=False,
)
args = parser.parse_args()
main(args)
experiments/calibration/LSTMUtils.py → experiments/stocks/LSTMUtils.py
View File
experiments/stocks/OutputTest.ipynb
-131
View File
File diff suppressed because one or more lines are too long
experiments/stocks/README.md
View File
experiments/stocks/__pycache__/GenerateMultiMeanPreds.cpython-38.pyc
BIN
View File
Binary file not shown.
experiments/stocks/__pycache__/LSTMUtils.cpython-38.pyc
BIN
View File
Binary file not shown.
experiments/stocks/example_tickers.txt
+3
View File
@@ -0,0 +1,3 @@
AAPL
MSFT
GOOG
experiments/stocks/lstm_scratch.ipynb
-408
View File
File diff suppressed because one or more lines are too long
experiments/stocks/plotter.ipynb
-454
View File
File diff suppressed because one or more lines are too long
experiments/stocks/saved-outputs/TSLA/lstm_2021-12-27.pt
BIN
View File
Binary file not shown.
experiments/stocks/saved-outputs/TSLA/lstm_2021-12-28.pt
BIN
View File
Binary file not shown.
experiments/stocks/scratch.ipynb
-722
View File
File diff suppressed because one or more lines are too long
experiments/trading/Calib_getter.ipynb
-393
View File
File diff suppressed because one or more lines are too long
experiments/trading/FullGenerator.py
-125
View File
@@ -1,125 +0,0 @@
import numpy as np
import torch
import pandas as pd
import gpytorch
import argparse
import datetime
import warnings
from voltron.data import make_ticker_list, GetStockHistory
from GenerateMultiMeanPreds import GenerateStockPredictions, GenerateBasicPredictions, GenerateGPCVPredictions
from gpytorch.utils.warnings import NumericalWarning
warnings.simplefilter("ignore", NumericalWarning)
def main(args):
data_path = "../../voltron/data/"
ticker_file = args.ticker_fname + ".txt"
tckr_list = make_ticker_list(data_path + ticker_file)
if args.end_date.lower() == "none":
end_date = datetime.date.today()
else:
end_date = datetime.datetime.strptime(args.end_date, "%Y-%m-%d").date()
for tckr in tckr_list:
try:
data = GetStockHistory(tckr, history=args.ntrain + args.lookback,
end_date=str(end_date))
except:
print(tckr, "FAILED")
data = None
if data is not None:
if args.kernel.lower() == 'volt':
GenerateStockPredictions(tckr, data, forecast_horizon=args.forecast_horizon,
train_iters=args.train_iters,
nsample=args.nsample,
ntrain=400, save=args.save, ntimes=args.ntimes,
vol_kernel=args.vol_kernel.lower())
elif args.kernel.lower() == 'gpcv':
GenerateGPCVPredictions(tckr, data, forecast_horizon=args.forecast_horizon,
train_iters=args.train_iters,
nsample=args.nsample,
ntrain=400, ntimes=args.ntimes)
else:
GenerateBasicPredictions(tckr, data, forecast_horizon=args.forecast_horizon,
kernel_name=args.kernel, mean_name=args.mean, k=args.k,
train_iters=args.train_iters,
nsample=args.nsample,
ntrain=args.ntrain, save=args.save)
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument(
"--ntimes",
type=int,
default=25,
)
parser.add_argument(
"--forecast_horizon",
type=int,
default=100,
)
parser.add_argument(
"--ticker_fname",
type=str,
default='nasdaq100',
)
parser.add_argument(
"--ntrain",
type=int,
default=400,
)
parser.add_argument(
'--kernel',
type=str,
default="volt",
)
parser.add_argument(
'--vol_kernel',
type=str,
default="bm",
)
parser.add_argument(
'--mean',
type=str,
default="ewma",
)
parser.add_argument(
"--nsample",
type=int,
default=1000,
)
parser.add_argument(
"--printing",
type=bool,
default=False
)
parser.add_argument(
"--train_iters",
type=int,
default=300,
)
parser.add_argument(
"--end_date",
default='2022-04-08',
)
parser.add_argument(
"--lookback",
type=int,
default=500,
)
parser.add_argument(
"--save",
type=bool,
default=True,
)
parser.add_argument(
"--k",
type=int,
default=100,
)
args = parser.parse_args()
main(args)
experiments/trading/GenerateDaysPredictions-old.py
-182
View File
@@ -1,182 +0,0 @@
import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np
import torch
import pandas as pd
import os
import gpytorch
import argparse
import datetime
from botorch.models import SingleTaskGP
from botorch.optim.fit import fit_gpytorch_torch
from gpytorch.likelihoods import GaussianLikelihood
from gpytorch.mlls import ExactMarginalLogLikelihood
from gpytorch.means import ConstantMean, LinearMean
from gpytorch.kernels import SpectralMixtureKernel, MaternKernel, RBFKernel, ScaleKernel
from voltron.means import EWMAMean, DEWMAMean, TEWMAMean
from voltron.train_utils import LearnGPCV, TrainVolModel, TrainVoltMagpieModel, TrainBasicModel
from voltron.models import VoltMagpie
from voltron.means import LogLinearMean
from voltron.rollout_utils import GeneratePrediction, Rollouts
from voltron.data import make_ticker_list, DataGetter
def main(args):
data_path = "../../voltron/data/"
ticker_file = "test_tickers.txt"
tckr_list = make_ticker_list(data_path + ticker_file)
print("Downloading Data.....")
if args.end_date.lower() == "none":
end_date = str(datetime.date.today())
else:
end_date = args.end_date
DataGetter(fpath = data_path, ticker_file=ticker_file, end_date=end_date)
print("Data Downloaded.")
use_cuda = torch.cuda.is_available()
ntest = 20
dt = 1./252
print("Producing Forecasts.....")
for tckr in tckr_list:
dat = pd.read_csv(data_path + tckr + ".csv")
train_x = torch.arange(dat.shape[0]-1) * dt
test_x = torch.arange(ntest) * dt + train_x[-1] + train_x[1]
train_y = torch.FloatTensor(dat.Close.to_numpy())
if use_cuda:
train_x = train_x.cuda()
test_x = test_x.cuda()
train_y = train_y.cuda()
train_iters=150
mean = 'ewma'
nsample = 1000
if args.kernel == "volt":
vol = LearnGPCV(train_x, train_y, train_iters=args.train_iters,
printing=False)
vmod, vlh = TrainVolModel(train_x, vol,
train_iters=args.train_iters, printing=False)
voltron, lh = TrainVoltMagpieModel(train_x, train_y[1:],
vmod, vlh, vol,
printing=False,
train_iters=args.train_iters,
k=300, mean_func=args.mean)
vmod.eval();
if args.mean in ['ewma', 'dewma', 'tewma']:
save_samples = Rollouts(train_x, train_y, test_x, voltron,
nsample=nsample)
else: ## VOLT + STANDARD MEAN
voltron.vol_model.eval()
predvol = voltron.vol_model(test_x).sample(torch.Size((nsample, ))).exp()
save_samples[idx, ::] = GeneratePrediction(train_x, train_y, test_x,
predvol, voltron).detach()
del predvol
del voltron, lh, vmod, vlh, vol
torch.cuda.empty_cache()
else:
kernel_possibilities = {"sm": SpectralMixtureKernel, "matern": MaternKernel, "rbf": RBFKernel}
kernel = kernel_possibilities[args.kernel.lower()]
if type(kernel) is not SpectralMixtureKernel:
kernel = ScaleKernel(kernel())
else:
kernel = kernel()
kernel.initialize_from_data_empspect(train_x, train_y.log())
train_y = train_y[1:]
model = SingleTaskGP(
train_x.view(-1,1),
train_y.log().reshape(-1, 1),
covar_module=kernel,
likelihood=GaussianLikelihood()
)
mean_name = args.mean.lower()
if mean_name == "loglinear":
model.mean_module = LogLinearMean(1)
model.mean_module.initialize_from_data(train_x, train_y.log())
elif mean_name == 'linear':
model.mean_module = LinearMean(1)
elif mean_name == "constant":
model.mean_module = ConstantMean()
elif mean_name == "ewma":
model.mean_module = EWMAMean(train_x, train_y.log(), k=args.k).to(train_x.device)
elif mean_name == "dewma":
model.mean_module = DEWMAMean(train_x, train_y.log(), k=args.k).to(train_x.device)
elif mean_name == "tewma":
model.mean_module = TEWMAMean(train_x, train_y.log(), k=args.k).to(train_x.device)
if use_cuda:
model = model.to(train_x.device)
mll = ExactMarginalLogLikelihood(model.likelihood, model)
fit_gpytorch_torch(mll, options={'maxiter':train_iters, 'disp':False})
if mean_name in ["loglinear", "constant", 'linear']:
save_samples[idx] = model.posterior(test_x).sample(torch.Size((nsample, ))).squeeze(-1).cpu().detach()
else:
save_samples[idx] = Rollouts(
train_x, train_y, test_x, model, nsample=nsample, method = "nonvol"
).cpu().detach()
torch.cuda.empty_cache()
del model, kernel
model_name = args.kernel + "_" + args.mean
savepath = "./saved-outputs/" + tckr + "/"
if not os.path.exists(savepath):
os.mkdir(savepath)
torch.save(save_samples, savepath + str(datetime.date.today()) + ".pt")
if args.printing:
print("\t" + tckr + " done.")
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument(
"--kernel",
type=str,
default="volt",
)
parser.add_argument(
"--mean",
type=str,
default="ewma",
)
parser.add_argument(
"--k",
type=int,
default=100,
)
parser.add_argument(
"--nsample",
type=int,
default=1000,
)
parser.add_argument(
"--printing",
type=bool,
default=False
)
parser.add_argument(
"--train_iters",
type=int,
default=400,
)
parser.add_argument(
"--end_date",
default="none",
)
args = parser.parse_args()
main(args)
experiments/trading/GenerateDaysPredictions.py
-138
View File
@@ -1,138 +0,0 @@
import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np
import torch
import pandas as pd
import os
import gpytorch
import argparse
import datetime
from botorch.models import SingleTaskGP
from botorch.optim.fit import fit_gpytorch_torch
from gpytorch.likelihoods import GaussianLikelihood
from gpytorch.mlls import ExactMarginalLogLikelihood
from gpytorch.means import ConstantMean, LinearMean
from gpytorch.kernels import SpectralMixtureKernel, MaternKernel, RBFKernel, ScaleKernel
from voltron.means import EWMAMean, DEWMAMean, TEWMAMean
from voltron.train_utils import LearnGPCV, TrainVolModel, TrainVoltMagpieModel, TrainBasicModel
from voltron.models import VoltMagpie
from voltron.means import LogLinearMean
from voltron.rollout_utils import GeneratePrediction, Rollouts
from voltron.data import make_ticker_list, DataGetter, GetStockHistory
def GenerateStockPredictions(ticker, dat,
forecast_horizon=20,
train_iters=400, nsample=1000,
ntrain=400, mean='ewma', kernel='volt',
save=False, k=300):
end_idxs = torch.arange(ntrain, dat.shape[0])
ntest = forecast_horizon
dt = 1./252
model_name = kernel + "_" + mean + str(k) + "_"
savepath = "./saved-outputs/" + ticker + "/"
for last_day in end_idxs:
date = str(dat.index[last_day.item()].date())
try:
train_y = torch.FloatTensor(dat.Close[last_day.item()-ntrain:last_day.item()].to_numpy())
train_x = torch.arange(train_y.shape[0]-1) * dt
test_x = torch.arange(ntest) * dt + train_x[-1] + train_x[1]
# try:
use_cuda = torch.cuda.is_available()
if use_cuda:
train_x = trin_x.cuda()
test_x = test_x.cuda()
train_y = train_y.cuda()
# print("Producing " + ticker + " Forecasts.....")
if kernel == "volt":
vol = LearnGPCV(train_x, train_y, train_iters=train_iters,
printing=False)
vmod, vlh = TrainVolModel(train_x, vol,
train_iters=train_iters, printing=False)
voltron, lh = TrainVoltMagpieModel(train_x, train_y[1:],
vmod, vlh, vol,
printing=False,
train_iters=train_iters,
k=k, mean_func=mean)
vmod.eval();
if mean in ['ewma', 'dewma', 'tewma']:
save_samples = Rollouts(train_x, train_y, test_x, voltron,
nsample=nsample)
else: ## VOLT + STANDARD MEAN
voltron.vol_model.eval()
predvol = voltron.vol_model(test_x).sample(torch.Size((nsample, ))).exp()
save_samples[idx, ::] = GeneratePrediction(train_x, train_y, test_x,
predvol, voltron).detach()
del predvol
del voltron, lh, vmod, vlh, vol
torch.cuda.empty_cache()
else:
kernel_possibilities = {"sm": SpectralMixtureKernel, "matern": MaternKernel, "rbf": RBFKernel}
kernel = kernel_possibilities[kernel.lower()]
if type(kernel) is not SpectralMixtureKernel:
kernel = ScaleKernel(kernel())
else:
kernel = kernel()
kernel.initialize_from_data_empspect(train_x, train_y.log())
train_y = train_y[1:]
model = SingleTaskGP(
train_x.view(-1,1),
train_y.log().reshape(-1, 1),
covar_module=kernel,
likelihood=GaussianLikelihood()
)
mean_name = mean.lower()
if mean_name == "loglinear":
model.mean_module = LogLinearMean(1)
model.mean_module.initialize_from_data(train_x, train_y.log())
elif mean_name == 'linear':
model.mean_module = LinearMean(1)
elif mean_name == "constant":
model.mean_module = ConstantMean()
elif mean_name == "ewma":
model.mean_module = EWMAMean(train_x, train_y.log(), k=k).to(train_x.device)
elif mean_name == "dewma":
model.mean_module = DEWMAMean(train_x, train_y.log(), k=k).to(train_x.device)
elif mean_name == "tewma":
model.mean_module = TEWMAMean(train_x, train_y.log(), k=k).to(train_x.device)
if use_cuda:
model = model.to(train_x.device)
mll = ExactMarginalLogLikelihood(model.likelihood, model)
fit_gpytorch_torch(mll, options={'maxiter':train_iters, 'disp':False})
if mean_name in ["loglinear", "constant", 'linear']:
save_samples[idx] = model.posterior(test_x).sample(torch.Size((nsample, ))).squeeze(-1).cpu().detach()
else:
save_samples[idx] = Rollouts(
train_x, train_y, test_x, model, nsample=nsample, method = "nonvol"
).cpu().detach()
torch.cuda.empty_cache()
del model, kernel
if save:
if not os.path.exists(savepath):
os.mkdir(savepath)
torch.save(save_samples, savepath + model_name + date + ".pt")
except:
nans = torch.ones(nsample, ntest) * torch.nan
if save:
if not os.path.exists(savepath):
os.mkdir(savepath)
torch.save(nans, savepath + model_name + date + ".pt")
return dat, save_samples
experiments/trading/Trader.py
-69
View File
@@ -1,69 +0,0 @@
import numpy as np
import torch
import pandas as pd
import gpytorch
import argparse
def ValueFunction(pred_samples, curr_px):
"""
pred_samples = (num samples) x (test times) matrix of forecast paths
curr_px = last observed price
"""
snr = (pred_samples.mean(0) - curr_px)/pred_samples.std(0)
if snr > 0.2:
return 1.
else:
return 0.
def main(args):
data_path = "../../voltron/data/"
ticker_file = "test_tickers.txt"
tckr_list = make_ticker_list(data_path + ticker_file)
if args.end_date
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument(
"--kernel",
type=str,
default="volt",
)
parser.add_argument(
"--mean",
type=str,
default="ewma",
)
parser.add_argument(
"--k",
type=int,
default=100,
)
parser.add_argument(
"--nsample",
type=int,
default=1000,
)
parser.add_argument(
"--printing",
type=bool,
default=False
)
parser.add_argument(
"--train_iters",
type=int,
default=400,
)
parser.add_argument(
"--end_date",
default="none",
)
args = parser.parse_args()
main(args)
experiments/trading/fbm_experiments.ipynb
-491
View File
File diff suppressed because one or more lines are too long
experiments/trading/greg_trading.ipynb
-395
View File
File diff suppressed because one or more lines are too long
experiments/trading/portfolio_management.ipynb
-1099
View File
File diff suppressed because one or more lines are too long
experiments/trading/shorter_time_scale.ipynb
-513
View File
File diff suppressed because one or more lines are too long
experiments/trading/simple-loader.ipynb
-104
View File
File diff suppressed because one or more lines are too long
experiments/trading/spy_template.py
-189
View File
@@ -1,189 +0,0 @@
import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np
import torch
import pandas as pd
import os
import gpytorch
import argparse
from botorch.models import SingleTaskGP
from botorch.optim.fit import fit_gpytorch_torch
from gpytorch.likelihoods import GaussianLikelihood
from gpytorch.mlls import ExactMarginalLogLikelihood
from gpytorch.means import ConstantMean, LinearMean
from gpytorch.kernels import SpectralMixtureKernel, MaternKernel, RBFKernel, ScaleKernel
import sys
sys.path.append("../../magpie/means/")
from EWMA import EWMAMean, DEWMAMean, TEWMAMean
sys.path.append("../../magpie/")
from train_utils import LearnGPCV, TrainVolModel, TrainVoltMagpieModel, TrainBasicModel
sys.path.append("../../magpie/models/")
from VoltMagpie import VoltMagpie
from voltron.means import LogLinearMean
sys.path.append("../spdr-forecasting/")
from rollout_utils import GeneratePrediction, Rollouts
def main(args):
savepath = "./saved-outputs/" + args.symbol + "/"
if not os.path.exists(savepath):
os.mkdir(savepath)
dpath = "../../magpie/data/"
full_data = pd.read_csv(dpath + args.symbol + ".csv")
full_data = pd.read_csv(dpath + args.symbol + ".csv")
full_data["Date"] = pd.to_datetime(full_data['Date'])
full_data = full_data.set_index(["Date"])
ntrain = 450
ntest = 20
nsample = args.nsample
train_iters = 500
train_x = torch.arange(ntrain) * 1./252
test_x = torch.arange(ntest) * 1./252 + train_x[-1] + train_x[1]
dt = train_x[1] - train_x[0]
start_date = pd.to_datetime("2011-11-21") - pd.Timedelta(ntrain, 'd')
px = torch.FloatTensor(full_data.loc[start_date:].Close).squeeze()
start_idxs = torch.arange(1, px.shape[0] - ntrain - ntest)
## save the typical stuff to use for plotting/validating ##
torch.save({"ntrain":ntrain, "ntest":ntest, "start_idxs":start_idxs},
"./saved-outputs/metadata.pt")
if torch.cuda.is_available():
use_cuda = True
train_x = train_x.cuda()
test_x = test_x.cuda()
else:
use_cuda = False
fname = args.kernel + "_" + args.mean + str(args.k) + ".pt"
save_samples = torch.zeros(start_idxs.numel(), nsample, ntest)
for idx, start_idx in enumerate(start_idxs):
train_y = px[start_idx-1:ntrain+start_idx].squeeze()
test_y = px[start_idx + ntrain:start_idx + ntrain+ntest].squeeze()
if use_cuda:
train_y = train_y.cuda()
test_y = test_y.cuda()
test_x = test_x.cuda()
if args.kernel.lower() == "volt":
dt = train_x[1] - train_x[0]
vol = LearnGPCV(train_x, train_y, train_iters=train_iters,
printing=False)
vmod, vlh = TrainVolModel(train_x, vol,
train_iters=train_iters, printing=False)
voltron, lh = TrainVoltMagpieModel(train_x, train_y[1:],
vmod, vlh, vol,
printing=False,
train_iters=train_iters,
k=args.k, mean_func=args.mean)
vmod.eval();
if args.mean in ['ewma', 'dewma', 'tewma']:
save_samples[idx, ::] = Rollouts(train_x, train_y, test_x, voltron,
nsample=nsample)
else: ## VOLT + STANDARD MEAN
voltron.vol_model.eval()
predvol = voltron.vol_model(test_x).sample(torch.Size((nsample, ))).exp()
save_samples[idx, ::] = GeneratePrediction(train_x, train_y, test_x,
predvol, voltron).detach()
del predvol
del voltron, lh, vmod, vlh, vol
else:
kernel_possibilities = {"sm": SpectralMixtureKernel, "matern": MaternKernel, "rbf": RBFKernel}
kernel = kernel_possibilities[args.kernel.lower()]
if type(kernel) is not SpectralMixtureKernel:
kernel = ScaleKernel(kernel())
else:
kernel = kernel()
kernel.initialize_from_data_empspect(train_x, train_y.log())
train_y = train_y[1:]
model = SingleTaskGP(
train_x.view(-1,1),
train_y.log().reshape(-1, 1),
covar_module=kernel,
likelihood=GaussianLikelihood()
)
mean_name = args.mean.lower()
if mean_name == "loglinear":
model.mean_module = LogLinearMean(1)
model.mean_module.initialize_from_data(train_x, train_y.log())
elif mean_name == 'linear':
model.mean_module = LinearMean(1)
elif mean_name == "constant":
model.mean_module = ConstantMean()
elif mean_name == "ewma":
model.mean_module = EWMAMean(train_x, train_y.log(), k=args.k).to(train_x.device)
elif mean_name == "dewma":
model.mean_module = DEWMAMean(train_x, train_y.log(), k=args.k).to(train_x.device)
elif mean_name == "tewma":
model.mean_module = TEWMAMean(train_x, train_y.log(), k=args.k).to(train_x.device)
if use_cuda:
model = model.to(train_x.device)
mll = ExactMarginalLogLikelihood(model.likelihood, model)
fit_gpytorch_torch(mll, options={'maxiter':train_iters, 'disp':False})
if mean_name in ["loglinear", "constant", 'linear']:
save_samples[idx] = model.posterior(test_x).sample(torch.Size((nsample, ))).squeeze(-1).cpu().detach()
else:
save_samples[idx] = Rollouts(
train_x, train_y, test_x, model, nsample=nsample, method = "nonvol"
).cpu().detach()
torch.cuda.empty_cache()
del model, kernel
print("Start Time = ", start_idx.item(), " out of ", len(start_idxs))
torch.cuda.empty_cache()
torch.save(save_samples, savepath + fname)
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument(
"--symbol",
type=str,
default="SPY",
)
parser.add_argument(
"--kernel",
type=str,
default="volt",
)
parser.add_argument(
"--mean",
type=str,
default="ewma",
)
parser.add_argument(
"--k",
type=int,
default=100,
)
parser.add_argument(
"--nsample",
type=int,
default=1000,
)
args = parser.parse_args()
main(args)
experiments/trading/trading_backtesting_with_active_management.ipynb
-1168
View File
File diff suppressed because one or more lines are too long
experiments/trading/trading_backtesting_with_active_management_v2-Copy1.ipynb
-1059
View File
File diff suppressed because one or more lines are too long
experiments/trading/trading_backtesting_with_active_management_v2-Copy2.ipynb
-1460
View File
File diff suppressed because one or more lines are too long
experiments/trading/trading_backtesting_with_active_management_v2-Copy3.ipynb
-1700
View File
File diff suppressed because one or more lines are too long
experiments/trading/trading_backtesting_with_active_management_v2.ipynb
-1010
View File
File diff suppressed because one or more lines are too long
experiments/trading/volt_const_runner.sh
-8
View File
@@ -1,8 +0,0 @@
#!/bin/bash
cat ../../voltron/data/nasdaq100.txt | while read line
do
for test_idx in {0..9}
do
python TickerSingleDayGenerator.py --kernel=volt --ntimes=25 --test_idx=${test_idx} --save=True --end_date="2022-01-12" --ticker=${line} --mean=constant
done
done

Some files were not shown because too many files have changed in this diff Show More