mirror of
https://github.com/wassname/catalyst.git
synced 2026-08-11 11:16:15 +08:00
ENH: Updates the cli, data bundles and extensions.
Adds the data bundle concept which makes it easy for users to register loading functions to build out minute and daily data along with an assets db and adjustments db. By default we have provided a `quandl` bundle which pulls from the public domain WIKI dataset. Users may register new bundles by decorating an ingest function with `zipline.data.bundles.register(<name>)`. This also provides a `yahoo_equities` function for creating an ingestion function that will load a static set of assets from yahoo. The cli is now structured as a couple of subcommands and has been changed to `python -m zipline`. The old behavior of `run_algo.py` has been moved to the `run` subcommand. This is almost entirely the same except that it now takes the name of the data bundle to use, defaulting to `quandl`. The next subcommand is `ingest` which takes the name of a data bundle to ingest. This will run the loading machinery and write the data to a specified location that `run` can find. There is also a `clean` subcommand which deletes the data that was written with `ingest`. Extensions have also been added to zipline. This is an experimental feature where users can provide an extra set of python files to run at the start of the process. These can be used to configure aspects of zipline. Right now the only thing that is supported in an extension file is the registration of a new data bundle.
This commit is contained in:
Joe Jevnik
+132
-228
@@ -51,53 +51,50 @@ My first algorithm
|
||||
Lets take a look at a very simple algorithm from the ``examples``
|
||||
directory, ``buyapple.py``:
|
||||
|
||||
.. code:: python
|
||||
.. code-block:: python
|
||||
|
||||
!tail ../../zipline/examples/buyapple.py
|
||||
from zipline.examples import buyapple
|
||||
buyapple??
|
||||
|
||||
|
||||
.. parsed-literal::
|
||||
.. code-block:: python
|
||||
|
||||
# Load price data from yahoo.
|
||||
data = load_from_yahoo(stocks=['AAPL'], indexes={}, start=start,
|
||||
end=end)
|
||||
from zipline.api import order, record, symbol
|
||||
|
||||
# Create and run the algorithm.
|
||||
algo = TradingAlgorithm(initialize=initialize, handle_data=handle_data,
|
||||
identifiers=['AAPL'])
|
||||
results = algo.run(data)
|
||||
|
||||
analyze(results=results)
|
||||
def initialize(context):
|
||||
pass
|
||||
|
||||
|
||||
def handle_data(context, data):
|
||||
order(symbol('AAPL'), 10)
|
||||
record(AAPL=data.current(symbol('AAPL'), 'price'))
|
||||
|
||||
|
||||
As you can see, we first have to import some functions we would like to
|
||||
use. All functions commonly used in your algorithm can be found in
|
||||
``zipline.api``. Here we are using ``order()`` which takes two arguments
|
||||
-- a security object, and a number specifying how many stocks you would
|
||||
like to order (if negative, ``order()`` will sell/short stocks). In this
|
||||
case we want to order 10 shares of Apple at each iteration. For more
|
||||
documentation on ``order()``, see the `Quantopian
|
||||
docs <https://www.quantopian.com/help#api-order>`__.
|
||||
``zipline.api``. Here we are using :func:`~zipline.api.order()` which takes two
|
||||
arguments: a security object, and a number specifying how many stocks you would
|
||||
like to order (if negative, :func:`~zipline.api.order()` will sell/short
|
||||
stocks). In this case we want to order 10 shares of Apple at each iteration. For
|
||||
more documentation on ``order()``, see the `Quantopian docs
|
||||
<https://www.quantopian.com/help#api-order>`__.
|
||||
|
||||
You don't have to use the ``symbol()`` function and could just pass in
|
||||
``AAPL`` directly but it is good practice as this way your code will be
|
||||
Quantopian compatible.
|
||||
|
||||
Finally, the ``record()`` function allows you to save the value of a
|
||||
variable at each iteration. You provide it with a name for the variable
|
||||
Finally, the :func:`~zipline.api.record` function allows you to save the value
|
||||
of a variable at each iteration. You provide it with a name for the variable
|
||||
together with the variable itself: ``varname=var``. After the algorithm
|
||||
finished running you will have access to each variable value you tracked
|
||||
with ``record()`` under the name you provided (we will see this further
|
||||
below). You also see how we can access the current price data of the
|
||||
with :func:`~zipline.api.record` under the name you provided (we will see this
|
||||
further below). You also see how we can access the current price data of the
|
||||
AAPL stock in the ``data`` event frame (for more information see
|
||||
`here <https://www.quantopian.com/help#api-event-properties>`__.
|
||||
|
||||
Running the algorithm
|
||||
~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
To now test this algorithm on financial data, ``zipline`` provides two
|
||||
interfaces. A command-line interface and an ``IPython Notebook``
|
||||
interface.
|
||||
To now test this algorithm on financial data, ``zipline`` provides three
|
||||
interfaces: A command-line interface, ``IPython Notebook`` magic, and
|
||||
:func:`~zipline.run_algorithm`.
|
||||
|
||||
Command line interface
|
||||
^^^^^^^^^^^^^^^^^^^^^^
|
||||
@@ -106,60 +103,59 @@ After you installed zipline you should be able to execute the following
|
||||
from your command line (e.g. ``cmd.exe`` on Windows, or the Terminal app
|
||||
on OSX):
|
||||
|
||||
.. code:: python
|
||||
|
||||
!run_algo.py --help
|
||||
.. code-block:: bash
|
||||
|
||||
$ python -m zipline run --help
|
||||
|
||||
.. parsed-literal::
|
||||
|
||||
usage: run_algo.py [-h] [-c FILE] [--algofile ALGOFILE] [--data-frequency {minute,daily}] [--start START] [--end END]
|
||||
[--capital_base CAPITAL_BASE] [--source {yahoo}] [--source_time_column SOURCE_TIME_COLUMN] [--symbols SYMBOLS]
|
||||
[--output OUTPUT] [--metadata_path METADATA_PATH] [--metadata_index METADATA_INDEX] [--print-algo] [--no-print-algo]
|
||||
Usage: __main__.py run [OPTIONS]
|
||||
|
||||
Zipline version 0.8.3.
|
||||
Run a backtest for the given algorithm.
|
||||
|
||||
optional arguments:
|
||||
-h, --help show this help message and exit
|
||||
-c FILE, --conf_file FILE
|
||||
Specify config file
|
||||
--algofile ALGOFILE, -f ALGOFILE
|
||||
--data-frequency {minute,daily}
|
||||
--start START, -s START
|
||||
--end END, -e END
|
||||
--capital_base CAPITAL_BASE
|
||||
--source {yahoo}, -d {yahoo}
|
||||
--source_time_column SOURCE_TIME_COLUMN, -t SOURCE_TIME_COLUMN
|
||||
--symbols SYMBOLS
|
||||
--output OUTPUT, -o OUTPUT
|
||||
--metadata_path METADATA_PATH, -m METADATA_PATH
|
||||
--metadata_index METADATA_INDEX, -x METADATA_INDEX
|
||||
--print-algo, -p
|
||||
--no-print-algo, -q
|
||||
Options:
|
||||
-f, --algofile FILENAME The file that contains the algorithm to run.
|
||||
-t, --algotext TEXT The algorithm script to run.
|
||||
-D, --define TEXT Define a name to be bound in the namespace
|
||||
before executing the algotext. For example
|
||||
'-Dname=value'. The value may be any python
|
||||
expression. These are evaluated in order so
|
||||
they may refer to previously defined names.
|
||||
--data-frequency [minute|daily]
|
||||
The data frequency of the simulation.
|
||||
[default: daily]
|
||||
--capital-base FLOAT The starting capital for the simulation.
|
||||
[default: 10000000.0]
|
||||
-b, --bundle BUNDLE-NAME The data bundle to use for the simulation.
|
||||
[default: quandl]
|
||||
--bundle-timestamp TIMESTAMP The date to lookup data on or before.
|
||||
[default: <current-time>]
|
||||
-s, --start DATE The start date of the simulation.
|
||||
-e, --end DATE The end date of the simulation.
|
||||
-o, --output FILENAME The location to write the perf data. If this
|
||||
is '-' the perf will be written to stdout.
|
||||
[default: -]
|
||||
--print-algo / --no-print-algo Print the algorithm to stdout.
|
||||
--help Show this message and exit.
|
||||
|
||||
|
||||
Note that you have to omit the preceding '!' when you call
|
||||
``run_algo.py``, this is only required by the IPython Notebook in which
|
||||
this tutorial was written.
|
||||
|
||||
As you can see there are a couple of flags that specify where to find
|
||||
your algorithm (``-f``) as well as parameters specifying which stock
|
||||
data to load from Yahoo! finance (``--symbols``) and the time-range
|
||||
(``--start`` and ``--end``). Finally, you'll want to save the
|
||||
performance metrics of your algorithm so that you can analyze how it
|
||||
performed. This is done via the ``--output`` flag and will cause it to
|
||||
write the performance ``DataFrame`` in the pickle Python file format.
|
||||
Note that you can also define a configuration file with these parameters
|
||||
that you can then conveniently pass to the ``-c`` option so that you
|
||||
don't have to supply the command line args all the time (see the .conf
|
||||
files in the examples directory).
|
||||
As you can see there are a couple of flags that specify where to find your
|
||||
algorithm (``-f``) as well as parameters specifying which data to use,
|
||||
defaulting to the :ref:`quandl-data-bundle`. There are also arguments for the
|
||||
date range to run the algorithm over (``--start`` and ``--end``). Finally,
|
||||
you'll want to save the performance metrics of your algorithm so that you can
|
||||
analyze how it performed. This is done via the ``--output`` flag and will cause
|
||||
it to write the performance ``DataFrame`` in the pickle Python file format.
|
||||
Note that you can also define a configuration file with these parameters that
|
||||
you can then conveniently pass to the ``-c`` option so that you don't have to
|
||||
supply the command line args all the time (see the .conf files in the examples
|
||||
directory).
|
||||
|
||||
Thus, to execute our algorithm from above and save the results to
|
||||
``buyapple_out.pickle`` we would call ``run_algo.py`` as follows:
|
||||
``buyapple_out.pickle`` we would call ``python -m zipline run`` as follows:
|
||||
|
||||
.. code:: python
|
||||
.. code-block:: python
|
||||
|
||||
!run_algo.py -f ../../zipline/examples/buyapple.py --start 2000-1-1 --end 2014-1-1 --symbols AAPL -o buyapple_out.pickle
|
||||
python -m zipline run -f ../../zipline/examples/buyapple.py --start 2000-1-1 --end 2014-1-1 --symbols AAPL -o buyapple_out.pickle
|
||||
|
||||
|
||||
.. parsed-literal::
|
||||
@@ -170,9 +166,7 @@ Thus, to execute our algorithm from above and save the results to
|
||||
[2015-11-04 22:45:32.820401] INFO: Performance: last close: 2013-12-31 21:00:00+00:00
|
||||
|
||||
|
||||
``run_algo.py`` first outputs the algorithm contents. It then fetches
|
||||
historical price and volume data of Apple from Yahoo! finance in the
|
||||
desired time range, calls the ``initialize()`` function, and then
|
||||
``run`` first calls the ``initialize()`` function, and then
|
||||
streams the historical stock price day-by-day through ``handle_data()``.
|
||||
After each call to ``handle_data()`` we instruct ``zipline`` to order 10
|
||||
stocks of AAPL. After the call of the ``order()`` function, ``zipline``
|
||||
@@ -187,31 +181,18 @@ slippage model that ``zipline`` uses, see the `Quantopian
|
||||
docs <https://www.quantopian.com/help#ide-slippage>`__ for more
|
||||
information).
|
||||
|
||||
Note that there is also an ``analyze()`` function printed.
|
||||
``run_algo.py`` will try and look for a file with the ending with
|
||||
``_analyze.py`` and the same name of the algorithm (so
|
||||
``buyapple_analyze.py``) or an ``analyze()`` function directly in the
|
||||
script. If an ``analyze()`` function is found it will be called *after*
|
||||
the simulation has finished and passed in the performance ``DataFrame``.
|
||||
(The reason for allowing specification of an ``analyze()`` function in a
|
||||
separate file is that this way ``buyapple.py`` remains a valid
|
||||
Quantopian algorithm that you can copy&paste to the platform).
|
||||
|
||||
Lets take a quick look at the performance ``DataFrame``. For this, we
|
||||
use ``pandas`` from inside the IPython Notebook and print the first ten
|
||||
rows. Note that ``zipline`` makes heavy usage of ``pandas``, especially
|
||||
for data input and outputting so it's worth spending some time to learn
|
||||
it.
|
||||
|
||||
.. code:: python
|
||||
.. code-block:: python
|
||||
|
||||
import pandas as pd
|
||||
perf = pd.read_pickle('buyapple_out.pickle') # read in perf DataFrame
|
||||
perf.head()
|
||||
|
||||
|
||||
|
||||
|
||||
.. raw:: html
|
||||
|
||||
<div style="max-height:1000px;max-width:1500px;overflow:auto;">
|
||||
@@ -378,7 +359,7 @@ and allows us to plot the price of apple. For example, we could easily
|
||||
examine now how our portfolio value changed over time compared to the
|
||||
AAPL stock price.
|
||||
|
||||
.. code:: python
|
||||
.. code-block:: python
|
||||
|
||||
%pylab inline
|
||||
figsize(12, 12)
|
||||
@@ -391,21 +372,14 @@ AAPL stock price.
|
||||
perf.AAPL.plot(ax=ax2)
|
||||
ax2.set_ylabel('AAPL stock price')
|
||||
|
||||
|
||||
.. parsed-literal::
|
||||
|
||||
Populating the interactive namespace from numpy and matplotlib
|
||||
|
||||
|
||||
|
||||
|
||||
.. parsed-literal::
|
||||
|
||||
<matplotlib.text.Text at 0x7ff5c6147f90>
|
||||
|
||||
|
||||
|
||||
|
||||
.. image:: tutorial_files/tutorial_11_2.png
|
||||
|
||||
|
||||
@@ -431,28 +405,21 @@ to run the algorithm from above with the same parameters we just have to
|
||||
execute the following cell after importing ``zipline`` to register the
|
||||
magic.
|
||||
|
||||
.. code:: python
|
||||
.. code-block:: python
|
||||
|
||||
import zipline
|
||||
%load_ext zipline
|
||||
|
||||
.. code:: python
|
||||
.. code-block:: python
|
||||
|
||||
%%zipline --start 2000-1-1 --end 2014-1-1 --symbols AAPL -o perf_ipython
|
||||
%%zipline --start 2000-1-1 --end 2014-1-1 --symbols AAPL
|
||||
from zipline.api import symbol, order, record
|
||||
|
||||
from zipline.api import symbol, order, record
|
||||
|
||||
def initialize(context):
|
||||
pass
|
||||
|
||||
def handle_data(context, data):
|
||||
order(symbol('AAPL'), 10)
|
||||
record(AAPL=data[symbol('AAPL')].price)
|
||||
|
||||
|
||||
.. parsed-literal::
|
||||
|
||||
AAPL
|
||||
def initialize(context):
|
||||
pass
|
||||
|
||||
def handle_data(context, data):
|
||||
order(symbol('AAPL'), 10)
|
||||
record(AAPL=data[symbol('AAPL')].price)
|
||||
|
||||
Note that we did not have to specify an input file as above since the
|
||||
magic will use the contents of the cell and look for your algorithm
|
||||
@@ -460,12 +427,9 @@ functions there. Also, instead of defining an output file we are
|
||||
specifying a variable name with ``-o`` that will be created in the name
|
||||
space and contain the performance ``DataFrame`` we looked at above.
|
||||
|
||||
.. code:: python
|
||||
|
||||
perf_ipython.head()
|
||||
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
_.head()
|
||||
|
||||
.. raw:: html
|
||||
|
||||
@@ -624,58 +588,6 @@ space and contain the performance ``DataFrame`` we looked at above.
|
||||
</div>
|
||||
|
||||
|
||||
|
||||
Manual (advanced)
|
||||
~~~~~~~~~~~~~~~~~
|
||||
|
||||
If you are happy with either way above you can safely skip this passage.
|
||||
To provide a closer look at how ``zipline`` actually works it is
|
||||
instructive to see how we run an algorithm without any of the interfaces
|
||||
demonstrated above which hide the actual ``zipline`` API.
|
||||
|
||||
.. code:: python
|
||||
|
||||
import pytz
|
||||
from datetime import datetime
|
||||
|
||||
from zipline.algorithm import TradingAlgorithm
|
||||
from zipline.utils.factory import load_bars_from_yahoo
|
||||
|
||||
# Load data manually from Yahoo! finance
|
||||
start = datetime(2000, 1, 1, 0, 0, 0, 0, pytz.utc)
|
||||
end = datetime(2012, 1, 1, 0, 0, 0, 0, pytz.utc)
|
||||
data = load_bars_from_yahoo(stocks=['AAPL'], start=start,
|
||||
end=end)
|
||||
|
||||
# Define algorithm
|
||||
def initialize(context):
|
||||
pass
|
||||
|
||||
def handle_data(context, data):
|
||||
order(symbol('AAPL'), 10)
|
||||
record(AAPL=data[symbol('AAPL')].price)
|
||||
|
||||
# Create algorithm object passing in initialize and
|
||||
# handle_data functions
|
||||
algo_obj = TradingAlgorithm(initialize=initialize,
|
||||
handle_data=handle_data)
|
||||
|
||||
# Run algorithm
|
||||
perf_manual = algo_obj.run(data)
|
||||
|
||||
|
||||
.. parsed-literal::
|
||||
|
||||
AAPL
|
||||
|
||||
|
||||
As you can see, we again define the functions as above but we manually
|
||||
pass them to the ``TradingAlgorithm`` class which is the main
|
||||
``zipline`` class for running algorithms. We also manually load the data
|
||||
using ``load_bars_from_yahoo()`` and pass it to the
|
||||
``TradingAlgorithm.run()`` method which kicks off the backtest
|
||||
simulation.
|
||||
|
||||
Access to previous prices using ``history``
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
@@ -706,81 +618,73 @@ you can directly use the ``history()`` function on Quantopian, in
|
||||
with ``add_history()`` and pass it the same arguments as the history
|
||||
function below. Lets look at the strategy which should make this clear:
|
||||
|
||||
.. code:: python
|
||||
.. code-block:: python
|
||||
|
||||
%%zipline --start 2000-1-1 --end 2014-1-1 --symbols AAPL -o perf_dma
|
||||
%%zipline --start 2000-1-1 --end 2014-1-1 -o perf_dma
|
||||
|
||||
|
||||
from zipline.api import order_target, record, symbol, history, add_history
|
||||
import numpy as np
|
||||
from zipline.api import order_target, record, symbol, history, add_history
|
||||
import numpy as np
|
||||
|
||||
def initialize(context):
|
||||
# Register 2 histories that track daily prices,
|
||||
# one with a 100 window and one with a 300 day window
|
||||
add_history(100, '1d', 'price')
|
||||
add_history(300, '1d', 'price')
|
||||
def initialize(context):
|
||||
# Register 2 histories that track daily prices,
|
||||
# one with a 100 window and one with a 300 day window
|
||||
add_history(100, '1d', 'price')
|
||||
add_history(300, '1d', 'price')
|
||||
|
||||
context.i = 0
|
||||
context.i = 0
|
||||
|
||||
|
||||
def handle_data(context, data):
|
||||
# Skip first 300 days to get full windows
|
||||
context.i += 1
|
||||
if context.i < 300:
|
||||
return
|
||||
def handle_data(context, data):
|
||||
# Skip first 300 days to get full windows
|
||||
context.i += 1
|
||||
if context.i < 300:
|
||||
return
|
||||
|
||||
# Compute averages
|
||||
# history() has to be called with the same params
|
||||
# from above and returns a pandas dataframe.
|
||||
short_mavg = history(100, '1d', 'price').mean()
|
||||
long_mavg = history(300, '1d', 'price').mean()
|
||||
# Compute averages
|
||||
# history() has to be called with the same params
|
||||
# from above and returns a pandas dataframe.
|
||||
short_mavg = history(100, '1d', 'price').mean()
|
||||
long_mavg = history(300, '1d', 'price').mean()
|
||||
|
||||
# Trading logic
|
||||
if short_mavg[0] > long_mavg[0]:
|
||||
# order_target orders as many shares as needed to
|
||||
# achieve the desired number of shares.
|
||||
order_target(symbol('AAPL'), 100)
|
||||
elif short_mavg[0] < long_mavg[0]:
|
||||
order_target(symbol('AAPL'), 0)
|
||||
# Trading logic
|
||||
if short_mavg[0] > long_mavg[0]:
|
||||
# order_target orders as many shares as needed to
|
||||
# achieve the desired number of shares.
|
||||
order_target(symbol('AAPL'), 100)
|
||||
elif short_mavg[0] < long_mavg[0]:
|
||||
order_target(symbol('AAPL'), 0)
|
||||
|
||||
# Save values for later inspection
|
||||
record(AAPL=data[symbol('AAPL')].price,
|
||||
short_mavg=short_mavg[0],
|
||||
long_mavg=long_mavg[0])
|
||||
# Save values for later inspection
|
||||
record(AAPL=data[symbol('AAPL')].price,
|
||||
short_mavg=short_mavg[0],
|
||||
long_mavg=long_mavg[0])
|
||||
|
||||
|
||||
def analyze(context, perf):
|
||||
fig = plt.figure()
|
||||
ax1 = fig.add_subplot(211)
|
||||
perf.portfolio_value.plot(ax=ax1)
|
||||
ax1.set_ylabel('portfolio value in $')
|
||||
|
||||
ax2 = fig.add_subplot(212)
|
||||
perf['AAPL'].plot(ax=ax2)
|
||||
perf[['short_mavg', 'long_mavg']].plot(ax=ax2)
|
||||
|
||||
perf_trans = perf.ix[[t != [] for t in perf.transactions]]
|
||||
buys = perf_trans.ix[[t[0]['amount'] > 0 for t in perf_trans.transactions]]
|
||||
sells = perf_trans.ix[
|
||||
[t[0]['amount'] < 0 for t in perf_trans.transactions]]
|
||||
ax2.plot(buys.index, perf.short_mavg.ix[buys.index],
|
||||
'^', markersize=10, color='m')
|
||||
ax2.plot(sells.index, perf.short_mavg.ix[sells.index],
|
||||
'v', markersize=10, color='k')
|
||||
ax2.set_ylabel('price in $')
|
||||
plt.legend(loc=0)
|
||||
plt.show()
|
||||
|
||||
|
||||
.. parsed-literal::
|
||||
|
||||
AAPL
|
||||
def analyze(context, perf):
|
||||
fig = plt.figure()
|
||||
ax1 = fig.add_subplot(211)
|
||||
perf.portfolio_value.plot(ax=ax1)
|
||||
ax1.set_ylabel('portfolio value in $')
|
||||
|
||||
ax2 = fig.add_subplot(212)
|
||||
perf['AAPL'].plot(ax=ax2)
|
||||
perf[['short_mavg', 'long_mavg']].plot(ax=ax2)
|
||||
|
||||
perf_trans = perf.ix[[t != [] for t in perf.transactions]]
|
||||
buys = perf_trans.ix[[t[0]['amount'] > 0 for t in perf_trans.transactions]]
|
||||
sells = perf_trans.ix[
|
||||
[t[0]['amount'] < 0 for t in perf_trans.transactions]]
|
||||
ax2.plot(buys.index, perf.short_mavg.ix[buys.index],
|
||||
'^', markersize=10, color='m')
|
||||
ax2.plot(sells.index, perf.short_mavg.ix[sells.index],
|
||||
'v', markersize=10, color='k')
|
||||
ax2.set_ylabel('price in $')
|
||||
plt.legend(loc=0)
|
||||
plt.show()
|
||||
|
||||
.. image:: tutorial_files/tutorial_22_1.png
|
||||
|
||||
|
||||
Here we are explicitly defining an ``analyze()`` function that gets
|
||||
automatically called once the backtest is done (this is not possible on
|
||||
Quantopian currently).
|
||||
|
||||
Reference in New Issue
Block a user