wassname/catalyst
1
0
Fork 0
mirror of https://github.com/wassname/catalyst.git synced 2026-09-10 11:50:32 +08:00
Code Issues Packages Projects Releases Wiki Activity

ENH: Updates the cli, data bundles and extensions.

Browse Source 
Adds the data bundle concept which makes it easy for users to register
loading functions to build out minute and daily data along with an
assets db and adjustments db. By default we have provided a `quandl`
bundle which pulls from the public domain WIKI dataset. Users may
register new bundles by decorating an ingest function with
`zipline.data.bundles.register(<name>)`. This also provides a
`yahoo_equities` function for creating an ingestion function that will
load a static set of assets from yahoo.

The cli is now structured as a couple of subcommands and has been
changed to `python -m zipline`. The old behavior of `run_algo.py` has
been moved to the `run` subcommand. This is almost entirely the same
except that it now takes the name of the data bundle to use, defaulting
to `quandl`.

The next subcommand is `ingest` which takes the name of
a data bundle to ingest. This will run the loading machinery and write
the data to a specified location that `run` can find.

There is also a `clean` subcommand which deletes the data that was
written with `ingest`.

Extensions have also been added to zipline. This is an experimental
feature where users can provide an extra set of python files to run at
the start of the process. These can be used to configure aspects of
zipline. Right now the only thing that is supported in an extension file
is the registration of a new data bundle.
This commit is contained in:
Joe Jevnik
2016-05-03 18:38:24 -04:00
parent efac476976
commit 59c8e371a2
70 changed files with 4444 additions and 1147 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/source/appendix.rst
+77 -13
View File
@@ -1,6 +1,11 @@
API Reference
-------------
Running a Backtest
~~~~~~~~~~~~~~~~~~
.. autofunction:: zipline.run_algorithm(...)
Algorithm API
~~~~~~~~~~~~~
@@ -85,29 +90,88 @@ Pipeline API
Asset Metadata
~~~~~~~~~~~~~~
.. autoclass:: zipline.assets.assets.Asset
.. autoclass:: zipline.assets.Asset
:members:
.. autoclass:: zipline.assets.assets.Equity
.. autoclass:: zipline.assets.Equity
:members:
.. autoclass:: zipline.assets.assets.Future
.. autoclass:: zipline.assets.Future
:members:
.. autoclass:: zipline.assets.assets.AssetFinder
:members:
.. autoclass:: zipline.assets.assets.AssetFinderCachedEquities
:members:
.. autoclass:: zipline.assets.asset_writer.AssetDBWriter
:members:
.. autoclass:: zipline.assets.assets.AssetConvertible
.. autoclass:: zipline.assets.AssetConvertible
:members:
Data API
~~~~~~~~
Writers
```````
.. autoclass:: zipline.data.minute_bars.BcolzMinuteBarWriter
:members:
.. autoclass:: zipline.data.us_equity_pricing.BcolzDailyBarWriter
:members:
.. autoclass:: zipline.data.us_equity_pricing.SQLiteAdjustmentWriter
:members:
.. autoclass:: zipline.assets.AssetDBWriter
:members:
Readers
```````
.. autoclass:: zipline.data.minute_bars.BcolzMinuteBarReader
:members:
.. autoclass:: zipline.data.us_equity_pricing.BcolzDailyBarReader
:members:
.. autoclass:: zipline.data.us_equity_pricing.SQLiteAdjustmentReader
:members:
.. autoclass:: zipline.assets.AssetFinder
:members:
.. autoclass:: zipline.assets.AssetFinderCachedEquities
:members:
Bundles
```````
.. autofunction:: zipline.data.bundles.register
.. autofunction:: zipline.data.bundles.ingest(name, environ=os.environ, date=None, show_progress=True)
.. autofunction:: zipline.data.bundles.load(name, environ=os.environ, date=None)
.. autofunction:: zipline.data.bundles.unregister
.. data:: zipline.data.bundles.bundles
The bundles that have been registered as a mapping from bundle name to bundle
data. This mapping is immutable and should only be updated through
:func:`~zipline.data.bundles.register` or
:func:`~zipline.data.bundles.unregister`.
.. autofunction:: zipline.data.bundles.yahoo_equities
Utilities
~~~~~~~~~
Caching
```````
.. autoclass:: zipline.utils.cache.CachedObject
.. autoclass:: zipline.utils.cache.ExpiringCache
.. autoclass:: zipline.utils.cache.dataframe_cache
.. autoclass:: zipline.utils.cache.working_file
.. autoclass:: zipline.utils.cache.working_dir
Command Line
````````````
.. autofunction:: zipline.utils.cli.maybe_show_progress
docs/source/beginner-tutorial.rst
+132 -228
View File
@@ -51,53 +51,50 @@ My first algorithm
Lets take a look at a very simple algorithm from the ``examples``
directory, ``buyapple.py``:
.. code:: python
.. code-block:: python
!tail ../../zipline/examples/buyapple.py
from zipline.examples import buyapple
buyapple??
.. parsed-literal::
.. code-block:: python
# Load price data from yahoo.
data = load_from_yahoo(stocks=['AAPL'], indexes={}, start=start,
end=end)
from zipline.api import order, record, symbol
# Create and run the algorithm.
algo = TradingAlgorithm(initialize=initialize, handle_data=handle_data,
identifiers=['AAPL'])
results = algo.run(data)
analyze(results=results)
def initialize(context):
pass
def handle_data(context, data):
order(symbol('AAPL'), 10)
record(AAPL=data.current(symbol('AAPL'), 'price'))
As you can see, we first have to import some functions we would like to
use. All functions commonly used in your algorithm can be found in
``zipline.api``. Here we are using ``order()`` which takes two arguments
-- a security object, and a number specifying how many stocks you would
like to order (if negative, ``order()`` will sell/short stocks). In this
case we want to order 10 shares of Apple at each iteration. For more
documentation on ``order()``, see the `Quantopian
docs <https://www.quantopian.com/help#api-order>`__.
``zipline.api``. Here we are using :func:`~zipline.api.order()` which takes two
arguments: a security object, and a number specifying how many stocks you would
like to order (if negative, :func:`~zipline.api.order()` will sell/short
stocks). In this case we want to order 10 shares of Apple at each iteration. For
more documentation on ``order()``, see the `Quantopian docs
<https://www.quantopian.com/help#api-order>`__.
You don't have to use the ``symbol()`` function and could just pass in
``AAPL`` directly but it is good practice as this way your code will be
Quantopian compatible.
Finally, the ``record()`` function allows you to save the value of a
variable at each iteration. You provide it with a name for the variable
Finally, the :func:`~zipline.api.record` function allows you to save the value
of a variable at each iteration. You provide it with a name for the variable
together with the variable itself: ``varname=var``. After the algorithm
finished running you will have access to each variable value you tracked
with ``record()`` under the name you provided (we will see this further
below). You also see how we can access the current price data of the
with :func:`~zipline.api.record` under the name you provided (we will see this
further below). You also see how we can access the current price data of the
AAPL stock in the ``data`` event frame (for more information see
`here <https://www.quantopian.com/help#api-event-properties>`__.
Running the algorithm
~~~~~~~~~~~~~~~~~~~~~
To now test this algorithm on financial data, ``zipline`` provides two
interfaces. A command-line interface and an ``IPython Notebook``
interface.
To now test this algorithm on financial data, ``zipline`` provides three
interfaces: A command-line interface, ``IPython Notebook`` magic, and
:func:`~zipline.run_algorithm`.
Command line interface
^^^^^^^^^^^^^^^^^^^^^^
@@ -106,60 +103,59 @@ After you installed zipline you should be able to execute the following
from your command line (e.g. ``cmd.exe`` on Windows, or the Terminal app
on OSX):
.. code:: python
!run_algo.py --help
.. code-block:: bash
$ python -m zipline run --help
.. parsed-literal::
usage: run_algo.py [-h] [-c FILE] [--algofile ALGOFILE] [--data-frequency {minute,daily}] [--start START] [--end END]
[--capital_base CAPITAL_BASE] [--source {yahoo}] [--source_time_column SOURCE_TIME_COLUMN] [--symbols SYMBOLS]
[--output OUTPUT] [--metadata_path METADATA_PATH] [--metadata_index METADATA_INDEX] [--print-algo] [--no-print-algo]
Usage: __main__.py run [OPTIONS]
Zipline version 0.8.3.
Run a backtest for the given algorithm.
optional arguments:
-h, --help show this help message and exit
-c FILE, --conf_file FILE
Specify config file
--algofile ALGOFILE, -f ALGOFILE
--data-frequency {minute,daily}
--start START, -s START
--end END, -e END
--capital_base CAPITAL_BASE
--source {yahoo}, -d {yahoo}
--source_time_column SOURCE_TIME_COLUMN, -t SOURCE_TIME_COLUMN
--symbols SYMBOLS
--output OUTPUT, -o OUTPUT
--metadata_path METADATA_PATH, -m METADATA_PATH
--metadata_index METADATA_INDEX, -x METADATA_INDEX
--print-algo, -p
--no-print-algo, -q
Options:
-f, --algofile FILENAME The file that contains the algorithm to run.
-t, --algotext TEXT The algorithm script to run.
-D, --define TEXT Define a name to be bound in the namespace
before executing the algotext. For example
'-Dname=value'. The value may be any python
expression. These are evaluated in order so
they may refer to previously defined names.
--data-frequency [minute|daily]
The data frequency of the simulation.
[default: daily]
--capital-base FLOAT The starting capital for the simulation.
[default: 10000000.0]
-b, --bundle BUNDLE-NAME The data bundle to use for the simulation.
[default: quandl]
--bundle-timestamp TIMESTAMP The date to lookup data on or before.
[default: <current-time>]
-s, --start DATE The start date of the simulation.
-e, --end DATE The end date of the simulation.
-o, --output FILENAME The location to write the perf data. If this
is '-' the perf will be written to stdout.
[default: -]
--print-algo / --no-print-algo Print the algorithm to stdout.
--help Show this message and exit.
Note that you have to omit the preceding '!' when you call
``run_algo.py``, this is only required by the IPython Notebook in which
this tutorial was written.
As you can see there are a couple of flags that specify where to find
your algorithm (``-f``) as well as parameters specifying which stock
data to load from Yahoo! finance (``--symbols``) and the time-range
(``--start`` and ``--end``). Finally, you'll want to save the
performance metrics of your algorithm so that you can analyze how it
performed. This is done via the ``--output`` flag and will cause it to
write the performance ``DataFrame`` in the pickle Python file format.
Note that you can also define a configuration file with these parameters
that you can then conveniently pass to the ``-c`` option so that you
don't have to supply the command line args all the time (see the .conf
files in the examples directory).
As you can see there are a couple of flags that specify where to find your
algorithm (``-f``) as well as parameters specifying which data to use,
defaulting to the :ref:`quandl-data-bundle`. There are also arguments for the
date range to run the algorithm over (``--start`` and ``--end``). Finally,
you'll want to save the performance metrics of your algorithm so that you can
analyze how it performed. This is done via the ``--output`` flag and will cause
it to write the performance ``DataFrame`` in the pickle Python file format.
Note that you can also define a configuration file with these parameters that
you can then conveniently pass to the ``-c`` option so that you don't have to
supply the command line args all the time (see the .conf files in the examples
directory).
Thus, to execute our algorithm from above and save the results to
``buyapple_out.pickle`` we would call ``run_algo.py`` as follows:
``buyapple_out.pickle`` we would call ``python -m zipline run`` as follows:
.. code:: python
.. code-block:: python
!run_algo.py -f ../../zipline/examples/buyapple.py --start 2000-1-1 --end 2014-1-1 --symbols AAPL -o buyapple_out.pickle
python -m zipline run -f ../../zipline/examples/buyapple.py --start 2000-1-1 --end 2014-1-1 --symbols AAPL -o buyapple_out.pickle
.. parsed-literal::
@@ -170,9 +166,7 @@ Thus, to execute our algorithm from above and save the results to
[2015-11-04 22:45:32.820401] INFO: Performance: last close: 2013-12-31 21:00:00+00:00
``run_algo.py`` first outputs the algorithm contents. It then fetches
historical price and volume data of Apple from Yahoo! finance in the
desired time range, calls the ``initialize()`` function, and then
``run`` first calls the ``initialize()`` function, and then
streams the historical stock price day-by-day through ``handle_data()``.
After each call to ``handle_data()`` we instruct ``zipline`` to order 10
stocks of AAPL. After the call of the ``order()`` function, ``zipline``
@@ -187,31 +181,18 @@ slippage model that ``zipline`` uses, see the `Quantopian
docs <https://www.quantopian.com/help#ide-slippage>`__ for more
information).
Note that there is also an ``analyze()`` function printed.
``run_algo.py`` will try and look for a file with the ending with
``_analyze.py`` and the same name of the algorithm (so
``buyapple_analyze.py``) or an ``analyze()`` function directly in the
script. If an ``analyze()`` function is found it will be called *after*
the simulation has finished and passed in the performance ``DataFrame``.
(The reason for allowing specification of an ``analyze()`` function in a
separate file is that this way ``buyapple.py`` remains a valid
Quantopian algorithm that you can copy&paste to the platform).
Lets take a quick look at the performance ``DataFrame``. For this, we
use ``pandas`` from inside the IPython Notebook and print the first ten
rows. Note that ``zipline`` makes heavy usage of ``pandas``, especially
for data input and outputting so it's worth spending some time to learn
it.
.. code:: python
.. code-block:: python
import pandas as pd
perf = pd.read_pickle('buyapple_out.pickle') # read in perf DataFrame
perf.head()
.. raw:: html
<div style="max-height:1000px;max-width:1500px;overflow:auto;">
@@ -378,7 +359,7 @@ and allows us to plot the price of apple. For example, we could easily
examine now how our portfolio value changed over time compared to the
AAPL stock price.
.. code:: python
.. code-block:: python
%pylab inline
figsize(12, 12)
@@ -391,21 +372,14 @@ AAPL stock price.
perf.AAPL.plot(ax=ax2)
ax2.set_ylabel('AAPL stock price')
.. parsed-literal::
Populating the interactive namespace from numpy and matplotlib
.. parsed-literal::
<matplotlib.text.Text at 0x7ff5c6147f90>
.. image:: tutorial_files/tutorial_11_2.png
@@ -431,28 +405,21 @@ to run the algorithm from above with the same parameters we just have to
execute the following cell after importing ``zipline`` to register the
magic.
.. code:: python
.. code-block:: python
import zipline
%load_ext zipline
.. code:: python
.. code-block:: python
%%zipline --start 2000-1-1 --end 2014-1-1 --symbols AAPL -o perf_ipython
%%zipline --start 2000-1-1 --end 2014-1-1 --symbols AAPL
from zipline.api import symbol, order, record
from zipline.api import symbol, order, record
def initialize(context):
pass
def handle_data(context, data):
order(symbol('AAPL'), 10)
record(AAPL=data[symbol('AAPL')].price)
.. parsed-literal::
AAPL
def initialize(context):
pass
def handle_data(context, data):
order(symbol('AAPL'), 10)
record(AAPL=data[symbol('AAPL')].price)
Note that we did not have to specify an input file as above since the
magic will use the contents of the cell and look for your algorithm
@@ -460,12 +427,9 @@ functions there. Also, instead of defining an output file we are
specifying a variable name with ``-o`` that will be created in the name
space and contain the performance ``DataFrame`` we looked at above.
.. code:: python
perf_ipython.head()
.. code-block:: python
_.head()
.. raw:: html
@@ -624,58 +588,6 @@ space and contain the performance ``DataFrame`` we looked at above.
</div>
Manual (advanced)
~~~~~~~~~~~~~~~~~
If you are happy with either way above you can safely skip this passage.
To provide a closer look at how ``zipline`` actually works it is
instructive to see how we run an algorithm without any of the interfaces
demonstrated above which hide the actual ``zipline`` API.
.. code:: python
import pytz
from datetime import datetime
from zipline.algorithm import TradingAlgorithm
from zipline.utils.factory import load_bars_from_yahoo
# Load data manually from Yahoo! finance
start = datetime(2000, 1, 1, 0, 0, 0, 0, pytz.utc)
end = datetime(2012, 1, 1, 0, 0, 0, 0, pytz.utc)
data = load_bars_from_yahoo(stocks=['AAPL'], start=start,
end=end)
# Define algorithm
def initialize(context):
pass
def handle_data(context, data):
order(symbol('AAPL'), 10)
record(AAPL=data[symbol('AAPL')].price)
# Create algorithm object passing in initialize and
# handle_data functions
algo_obj = TradingAlgorithm(initialize=initialize,
handle_data=handle_data)
# Run algorithm
perf_manual = algo_obj.run(data)
.. parsed-literal::
AAPL
As you can see, we again define the functions as above but we manually
pass them to the ``TradingAlgorithm`` class which is the main
``zipline`` class for running algorithms. We also manually load the data
using ``load_bars_from_yahoo()`` and pass it to the
``TradingAlgorithm.run()`` method which kicks off the backtest
simulation.
Access to previous prices using ``history``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -706,81 +618,73 @@ you can directly use the ``history()`` function on Quantopian, in
with ``add_history()`` and pass it the same arguments as the history
function below. Lets look at the strategy which should make this clear:
.. code:: python
.. code-block:: python
%%zipline --start 2000-1-1 --end 2014-1-1 --symbols AAPL -o perf_dma
%%zipline --start 2000-1-1 --end 2014-1-1 -o perf_dma
from zipline.api import order_target, record, symbol, history, add_history
import numpy as np
from zipline.api import order_target, record, symbol, history, add_history
import numpy as np
def initialize(context):
# Register 2 histories that track daily prices,
# one with a 100 window and one with a 300 day window
add_history(100, '1d', 'price')
add_history(300, '1d', 'price')
def initialize(context):
# Register 2 histories that track daily prices,
# one with a 100 window and one with a 300 day window
add_history(100, '1d', 'price')
add_history(300, '1d', 'price')
context.i = 0
context.i = 0
def handle_data(context, data):
# Skip first 300 days to get full windows
context.i += 1
if context.i < 300:
return
def handle_data(context, data):
# Skip first 300 days to get full windows
context.i += 1
if context.i < 300:
return
# Compute averages
# history() has to be called with the same params
# from above and returns a pandas dataframe.
short_mavg = history(100, '1d', 'price').mean()
long_mavg = history(300, '1d', 'price').mean()
# Compute averages
# history() has to be called with the same params
# from above and returns a pandas dataframe.
short_mavg = history(100, '1d', 'price').mean()
long_mavg = history(300, '1d', 'price').mean()
# Trading logic
if short_mavg[0] > long_mavg[0]:
# order_target orders as many shares as needed to
# achieve the desired number of shares.
order_target(symbol('AAPL'), 100)
elif short_mavg[0] < long_mavg[0]:
order_target(symbol('AAPL'), 0)
# Trading logic
if short_mavg[0] > long_mavg[0]:
# order_target orders as many shares as needed to
# achieve the desired number of shares.
order_target(symbol('AAPL'), 100)
elif short_mavg[0] < long_mavg[0]:
order_target(symbol('AAPL'), 0)
# Save values for later inspection
record(AAPL=data[symbol('AAPL')].price,
short_mavg=short_mavg[0],
long_mavg=long_mavg[0])
# Save values for later inspection
record(AAPL=data[symbol('AAPL')].price,
short_mavg=short_mavg[0],
long_mavg=long_mavg[0])
def analyze(context, perf):
fig = plt.figure()
ax1 = fig.add_subplot(211)
perf.portfolio_value.plot(ax=ax1)
ax1.set_ylabel('portfolio value in $')
ax2 = fig.add_subplot(212)
perf['AAPL'].plot(ax=ax2)
perf[['short_mavg', 'long_mavg']].plot(ax=ax2)
perf_trans = perf.ix[[t != [] for t in perf.transactions]]
buys = perf_trans.ix[[t[0]['amount'] > 0 for t in perf_trans.transactions]]
sells = perf_trans.ix[
[t[0]['amount'] < 0 for t in perf_trans.transactions]]
ax2.plot(buys.index, perf.short_mavg.ix[buys.index],
'^', markersize=10, color='m')
ax2.plot(sells.index, perf.short_mavg.ix[sells.index],
'v', markersize=10, color='k')
ax2.set_ylabel('price in $')
plt.legend(loc=0)
plt.show()
.. parsed-literal::
AAPL
def analyze(context, perf):
fig = plt.figure()
ax1 = fig.add_subplot(211)
perf.portfolio_value.plot(ax=ax1)
ax1.set_ylabel('portfolio value in $')
ax2 = fig.add_subplot(212)
perf['AAPL'].plot(ax=ax2)
perf[['short_mavg', 'long_mavg']].plot(ax=ax2)
perf_trans = perf.ix[[t != [] for t in perf.transactions]]
buys = perf_trans.ix[[t[0]['amount'] > 0 for t in perf_trans.transactions]]
sells = perf_trans.ix[
[t[0]['amount'] < 0 for t in perf_trans.transactions]]
ax2.plot(buys.index, perf.short_mavg.ix[buys.index],
'^', markersize=10, color='m')
ax2.plot(sells.index, perf.short_mavg.ix[sells.index],
'v', markersize=10, color='k')
ax2.set_ylabel('price in $')
plt.legend(loc=0)
plt.show()
.. image:: tutorial_files/tutorial_22_1.png
Here we are explicitly defining an ``analyze()`` function that gets
automatically called once the backtest is done (this is not possible on
Quantopian currently).
docs/source/bundles.rst
+277
View File
@@ -0,0 +1,277 @@
Data Bundles
------------
A data bundle is a collection of pricing data, adjustment data, and an asset
database. Bundles allow us to preload all of the data we will need to run
backtests and store the data for future runs.
Ingesting Data
~~~~~~~~~~~~~~
The first step to using a data bundle is to ingest the data. This will invoke
some custom bundle command and then write the data to a standard location that
zipline can find. By default this location is ``$ZIPLINE_ROOT/data/<bundle>``
where by default ``ZIPLINE_ROOT=~/.zipline``. This step may take some time as it
could involve downloading and processing a lot of data. This can be run with:
.. code-block:: bash
$ python -m zipline ingest <bundle>
where ``<bundle>`` is the name of the bundle to ingest.
Old Data
~~~~~~~~
When the ``ingest`` command is used it will write the new data to a subdirectory
of ``$ZIPLINE_ROOT/data/<bundle>`` which is named with the current date. This
makes it possible to look at older data or even run backtests with this older
copy. This makers it easier to reproduce backtest results later.
One drawback of saving all of this data by default is that the data directory
may grow quite large even if you do not want to use the data. To solve this
problem there is another command ``clean`` which will clear data bundles based
on some time constraints.
For example:
.. code-block:: bash
# clean everything older than <date>
$ python -m zipline clean <bundle> --before <date>
# clean everything newer than <date>
$ python -m zipline clean <bundle> --after <date>
# keep everything in the range of [before, after] and delete the rest
$ python -m zipline clean <bundle> --before <date> --after <after>
# clean all but the last <int> runs
$ python -m zipline clean <bundle> --keep-last <int>
Running Backtests with Data Bundles
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Now that the data has been ingested we can use it to run backtests with the
``run`` command. This can be specified with the ``--bundle`` option like:
.. code-block:: bash
$ python -m zipline run --bundle <bundle> --algofile algo.py ...
We may also specify the date to use to look up the bundle data with the
``--bundle-date`` option. This will cause us to the the most recent bundle
ingestion that is less than or equal to the ``bundle-date``. This is how we can
run backtests with older data. The reason that this uses a less than or equal to
relationship is that we can specify the date that we ran an old backtest and get
the same data that would have been available to us on that date. The
``bundle-date`` defaults to the current day to use the most recent data.
Default Data Bundles
~~~~~~~~~~~~~~~~~~~~
.. _quandl-data-bundle:
Quandl WIKI Bundle
``````````````````
By default zipline comes with the ``quandl`` data bundle which uses quandl's
`WIKI dataset <https://www.quandl.com/data/WIKI>`_. The quandl data bundle
includes daily pricing data, splits, cash dividends, and asset metadata. This is
the bundle that ``run`` will use by default if no other bundle is specified. To
ingest this data bundle we recommend creating an account on quandl.com to get an
API key to be able to make more API requests per day. Once we have an API key we
may run:
.. code-block:: bash
$ QUANDL_API_KEY=<api-key> python -m zipline ingest quandl
though we may still run ``ingest`` as an anonymous quandl user (with no API
key). We may also set the ``QUANDL_DOWNLOAD_ATTEMPTS`` environment variable to
an integer which is the number of attempts that should be made to download data
from quandls servers. By default this will be 5, meaning that we will retry each
attempt 5 times.
.. note::
``QUANDL_DOWNLOAD_ATTEMPTS`` is not the total number of allowed failures,
just the number of allowed failures per request. The quandl loader will make
one request per 100 equities for the metadata followed by one request per
equity.
Yahoo Bundle Factories
``````````````````````
Zipline also ships with a factory function for creating a data bundle out of a
set of tickers from yahoo: :func:`~zipline.data.bundles.yahoo_equities`.
This makes it easy to pre-download and cache the data for a set of equities from
yahoo. This includes daily pricing data along with splits, cash dividends, and
inferred asset metadata. To create a bundle from a set of equities, add the
following to your ``~/.zipline/extensions.py`` file:
.. code-block:: python
from zipline.bundles import register, yahoo_equities
# these are the tickers you would like data for
equities = {
'AAPL',
'MSFT',
'GOOG',
}
register(
'my-yahoo-equities-bundle', # name this whatever you like
yahoo_equities(equities),
)
This may now be used like:
.. code-block:: bash
$ python -m zipline ingest my-yahoo-equities-bundle
$ python -m zipline run -f algo.py --bundle my-yahoo-equities-bundle
More than one yahoo equities bundle may be registered as long as they use
different names.
Writing a New Bundle
~~~~~~~~~~~~~~~~~~~~
Data bundles exist to make it easy to use different data sources with
zipline. To add a new bundle, one must implement an ingest function.
This function is responsible for loading the data into memory and passing it to
a set of writer objects provided by zipline to convert the data to zipline's
internal format. The ingest function may work by downloading data from a remote
location like the ``quandl`` bundle or yahoo bundles or it may just load files
that are already on the machine. The function is provided with writers that will
write the data to the correct location transactionally. If an ingestion fails
part way through the bundle will not be written in an incomplete state.
The signature of the ingest function should be:
.. code-block:: python
ingest(environ,
asset_db_writer,
minute_bar_writer,
daily_bar_writer,
adjustment_writer,
calendar,
cache,
show_progress)
``environ``
```````````
``environ`` is a mapping representing the environment variables to use. This is
where any custom arguments needed for the ingestion should be passed, for
example: the ``quandl`` bundle uses the enviornment to pass the API key and the
download retry attempt count.
``asset_db_writer``
```````````````````
``asset_db_writer`` is an instance of :class:`~zipline.assets.AssetDBWriter`.
This is the writer for the asset metadata which provides the asset lifetimes and
the symbol to asset id (sid) mapping. This may also contain the asset name,
exchange and a few other columns. To write data, invoke
:meth:`~zipline.assets.AssetDBWriter.write` with dataframes for the various
pieces of metadata. More information about the format of the data exists in the
docs for write.
``minute_bar_writer``
`````````````````````
``minute_bar_writer`` is an instance of
:class:`~zipline.data.minute_bars.BcolzMinuteBarWriter`. This writer is used to
convert data to zipline's internal bcolz format to later be read by a
:class:`~zipline.data.minute_bars.BcolzMinuteBarReader`. If minute data is
provided, users should call
:meth:`~zipline.data.minute_bars.BcolzMinuteBarWriter.write` with an iterable of
(sid, dataframe) tuples. The ``show_progress`` argument should also be forwarded
to this method. If the data source does not provide minute level data, then
there is no need to call the write method. It is also acceptable to pass an
empty iterator to :meth:`~zipline.data.minute_bars.BcolzMinuteBarWriter.write`
to signal that there is no minutely data.
.. note::
The data passed to
:meth:`~zipline.data.minute_bars.BcolzMinuteBarWriter.write` may be a lazy
iterator or generator to avoid loading all of the minute data into memory at
a single time. A given sid may also appear multiple times in the data as long
as the dates are strictly increasing.
``daily_bar_writer``
````````````````````
``daily_bar_writer`` is an instance of
:class:`~zipline.data.us_equity_pricing.BcolzDailyBarWriter`. This writer is
used to convert data into zipline's internal bcolz format to later be read by a
:class:`~zipline.data.us_equity_pricing.BcolzDailyBarReader`. If daily data is
provided, users should call
:meth:`~zipline.data.minute_bars.BcolzDailyBarWriter.write` with an iterable of
(sid dataframe) tuples. The ``show_progress`` argument should also be forwarded
to this method. If the data shource does not provide daily data, then there is
no need to call the write method. It is also acceptable to pass an empty
iterable to :meth:`~zipline.data.minute_bars.BcolzMinuteBarWriter.write` to
signal that there is no daily data. If no daily data is provided but minute data
is provided, a daily rollup will happen to service daily history requests.
.. note::
Like the ``minute_bar_writer``, the data passed to
:meth:`~zipline.data.minute_bars.BcolzMinuteBarWriter.write` may be a lazy
iterable or generator to avoid loading all of the data into memory at once.
Unlike the ``minute_bar_writer``, a sid may only appear once in the data
iterable.
``adjustment_writer``
`````````````````````
``adjustment_writer`` is an instance of
:class:`~zipline.data.us_equity_pricing.SQLiteAdjustmentWriter`. This writer is
used to store splits, mergers, dividends, and stock dividends. The data should
be provided as dataframes and passed to
:meth:`~zipline.data.us_equity_pricing.SQLiteAdjustmentWriter.write`. Each of
these fields are optional, but the writer can accept as much of the data as you
have.
``calendar``
````````````
``calendar`` is a ``pandas.DatetimeIndex`` object holding all of the trading
days that the bundle should load data for. This is to help some bundles generate
queries for the days needed.
``cache``
`````````
``cache`` is an instance of :class:`~zipline.utils.cache.dataframe_cache`. This
object is a mapping from strings to dataframes. This object is provided in case
an ingestion crashes part way through. The idea is that the ingest function
should check the cache for raw data, if it doesn't exist in the cache, it should
acquire it and then store it in the cache. Then it can parse and write the
data. The cache will be cleared only after a successful load, this prevents the
ingest function from needing to redownload all the data if there is some bug in
the parsing. If it is very fast to get the data, for example if it is coming
from another local file, then there is no need to use this cache.
``show_progress``
`````````````````
``show_progress`` is a boolean indicating that the user would like to receive
feedback about the ingest function's progress fetching and writing the
data. Some examples for where to show how many files you have downloaded out of
the total needed, or how far into some data conversion the ingest function
is. One tool that may help with implementing ``show_progress`` for a loop is
:class:`~zipline.utils.cli.maybe_show_progress`. This argument should always be
forwarded to ``minute_bar_writer.write`` and ``daily_bar_writer.write``.
docs/source/index.rst
+1
View File
@@ -5,6 +5,7 @@
install
beginner-tutorial
bundles
releases
appendix
release-process
docs/source/whatsnew/1.0.0.txt
+45 -1
View File
@@ -12,7 +12,51 @@ Development
Highlights
~~~~~~~~~~
None
New Entry Points (:issue:`xxxx`)
````````````````````````````````
In order to make it easier to use zipline we have updated the entry points for
a backtest. The three supported ways to run a backtest are now:
1. :func:`zipline.run_algo`
2. ``$ python -m zipline run``
3. ``%zipline`` (IPython magic)
Data Bundles (:issue:`xxxx`)
````````````````````````````
1.0.0 introduces data bundles. Data bundles are groups of data that should be
preloaded and used to run backtests later. This allows users to not need to to
specify which tickers they are interested in each time they run an
algorithm. This also allows us to cache the data between runs.
By default, the ``quandl`` bundle will be used which pulls data from quandl's
`WIKI dataset <https://www.quandl.com/data/WIKI>`_. New bundles may be
registered with :func:`zipline.data.bundles.register` like:
.. code-block:: python
@zipline.data.bundles.register('my-new-bundle')
def my_new_bundle_ingest(environ,
asset_db_writer,
minute_bar_writer,
daily_bar_writer,
adjustment_writer,
calendar,
cache,
show_progress):
...
This function should retrieve the data it needs and then use the writers that
have been passed to write that data to disc in a location that zipline can find
later.
This data can be used in backtests by passing the name as the ``-b / --bundle``
argument to ``$ python -m zipline run`` or as the ``bundle`` argument to
:func:`zipline.run_algo`.
For more information see `Data Bundles`_ for more information.
Enhancements
~~~~~~~~~~~~