wassname/catalyst
1
0
Fork 0
mirror of https://github.com/wassname/catalyst.git synced 2026-07-11 11:40:40 +08:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #1173 from quantopian/quandl-wiki-loader

Browse Source 
Quandl wiki loader
This commit is contained in:
Joe Jevnik
2016-05-03 19:11:18 -04:00
parent 21bc598d85 59c8e371a2
commit f3e436a1bf
77 changed files with 4609 additions and 1310 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/source/appendix.rst
+77 -13
View File
@@ -1,6 +1,11 @@
API Reference
-------------
Running a Backtest
~~~~~~~~~~~~~~~~~~
.. autofunction:: zipline.run_algorithm(...)
Algorithm API
~~~~~~~~~~~~~
@@ -85,29 +90,88 @@ Pipeline API
Asset Metadata
~~~~~~~~~~~~~~
.. autoclass:: zipline.assets.assets.Asset
.. autoclass:: zipline.assets.Asset
:members:
.. autoclass:: zipline.assets.assets.Equity
.. autoclass:: zipline.assets.Equity
:members:
.. autoclass:: zipline.assets.assets.Future
.. autoclass:: zipline.assets.Future
:members:
.. autoclass:: zipline.assets.assets.AssetFinder
:members:
.. autoclass:: zipline.assets.assets.AssetFinderCachedEquities
:members:
.. autoclass:: zipline.assets.asset_writer.AssetDBWriter
:members:
.. autoclass:: zipline.assets.assets.AssetConvertible
.. autoclass:: zipline.assets.AssetConvertible
:members:
Data API
~~~~~~~~
Writers
```````
.. autoclass:: zipline.data.minute_bars.BcolzMinuteBarWriter
:members:
.. autoclass:: zipline.data.us_equity_pricing.BcolzDailyBarWriter
:members:
.. autoclass:: zipline.data.us_equity_pricing.SQLiteAdjustmentWriter
:members:
.. autoclass:: zipline.assets.AssetDBWriter
:members:
Readers
```````
.. autoclass:: zipline.data.minute_bars.BcolzMinuteBarReader
:members:
.. autoclass:: zipline.data.us_equity_pricing.BcolzDailyBarReader
:members:
.. autoclass:: zipline.data.us_equity_pricing.SQLiteAdjustmentReader
:members:
.. autoclass:: zipline.assets.AssetFinder
:members:
.. autoclass:: zipline.assets.AssetFinderCachedEquities
:members:
Bundles
```````
.. autofunction:: zipline.data.bundles.register
.. autofunction:: zipline.data.bundles.ingest(name, environ=os.environ, date=None, show_progress=True)
.. autofunction:: zipline.data.bundles.load(name, environ=os.environ, date=None)
.. autofunction:: zipline.data.bundles.unregister
.. data:: zipline.data.bundles.bundles
The bundles that have been registered as a mapping from bundle name to bundle
data. This mapping is immutable and should only be updated through
:func:`~zipline.data.bundles.register` or
:func:`~zipline.data.bundles.unregister`.
.. autofunction:: zipline.data.bundles.yahoo_equities
Utilities
~~~~~~~~~
Caching
```````
.. autoclass:: zipline.utils.cache.CachedObject
.. autoclass:: zipline.utils.cache.ExpiringCache
.. autoclass:: zipline.utils.cache.dataframe_cache
.. autoclass:: zipline.utils.cache.working_file
.. autoclass:: zipline.utils.cache.working_dir
Command Line
````````````
.. autofunction:: zipline.utils.cli.maybe_show_progress
docs/source/beginner-tutorial.rst
+132 -228
View File
@@ -51,53 +51,50 @@ My first algorithm
Lets take a look at a very simple algorithm from the ``examples``
directory, ``buyapple.py``:
.. code:: python
.. code-block:: python
!tail ../../zipline/examples/buyapple.py
from zipline.examples import buyapple
buyapple??
.. parsed-literal::
.. code-block:: python
# Load price data from yahoo.
data = load_from_yahoo(stocks=['AAPL'], indexes={}, start=start,
end=end)
from zipline.api import order, record, symbol
# Create and run the algorithm.
algo = TradingAlgorithm(initialize=initialize, handle_data=handle_data,
identifiers=['AAPL'])
results = algo.run(data)
analyze(results=results)
def initialize(context):
pass
def handle_data(context, data):
order(symbol('AAPL'), 10)
record(AAPL=data.current(symbol('AAPL'), 'price'))
As you can see, we first have to import some functions we would like to
use. All functions commonly used in your algorithm can be found in
``zipline.api``. Here we are using ``order()`` which takes two arguments
-- a security object, and a number specifying how many stocks you would
like to order (if negative, ``order()`` will sell/short stocks). In this
case we want to order 10 shares of Apple at each iteration. For more
documentation on ``order()``, see the `Quantopian
docs <https://www.quantopian.com/help#api-order>`__.
``zipline.api``. Here we are using :func:`~zipline.api.order()` which takes two
arguments: a security object, and a number specifying how many stocks you would
like to order (if negative, :func:`~zipline.api.order()` will sell/short
stocks). In this case we want to order 10 shares of Apple at each iteration. For
more documentation on ``order()``, see the `Quantopian docs
<https://www.quantopian.com/help#api-order>`__.
You don't have to use the ``symbol()`` function and could just pass in
``AAPL`` directly but it is good practice as this way your code will be
Quantopian compatible.
Finally, the ``record()`` function allows you to save the value of a
variable at each iteration. You provide it with a name for the variable
Finally, the :func:`~zipline.api.record` function allows you to save the value
of a variable at each iteration. You provide it with a name for the variable
together with the variable itself: ``varname=var``. After the algorithm
finished running you will have access to each variable value you tracked
with ``record()`` under the name you provided (we will see this further
below). You also see how we can access the current price data of the
with :func:`~zipline.api.record` under the name you provided (we will see this
further below). You also see how we can access the current price data of the
AAPL stock in the ``data`` event frame (for more information see
`here <https://www.quantopian.com/help#api-event-properties>`__.
Running the algorithm
~~~~~~~~~~~~~~~~~~~~~
To now test this algorithm on financial data, ``zipline`` provides two
interfaces. A command-line interface and an ``IPython Notebook``
interface.
To now test this algorithm on financial data, ``zipline`` provides three
interfaces: A command-line interface, ``IPython Notebook`` magic, and
:func:`~zipline.run_algorithm`.
Command line interface
^^^^^^^^^^^^^^^^^^^^^^
@@ -106,60 +103,59 @@ After you installed zipline you should be able to execute the following
from your command line (e.g. ``cmd.exe`` on Windows, or the Terminal app
on OSX):
.. code:: python
!run_algo.py --help
.. code-block:: bash
$ python -m zipline run --help
.. parsed-literal::
usage: run_algo.py [-h] [-c FILE] [--algofile ALGOFILE] [--data-frequency {minute,daily}] [--start START] [--end END]
[--capital_base CAPITAL_BASE] [--source {yahoo}] [--source_time_column SOURCE_TIME_COLUMN] [--symbols SYMBOLS]
[--output OUTPUT] [--metadata_path METADATA_PATH] [--metadata_index METADATA_INDEX] [--print-algo] [--no-print-algo]
Usage: __main__.py run [OPTIONS]
Zipline version 0.8.3.
Run a backtest for the given algorithm.
optional arguments:
-h, --help show this help message and exit
-c FILE, --conf_file FILE
Specify config file
--algofile ALGOFILE, -f ALGOFILE
--data-frequency {minute,daily}
--start START, -s START
--end END, -e END
--capital_base CAPITAL_BASE
--source {yahoo}, -d {yahoo}
--source_time_column SOURCE_TIME_COLUMN, -t SOURCE_TIME_COLUMN
--symbols SYMBOLS
--output OUTPUT, -o OUTPUT
--metadata_path METADATA_PATH, -m METADATA_PATH
--metadata_index METADATA_INDEX, -x METADATA_INDEX
--print-algo, -p
--no-print-algo, -q
Options:
-f, --algofile FILENAME The file that contains the algorithm to run.
-t, --algotext TEXT The algorithm script to run.
-D, --define TEXT Define a name to be bound in the namespace
before executing the algotext. For example
'-Dname=value'. The value may be any python
expression. These are evaluated in order so
they may refer to previously defined names.
--data-frequency [minute|daily]
The data frequency of the simulation.
[default: daily]
--capital-base FLOAT The starting capital for the simulation.
[default: 10000000.0]
-b, --bundle BUNDLE-NAME The data bundle to use for the simulation.
[default: quandl]
--bundle-timestamp TIMESTAMP The date to lookup data on or before.
[default: <current-time>]
-s, --start DATE The start date of the simulation.
-e, --end DATE The end date of the simulation.
-o, --output FILENAME The location to write the perf data. If this
is '-' the perf will be written to stdout.
[default: -]
--print-algo / --no-print-algo Print the algorithm to stdout.
--help Show this message and exit.
Note that you have to omit the preceding '!' when you call
``run_algo.py``, this is only required by the IPython Notebook in which
this tutorial was written.
As you can see there are a couple of flags that specify where to find
your algorithm (``-f``) as well as parameters specifying which stock
data to load from Yahoo! finance (``--symbols``) and the time-range
(``--start`` and ``--end``). Finally, you'll want to save the
performance metrics of your algorithm so that you can analyze how it
performed. This is done via the ``--output`` flag and will cause it to
write the performance ``DataFrame`` in the pickle Python file format.
Note that you can also define a configuration file with these parameters
that you can then conveniently pass to the ``-c`` option so that you
don't have to supply the command line args all the time (see the .conf
files in the examples directory).
As you can see there are a couple of flags that specify where to find your
algorithm (``-f``) as well as parameters specifying which data to use,
defaulting to the :ref:`quandl-data-bundle`. There are also arguments for the
date range to run the algorithm over (``--start`` and ``--end``). Finally,
you'll want to save the performance metrics of your algorithm so that you can
analyze how it performed. This is done via the ``--output`` flag and will cause
it to write the performance ``DataFrame`` in the pickle Python file format.
Note that you can also define a configuration file with these parameters that
you can then conveniently pass to the ``-c`` option so that you don't have to
supply the command line args all the time (see the .conf files in the examples
directory).
Thus, to execute our algorithm from above and save the results to
``buyapple_out.pickle`` we would call ``run_algo.py`` as follows:
``buyapple_out.pickle`` we would call ``python -m zipline run`` as follows:
.. code:: python
.. code-block:: python
!run_algo.py -f ../../zipline/examples/buyapple.py --start 2000-1-1 --end 2014-1-1 --symbols AAPL -o buyapple_out.pickle
python -m zipline run -f ../../zipline/examples/buyapple.py --start 2000-1-1 --end 2014-1-1 --symbols AAPL -o buyapple_out.pickle
.. parsed-literal::
@@ -170,9 +166,7 @@ Thus, to execute our algorithm from above and save the results to
[2015-11-04 22:45:32.820401] INFO: Performance: last close: 2013-12-31 21:00:00+00:00
``run_algo.py`` first outputs the algorithm contents. It then fetches
historical price and volume data of Apple from Yahoo! finance in the
desired time range, calls the ``initialize()`` function, and then
``run`` first calls the ``initialize()`` function, and then
streams the historical stock price day-by-day through ``handle_data()``.
After each call to ``handle_data()`` we instruct ``zipline`` to order 10
stocks of AAPL. After the call of the ``order()`` function, ``zipline``
@@ -187,31 +181,18 @@ slippage model that ``zipline`` uses, see the `Quantopian
docs <https://www.quantopian.com/help#ide-slippage>`__ for more
information).
Note that there is also an ``analyze()`` function printed.
``run_algo.py`` will try and look for a file with the ending with
``_analyze.py`` and the same name of the algorithm (so
``buyapple_analyze.py``) or an ``analyze()`` function directly in the
script. If an ``analyze()`` function is found it will be called *after*
the simulation has finished and passed in the performance ``DataFrame``.
(The reason for allowing specification of an ``analyze()`` function in a
separate file is that this way ``buyapple.py`` remains a valid
Quantopian algorithm that you can copy&paste to the platform).
Lets take a quick look at the performance ``DataFrame``. For this, we
use ``pandas`` from inside the IPython Notebook and print the first ten
rows. Note that ``zipline`` makes heavy usage of ``pandas``, especially
for data input and outputting so it's worth spending some time to learn
it.
.. code:: python
.. code-block:: python
import pandas as pd
perf = pd.read_pickle('buyapple_out.pickle') # read in perf DataFrame
perf.head()
.. raw:: html
<div style="max-height:1000px;max-width:1500px;overflow:auto;">
@@ -378,7 +359,7 @@ and allows us to plot the price of apple. For example, we could easily
examine now how our portfolio value changed over time compared to the
AAPL stock price.
.. code:: python
.. code-block:: python
%pylab inline
figsize(12, 12)
@@ -391,21 +372,14 @@ AAPL stock price.
perf.AAPL.plot(ax=ax2)
ax2.set_ylabel('AAPL stock price')
.. parsed-literal::
Populating the interactive namespace from numpy and matplotlib
.. parsed-literal::
<matplotlib.text.Text at 0x7ff5c6147f90>
.. image:: tutorial_files/tutorial_11_2.png
@@ -431,28 +405,21 @@ to run the algorithm from above with the same parameters we just have to
execute the following cell after importing ``zipline`` to register the
magic.
.. code:: python
.. code-block:: python
import zipline
%load_ext zipline
.. code:: python
.. code-block:: python
%%zipline --start 2000-1-1 --end 2014-1-1 --symbols AAPL -o perf_ipython
%%zipline --start 2000-1-1 --end 2014-1-1 --symbols AAPL
from zipline.api import symbol, order, record
from zipline.api import symbol, order, record
def initialize(context):
pass
def handle_data(context, data):
order(symbol('AAPL'), 10)
record(AAPL=data[symbol('AAPL')].price)
.. parsed-literal::
AAPL
def initialize(context):
pass
def handle_data(context, data):
order(symbol('AAPL'), 10)
record(AAPL=data[symbol('AAPL')].price)
Note that we did not have to specify an input file as above since the
magic will use the contents of the cell and look for your algorithm
@@ -460,12 +427,9 @@ functions there. Also, instead of defining an output file we are
specifying a variable name with ``-o`` that will be created in the name
space and contain the performance ``DataFrame`` we looked at above.
.. code:: python
perf_ipython.head()
.. code-block:: python
_.head()
.. raw:: html
@@ -624,58 +588,6 @@ space and contain the performance ``DataFrame`` we looked at above.
</div>
Manual (advanced)
~~~~~~~~~~~~~~~~~
If you are happy with either way above you can safely skip this passage.
To provide a closer look at how ``zipline`` actually works it is
instructive to see how we run an algorithm without any of the interfaces
demonstrated above which hide the actual ``zipline`` API.
.. code:: python
import pytz
from datetime import datetime
from zipline.algorithm import TradingAlgorithm
from zipline.utils.factory import load_bars_from_yahoo
# Load data manually from Yahoo! finance
start = datetime(2000, 1, 1, 0, 0, 0, 0, pytz.utc)
end = datetime(2012, 1, 1, 0, 0, 0, 0, pytz.utc)
data = load_bars_from_yahoo(stocks=['AAPL'], start=start,
end=end)
# Define algorithm
def initialize(context):
pass
def handle_data(context, data):
order(symbol('AAPL'), 10)
record(AAPL=data[symbol('AAPL')].price)
# Create algorithm object passing in initialize and
# handle_data functions
algo_obj = TradingAlgorithm(initialize=initialize,
handle_data=handle_data)
# Run algorithm
perf_manual = algo_obj.run(data)
.. parsed-literal::
AAPL
As you can see, we again define the functions as above but we manually
pass them to the ``TradingAlgorithm`` class which is the main
``zipline`` class for running algorithms. We also manually load the data
using ``load_bars_from_yahoo()`` and pass it to the
``TradingAlgorithm.run()`` method which kicks off the backtest
simulation.
Access to previous prices using ``history``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -706,81 +618,73 @@ you can directly use the ``history()`` function on Quantopian, in
with ``add_history()`` and pass it the same arguments as the history
function below. Lets look at the strategy which should make this clear:
.. code:: python
.. code-block:: python
%%zipline --start 2000-1-1 --end 2014-1-1 --symbols AAPL -o perf_dma
%%zipline --start 2000-1-1 --end 2014-1-1 -o perf_dma
from zipline.api import order_target, record, symbol, history, add_history
import numpy as np
from zipline.api import order_target, record, symbol, history, add_history
import numpy as np
def initialize(context):
# Register 2 histories that track daily prices,
# one with a 100 window and one with a 300 day window
add_history(100, '1d', 'price')
add_history(300, '1d', 'price')
def initialize(context):
# Register 2 histories that track daily prices,
# one with a 100 window and one with a 300 day window
add_history(100, '1d', 'price')
add_history(300, '1d', 'price')
context.i = 0
context.i = 0
def handle_data(context, data):
# Skip first 300 days to get full windows
context.i += 1
if context.i < 300:
return
def handle_data(context, data):
# Skip first 300 days to get full windows
context.i += 1
if context.i < 300:
return
# Compute averages
# history() has to be called with the same params
# from above and returns a pandas dataframe.
short_mavg = history(100, '1d', 'price').mean()
long_mavg = history(300, '1d', 'price').mean()
# Compute averages
# history() has to be called with the same params
# from above and returns a pandas dataframe.
short_mavg = history(100, '1d', 'price').mean()
long_mavg = history(300, '1d', 'price').mean()
# Trading logic
if short_mavg[0] > long_mavg[0]:
# order_target orders as many shares as needed to
# achieve the desired number of shares.
order_target(symbol('AAPL'), 100)
elif short_mavg[0] < long_mavg[0]:
order_target(symbol('AAPL'), 0)
# Trading logic
if short_mavg[0] > long_mavg[0]:
# order_target orders as many shares as needed to
# achieve the desired number of shares.
order_target(symbol('AAPL'), 100)
elif short_mavg[0] < long_mavg[0]:
order_target(symbol('AAPL'), 0)
# Save values for later inspection
record(AAPL=data[symbol('AAPL')].price,
short_mavg=short_mavg[0],
long_mavg=long_mavg[0])
# Save values for later inspection
record(AAPL=data[symbol('AAPL')].price,
short_mavg=short_mavg[0],
long_mavg=long_mavg[0])
def analyze(context, perf):
fig = plt.figure()
ax1 = fig.add_subplot(211)
perf.portfolio_value.plot(ax=ax1)
ax1.set_ylabel('portfolio value in $')
ax2 = fig.add_subplot(212)
perf['AAPL'].plot(ax=ax2)
perf[['short_mavg', 'long_mavg']].plot(ax=ax2)
perf_trans = perf.ix[[t != [] for t in perf.transactions]]
buys = perf_trans.ix[[t[0]['amount'] > 0 for t in perf_trans.transactions]]
sells = perf_trans.ix[
[t[0]['amount'] < 0 for t in perf_trans.transactions]]
ax2.plot(buys.index, perf.short_mavg.ix[buys.index],
'^', markersize=10, color='m')
ax2.plot(sells.index, perf.short_mavg.ix[sells.index],
'v', markersize=10, color='k')
ax2.set_ylabel('price in $')
plt.legend(loc=0)
plt.show()
.. parsed-literal::
AAPL
def analyze(context, perf):
fig = plt.figure()
ax1 = fig.add_subplot(211)
perf.portfolio_value.plot(ax=ax1)
ax1.set_ylabel('portfolio value in $')
ax2 = fig.add_subplot(212)
perf['AAPL'].plot(ax=ax2)
perf[['short_mavg', 'long_mavg']].plot(ax=ax2)
perf_trans = perf.ix[[t != [] for t in perf.transactions]]
buys = perf_trans.ix[[t[0]['amount'] > 0 for t in perf_trans.transactions]]
sells = perf_trans.ix[
[t[0]['amount'] < 0 for t in perf_trans.transactions]]
ax2.plot(buys.index, perf.short_mavg.ix[buys.index],
'^', markersize=10, color='m')
ax2.plot(sells.index, perf.short_mavg.ix[sells.index],
'v', markersize=10, color='k')
ax2.set_ylabel('price in $')
plt.legend(loc=0)
plt.show()
.. image:: tutorial_files/tutorial_22_1.png
Here we are explicitly defining an ``analyze()`` function that gets
automatically called once the backtest is done (this is not possible on
Quantopian currently).
docs/source/bundles.rst
+277
View File
@@ -0,0 +1,277 @@
Data Bundles
------------
A data bundle is a collection of pricing data, adjustment data, and an asset
database. Bundles allow us to preload all of the data we will need to run
backtests and store the data for future runs.
Ingesting Data
~~~~~~~~~~~~~~
The first step to using a data bundle is to ingest the data. This will invoke
some custom bundle command and then write the data to a standard location that
zipline can find. By default this location is ``$ZIPLINE_ROOT/data/<bundle>``
where by default ``ZIPLINE_ROOT=~/.zipline``. This step may take some time as it
could involve downloading and processing a lot of data. This can be run with:
.. code-block:: bash
$ python -m zipline ingest <bundle>
where ``<bundle>`` is the name of the bundle to ingest.
Old Data
~~~~~~~~
When the ``ingest`` command is used it will write the new data to a subdirectory
of ``$ZIPLINE_ROOT/data/<bundle>`` which is named with the current date. This
makes it possible to look at older data or even run backtests with this older
copy. This makers it easier to reproduce backtest results later.
One drawback of saving all of this data by default is that the data directory
may grow quite large even if you do not want to use the data. To solve this
problem there is another command ``clean`` which will clear data bundles based
on some time constraints.
For example:
.. code-block:: bash
# clean everything older than <date>
$ python -m zipline clean <bundle> --before <date>
# clean everything newer than <date>
$ python -m zipline clean <bundle> --after <date>
# keep everything in the range of [before, after] and delete the rest
$ python -m zipline clean <bundle> --before <date> --after <after>
# clean all but the last <int> runs
$ python -m zipline clean <bundle> --keep-last <int>
Running Backtests with Data Bundles
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Now that the data has been ingested we can use it to run backtests with the
``run`` command. This can be specified with the ``--bundle`` option like:
.. code-block:: bash
$ python -m zipline run --bundle <bundle> --algofile algo.py ...
We may also specify the date to use to look up the bundle data with the
``--bundle-date`` option. This will cause us to the the most recent bundle
ingestion that is less than or equal to the ``bundle-date``. This is how we can
run backtests with older data. The reason that this uses a less than or equal to
relationship is that we can specify the date that we ran an old backtest and get
the same data that would have been available to us on that date. The
``bundle-date`` defaults to the current day to use the most recent data.
Default Data Bundles
~~~~~~~~~~~~~~~~~~~~
.. _quandl-data-bundle:
Quandl WIKI Bundle
``````````````````
By default zipline comes with the ``quandl`` data bundle which uses quandl's
`WIKI dataset <https://www.quandl.com/data/WIKI>`_. The quandl data bundle
includes daily pricing data, splits, cash dividends, and asset metadata. This is
the bundle that ``run`` will use by default if no other bundle is specified. To
ingest this data bundle we recommend creating an account on quandl.com to get an
API key to be able to make more API requests per day. Once we have an API key we
may run:
.. code-block:: bash
$ QUANDL_API_KEY=<api-key> python -m zipline ingest quandl
though we may still run ``ingest`` as an anonymous quandl user (with no API
key). We may also set the ``QUANDL_DOWNLOAD_ATTEMPTS`` environment variable to
an integer which is the number of attempts that should be made to download data
from quandls servers. By default this will be 5, meaning that we will retry each
attempt 5 times.
.. note::
``QUANDL_DOWNLOAD_ATTEMPTS`` is not the total number of allowed failures,
just the number of allowed failures per request. The quandl loader will make
one request per 100 equities for the metadata followed by one request per
equity.
Yahoo Bundle Factories
``````````````````````
Zipline also ships with a factory function for creating a data bundle out of a
set of tickers from yahoo: :func:`~zipline.data.bundles.yahoo_equities`.
This makes it easy to pre-download and cache the data for a set of equities from
yahoo. This includes daily pricing data along with splits, cash dividends, and
inferred asset metadata. To create a bundle from a set of equities, add the
following to your ``~/.zipline/extensions.py`` file:
.. code-block:: python
from zipline.bundles import register, yahoo_equities
# these are the tickers you would like data for
equities = {
'AAPL',
'MSFT',
'GOOG',
}
register(
'my-yahoo-equities-bundle', # name this whatever you like
yahoo_equities(equities),
)
This may now be used like:
.. code-block:: bash
$ python -m zipline ingest my-yahoo-equities-bundle
$ python -m zipline run -f algo.py --bundle my-yahoo-equities-bundle
More than one yahoo equities bundle may be registered as long as they use
different names.
Writing a New Bundle
~~~~~~~~~~~~~~~~~~~~
Data bundles exist to make it easy to use different data sources with
zipline. To add a new bundle, one must implement an ingest function.
This function is responsible for loading the data into memory and passing it to
a set of writer objects provided by zipline to convert the data to zipline's
internal format. The ingest function may work by downloading data from a remote
location like the ``quandl`` bundle or yahoo bundles or it may just load files
that are already on the machine. The function is provided with writers that will
write the data to the correct location transactionally. If an ingestion fails
part way through the bundle will not be written in an incomplete state.
The signature of the ingest function should be:
.. code-block:: python
ingest(environ,
asset_db_writer,
minute_bar_writer,
daily_bar_writer,
adjustment_writer,
calendar,
cache,
show_progress)
``environ``
```````````
``environ`` is a mapping representing the environment variables to use. This is
where any custom arguments needed for the ingestion should be passed, for
example: the ``quandl`` bundle uses the enviornment to pass the API key and the
download retry attempt count.
``asset_db_writer``
```````````````````
``asset_db_writer`` is an instance of :class:`~zipline.assets.AssetDBWriter`.
This is the writer for the asset metadata which provides the asset lifetimes and
the symbol to asset id (sid) mapping. This may also contain the asset name,
exchange and a few other columns. To write data, invoke
:meth:`~zipline.assets.AssetDBWriter.write` with dataframes for the various
pieces of metadata. More information about the format of the data exists in the
docs for write.
``minute_bar_writer``
`````````````````````
``minute_bar_writer`` is an instance of
:class:`~zipline.data.minute_bars.BcolzMinuteBarWriter`. This writer is used to
convert data to zipline's internal bcolz format to later be read by a
:class:`~zipline.data.minute_bars.BcolzMinuteBarReader`. If minute data is
provided, users should call
:meth:`~zipline.data.minute_bars.BcolzMinuteBarWriter.write` with an iterable of
(sid, dataframe) tuples. The ``show_progress`` argument should also be forwarded
to this method. If the data source does not provide minute level data, then
there is no need to call the write method. It is also acceptable to pass an
empty iterator to :meth:`~zipline.data.minute_bars.BcolzMinuteBarWriter.write`
to signal that there is no minutely data.
.. note::
The data passed to
:meth:`~zipline.data.minute_bars.BcolzMinuteBarWriter.write` may be a lazy
iterator or generator to avoid loading all of the minute data into memory at
a single time. A given sid may also appear multiple times in the data as long
as the dates are strictly increasing.
``daily_bar_writer``
````````````````````
``daily_bar_writer`` is an instance of
:class:`~zipline.data.us_equity_pricing.BcolzDailyBarWriter`. This writer is
used to convert data into zipline's internal bcolz format to later be read by a
:class:`~zipline.data.us_equity_pricing.BcolzDailyBarReader`. If daily data is
provided, users should call
:meth:`~zipline.data.minute_bars.BcolzDailyBarWriter.write` with an iterable of
(sid dataframe) tuples. The ``show_progress`` argument should also be forwarded
to this method. If the data shource does not provide daily data, then there is
no need to call the write method. It is also acceptable to pass an empty
iterable to :meth:`~zipline.data.minute_bars.BcolzMinuteBarWriter.write` to
signal that there is no daily data. If no daily data is provided but minute data
is provided, a daily rollup will happen to service daily history requests.
.. note::
Like the ``minute_bar_writer``, the data passed to
:meth:`~zipline.data.minute_bars.BcolzMinuteBarWriter.write` may be a lazy
iterable or generator to avoid loading all of the data into memory at once.
Unlike the ``minute_bar_writer``, a sid may only appear once in the data
iterable.
``adjustment_writer``
`````````````````````
``adjustment_writer`` is an instance of
:class:`~zipline.data.us_equity_pricing.SQLiteAdjustmentWriter`. This writer is
used to store splits, mergers, dividends, and stock dividends. The data should
be provided as dataframes and passed to
:meth:`~zipline.data.us_equity_pricing.SQLiteAdjustmentWriter.write`. Each of
these fields are optional, but the writer can accept as much of the data as you
have.
``calendar``
````````````
``calendar`` is a ``pandas.DatetimeIndex`` object holding all of the trading
days that the bundle should load data for. This is to help some bundles generate
queries for the days needed.
``cache``
`````````
``cache`` is an instance of :class:`~zipline.utils.cache.dataframe_cache`. This
object is a mapping from strings to dataframes. This object is provided in case
an ingestion crashes part way through. The idea is that the ingest function
should check the cache for raw data, if it doesn't exist in the cache, it should
acquire it and then store it in the cache. Then it can parse and write the
data. The cache will be cleared only after a successful load, this prevents the
ingest function from needing to redownload all the data if there is some bug in
the parsing. If it is very fast to get the data, for example if it is coming
from another local file, then there is no need to use this cache.
``show_progress``
`````````````````
``show_progress`` is a boolean indicating that the user would like to receive
feedback about the ingest function's progress fetching and writing the
data. Some examples for where to show how many files you have downloaded out of
the total needed, or how far into some data conversion the ingest function
is. One tool that may help with implementing ``show_progress`` for a loop is
:class:`~zipline.utils.cli.maybe_show_progress`. This argument should always be
forwarded to ``minute_bar_writer.write`` and ``daily_bar_writer.write``.
docs/source/index.rst
+1
View File
@@ -5,6 +5,7 @@
install
beginner-tutorial
bundles
releases
appendix
release-process
docs/source/whatsnew/1.0.0.txt
+49 -4
View File
@@ -12,7 +12,51 @@ Development
Highlights
~~~~~~~~~~
None
New Entry Points (:issue:`xxxx`)
````````````````````````````````
In order to make it easier to use zipline we have updated the entry points for
a backtest. The three supported ways to run a backtest are now:
1. :func:`zipline.run_algo`
2. ``$ python -m zipline run``
3. ``%zipline`` (IPython magic)
Data Bundles (:issue:`xxxx`)
````````````````````````````
1.0.0 introduces data bundles. Data bundles are groups of data that should be
preloaded and used to run backtests later. This allows users to not need to to
specify which tickers they are interested in each time they run an
algorithm. This also allows us to cache the data between runs.
By default, the ``quandl`` bundle will be used which pulls data from quandl's
`WIKI dataset <https://www.quandl.com/data/WIKI>`_. New bundles may be
registered with :func:`zipline.data.bundles.register` like:
.. code-block:: python
@zipline.data.bundles.register('my-new-bundle')
def my_new_bundle_ingest(environ,
asset_db_writer,
minute_bar_writer,
daily_bar_writer,
adjustment_writer,
calendar,
cache,
show_progress):
...
This function should retrieve the data it needs and then use the writers that
have been passed to write that data to disc in a location that zipline can find
later.
This data can be used in backtests by passing the name as the ``-b / --bundle``
argument to ``$ python -m zipline run`` or as the ``bundle`` argument to
:func:`zipline.run_algo`.
For more information see `Data Bundles`_ for more information.
Enhancements
~~~~~~~~~~~~
@@ -20,9 +64,10 @@ Enhancements
* Made the data loading classes have more consistent interfaces. This includes
the equity bar writers, adjustment writer, and asset db writer. The new
interface is that the resource to be written to is passed at construction time
and the data to write is provided later to the `write` method as a
dataframe. This model allows us to pass these writer objects around as a
resource for other classes and functions to consume (:issue:`1109`).
and the data to write is provided later to the `write` method as
dataframes or some iterator of dataframes. This model allows us to pass these
writer objects around as a resource for other classes and functions to
consume (:issue:`1109` and :issue:`1149`).
* Added masking to :class:`zipline.pipeline.CustomFactor`.
Custom factors can now be passed a Filter upon instantiation. This tells the
etc/requirements.txt
+1
View File
@@ -15,6 +15,7 @@ numpy==1.9.2
# statsmodels in turn is required for some pandas packages
scipy==0.15.1
pandas==0.16.1
pandas-datareader==0.2.1
# Needed for parts of pandas.stats
patsy==0.4.0
statsmodels==0.6.1
scripts/run_algo.py
-26
View File
@@ -1,26 +0,0 @@
#!/usr/bin/env python
#
# Copyright 2014 Quantopian, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import logbook
import sys
from zipline.utils import parse_args, run_pipeline
if __name__ == "__main__":
logbook.StderrHandler().push_application()
parsed = parse_args(sys.argv[1:])
run_pipeline(**parsed)
sys.exit(0)
setup.py
-1
View File
@@ -264,7 +264,6 @@ setup(
author_email='opensource@quantopian.com',
packages=find_packages('.', include=['zipline', 'zipline.*']),
ext_modules=ext_modules,
scripts=['scripts/run_algo.py'],
include_package_data=True,
license='Apache 2.0',
classifiers=[
tests/data/bundles/test_core.py
+218
View File
@@ -0,0 +1,218 @@
import pandas as pd
from toolz import valmap
import toolz.curried.operator as op
from zipline.assets.synthetic import make_simple_equity_info
from zipline.data.bundles import load
from zipline.data.bundles.core import _make_bundle_core
from zipline.lib.adjustment import Float64Multiply
from zipline.pipeline.loaders.synthetic import (
make_bar_data,
expected_bar_values_2d,
)
from zipline.testing import (
subtest,
tmp_dir,
str_to_seconds,
tmp_trading_env,
)
from zipline.testing.fixtures import ZiplineTestCase
from zipline.testing.predicates import (
assert_equal,
assert_false,
assert_in,
assert_is,
assert_is_instance,
)
from zipline.utils.cache import dataframe_cache
from zipline.utils.functional import apply
from zipline.utils.tradingcalendar import trading_days
class BundleCoreTestCase(ZiplineTestCase):
def init_instance_fixtures(self):
super(BundleCoreTestCase, self).init_instance_fixtures()
(self.bundles,
self.register,
self.unregister,
self.ingest) = _make_bundle_core()
def test_register_decorator(self):
@apply
@subtest(((c,) for c in 'abcde'), 'name')
def _(name):
@self.register(name)
def ingest(*args):
pass
assert_in(name, self.bundles)
assert_is(self.bundles[name].ingest, ingest)
self._check_bundles(set('abcde'))
def test_register_call(self):
def ingest(*args):
pass
@apply
@subtest(((c,) for c in 'abcde'), 'name')
def _(name):
self.register(name, ingest)
assert_in(name, self.bundles)
assert_is(self.bundles[name].ingest, ingest)
assert_equal(
valmap(op.attrgetter('ingest'), self.bundles),
{k: ingest for k in 'abcde'},
)
self._check_bundles(set('abcde'))
def _check_bundles(self, names):
assert_equal(set(self.bundles.keys()), names)
for name in names:
self.unregister(name)
assert_false(self.bundles)
def test_ingest(self):
zipline_root = self.enter_instance_context(tmp_dir()).path
env = self.enter_instance_context(tmp_trading_env())
start = pd.Timestamp('2014-01-06', tz='utc')
end = pd.Timestamp('2014-01-10', tz='utc')
calendar = trading_days[trading_days.slice_indexer(start, end)]
minutes = env.minutes_for_days_in_range(calendar[0], calendar[-1])
outer_environ = {
'ZIPLINE_ROOT': zipline_root,
}
sids = tuple(range(3))
equities = make_simple_equity_info(
sids,
calendar[0],
calendar[-1],
)
daily_bar_data = make_bar_data(equities, calendar)
minute_bar_data = make_bar_data(equities, minutes)
first_split_ratio = 0.5
second_split_ratio = 0.1
splits = pd.DataFrame.from_records([
{
'effective_date': str_to_seconds('2014-01-08'),
'ratio': first_split_ratio,
'sid': 0,
},
{
'effective_date': str_to_seconds('2014-01-09'),
'ratio': second_split_ratio,
'sid': 1,
},
])
@self.register('bundle',
calendar=calendar,
opens=env.opens_in_range(calendar[0], calendar[-1]),
closes=env.closes_in_range(calendar[0], calendar[-1]))
def bundle_ingest(environ,
asset_db_writer,
minute_bar_writer,
daily_bar_writer,
adjustment_writer,
calendar,
cache,
show_progress):
assert_is(environ, outer_environ)
asset_db_writer.write(equities=equities)
minute_bar_writer.write(minute_bar_data)
daily_bar_writer.write(daily_bar_data)
adjustment_writer.write(splits=splits)
assert_is_instance(calendar, pd.DatetimeIndex)
assert_is_instance(cache, dataframe_cache)
assert_is_instance(show_progress, bool)
self.ingest('bundle', environ=outer_environ)
bundle = load('bundle', environ=outer_environ)
assert_equal(set(bundle.asset_finder.sids), set(sids))
columns = 'open', 'high', 'low', 'close', 'volume'
actual = bundle.minute_bar_reader.load_raw_arrays(
columns,
minutes[0],
minutes[-1],
sids,
)
for actual_column, colname in zip(actual, columns):
assert_equal(
actual_column,
expected_bar_values_2d(minutes, equities, colname),
msg=colname,
)
actual = bundle.daily_bar_reader.load_raw_arrays(
columns,
calendar[0],
calendar[-1],
sids,
)
for actual_column, colname in zip(actual, columns):
assert_equal(
actual_column,
expected_bar_values_2d(calendar, equities, colname),
msg=colname,
)
adjustments_for_cols = bundle.adjustment_reader.load_adjustments(
columns,
calendar,
pd.Index(sids),
)
for column, adjustments in zip(columns, adjustments_for_cols[:-1]):
# iterate over all the adjustments but `volume`
assert_equal(
adjustments,
{
2: [Float64Multiply(
first_row=0,
last_row=2,
first_col=0,
last_col=0,
value=first_split_ratio,
)],
3: [Float64Multiply(
first_row=0,
last_row=3,
first_col=1,
last_col=1,
value=second_split_ratio,
)],
},
msg=column,
)
# check the volume, the value should be 1/ratio
assert_equal(
adjustments_for_cols[-1],
{
2: [Float64Multiply(
first_row=0,
last_row=2,
first_col=0,
last_col=0,
value=1 / first_split_ratio,
)],
3: [Float64Multiply(
first_row=0,
last_row=3,
first_col=1,
last_col=1,
value=1 / second_split_ratio,
)],
},
msg='volume',
)
tests/data/bundles/test_quandl.py
+243
View File
@@ -0,0 +1,243 @@
from __future__ import division
import numpy as np
import pandas as pd
from toolz import merge
import toolz.curried.operator as op
from zipline.data.bundles import ingest, load, bundles
from zipline.data.bundles.quandl import (
format_wiki_url,
format_metadata_url,
)
from zipline.lib.adjustment import Float64Multiply
from zipline.testing import (
test_resource_path,
tmp_dir,
patch_read_csv,
)
from zipline.testing.fixtures import ZiplineTestCase
from zipline.testing.predicates import (
assert_equal,
)
from zipline.utils.functional import apply
class QuandlBundleTestCase(ZiplineTestCase):
symbols = 'AAPL', 'BRK_A', 'MSFT', 'ZEN'
asset_start = pd.Timestamp('2014-01', tz='utc')
asset_end = pd.Timestamp('2015-01', tz='utc')
calendar = bundles['quandl'].calendar
start_date = calendar[0]
end_date = calendar[-1]
api_key = 'ayylmao'
columns = 'open', 'high', 'low', 'close', 'volume'
def _expected_data(self, asset_finder):
sids = {
symbol: asset_finder.lookup_symbol(
symbol,
self.asset_start,
).sid
for symbol in self.symbols
}
def per_symbol(symbol):
df = pd.read_csv(
test_resource_path('quandl_samples', symbol + '.csv.gz'),
parse_dates=['Date'],
index_col='Date',
usecols=[
'Open',
'High',
'Low',
'Close',
'Volume',
'Date',
'Ex-Dividend',
'Split Ratio',
],
na_values=['NA'],
).rename(columns={
'Open': 'open',
'High': 'high',
'Low': 'low',
'Close': 'close',
'Volume': 'volume',
'Date': 'date',
'Ex-Dividend': 'ex_dividend',
'Split Ratio': 'split_ratio',
})
df['sid'] = sids[symbol]
return df
all_ = pd.concat(map(per_symbol, self.symbols)).set_index(
'sid',
append=True,
).unstack()
# fancy list comprehension with statements
@list
@apply
def pricing():
for column in self.columns:
vs = all_[column].values
if column == 'volume':
vs = np.nan_to_num(vs)
yield vs
# the first index our written data will appear in the files on disk
start_idx = self.calendar.get_loc(self.asset_start, 'ffill') + 1
# convert an index into the raw dataframe into an index into the
# final data
i = op.add(start_idx)
def expected_dividend_adjustment(idx, symbol):
sid = sids[symbol]
return (
1 -
all_.ix[idx, ('ex_dividend', sid)] /
all_.ix[idx - 1, ('close', sid)]
)
adjustments = [
# ohlc
{
# dividends
i(24): [Float64Multiply(
first_row=0,
last_row=i(24),
first_col=sids['AAPL'],
last_col=sids['AAPL'],
value=expected_dividend_adjustment(24, 'AAPL'),
)],
i(87): [Float64Multiply(
first_row=0,
last_row=i(87),
first_col=sids['AAPL'],
last_col=sids['AAPL'],
value=expected_dividend_adjustment(87, 'AAPL'),
)],
i(150): [Float64Multiply(
first_row=0,
last_row=i(150),
first_col=sids['AAPL'],
last_col=sids['AAPL'],
value=expected_dividend_adjustment(150, 'AAPL'),
)],
i(214): [Float64Multiply(
first_row=0,
last_row=i(214),
first_col=sids['AAPL'],
last_col=sids['AAPL'],
value=expected_dividend_adjustment(214, 'AAPL'),
)],
i(31): [Float64Multiply(
first_row=0,
last_row=i(31),
first_col=sids['MSFT'],
last_col=sids['MSFT'],
value=expected_dividend_adjustment(31, 'MSFT'),
)],
i(90): [Float64Multiply(
first_row=0,
last_row=i(90),
first_col=sids['MSFT'],
last_col=sids['MSFT'],
value=expected_dividend_adjustment(90, 'MSFT'),
)],
i(222): [Float64Multiply(
first_row=0,
last_row=i(222),
first_col=sids['MSFT'],
last_col=sids['MSFT'],
value=expected_dividend_adjustment(222, 'MSFT'),
)],
# splits
i(108): [Float64Multiply(
first_row=0,
last_row=i(108),
first_col=sids['AAPL'],
last_col=sids['AAPL'],
value=7.0,
)],
},
] * (len(self.columns) - 1) + [
# volume
{
i(108): [Float64Multiply(
first_row=0,
last_row=i(108),
first_col=sids['AAPL'],
last_col=sids['AAPL'],
value=1 / 7,
)],
}
]
return pricing, adjustments
def test_bundle(self):
url_map = merge(
{
format_wiki_url(
self.api_key,
symbol,
self.start_date,
self.end_date,
): test_resource_path('quandl_samples', symbol + '.csv.gz')
for symbol in self.symbols
},
{
format_metadata_url(self.api_key, n): test_resource_path(
'quandl_samples',
'metadata-%d.csv.gz' % n,
)
for n in (1, 2)
},
)
zipline_root = self.enter_instance_context(tmp_dir()).path
environ = {
'ZIPLINE_ROOT': zipline_root,
'QUANDL_API_KEY': self.api_key,
}
with patch_read_csv(url_map, strict=True):
ingest('quandl', environ=environ)
bundle = load('quandl', environ=environ)
sids = 0, 1, 2, 3
assert_equal(set(bundle.asset_finder.sids), set(sids))
for equity in bundle.asset_finder.retrieve_all(sids):
assert_equal(equity.start_date, self.asset_start, msg=equity)
assert_equal(equity.end_date, self.asset_end, msg=equity)
cal = self.calendar
actual = bundle.daily_bar_reader.load_raw_arrays(
self.columns,
cal[cal.get_loc(self.asset_start, 'bfill')],
cal[cal.get_loc(self.asset_end, 'ffill')],
sids,
)
expected_pricing, expected_adjustments = self._expected_data(
bundle.asset_finder,
)
assert_equal(actual, expected_pricing, array_decimal=2)
adjustments_for_cols = bundle.adjustment_reader.load_adjustments(
self.columns,
cal,
pd.Index(sids),
)
for column, adjustments, expected in zip(self.columns,
adjustments_for_cols,
expected_adjustments):
assert_equal(
adjustments,
expected,
msg=column,
)
tests/data/bundles/test_yahoo.py
+201
View File
@@ -0,0 +1,201 @@
import numpy as np
import pandas as pd
from six.moves.urllib.parse import urlparse, parse_qs
from toolz import flip, identity
from toolz.curried import merge_with, operator as op
from zipline.data.bundles.core import _make_bundle_core
from zipline.data.bundles import yahoo_equities, load
from zipline.lib.adjustment import Float64Multiply
from zipline.testing import test_resource_path, tmp_dir, read_compressed
from zipline.testing.fixtures import WithResponses, ZiplineTestCase
from zipline.testing.predicates import assert_equal
from zipline.utils.tradingcalendar import trading_days
class YahooBundleTestCase(WithResponses, ZiplineTestCase):
symbols = 'AAPL', 'IBM', 'MSFT'
columns = 'open', 'high', 'low', 'close', 'volume'
asset_start = pd.Timestamp('2014-01-02', tz='utc')
asset_end = pd.Timestamp('2014-12-31', tz='utc')
calendar = trading_days[
(trading_days >= asset_start) &
(trading_days <= asset_end)
]
@classmethod
def init_class_fixtures(cls):
super(YahooBundleTestCase, cls).init_class_fixtures()
(cls.bundles,
cls.register,
cls.unregister,
cls.ingest) = map(staticmethod, _make_bundle_core())
def _expected_data(self):
sids = 0, 1, 2
modifier = {
'low': 0,
'open': 1,
'close': 2,
'high': 3,
'volume': 0,
}
pricing = [
np.hstack((
np.arange(252, dtype='float64')[:, np.newaxis] +
1 +
sid * 10000 +
modifier[column] * 1000
for sid in sorted(sids)
))
for column in self.columns
]
# There are two dividends and 1 split for each company.
def dividend_adjustment(sid, which):
"""The dividends occur at indices 252 // 4 and 3 * 252 / 4
with a cash amount of sid + 1 / 10 and sid + 2 / 10
"""
if which == 'first':
idx = 252 // 4
else:
idx = 3 * 252 // 4
return {
idx: [Float64Multiply(
first_row=0,
last_row=idx,
first_col=sid,
last_col=sid,
value=float(
1 -
((sid + 1 + (which == 'second')) / 10) /
(idx - 1 + sid * 10000 + 2000)
),
)],
}
def split_adjustment(sid, volume):
"""The splits occur at index 252 // 2 with a ratio of (sid + 1):1
"""
idx = 252 // 2
return {
idx: [Float64Multiply(
first_row=0,
last_row=idx,
first_col=sid,
last_col=sid,
value=(identity if volume else op.truediv(1))(sid + 2),
)],
}
merge_adjustments = merge_with(flip(sum, []))
adjustments = [
# ohlc
merge_adjustments(
*tuple(dividend_adjustment(sid, 'first') for sid in sids) +
tuple(dividend_adjustment(sid, 'second') for sid in sids) +
tuple(split_adjustment(sid, volume=False) for sid in sids)
)
] * (len(self.columns) - 1) + [
# volume
merge_adjustments(
split_adjustment(sid, volume=True) for sid in sids
),
]
return pricing, adjustments
def test_bundle(self):
def get_symbol_from_url(url):
params = parse_qs(urlparse(url).query)
symbol, = params['s']
return symbol
def pricing_callback(request):
headers = {
'content-encoding': 'gzip',
'content-type': 'text/csv',
}
path = test_resource_path(
'yahoo_samples',
get_symbol_from_url(request.url) + '.csv.gz',
)
with open(path, 'rb') as f:
return (
200,
headers,
f.read(),
)
for _ in range(3):
self.responses.add_callback(
self.responses.GET,
'http://ichart.finance.yahoo.com/table.csv',
pricing_callback,
)
def adjustments_callback(request):
path = test_resource_path(
'yahoo_samples',
get_symbol_from_url(request.url) + '.adjustments.gz',
)
return 200, {}, read_compressed(path)
for _ in range(3):
self.responses.add_callback(
self.responses.GET,
'http://ichart.finance.yahoo.com/x',
adjustments_callback,
)
cal = self.calendar
self.register(
'bundle',
yahoo_equities(self.symbols),
calendar=cal,
)
zipline_root = self.enter_instance_context(tmp_dir()).path
environ = {
'ZIPLINE_ROOT': zipline_root,
}
self.ingest('bundle', environ=environ)
bundle = load('bundle', environ=environ)
sids = 0, 1, 2
equities = bundle.asset_finder.retrieve_all(sids)
for equity, expected_symbol in zip(equities, self.symbols):
assert_equal(equity.symbol, expected_symbol)
for equity in bundle.asset_finder.retrieve_all(sids):
assert_equal(equity.start_date, self.asset_start, msg=equity)
assert_equal(equity.end_date, self.asset_end, msg=equity)
actual = bundle.daily_bar_reader.load_raw_arrays(
self.columns,
cal[cal.get_loc(self.asset_start, 'bfill')],
cal[cal.get_loc(self.asset_end, 'ffill')],
sids,
)
expected_pricing, expected_adjustments = self._expected_data()
assert_equal(actual, expected_pricing, array_decimal=2)
adjustments_for_cols = bundle.adjustment_reader.load_adjustments(
self.columns,
cal,
pd.Index(sids),
)
for column, adjustments, expected in zip(self.columns,
adjustments_for_cols,
expected_adjustments):
assert_equal(
adjustments,
expected,
msg=column,
)
tests/data/test_minute_bars.py
+29 -24
View File
@@ -24,6 +24,7 @@ from numpy import (
float64,
full,
nan,
transpose,
zeros,
)
from numpy.testing import assert_almost_equal, assert_array_equal
@@ -100,7 +101,7 @@ class BcolzMinuteBarTestCase(TestCase):
'volume': [50.0]
},
index=[minute])
self.writer.write(sid, data)
self.writer.write_sid(sid, data)
open_price = self.reader.get_value(sid, minute, 'open')
@@ -135,7 +136,7 @@ class BcolzMinuteBarTestCase(TestCase):
'volume': [50.0, 51.0]
},
index=[minute_0, minute_1])
self.writer.write(sid, data)
self.writer.write_sid(sid, data)
open_price = self.reader.get_value(sid, minute_0, 'open')
@@ -190,7 +191,7 @@ class BcolzMinuteBarTestCase(TestCase):
'volume': [50.0]
},
index=[minute])
self.writer.write(sid, data)
self.writer.write_sid(sid, data)
open_price = self.reader.get_value(sid, minute, 'open')
@@ -224,7 +225,7 @@ class BcolzMinuteBarTestCase(TestCase):
'volume': [0]
},
index=[minute])
self.writer.write(sid, data)
self.writer.write_sid(sid, data)
open_price = self.reader.get_value(sid, minute, 'open')
@@ -267,7 +268,7 @@ class BcolzMinuteBarTestCase(TestCase):
'volume': [50.0, 51.0]
},
index=minutes)
self.writer.write(sid, data)
self.writer.write_sid(sid, data)
minute = minutes[0]
@@ -325,10 +326,10 @@ class BcolzMinuteBarTestCase(TestCase):
'volume': [50.0]
},
index=[minute])
self.writer.write(sid, data)
self.writer.write_sid(sid, data)
with self.assertRaises(BcolzMinuteOverlappingData):
self.writer.write(sid, data)
self.writer.write_sid(sid, data)
def test_write_multiple_sids(self):
"""
@@ -361,7 +362,7 @@ class BcolzMinuteBarTestCase(TestCase):
'volume': [100.0]
},
index=[minute])
self.writer.write(sids[0], data)
self.writer.write_sid(sids[0], data)
data = DataFrame(
data={
@@ -372,7 +373,7 @@ class BcolzMinuteBarTestCase(TestCase):
'volume': [200.0]
},
index=[minute])
self.writer.write(sids[1], data)
self.writer.write_sid(sids[1], data)
sid = sids[0]
@@ -442,7 +443,7 @@ class BcolzMinuteBarTestCase(TestCase):
'volume': [100.0]
},
index=[minute])
self.writer.write(sid, data)
self.writer.write_sid(sid, data)
open_price = self.reader.get_value(sid, minute, 'open')
@@ -489,12 +490,13 @@ class BcolzMinuteBarTestCase(TestCase):
'volume': full(9, 0),
},
index=[minutes])
self.writer.write(sid, data)
self.writer.write_sid(sid, data)
fields = ['open', 'high', 'low', 'close', 'volume']
ohlcv_window = self.reader.unadjusted_window(
fields, minutes[0], minutes[-1], [sid])
ohlcv_window = list(map(transpose, self.reader.load_raw_arrays(
fields, minutes[0], minutes[-1], [sid],
)))
for i, field in enumerate(fields):
if field != 'volume':
@@ -531,12 +533,13 @@ class BcolzMinuteBarTestCase(TestCase):
'volume': full(9, 0),
},
index=[minutes])
self.writer.write(sid, data)
self.writer.write_sid(sid, data)
fields = ['open', 'high', 'low', 'close', 'volume']
ohlcv_window = self.reader.unadjusted_window(
fields, minutes[0], minutes[-1], [sid])
ohlcv_window = list(map(transpose, self.reader.load_raw_arrays(
fields, minutes[0], minutes[-1], [sid],
)))
for i, field in enumerate(fields):
if field != 'volume':
@@ -630,7 +633,7 @@ class BcolzMinuteBarTestCase(TestCase):
'volume': [1000, 0, 1001]
},
index=minutes)
self.writer.write(sids[0], data_1)
self.writer.write_sid(sids[0], data_1)
data_2 = DataFrame(
data={
@@ -641,14 +644,15 @@ class BcolzMinuteBarTestCase(TestCase):
'volume': [2000, 0, 2001]
},
index=minutes)
self.writer.write(sids[1], data_2)
self.writer.write_sid(sids[1], data_2)
reader = BcolzMinuteBarReader(self.dest)
columns = ['open', 'high', 'low', 'close', 'volume']
sids = [sids[0], sids[1]]
arrays = reader.unadjusted_window(
columns, minutes[0], minutes[-1], sids)
arrays = list(map(transpose, reader.load_raw_arrays(
columns, minutes[0], minutes[-1], sids,
)))
data = {sids[0]: data_1, sids[1]: data_2}
@@ -681,7 +685,7 @@ class BcolzMinuteBarTestCase(TestCase):
'volume': [1000, 1001, 1002],
},
index=minutes)
self.writer.write(sids[0], data_1)
self.writer.write_sid(sids[0], data_1)
data_2 = DataFrame(
data={
@@ -692,14 +696,15 @@ class BcolzMinuteBarTestCase(TestCase):
'volume': [2000, 2001, 2002],
},
index=minutes)
self.writer.write(sids[1], data_2)
self.writer.write_sid(sids[1], data_2)
reader = BcolzMinuteBarReader(self.dest)
columns = ['open', 'high', 'low', 'close', 'volume']
sids = [sids[0], sids[1]]
arrays = reader.unadjusted_window(
columns, minutes[0], minutes[-1], sids)
arrays = list(map(transpose, reader.load_raw_arrays(
columns, minutes[0], minutes[-1], sids,
)))
data = {sids[0]: data_1, sids[1]: data_2}
tests/data/test_us_equity_pricing.py
+15 -16
View File
@@ -33,14 +33,13 @@ from zipline.data.us_equity_pricing import (
BcolzDailyBarReader,
NoDataOnDate,
)
from zipline.pipeline.data import USEquityPricing
from zipline.pipeline.loaders.synthetic import (
OHLCV,
asset_start,
asset_end,
expected_daily_bar_value,
expected_daily_bar_values_2d,
make_daily_bar_data,
expected_bar_value,
expected_bar_values_2d,
make_bar_data,
)
from zipline.testing import seconds_to_timestamp
from zipline.testing.fixtures import (
@@ -89,7 +88,7 @@ class BcolzDailyBarTestCase(WithBcolzDailyBarReader, ZiplineTestCase):
@classmethod
def make_daily_bar_data(cls):
return make_daily_bar_data(
return make_bar_data(
EQUITY_INFO,
cls.bcolz_daily_bar_days,
)
@@ -129,7 +128,7 @@ class BcolzDailyBarTestCase(WithBcolzDailyBarReader, ZiplineTestCase):
for asset_id in self.assets:
for date in self.dates_for_asset(asset_id):
self.assertEqual(
expected_daily_bar_value(
expected_bar_value(
asset_id,
date,
column
@@ -198,18 +197,18 @@ class BcolzDailyBarTestCase(WithBcolzDailyBarReader, ZiplineTestCase):
for column, result in zip(columns, results):
assert_array_equal(
result,
expected_daily_bar_values_2d(
expected_bar_values_2d(
dates,
EQUITY_INFO,
column.name,
column,
)
)
@parameterized.expand([
([USEquityPricing.open],),
([USEquityPricing.close, USEquityPricing.volume],),
([USEquityPricing.volume, USEquityPricing.high, USEquityPricing.low],),
(USEquityPricing.columns,),
(['open'],),
(['close', 'volume'],),
(['volume', 'high', 'low'],),
(['open', 'high', 'low', 'close', 'volume'],),
])
def test_read(self, columns):
self._check_read_results(
@@ -224,7 +223,7 @@ class BcolzDailyBarTestCase(WithBcolzDailyBarReader, ZiplineTestCase):
Test loading with queries that starts on the first day of each asset's
lifetime.
"""
columns = [USEquityPricing.high, USEquityPricing.volume]
columns = ['high', 'volume']
for asset in self.assets:
self._check_read_results(
columns,
@@ -238,7 +237,7 @@ class BcolzDailyBarTestCase(WithBcolzDailyBarReader, ZiplineTestCase):
Test loading with queries that start on the last day of each asset's
lifetime.
"""
columns = [USEquityPricing.close, USEquityPricing.volume]
columns = ['close', 'volume']
for asset in self.assets:
self._check_read_results(
columns,
@@ -252,7 +251,7 @@ class BcolzDailyBarTestCase(WithBcolzDailyBarReader, ZiplineTestCase):
Test loading with queries that end on the first day of each asset's
lifetime.
"""
columns = [USEquityPricing.close, USEquityPricing.volume]
columns = ['close', 'volume']
for asset in self.assets:
self._check_read_results(
columns,
@@ -266,7 +265,7 @@ class BcolzDailyBarTestCase(WithBcolzDailyBarReader, ZiplineTestCase):
Test loading with queries that end on the last day of each asset's
lifetime.
"""
columns = [USEquityPricing.close, USEquityPricing.volume]
columns = ['close', 'volume']
for asset in self.assets:
self._check_read_results(
columns,
tests/finance/test_slippage.py
+18 -20
View File
@@ -58,18 +58,16 @@ class SlippageTestCase(WithSimParams, WithDataPortal, ZiplineTestCase):
@classmethod
def make_minute_bar_data(cls):
return {
133: pd.DataFrame(
{
'open': [3.0, 3.0, 3.5, 4.0, 3.5],
'high': [3.15, 3.15, 3.15, 3.15, 3.15],
'low': [2.85, 2.85, 2.85, 2.85, 2.85],
'close': [3.0, 3.5, 4.0, 3.5, 3.0],
'volume': [2000, 2000, 2000, 2000, 2000],
},
index=cls.minutes,
),
}
yield 133, pd.DataFrame(
{
'open': [3.0, 3.0, 3.5, 4.0, 3.5],
'high': [3.15, 3.15, 3.15, 3.15, 3.15],
'low': [2.85, 2.85, 2.85, 2.85, 2.85],
'close': [3.0, 3.5, 4.0, 3.5, 3.0],
'volume': [2000, 2000, 2000, 2000, 2000],
},
index=cls.minutes,
)
@classmethod
def init_class_fixtures(cls):
@@ -77,8 +75,8 @@ class SlippageTestCase(WithSimParams, WithDataPortal, ZiplineTestCase):
cls.ASSET133 = cls.env.asset_finder.retrieve_asset(133)
def test_volume_share_slippage(self):
assets = {
133: pd.DataFrame(
assets = (
(133, pd.DataFrame(
{
'open': [3.00],
'high': [3.15],
@@ -87,8 +85,8 @@ class SlippageTestCase(WithSimParams, WithDataPortal, ZiplineTestCase):
'volume': [200],
},
index=[self.minutes[0]],
),
}
)),
)
days = pd.date_range(
start=normalize_date(self.minutes[0]),
end=normalize_date(self.minutes[-1])
@@ -465,8 +463,8 @@ class SlippageTestCase(WithSimParams, WithDataPortal, ZiplineTestCase):
data['sid'] = self.ASSET133
order = Order(**data)
assets = {
133: pd.DataFrame(
assets = (
(133, pd.DataFrame(
{
'open': [event_data['open']],
'high': [event_data['high']],
@@ -475,8 +473,8 @@ class SlippageTestCase(WithSimParams, WithDataPortal, ZiplineTestCase):
'volume': [event_data['volume']],
},
index=[pd.Timestamp('2006-01-05 14:31', tz='UTC')],
),
}
)),
)
days = pd.date_range(
start=normalize_date(self.minutes[0]),
end=normalize_date(self.minutes[-1])
tests/pipeline/test_engine.py
+13 -12
View File
@@ -40,8 +40,17 @@ from toolz import merge
from zipline.assets.synthetic import make_rotating_equity_info
from zipline.lib.adjustment import MULTIPLY
from zipline.pipeline import CustomFactor, Pipeline
from zipline.pipeline.data import Column, DataSet, USEquityPricing
from zipline.pipeline.loaders.synthetic import PrecomputedLoader
from zipline.pipeline import Pipeline
from zipline.pipeline.data import USEquityPricing, DataSet, Column
from zipline.pipeline.loaders.equity_pricing_loader import (
USEquityPricingLoader,
)
from zipline.pipeline.factors import CustomFactor
from zipline.pipeline.loaders.synthetic import (
make_bar_data,
expected_bar_values_2d,
)
from zipline.pipeline.engine import SimplePipelineEngine
from zipline.pipeline.factors import (
AverageDollarVolume,
@@ -52,15 +61,7 @@ from zipline.pipeline.factors import (
MaxDrawdown,
SimpleMovingAverage,
)
from zipline.pipeline.loaders.equity_pricing_loader import (
USEquityPricingLoader,
)
from zipline.pipeline.loaders.frame import DataFrameLoader
from zipline.pipeline.loaders.synthetic import (
expected_daily_bar_values_2d,
make_daily_bar_data,
PrecomputedLoader,
)
from zipline.pipeline.term import NotSpecified
from zipline.testing import (
product_upper_triangle,
@@ -925,7 +926,7 @@ class SyntheticBcolzTestCase(WithAdjustmentReader,
@classmethod
def make_daily_bar_data(cls):
return make_daily_bar_data(
return make_bar_data(
cls.equity_info,
cls.bcolz_daily_bar_days,
)
@@ -999,7 +1000,7 @@ class SyntheticBcolzTestCase(WithAdjustmentReader,
# computed results to be computed using values anchored on the
# **previous** day's data.
expected_raw = rolling_mean(
expected_daily_bar_values_2d(
expected_bar_values_2d(
dates - self.env.trading_day,
self.equity_info,
'close',
tests/pipeline/test_us_equity_pricing_loader.py
+9 -9
View File
@@ -37,8 +37,8 @@ from toolz.curried.operator import getitem
from zipline.lib.adjustment import Float64Multiply
from zipline.pipeline.loaders.synthetic import (
NullAdjustmentReader,
make_daily_bar_data,
expected_daily_bar_values_2d,
make_bar_data,
expected_bar_values_2d,
)
from zipline.pipeline.loaders.equity_pricing_loader import (
USEquityPricingLoader,
@@ -282,7 +282,7 @@ class USEquityPricingLoaderTestCase(WithAdjustmentReader,
@classmethod
def make_daily_bar_data(cls):
return make_daily_bar_data(
return make_bar_data(
EQUITY_INFO,
cls.bcolz_daily_bar_days,
)
@@ -364,7 +364,7 @@ class USEquityPricingLoaderTestCase(WithAdjustmentReader,
)
adjustments = self.adjustment_reader.load_adjustments(
columns,
[c.name for c in columns],
query_days,
self.assets,
)
@@ -410,7 +410,7 @@ class USEquityPricingLoaderTestCase(WithAdjustmentReader,
)
adjustments = adjustment_reader.load_adjustments(
columns,
[c.name for c in columns],
query_days,
self.assets,
)
@@ -429,12 +429,12 @@ class USEquityPricingLoaderTestCase(WithAdjustmentReader,
)
closes, volumes = map(getitem(results), columns)
expected_baseline_closes = expected_daily_bar_values_2d(
expected_baseline_closes = expected_bar_values_2d(
shifted_query_days,
self.asset_info,
'close',
)
expected_baseline_volumes = expected_daily_bar_values_2d(
expected_baseline_volumes = expected_bar_values_2d(
shifted_query_days,
self.asset_info,
'volume',
@@ -506,12 +506,12 @@ class USEquityPricingLoaderTestCase(WithAdjustmentReader,
)
highs, volumes = map(getitem(results), columns)
expected_baseline_highs = expected_daily_bar_values_2d(
expected_baseline_highs = expected_bar_values_2d(
shifted_query_days,
self.asset_info,
'high',
)
expected_baseline_volumes = expected_daily_bar_values_2d(
expected_baseline_volumes = expected_bar_values_2d(
shifted_query_days,
self.asset_info,
'volume',
tests/resources/example_data.tar.gz
BIN
View File
Binary file not shown.
tests/resources/quandl_samples/AAPL.csv.gz
BIN
View File
Binary file not shown.
tests/resources/quandl_samples/BRK_A.csv.gz
BIN
View File
Binary file not shown.
tests/resources/quandl_samples/MSFT.csv.gz
BIN
View File
Binary file not shown.
tests/resources/quandl_samples/ZEN.csv.gz
BIN
View File
Binary file not shown.
tests/resources/quandl_samples/metadata-1.csv.gz
BIN
View File
Binary file not shown.
tests/resources/quandl_samples/metadata-2.csv.gz
BIN
View File
Binary file not shown.
tests/resources/quandl_samples/rebuild_samples.py
+29 -4
View File
@@ -2,13 +2,14 @@
Script for rebuilding the samples for the Quandl tests.
"""
from __future__ import print_function
from operator import methodcaller
import pandas as pd
import requests
from zipline.data.quandl import format_wiki_url
from zipline.utils.test_utils import test_resource_path, write_compressed
from zipline.testing import test_resource_path, write_compressed
def zipfile_path(symbol):
@@ -18,7 +19,14 @@ def zipfile_path(symbol):
def main():
start_date = pd.Timestamp('2014')
end_date = pd.Timestamp('2015')
symbols = ['AAPL', 'MSFT', 'BRK_A', 'ZEN']
symbols = 'AAPL', 'MSFT', 'BRK_A', 'ZEN'
names = (
'Apple Inc.',
'Microsoft Corporation',
'Berkshire Hathaway Inc. Class A',
'Zendesk Inc',
)
print('Downloading equity data')
for sym in symbols:
url = format_wiki_url(
api_key=None,
@@ -26,14 +34,31 @@ def main():
start_date=start_date,
end_date=end_date,
)
print("Fetching from %s" % url)
print('Fetching from %s' % url)
response = requests.get(url)
response.raise_for_status()
path = zipfile_path(sym)
print("Writing compressed data to %s" % path)
print('Writing compressed data to %s' % path)
write_compressed(path, response.content)
print('Writing mock metadata')
cols = b'dataset_code,name,oldest_available_date,newest_available_date\n'
metadata = cols + b'\n'.join(
b','.join(map(methodcaller('encode', 'ascii'), (
symbol,
name,
str(start_date.date()), str(end_date.date()))
))
for symbol, name in zip(symbols, names)
)
path = zipfile_path('metadata-1')
print('Writing compressed data to %s' % path)
write_compressed(path, metadata)
path = zipfile_path('metadata-2')
print('Writing compressed data to %s' % path)
write_compressed(path, cols)
if __name__ == '__main__':
main()
tests/resources/rebuild_example_data
Executable
+120
View File
@@ -0,0 +1,120 @@
#!/usr/bin/env python
from code import InteractiveConsole
import readline # noqa
import shutil
import tarfile
import click
import numpy as np
import pandas as pd
from zipline import examples, run_algorithm
from zipline.testing import test_resource_path, tmp_dir
from zipline.utils.cache import dataframe_cache
banner = """
Please verify that the new perfomance is more correct than the old performance.
To do this, please inspect `new` and `old` which are mappings from the name of
the example to the results.
If you are sure that the new results are more correct, or that the difference
is acceptable, please call `correct()`. Otherwise, call `incorrect()`.
Note
----
Remember to run this with the other supported versions of pandas!
"""
def eof(*args, **kwargs):
raise EOFError()
@click.command()
@click.pass_context
def main(ctx):
"""Rebuild the perf data for test_examples
"""
example_path = test_resource_path('example_data.tar.gz')
with tmp_dir() as d:
with tarfile.open(example_path) as tar:
tar.extractall(d.path)
mods = (
(e, getattr(examples, e))
for e in dir(examples)
if not e.startswith('_')
)
new_perf_path = d.getpath(
'example_data/new_perf/%s' % pd.__version__.replace('.', '-'),
)
c = dataframe_cache(
new_perf_path,
serialization='pickle:2',
)
with c:
for name, mod in mods:
c[name] = run_algorithm(
handle_data=mod.handle_data,
initialize=mod.initialize,
before_trading_start=getattr(
mod, 'before_trading_start', None,
),
analyze=getattr(mod, 'analyze', None),
bundle='test',
environ={
'ZIPLINE_ROOT': d.getpath('example_data/root'),
},
**mod._test_args()
)
correct_called = [False]
console = None
def _exit(*args, **kwargs):
console.raw_input = eof
def correct():
correct_called[0] = True
_exit()
expected_perf_path = d.getpath(
'example_data/expected_perf/%s' %
pd.__version__.replace('.', '-'),
)
# allow users to run some analysis to make sure that the new
# results check out
console = InteractiveConsole({
'correct': correct,
'exit': _exit,
'incorrect': _exit,
'new': c,
'np': np,
'old': dataframe_cache(
expected_perf_path,
serialization='pickle',
),
'pd': pd,
})
console.interact(banner)
if not correct_called[0]:
ctx.fail(
'`correct()` was not called! This means that the new'
' results will not be written',
)
# move the new results to the expected path
shutil.rmtree(expected_perf_path)
shutil.copytree(new_perf_path, expected_perf_path)
with tarfile.open(example_path, 'w|gz') as tar:
tar.add(d.getpath('example_data'), 'example_data')
if __name__ == '__main__':
main()
tests/resources/yahoo_samples/AAPL.adjustments.gz
BIN
View File
Binary file not shown.
tests/resources/yahoo_samples/AAPL.csv.gz
BIN
View File
Binary file not shown.
tests/resources/yahoo_samples/IBM.adjustments.gz
BIN
View File
Binary file not shown.
tests/resources/yahoo_samples/IBM.csv.gz
BIN
View File
Binary file not shown.
tests/resources/yahoo_samples/MSFT.adjustments.gz
BIN
View File
Binary file not shown.
tests/resources/yahoo_samples/MSFT.csv.gz
BIN
View File
Binary file not shown.
tests/resources/yahoo_samples/rebuild_samples
+79
View File
@@ -0,0 +1,79 @@
#!/usr/bin/env python
"""
Script for rebuilding the samples for the Yahoo tests.
"""
from textwrap import dedent
import numpy as np
import pandas as pd
from zipline.testing import test_resource_path, write_compressed
from zipline.utils.tradingcalendar import trading_days
def zipfile_path(symbol, ext):
return test_resource_path('yahoo_samples', symbol + ext + '.gz')
def pricing_for_sid(sid):
modifier = {
'Low': 0,
'Open': 1,
'Close': 2,
'High': 3,
'Volume': 0,
}
def column(name):
return np.arange(252) + 1 + sid * 10000 + modifier[name] * 1000
return pd.DataFrame(
data={
'Date': trading_days[
(trading_days >= pd.Timestamp('2014')) &
(trading_days < pd.Timestamp('2015'))
],
'Open': column('Open'),
'High': column('High'),
'Low': column('Low'),
'Close': column('Close'),
'Volume': column('Volume'),
'Adj Close': 0,
},
columns=[
'Date', 'Open', 'High', 'Low', 'Close', 'Volume', 'Adj Close',
],
).to_csv(index=False, date_format='%Y-%m-%d').encode('ascii')
def adjustments_for_sid(sid):
"""This is not exactly a csv... thanks yahoo.
"""
return dedent(
"""\
Date,Dividends
DIVIDEND, 20140403,0.{p1}00000
SPLIT, 20140703,{p2}:1
DIVIDEND, 20141002,0.{p2}00000
STARTDATE, 20140102
ENDDATE, 20141231
TOTALSIZE, 2
""".format(p1=sid + 1, p2=sid + 2),
).encode('ascii')
def main():
symbols = 'AAPL', 'IBM', 'MSFT'
for sid, symbol in enumerate(symbols):
write_compressed(
zipfile_path(symbol, '.csv'),
pricing_for_sid(sid),
)
write_compressed(
zipfile_path(symbol, '.adjustments'),
adjustments_for_sid(sid),
)
if __name__ == '__main__':
main()
tests/test_algorithm.py
+23 -31
View File
@@ -29,7 +29,6 @@ from testfixtures import TempDirectory
import numpy as np
import pandas as pd
import pytz
from toolz import merge
from zipline import TradingAlgorithm
from zipline.api import FixedSlippage
@@ -1039,25 +1038,20 @@ class TestBeforeTradingStart(WithDataPortal,
index=asset_minutes,
)
split_data.iloc[780:] = split_data.iloc[780:] / 2.0
return merge(
{
sid: create_minute_df_for_asset(
cls.env,
cls.data_start,
cls.sim_params.period_end,
)
for sid in (1, 8554)
},
{
2: create_minute_df_for_asset(
cls.env,
cls.data_start,
cls.sim_params.period_end,
50,
),
cls.SPLIT_ASSET_SID: split_data,
},
for sid in (1, 8554):
yield sid, create_minute_df_for_asset(
cls.env,
cls.data_start,
cls.sim_params.period_end,
)
yield 2, create_minute_df_for_asset(
cls.env,
cls.data_start,
cls.sim_params.period_end,
50,
)
yield cls.SPLIT_ASSET_SID, split_data
@classmethod
def make_splits_data(cls):
@@ -2552,18 +2546,16 @@ class TestOrderCancelation(WithDataPortal,
minutes_arr = np.arange(1, 1 + minutes_count)
# normal test data, but volume is pinned at 1 share per minute
return {
1: pd.DataFrame(
{
'open': minutes_arr + 1,
'high': minutes_arr + 2,
'low': minutes_arr - 1,
'close': minutes_arr,
'volume': np.full(minutes_count, 1),
},
index=asset_minutes,
),
}
yield 1, pd.DataFrame(
{
'open': minutes_arr + 1,
'high': minutes_arr + 2,
'low': minutes_arr - 1,
'close': minutes_arr,
'volume': np.full(minutes_count, 1),
},
index=asset_minutes,
)
@classmethod
def make_daily_bar_data(cls):
tests/test_api_shim.py
+2 -4
View File
@@ -122,14 +122,12 @@ class TestAPIShim(WithDataPortal, WithSimParams, ZiplineTestCase):
@classmethod
def make_minute_bar_data(cls):
return {
sid: create_minute_df_for_asset(
for sid in cls.sids:
yield sid, create_minute_df_for_asset(
cls.env,
cls.SIM_PARAMS_START,
cls.SIM_PARAMS_END,
)
for sid in cls.sids
}
@classmethod
def make_daily_bar_data(cls):
tests/test_bar_data.py
+20 -27
View File
@@ -15,7 +15,6 @@
from nose_parameterized import parameterized
import numpy as np
import pandas as pd
from toolz import merge
from zipline._protocol import handle_non_market_minutes
from zipline.protocol import BarData
@@ -109,32 +108,26 @@ class TestMinuteBarData(WithBarDataChecks,
# asset2 has trades every 10 minutes
# split_asset trades every minute
# illiquid_split_asset trades every 10 minutes
return merge(
{
sid: create_minute_df_for_asset(
cls.env,
cls.bcolz_minute_bar_days[0],
cls.bcolz_minute_bar_days[-1],
)
for sid in (1, cls.SPLIT_ASSET_SID)
},
{
sid: create_minute_df_for_asset(
cls.env,
cls.bcolz_minute_bar_days[0],
cls.bcolz_minute_bar_days[-1],
10,
)
for sid in (2, cls.ILLIQUID_SPLIT_ASSET_SID)
},
{
cls.HILARIOUSLY_ILLIQUID_ASSET_SID: create_minute_df_for_asset(
cls.env,
cls.bcolz_minute_bar_days[0],
cls.bcolz_minute_bar_days[-1],
50,
)
},
for sid in (1, cls.SPLIT_ASSET_SID):
yield sid, create_minute_df_for_asset(
cls.env,
cls.bcolz_minute_bar_days[0],
cls.bcolz_minute_bar_days[-1],
)
for sid in (2, cls.ILLIQUID_SPLIT_ASSET_SID):
yield sid, create_minute_df_for_asset(
cls.env,
cls.bcolz_minute_bar_days[0],
cls.bcolz_minute_bar_days[-1],
10,
)
yield cls.HILARIOUSLY_ILLIQUID_ASSET_SID, create_minute_df_for_asset(
cls.env,
cls.bcolz_minute_bar_days[0],
cls.bcolz_minute_bar_days[-1],
50,
)
@classmethod
tests/test_cli.py
-70
View File
@@ -1,70 +0,0 @@
#
# Copyright 2014 Quantopian, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import os
from unittest import TestCase
from six import iteritems
from zipline.utils import parse_args
from zipline.utils import cli
class TestParseArgs(TestCase):
def test_defaults(self):
args = parse_args([])
for k, v in iteritems(cli.DEFAULTS):
self.assertEqual(v, args[k])
def write_conf_file(self):
conf_str = """
[Defaults]
algofile=test.py
symbols=test_symbols
start=1990-1-1
"""
with open('test.conf', 'w') as fd:
fd.write(conf_str)
def test_conf_file(self):
self.write_conf_file()
try:
args = parse_args(['-c', 'test.conf'])
self.assertEqual(args['algofile'], 'test.py')
self.assertEqual(args['symbols'], 'test_symbols')
self.assertEqual(args['start'], '1990-1-1')
self.assertEqual(args['data_frequency'],
cli.DEFAULTS['data_frequency'])
finally:
os.remove('test.conf')
def test_overwrite(self):
self.write_conf_file()
try:
args = parse_args(['-c', 'test.conf', '--start', '1992-1-1',
'--algofile', 'test2.py'])
# Overwritten values
self.assertEqual(args['algofile'], 'test2.py')
self.assertEqual(args['start'], '1992-1-1')
# Non-overwritten values
self.assertEqual(args['symbols'], 'test_symbols')
# Default values
self.assertEqual(args['data_frequency'],
cli.DEFAULTS['data_frequency'])
finally:
os.remove('test.conf')
tests/test_examples.py
+81 -21
View File
@@ -15,15 +15,17 @@
# This code is based on a unittest written by John Salvatier:
# https://github.com/pymc-devs/pymc/blob/pymc3/tests/test_examples.py
import tarfile
import glob
import matplotlib
from nose_parameterized import parameterized
import os
import runpy
from unittest import TestCase
import pandas as pd
from zipline.utils import parse_args, run_pipeline
from zipline import examples, run_algorithm
from zipline.testing import test_resource_path
from zipline.testing.fixtures import WithTmpDir, ZiplineTestCase
from zipline.testing.predicates import assert_equal
from zipline.utils.cache import dataframe_cache
# Otherwise the next line sometimes complains about being run too late.
_multiprocess_can_split_ = False
@@ -31,22 +33,80 @@ _multiprocess_can_split_ = False
matplotlib.use('Agg')
def example_dir():
import zipline
d = os.path.dirname(zipline.__file__)
return os.path.join(os.path.abspath(d), 'examples')
class ExamplesTests(WithTmpDir, ZiplineTestCase):
# some columns contain values with unique ids that will not be the same
cols_to_check = [
'algo_volatility',
'algorithm_period_return',
'alpha',
'benchmark_period_return',
'benchmark_volatility',
'beta',
'capital_used',
'ending_cash',
'ending_exposure',
'ending_value',
'excess_return',
'gross_leverage',
'long_exposure',
'long_value',
'longs_count',
'max_drawdown',
'max_leverage',
'net_leverage',
'period_close',
'period_label',
'period_open',
'pnl',
'portfolio_value',
'positions',
'returns',
'short_exposure',
'short_value',
'shorts_count',
'sortino',
'starting_cash',
'starting_exposure',
'starting_value',
'trading_days',
'treasury_period_return',
]
@classmethod
def init_class_fixtures(cls):
super(ExamplesTests, cls).init_class_fixtures()
class ExamplesTests(TestCase):
# Test algorithms as if they are executed directly from the command line.
@parameterized.expand(((os.path.basename(f).replace('.', '_'), f) for f in
glob.glob(os.path.join(example_dir(), '*.py'))))
def test_example(self, name, example):
runpy.run_path(example, run_name='__main__')
with tarfile.open(test_resource_path('example_data.tar.gz')) as tar:
tar.extractall(cls.tmpdir.path)
# Test algorithm as if scripts/run_algo.py is being used.
def test_example_run_pipeline(self):
example = os.path.join(example_dir(), 'buyapple.py')
confs = ['-f', example, '--start', '2011-1-1', '--end', '2012-1-1']
parsed_args = parse_args(confs)
run_pipeline(**parsed_args)
cls.expected_perf = dataframe_cache(
cls.tmpdir.getpath(
'example_data/expected_perf/%s' %
pd.__version__.replace('.', '-'),
),
serialization='pickle',
)
@parameterized.expand(e for e in dir(examples) if not e.startswith('_'))
def test_example(self, example):
mod = getattr(examples, example)
actual_perf = run_algorithm(
handle_data=mod.handle_data,
initialize=mod.initialize,
before_trading_start=getattr(mod, 'before_trading_start', None),
analyze=getattr(mod, 'analyze', None),
bundle='test',
environ={
'ZIPLINE_ROOT': self.tmpdir.getpath('example_data/root'),
},
capital_base=1e7,
**mod._test_args()
)
assert_equal(
actual_perf[self.cols_to_check],
self.expected_perf[example][self.cols_to_check],
# There is a difference in the datetime columns in pandas
# 0.16 and 0.17 because in 16 they are object and in 17 they are
# datetime[ns, UTC]. We will just ignore the dtypes for now.
check_dtype=False,
)
tests/test_finance.py
+2 -1
View File
@@ -23,6 +23,7 @@ from nose.tools import timed
import numpy as np
import pandas as pd
import pytz
from six import iteritems
from six.moves import range
from testfixtures import TempDirectory
@@ -219,7 +220,7 @@ class FinanceTestCase(WithLogger,
env,
env.days_in_range(minutes[0], minutes[-1]),
tempdir.path,
assets
iteritems(assets),
)
equity_minute_reader = BcolzMinuteBarReader(tempdir.path)
tests/test_history.py
+44 -48
View File
@@ -2,12 +2,12 @@ from textwrap import dedent
from numbers import Real
import pandas as pd
from nose_parameterized import parameterized
import numpy as np
from numpy import nan
from numpy.testing import assert_almost_equal
from nose_parameterized import parameterized
import pandas as pd
from six import iteritems
from zipline import TradingAlgorithm
from zipline._protocol import handle_non_market_minutes
@@ -473,7 +473,7 @@ class MinuteEquityHistoryTestCase(WithHistory, ZiplineTestCase):
start_val=2,
interval=10,
)
return data
return iteritems(data)
def test_history_in_initialize(self):
algo_text = dedent(
@@ -986,24 +986,22 @@ class DailyEquityHistoryTestCase(WithHistory, ZiplineTestCase):
def make_minute_bar_data(cls):
asset1 = cls.asset_finder.retrieve_asset(1)
asset2 = cls.asset_finder.retrieve_asset(2)
return {
asset1.sid: create_minute_df_for_asset(
cls.env,
asset1.start_date,
asset1.end_date,
start_val=2,
),
asset2.sid: create_minute_df_for_asset(
cls.env,
asset2.start_date,
cls.env.previous_trading_day(asset2.end_date),
start_val=2,
minute_blacklist=[
pd.Timestamp('2015-01-08 14:31', tz='UTC'),
pd.Timestamp('2015-01-08 21:00', tz='UTC'),
],
),
}
yield asset1.sid, create_minute_df_for_asset(
cls.env,
asset1.start_date,
asset1.end_date,
start_val=2,
)
yield asset2.sid, create_minute_df_for_asset(
cls.env,
asset2.start_date,
cls.env.previous_trading_day(asset2.end_date),
start_val=2,
minute_blacklist=[
pd.Timestamp('2015-01-08 14:31', tz='UTC'),
pd.Timestamp('2015-01-08 21:00', tz='UTC'),
],
)
@classmethod
def create_df_for_asset(cls, start_day, end_day, interval=1,
@@ -1548,32 +1546,30 @@ class MinuteToDailyAggregationTestCase(WithBcolzMinuteBarReader,
@classmethod
def make_minute_bar_data(cls):
return {
# sid data is created so that at least one high is lower than a
# previous high, and the inverse for low
1: pd.DataFrame(
{
'open': [nan, 103.50, 102.50, 104.50, 101.50, nan],
'high': [nan, 103.90, 102.90, 104.90, 101.90, nan],
'low': [nan, 103.10, 102.10, 104.10, 101.10, nan],
'close': [nan, 103.30, 102.30, 104.30, 101.30, nan],
'volume': [0, 1003, 1002, 1004, 1001, 0]
},
index=cls.minutes,
),
# sid 2 is included to provide data on different bars than sid 1,
# as will as illiquidty mid-day
2: pd.DataFrame(
{
'open': [201.50, nan, 204.50, nan, 200.50, 202.50],
'high': [201.90, nan, 204.90, nan, 200.90, 202.90],
'low': [201.10, nan, 204.10, nan, 200.10, 202.10],
'close': [201.30, nan, 203.50, nan, 200.30, 202.30],
'volume': [2001, 0, 2004, 0, 2000, 2002],
},
index=cls.minutes,
),
}
# sid data is created so that at least one high is lower than a
# previous high, and the inverse for low
yield 1, pd.DataFrame(
{
'open': [nan, 103.50, 102.50, 104.50, 101.50, nan],
'high': [nan, 103.90, 102.90, 104.90, 101.90, nan],
'low': [nan, 103.10, 102.10, 104.10, 101.10, nan],
'close': [nan, 103.30, 102.30, 104.30, 101.30, nan],
'volume': [0, 1003, 1002, 1004, 1001, 0]
},
index=cls.minutes,
)
# sid 2 is included to provide data on different bars than sid 1,
# as will as illiquidty mid-day
yield 2, pd.DataFrame(
{
'open': [201.50, nan, 204.50, nan, 200.50, 202.50],
'high': [201.90, nan, 204.90, nan, 200.90, 202.90],
'low': [201.10, nan, 204.10, nan, 200.10, 202.10],
'close': [201.30, nan, 203.50, nan, 200.30, 202.30],
'volume': [2001, 0, 2004, 0, 2000, 2002],
},
index=cls.minutes,
)
expected_values = {
1: pd.DataFrame(
zipline/__init__.py
+9 -9
View File
@@ -20,6 +20,7 @@ from . import data
from . import finance
from . import gens
from . import utils
from .utils.run_algo import run_algorithm
from ._version import get_versions
# These need to happen after the other imports.
from . algorithm import TradingAlgorithm
@@ -28,19 +29,18 @@ from . import api
__version__ = get_versions()['version']
del get_versions
try:
ip = get_ipython() # flake8: noqa
except NameError:
pass
else:
ip.register_magic_function(utils.parse_cell_magic, "line_cell", "zipline")
del ip
def load_ipython_extension(ipython):
from .__main__ import zipline_magic
ipython.register_magic_function(zipline_magic, 'line_cell', 'zipline')
__all__ = [
'TradingAlgorithm',
'api',
'data',
'finance',
'gens',
'run_algorithm',
'utils',
'api',
'TradingAlgorithm',
]
zipline/__main__.py
+332
View File
@@ -0,0 +1,332 @@
import datetime
import os
from functools import wraps
import click
import logbook
import pandas as pd
from zipline.data import bundles
from zipline.utils.cli import Date, Timestamp
from zipline.utils.run_algo import _run, load_extensions
try:
__IPYTHON__
except NameError:
__IPYTHON__ = False
@click.group()
@click.option(
'-e',
'--extension',
multiple=True,
help='File or module path to a zipline extension to load.',
)
@click.option(
'--strict-extensions/--non-strict-extensions',
is_flag=True,
help='If --strict-extensions is passed then zipline will not run if it'
' cannot load all of the specified extensions. If this is not passed or'
' --non-strict-extensions is passed then the failure will be logged but'
' execution will continue.',
)
@click.option(
'--default-extension/--no-default-extension',
is_flag=True,
default=True,
help="Don't load the default zipline extension.py file in $ZIPLINE_HOME.",
)
def cli(extension, strict_extensions, default_extension):
"""Top level zipline entry point.
"""
# install a logbook handler before performing any other operations
logbook.StderrHandler().push_application()
load_extensions(
default_extension,
extension,
strict_extensions,
os.environ,
)
def extract_option_object(option):
"""Convert a click.option call into a click.Option object.
Parameters
----------
option : decorator
A click.option decorator.
Returns
-------
option_object : click.Option
The option object that this decorator will create.
"""
@option
def opt():
pass
return opt.__click_params__[0]
def ipython_only(option):
"""Mark that an option should only be exposed in IPython.
Parameters
----------
option : decorator
A click.option decorator.
Returns
-------
ipython_only_dec : decorator
A decorator that correctly applies the argument even when not
using IPython mode.
"""
if __IPYTHON__:
return option
argname = extract_option_object(option).name
def d(f):
@wraps(f)
def _(*args, **kwargs):
kwargs[argname] = None
return f(*args, **kwargs)
return _
return d
@cli.command()
@click.option(
'-f',
'--algofile',
default=None,
type=click.File('r'),
help='The file that contains the algorithm to run.',
)
@click.option(
'-t',
'--algotext',
help='The algorithm script to run.',
)
@click.option(
'-D',
'--define',
multiple=True,
help="Define a name to be bound in the namespace before executing"
" the algotext. For example '-Dname=value'. The value may be any python"
" expression. These are evaluated in order so they may refer to previously"
" defined names.",
)
@click.option(
'--data-frequency',
type=click.Choice({'daily', 'minute'}),
default='daily',
show_default=True,
help='The data frequency of the simulation.',
)
@click.option(
'--capital-base',
type=float,
default=10e6,
show_default=True,
help='The starting capital for the simulation.',
)
@click.option(
'-b',
'--bundle',
default='quandl',
metavar='BUNDLE-NAME',
show_default=True,
help='The data bundle to use for the simulation.',
)
@click.option(
'--bundle-timestamp',
type=Timestamp(),
default=pd.Timestamp.utcnow(),
show_default=False,
help='The date to lookup data on or before.\n'
'[default: <current-time>]'
)
@click.option(
'-s',
'--start',
type=Date(tz='utc', as_timestamp=True),
help='The start date of the simulation.',
)
@click.option(
'-e',
'--end',
type=Date(tz='utc', as_timestamp=True),
help='The end date of the simulation.',
)
@click.option(
'-o',
'--output',
default='-',
metavar='FILENAME',
show_default=True,
help="The location to write the perf data. If this is '-' the perf will"
" be written to stdout.",
)
@click.option(
'--print-algo/--no-print-algo',
is_flag=True,
default=False,
help='Print the algorithm to stdout.',
)
@ipython_only(click.option(
'--local-namespace/--no-local-namespace',
is_flag=True,
default=None,
help='Should the algorithm methods be resolved in the local namespace.'
))
@click.pass_context
def run(ctx,
algofile,
algotext,
define,
data_frequency,
capital_base,
bundle,
bundle_timestamp,
start,
end,
output,
print_algo,
local_namespace):
"""Run a backtest for the given algorithm.
"""
# check that the start and end dates are passed correctly
if start is None and end is None:
# check both at the same time to avoid the case where a user
# does not pass either of these and then passes the first only
# to be told they need to pass the second argument also
ctx.fail(
"must specify dates with '-s' / '--start' and '-e' / '--end'",
)
if start is None:
ctx.fail("must specify a start date with '-s' / '--start'")
if end is None:
ctx.fail("must specify an end date with '-s' / '--end'")
if (algotext is not None) == (algofile is not None):
ctx.fail(
"must specify exactly one of '-f' / '--algofile' or"
" '-t' / '--algotext'",
)
perf = _run(
initialize=None,
handle_data=None,
before_trading_start=None,
analyze=None,
algofile=algofile,
algotext=algotext,
defines=define,
data_frequency=data_frequency,
capital_base=capital_base,
data=None,
bundle=bundle,
bundle_timestamp=bundle_timestamp,
start=start,
end=end,
output=output,
print_algo=print_algo,
local_namespace=local_namespace,
environ=os.environ,
)
if output == '-':
click.echo(str(perf))
elif output != os.devnull: # make the zipline magic not write any data
perf.to_pickle(output)
return perf
def zipline_magic(line, cell=None):
"""The zipline IPython cell magic.
"""
try:
return run.main(
# put our overrides at the start of the parameter list so that
# users may pass values with higher precedence
[
'--algotext', cell,
'--output', os.devnull, # don't write the results by default
] + ([
# these options are set when running in line magic mode
# set a non None algo text to use the ipython user_ns
'--algotext', '',
'--local-namespace',
] if cell is None else []) + line.split(),
'%s%%zipline' % ((cell or '') and '%'),
# don't use system exit and propogate errors to the caller
standalone_mode=False,
)
except SystemExit as e:
# https://github.com/mitsuhiko/click/pull/533
# even in standalone_mode=False `--help` really wants to kill us ;_;
if e.code:
raise ValueError('main returned non-zero status code: %d' % e.code)
@cli.command()
@click.argument('BUNDLE-NAME')
@click.option(
'--show-progress/--no-show-progress',
is_flag=True,
default=True,
help='Print progress information to the terminal.'
)
def ingest(bundle_name, show_progress):
"""Ingest the data for the given bundle.
"""
bundles.ingest(
bundle_name,
os.environ,
datetime.date.today(),
show_progress,
)
@cli.command()
@click.argument('BUNDLE-NAME')
@click.option(
'-b',
'--before',
type=Timestamp(),
help='Clear all data before TIMESTAMP.'
' This may not be passed with -k / --keep-last',
)
@click.option(
'-a',
'--after',
type=Timestamp(),
help='Clear all data after TIMESTAMP'
' This may not be passed with -k / --keep-last',
)
@click.option(
'-k',
'--keep-last',
type=int,
metavar='N',
help='Clear all but the last N downloads.'
' This may not be passed with -b / --before or -a / --after',
)
def clean(bundle_name, before, after, keep_last):
"""Clean up data downloaded with the ingest command.
"""
bundles.clean(
bundle_name,
before,
after,
keep_last,
)
if __name__ == '__main__':
cli()
zipline/algorithm.py
+1 -1
View File
@@ -116,7 +116,7 @@ from zipline.gens.sim_engine import (
from zipline.sources.benchmark_source import BenchmarkSource
from zipline.zipline_warnings import ZiplineDeprecationWarning
DEFAULT_CAPITAL_BASE = float("1.0e5")
DEFAULT_CAPITAL_BASE = 1e5
log = logbook.Logger("ZiplineLog")
zipline/assets/asset_writer.py
+89
View File
@@ -257,6 +257,95 @@ class AssetDBWriter(object):
exchanges=None,
root_symbols=None,
chunk_size=DEFAULT_CHUNK_SIZE):
"""Write asset metadata to a sqlite database.
Parameters
----------
equities : pd.DataFrame, optional
The equity metadata. The columns for this dataframe are:
symbol : str
The ticker symbol for this equity.
fuzzy_symbol : str, optional
The fuzzy symbol for this equity. This is the symbol
without any delimiting characters like '.' or '_'.
asset_name : str
The full name for this asset.
start_date : datetime
The date when this asset was created.
end_date : datetime, optional
The last date we have trade data for this asset.
first_traded : datetime, optional
The first date we have trade data for this asset.
auto_close_date : datetime, optional
The date on which to close any positions in this asset.
exchange : str, optional
The exchange where this asset is traded.
The index of this dataframe should contain the sids.
futures : pd.Dataframe, optional
The future contract metadata. The columns for this dataframe are:
symbol : str
The ticker symbol for this futures contract.
root_symbol : str
The root symbol, or the symbol with the expiration stripped
out.
asset_name : str
The full name for this asset.
start_date : datetime, optional
The date when this asset was created.
end_date : datetime, optional
The last date we have trade data for this asset.
first_traded : datetime, optional
The first date we have trade data for this asset.
exchange : str, optional
The exchange where this asset is traded.
notice_date : datetime
The date when the owner of the contract may be forced
to take physical delivery of the contract's asset.
expiration_date : datetime
The date when the contract expires.
auto_close_date : datetime
The date when the broker will automatically close any
positions in this contract.
tick_size : float
The minimum price movement of the contract.
multiplier: float
The amount of the underlying asset represented by this
contract.
exchanges : pd.Dataframe, optional
The exchanges where assets can be traded. The columns of this
dataframe are:
exchange : str
The name of the exchange.
timezone : str
The timezone of the exchange.
root_symbols : pd.Dataframe, optional
The root symbols for the futures contracts. The columns for this
dataframe are:
root_symbol : str
The root symbol name.
root_symbol_id : int
The unique id for this root symbol.
sector : string, optional
The sector of this root symbol.
description : string, optional
A short description of this root symbol.
exchange : str
The exchange where this root symbol is traded.
chunk_size : int, optional
The amount of rows to write to the SQLite table at once.
This defaults to the default number of bind params in sqlite.
If you have compiled sqlite3 with more bind or less params you may
want to pass that value here.
See Also
--------
zipline.assets.asset_finder
"""
with self.engine.begin() as txn:
# Create SQL tables if they do not exist.
zipline/assets/assets.py
+1 -1
View File
@@ -94,7 +94,7 @@ class AssetFinder(object):
See Also
--------
:class:`zipline.assets.asset_writer.AssetDBWriter`
:class:`zipline.assets.AssetDBWriter`
"""
# Token used as a substitute for pickling objects that contain a
# reference to an AssetFinder.
zipline/data/__init__.py
+12 -4
View File
@@ -1,8 +1,16 @@
from . import loader
from .loader import (
load_from_yahoo, load_bars_from_yahoo, load_prices_from_csv,
load_prices_from_csv_folder
load_from_yahoo,
load_bars_from_yahoo,
load_prices_from_csv,
load_prices_from_csv_folder,
)
__all__ = ['loader', 'load_from_yahoo', 'load_bars_from_yahoo',
'load_prices_from_csv', 'load_prices_from_csv_folder']
__all__ = [
'load_bars_from_yahoo',
'load_from_yahoo',
'load_prices_from_csv',
'load_prices_from_csv_folder',
'loader',
]
zipline/data/bundles/__init__.py
+20
View File
@@ -0,0 +1,20 @@
from .core import (
bundles,
clean,
ingest,
load,
register,
unregister,
)
from .yahoo import yahoo_equities
__all__ = [
'bundles',
'clean',
'ingest',
'load',
'register',
'unregister',
'yahoo_equities',
]
zipline/data/bundles/core.py
+469
View File
@@ -0,0 +1,469 @@
from collections import namedtuple
import errno
import os
import shutil
import warnings
import click
import pandas as pd
from toolz import curry, complement, compose
from ..us_equity_pricing import (
BcolzDailyBarReader,
BcolzDailyBarWriter,
SQLiteAdjustmentReader,
SQLiteAdjustmentWriter,
)
from ..minute_bars import (
BcolzMinuteBarReader,
BcolzMinuteBarWriter,
)
from zipline.assets import AssetDBWriter, AssetFinder, ASSET_DB_VERSION
from zipline.utils.cache import (
dataframe_cache,
working_file,
working_dir,
)
from zipline.utils.compat import mappingproxy
from zipline.utils.input_validation import ensure_timestamp, optionally
import zipline.utils.paths as pth
from zipline.utils.preprocess import preprocess
from zipline.utils.tradingcalendar import trading_days, open_and_closes
def asset_db_path(bundle_name, timestr, environ=None):
return pth.data_path(
[bundle_name, timestr, 'assets-%d.sqlite' % ASSET_DB_VERSION],
environ=environ,
)
def minute_equity_path(bundle_name, timestr, environ=None):
return pth.data_path(
[bundle_name, timestr, 'minute_equities.bcolz'],
environ=environ,
)
def daily_equity_path(bundle_name, timestr, environ=None):
return pth.data_path(
[bundle_name, timestr, 'daily_equities.bcolz'],
environ=environ,
)
def adjustment_db_path(bundle_name, timestr, environ=None):
return pth.data_path(
[bundle_name, timestr, 'adjustments.sqlite'],
environ=environ,
)
def cache_path(bundle_name, timestr, environ=None):
return pth.data_path(
[bundle_name, timestr, '.cache'],
environ=environ,
)
_BundlePayload = namedtuple(
'_BundlePayload',
'calendar opens closes minutes_per_day ingest',
)
class UnknownBundle(click.ClickException, LookupError):
"""Raised if no bundle with the given name was registered.
"""
exit_code = 1
def __init__(self, name):
super(UnknownBundle, self).__init__(
'No bundle registered with the name %r' % name,
)
self.name = name
def __str__(self):
return self.message
def _make_bundle_core():
"""Create a family of data bundle functions that read from the same
bundle mapping.
Returns
-------
bundles : mappingproxy
The mapping of bundles to bundle payloads.
register : callable
The function which registers new bundles in the ``bundles`` mapping.
unregister : callable
The function which deregisters bundles from the ``bundles`` mapping.
ingest_bundle : callable
The function which downloads and write data for a given data bundle.
"""
_bundles = {} # the registered bundles
# Expose _bundles through a proxy so that users cannot mutate this
# accidentally. Users may go through `register` to update this which will
# warn when trampling another bundle.
bundles = mappingproxy(_bundles)
@curry
def register(name,
f,
calendar=trading_days,
opens=open_and_closes['market_open'],
closes=open_and_closes['market_close'],
minutes_per_day=390):
"""Register a data bundle ingest function.
Parameters
----------
name : str
The name of the bundle.
f : callable
The ingest function. This function will be passed:
environ : mapping
The environment this is being run with.
asset_db_writer : AssetDBWriter
The asset db writer to write into.
minute_bar_writer : BcolzMinuteBarWriter
The minute bar writer to write into.
daily_bar_writer : BcolzDailyBarWriter
The daily bar writer to write into.
adjustment_writer : SQLiteAdjustmentWriter
The adjustment db writer to write into.
calendar : pd.DatetimeIndex
The trading calendar to ingest for.
cache : DataFrameCache
A mapping object to temporarily store dataframes.
This should be used to cache intermediates in case the load
fails. This will be automatically cleaned up after a
successful load.
show_progress : bool
Show the progress for the current load where possible.
calendar : pd.DatetimeIndex, optional
The exchange calendar to align the data to. This defaults to the
NYSE calendar.
market_open : pd.DatetimeIndex, optional
The minute when the market opens each day. This defaults to the
NYSE calendar.
market_close : pd.DatetimeIndex, optional
The minute when the market closes each day. This defaults to the
NYSE calendar.
minutes_per_day : int, optional
The number of minutes in each normal trading day.
Notes
-----
This function my be used as a decorator, for example:
.. code-block:: python
@register('quandl')
def quandl_ingest_function(...):
...
See Also
--------
zipline.data.bundles.bundles
"""
if name in bundles:
warnings.warn(
'Overwriting bundle with name %r' % name,
stacklevel=3,
)
_bundles[name] = _BundlePayload(
calendar,
opens,
closes,
minutes_per_day,
f,
)
return f
def unregister(name):
"""Unregister a bundle.
Parameters
----------
name : str
The name of the bundle to unregister.
Raises
------
UnknownBundle
Raised when no bundle has been registered with the given name.
See Also
--------
zipline.data.bundles.bundles
"""
try:
del _bundles[name]
except KeyError:
raise UnknownBundle(name)
def ingest(name,
environ=os.environ,
timestamp=None,
show_progress=True):
"""Ingest data for a given bundle.
Parameters
----------
name : str
The name of the bundle.
environ : mapping, optional
The environment variables. By default this is os.environ.
timestamp : datetime, optional
The timestamp to use for the load.
By default this is the current time.
show_progress : bool, optional
Tell the ingest function to display the progress where possible.
"""
try:
bundle = bundles[name]
except KeyError:
raise UnknownBundle(name)
if timestamp is None:
timestamp = pd.Timestamp.utcnow()
timestamp = timestamp.tz_convert('utc').tz_localize(None)
timestr = str(timestamp.value)
cachepath = cache_path(name, timestr, environ=environ)
pth.ensure_directory(cachepath)
with dataframe_cache(cachepath, clean_on_failure=False) as cache, \
working_dir(
daily_equity_path(name, timestr, environ=environ),
) as daily_bars_dir, \
working_dir(
minute_equity_path(name, timestr, environ=environ),
) as minute_bars_dir, \
working_file(
asset_db_path(name, timestr, environ=environ),
) as asset_db_file, \
working_file(
adjustment_db_path(name, timestr, environ=environ),
) as adjustment_db_file:
# we use `cleanup_on_failure=False` so that we don't purge the
# cache directory if the load fails in the middle
daily_bar_writer = BcolzDailyBarWriter(
daily_bars_dir.name,
bundle.calendar,
)
# Do an empty write to ensure that the daily ctables exist
# when we create the SQLiteAdjustmentWriter below. The
# SQLiteAdjustmentWriter needs to open the daily ctables so that
# it can compute the adjustment ratios for the dividends.
daily_bar_writer.write(())
bundle.ingest(
environ,
AssetDBWriter(asset_db_file.name),
BcolzMinuteBarWriter(
bundle.calendar[0],
minute_bars_dir.name,
bundle.opens,
bundle.closes,
minutes_per_day=bundle.minutes_per_day,
),
daily_bar_writer,
SQLiteAdjustmentWriter(
adjustment_db_file.name,
BcolzDailyBarReader(daily_bars_dir.name),
bundle.calendar,
overwrite=True,
),
bundle.calendar,
cache,
show_progress,
)
return bundles, register, unregister, ingest
bundles, register, unregister, ingest = _make_bundle_core()
BundleData = namedtuple(
'BundleData',
'asset_finder minute_bar_reader daily_bar_reader adjustment_reader',
)
def most_recent_data(bundle_name, timestamp, environ=None):
"""Get the path to the most recent data after ``date``for the given bundle.
Parameters
----------
bundle_name : str
The name of the bundle to lookup.
timestamp : datetime
The timestamp to begin searching on or before.
environ : dict, optional
An environment dict to forward to zipline_root.
"""
try:
candidates = os.listdir(pth.data_path([bundle_name], environ=environ))
return pth.data_path(
[bundle_name,
max(
filter(complement(pth.hidden), candidates),
key=compose(pd.Timestamp, int),
)],
environ=environ,
)
except ValueError:
raise ValueError(
'no data for bundle %r on or before %s' % (
bundle_name,
timestamp,
),
)
except OSError as e:
if e.errno != errno.ENOENT:
raise
raise UnknownBundle(bundle_name)
def load(name, environ=os.environ, timestamp=None):
"""Loads a previously ingested bundle.
Parameters
----------
name : str
The name of the bundle.
environ : mapping, optional
The environment variables. Defaults of os.environ.
timestamp : datetime, optional
The timestamp of the data to lookup.
Defaults to the current time.
Returns
-------
bundle_data : BundleData
The raw data readers for this bundle.
"""
if timestamp is None:
timestamp = pd.Timestamp.utcnow()
timestr = most_recent_data(name, timestamp, environ=environ)
return BundleData(
asset_finder=AssetFinder(
asset_db_path(name, timestr, environ=environ),
),
minute_bar_reader=BcolzMinuteBarReader(
minute_equity_path(name, timestr, environ=environ),
),
daily_bar_reader=BcolzDailyBarReader(
daily_equity_path(name, timestr, environ=environ),
),
adjustment_reader=SQLiteAdjustmentReader(
adjustment_db_path(name, timestr, environ=environ),
),
)
class BadClean(click.ClickException, ValueError):
"""Exception indicating that an invalid argument set was passed to
``clean``.
Parameters
----------
before, after, keep_last : any
The bad arguments to ``clean``.
See Also
--------
clean
"""
def __init__(self, before, after, keep_last):
super(BadClean, self).__init__(
'Cannot pass a combination of `before` and `after` with'
'`keep_last`. Got: before=%r, after=%r, keep_n=%r\n' % (
before,
after,
keep_last,
),
)
def __str__(self):
return self.message
@preprocess(
before=optionally(ensure_timestamp),
after=optionally(ensure_timestamp),
)
def clean(name, before=None, after=None, keep_last=None, environ=os.environ):
"""Clean up data that was created with ``ingest`` or
``$ python -m zipline ingest``
Parameters
----------
name : str
The name of the bundle to remove data for.
before : datetime, optional
Remove data ingested before this date.
This argument is mutually exclusive with: keep_last
after : datetime, optional
Remove data ingested after this date.
This argument is mutually exclusive with: keep_last
keep_last : int, optional
Remove all but the last ``keep_last`` ingestions.
This argument is mutually exclusive with:
before
after
Returns
-------
cleaned : set[str]
The names of the runs that were removed.
Raises
------
BadClean
Raised when ``before`` and or ``after`` are passed with ``keep_last``.
This is a subclass of ``ValueError``.
"""
try:
all_runs = sorted(
pd.Timestamp(f)
for f in os.listdir(pth.data_path([name], environ=environ))
if not pth.hidden(f)
)
except OSError as e:
if e.errno != errno.ENOENT:
raise
raise UnknownBundle(name)
if (before is not None or after is not None) and keep_last is not None:
raise BadClean(before, after, keep_last)
if keep_last is None:
def in_last_n(dt):
return False
else:
last_n_dts = set(all_runs[:keep_last])
def in_last_n(dt):
return dt in last_n_dts
def should_clean(name):
dt = pd.Timestamp(name)
return (
(
(before is not None and dt < before) or
(after is not None and dt > after)
) and
not in_last_n(dt)
)
cleaned = set()
for run in all_runs:
if should_clean(run):
shutil.rmdir(run)
cleaned.add(run)
return cleaned
zipline/data/bundles/quandl.py
+298
View File
@@ -0,0 +1,298 @@
"""
Module for building a complete daily dataset from Quandl's WIKI dataset.
"""
from itertools import count
from time import time, sleep
from logbook import Logger
import pandas as pd
from six.moves.urllib.parse import urlencode
from zipline.utils.cli import maybe_show_progress
from zipline.data import bundles
log = Logger(__name__)
seconds_per_call = (pd.Timedelta('10 minutes') / 2000).total_seconds()
def _fetch_raw_metadata(api_key, cache, retries, environ):
"""Generator that yields each page of data from the metadata endpoint
as a dataframe.
"""
for page_number in count(1):
key = 'metadata-page-%d' % page_number
try:
raw = cache[key]
except KeyError:
for _ in range(retries):
try:
raw = pd.read_csv(
format_metadata_url(api_key, page_number),
parse_dates=[
'oldest_available_date',
'newest_available_date',
],
usecols=[
'dataset_code',
'name',
'oldest_available_date',
'newest_available_date',
],
)
break
except ValueError:
# when we are past the last page we will get a value
# error because there will be no columns
raw = pd.DataFrame([])
break
except Exception:
pass
else:
raise ValueError(
'Failed to download metadata page %d after %d'
' attempts.' % (page_number, retries),
)
cache[key] = raw
if raw.empty:
# use the empty dataframe to signal completion
break
yield raw
def fetch_symbol_metadata_frame(api_key,
cache,
retries=5,
environ=None,
show_progress=False):
"""
Download Quandl symbol metadata.
Parameters
----------
api_key : str
The quandl api key to use. If this is None then no api key will be
sent.
cache : DataFrameCache
The cache to use for persisting the intermediate data.
retries : int, optional
The number of times to retry each request before failing.
environ : mapping[str -> str], optional
The environment to use to find the zipline home. By default this
is ``os.environ``.
show_progress : bool, optional
Show a progress bar for the download of this data.
Returns
-------
metadata_frame : pd.DataFrame
A dataframe with the following columns:
symbol: the asset's symbol
name: the full name of the asset
start_date: the first date of data for this asset
end_date: the last date of data for this asset
exchange: the exchange for the asset; this is always 'quandl'
The index of the dataframe will be used for symbol->sid mappings but
otherwise does not have specific meaning.
"""
raw_iter = _fetch_raw_metadata(api_key, cache, retries, environ)
def item_show_func(_, _it=iter(count())):
'Downloading page: %d' % next(_it)
with maybe_show_progress(raw_iter,
show_progress,
item_show_func=item_show_func,
label='Downloading WIKI metadata: ') as blocks:
data = pd.concat(blocks, ignore_index=True).rename(columns={
'dataset_code': 'symbol',
'name': 'asset_name',
'oldest_available_date': 'start_date',
'newest_available_date': 'end_date',
}).sort('symbol')
# cut out all the other stuff in the name column
# we need to escape the paren because it is actually splitting on a regex
data.asset_name = data.asset_name.str.split(r' \(', 1).str.get(0)
data['exchange'] = 'quandl'
return data
def format_metadata_url(api_key, page_number):
"""Build the query RL for the quandl WIKI metadata.
"""
query_params = [
('per_page', '100'),
('sort_by', 'id'),
('page', str(page_number)),
('database_code', 'WIKI'),
]
if api_key is not None:
query_params = [('api_key', api_key)] + query_params
return (
'https://www.quandl.com/api/v3/datasets.csv?' + urlencode(query_params)
)
def format_wiki_url(api_key, symbol, start_date, end_date):
"""
Build a query URL for a quandl WIKI dataset.
"""
query_params = [
('start_date', start_date.strftime('%Y-%m-%d')),
('end_date', end_date.strftime('%Y-%m-%d')),
('order', 'asc'),
]
if api_key is not None:
query_params = [('api_key', api_key)] + query_params
return (
"https://www.quandl.com/api/v3/datasets/WIKI/"
"{symbol}.csv?{query}".format(
symbol=symbol,
query=urlencode(query_params),
)
)
def fetch_single_equity(api_key,
symbol,
start_date,
end_date,
retries=5):
"""
Download data for a single equity.
"""
for _ in range(retries):
try:
return pd.read_csv(
format_wiki_url(api_key, symbol, start_date, end_date),
parse_dates=['Date'],
index_col='Date',
usecols=[
'Open',
'High',
'Low',
'Close',
'Volume',
'Date',
'Ex-Dividend',
'Split Ratio',
],
na_values=['NA'],
).rename(columns={
'Open': 'open',
'High': 'high',
'Low': 'low',
'Close': 'close',
'Volume': 'volume',
'Date': 'date',
'Ex-Dividend': 'ex_dividend',
'Split Ratio': 'split_ratio',
})
except Exception:
log.exception("Exception raised reading Quandl data. Retrying.")
else:
raise ValueError(
"Failed to download data for %r after %d attempts." % (
symbol, retries
)
)
def _update_splits(splits, asset_id, raw_data):
split_ratios = raw_data.split_ratio
df = pd.DataFrame({'ratio': split_ratios[split_ratios != 1]})
df.index.name = 'effective_date'
df.reset_index(inplace=True)
df['sid'] = asset_id
splits.append(df)
def _update_dividends(dividends, asset_id, raw_data):
divs = raw_data.ex_dividend
df = pd.DataFrame({'amount': divs[divs != 0]})
df.index.name = 'ex_date'
df.reset_index(inplace=True)
df['sid'] = asset_id
# we do not have this data in the WIKI dataset
df['record_date'] = df['declared_date'] = df['pay_date'] = pd.NaT
dividends.append(df)
def gen_symbol_data(api_key,
cache,
symbol_map,
calendar,
splits,
dividends,
retries):
start_date = calendar[0]
end_date = calendar[-1]
for asset_id, symbol in symbol_map.iteritems():
start_time = time()
try:
# see if we have this data cached.
raw_data = cache[symbol]
should_sleep = False
except KeyError:
# we need to fetch the data and then write it to our cache
raw_data = cache[symbol] = fetch_single_equity(
api_key,
symbol,
start_date=start_date,
end_date=end_date,
)
should_sleep = True
_update_splits(splits, asset_id, raw_data)
_update_dividends(dividends, asset_id, raw_data)
raw_data = raw_data.reindex(calendar, copy=False).fillna(0.0)
yield asset_id, raw_data
if should_sleep:
remaining = seconds_per_call - time() - start_time
if remaining > 0:
sleep(remaining)
@bundles.register('quandl')
def quandl_bundle(environ,
asset_db_writer,
minute_bar_writer, # unused
daily_bar_writer,
adjustment_writer,
calendar,
cache,
show_progress):
"""Build a zipline data bundle from the Quandl WIKI dataset.
"""
api_key = environ.get('QUANDL_API_KEY')
metadata = fetch_symbol_metadata_frame(
api_key,
cache=cache,
show_progress=show_progress,
)
symbol_map = metadata.symbol
# data we will collect in `gen_symbol_data`
splits = []
dividends = []
asset_db_writer.write(metadata)
daily_bar_writer.write(
gen_symbol_data(
api_key,
cache,
symbol_map,
calendar,
splits,
dividends,
environ.get('QUANDL_DOWNLOAD_ATTEMPTS', 5),
),
)
adjustment_writer.write(
splits=pd.concat(splits, ignore_index=True),
dividends=pd.concat(dividends, ignore_index=True),
)
zipline/data/bundles/yahoo.py
+164
View File
@@ -0,0 +1,164 @@
import os
import numpy as np
import pandas as pd
from pandas_datareader.data import DataReader
import requests
from zipline.utils.cli import maybe_show_progress
def _cachpath(symbol, type_):
return '-'.join((symbol.replace(os.path.sep, '_'), type_))
def yahoo_equities(symbols, start=None, end=None):
"""Create a data bundle ingest function from a set of symbols loaded from
yahoo.
Parameters
----------
symbols : iterable[str]
The ticker symbols to load data for.
start : datetime, optional
The start date to query for. By default this pulls the full history
for the calendar.
end : datetime, optional
The end date to query for. By default this pulls the full history
for the calendar.
Returns
-------
ingest : callable
The bundle ingest function for the given set of symbols.
Examples
--------
This code should be added to ~/.zipline/extension.py
.. code-block:: python
from zipline.data.bundles import yahoo_equities, register
symbols = (
'AAPL',
'IBM',
'MSFT',
)
register('my_bundle', yahoo_equities(symbols))
Notes
-----
The sids for each symbol will be the index into the symbols sequence.
"""
# strict this in memory so that we can reiterate over it
symbols = tuple(symbols)
def ingest(environ,
asset_db_writer,
minute_bar_writer, # unused
daily_bar_writer,
adjustment_writer,
calendar,
cache,
show_progress,
# pass these as defaults to make them 'nonlocal' in py2
start=start,
end=end):
if start is None:
start = calendar[0]
if end is None:
end = None
metadata = pd.DataFrame(np.empty(len(symbols), dtype=[
('start_date', 'datetime64[ns]'),
('end_date', 'datetime64[ns]'),
('symbol', 'object'),
]))
def _pricing_iter():
sid = 0
with maybe_show_progress(
symbols,
show_progress,
label='Downloading Yahoo pricing data: ') as it, \
requests.Session() as session:
for symbol in it:
path = _cachpath(symbol, 'ohlcv')
try:
df = cache[path]
except KeyError:
df = cache[path] = DataReader(
symbol,
'yahoo',
start,
end,
session=session,
).sort_index()
# the start date is the date of the first trade and
# the end date is the date of the last trade
metadata.iloc[sid] = df.index[0], df.index[-1], symbol
df.rename(
columns={
'Open': 'open',
'High': 'high',
'Low': 'low',
'Close': 'close',
'Volume': 'volume',
},
inplace=True,
)
yield sid, df
sid += 1
daily_bar_writer.write(_pricing_iter(), show_progress=True)
symbol_map = pd.Series(metadata.symbol.index, metadata.symbol)
asset_db_writer.write(equities=metadata)
adjustments = []
with maybe_show_progress(
symbols,
show_progress,
label='Downloading Yahoo adjustment data: ') as it, \
requests.Session() as session:
for symbol in it:
path = _cachpath(symbol, 'adjustment')
try:
df = cache[path]
except KeyError:
df = cache[path] = DataReader(
symbol,
'yahoo-actions',
start,
end,
session=session,
).sort_index()
df['sid'] = symbol_map[symbol]
adjustments.append(df)
adj_df = pd.concat(adjustments)
adj_df.index.name = 'date'
adj_df.reset_index(inplace=True)
splits = adj_df[adj_df.action == 'SPLIT']
splits = splits.rename(
columns={'value': 'ratio', 'date': 'effective_date'},
)
splits.drop('action', axis=1, inplace=True)
dividends = adj_df[adj_df.action == 'DIVIDEND']
dividends = dividends.rename(
columns={'value': 'amount', 'date': 'ex_date'},
)
dividends.drop('action', axis=1, inplace=True)
# we do not have this data in the yahoo dataset
dividends['record_date'] = pd.NaT
dividends['declared_date'] = pd.NaT
dividends['pay_date'] = pd.NaT
adjustment_writer.write(splits=splits, dividends=dividends)
return ingest
zipline/data/data_portal.py
+48 -16
View File
@@ -163,8 +163,12 @@ class DailyHistoryAggregator(object):
else:
after_last = pd.Timestamp(
last_visited_dt + self._one_min, tz='UTC')
window = self._minute_reader.unadjusted_window(
['open'], after_last, dt, [asset])[0]
window = self._minute_reader.load_raw_arrays(
['open'],
after_last,
dt,
[asset],
)[0]
nonnan = window[~pd.isnull(window)]
if len(nonnan):
val = nonnan[0]
@@ -174,8 +178,12 @@ class DailyHistoryAggregator(object):
opens.append(val)
continue
except KeyError:
window = self._minute_reader.unadjusted_window(
['open'], market_open, dt, [asset])[0]
window = self._minute_reader.load_raw_arrays(
['open'],
market_open,
dt,
[asset],
)[0]
nonnan = window[~pd.isnull(window)]
if len(nonnan):
val = nonnan[0]
@@ -232,15 +240,23 @@ class DailyHistoryAggregator(object):
else:
after_last = pd.Timestamp(
last_visited_dt + self._one_min, tz='UTC')
window = self._minute_reader.unadjusted_window(
['high'], after_last, dt, [asset])
window = self._minute_reader.load_raw_arrays(
['high'],
after_last,
dt,
[asset],
)[0].T
val = max(last_max, np.nanmax(window))
entries[asset] = (dt_value, val)
highs.append(val)
continue
except KeyError:
window = self._minute_reader.unadjusted_window(
['high'], market_open, dt, [asset])
window = self._minute_reader.load_raw_arrays(
['high'],
market_open,
dt,
[asset],
)[0].T
val = np.nanmax(window)
entries[asset] = (dt_value, val)
highs.append(val)
@@ -288,8 +304,12 @@ class DailyHistoryAggregator(object):
else:
after_last = pd.Timestamp(
last_visited_dt + self._one_min, tz='UTC')
window = self._minute_reader.unadjusted_window(
['low'], after_last, dt, [asset])
window = self._minute_reader.load_raw_arrays(
['low'],
after_last,
dt,
[asset],
)[0].T
window_min = np.nanmin(window)
if pd.isnull(window_min):
val = last_min
@@ -299,8 +319,12 @@ class DailyHistoryAggregator(object):
lows.append(val)
continue
except KeyError:
window = self._minute_reader.unadjusted_window(
['low'], market_open, dt, [asset])
window = self._minute_reader.load_raw_arrays(
['low'],
market_open,
dt,
[asset],
)[0].T
val = np.nanmin(window)
entries[asset] = (dt_value, val)
lows.append(val)
@@ -410,15 +434,23 @@ class DailyHistoryAggregator(object):
else:
after_last = pd.Timestamp(
last_visited_dt + self._one_min, tz='UTC')
window = self._minute_reader.unadjusted_window(
['volume'], after_last, dt, [asset])
window = self._minute_reader.load_raw_arrays(
['volume'],
after_last,
dt,
[asset],
)[0]
val = np.nansum(window) + last_total
entries[asset] = (dt_value, val)
volumes.append(val)
continue
except KeyError:
window = self._minute_reader.unadjusted_window(
['volume'], market_open, dt, [asset])
window = self._minute_reader.load_raw_arrays(
['volume'],
market_open,
dt,
[asset],
)[0]
val = np.nansum(window)
entries[asset] = (dt_value, val)
volumes.append(val)
zipline/data/loader.py
+8 -6
View File
@@ -16,22 +16,20 @@ import os
from collections import OrderedDict
import logbook
import pandas as pd
from pandas.io.data import DataReader
import pytz
from six import iteritems
from six.moves.urllib_error import HTTPError
from . benchmarks import get_benchmark_returns
from .benchmarks import get_benchmark_returns
from . import treasuries, treasuries_can
from .paths import (
from ..utils.paths import (
cache_root,
data_root,
)
from zipline.utils.tradingcalendar import (
from ..utils.deprecate import deprecated
from ..utils.tradingcalendar import (
trading_day as trading_day_nyse,
trading_days as trading_days_nyse,
)
@@ -413,6 +411,10 @@ def load_from_yahoo(indexes=None,
return df
@deprecated(
'load_bars_from_yahoo is deprecated, please register a'
' yahoo_equities data bundle instead',
)
def load_bars_from_yahoo(indexes=None,
stocks=None,
start=None,
zipline/data/minute_bars.py
+120 -84
View File
@@ -29,6 +29,7 @@ from zipline.data._minute_bar_internal import (
)
from zipline.gens.sim_engine import NANOS_IN_MINUTE
from zipline.utils.cli import maybe_show_progress
from zipline.utils.memoize import lazyval
US_EQUITIES_MINUTES_PER_DAY = 390
@@ -91,6 +92,22 @@ def _sid_subdir_path(sid):
class BcolzMinuteBarMetadata(object):
"""
Parameters
----------
first_trading_day : datetime-like
UTC midnight of the first day available in the dataset.
minute_index : pd.DatetimeIndex
The minutes which act as an index into the corresponding values
written into each sid's ctable.
market_opens : pd.DatetimeIndex
The market opens for each day in the data set. (Not yet required.)
market_closes : pd.DatetimeIndex
The market closes for each day in the data set. (Not yet required.)
ohlc_ratio : int
The factor by which the pricing data is multiplied so that the
float data can be stored as an integer.
"""
METADATA_FILENAME = 'metadata.json'
@@ -122,22 +139,6 @@ class BcolzMinuteBarMetadata(object):
market_opens,
market_closes,
ohlc_ratio):
"""
Parameters:
-----------
first_trading_day : datetime-like
UTC midnight of the first day available in the dataset.
minute_index : pd.DatetimeIndex
The minutes which act as an index into the corresponding values
written into each sid's ctable.
market_opens : pd.DatetimeIndex
The market opens for each day in the data set. (Not yet required.)
market_closes : pd.DatetimeIndex
The market closes for each day in the data set. (Not yet required.)
ohlc_ratio : int
The factor by which the pricing data is multiplied so that the
float data can be stored as an integer.
"""
self.first_trading_day = first_trading_day
self.market_opens = market_opens
self.market_closes = market_closes
@@ -176,6 +177,55 @@ class BcolzMinuteBarWriter(object):
"""
Class capable of writing minute OHLCV data to disk into bcolz format.
Parameters
----------
first_trading_day : datetime
The first trading day in the data set.
rootdir : string
Path to the root directory into which to write the metadata and
bcolz subdirectories.
market_opens : pd.Series
The market opens used as a starting point for each periodic span of
minutes in the index.
The index of the series is expected to be a DatetimeIndex of the
UTC midnight of each trading day.
The values are datetime64-like UTC market opens for each day in the
index.
market_closes : pd.Series
The market closes that correspond with the market opens,
The index of the series is expected to be a DatetimeIndex of the
UTC midnight of each trading day.
The values are datetime64-like UTC market opens for each day in the
index.
The closes are written so that the reader can filter out non-market
minutes even though the tail end of early closes are written in
the data arrays to keep a regular shape.
minutes_per_day : int
The number of minutes per each period. Defaults to 390, the mode
of minutes in NYSE trading days.
ohlc_ratio : int, optional
The ratio by which to multiply the pricing data to convert the
floats from floats to an integer to fit within the np.uint32.
The default is 1000 to support pricing data which comes in to the
thousands place.
expectedlen : int, optional
The expected length of the dataset, used when creating the initial
bcolz ctable.
If the expectedlen is not used, the chunksize and corresponding
compression ratios are not ideal.
Defaults to supporting 15 years of NYSE equity market data.
see: http://bcolz.blosc.org/opt-tips.html#informing-about-the-length-of-your-carrays # noqa
Notes
-----
Writes a bcolz directory for each individual sid, all contained within
a root directory which also contains metadata about the entire dataset.
@@ -214,6 +264,10 @@ class BcolzMinuteBarWriter(object):
The datetimes which correspond to each position are written in the metadata
as integer nanoseconds since the epoch into the `minute_index` key.
See Also
--------
zipline.data.minute_bars.BcolzMinuteBarReader
"""
COL_NAMES = ('open', 'high', 'low', 'close', 'volume')
@@ -225,61 +279,6 @@ class BcolzMinuteBarWriter(object):
minutes_per_day,
ohlc_ratio=OHLC_RATIO,
expectedlen=DEFAULT_EXPECTEDLEN):
"""
Parameters:
-----------
first_trading_day : datetime-like
The first trading day in the data set.
rootdir : string
Path to the root directory into which to write the metadata and
bcolz subdirectories.
market_opens : pd.Series
The market opens used as a starting point for each periodic span of
minutes in the index.
The index of the series is expected to be a DatetimeIndex of the
UTC midnight of each trading day.
The values are datetime64-like UTC market opens for each day in the
index.
market_closes : pd.Series
The market closes that correspond with the market opens,
The index of the series is expected to be a DatetimeIndex of the
UTC midnight of each trading day.
The values are datetime64-like UTC market opens for each day in the
index.
The closes are written so that the reader can filter out non-market
minutes even though the tail end of early closes are written in
the data arrays to keep a regular shape.
minutes_per_day : int
The number of minutes per each period. Defaults to 390, the mode
of minutes in NYSE trading days.
ohlc_ratio : int
The ratio by which to multiply the pricing data to convert the
floats from floats to an integer to fit within the np.uint32.
The default is 1000 to support pricing data which comes in to the
thousands place.
expectedlen : int
The expected length of the dataset, used when creating the initial
bcolz ctable.
If the expectedlen is not used, the chunksize and corresponding
compression ratios are not ideal.
Defaults to supporting 15 years of NYSE equity market data.
see: http://bcolz.blosc.org/opt-tips.html#informing-about-the-length-of-your-carrays # noqa
"""
self._rootdir = rootdir
self._first_trading_day = first_trading_day
self._market_opens = market_opens[
@@ -441,7 +440,38 @@ class BcolzMinuteBarWriter(object):
assert new_last_date == date, "new_last_date={0} != date={1}".format(
new_last_date, date)
def write(self, sid, df):
def write(self, data, show_progress=False):
"""Write a stream of minute data.
Parameters
----------
data : iterable[(int, pd.DataFrame)]
The data to write. Each element should be a tuple of sid, data
where data has the following format:
columns : ('open', 'high', 'low', 'close', 'volume')
open : float64
high : float64
low : float64
close : float64
volume : float64|int64
index : DatetimeIndex of market minutes.
A given sid may appear more than once in ``data``; however,
the dates must be strictly increasing.
show_progress : bool, optional
Whether or not to show a progress bar while writing.
"""
ctx = maybe_show_progress(
data,
show_progress=show_progress,
item_show_func=lambda e: e if e is None else str(e[0]),
label="Merging minute equity files:",
)
write_sid = self.write_sid
with ctx as it:
for e in it:
write_sid(*e)
def write_sid(self, sid, df):
"""
Write the OHLCV data for the given sid.
If there is no bcolz ctable yet created for the sid, create it.
@@ -585,17 +615,21 @@ class BcolzMinuteBarWriter(object):
class BcolzMinuteBarReader(object):
"""
Reader for data written by BcolzMinuteBarWriter
Parameters:
-----------
rootdir : string
The root directory containing the metadata and asset bcolz
directories.
See Also
--------
zipline.data.minute_bars.BcolzMinuteBarWriter
"""
def __init__(self, rootdir):
"""
Reader for data written by BcolzMinuteBarWriter
Parameters:
-----------
rootdir : string
The root directory containing the metadata and asset bcolz
directories.
"""
self._rootdir = rootdir
metadata = self._get_metadata()
@@ -826,7 +860,7 @@ class BcolzMinuteBarReader(object):
US_EQUITIES_MINUTES_PER_DAY,
)
def unadjusted_window(self, fields, start_dt, end_dt, sids):
def load_raw_arrays(self, fields, start_dt, end_dt, sids):
"""
Parameters
----------
@@ -843,7 +877,7 @@ class BcolzMinuteBarReader(object):
-------
list of np.ndarray
A list with an entry per field of ndarrays with shape
(sids, minutes in range) with a dtype of float64, containing the
(minutes in range, sids) with a dtype of float64, containing the
values for the respective field over start and end dt range.
"""
start_idx = self._find_position_of_minute(start_dt)
@@ -860,7 +894,7 @@ class BcolzMinuteBarReader(object):
length = excl_stop - excl_start + 1
num_minutes -= length
shape = (len(sids), num_minutes)
shape = num_minutes, len(sids)
for field in fields:
if field != 'volume':
@@ -877,7 +911,9 @@ class BcolzMinuteBarReader(object):
excl_start - start_idx:excl_stop - start_idx + 1]
values = np.delete(values, excl_slice)
where = values != 0
out[i, where] = values[where]
# first slice down to len(where) because we might not have
# written data for all the minutes requested
out[:len(where), i][where] = values[where]
if field != 'volume':
out *= self._ohlc_inverse
results.append(out)
zipline/data/paths.py
-91
View File
@@ -1,91 +0,0 @@
"""
Canonical path locations for zipline data.
Paths are rooted at $ZIPLINE_ROOT if that environment variable is set.
Otherwise default to expanduser(~/.zipline)
"""
import os
from os.path import (
expanduser,
join,
)
def zipline_root(environ=None):
"""
Get the root directory for all zipline-managed files.
For testing purposes, this accepts a dictionary to interpret as the os
environment.
Parameters
----------
environ : dict, optional
A dict to interpret as the os environment.
Returns
-------
root : string
Path to the zipline root dir.
"""
if environ is None:
environ = os.environ.copy()
root = environ.get('ZIPLINE_ROOT', None)
if root is None:
root = expanduser('~/.zipline')
return root
def zipline_root_path(path, environ=None):
"""
Get a path relative to the zipline root.
Parameters
----------
path : str
The requested path.
environ : dict, optional
An environment dict to forward to zipline_root.
Returns
-------
newpath : str
The requested path joined with the zipline root.
"""
return join(zipline_root(environ=environ), path)
def data_root(environ=None):
"""
The root directory for zipline data files.
Parameters
----------
environ : dict, optional
An environment dict to forward to zipline_root.
Returns
-------
data_root : str
The zipline data root.
"""
return zipline_root_path('data', environ=environ)
def cache_root(environ=None):
"""
The root directory for zipline cache files.
Parameters
----------
environ : dict, optional
An environment dict to forward to zipline_root.
Returns
-------
cache_root : str
The zipline cache root.
"""
return zipline_root_path('cache', environ=environ)
zipline/data/us_equity_loader.py
+11 -5
View File
@@ -24,7 +24,6 @@ from pandas.tslib import normalize_date
from six import with_metaclass
from zipline.pipeline.data.equity_pricing import USEquityPricing
from zipline.lib._float64window import AdjustedArrayWindow as Float64Window
from zipline.lib.adjustment import Float64Multiply
from zipline.utils.cache import ExpiringCache
@@ -300,9 +299,12 @@ class USEquityDailyHistoryLoader(USEquityHistoryLoader):
return self._reader._calendar
def _array(self, dts, assets, field):
col = getattr(USEquityPricing, field)
return self._reader.load_raw_arrays(
[col], dts[0], dts[-1], assets)[0]
[field],
dts[0],
dts[-1],
assets,
)[0]
class USEquityMinuteHistoryLoader(USEquityHistoryLoader):
@@ -318,5 +320,9 @@ class USEquityMinuteHistoryLoader(USEquityHistoryLoader):
end=self._reader.last_available_dt)]
def _array(self, dts, assets, field):
return self._reader.unadjusted_window(
[field], dts[0], dts[-1], assets)[0].T
return self._reader.load_raw_arrays(
[field],
dts[0],
dts[-1],
assets,
)[0]
zipline/data/us_equity_pricing.py
+94 -86
View File
@@ -22,7 +22,6 @@ import warnings
from bcolz import (
carray,
ctable,
open as open_ctable,
)
from collections import namedtuple
import logbook
@@ -193,7 +192,7 @@ class BcolzDailyBarWriter(object):
See Also
--------
BcolzDailyBarReader : Consumer of the data written by this class.
zipline.data.us_equity_pricing.BcolzDailyBarReader
"""
_csv_dtypes = {
'open': float64,
@@ -209,7 +208,7 @@ class BcolzDailyBarWriter(object):
@property
def progress_bar_message(self):
return "Merging asset files:"
return "Merging daily equity files:"
def progress_bar_item_show_func(self, value):
return value if value is None else str(value[0])
@@ -229,9 +228,9 @@ class BcolzDailyBarWriter(object):
The assets that should be in ``data``. If this is provided
we will check ``data`` against the assets and provide better
progress information.
show_progress : bool
show_progress : bool, optional
Whether or not to show a progress bar while writing.
invalid_data_behavior : {'warn', 'raise', 'ignore'}
invalid_data_behavior : {'warn', 'raise', 'ignore'}, optional
What to do when data is encountered that is outside the range of
a uint32.
@@ -365,6 +364,7 @@ class BcolzDailyBarWriter(object):
full_table.attrs['last_row'] = last_row
full_table.attrs['calendar_offset'] = calendar_offset
full_table.attrs['calendar'] = calendar.asi8.tolist()
full_table.flush()
return full_table
@@ -451,11 +451,13 @@ class BcolzDailyBarReader(DailyBarReader):
We use calendar_offset and calendar to orient loaded blocks within a
range of queried dates.
"""
@preprocess(table=coerce_string(open_ctable, mode='r'))
def __init__(self, table, read_all_threshold=3000):
self._table = table
See Also
--------
zipline.data.us_equity_pricing.BcolzDailyBarWriter
"""
def __init__(self, table, read_all_threshold=3000):
self._maybe_table_rootdir = table
# Cache of fully read np.array for the carrays in the daily bar table.
# raw_array does not use the same cache, but it could.
# Need to test keeping the entire array in memory for the course of a
@@ -464,6 +466,13 @@ class BcolzDailyBarReader(DailyBarReader):
self.PRICE_ADJUSTMENT_FACTOR = 0.001
self._read_all_threshold = read_all_threshold
@lazyval
def _table(self):
maybe_table_rootdir = self._maybe_table_rootdir
if isinstance(maybe_table_rootdir, ctable):
return maybe_table_rootdir
return ctable(rootdir=maybe_table_rootdir, mode='r')
@lazyval
def _calendar(self):
return DatetimeIndex(self._table.attrs['calendar'], tz='UTC')
@@ -565,7 +574,7 @@ class BcolzDailyBarReader(DailyBarReader):
return _read_bcolz_data(
self._table,
(end_idx - start_idx + 1, len(assets)),
[column.name for column in columns],
list(columns),
first_rows,
last_rows,
offsets,
@@ -717,12 +726,12 @@ class PanelDailyBarReader(DailyBarReader):
return self._calendar[-1]
def load_raw_arrays(self, columns, start_date, end_date, assets):
col_names = [col.name for col in columns]
columns = list(columns)
cal = self._calendar
index = cal[cal.slice_indexer(start_date, end_date)]
shape = (len(index), len(assets))
results = []
for col in col_names:
for col in columns:
outbuf = zeros(shape=shape)
for i, asset in enumerate(assets):
data = self.panel.loc[asset, start_date:end_date, col]
@@ -792,7 +801,7 @@ class SQLiteAdjustmentWriter(object):
See Also
--------
SQLiteAdjustmentReader
zipline.data.us_equity_pricing.SQLiteAdjustmentReader
"""
def __init__(self,
@@ -862,6 +871,11 @@ class SQLiteAdjustmentWriter(object):
SQLITE_ADJUSTMENT_TABLENAMES,
)
)
if not (frame is None or frame.empty):
frame = frame.copy()
frame['effective_date'] = frame['effective_date'].values.astype(
'datetime64[s]',
).astype('int64')
return self._write(
tablename,
SQLITE_ADJUSTMENT_COLUMN_DTYPES,
@@ -1016,82 +1030,72 @@ class SQLiteAdjustmentWriter(object):
Parameters
----------
splits : pandas.DataFrame
Dataframe containing split data.
mergers : pandas.DataFrame
DataFrame containing merger data.
dividends : pandas.DataFrame
DataFrame containing dividend data.
splits : pandas.DataFrame, optional
Dataframe containing split data. The format of this dataframe is:
effective_date : int
The date, represented as seconds since Unix epoch, on which
the adjustment should be applied.
ratio : float
A value to apply to all data earlier than the effective date.
For open, high, low, and close those values are multiplied by
the ratio. Volume is divided by this value.
sid : int
The asset id associated with this adjustment.
mergers : pandas.DataFrame, optional
DataFrame containing merger data. The format of this dataframe is:
effective_date : int
The date, represented as seconds since Unix epoch, on which
the adjustment should be applied.
ratio : float
A value to apply to all data earlier than the effective date.
For open, high, low, and close those values are multiplied by
the ratio. Volume is unaffected.
sid : int
The asset id associated with this adjustment.
dividends : pandas.DataFrame, optional
DataFrame containing dividend data. The format of the dataframe is:
sid : int
The asset id associated with this adjustment.
ex_date : datetime64
The date on which an equity must be held to be eligible to
receive payment.
declared_date : datetime64
The date on which the dividend is announced to the public.
pay_date : datetime64
The date on which the dividend is distributed.
record_date : datetime64
The date on which the stock ownership is checked to determine
distribution of dividends.
amount : float
The cash amount paid for each share.
Notes
-----
DataFrame input (`splits`, `mergers`) should all have
the following columns:
effective_date : int
The date, represented as seconds since Unix epoch, on which the
adjustment should be applied.
ratio : float
A value to apply to all data earlier than the effective date.
sid : int
The asset id associated with this adjustment.
The ratio column is interpreted as follows:
- For all adjustment types, multiply price fields ('open', 'high',
'low', and 'close') by the ratio.
- For **splits only**, **divide** volume by the adjustment ratio.
DataFrame input, 'dividends' should have the following columns:
sid : int
The asset id associated with this adjustment.
ex_date : datetime64
The date on which an equity must be held to be eligible to receive
payment.
declared_date : datetime64
The date on which the dividend is announced to the public.
pay_date : datetime64
The date on which the dividend is distributed.
record_date : datetime64
The date on which the stock ownership is checked to determine
distribution of dividends.
amount : float
The cash amount paid for each share.
Dividend ratios are calculated as
1.0 - (dividend_value / "close on day prior to dividend ex_date").
DataFrame input, 'stock_dividends' should have the following columns:
sid : int
The asset id associated with this adjustment.
ex_date : datetime64
The date on which an equity must be held to be eligible to receive
payment.
declared_date : datetime64
The date on which the dividend is announced to the public.
pay_date : datetime64
The date on which the dividend is distributed.
record_date : datetime64
The date on which the stock ownership is checked to determine
distribution of dividends.
payment_sid : int
The asset id of the shares that should be paid instead of cash.
ratio: float
The ratio of currently held shares in the held sid that should
be paid with new shares of the payment_sid.
stock_dividends is optional.
Returns
-------
None
Dividend ratios are calculated as:
``1.0 - (dividend_value / "close on day prior to ex_date")``
stock_dividends : pandas.DataFrame, optional
DataFrame containing stock dividend data. The format of the
dataframe is:
sid : int
The asset id associated with this adjustment.
ex_date : datetime64
The date on which an equity must be held to be eligible to
receive payment.
declared_date : datetime64
The date on which the dividend is announced to the public.
pay_date : datetime64
The date on which the dividend is distributed.
record_date : datetime64
The date on which the stock ownership is checked to determine
distribution of dividends.
payment_sid : int
The asset id of the shares that should be paid instead of
cash.
ratio : float
The ratio of currently held shares in the held sid that
should be paid with new shares of the payment_sid.
See Also
--------
SQLiteAdjustmentReader : Consumer for the data written by this class
zipline.data.us_equity_pricing.SQLiteAdjustmentReader
"""
self.write_frame('splits', splits)
self.write_frame('mergers', mergers)
@@ -1168,6 +1172,10 @@ class SQLiteAdjustmentReader(object):
----------
conn : str or sqlite3.Connection
Connection from which to load data.
See Also
--------
zipline.data.us_equity_pricing.SQLiteAdjustmentWriter
"""
@preprocess(conn=coerce_string(sqlite3.connect))
@@ -1177,7 +1185,7 @@ class SQLiteAdjustmentReader(object):
def load_adjustments(self, columns, dates, assets):
return load_adjustments_from_sqlite(
self.conn,
[column.name for column in columns],
list(columns),
dates,
assets,
)
zipline/examples/__init__.py
+17
View File
@@ -0,0 +1,17 @@
from glob import glob
from importlib import import_module
import os
for f in os.listdir(os.path.dirname(__file__)):
if not f.endswith('.py') or f == '__init__.py':
continue
modname = f[:-len('.py')]
globals()[modname] = import_module('.' + modname, package=__name__)
del f
try:
del modname
except NameError:
pass
del os, import_module, glob
zipline/examples/buy_and_hold.py
+8 -17
View File
@@ -13,10 +13,7 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import pandas as pd
from zipline import TradingAlgorithm
from zipline.api import order, symbol
from zipline.data.loader import load_bars_from_yahoo
stocks = ['AAPL', 'MSFT']
@@ -33,18 +30,12 @@ def handle_data(context, data):
context.has_ordered = True
if __name__ == '__main__':
def _test_args():
"""Extra arguments to use when zipline's automated tests run this example.
"""
import pandas as pd
# creating time interval
start = pd.Timestamp('2008-01-01', tz='UTC')
end = pd.Timestamp('2013-01-01', tz='UTC')
# loading the data
input_data = load_bars_from_yahoo(
stocks=stocks,
start=start,
end=end,
)
algo = TradingAlgorithm(initialize=initialize, handle_data=handle_data)
results = algo.run(input_data)
return {
'start': pd.Timestamp('2008', tz='utc'),
'end': pd.Timestamp('2013', tz='utc'),
}
zipline/examples/buyapple.conf
-3
View File
@@ -1,3 +0,0 @@
[Defaults]
algofile=buyapple.py
symbols=AAPL
zipline/examples/buyapple.py
Executable → Regular
+9 -21
View File
@@ -23,7 +23,7 @@ def initialize(context):
def handle_data(context, data):
order(symbol('AAPL'), 10)
record(AAPL=data.current(symbol('AAPL'), "price"))
record(AAPL=data.current(symbol('AAPL'), 'price'))
# Note: this function can be removed if running
@@ -43,24 +43,12 @@ def analyze(context=None, results=None):
plt.show()
# Note: this if-block should be removed if running
# this algorithm on quantopian.com
if __name__ == '__main__':
from datetime import datetime
import pytz
from zipline.algorithm import TradingAlgorithm
from zipline.utils.factory import load_from_yahoo
def _test_args():
"""Extra arguments to use when zipline's automated tests run this example.
"""
import pandas as pd
# Set the simulation start and end dates
start = datetime(2014, 1, 1, 0, 0, 0, 0, pytz.utc)
end = datetime(2014, 11, 1, 0, 0, 0, 0, pytz.utc)
# Load price data from yahoo.
data = load_from_yahoo(stocks=['AAPL'], indexes={}, start=start,
end=end)
# Create and run the algorithm.
algo = TradingAlgorithm(initialize=initialize, handle_data=handle_data)
results = algo.run(data)
analyze(results=results)
return {
'start': pd.Timestamp('2014-01-01', tz='utc'),
'end': pd.Timestamp('2014-11-01', tz='utc'),
}
zipline/examples/dual_ema_talib.py
Executable → Regular
+8 -21
View File
@@ -98,25 +98,12 @@ def analyze(context=None, results=None):
plt.show()
# Note: this if-block should be removed if running
# this algorithm on quantopian.com
if __name__ == '__main__':
from datetime import datetime
import pytz
from zipline.algorithm import TradingAlgorithm
from zipline.utils.factory import load_from_yahoo
def _test_args():
"""Extra arguments to use when zipline's automated tests run this example.
"""
import pandas as pd
# Set the simulation start and end dates.
start = datetime(2014, 1, 1, 0, 0, 0, 0, pytz.utc)
end = datetime(2014, 11, 1, 0, 0, 0, 0, pytz.utc)
# Load price data from yahoo.
data = load_from_yahoo(stocks=['AAPL'], indexes={}, start=start,
end=end)
# Create and run the algorithm.
algo = TradingAlgorithm(initialize=initialize, handle_data=handle_data)
results = algo.run(data).dropna()
# Plot the portfolio and asset data.
analyze(results=results)
return {
'start': pd.Timestamp('2014-01-01', tz='utc'),
'end': pd.Timestamp('2014-11-01', tz='utc'),
}
zipline/examples/dual_moving_average.conf
-5
View File
@@ -1,5 +0,0 @@
[Defaults]
algofile=dual_moving_average.py
symbols=AAPL
start=2000-1-1
end=2014-1-1
zipline/examples/dual_moving_average.py
Executable → Regular
+8 -25
View File
@@ -97,29 +97,12 @@ def analyze(context=None, results=None):
plt.show()
# Note: this if-block should be removed if running
# this algorithm on quantopian.com
if __name__ == '__main__':
from datetime import datetime
import pytz
from zipline.algorithm import TradingAlgorithm
from zipline.utils.factory import load_bars_from_yahoo
def _test_args():
"""Extra arguments to use when zipline's automated tests run this example.
"""
import pandas as pd
# Set the simulation start and end dates.
start = datetime(2011, 1, 1, 0, 0, 0, 0, pytz.utc)
end = datetime(2013, 1, 1, 0, 0, 0, 0, pytz.utc)
# Load price data from yahoo.
data = load_bars_from_yahoo(
stocks=['AAPL'],
indexes={},
start=start,
end=end,
)
# Create and run the algorithm.
algo = TradingAlgorithm(initialize=initialize, handle_data=handle_data)
results = algo.run(data)
# Plot the portfolio and asset data.
analyze(results=results)
return {
'start': pd.Timestamp('2011', tz='utc'),
'end': pd.Timestamp('2013', tz='utc'),
}
zipline/examples/olmar.py
+8 -20
View File
@@ -1,11 +1,7 @@
import sys
import logbook
import numpy as np
from datetime import datetime
import pytz
from zipline.algorithm import TradingAlgorithm
from zipline.utils.factory import load_from_yahoo
from zipline.finance import commission
zipline_logging = logbook.NestedSetup([
@@ -161,20 +157,12 @@ def analyze(context=None, results=None):
plt.show()
# Note: this if-block should be removed if running
# this algorithm on quantopian.com
if __name__ == '__main__':
# Set the simulation start and end dates.
start = datetime(2004, 1, 1, 0, 0, 0, 0, pytz.utc)
end = datetime(2008, 1, 1, 0, 0, 0, 0, pytz.utc)
def _test_args():
"""Extra arguments to use when zipline's automated tests run this example.
"""
import pandas as pd
# Load price data from yahoo.
data = load_from_yahoo(stocks=STOCKS, indexes={}, start=start, end=end)
data = data.dropna()
# Create and run the algorithm.
olmar = TradingAlgorithm(handle_data=handle_data, initialize=initialize)
results = olmar.run(data)
# Plot the portfolio data.
analyze(results=results)
return {
'start': pd.Timestamp('2004', tz='utc'),
'end': pd.Timestamp('2008', tz='utc'),
}
zipline/pipeline/loaders/equity_pricing_loader.py
+3 -3
View File
@@ -69,15 +69,15 @@ class USEquityPricingLoader(PipelineLoader):
start_date, end_date = _shift_dates(
self._calendar, dates[0], dates[-1], shift=1,
)
colnames = [c.name for c in columns]
raw_arrays = self.raw_price_loader.load_raw_arrays(
columns,
colnames,
start_date,
end_date,
assets,
)
adjustments = self.adjustments_loader.load_adjustments(
columns,
colnames,
dates,
assets,
)
zipline/pipeline/loaders/synthetic.py
+10 -10
View File
@@ -213,7 +213,7 @@ def asset_end(asset_info, asset):
return ret
def make_daily_bar_data(asset_info, calendar):
def make_bar_data(asset_info, calendar):
"""
For a given asset/date/column combination, we generate a corresponding raw
@@ -249,7 +249,7 @@ def make_daily_bar_data(asset_info, calendar):
assert (
# Using .value here to avoid having to care about UTC-aware dates.
PSEUDO_EPOCH.value <
calendar.min().value <=
calendar.normalize().min().value <=
asset_info['start_date'].min().value
), "calendar.min(): %s\nasset_info['start_date'].min(): %s" % (
calendar.min(),
@@ -266,13 +266,13 @@ def make_daily_bar_data(asset_info, calendar):
"""
# Get the dates for which this asset existed according to our asset
# info.
dates = calendar[calendar.slice_indexer(
datetimes = calendar[calendar.slice_indexer(
asset_start(asset_info, asset_id),
asset_end(asset_info, asset_id),
)]
data = full(
(len(dates), len(US_EQUITY_PRICING_BCOLZ_COLUMNS)),
(len(datetimes), len(US_EQUITY_PRICING_BCOLZ_COLUMNS)),
asset_id * 100 * 1000,
dtype=uint32,
)
@@ -281,15 +281,15 @@ def make_daily_bar_data(asset_info, calendar):
data[:, :5] += arange(5, dtype=uint32) * 1000
# Add days since Jan 1 2001 for OHLCV columns.
data[:, :5] += (dates - PSEUDO_EPOCH).days[:, None].astype(uint32)
data[:, :5] += (datetimes - PSEUDO_EPOCH).days[:, None].astype(uint32)
frame = DataFrame(
data,
index=dates,
index=datetimes,
columns=US_EQUITY_PRICING_BCOLZ_COLUMNS,
)
frame['day'] = nanos_to_seconds(dates.asi8)
frame['day'] = nanos_to_seconds(datetimes.asi8)
frame['id'] = asset_id
return frame
@@ -297,7 +297,7 @@ def make_daily_bar_data(asset_info, calendar):
yield asset, _raw_data_for_asset(asset)
def expected_daily_bar_value(asset_id, date, colname):
def expected_bar_value(asset_id, date, colname):
"""
Check that the raw value for an asset/date/column triple is as
expected.
@@ -310,7 +310,7 @@ def expected_daily_bar_value(asset_id, date, colname):
return from_asset + from_colname + from_date
def expected_daily_bar_values_2d(dates, asset_info, colname):
def expected_bar_values_2d(dates, asset_info, colname):
"""
Return an 2D array containing cls.expected_value(asset_id, date,
colname) for each date/asset pair in the inputs.
@@ -336,7 +336,7 @@ def expected_daily_bar_values_2d(dates, asset_info, colname):
# date.
if not (start <= date <= end):
continue
data[i, j] = expected_daily_bar_value(asset, date, colname)
data[i, j] = expected_bar_value(asset, date, colname)
return data
zipline/testing/__init__.py
+1
View File
@@ -29,6 +29,7 @@ from .core import ( # noqa
parameter_space,
parameter_space,
patch_os_environment,
patch_read_csv,
permute_rows,
powerset,
product_upper_triangle,
zipline/testing/core.py
+41 -11
View File
@@ -349,7 +349,7 @@ def make_trade_data_for_asset_info(dates,
)
if writer:
writer.write(sid, df)
writer.write_sid(sid, df)
trade_data[sid] = df
@@ -424,8 +424,8 @@ def write_minute_data(env, tempdir, minutes, sids):
def create_minute_bar_data(minutes, sids):
length = len(minutes)
return {
sid: pd.DataFrame(
for sid_idx, sid in enumerate(sids):
yield sid, pd.DataFrame(
{
'open': np.arange(length) + 10 + sid_idx,
'high': np.arange(length) + 15 + sid_idx,
@@ -435,8 +435,6 @@ def create_minute_bar_data(minutes, sids):
},
index=minutes,
)
for sid_idx, sid in enumerate(sids)
}
def create_daily_bar_data(trading_days, sids):
@@ -492,20 +490,17 @@ def create_data_portal(env, tempdir, sim_params, sids, adjustment_reader=None):
)
def write_bcolz_minute_data(env, days, path, df_dict):
def write_bcolz_minute_data(env, days, path, data):
market_opens = env.open_and_closes.market_open.loc[days]
market_closes = env.open_and_closes.market_close.loc[days]
writer = BcolzMinuteBarWriter(
BcolzMinuteBarWriter(
days[0],
path,
market_opens,
market_closes,
US_EQUITIES_MINUTES_PER_DAY
)
for sid, df in iteritems(df_dict):
writer.write(sid, df)
).write(data)
def create_minute_df_for_asset(env,
@@ -1183,6 +1178,7 @@ zipline_git_root = abspath(
)
@nottest
def test_resource_path(*path_parts):
return os.path.join(zipline_git_root, 'tests', 'resources', *path_parts)
@@ -1313,3 +1309,37 @@ class tmp_bcolz_daily_bar_reader(_TmpBarReader):
@staticmethod
def _write(env, days, path, data):
BcolzDailyBarWriter(path, days).write(data)
@contextmanager
def patch_read_csv(url_map, module=pd, strict=False):
"""Patch pandas.read_csv to map lookups from url to another.
Parameters
----------
url_map : mapping[str or file-like object -> str or file-like object]
The mapping to use to redirect read_csv calls.
module : module, optional
The module to patch ``read_csv`` on. By default this is ``pandas``.
This should be set to another module if ``read_csv`` is early-bound
like ``from pandas import read_csv`` instead of late-bound like:
``import pandas as pd; pd.read_csv``.
strict : bool, optional
If true, then this will assert that ``read_csv`` is only called with
elements in the ``url_map``.
"""
read_csv = pd.read_csv
def patched_read_csv(filepath_or_buffer, *args, **kwargs):
if filepath_or_buffer in url_map:
return read_csv(url_map[filepath_or_buffer], *args, **kwargs)
elif not strict:
return read_csv(filepath_or_buffer, *args, **kwargs)
else:
raise AssertionError(
'attempted to call read_csv on %r which not in the url map' %
filepath_or_buffer,
)
with patch.object(module, 'read_csv', patched_read_csv):
yield
zipline/testing/fixtures.py
+3 -5
View File
@@ -6,7 +6,7 @@ from contextlib2 import ExitStack
from logbook import NullHandler, Logger
from nose_parameterized import parameterized
from pandas.util.testing import assert_series_equal
from six import with_metaclass, iteritems
from six import with_metaclass
from toolz import flip
import numpy as np
import pandas as pd
@@ -681,7 +681,7 @@ class WithBcolzMinuteBarReader(WithTradingEnvironment, WithTmpDir):
Methods
-------
make_minute_bar_data() -> dict[int -> pd.DataFrame]
make_minute_bar_data() -> iterable[(int, pd.DataFrame)]
A class method that returns a dict mapping sid to dataframe
which will be written to the bcolz files that the class's
``BcolzMinuteBarReader`` will read from. By default this creates
@@ -734,9 +734,7 @@ class WithBcolzMinuteBarReader(WithTradingEnvironment, WithTmpDir):
cls.env.open_and_closes.market_close.loc[days],
US_EQUITIES_MINUTES_PER_DAY
)
cls.bcolz_minute_bar_data = cls.make_minute_bar_data()
for sid, df in iteritems(cls.bcolz_minute_bar_data):
writer.write(sid, df)
writer.write(cls.make_minute_bar_data())
cls.bcolz_minute_bar_reader = BcolzMinuteBarReader(p)
zipline/testing/predicates.py
+193 -11
View File
@@ -1,7 +1,89 @@
from functools import partial
import inspect
from nose.tools import ( # noqa
assert_almost_equal,
assert_almost_equals,
assert_dict_contains_subset,
assert_false,
assert_greater,
assert_greater_equal,
assert_in,
assert_is,
assert_is_instance,
assert_is_none,
assert_is_not,
assert_is_not_none,
assert_less,
assert_less_equal,
assert_multi_line_equal,
assert_not_almost_equal,
assert_not_almost_equals,
assert_not_equal,
assert_not_equals,
assert_not_in,
assert_not_is_instance,
assert_raises,
assert_raises_regexp,
assert_regexp_matches,
assert_sequence_equal,
assert_set_equal,
assert_true,
assert_tuple_equal,
)
import numpy as np
import pandas as pd
from pandas.util.testing import assert_frame_equal
from six import iteritems, viewkeys, PY2
from toolz import dissoc, keyfilter
import toolz.curried.operator as op
from zipline.dispatch import dispatch
from zipline.lib.adjustment import Adjustment
from zipline.utils.functional import dzip_exact
from zipline.utils.math_utils import tolerant_equals
def keywords(func):
"""Get the argument names of a function
>>> def f(x, y=2):
... pass
>>> keywords(f)
['x', 'y']
Notes
-----
Taken from odo.utils
"""
if isinstance(func, type):
return keywords(func.__init__)
return inspect.getargspec(func).args
def filter_kwargs(f, kwargs):
"""Return a dict of valid kwargs for `f` from a subset of `kwargs`
Examples
--------
>>> def f(a, b=1, c=2):
... return a + b + c
...
>>> raw_kwargs = dict(a=1, b=3, d=4)
>>> f(**raw_kwargs)
Traceback (most recent call last):
...
TypeError: f() got an unexpected keyword argument 'd'
>>> kwargs = filter_kwargs(f, raw_kwargs)
>>> f(**kwargs)
6
Notes
-----
Taken from odo.utils
"""
return keyfilter(op.contains(keywords(f)), kwargs)
def _s(word, seq, suffix='s'):
@@ -41,8 +123,26 @@ def _fmt_path(path):
return 'path: _' + ''.join(path)
def _fmt_msg(msg):
"""Format the message for final display.
Parameters
----------
msg : str
The message to show to the user to provide additional context.
returns
-------
fmtd : str
The formatted message to put into the error message.
"""
if not msg:
return ''
return msg + '\n'
@dispatch(object, object)
def assert_equal(result, expected, path=(), **kwargs):
def assert_equal(result, expected, path=(), msg='', **kwargs):
"""Assert that two objects are equal using the ``==`` operator.
Parameters
@@ -57,15 +157,42 @@ def assert_equal(result, expected, path=(), **kwargs):
AssertionError
Raised when ``result`` is not equal to ``expected``.
"""
assert result == expected, '%s != %s\n%s' % (
assert result == expected, '%s%s != %s\n%s' % (
_fmt_msg(msg),
result,
expected,
_fmt_path(path),
)
@assert_equal.register(float, float)
def assert_float_equal(result,
expected,
path=(),
msg='',
float_rtol=10e-7,
float_atol=10e-7,
float_equal_nan=True,
**kwargs):
assert tolerant_equals(
result,
expected,
rtol=float_rtol,
atol=float_atol,
equal_nan=float_equal_nan,
), '%s%s != %s with rtol=%s and atol=%s%s\n%s' % (
_fmt_msg(msg),
result,
expected,
float_rtol,
float_atol,
(' (with nan != nan)' if not float_equal_nan else ''),
_fmt_path(path),
)
@assert_equal.register(dict, dict)
def assert_dict_equal(result, expected, path=(), **kwargs):
def assert_dict_equal(result, expected, path=(), msg='', **kwargs):
if path is None:
path = ()
@@ -89,8 +216,8 @@ def assert_dict_equal(result, expected, path=(), **kwargs):
in_expected,
)
raise AssertionError(
'dict keys do not match\n%s\n%s' % (
msg,
'%sdict keys do not match\n%s' % (
_fmt_msg(msg),
_fmt_path(path + ('.%s()' % ('viewkeys' if PY2 else 'keys'),)),
),
)
@@ -102,6 +229,7 @@ def assert_dict_equal(result, expected, path=(), **kwargs):
resultv,
expectedv,
path=path + ('[%r]' % k,),
msg=msg,
**kwargs
)
except AssertionError as e:
@@ -112,14 +240,16 @@ def assert_dict_equal(result, expected, path=(), **kwargs):
@assert_equal.register(list, list) # noqa
def assert_list_equal(result, expected, path=(), **kwargs):
def assert_list_equal(result, expected, path=(), msg='', **kwargs):
result_len = len(result)
expected_len = len(expected)
assert result_len == expected_len, (
'list lengths do not match: %d != %d\n%s' %
result_len,
expected_len,
_fmt_path(path),
'%slist lengths do not match: %d != %d\n%s' % (
_fmt_msg(msg),
result_len,
expected_len,
_fmt_path(path),
)
)
for n, (resultv, expectedv) in enumerate(zip(result, expected)):
@@ -127,6 +257,56 @@ def assert_list_equal(result, expected, path=(), **kwargs):
resultv,
expectedv,
path=path + ('[%d]' % n,),
msg=msg,
**kwargs
)
@assert_equal.register(np.ndarray, np.ndarray)
def assert_array_equal(result,
expected,
path=(),
msg='',
array_verbose=True,
array_decimal=None,
**kwargs):
f = (
np.testing.assert_array_equal
if array_decimal is None else
partial(np.testing.assert_array_almost_equal, decimal=array_decimal)
)
try:
f(
result,
expected,
verbose=array_verbose,
err_msg=msg,
)
except AssertionError as e:
raise AssertionError('\n'.join((str(e), _fmt_path(path))))
@assert_equal.register(pd.DataFrame, pd.DataFrame)
def assert_dataframe_equal(result, expected, path=(), msg='', **kwargs):
try:
assert_frame_equal(
result,
expected,
**filter_kwargs(assert_frame_equal, kwargs)
)
except AssertionError as e:
raise AssertionError(
_fmt_msg(msg) + '\n'.join((str(e), _fmt_path(path))),
)
@assert_equal.register(Adjustment, Adjustment)
def assert_adjustment_equal(result, expected, path=(), **kwargs):
for attr in ('first_row', 'last_row', 'first_col', 'last_col', 'value'):
assert_equal(
getattr(result, attr),
getattr(expected, attr),
path=path + ('.' + attr,),
**kwargs
)
@@ -137,4 +317,6 @@ try:
except ImportError:
pass
else:
assert_equal.funcs.update(assert_dshape_equal.funcs)
assert_equal.funcs.update(
dissoc(assert_dshape_equal.funcs, (object, object)),
)
zipline/utils/__init__.py
-18
View File
@@ -1,18 +0,0 @@
#
# Copyright 2014 Quantopian, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from .cli import run_pipeline, parse_args, parse_cell_magic
__all__ = ['run_pipeline', 'parse_args', 'parse_cell_magic']
zipline/utils/cache.py
+238 -19
View File
@@ -1,11 +1,22 @@
"""
Caching utilities for zipline
"""
from collections import namedtuple
from collections import namedtuple, MutableMapping
import errno
import os
import pickle
from shutil import rmtree, copyfile, copytree
from tempfile import mkdtemp, NamedTemporaryFile
import pandas as pd
from .context_tricks import nop_context
from .paths import ensure_directory
class Expired(Exception):
pass
"""Marks that a :class:`CachedObject` has expired.
"""
class CachedObject(namedtuple("_CachedObject", "value expires")):
@@ -20,11 +31,6 @@ class CachedObject(namedtuple("_CachedObject", "value expires")):
Expiration date of `value`. The cache is considered invalid for dates
**strictly greater** than `expires`.
Methods
-------
get(self, dt)
Get the cached object.
Usage
-----
>>> from pandas import Timestamp, Timedelta
@@ -66,22 +72,11 @@ class ExpiringCache(object):
Parameters
----------
cache : dict-like
cache : dict-like, optional
An instance of a dict-like object which needs to support at least:
`__del__`, `__getitem__`, `__setitem__`
If `None`, than a dict is used as a default.
Methods
-------
get(self, key, dt)
Get the value of a cached object for the given `key` at `dt`, if the
CachedObject has expired then the object is removed from the cache,
and `KeyError` is raised.
set(self, key, value, expiration_dt)
Add a new `value` to the cache at `dt` wrapped in a CachedObject which
expires at `expiration_dt`.
Usage
-----
>>> from pandas import Timestamp, Timedelta
@@ -104,6 +99,26 @@ class ExpiringCache(object):
self._cache = {}
def get(self, key, dt):
"""Get the value of a cached object.
Parameters
----------
key : any
The key to lookup.
dt : datetime
The time of the lookup.
Returns
-------
result : any
The value for ``key``.
Raises
------
KeyError
Raised if the key is not in the cache or the value for the key
has expired.
"""
try:
return self._cache[key].unwrap(dt)
except Expired:
@@ -111,4 +126,208 @@ class ExpiringCache(object):
raise KeyError(key)
def set(self, key, value, expiration_dt):
"""Adds a new key value pair to the cache.
Parameters
----------
key : any
The key to use for the pair.
value : any
The value to store under the name ``key``.
expiration_dt : datetime
When should this mapping expire? The cache is considered invalid
for dates **strictly greater** than ``expiration_dt``.
"""
self._cache[key] = CachedObject(value, expiration_dt)
class dataframe_cache(MutableMapping):
"""A disk-backed cache for dataframes.
``dataframe_cache`` is a mutable mapping from string names to pandas
DataFrame objects.
This object may be used as a context manager to delete the cache directory
on exit.
Parameters
----------
path : str, optional
The directory path to the cache. Files will be written as
``path/<keyname>``.
lock : Lock, optional
Thread lock for multithreaded/multiprocessed access to the cache.
If not provided no locking will be used.
clean_on_failure : bool, optional
Should the directory be cleaned up if an exception is raised in the
context manager.
serialize : {'msgpack', 'pickle:<n>'}, optional
How should the data be serialized. If ``'pickle'`` is passed, an
optional pickle protocol can be passed like: ``'pickle:3'`` which says
to use pickle protocol 3.
Notes
-----
The syntax ``cache[:]`` will load all key:value pairs into memory as a
dictionary.
The cache uses a temporary file format that is subject to change between
versions of zipline.
"""
def __init__(self,
path=None,
lock=None,
clean_on_failure=True,
serialization='msgpack'):
self.path = path if path is not None else mkdtemp()
self.lock = lock if lock is not None else nop_context
self.clean_on_failure = clean_on_failure
if serialization == 'msgpack':
self.serialize = pd.DataFrame.to_msgpack
self.deserialize = pd.read_msgpack
self._protocol = None
else:
s = serialization.split(':', 1)
if s[0] != 'pickle':
raise ValueError(
"'serialization' must be either 'msgpack' or 'pickle[:n]'",
)
self._protocol = int(s[1]) if len(s) == 2 else None
self.serialize = self._serialize_pickle
self.deserialize = self._deserialize_pickle
ensure_directory(self.path)
def _serialize_pickle(self, df, path):
with open(path, 'wb') as f:
pickle.dump(df, f, protocol=self._protocol)
def _deserialize_pickle(self, path):
with open(path, 'rb') as f:
return pickle.load(f)
def _keypath(self, key):
return os.path.join(self.path, key)
def __enter__(self):
return self
def __exit__(self, type_, value, tb):
if not (self.clean_on_failure or value is None):
# we are not cleaning up after a failure and there was an exception
return
with self.lock:
rmtree(self.path)
def __getitem__(self, key):
if key == slice(None):
return dict(self.items())
with self.lock:
try:
return self.deserialize(self._keypath(key))
except UnboundLocalError:
# This is how pandas fails if the file doesn't exist! #pandas
raise KeyError(key)
def __setitem__(self, key, value):
with self.lock:
self.serialize(value, self._keypath(key))
def __delitem__(self, key):
with self.lock:
try:
os.remove(self._keypath(key))
except OSError as e:
if e.errno == errno.ENOENT:
# raise a keyerror if this directory did not exist
raise KeyError(key)
# reraise the actual oserror otherwise
raise
def __iter__(self):
return iter(os.listdir(self.path))
def __len__(self):
return len(os.listdir(self.path))
def __repr__(self):
return '<%s: keys={%s}>' % (
type(self).__name__,
', '.join(map(repr, sorted(self))),
)
class working_file(object):
"""A context manager for managing a temporary file that will be moved
to a non-temporary location if no exceptions are raised in the context.
Parameters
----------
final_path : str
The location to move the file when committing.
*args, **kwargs
Forwarded to NamedTemporaryFile.
Notes
-----
The file is moved on __exit__ if there are no exceptions.
``working_file`` uses :func:`shutil.copyfile` to move the actual files,
meaning it has as strong of guarantees as :func:`shutil.copyfile`.
"""
def __init__(self, final_path, *args, **kwargs):
self._tmpfile = NamedTemporaryFile(*args, **kwargs)
self._final_path = final_path
def _commit(self):
"""Sync the temporary file to the final path.
"""
copyfile(self.name, self._final_path)
def __getattr__(self, attr):
return getattr(self._tmpfile, attr)
def __enter__(self):
self._tmpfile.__enter__()
return self
def __exit__(self, *exc_info):
if exc_info[0] is None:
self._commit()
self._tmpfile.__exit__(*exc_info)
class working_dir(object):
"""A context manager for managing a temporary directory that will be moved
to a non-temporary location if no exceptions are raised in the context.
Parameters
----------
final_path : str
The location to move the file when committing.
*args, **kwargs
Forwarded to tmp_dir.
Notes
-----
The file is moved on __exit__ if there are no exceptions.
``working_dir`` uses :func:`shutil.copytree` to move the actual files,
meaning it has as strong of guarantees as :func:`shutil.copytree`.
"""
def __init__(self, final_path, *args, **kwargs):
self.name = mkdtemp()
self._final_path = final_path
def _commit(self):
"""Sync the temporary directory to the final path.
"""
copytree(self.name, self._final_path)
def __enter__(self):
return self
def __exit__(self, *exc_info):
if exc_info[0] is None:
self._commit()
rmtree(self.name)
zipline/utils/cli.py
+87 -254
View File
@@ -1,259 +1,8 @@
#
# Copyright 2014 Quantopian, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import sys
import os
import argparse
from copy import copy
import click
from six import print_
from six.moves import configparser
import pandas as pd
try:
from pygments import highlight
from pygments.lexers import PythonLexer
from pygments.formatters import TerminalFormatter
PYGMENTS = True
except:
PYGMENTS = False
import zipline
from zipline.errors import NoSourceError, PipelineDateError
from .context_tricks import CallbackManager
DEFAULTS = {
'data_frequency': 'daily',
'capital_base': '10e6',
'source': 'yahoo',
'symbols': 'AAPL',
'metadata_index': 'symbol',
'source_time_column': 'Date',
}
def parse_args(argv, ipython_mode=False):
"""Parse list of arguments.
If a config file is provided (via -c), it will read in the
supplied options and overwrite any global defaults.
All other directly supplied arguments will overwrite the config
file settings.
Arguments:
* argv : list of strings
List of arguments, e.g. ['-c', 'my.conf']
* ipython_mode : bool <default=True>
Whether to parse IPython specific arguments
like --local_namespace
Notes:
Default settings can be found in zipline.utils.cli.DEFAULTS.
"""
# Parse any conf_file specification
# We make this parser with add_help=False so that
# it doesn't parse -h and print help.
conf_parser = argparse.ArgumentParser(
# Don't mess with format of description
formatter_class=argparse.RawDescriptionHelpFormatter,
# Turn off help, so we print all options in response to -h
add_help=False
)
conf_parser.add_argument("-c", "--conf_file",
help="Specify config file",
metavar="FILE")
args, remaining_argv = conf_parser.parse_known_args(argv)
defaults = copy(DEFAULTS)
if args.conf_file:
config = configparser.SafeConfigParser()
config.read([args.conf_file])
defaults.update(dict(config.items("Defaults")))
# Parse rest of arguments
# Don't suppress add_help here so it will handle -h
parser = argparse.ArgumentParser(
# Inherit options from config_parser
description="Zipline version %s." % zipline.__version__,
parents=[conf_parser]
)
parser.set_defaults(**defaults)
parser.add_argument('--algofile', '-f')
parser.add_argument('--data-frequency',
choices=('minute', 'daily'))
parser.add_argument('--start', '-s')
parser.add_argument('--end', '-e')
parser.add_argument('--capital_base')
parser.add_argument('--source', '-d', choices=('yahoo',))
parser.add_argument('--source_time_column', '-t')
parser.add_argument('--symbols')
parser.add_argument('--output', '-o')
parser.add_argument('--metadata_path', '-m')
parser.add_argument('--metadata_index', '-x')
parser.add_argument('--print-algo', '-p', dest='print_algo',
action='store_true')
parser.add_argument('--no-print-algo', '-q', dest='print_algo',
action='store_false')
if ipython_mode:
parser.add_argument('--local_namespace', action='store_true')
args = parser.parse_args(remaining_argv)
return(vars(args))
def parse_cell_magic(line, cell):
"""Parse IPython magic
"""
args_list = line.split(' ')
args = parse_args(args_list, ipython_mode=True)
# Remove print_algo kwarg to overwrite below.
args.pop('print_algo')
local_namespace = args.pop('local_namespace', False)
# By default, execute inside IPython namespace
if not local_namespace:
args['namespace'] = get_ipython().user_ns # flake8: noqa
# If we are running inside NB, do not output to file but create a
# variable instead
output_var_name = args.pop('output', None)
perf = run_pipeline(print_algo=False, algo_text=cell, **args)
if output_var_name is not None:
get_ipython().user_ns[output_var_name] = perf # flake8: noqa
def run_pipeline(print_algo=True, **kwargs):
"""Runs a full zipline pipeline given configuration keyword
arguments.
1. Load data (start and end dates can be provided a strings as
well as the source and symobls).
2. Instantiate algorithm (supply either algo_text or algofile
kwargs containing initialize() and handle_data() functions). If
algofile is supplied, will try to look for algofile_analyze.py and
append it.
3. Run algorithm (supply capital_base as float).
4. Return performance dataframe.
:Arguments:
* print_algo : bool <default=True>
Whether to print the algorithm to command line. Will use
pygments syntax coloring if pygments is found.
"""
start = kwargs['start']
end = kwargs['end']
# Compare against None because strings/timestamps may have been given
if start is not None:
start = pd.Timestamp(start, tz='UTC')
if end is not None:
end = pd.Timestamp(end, tz='UTC')
# Fail out if only one bound is provided
if ((start is None) or (end is None)) and (start != end):
raise PipelineDateError(start=start, end=end)
# Check if start and end are provided, and if the sim_params need to read
# a start and end from the DataSource
if start is None:
overwrite_sim_params = True
else:
overwrite_sim_params = False
symbols = kwargs['symbols'].split(',')
asset_identifier = kwargs['metadata_index']
# Pull asset metadata
asset_metadata = kwargs.get('asset_metadata', None)
asset_metadata_path = kwargs['metadata_path']
# Read in a CSV file, if applicable
if asset_metadata_path is not None:
if os.path.isfile(asset_metadata_path):
asset_metadata = pd.read_csv(asset_metadata_path,
index_col=asset_identifier)
source_arg = kwargs['source']
source_time_column = kwargs['source_time_column']
if source_arg is None:
raise NoSourceError()
elif source_arg == 'yahoo':
source = zipline.data.load_bars_from_yahoo(
stocks=symbols, start=start, end=end)
elif os.path.isfile(source_arg):
source = zipline.data.load_prices_from_csv(
filepath=source_arg,
identifier_col=source_time_column
)
elif os.path.isdir(source_arg):
source = zipline.data.load_prices_from_csv_folder(
folderpath=source_arg,
identifier_col=source_time_column
)
else:
raise NotImplementedError(
'Source %s not implemented.' % kwargs['source'])
algo_text = kwargs.get('algo_text', None)
if algo_text is None:
# Expect algofile to be set
algo_fname = kwargs['algofile']
with open(algo_fname, 'r') as fd:
algo_text = fd.read()
if print_algo:
if PYGMENTS:
highlight(algo_text, PythonLexer(), TerminalFormatter(),
outfile=sys.stdout)
else:
print_(algo_text)
algo = zipline.TradingAlgorithm(script=algo_text,
namespace=kwargs.get('namespace', {}),
capital_base=float(kwargs['capital_base']),
algo_filename=kwargs.get('algofile'),
equities_metadata=asset_metadata,
start=start,
end=end)
perf = algo.run(source, overwrite_sim_params=overwrite_sim_params)
output_fname = kwargs.get('output', None)
if output_fname is not None:
perf.to_pickle(output_fname)
return perf
def maybe_show_progress(it, show_progress, **kwargs):
"""Optionally show a progress bar for the given iterator.
@@ -274,12 +23,96 @@ def maybe_show_progress(it, show_progress, **kwargs):
Examples
--------
with maybe_show_progress([1, 2, 3], True) as ns:
for n in ns:
...
.. code-block:: python
with maybe_show_progress([1, 2, 3], True) as ns:
for n in ns:
...
"""
if show_progress:
return click.progressbar(it, **kwargs)
# context manager that just return `it` when we enter it
return CallbackManager(lambda it=it: it)
class _DatetimeParam(click.ParamType):
def __init__(self, tz=None):
self.tz = tz
def parser(self, value):
return pd.Timestamp(value, tz=self.tz)
@property
def name(self):
return type(self).__name__.upper()
def convert(self, value, param, ctx):
try:
return self.parser(value)
except ValueError:
self.fail(
'%s is not a valid %s' % (value, self.name.lower()),
param,
ctx,
)
class Timestamp(_DatetimeParam):
"""A click parameter that parses the value into pandas.Timestamp objects.
Parameters
----------
tz : timezone-coercable, optional
The timezone to parse the string as.
By default the timezone will be infered from the string or naiive.
"""
class Date(_DatetimeParam):
"""A click parameter that parses the value into datetime.date objects.
Parameters
----------
tz : timezone-coercable, optional
The timezone to parse the string as.
By default the timezone will be infered from the string or naiive.
as_timestamp : bool, optional
If True, return the value as a pd.Timestamp object normalized to
midnight.
"""
def __init__(self, tz=None, as_timestamp=False):
super(Date, self).__init__(tz=tz)
self.as_timestamp = as_timestamp
def parser(self, value):
ts = super(Date, self).parser(value)
return ts.normalize() if self.as_timestamp else ts.date()
class Time(_DatetimeParam):
"""A click parameter that parses the value into timetime.time objects.
Parameters
----------
tz : timezone-coercable, optional
The timezone to parse the string as.
By default the timezone will be infered from the string or naiive.
"""
def parser(self, value):
return super(Time, self).parser(value).time()
class Timedelta(_DatetimeParam):
"""A click parameter that parses values into pd.Timedelta objects.
Parameters
----------
unit : {'D', 'h', 'm', 's', 'ms', 'us', 'ns'}, optional
Denotes the unit of the input if the input is an integer.
"""
def __init__(self, unit='ns'):
self.unit = unit
def parser(self, value):
return pd.Timedelta(value, unit=self.unit)
zipline/utils/compat.py
-3
View File
@@ -2,7 +2,6 @@ from six import PY2
if PY2:
from functools32 import lru_cache
from ctypes import py_object, pythonapi
mappingproxy = pythonapi.PyDictProxy_New
@@ -10,10 +9,8 @@ if PY2:
mappingproxy.restype = py_object
else:
from functools import lru_cache
from types import MappingProxyType as mappingproxy
__all__ = [
'lru_cache',
'mappingproxy',
]
zipline/utils/input_validation.py
+30
View File
@@ -16,6 +16,7 @@ from functools import partial, wraps
from operator import attrgetter
from numpy import dtype
import pandas as pd
from pytz import timezone
from six import iteritems, string_types, PY3
from toolz import valmap, complement, compose
@@ -133,6 +134,35 @@ def ensure_timezone(func, argname, arg):
)
def ensure_timestamp(func, argname, arg):
"""Argument preprocessor that converts the input into a pandas Timestamp
object.
Usage
-----
>>> from zipline.utils.preprocess import preprocess
>>> @preprocess(ts=ensure_timestamp)
... def foo(ts):
... return ts
>>> foo('2014-01-01')
Timestamp('2014-01-01 00:00:00')
"""
try:
return pd.Timestamp(arg)
except ValueError as e:
raise TypeError(
"{func}() couldn't convert argument "
"{argname}={arg!r} to a pandas Timestamp.\n"
"Original error was: {t}: {e}".format(
func=_qualified_name(func),
argname=argname,
arg=arg,
t=_qualified_name(type(e)),
e=e,
),
)
def expect_dtypes(*_pos, **named):
"""
Preprocessing decorator that verifies inputs have expected numpy dtypes.
zipline/utils/math_utils.py
+28 -2
View File
@@ -12,11 +12,37 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import math
from numpy import isnan
def tolerant_equals(a, b, atol=10e-7, rtol=10e-7):
def tolerant_equals(a, b, atol=10e-7, rtol=10e-7, equal_nan=False):
"""Check if a and b are equal with some tolerance.
Parameters
----------
a, b : float
The floats to check for equality.
atol : float, optional
The absolute tolerance.
rtol : float, optional
The relative tolerance.
equal_nan : bool, optional
Should NaN compare equal?
See Also
--------
numpy.isclose
Notes
-----
This function is just a scalar version of numpy.isclose for performance.
See the docstring of ``isclose`` for more information about ``atol`` and
``rtol``.
"""
if equal_nan and isnan(a) and isnan(b):
return True
return math.fabs(a - b) <= (atol + rtol * math.fabs(b))
zipline/utils/paths.py
+223
View File
@@ -0,0 +1,223 @@
"""
Canonical path locations for zipline data.
Paths are rooted at $ZIPLINE_ROOT if that environment variable is set.
Otherwise default to expanduser(~/.zipline)
"""
from errno import EEXIST
import os
from os.path import exists, expanduser, join
import pandas as pd
def hidden(path):
"""Check if a path is hidden.
Parameters
----------
path : str
A filepath.
"""
return path.startswith('.')
def ensure_directory(path):
"""
Ensure that a directory named "path" exists.
"""
try:
os.makedirs(path)
except OSError as exc:
if exc.errno == EEXIST and os.path.isdir(path):
return
raise
def ensure_directory_containing(path):
"""
Ensure that the directory containing `path` exists.
This is just a convenience wrapper for doing::
ensure_directory(os.path.dirname(path))
"""
ensure_directory(os.path.dirname(path))
def last_modified_time(path):
"""
Get the last modified time of path as a Timestamp.
"""
return pd.Timestamp(os.path.getmtime(path), unit='s', tz='UTC')
def modified_since(path, dt):
"""
Check whether `path` was modified since `dt`.
Returns False if path doesn't exist.
Parameters
----------
path : str
Path to the file to be checked.
dt : pd.Timestamp
The date against which to compare last_modified_time(path).
Returns
-------
was_modified : bool
Will be ``False`` if path doesn't exists, or if its last modified date
is earlier than or equal to `dt`
"""
return exists(path) and last_modified_time(path) > dt
def zipline_root(environ=None):
"""
Get the root directory for all zipline-managed files.
For testing purposes, this accepts a dictionary to interpret as the os
environment.
Parameters
----------
environ : dict, optional
A dict to interpret as the os environment.
Returns
-------
root : string
Path to the zipline root dir.
"""
if environ is None:
environ = os.environ
root = environ.get('ZIPLINE_ROOT', None)
if root is None:
root = expanduser('~/.zipline')
return root
def zipline_path(paths, environ=None):
"""
Get a path relative to the zipline root.
Parameters
----------
paths : list[str]
List of requested path pieces.
environ : dict, optional
An environment dict to forward to zipline_root.
Returns
-------
newpath : str
The requested path joined with the zipline root.
"""
return join(zipline_root(environ=environ), *paths)
def default_extension(environ=None):
"""
Get the path to the default zipline extension file.
Parameters
----------
environ : dict, optional
An environment dict to forwart to zipline_root.
Returns
-------
default_extension_path : str
The file path to the default zipline extension file.
"""
return zipline_path(['extension.py'], environ=environ)
def data_root(environ=None):
"""
The root directory for zipline data files.
Parameters
----------
environ : dict, optional
An environment dict to forward to zipline_root.
Returns
-------
data_root : str
The zipline data root.
"""
return zipline_path(['data'], environ=environ)
def ensure_data_root(environ=None):
"""
Ensure that the data root exists.
"""
ensure_directory(data_root(environ=environ))
def data_path(paths, environ=None):
"""
Get a path relative to the zipline data directory.
Parameters
----------
paths : iterable[str]
List of requested path pieces.
environ : dict, optional
An environment dict to forward to zipline_root.
Returns
-------
newpath : str
The requested path joined with the zipline data root.
"""
return zipline_path(['data'] + list(paths), environ=environ)
def cache_root(environ=None):
"""
The root directory for zipline cache files.
Parameters
----------
environ : dict, optional
An environment dict to forward to zipline_root.
Returns
-------
cache_root : str
The zipline cache root.
"""
return zipline_path(['cache'], environ=environ)
def ensure_cache_root(environ=None):
"""
Ensure that the data root exists.
"""
ensure_directory(cache_root(environ=environ))
def cache_path(paths, environ=None):
"""
Get a path relative to the zipline cache directory.
Parameters
----------
paths : iterable[str]
List of requested path pieces.
environ : dict, optional
An environment dict to forward to zipline_root.
Returns
-------
newpath : str
The requested path joined with the zipline cache root.
"""
return zipline_path(['cache'] + list(paths), environ=environ)
zipline/utils/run_algo.py
+335
View File
@@ -0,0 +1,335 @@
import os
import re
from runpy import run_path
import sys
import warnings
import click
try:
from pygments import highlight
from pygments.lexers import PythonLexer
from pygments.formatters import TerminalFormatter
PYGMENTS = True
except:
PYGMENTS = False
from toolz import valfilter, concatv
from zipline.algorithm import TradingAlgorithm
from zipline.data.bundles.core import load
from zipline.data.data_portal import DataPortal
from zipline.finance.trading import TradingEnvironment
import zipline.utils.paths as pth
class _RunAlgoError(click.ClickException, ValueError):
"""Signal an error that should have a different message if invoked from
the cli.
Parameters
----------
pyfunc_msg : str
The message that will be shown when called as a python function.
cmdline_msg : str
The message that will be shown on the command line.
"""
exit_code = 1
def __init__(self, pyfunc_msg, cmdline_msg):
super(_RunAlgoError, self).__init__(cmdline_msg)
self.pyfunc_msg = pyfunc_msg
def __str__(self):
return self.pyfunc_msg
def _run(handle_data,
initialize,
before_trading_start,
analyze,
algofile,
algotext,
defines,
data_frequency,
capital_base,
data,
bundle,
bundle_timestamp,
start,
end,
output,
print_algo,
local_namespace,
environ):
"""Run a backtest for the given algorithm.
This is shared between the cli and :func:`zipline.run_algo`.
"""
if algotext is not None:
if local_namespace:
ip = get_ipython() # noqa
namespace = ip.user_ns
else:
namespace = {}
for assign in defines:
try:
name, value = assign.split('=', 2)
except ValueError:
raise ValueError(
'invalid define %r, should be of the form name=value' %
assign,
)
try:
# evaluate in the same namespace so names may refer to
# eachother
namespace[name] = eval(value, namespace)
except Exception as e:
raise ValueError(
'failed to execute definition for name %r: %s' % (name, e),
)
elif defines:
raise _RunAlgoError(
'cannot pass define without `algotext`',
"cannot pass '-D' / '--define' without '-t' / '--algotext'",
)
else:
namespace = {}
if algofile is not None:
algotext = algofile.read()
if print_algo:
if PYGMENTS:
highlight(
algotext,
PythonLexer(),
TerminalFormatter(),
outfile=sys.stdout,
)
else:
click.echo(algotext)
if bundle is not None:
bundle_data = load(
bundle,
environ,
bundle_timestamp,
)
prefix, connstr = re.split(
r'sqlite:///',
str(bundle_data.asset_finder.engine.url),
maxsplit=1,
)
if prefix:
raise ValueError(
"invalid url %r, must begin with 'sqlite:///'" %
str(bundle_data.asset_finder.engine.url),
)
env = TradingEnvironment(asset_db_path=connstr)
data = DataPortal(
env,
equity_minute_reader=bundle_data.minute_bar_reader,
equity_daily_reader=bundle_data.daily_bar_reader,
adjustment_reader=bundle_data.adjustment_reader,
)
perf = TradingAlgorithm(
namespace=namespace,
capital_base=capital_base,
start=start,
end=end,
env=env,
**{
'initialize': initialize,
'handle_data': handle_data,
'before_trading_start': before_trading_start,
'analyze': analyze,
} if algotext is None else {
'algo_filename': algofile,
'script': algotext,
}
).run(
data,
overwrite_sim_params=False,
)
if output == '-':
click.echo(str(perf))
elif output != os.devnull: # make the zipline magic not write any data
perf.to_pickle(output)
return perf
# All of the loaded extensions. We don't want to load an extension twice.
_loaded_extensions = set()
def load_extensions(default, extensions, strict, environ, reload=False):
"""Load all of the given extensions. This should be called by run_algo
or the cli.
Parameters
----------
default : bool
Load the default exension (~/.zipline/extension.py)?
extension : iterable[str]
The paths to the extensions to load. If the path ends in ``.py`` it is
treated as a script and executed. If it does not end in ``.py`` it is
treated as a module to be imported.
strict : bool
Should failure to load an extension raise. If this is false it will
still warn.
environ : mapping
The environment to use to find the default extension path.
reload : bool, optional
Reload any extensions that have already been loaded.
"""
if default:
default_extension_path = pth.default_extension(environ=environ)
open(default_extension_path, 'a+').close() # touch the file
# put the default extension first so other extensions can depend on
# the order they are loaded
extensions = concatv([default_extension_path], extensions)
for ext in extensions:
if ext in _loaded_extensions and not reload:
continue
try:
# load all of the zipline extensionss
if ext.endswith('.py'):
run_path(ext, run_name='<extension>')
else:
__import__(ext)
except Exception as e:
if strict:
# if `strict` we should raise the actual exception and fail
raise
# without `strict` we should just log the failure
warnings.warn(
'Failed to load extension: %r\n%s' % (ext, e),
stacklevel=2
)
else:
_loaded_extensions.add(ext)
def run_algorithm(start,
end,
initialize,
capital_base,
handle_data=None,
before_trading_start=None,
analyze=None,
data_frequency='daily',
data=None,
bundle=None,
bundle_timestamp=None,
default_extension=True,
extensions=(),
strict_extensions=True,
environ=os.environ):
"""Run a trading algorithm.
Parameters
----------
start : datetime
The start date of the backtest.
end : datetime
The end date of the backtest..
initialize : callable[context -> None]
The initialize function to use for the algorithm. This is called once
at the very begining of the backtest and should be used to set up
any state needed by the algorithm.
capital_base : float
The starting capital for the backtest.
handle_data : callable[(context, BarData) -> None], optional
The handle_data function to use for the algorithm. This is called
every minute when ``data_frequency == 'minute'`` or every day
when ``data_frequency == 'daily'``.
before_trading_start : callable[(context, BarData) -> None], optional
The before_trading_start function for the algorithm. This is called
once before each trading day (after initialize on the first day).
analyze : callable[(context, pd.DataFrame) -> None], optional
The analyze function to use for the algorithm. This function is called
once at the end of the backtest and is passed the context and the
performance data.
data_frequency : {'daily', 'minute'}, optional
The data frequency to run the algorithm at.
data : pd.DataFrame, pd.Panel, or DataPortal, optional
The ohlcv data to run the backtest with.
This argument is mutually exclusive with:
``bundle``
``bundle_timestamp``
bundle : str, optional
The name of the data bundle to use to load the data to run the backtest
with. This defaults to 'quandl'.
This argument is mutually exclusive with ``data``.
bundle_timestamp : datetime, optional
The datetime to lookup the bundle data for. This defaults to the
current time.
This argument is mutually exclusive with ``data``.
default_extension : bool, optional
Should the default zipline extension be loaded. This is found at
``$ZIPLINE_ROOT/extension.py``
extensions : iterable[str], optional
The names of any other extensions to load. Each element may either be
a dotted module path like ``a.b.c`` or a path to a python file ending
in ``.py`` like ``a/b/c.py``.
strict_extensions : bool, optional
Should the run fail if any extensions fail to load. If this is false,
a warning will be raised instead.
environ : mapping[str -> str], optional
The os environment to use. Many extensions use this to get parameters.
This defaults to ``os.environ``.
Returns
-------
perf : pd.DataFrame
The daily performance of the algorithm.
See Also
--------
zipline.data.bundles.bundles : The available data bundles.
"""
load_extensions(default_extension, extensions, strict_extensions, environ)
non_none_data = valfilter(bool, {
'data': data,
'bundle': bundle,
})
if not non_none_data:
# if neither data nor bundle are passed use 'quandl'
bundle = 'quandl'
if len(non_none_data) != 1:
raise ValueError(
'must specify one of `data`, `data_portal`, or `bundle`,'
' got: %r' % non_none_data,
)
if 'bundle' not in non_none_data and bundle_timestamp is not None:
raise ValueError(
'cannot specify `bundle_timestamp` without passing `bundle`',
)
return _run(
handle_data=handle_data,
initialize=initialize,
before_trading_start=before_trading_start,
analyze=analyze,
algofile=None,
algotext=None,
defines=(),
data_frequency=data_frequency,
capital_base=capital_base,
data=data,
bundle=bundle,
bundle_timestamp=bundle_timestamp,
start=start,
end=end,
output=os.devnull,
print_algo=False,
local_namespace=False,
environ=environ,
)