diff --git a/docs/source/appendix.rst b/docs/source/appendix.rst index 27d28c2d..1ef2ded5 100644 --- a/docs/source/appendix.rst +++ b/docs/source/appendix.rst @@ -1,6 +1,11 @@ API Reference ------------- +Running a Backtest +~~~~~~~~~~~~~~~~~~ + +.. autofunction:: zipline.run_algorithm(...) + Algorithm API ~~~~~~~~~~~~~ @@ -85,29 +90,88 @@ Pipeline API Asset Metadata ~~~~~~~~~~~~~~ -.. autoclass:: zipline.assets.assets.Asset +.. autoclass:: zipline.assets.Asset :members: -.. autoclass:: zipline.assets.assets.Equity +.. autoclass:: zipline.assets.Equity :members: -.. autoclass:: zipline.assets.assets.Future +.. autoclass:: zipline.assets.Future :members: -.. autoclass:: zipline.assets.assets.AssetFinder - :members: - -.. autoclass:: zipline.assets.assets.AssetFinderCachedEquities - :members: - -.. autoclass:: zipline.assets.asset_writer.AssetDBWriter - :members: - -.. autoclass:: zipline.assets.assets.AssetConvertible +.. autoclass:: zipline.assets.AssetConvertible :members: Data API ~~~~~~~~ +Writers +``````` .. autoclass:: zipline.data.minute_bars.BcolzMinuteBarWriter :members: + +.. autoclass:: zipline.data.us_equity_pricing.BcolzDailyBarWriter + :members: + +.. autoclass:: zipline.data.us_equity_pricing.SQLiteAdjustmentWriter + :members: + +.. autoclass:: zipline.assets.AssetDBWriter + :members: + +Readers +``````` +.. autoclass:: zipline.data.minute_bars.BcolzMinuteBarReader + :members: + +.. autoclass:: zipline.data.us_equity_pricing.BcolzDailyBarReader + :members: + +.. autoclass:: zipline.data.us_equity_pricing.SQLiteAdjustmentReader + :members: + +.. autoclass:: zipline.assets.AssetFinder + :members: + +.. autoclass:: zipline.assets.AssetFinderCachedEquities + :members: + +Bundles +``````` +.. autofunction:: zipline.data.bundles.register + +.. autofunction:: zipline.data.bundles.ingest(name, environ=os.environ, date=None, show_progress=True) + +.. autofunction:: zipline.data.bundles.load(name, environ=os.environ, date=None) + +.. autofunction:: zipline.data.bundles.unregister + +.. data:: zipline.data.bundles.bundles + + The bundles that have been registered as a mapping from bundle name to bundle + data. This mapping is immutable and should only be updated through + :func:`~zipline.data.bundles.register` or + :func:`~zipline.data.bundles.unregister`. + +.. autofunction:: zipline.data.bundles.yahoo_equities + + +Utilities +~~~~~~~~~ + +Caching +``````` + +.. autoclass:: zipline.utils.cache.CachedObject + +.. autoclass:: zipline.utils.cache.ExpiringCache + +.. autoclass:: zipline.utils.cache.dataframe_cache + +.. autoclass:: zipline.utils.cache.working_file + +.. autoclass:: zipline.utils.cache.working_dir + +Command Line +```````````` +.. autofunction:: zipline.utils.cli.maybe_show_progress diff --git a/docs/source/beginner-tutorial.rst b/docs/source/beginner-tutorial.rst index 80a81205..3bda3cea 100644 --- a/docs/source/beginner-tutorial.rst +++ b/docs/source/beginner-tutorial.rst @@ -51,53 +51,50 @@ My first algorithm Lets take a look at a very simple algorithm from the ``examples`` directory, ``buyapple.py``: -.. code:: python +.. code-block:: python - !tail ../../zipline/examples/buyapple.py + from zipline.examples import buyapple + buyapple?? -.. parsed-literal:: +.. code-block:: python - # Load price data from yahoo. - data = load_from_yahoo(stocks=['AAPL'], indexes={}, start=start, - end=end) + from zipline.api import order, record, symbol - # Create and run the algorithm. - algo = TradingAlgorithm(initialize=initialize, handle_data=handle_data, - identifiers=['AAPL']) - results = algo.run(data) - analyze(results=results) + def initialize(context): + pass + + + def handle_data(context, data): + order(symbol('AAPL'), 10) + record(AAPL=data.current(symbol('AAPL'), 'price')) As you can see, we first have to import some functions we would like to use. All functions commonly used in your algorithm can be found in -``zipline.api``. Here we are using ``order()`` which takes two arguments --- a security object, and a number specifying how many stocks you would -like to order (if negative, ``order()`` will sell/short stocks). In this -case we want to order 10 shares of Apple at each iteration. For more -documentation on ``order()``, see the `Quantopian -docs `__. +``zipline.api``. Here we are using :func:`~zipline.api.order()` which takes two +arguments: a security object, and a number specifying how many stocks you would +like to order (if negative, :func:`~zipline.api.order()` will sell/short +stocks). In this case we want to order 10 shares of Apple at each iteration. For +more documentation on ``order()``, see the `Quantopian docs +`__. -You don't have to use the ``symbol()`` function and could just pass in -``AAPL`` directly but it is good practice as this way your code will be -Quantopian compatible. - -Finally, the ``record()`` function allows you to save the value of a -variable at each iteration. You provide it with a name for the variable +Finally, the :func:`~zipline.api.record` function allows you to save the value +of a variable at each iteration. You provide it with a name for the variable together with the variable itself: ``varname=var``. After the algorithm finished running you will have access to each variable value you tracked -with ``record()`` under the name you provided (we will see this further -below). You also see how we can access the current price data of the +with :func:`~zipline.api.record` under the name you provided (we will see this +further below). You also see how we can access the current price data of the AAPL stock in the ``data`` event frame (for more information see `here `__. Running the algorithm ~~~~~~~~~~~~~~~~~~~~~ -To now test this algorithm on financial data, ``zipline`` provides two -interfaces. A command-line interface and an ``IPython Notebook`` -interface. +To now test this algorithm on financial data, ``zipline`` provides three +interfaces: A command-line interface, ``IPython Notebook`` magic, and +:func:`~zipline.run_algorithm`. Command line interface ^^^^^^^^^^^^^^^^^^^^^^ @@ -106,60 +103,59 @@ After you installed zipline you should be able to execute the following from your command line (e.g. ``cmd.exe`` on Windows, or the Terminal app on OSX): -.. code:: python - - !run_algo.py --help +.. code-block:: bash + $ python -m zipline run --help .. parsed-literal:: - usage: run_algo.py [-h] [-c FILE] [--algofile ALGOFILE] [--data-frequency {minute,daily}] [--start START] [--end END] - [--capital_base CAPITAL_BASE] [--source {yahoo}] [--source_time_column SOURCE_TIME_COLUMN] [--symbols SYMBOLS] - [--output OUTPUT] [--metadata_path METADATA_PATH] [--metadata_index METADATA_INDEX] [--print-algo] [--no-print-algo] + Usage: __main__.py run [OPTIONS] - Zipline version 0.8.3. + Run a backtest for the given algorithm. - optional arguments: - -h, --help show this help message and exit - -c FILE, --conf_file FILE - Specify config file - --algofile ALGOFILE, -f ALGOFILE - --data-frequency {minute,daily} - --start START, -s START - --end END, -e END - --capital_base CAPITAL_BASE - --source {yahoo}, -d {yahoo} - --source_time_column SOURCE_TIME_COLUMN, -t SOURCE_TIME_COLUMN - --symbols SYMBOLS - --output OUTPUT, -o OUTPUT - --metadata_path METADATA_PATH, -m METADATA_PATH - --metadata_index METADATA_INDEX, -x METADATA_INDEX - --print-algo, -p - --no-print-algo, -q + Options: + -f, --algofile FILENAME The file that contains the algorithm to run. + -t, --algotext TEXT The algorithm script to run. + -D, --define TEXT Define a name to be bound in the namespace + before executing the algotext. For example + '-Dname=value'. The value may be any python + expression. These are evaluated in order so + they may refer to previously defined names. + --data-frequency [minute|daily] + The data frequency of the simulation. + [default: daily] + --capital-base FLOAT The starting capital for the simulation. + [default: 10000000.0] + -b, --bundle BUNDLE-NAME The data bundle to use for the simulation. + [default: quandl] + --bundle-timestamp TIMESTAMP The date to lookup data on or before. + [default: ] + -s, --start DATE The start date of the simulation. + -e, --end DATE The end date of the simulation. + -o, --output FILENAME The location to write the perf data. If this + is '-' the perf will be written to stdout. + [default: -] + --print-algo / --no-print-algo Print the algorithm to stdout. + --help Show this message and exit. - -Note that you have to omit the preceding '!' when you call -``run_algo.py``, this is only required by the IPython Notebook in which -this tutorial was written. - -As you can see there are a couple of flags that specify where to find -your algorithm (``-f``) as well as parameters specifying which stock -data to load from Yahoo! finance (``--symbols``) and the time-range -(``--start`` and ``--end``). Finally, you'll want to save the -performance metrics of your algorithm so that you can analyze how it -performed. This is done via the ``--output`` flag and will cause it to -write the performance ``DataFrame`` in the pickle Python file format. -Note that you can also define a configuration file with these parameters -that you can then conveniently pass to the ``-c`` option so that you -don't have to supply the command line args all the time (see the .conf -files in the examples directory). +As you can see there are a couple of flags that specify where to find your +algorithm (``-f``) as well as parameters specifying which data to use, +defaulting to the :ref:`quandl-data-bundle`. There are also arguments for the +date range to run the algorithm over (``--start`` and ``--end``). Finally, +you'll want to save the performance metrics of your algorithm so that you can +analyze how it performed. This is done via the ``--output`` flag and will cause +it to write the performance ``DataFrame`` in the pickle Python file format. +Note that you can also define a configuration file with these parameters that +you can then conveniently pass to the ``-c`` option so that you don't have to +supply the command line args all the time (see the .conf files in the examples +directory). Thus, to execute our algorithm from above and save the results to -``buyapple_out.pickle`` we would call ``run_algo.py`` as follows: +``buyapple_out.pickle`` we would call ``python -m zipline run`` as follows: -.. code:: python +.. code-block:: python - !run_algo.py -f ../../zipline/examples/buyapple.py --start 2000-1-1 --end 2014-1-1 --symbols AAPL -o buyapple_out.pickle + python -m zipline run -f ../../zipline/examples/buyapple.py --start 2000-1-1 --end 2014-1-1 --symbols AAPL -o buyapple_out.pickle .. parsed-literal:: @@ -170,9 +166,7 @@ Thus, to execute our algorithm from above and save the results to [2015-11-04 22:45:32.820401] INFO: Performance: last close: 2013-12-31 21:00:00+00:00 -``run_algo.py`` first outputs the algorithm contents. It then fetches -historical price and volume data of Apple from Yahoo! finance in the -desired time range, calls the ``initialize()`` function, and then +``run`` first calls the ``initialize()`` function, and then streams the historical stock price day-by-day through ``handle_data()``. After each call to ``handle_data()`` we instruct ``zipline`` to order 10 stocks of AAPL. After the call of the ``order()`` function, ``zipline`` @@ -187,31 +181,18 @@ slippage model that ``zipline`` uses, see the `Quantopian docs `__ for more information). -Note that there is also an ``analyze()`` function printed. -``run_algo.py`` will try and look for a file with the ending with -``_analyze.py`` and the same name of the algorithm (so -``buyapple_analyze.py``) or an ``analyze()`` function directly in the -script. If an ``analyze()`` function is found it will be called *after* -the simulation has finished and passed in the performance ``DataFrame``. -(The reason for allowing specification of an ``analyze()`` function in a -separate file is that this way ``buyapple.py`` remains a valid -Quantopian algorithm that you can copy&paste to the platform). - Lets take a quick look at the performance ``DataFrame``. For this, we use ``pandas`` from inside the IPython Notebook and print the first ten rows. Note that ``zipline`` makes heavy usage of ``pandas``, especially for data input and outputting so it's worth spending some time to learn it. -.. code:: python +.. code-block:: python import pandas as pd perf = pd.read_pickle('buyapple_out.pickle') # read in perf DataFrame perf.head() - - - .. raw:: html
@@ -378,7 +359,7 @@ and allows us to plot the price of apple. For example, we could easily examine now how our portfolio value changed over time compared to the AAPL stock price. -.. code:: python +.. code-block:: python %pylab inline figsize(12, 12) @@ -391,21 +372,14 @@ AAPL stock price. perf.AAPL.plot(ax=ax2) ax2.set_ylabel('AAPL stock price') - .. parsed-literal:: Populating the interactive namespace from numpy and matplotlib - - - .. parsed-literal:: - - - .. image:: tutorial_files/tutorial_11_2.png @@ -431,28 +405,21 @@ to run the algorithm from above with the same parameters we just have to execute the following cell after importing ``zipline`` to register the magic. -.. code:: python +.. code-block:: python - import zipline + %load_ext zipline -.. code:: python +.. code-block:: python - %%zipline --start 2000-1-1 --end 2014-1-1 --symbols AAPL -o perf_ipython + %%zipline --start 2000-1-1 --end 2014-1-1 --symbols AAPL + from zipline.api import symbol, order, record - from zipline.api import symbol, order, record - - def initialize(context): - pass - - def handle_data(context, data): - order(symbol('AAPL'), 10) - record(AAPL=data[symbol('AAPL')].price) - - -.. parsed-literal:: - - AAPL + def initialize(context): + pass + def handle_data(context, data): + order(symbol('AAPL'), 10) + record(AAPL=data[symbol('AAPL')].price) Note that we did not have to specify an input file as above since the magic will use the contents of the cell and look for your algorithm @@ -460,12 +427,9 @@ functions there. Also, instead of defining an output file we are specifying a variable name with ``-o`` that will be created in the name space and contain the performance ``DataFrame`` we looked at above. -.. code:: python - - perf_ipython.head() - - +.. code-block:: python + _.head() .. raw:: html @@ -624,58 +588,6 @@ space and contain the performance ``DataFrame`` we looked at above.
- -Manual (advanced) -~~~~~~~~~~~~~~~~~ - -If you are happy with either way above you can safely skip this passage. -To provide a closer look at how ``zipline`` actually works it is -instructive to see how we run an algorithm without any of the interfaces -demonstrated above which hide the actual ``zipline`` API. - -.. code:: python - - import pytz - from datetime import datetime - - from zipline.algorithm import TradingAlgorithm - from zipline.utils.factory import load_bars_from_yahoo - - # Load data manually from Yahoo! finance - start = datetime(2000, 1, 1, 0, 0, 0, 0, pytz.utc) - end = datetime(2012, 1, 1, 0, 0, 0, 0, pytz.utc) - data = load_bars_from_yahoo(stocks=['AAPL'], start=start, - end=end) - - # Define algorithm - def initialize(context): - pass - - def handle_data(context, data): - order(symbol('AAPL'), 10) - record(AAPL=data[symbol('AAPL')].price) - - # Create algorithm object passing in initialize and - # handle_data functions - algo_obj = TradingAlgorithm(initialize=initialize, - handle_data=handle_data) - - # Run algorithm - perf_manual = algo_obj.run(data) - - -.. parsed-literal:: - - AAPL - - -As you can see, we again define the functions as above but we manually -pass them to the ``TradingAlgorithm`` class which is the main -``zipline`` class for running algorithms. We also manually load the data -using ``load_bars_from_yahoo()`` and pass it to the -``TradingAlgorithm.run()`` method which kicks off the backtest -simulation. - Access to previous prices using ``history`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -706,81 +618,73 @@ you can directly use the ``history()`` function on Quantopian, in with ``add_history()`` and pass it the same arguments as the history function below. Lets look at the strategy which should make this clear: -.. code:: python +.. code-block:: python - %%zipline --start 2000-1-1 --end 2014-1-1 --symbols AAPL -o perf_dma + %%zipline --start 2000-1-1 --end 2014-1-1 -o perf_dma - from zipline.api import order_target, record, symbol, history, add_history - import numpy as np + from zipline.api import order_target, record, symbol, history, add_history + import numpy as np - def initialize(context): - # Register 2 histories that track daily prices, - # one with a 100 window and one with a 300 day window - add_history(100, '1d', 'price') - add_history(300, '1d', 'price') + def initialize(context): + # Register 2 histories that track daily prices, + # one with a 100 window and one with a 300 day window + add_history(100, '1d', 'price') + add_history(300, '1d', 'price') - context.i = 0 + context.i = 0 - def handle_data(context, data): - # Skip first 300 days to get full windows - context.i += 1 - if context.i < 300: - return + def handle_data(context, data): + # Skip first 300 days to get full windows + context.i += 1 + if context.i < 300: + return - # Compute averages - # history() has to be called with the same params - # from above and returns a pandas dataframe. - short_mavg = history(100, '1d', 'price').mean() - long_mavg = history(300, '1d', 'price').mean() + # Compute averages + # history() has to be called with the same params + # from above and returns a pandas dataframe. + short_mavg = history(100, '1d', 'price').mean() + long_mavg = history(300, '1d', 'price').mean() - # Trading logic - if short_mavg[0] > long_mavg[0]: - # order_target orders as many shares as needed to - # achieve the desired number of shares. - order_target(symbol('AAPL'), 100) - elif short_mavg[0] < long_mavg[0]: - order_target(symbol('AAPL'), 0) + # Trading logic + if short_mavg[0] > long_mavg[0]: + # order_target orders as many shares as needed to + # achieve the desired number of shares. + order_target(symbol('AAPL'), 100) + elif short_mavg[0] < long_mavg[0]: + order_target(symbol('AAPL'), 0) - # Save values for later inspection - record(AAPL=data[symbol('AAPL')].price, - short_mavg=short_mavg[0], - long_mavg=long_mavg[0]) + # Save values for later inspection + record(AAPL=data[symbol('AAPL')].price, + short_mavg=short_mavg[0], + long_mavg=long_mavg[0]) - def analyze(context, perf): - fig = plt.figure() - ax1 = fig.add_subplot(211) - perf.portfolio_value.plot(ax=ax1) - ax1.set_ylabel('portfolio value in $') - - ax2 = fig.add_subplot(212) - perf['AAPL'].plot(ax=ax2) - perf[['short_mavg', 'long_mavg']].plot(ax=ax2) - - perf_trans = perf.ix[[t != [] for t in perf.transactions]] - buys = perf_trans.ix[[t[0]['amount'] > 0 for t in perf_trans.transactions]] - sells = perf_trans.ix[ - [t[0]['amount'] < 0 for t in perf_trans.transactions]] - ax2.plot(buys.index, perf.short_mavg.ix[buys.index], - '^', markersize=10, color='m') - ax2.plot(sells.index, perf.short_mavg.ix[sells.index], - 'v', markersize=10, color='k') - ax2.set_ylabel('price in $') - plt.legend(loc=0) - plt.show() - - -.. parsed-literal:: - - AAPL + def analyze(context, perf): + fig = plt.figure() + ax1 = fig.add_subplot(211) + perf.portfolio_value.plot(ax=ax1) + ax1.set_ylabel('portfolio value in $') + ax2 = fig.add_subplot(212) + perf['AAPL'].plot(ax=ax2) + perf[['short_mavg', 'long_mavg']].plot(ax=ax2) + perf_trans = perf.ix[[t != [] for t in perf.transactions]] + buys = perf_trans.ix[[t[0]['amount'] > 0 for t in perf_trans.transactions]] + sells = perf_trans.ix[ + [t[0]['amount'] < 0 for t in perf_trans.transactions]] + ax2.plot(buys.index, perf.short_mavg.ix[buys.index], + '^', markersize=10, color='m') + ax2.plot(sells.index, perf.short_mavg.ix[sells.index], + 'v', markersize=10, color='k') + ax2.set_ylabel('price in $') + plt.legend(loc=0) + plt.show() .. image:: tutorial_files/tutorial_22_1.png - Here we are explicitly defining an ``analyze()`` function that gets automatically called once the backtest is done (this is not possible on Quantopian currently). diff --git a/docs/source/bundles.rst b/docs/source/bundles.rst new file mode 100644 index 00000000..5134ec7f --- /dev/null +++ b/docs/source/bundles.rst @@ -0,0 +1,277 @@ +Data Bundles +------------ + +A data bundle is a collection of pricing data, adjustment data, and an asset +database. Bundles allow us to preload all of the data we will need to run +backtests and store the data for future runs. + +Ingesting Data +~~~~~~~~~~~~~~ + +The first step to using a data bundle is to ingest the data. This will invoke +some custom bundle command and then write the data to a standard location that +zipline can find. By default this location is ``$ZIPLINE_ROOT/data/`` +where by default ``ZIPLINE_ROOT=~/.zipline``. This step may take some time as it +could involve downloading and processing a lot of data. This can be run with: + +.. code-block:: bash + + $ python -m zipline ingest + + +where ```` is the name of the bundle to ingest. + +Old Data +~~~~~~~~ + +When the ``ingest`` command is used it will write the new data to a subdirectory +of ``$ZIPLINE_ROOT/data/`` which is named with the current date. This +makes it possible to look at older data or even run backtests with this older +copy. This makers it easier to reproduce backtest results later. + +One drawback of saving all of this data by default is that the data directory +may grow quite large even if you do not want to use the data. To solve this +problem there is another command ``clean`` which will clear data bundles based +on some time constraints. + +For example: + +.. code-block:: bash + + # clean everything older than + $ python -m zipline clean --before + + # clean everything newer than + $ python -m zipline clean --after + + # keep everything in the range of [before, after] and delete the rest + $ python -m zipline clean --before --after + + # clean all but the last runs + $ python -m zipline clean --keep-last + + +Running Backtests with Data Bundles +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Now that the data has been ingested we can use it to run backtests with the +``run`` command. This can be specified with the ``--bundle`` option like: + +.. code-block:: bash + + $ python -m zipline run --bundle --algofile algo.py ... + + +We may also specify the date to use to look up the bundle data with the +``--bundle-date`` option. This will cause us to the the most recent bundle +ingestion that is less than or equal to the ``bundle-date``. This is how we can +run backtests with older data. The reason that this uses a less than or equal to +relationship is that we can specify the date that we ran an old backtest and get +the same data that would have been available to us on that date. The +``bundle-date`` defaults to the current day to use the most recent data. + +Default Data Bundles +~~~~~~~~~~~~~~~~~~~~ + +.. _quandl-data-bundle: + +Quandl WIKI Bundle +`````````````````` + +By default zipline comes with the ``quandl`` data bundle which uses quandl's +`WIKI dataset `_. The quandl data bundle +includes daily pricing data, splits, cash dividends, and asset metadata. This is +the bundle that ``run`` will use by default if no other bundle is specified. To +ingest this data bundle we recommend creating an account on quandl.com to get an +API key to be able to make more API requests per day. Once we have an API key we +may run: + +.. code-block:: bash + + $ QUANDL_API_KEY= python -m zipline ingest quandl + +though we may still run ``ingest`` as an anonymous quandl user (with no API +key). We may also set the ``QUANDL_DOWNLOAD_ATTEMPTS`` environment variable to +an integer which is the number of attempts that should be made to download data +from quandls servers. By default this will be 5, meaning that we will retry each +attempt 5 times. + +.. note:: + + ``QUANDL_DOWNLOAD_ATTEMPTS`` is not the total number of allowed failures, + just the number of allowed failures per request. The quandl loader will make + one request per 100 equities for the metadata followed by one request per + equity. + + +Yahoo Bundle Factories +`````````````````````` + +Zipline also ships with a factory function for creating a data bundle out of a +set of tickers from yahoo: :func:`~zipline.data.bundles.yahoo_equities`. +This makes it easy to pre-download and cache the data for a set of equities from +yahoo. This includes daily pricing data along with splits, cash dividends, and +inferred asset metadata. To create a bundle from a set of equities, add the +following to your ``~/.zipline/extensions.py`` file: + +.. code-block:: python + + from zipline.bundles import register, yahoo_equities + + # these are the tickers you would like data for + equities = { + 'AAPL', + 'MSFT', + 'GOOG', + } + register( + 'my-yahoo-equities-bundle', # name this whatever you like + yahoo_equities(equities), + ) + + +This may now be used like: + +.. code-block:: bash + + $ python -m zipline ingest my-yahoo-equities-bundle + $ python -m zipline run -f algo.py --bundle my-yahoo-equities-bundle + + +More than one yahoo equities bundle may be registered as long as they use +different names. + +Writing a New Bundle +~~~~~~~~~~~~~~~~~~~~ + +Data bundles exist to make it easy to use different data sources with +zipline. To add a new bundle, one must implement an ingest function. + +This function is responsible for loading the data into memory and passing it to +a set of writer objects provided by zipline to convert the data to zipline's +internal format. The ingest function may work by downloading data from a remote +location like the ``quandl`` bundle or yahoo bundles or it may just load files +that are already on the machine. The function is provided with writers that will +write the data to the correct location transactionally. If an ingestion fails +part way through the bundle will not be written in an incomplete state. + +The signature of the ingest function should be: + +.. code-block:: python + + ingest(environ, + asset_db_writer, + minute_bar_writer, + daily_bar_writer, + adjustment_writer, + calendar, + cache, + show_progress) + +``environ`` +``````````` + +``environ`` is a mapping representing the environment variables to use. This is +where any custom arguments needed for the ingestion should be passed, for +example: the ``quandl`` bundle uses the enviornment to pass the API key and the +download retry attempt count. + +``asset_db_writer`` +``````````````````` + +``asset_db_writer`` is an instance of :class:`~zipline.assets.AssetDBWriter`. +This is the writer for the asset metadata which provides the asset lifetimes and +the symbol to asset id (sid) mapping. This may also contain the asset name, +exchange and a few other columns. To write data, invoke +:meth:`~zipline.assets.AssetDBWriter.write` with dataframes for the various +pieces of metadata. More information about the format of the data exists in the +docs for write. + +``minute_bar_writer`` +````````````````````` + +``minute_bar_writer`` is an instance of +:class:`~zipline.data.minute_bars.BcolzMinuteBarWriter`. This writer is used to +convert data to zipline's internal bcolz format to later be read by a +:class:`~zipline.data.minute_bars.BcolzMinuteBarReader`. If minute data is +provided, users should call +:meth:`~zipline.data.minute_bars.BcolzMinuteBarWriter.write` with an iterable of +(sid, dataframe) tuples. The ``show_progress`` argument should also be forwarded +to this method. If the data source does not provide minute level data, then +there is no need to call the write method. It is also acceptable to pass an +empty iterator to :meth:`~zipline.data.minute_bars.BcolzMinuteBarWriter.write` +to signal that there is no minutely data. + +.. note:: + + The data passed to + :meth:`~zipline.data.minute_bars.BcolzMinuteBarWriter.write` may be a lazy + iterator or generator to avoid loading all of the minute data into memory at + a single time. A given sid may also appear multiple times in the data as long + as the dates are strictly increasing. + +``daily_bar_writer`` +```````````````````` + +``daily_bar_writer`` is an instance of +:class:`~zipline.data.us_equity_pricing.BcolzDailyBarWriter`. This writer is +used to convert data into zipline's internal bcolz format to later be read by a +:class:`~zipline.data.us_equity_pricing.BcolzDailyBarReader`. If daily data is +provided, users should call +:meth:`~zipline.data.minute_bars.BcolzDailyBarWriter.write` with an iterable of +(sid dataframe) tuples. The ``show_progress`` argument should also be forwarded +to this method. If the data shource does not provide daily data, then there is +no need to call the write method. It is also acceptable to pass an empty +iterable to :meth:`~zipline.data.minute_bars.BcolzMinuteBarWriter.write` to +signal that there is no daily data. If no daily data is provided but minute data +is provided, a daily rollup will happen to service daily history requests. + +.. note:: + + Like the ``minute_bar_writer``, the data passed to + :meth:`~zipline.data.minute_bars.BcolzMinuteBarWriter.write` may be a lazy + iterable or generator to avoid loading all of the data into memory at once. + Unlike the ``minute_bar_writer``, a sid may only appear once in the data + iterable. + +``adjustment_writer`` +````````````````````` + +``adjustment_writer`` is an instance of +:class:`~zipline.data.us_equity_pricing.SQLiteAdjustmentWriter`. This writer is +used to store splits, mergers, dividends, and stock dividends. The data should +be provided as dataframes and passed to +:meth:`~zipline.data.us_equity_pricing.SQLiteAdjustmentWriter.write`. Each of +these fields are optional, but the writer can accept as much of the data as you +have. + +``calendar`` +```````````` + +``calendar`` is a ``pandas.DatetimeIndex`` object holding all of the trading +days that the bundle should load data for. This is to help some bundles generate +queries for the days needed. + +``cache`` +````````` + +``cache`` is an instance of :class:`~zipline.utils.cache.dataframe_cache`. This +object is a mapping from strings to dataframes. This object is provided in case +an ingestion crashes part way through. The idea is that the ingest function +should check the cache for raw data, if it doesn't exist in the cache, it should +acquire it and then store it in the cache. Then it can parse and write the +data. The cache will be cleared only after a successful load, this prevents the +ingest function from needing to redownload all the data if there is some bug in +the parsing. If it is very fast to get the data, for example if it is coming +from another local file, then there is no need to use this cache. + +``show_progress`` +````````````````` + +``show_progress`` is a boolean indicating that the user would like to receive +feedback about the ingest function's progress fetching and writing the +data. Some examples for where to show how many files you have downloaded out of +the total needed, or how far into some data conversion the ingest function +is. One tool that may help with implementing ``show_progress`` for a loop is +:class:`~zipline.utils.cli.maybe_show_progress`. This argument should always be +forwarded to ``minute_bar_writer.write`` and ``daily_bar_writer.write``. diff --git a/docs/source/index.rst b/docs/source/index.rst index 9dad6958..b8f24239 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -5,6 +5,7 @@ install beginner-tutorial + bundles releases appendix release-process diff --git a/docs/source/whatsnew/1.0.0.txt b/docs/source/whatsnew/1.0.0.txt index 5d9b581b..3a11be8a 100644 --- a/docs/source/whatsnew/1.0.0.txt +++ b/docs/source/whatsnew/1.0.0.txt @@ -12,7 +12,51 @@ Development Highlights ~~~~~~~~~~ -None +New Entry Points (:issue:`xxxx`) +```````````````````````````````` + +In order to make it easier to use zipline we have updated the entry points for +a backtest. The three supported ways to run a backtest are now: + +1. :func:`zipline.run_algo` +2. ``$ python -m zipline run`` +3. ``%zipline`` (IPython magic) + +Data Bundles (:issue:`xxxx`) +```````````````````````````` + +1.0.0 introduces data bundles. Data bundles are groups of data that should be +preloaded and used to run backtests later. This allows users to not need to to +specify which tickers they are interested in each time they run an +algorithm. This also allows us to cache the data between runs. + +By default, the ``quandl`` bundle will be used which pulls data from quandl's +`WIKI dataset `_. New bundles may be +registered with :func:`zipline.data.bundles.register` like: + +.. code-block:: python + + @zipline.data.bundles.register('my-new-bundle') + def my_new_bundle_ingest(environ, + asset_db_writer, + minute_bar_writer, + daily_bar_writer, + adjustment_writer, + calendar, + cache, + show_progress): + ... + + +This function should retrieve the data it needs and then use the writers that +have been passed to write that data to disc in a location that zipline can find +later. + +This data can be used in backtests by passing the name as the ``-b / --bundle`` +argument to ``$ python -m zipline run`` or as the ``bundle`` argument to +:func:`zipline.run_algo`. + +For more information see `Data Bundles`_ for more information. Enhancements ~~~~~~~~~~~~ diff --git a/etc/requirements.txt b/etc/requirements.txt index a211f690..4875e914 100644 --- a/etc/requirements.txt +++ b/etc/requirements.txt @@ -15,6 +15,7 @@ numpy==1.9.2 # statsmodels in turn is required for some pandas packages scipy==0.15.1 pandas==0.16.1 +pandas-datareader==0.2.1 # Needed for parts of pandas.stats patsy==0.4.0 statsmodels==0.6.1 diff --git a/scripts/run_algo.py b/scripts/run_algo.py deleted file mode 100755 index de657a77..00000000 --- a/scripts/run_algo.py +++ /dev/null @@ -1,26 +0,0 @@ -#!/usr/bin/env python -# -# Copyright 2014 Quantopian, Inc. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -import logbook -import sys - -from zipline.utils import parse_args, run_pipeline - -if __name__ == "__main__": - logbook.StderrHandler().push_application() - parsed = parse_args(sys.argv[1:]) - run_pipeline(**parsed) - sys.exit(0) diff --git a/setup.py b/setup.py index c7e5594d..78063820 100644 --- a/setup.py +++ b/setup.py @@ -264,7 +264,6 @@ setup( author_email='opensource@quantopian.com', packages=find_packages('.', include=['zipline', 'zipline.*']), ext_modules=ext_modules, - scripts=['scripts/run_algo.py'], include_package_data=True, license='Apache 2.0', classifiers=[ diff --git a/tests/data/bundles/test_core.py b/tests/data/bundles/test_core.py new file mode 100644 index 00000000..7e61ac0d --- /dev/null +++ b/tests/data/bundles/test_core.py @@ -0,0 +1,218 @@ +import pandas as pd +from toolz import valmap +import toolz.curried.operator as op + +from zipline.assets.synthetic import make_simple_equity_info +from zipline.data.bundles import load +from zipline.data.bundles.core import _make_bundle_core +from zipline.lib.adjustment import Float64Multiply +from zipline.pipeline.loaders.synthetic import ( + make_bar_data, + expected_bar_values_2d, +) +from zipline.testing import ( + subtest, + tmp_dir, + str_to_seconds, + tmp_trading_env, +) +from zipline.testing.fixtures import ZiplineTestCase +from zipline.testing.predicates import ( + assert_equal, + assert_false, + assert_in, + assert_is, + assert_is_instance, +) +from zipline.utils.cache import dataframe_cache +from zipline.utils.functional import apply +from zipline.utils.tradingcalendar import trading_days + + +class BundleCoreTestCase(ZiplineTestCase): + def init_instance_fixtures(self): + super(BundleCoreTestCase, self).init_instance_fixtures() + (self.bundles, + self.register, + self.unregister, + self.ingest) = _make_bundle_core() + + def test_register_decorator(self): + @apply + @subtest(((c,) for c in 'abcde'), 'name') + def _(name): + @self.register(name) + def ingest(*args): + pass + + assert_in(name, self.bundles) + assert_is(self.bundles[name].ingest, ingest) + + self._check_bundles(set('abcde')) + + def test_register_call(self): + def ingest(*args): + pass + + @apply + @subtest(((c,) for c in 'abcde'), 'name') + def _(name): + self.register(name, ingest) + assert_in(name, self.bundles) + assert_is(self.bundles[name].ingest, ingest) + + assert_equal( + valmap(op.attrgetter('ingest'), self.bundles), + {k: ingest for k in 'abcde'}, + ) + self._check_bundles(set('abcde')) + + def _check_bundles(self, names): + assert_equal(set(self.bundles.keys()), names) + + for name in names: + self.unregister(name) + + assert_false(self.bundles) + + def test_ingest(self): + zipline_root = self.enter_instance_context(tmp_dir()).path + env = self.enter_instance_context(tmp_trading_env()) + + start = pd.Timestamp('2014-01-06', tz='utc') + end = pd.Timestamp('2014-01-10', tz='utc') + calendar = trading_days[trading_days.slice_indexer(start, end)] + minutes = env.minutes_for_days_in_range(calendar[0], calendar[-1]) + outer_environ = { + 'ZIPLINE_ROOT': zipline_root, + } + + sids = tuple(range(3)) + equities = make_simple_equity_info( + sids, + calendar[0], + calendar[-1], + ) + + daily_bar_data = make_bar_data(equities, calendar) + minute_bar_data = make_bar_data(equities, minutes) + first_split_ratio = 0.5 + second_split_ratio = 0.1 + splits = pd.DataFrame.from_records([ + { + 'effective_date': str_to_seconds('2014-01-08'), + 'ratio': first_split_ratio, + 'sid': 0, + }, + { + 'effective_date': str_to_seconds('2014-01-09'), + 'ratio': second_split_ratio, + 'sid': 1, + }, + ]) + + @self.register('bundle', + calendar=calendar, + opens=env.opens_in_range(calendar[0], calendar[-1]), + closes=env.closes_in_range(calendar[0], calendar[-1])) + def bundle_ingest(environ, + asset_db_writer, + minute_bar_writer, + daily_bar_writer, + adjustment_writer, + calendar, + cache, + show_progress): + assert_is(environ, outer_environ) + + asset_db_writer.write(equities=equities) + minute_bar_writer.write(minute_bar_data) + daily_bar_writer.write(daily_bar_data) + adjustment_writer.write(splits=splits) + + assert_is_instance(calendar, pd.DatetimeIndex) + assert_is_instance(cache, dataframe_cache) + assert_is_instance(show_progress, bool) + + self.ingest('bundle', environ=outer_environ) + bundle = load('bundle', environ=outer_environ) + + assert_equal(set(bundle.asset_finder.sids), set(sids)) + + columns = 'open', 'high', 'low', 'close', 'volume' + + actual = bundle.minute_bar_reader.load_raw_arrays( + columns, + minutes[0], + minutes[-1], + sids, + ) + + for actual_column, colname in zip(actual, columns): + assert_equal( + actual_column, + expected_bar_values_2d(minutes, equities, colname), + msg=colname, + ) + + actual = bundle.daily_bar_reader.load_raw_arrays( + columns, + calendar[0], + calendar[-1], + sids, + ) + for actual_column, colname in zip(actual, columns): + assert_equal( + actual_column, + expected_bar_values_2d(calendar, equities, colname), + msg=colname, + ) + adjustments_for_cols = bundle.adjustment_reader.load_adjustments( + columns, + calendar, + pd.Index(sids), + ) + for column, adjustments in zip(columns, adjustments_for_cols[:-1]): + # iterate over all the adjustments but `volume` + assert_equal( + adjustments, + { + 2: [Float64Multiply( + first_row=0, + last_row=2, + first_col=0, + last_col=0, + value=first_split_ratio, + )], + 3: [Float64Multiply( + first_row=0, + last_row=3, + first_col=1, + last_col=1, + value=second_split_ratio, + )], + }, + msg=column, + ) + + # check the volume, the value should be 1/ratio + assert_equal( + adjustments_for_cols[-1], + { + 2: [Float64Multiply( + first_row=0, + last_row=2, + first_col=0, + last_col=0, + value=1 / first_split_ratio, + )], + 3: [Float64Multiply( + first_row=0, + last_row=3, + first_col=1, + last_col=1, + value=1 / second_split_ratio, + )], + }, + msg='volume', + ) diff --git a/tests/data/bundles/test_quandl.py b/tests/data/bundles/test_quandl.py new file mode 100644 index 00000000..31627ae3 --- /dev/null +++ b/tests/data/bundles/test_quandl.py @@ -0,0 +1,243 @@ +from __future__ import division + +import numpy as np +import pandas as pd +from toolz import merge +import toolz.curried.operator as op + +from zipline.data.bundles import ingest, load, bundles +from zipline.data.bundles.quandl import ( + format_wiki_url, + format_metadata_url, +) +from zipline.lib.adjustment import Float64Multiply +from zipline.testing import ( + test_resource_path, + tmp_dir, + patch_read_csv, +) +from zipline.testing.fixtures import ZiplineTestCase +from zipline.testing.predicates import ( + assert_equal, +) +from zipline.utils.functional import apply + + +class QuandlBundleTestCase(ZiplineTestCase): + symbols = 'AAPL', 'BRK_A', 'MSFT', 'ZEN' + asset_start = pd.Timestamp('2014-01', tz='utc') + asset_end = pd.Timestamp('2015-01', tz='utc') + calendar = bundles['quandl'].calendar + start_date = calendar[0] + end_date = calendar[-1] + api_key = 'ayylmao' + columns = 'open', 'high', 'low', 'close', 'volume' + + def _expected_data(self, asset_finder): + sids = { + symbol: asset_finder.lookup_symbol( + symbol, + self.asset_start, + ).sid + for symbol in self.symbols + } + + def per_symbol(symbol): + df = pd.read_csv( + test_resource_path('quandl_samples', symbol + '.csv.gz'), + parse_dates=['Date'], + index_col='Date', + usecols=[ + 'Open', + 'High', + 'Low', + 'Close', + 'Volume', + 'Date', + 'Ex-Dividend', + 'Split Ratio', + ], + na_values=['NA'], + ).rename(columns={ + 'Open': 'open', + 'High': 'high', + 'Low': 'low', + 'Close': 'close', + 'Volume': 'volume', + 'Date': 'date', + 'Ex-Dividend': 'ex_dividend', + 'Split Ratio': 'split_ratio', + }) + df['sid'] = sids[symbol] + return df + + all_ = pd.concat(map(per_symbol, self.symbols)).set_index( + 'sid', + append=True, + ).unstack() + + # fancy list comprehension with statements + @list + @apply + def pricing(): + for column in self.columns: + vs = all_[column].values + if column == 'volume': + vs = np.nan_to_num(vs) + yield vs + + # the first index our written data will appear in the files on disk + start_idx = self.calendar.get_loc(self.asset_start, 'ffill') + 1 + + # convert an index into the raw dataframe into an index into the + # final data + i = op.add(start_idx) + + def expected_dividend_adjustment(idx, symbol): + sid = sids[symbol] + return ( + 1 - + all_.ix[idx, ('ex_dividend', sid)] / + all_.ix[idx - 1, ('close', sid)] + ) + + adjustments = [ + # ohlc + { + # dividends + i(24): [Float64Multiply( + first_row=0, + last_row=i(24), + first_col=sids['AAPL'], + last_col=sids['AAPL'], + value=expected_dividend_adjustment(24, 'AAPL'), + )], + i(87): [Float64Multiply( + first_row=0, + last_row=i(87), + first_col=sids['AAPL'], + last_col=sids['AAPL'], + value=expected_dividend_adjustment(87, 'AAPL'), + )], + i(150): [Float64Multiply( + first_row=0, + last_row=i(150), + first_col=sids['AAPL'], + last_col=sids['AAPL'], + value=expected_dividend_adjustment(150, 'AAPL'), + )], + i(214): [Float64Multiply( + first_row=0, + last_row=i(214), + first_col=sids['AAPL'], + last_col=sids['AAPL'], + value=expected_dividend_adjustment(214, 'AAPL'), + )], + + i(31): [Float64Multiply( + first_row=0, + last_row=i(31), + first_col=sids['MSFT'], + last_col=sids['MSFT'], + value=expected_dividend_adjustment(31, 'MSFT'), + )], + i(90): [Float64Multiply( + first_row=0, + last_row=i(90), + first_col=sids['MSFT'], + last_col=sids['MSFT'], + value=expected_dividend_adjustment(90, 'MSFT'), + )], + i(222): [Float64Multiply( + first_row=0, + last_row=i(222), + first_col=sids['MSFT'], + last_col=sids['MSFT'], + value=expected_dividend_adjustment(222, 'MSFT'), + )], + + # splits + i(108): [Float64Multiply( + first_row=0, + last_row=i(108), + first_col=sids['AAPL'], + last_col=sids['AAPL'], + value=7.0, + )], + }, + ] * (len(self.columns) - 1) + [ + # volume + { + i(108): [Float64Multiply( + first_row=0, + last_row=i(108), + first_col=sids['AAPL'], + last_col=sids['AAPL'], + value=1 / 7, + )], + } + ] + return pricing, adjustments + + def test_bundle(self): + url_map = merge( + { + format_wiki_url( + self.api_key, + symbol, + self.start_date, + self.end_date, + ): test_resource_path('quandl_samples', symbol + '.csv.gz') + for symbol in self.symbols + }, + { + format_metadata_url(self.api_key, n): test_resource_path( + 'quandl_samples', + 'metadata-%d.csv.gz' % n, + ) + for n in (1, 2) + }, + ) + zipline_root = self.enter_instance_context(tmp_dir()).path + environ = { + 'ZIPLINE_ROOT': zipline_root, + 'QUANDL_API_KEY': self.api_key, + } + + with patch_read_csv(url_map, strict=True): + ingest('quandl', environ=environ) + + bundle = load('quandl', environ=environ) + sids = 0, 1, 2, 3 + assert_equal(set(bundle.asset_finder.sids), set(sids)) + + for equity in bundle.asset_finder.retrieve_all(sids): + assert_equal(equity.start_date, self.asset_start, msg=equity) + assert_equal(equity.end_date, self.asset_end, msg=equity) + + cal = self.calendar + actual = bundle.daily_bar_reader.load_raw_arrays( + self.columns, + cal[cal.get_loc(self.asset_start, 'bfill')], + cal[cal.get_loc(self.asset_end, 'ffill')], + sids, + ) + expected_pricing, expected_adjustments = self._expected_data( + bundle.asset_finder, + ) + assert_equal(actual, expected_pricing, array_decimal=2) + + adjustments_for_cols = bundle.adjustment_reader.load_adjustments( + self.columns, + cal, + pd.Index(sids), + ) + + for column, adjustments, expected in zip(self.columns, + adjustments_for_cols, + expected_adjustments): + assert_equal( + adjustments, + expected, + msg=column, + ) diff --git a/tests/data/bundles/test_yahoo.py b/tests/data/bundles/test_yahoo.py new file mode 100644 index 00000000..03bf4ad9 --- /dev/null +++ b/tests/data/bundles/test_yahoo.py @@ -0,0 +1,201 @@ +import numpy as np +import pandas as pd +from six.moves.urllib.parse import urlparse, parse_qs +from toolz import flip, identity +from toolz.curried import merge_with, operator as op + +from zipline.data.bundles.core import _make_bundle_core +from zipline.data.bundles import yahoo_equities, load +from zipline.lib.adjustment import Float64Multiply +from zipline.testing import test_resource_path, tmp_dir, read_compressed +from zipline.testing.fixtures import WithResponses, ZiplineTestCase +from zipline.testing.predicates import assert_equal +from zipline.utils.tradingcalendar import trading_days + + +class YahooBundleTestCase(WithResponses, ZiplineTestCase): + symbols = 'AAPL', 'IBM', 'MSFT' + columns = 'open', 'high', 'low', 'close', 'volume' + asset_start = pd.Timestamp('2014-01-02', tz='utc') + asset_end = pd.Timestamp('2014-12-31', tz='utc') + calendar = trading_days[ + (trading_days >= asset_start) & + (trading_days <= asset_end) + ] + + @classmethod + def init_class_fixtures(cls): + super(YahooBundleTestCase, cls).init_class_fixtures() + (cls.bundles, + cls.register, + cls.unregister, + cls.ingest) = map(staticmethod, _make_bundle_core()) + + def _expected_data(self): + sids = 0, 1, 2 + modifier = { + 'low': 0, + 'open': 1, + 'close': 2, + 'high': 3, + 'volume': 0, + } + pricing = [ + np.hstack(( + np.arange(252, dtype='float64')[:, np.newaxis] + + 1 + + sid * 10000 + + modifier[column] * 1000 + for sid in sorted(sids) + )) + for column in self.columns + ] + + # There are two dividends and 1 split for each company. + + def dividend_adjustment(sid, which): + """The dividends occur at indices 252 // 4 and 3 * 252 / 4 + with a cash amount of sid + 1 / 10 and sid + 2 / 10 + """ + if which == 'first': + idx = 252 // 4 + else: + idx = 3 * 252 // 4 + + return { + idx: [Float64Multiply( + first_row=0, + last_row=idx, + first_col=sid, + last_col=sid, + value=float( + 1 - + ((sid + 1 + (which == 'second')) / 10) / + (idx - 1 + sid * 10000 + 2000) + ), + )], + } + + def split_adjustment(sid, volume): + """The splits occur at index 252 // 2 with a ratio of (sid + 1):1 + """ + idx = 252 // 2 + return { + idx: [Float64Multiply( + first_row=0, + last_row=idx, + first_col=sid, + last_col=sid, + value=(identity if volume else op.truediv(1))(sid + 2), + )], + } + + merge_adjustments = merge_with(flip(sum, [])) + + adjustments = [ + # ohlc + merge_adjustments( + *tuple(dividend_adjustment(sid, 'first') for sid in sids) + + tuple(dividend_adjustment(sid, 'second') for sid in sids) + + tuple(split_adjustment(sid, volume=False) for sid in sids) + ) + ] * (len(self.columns) - 1) + [ + # volume + merge_adjustments( + split_adjustment(sid, volume=True) for sid in sids + ), + ] + + return pricing, adjustments + + def test_bundle(self): + + def get_symbol_from_url(url): + params = parse_qs(urlparse(url).query) + symbol, = params['s'] + return symbol + + def pricing_callback(request): + headers = { + 'content-encoding': 'gzip', + 'content-type': 'text/csv', + } + path = test_resource_path( + 'yahoo_samples', + get_symbol_from_url(request.url) + '.csv.gz', + ) + with open(path, 'rb') as f: + return ( + 200, + headers, + f.read(), + ) + + for _ in range(3): + self.responses.add_callback( + self.responses.GET, + 'http://ichart.finance.yahoo.com/table.csv', + pricing_callback, + ) + + def adjustments_callback(request): + path = test_resource_path( + 'yahoo_samples', + get_symbol_from_url(request.url) + '.adjustments.gz', + ) + return 200, {}, read_compressed(path) + + for _ in range(3): + self.responses.add_callback( + self.responses.GET, + 'http://ichart.finance.yahoo.com/x', + adjustments_callback, + ) + + cal = self.calendar + self.register( + 'bundle', + yahoo_equities(self.symbols), + calendar=cal, + ) + + zipline_root = self.enter_instance_context(tmp_dir()).path + environ = { + 'ZIPLINE_ROOT': zipline_root, + } + + self.ingest('bundle', environ=environ) + bundle = load('bundle', environ=environ) + + sids = 0, 1, 2 + equities = bundle.asset_finder.retrieve_all(sids) + for equity, expected_symbol in zip(equities, self.symbols): + assert_equal(equity.symbol, expected_symbol) + + for equity in bundle.asset_finder.retrieve_all(sids): + assert_equal(equity.start_date, self.asset_start, msg=equity) + assert_equal(equity.end_date, self.asset_end, msg=equity) + + actual = bundle.daily_bar_reader.load_raw_arrays( + self.columns, + cal[cal.get_loc(self.asset_start, 'bfill')], + cal[cal.get_loc(self.asset_end, 'ffill')], + sids, + ) + expected_pricing, expected_adjustments = self._expected_data() + assert_equal(actual, expected_pricing, array_decimal=2) + + adjustments_for_cols = bundle.adjustment_reader.load_adjustments( + self.columns, + cal, + pd.Index(sids), + ) + + for column, adjustments, expected in zip(self.columns, + adjustments_for_cols, + expected_adjustments): + assert_equal( + adjustments, + expected, + msg=column, + ) diff --git a/tests/data/test_minute_bars.py b/tests/data/test_minute_bars.py index 9abdffbb..edddf060 100644 --- a/tests/data/test_minute_bars.py +++ b/tests/data/test_minute_bars.py @@ -24,6 +24,7 @@ from numpy import ( float64, full, nan, + transpose, zeros, ) from numpy.testing import assert_almost_equal, assert_array_equal @@ -493,8 +494,9 @@ class BcolzMinuteBarTestCase(TestCase): fields = ['open', 'high', 'low', 'close', 'volume'] - ohlcv_window = self.reader.unadjusted_window( - fields, minutes[0], minutes[-1], [sid]) + ohlcv_window = list(map(transpose, self.reader.load_raw_arrays( + fields, minutes[0], minutes[-1], [sid], + ))) for i, field in enumerate(fields): if field != 'volume': @@ -535,8 +537,9 @@ class BcolzMinuteBarTestCase(TestCase): fields = ['open', 'high', 'low', 'close', 'volume'] - ohlcv_window = self.reader.unadjusted_window( - fields, minutes[0], minutes[-1], [sid]) + ohlcv_window = list(map(transpose, self.reader.load_raw_arrays( + fields, minutes[0], minutes[-1], [sid], + ))) for i, field in enumerate(fields): if field != 'volume': @@ -647,8 +650,9 @@ class BcolzMinuteBarTestCase(TestCase): columns = ['open', 'high', 'low', 'close', 'volume'] sids = [sids[0], sids[1]] - arrays = reader.unadjusted_window( - columns, minutes[0], minutes[-1], sids) + arrays = list(map(transpose, reader.load_raw_arrays( + columns, minutes[0], minutes[-1], sids, + ))) data = {sids[0]: data_1, sids[1]: data_2} @@ -698,8 +702,9 @@ class BcolzMinuteBarTestCase(TestCase): columns = ['open', 'high', 'low', 'close', 'volume'] sids = [sids[0], sids[1]] - arrays = reader.unadjusted_window( - columns, minutes[0], minutes[-1], sids) + arrays = list(map(transpose, reader.load_raw_arrays( + columns, minutes[0], minutes[-1], sids, + ))) data = {sids[0]: data_1, sids[1]: data_2} diff --git a/tests/data/test_us_equity_pricing.py b/tests/data/test_us_equity_pricing.py index 9391809b..261c2dc0 100644 --- a/tests/data/test_us_equity_pricing.py +++ b/tests/data/test_us_equity_pricing.py @@ -33,14 +33,13 @@ from zipline.data.us_equity_pricing import ( BcolzDailyBarReader, NoDataOnDate, ) -from zipline.pipeline.data import USEquityPricing from zipline.pipeline.loaders.synthetic import ( OHLCV, asset_start, asset_end, - expected_daily_bar_value, - expected_daily_bar_values_2d, - make_daily_bar_data, + expected_bar_value, + expected_bar_values_2d, + make_bar_data, ) from zipline.testing import seconds_to_timestamp from zipline.testing.fixtures import ( @@ -89,7 +88,7 @@ class BcolzDailyBarTestCase(WithBcolzDailyBarReader, ZiplineTestCase): @classmethod def make_daily_bar_data(cls): - return make_daily_bar_data( + return make_bar_data( EQUITY_INFO, cls.bcolz_daily_bar_days, ) @@ -129,7 +128,7 @@ class BcolzDailyBarTestCase(WithBcolzDailyBarReader, ZiplineTestCase): for asset_id in self.assets: for date in self.dates_for_asset(asset_id): self.assertEqual( - expected_daily_bar_value( + expected_bar_value( asset_id, date, column @@ -198,18 +197,18 @@ class BcolzDailyBarTestCase(WithBcolzDailyBarReader, ZiplineTestCase): for column, result in zip(columns, results): assert_array_equal( result, - expected_daily_bar_values_2d( + expected_bar_values_2d( dates, EQUITY_INFO, - column.name, + column, ) ) @parameterized.expand([ - ([USEquityPricing.open],), - ([USEquityPricing.close, USEquityPricing.volume],), - ([USEquityPricing.volume, USEquityPricing.high, USEquityPricing.low],), - (USEquityPricing.columns,), + (['open'],), + (['close', 'volume'],), + (['volume', 'high', 'low'],), + (['open', 'high', 'low', 'close', 'volume'],), ]) def test_read(self, columns): self._check_read_results( @@ -224,7 +223,7 @@ class BcolzDailyBarTestCase(WithBcolzDailyBarReader, ZiplineTestCase): Test loading with queries that starts on the first day of each asset's lifetime. """ - columns = [USEquityPricing.high, USEquityPricing.volume] + columns = ['high', 'volume'] for asset in self.assets: self._check_read_results( columns, @@ -238,7 +237,7 @@ class BcolzDailyBarTestCase(WithBcolzDailyBarReader, ZiplineTestCase): Test loading with queries that start on the last day of each asset's lifetime. """ - columns = [USEquityPricing.close, USEquityPricing.volume] + columns = ['close', 'volume'] for asset in self.assets: self._check_read_results( columns, @@ -252,7 +251,7 @@ class BcolzDailyBarTestCase(WithBcolzDailyBarReader, ZiplineTestCase): Test loading with queries that end on the first day of each asset's lifetime. """ - columns = [USEquityPricing.close, USEquityPricing.volume] + columns = ['close', 'volume'] for asset in self.assets: self._check_read_results( columns, @@ -266,7 +265,7 @@ class BcolzDailyBarTestCase(WithBcolzDailyBarReader, ZiplineTestCase): Test loading with queries that end on the last day of each asset's lifetime. """ - columns = [USEquityPricing.close, USEquityPricing.volume] + columns = ['close', 'volume'] for asset in self.assets: self._check_read_results( columns, diff --git a/tests/pipeline/test_engine.py b/tests/pipeline/test_engine.py index 8eba9dfb..f00d349a 100644 --- a/tests/pipeline/test_engine.py +++ b/tests/pipeline/test_engine.py @@ -40,8 +40,17 @@ from toolz import merge from zipline.assets.synthetic import make_rotating_equity_info from zipline.lib.adjustment import MULTIPLY -from zipline.pipeline import CustomFactor, Pipeline -from zipline.pipeline.data import Column, DataSet, USEquityPricing +from zipline.pipeline.loaders.synthetic import PrecomputedLoader +from zipline.pipeline import Pipeline +from zipline.pipeline.data import USEquityPricing, DataSet, Column +from zipline.pipeline.loaders.equity_pricing_loader import ( + USEquityPricingLoader, +) +from zipline.pipeline.factors import CustomFactor +from zipline.pipeline.loaders.synthetic import ( + make_bar_data, + expected_bar_values_2d, +) from zipline.pipeline.engine import SimplePipelineEngine from zipline.pipeline.factors import ( AverageDollarVolume, @@ -52,15 +61,7 @@ from zipline.pipeline.factors import ( MaxDrawdown, SimpleMovingAverage, ) -from zipline.pipeline.loaders.equity_pricing_loader import ( - USEquityPricingLoader, -) from zipline.pipeline.loaders.frame import DataFrameLoader -from zipline.pipeline.loaders.synthetic import ( - expected_daily_bar_values_2d, - make_daily_bar_data, - PrecomputedLoader, -) from zipline.pipeline.term import NotSpecified from zipline.testing import ( product_upper_triangle, @@ -925,7 +926,7 @@ class SyntheticBcolzTestCase(WithAdjustmentReader, @classmethod def make_daily_bar_data(cls): - return make_daily_bar_data( + return make_bar_data( cls.equity_info, cls.bcolz_daily_bar_days, ) @@ -999,7 +1000,7 @@ class SyntheticBcolzTestCase(WithAdjustmentReader, # computed results to be computed using values anchored on the # **previous** day's data. expected_raw = rolling_mean( - expected_daily_bar_values_2d( + expected_bar_values_2d( dates - self.env.trading_day, self.equity_info, 'close', diff --git a/tests/pipeline/test_us_equity_pricing_loader.py b/tests/pipeline/test_us_equity_pricing_loader.py index 12a44f95..1b319235 100644 --- a/tests/pipeline/test_us_equity_pricing_loader.py +++ b/tests/pipeline/test_us_equity_pricing_loader.py @@ -37,8 +37,8 @@ from toolz.curried.operator import getitem from zipline.lib.adjustment import Float64Multiply from zipline.pipeline.loaders.synthetic import ( NullAdjustmentReader, - make_daily_bar_data, - expected_daily_bar_values_2d, + make_bar_data, + expected_bar_values_2d, ) from zipline.pipeline.loaders.equity_pricing_loader import ( USEquityPricingLoader, @@ -282,7 +282,7 @@ class USEquityPricingLoaderTestCase(WithAdjustmentReader, @classmethod def make_daily_bar_data(cls): - return make_daily_bar_data( + return make_bar_data( EQUITY_INFO, cls.bcolz_daily_bar_days, ) @@ -364,7 +364,7 @@ class USEquityPricingLoaderTestCase(WithAdjustmentReader, ) adjustments = self.adjustment_reader.load_adjustments( - columns, + [c.name for c in columns], query_days, self.assets, ) @@ -410,7 +410,7 @@ class USEquityPricingLoaderTestCase(WithAdjustmentReader, ) adjustments = adjustment_reader.load_adjustments( - columns, + [c.name for c in columns], query_days, self.assets, ) @@ -429,12 +429,12 @@ class USEquityPricingLoaderTestCase(WithAdjustmentReader, ) closes, volumes = map(getitem(results), columns) - expected_baseline_closes = expected_daily_bar_values_2d( + expected_baseline_closes = expected_bar_values_2d( shifted_query_days, self.asset_info, 'close', ) - expected_baseline_volumes = expected_daily_bar_values_2d( + expected_baseline_volumes = expected_bar_values_2d( shifted_query_days, self.asset_info, 'volume', @@ -506,12 +506,12 @@ class USEquityPricingLoaderTestCase(WithAdjustmentReader, ) highs, volumes = map(getitem(results), columns) - expected_baseline_highs = expected_daily_bar_values_2d( + expected_baseline_highs = expected_bar_values_2d( shifted_query_days, self.asset_info, 'high', ) - expected_baseline_volumes = expected_daily_bar_values_2d( + expected_baseline_volumes = expected_bar_values_2d( shifted_query_days, self.asset_info, 'volume', diff --git a/tests/resources/example_data.tar.gz b/tests/resources/example_data.tar.gz new file mode 100644 index 00000000..7957345e Binary files /dev/null and b/tests/resources/example_data.tar.gz differ diff --git a/tests/resources/quandl_samples/AAPL.csv.gz b/tests/resources/quandl_samples/AAPL.csv.gz index 4c2c4e23..94a97af4 100644 Binary files a/tests/resources/quandl_samples/AAPL.csv.gz and b/tests/resources/quandl_samples/AAPL.csv.gz differ diff --git a/tests/resources/quandl_samples/BRK_A.csv.gz b/tests/resources/quandl_samples/BRK_A.csv.gz index 9a79fe1e..1893fabc 100644 Binary files a/tests/resources/quandl_samples/BRK_A.csv.gz and b/tests/resources/quandl_samples/BRK_A.csv.gz differ diff --git a/tests/resources/quandl_samples/MSFT.csv.gz b/tests/resources/quandl_samples/MSFT.csv.gz index b55a4ec2..f1e374f1 100644 Binary files a/tests/resources/quandl_samples/MSFT.csv.gz and b/tests/resources/quandl_samples/MSFT.csv.gz differ diff --git a/tests/resources/quandl_samples/ZEN.csv.gz b/tests/resources/quandl_samples/ZEN.csv.gz index 1f5f493b..b0d8ebb8 100644 Binary files a/tests/resources/quandl_samples/ZEN.csv.gz and b/tests/resources/quandl_samples/ZEN.csv.gz differ diff --git a/tests/resources/quandl_samples/metadata-1.csv.gz b/tests/resources/quandl_samples/metadata-1.csv.gz new file mode 100644 index 00000000..4bf475bc Binary files /dev/null and b/tests/resources/quandl_samples/metadata-1.csv.gz differ diff --git a/tests/resources/quandl_samples/metadata-2.csv.gz b/tests/resources/quandl_samples/metadata-2.csv.gz new file mode 100644 index 00000000..24a71e38 Binary files /dev/null and b/tests/resources/quandl_samples/metadata-2.csv.gz differ diff --git a/tests/resources/quandl_samples/rebuild_samples.py b/tests/resources/quandl_samples/rebuild_samples.py index 6135ff52..377d90ff 100644 --- a/tests/resources/quandl_samples/rebuild_samples.py +++ b/tests/resources/quandl_samples/rebuild_samples.py @@ -2,13 +2,14 @@ Script for rebuilding the samples for the Quandl tests. """ from __future__ import print_function +from operator import methodcaller import pandas as pd import requests from zipline.data.quandl import format_wiki_url -from zipline.utils.test_utils import test_resource_path, write_compressed +from zipline.testing import test_resource_path, write_compressed def zipfile_path(symbol): @@ -18,7 +19,14 @@ def zipfile_path(symbol): def main(): start_date = pd.Timestamp('2014') end_date = pd.Timestamp('2015') - symbols = ['AAPL', 'MSFT', 'BRK_A', 'ZEN'] + symbols = 'AAPL', 'MSFT', 'BRK_A', 'ZEN' + names = ( + 'Apple Inc.', + 'Microsoft Corporation', + 'Berkshire Hathaway Inc. Class A', + 'Zendesk Inc', + ) + print('Downloading equity data') for sym in symbols: url = format_wiki_url( api_key=None, @@ -26,14 +34,31 @@ def main(): start_date=start_date, end_date=end_date, ) - print("Fetching from %s" % url) + print('Fetching from %s' % url) response = requests.get(url) response.raise_for_status() path = zipfile_path(sym) - print("Writing compressed data to %s" % path) + print('Writing compressed data to %s' % path) write_compressed(path, response.content) + print('Writing mock metadata') + cols = b'dataset_code,name,oldest_available_date,newest_available_date\n' + metadata = cols + b'\n'.join( + b','.join(map(methodcaller('encode', 'ascii'), ( + symbol, + name, + str(start_date.date()), str(end_date.date())) + )) + for symbol, name in zip(symbols, names) + ) + path = zipfile_path('metadata-1') + print('Writing compressed data to %s' % path) + write_compressed(path, metadata) + path = zipfile_path('metadata-2') + print('Writing compressed data to %s' % path) + write_compressed(path, cols) + if __name__ == '__main__': main() diff --git a/tests/resources/rebuild_example_data b/tests/resources/rebuild_example_data new file mode 100755 index 00000000..957d72f7 --- /dev/null +++ b/tests/resources/rebuild_example_data @@ -0,0 +1,120 @@ +#!/usr/bin/env python +from code import InteractiveConsole +import readline # noqa +import shutil +import tarfile + +import click +import numpy as np +import pandas as pd + +from zipline import examples, run_algorithm +from zipline.testing import test_resource_path, tmp_dir +from zipline.utils.cache import dataframe_cache + +banner = """ +Please verify that the new perfomance is more correct than the old performance. + +To do this, please inspect `new` and `old` which are mappings from the name of +the example to the results. + +If you are sure that the new results are more correct, or that the difference +is acceptable, please call `correct()`. Otherwise, call `incorrect()`. + +Note +---- +Remember to run this with the other supported versions of pandas! +""" + + +def eof(*args, **kwargs): + raise EOFError() + + +@click.command() +@click.pass_context +def main(ctx): + """Rebuild the perf data for test_examples + """ + example_path = test_resource_path('example_data.tar.gz') + with tmp_dir() as d: + with tarfile.open(example_path) as tar: + tar.extractall(d.path) + + mods = ( + (e, getattr(examples, e)) + for e in dir(examples) + if not e.startswith('_') + ) + + new_perf_path = d.getpath( + 'example_data/new_perf/%s' % pd.__version__.replace('.', '-'), + ) + c = dataframe_cache( + new_perf_path, + serialization='pickle:2', + ) + with c: + for name, mod in mods: + c[name] = run_algorithm( + handle_data=mod.handle_data, + initialize=mod.initialize, + before_trading_start=getattr( + mod, 'before_trading_start', None, + ), + analyze=getattr(mod, 'analyze', None), + bundle='test', + environ={ + 'ZIPLINE_ROOT': d.getpath('example_data/root'), + }, + **mod._test_args() + ) + + correct_called = [False] + + console = None + + def _exit(*args, **kwargs): + console.raw_input = eof + + def correct(): + correct_called[0] = True + _exit() + + expected_perf_path = d.getpath( + 'example_data/expected_perf/%s' % + pd.__version__.replace('.', '-'), + ) + + # allow users to run some analysis to make sure that the new + # results check out + console = InteractiveConsole({ + 'correct': correct, + 'exit': _exit, + 'incorrect': _exit, + 'new': c, + 'np': np, + 'old': dataframe_cache( + expected_perf_path, + serialization='pickle', + ), + 'pd': pd, + }) + console.interact(banner) + + if not correct_called[0]: + ctx.fail( + '`correct()` was not called! This means that the new' + ' results will not be written', + ) + + # move the new results to the expected path + shutil.rmtree(expected_perf_path) + shutil.copytree(new_perf_path, expected_perf_path) + + with tarfile.open(example_path, 'w|gz') as tar: + tar.add(d.getpath('example_data'), 'example_data') + + +if __name__ == '__main__': + main() diff --git a/tests/resources/yahoo_samples/AAPL.adjustments.gz b/tests/resources/yahoo_samples/AAPL.adjustments.gz new file mode 100644 index 00000000..cfcc5d4e Binary files /dev/null and b/tests/resources/yahoo_samples/AAPL.adjustments.gz differ diff --git a/tests/resources/yahoo_samples/AAPL.csv.gz b/tests/resources/yahoo_samples/AAPL.csv.gz new file mode 100644 index 00000000..3ddf5813 Binary files /dev/null and b/tests/resources/yahoo_samples/AAPL.csv.gz differ diff --git a/tests/resources/yahoo_samples/IBM.adjustments.gz b/tests/resources/yahoo_samples/IBM.adjustments.gz new file mode 100644 index 00000000..98a5b74d Binary files /dev/null and b/tests/resources/yahoo_samples/IBM.adjustments.gz differ diff --git a/tests/resources/yahoo_samples/IBM.csv.gz b/tests/resources/yahoo_samples/IBM.csv.gz new file mode 100644 index 00000000..fbc42c69 Binary files /dev/null and b/tests/resources/yahoo_samples/IBM.csv.gz differ diff --git a/tests/resources/yahoo_samples/MSFT.adjustments.gz b/tests/resources/yahoo_samples/MSFT.adjustments.gz new file mode 100644 index 00000000..9367baf3 Binary files /dev/null and b/tests/resources/yahoo_samples/MSFT.adjustments.gz differ diff --git a/tests/resources/yahoo_samples/MSFT.csv.gz b/tests/resources/yahoo_samples/MSFT.csv.gz new file mode 100644 index 00000000..690fdbec Binary files /dev/null and b/tests/resources/yahoo_samples/MSFT.csv.gz differ diff --git a/tests/resources/yahoo_samples/rebuild_samples b/tests/resources/yahoo_samples/rebuild_samples new file mode 100644 index 00000000..7c45e4a6 --- /dev/null +++ b/tests/resources/yahoo_samples/rebuild_samples @@ -0,0 +1,79 @@ +#!/usr/bin/env python +""" +Script for rebuilding the samples for the Yahoo tests. +""" +from textwrap import dedent + +import numpy as np +import pandas as pd + +from zipline.testing import test_resource_path, write_compressed +from zipline.utils.tradingcalendar import trading_days + + +def zipfile_path(symbol, ext): + return test_resource_path('yahoo_samples', symbol + ext + '.gz') + + +def pricing_for_sid(sid): + modifier = { + 'Low': 0, + 'Open': 1, + 'Close': 2, + 'High': 3, + 'Volume': 0, + } + + def column(name): + return np.arange(252) + 1 + sid * 10000 + modifier[name] * 1000 + + return pd.DataFrame( + data={ + 'Date': trading_days[ + (trading_days >= pd.Timestamp('2014')) & + (trading_days < pd.Timestamp('2015')) + ], + 'Open': column('Open'), + 'High': column('High'), + 'Low': column('Low'), + 'Close': column('Close'), + 'Volume': column('Volume'), + 'Adj Close': 0, + }, + columns=[ + 'Date', 'Open', 'High', 'Low', 'Close', 'Volume', 'Adj Close', + ], + ).to_csv(index=False, date_format='%Y-%m-%d').encode('ascii') + + +def adjustments_for_sid(sid): + """This is not exactly a csv... thanks yahoo. + """ + return dedent( + """\ + Date,Dividends + DIVIDEND, 20140403,0.{p1}00000 + SPLIT, 20140703,{p2}:1 + DIVIDEND, 20141002,0.{p2}00000 + STARTDATE, 20140102 + ENDDATE, 20141231 + TOTALSIZE, 2 + """.format(p1=sid + 1, p2=sid + 2), + ).encode('ascii') + + +def main(): + symbols = 'AAPL', 'IBM', 'MSFT' + + for sid, symbol in enumerate(symbols): + write_compressed( + zipfile_path(symbol, '.csv'), + pricing_for_sid(sid), + ) + write_compressed( + zipfile_path(symbol, '.adjustments'), + adjustments_for_sid(sid), + ) + +if __name__ == '__main__': + main() diff --git a/tests/test_cli.py b/tests/test_cli.py deleted file mode 100644 index d7c187b2..00000000 --- a/tests/test_cli.py +++ /dev/null @@ -1,70 +0,0 @@ -# -# Copyright 2014 Quantopian, Inc. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -import os -from unittest import TestCase -from six import iteritems - -from zipline.utils import parse_args -from zipline.utils import cli - - -class TestParseArgs(TestCase): - def test_defaults(self): - args = parse_args([]) - for k, v in iteritems(cli.DEFAULTS): - self.assertEqual(v, args[k]) - - def write_conf_file(self): - conf_str = """ -[Defaults] -algofile=test.py -symbols=test_symbols -start=1990-1-1 - """ - - with open('test.conf', 'w') as fd: - fd.write(conf_str) - - def test_conf_file(self): - self.write_conf_file() - try: - args = parse_args(['-c', 'test.conf']) - - self.assertEqual(args['algofile'], 'test.py') - self.assertEqual(args['symbols'], 'test_symbols') - self.assertEqual(args['start'], '1990-1-1') - self.assertEqual(args['data_frequency'], - cli.DEFAULTS['data_frequency']) - finally: - os.remove('test.conf') - - def test_overwrite(self): - self.write_conf_file() - - try: - args = parse_args(['-c', 'test.conf', '--start', '1992-1-1', - '--algofile', 'test2.py']) - - # Overwritten values - self.assertEqual(args['algofile'], 'test2.py') - self.assertEqual(args['start'], '1992-1-1') - # Non-overwritten values - self.assertEqual(args['symbols'], 'test_symbols') - # Default values - self.assertEqual(args['data_frequency'], - cli.DEFAULTS['data_frequency']) - finally: - os.remove('test.conf') diff --git a/tests/test_examples.py b/tests/test_examples.py index 14080655..e074c9e4 100644 --- a/tests/test_examples.py +++ b/tests/test_examples.py @@ -15,15 +15,17 @@ # This code is based on a unittest written by John Salvatier: # https://github.com/pymc-devs/pymc/blob/pymc3/tests/test_examples.py +import tarfile -import glob import matplotlib from nose_parameterized import parameterized -import os -import runpy -from unittest import TestCase +import pandas as pd -from zipline.utils import parse_args, run_pipeline +from zipline import examples, run_algorithm +from zipline.testing import test_resource_path +from zipline.testing.fixtures import WithTmpDir, ZiplineTestCase +from zipline.testing.predicates import assert_equal +from zipline.utils.cache import dataframe_cache # Otherwise the next line sometimes complains about being run too late. _multiprocess_can_split_ = False @@ -31,22 +33,80 @@ _multiprocess_can_split_ = False matplotlib.use('Agg') -def example_dir(): - import zipline - d = os.path.dirname(zipline.__file__) - return os.path.join(os.path.abspath(d), 'examples') +class ExamplesTests(WithTmpDir, ZiplineTestCase): + # some columns contain values with unique ids that will not be the same + cols_to_check = [ + 'algo_volatility', + 'algorithm_period_return', + 'alpha', + 'benchmark_period_return', + 'benchmark_volatility', + 'beta', + 'capital_used', + 'ending_cash', + 'ending_exposure', + 'ending_value', + 'excess_return', + 'gross_leverage', + 'long_exposure', + 'long_value', + 'longs_count', + 'max_drawdown', + 'max_leverage', + 'net_leverage', + 'period_close', + 'period_label', + 'period_open', + 'pnl', + 'portfolio_value', + 'positions', + 'returns', + 'short_exposure', + 'short_value', + 'shorts_count', + 'sortino', + 'starting_cash', + 'starting_exposure', + 'starting_value', + 'trading_days', + 'treasury_period_return', + ] + @classmethod + def init_class_fixtures(cls): + super(ExamplesTests, cls).init_class_fixtures() -class ExamplesTests(TestCase): - # Test algorithms as if they are executed directly from the command line. - @parameterized.expand(((os.path.basename(f).replace('.', '_'), f) for f in - glob.glob(os.path.join(example_dir(), '*.py')))) - def test_example(self, name, example): - runpy.run_path(example, run_name='__main__') + with tarfile.open(test_resource_path('example_data.tar.gz')) as tar: + tar.extractall(cls.tmpdir.path) - # Test algorithm as if scripts/run_algo.py is being used. - def test_example_run_pipeline(self): - example = os.path.join(example_dir(), 'buyapple.py') - confs = ['-f', example, '--start', '2011-1-1', '--end', '2012-1-1'] - parsed_args = parse_args(confs) - run_pipeline(**parsed_args) + cls.expected_perf = dataframe_cache( + cls.tmpdir.getpath( + 'example_data/expected_perf/%s' % + pd.__version__.replace('.', '-'), + ), + serialization='pickle', + ) + + @parameterized.expand(e for e in dir(examples) if not e.startswith('_')) + def test_example(self, example): + mod = getattr(examples, example) + actual_perf = run_algorithm( + handle_data=mod.handle_data, + initialize=mod.initialize, + before_trading_start=getattr(mod, 'before_trading_start', None), + analyze=getattr(mod, 'analyze', None), + bundle='test', + environ={ + 'ZIPLINE_ROOT': self.tmpdir.getpath('example_data/root'), + }, + capital_base=1e7, + **mod._test_args() + ) + assert_equal( + actual_perf[self.cols_to_check], + self.expected_perf[example][self.cols_to_check], + # There is a difference in the datetime columns in pandas + # 0.16 and 0.17 because in 16 they are object and in 17 they are + # datetime[ns, UTC]. We will just ignore the dtypes for now. + check_dtype=False, + ) diff --git a/zipline/__init__.py b/zipline/__init__.py index 9166faa7..5dfe287b 100644 --- a/zipline/__init__.py +++ b/zipline/__init__.py @@ -20,6 +20,7 @@ from . import data from . import finance from . import gens from . import utils +from .utils.run_algo import run_algorithm from ._version import get_versions # These need to happen after the other imports. from . algorithm import TradingAlgorithm @@ -28,19 +29,18 @@ from . import api __version__ = get_versions()['version'] del get_versions -try: - ip = get_ipython() # flake8: noqa -except NameError: - pass -else: - ip.register_magic_function(utils.parse_cell_magic, "line_cell", "zipline") - del ip + +def load_ipython_extension(ipython): + from .__main__ import zipline_magic + ipython.register_magic_function(zipline_magic, 'line_cell', 'zipline') + __all__ = [ + 'TradingAlgorithm', + 'api', 'data', 'finance', 'gens', + 'run_algorithm', 'utils', - 'api', - 'TradingAlgorithm', ] diff --git a/zipline/__main__.py b/zipline/__main__.py new file mode 100644 index 00000000..df660d57 --- /dev/null +++ b/zipline/__main__.py @@ -0,0 +1,332 @@ +import datetime +import os +from functools import wraps + +import click +import logbook +import pandas as pd + +from zipline.data import bundles +from zipline.utils.cli import Date, Timestamp +from zipline.utils.run_algo import _run, load_extensions + +try: + __IPYTHON__ +except NameError: + __IPYTHON__ = False + + +@click.group() +@click.option( + '-e', + '--extension', + multiple=True, + help='File or module path to a zipline extension to load.', +) +@click.option( + '--strict-extensions/--non-strict-extensions', + is_flag=True, + help='If --strict-extensions is passed then zipline will not run if it' + ' cannot load all of the specified extensions. If this is not passed or' + ' --non-strict-extensions is passed then the failure will be logged but' + ' execution will continue.', +) +@click.option( + '--default-extension/--no-default-extension', + is_flag=True, + default=True, + help="Don't load the default zipline extension.py file in $ZIPLINE_HOME.", +) +def cli(extension, strict_extensions, default_extension): + """Top level zipline entry point. + """ + # install a logbook handler before performing any other operations + logbook.StderrHandler().push_application() + load_extensions( + default_extension, + extension, + strict_extensions, + os.environ, + ) + + +def extract_option_object(option): + """Convert a click.option call into a click.Option object. + + Parameters + ---------- + option : decorator + A click.option decorator. + + Returns + ------- + option_object : click.Option + The option object that this decorator will create. + """ + @option + def opt(): + pass + + return opt.__click_params__[0] + + +def ipython_only(option): + """Mark that an option should only be exposed in IPython. + + Parameters + ---------- + option : decorator + A click.option decorator. + + Returns + ------- + ipython_only_dec : decorator + A decorator that correctly applies the argument even when not + using IPython mode. + """ + if __IPYTHON__: + return option + + argname = extract_option_object(option).name + + def d(f): + @wraps(f) + def _(*args, **kwargs): + kwargs[argname] = None + return f(*args, **kwargs) + return _ + return d + + +@cli.command() +@click.option( + '-f', + '--algofile', + default=None, + type=click.File('r'), + help='The file that contains the algorithm to run.', +) +@click.option( + '-t', + '--algotext', + help='The algorithm script to run.', +) +@click.option( + '-D', + '--define', + multiple=True, + help="Define a name to be bound in the namespace before executing" + " the algotext. For example '-Dname=value'. The value may be any python" + " expression. These are evaluated in order so they may refer to previously" + " defined names.", +) +@click.option( + '--data-frequency', + type=click.Choice({'daily', 'minute'}), + default='daily', + show_default=True, + help='The data frequency of the simulation.', +) +@click.option( + '--capital-base', + type=float, + default=10e6, + show_default=True, + help='The starting capital for the simulation.', +) +@click.option( + '-b', + '--bundle', + default='quandl', + metavar='BUNDLE-NAME', + show_default=True, + help='The data bundle to use for the simulation.', +) +@click.option( + '--bundle-timestamp', + type=Timestamp(), + default=pd.Timestamp.utcnow(), + show_default=False, + help='The date to lookup data on or before.\n' + '[default: ]' +) +@click.option( + '-s', + '--start', + type=Date(tz='utc', as_timestamp=True), + help='The start date of the simulation.', +) +@click.option( + '-e', + '--end', + type=Date(tz='utc', as_timestamp=True), + help='The end date of the simulation.', +) +@click.option( + '-o', + '--output', + default='-', + metavar='FILENAME', + show_default=True, + help="The location to write the perf data. If this is '-' the perf will" + " be written to stdout.", +) +@click.option( + '--print-algo/--no-print-algo', + is_flag=True, + default=False, + help='Print the algorithm to stdout.', +) +@ipython_only(click.option( + '--local-namespace/--no-local-namespace', + is_flag=True, + default=None, + help='Should the algorithm methods be resolved in the local namespace.' +)) +@click.pass_context +def run(ctx, + algofile, + algotext, + define, + data_frequency, + capital_base, + bundle, + bundle_timestamp, + start, + end, + output, + print_algo, + local_namespace): + """Run a backtest for the given algorithm. + """ + # check that the start and end dates are passed correctly + if start is None and end is None: + # check both at the same time to avoid the case where a user + # does not pass either of these and then passes the first only + # to be told they need to pass the second argument also + ctx.fail( + "must specify dates with '-s' / '--start' and '-e' / '--end'", + ) + if start is None: + ctx.fail("must specify a start date with '-s' / '--start'") + if end is None: + ctx.fail("must specify an end date with '-s' / '--end'") + + if (algotext is not None) == (algofile is not None): + ctx.fail( + "must specify exactly one of '-f' / '--algofile' or" + " '-t' / '--algotext'", + ) + + perf = _run( + initialize=None, + handle_data=None, + before_trading_start=None, + analyze=None, + algofile=algofile, + algotext=algotext, + defines=define, + data_frequency=data_frequency, + capital_base=capital_base, + data=None, + bundle=bundle, + bundle_timestamp=bundle_timestamp, + start=start, + end=end, + output=output, + print_algo=print_algo, + local_namespace=local_namespace, + environ=os.environ, + ) + + if output == '-': + click.echo(str(perf)) + elif output != os.devnull: # make the zipline magic not write any data + perf.to_pickle(output) + + return perf + + +def zipline_magic(line, cell=None): + """The zipline IPython cell magic. + """ + try: + return run.main( + # put our overrides at the start of the parameter list so that + # users may pass values with higher precedence + [ + '--algotext', cell, + '--output', os.devnull, # don't write the results by default + ] + ([ + # these options are set when running in line magic mode + # set a non None algo text to use the ipython user_ns + '--algotext', '', + '--local-namespace', + ] if cell is None else []) + line.split(), + '%s%%zipline' % ((cell or '') and '%'), + # don't use system exit and propogate errors to the caller + standalone_mode=False, + ) + except SystemExit as e: + # https://github.com/mitsuhiko/click/pull/533 + # even in standalone_mode=False `--help` really wants to kill us ;_; + if e.code: + raise ValueError('main returned non-zero status code: %d' % e.code) + + +@cli.command() +@click.argument('BUNDLE-NAME') +@click.option( + '--show-progress/--no-show-progress', + is_flag=True, + default=True, + help='Print progress information to the terminal.' +) +def ingest(bundle_name, show_progress): + """Ingest the data for the given bundle. + """ + bundles.ingest( + bundle_name, + os.environ, + datetime.date.today(), + show_progress, + ) + + +@cli.command() +@click.argument('BUNDLE-NAME') +@click.option( + '-b', + '--before', + type=Timestamp(), + help='Clear all data before TIMESTAMP.' + ' This may not be passed with -k / --keep-last', +) +@click.option( + '-a', + '--after', + type=Timestamp(), + help='Clear all data after TIMESTAMP' + ' This may not be passed with -k / --keep-last', +) +@click.option( + '-k', + '--keep-last', + type=int, + metavar='N', + help='Clear all but the last N downloads.' + ' This may not be passed with -b / --before or -a / --after', +) +def clean(bundle_name, before, after, keep_last): + """Clean up data downloaded with the ingest command. + """ + bundles.clean( + bundle_name, + before, + after, + keep_last, + ) + + +if __name__ == '__main__': + cli() diff --git a/zipline/algorithm.py b/zipline/algorithm.py index ed0f124d..496a6a56 100644 --- a/zipline/algorithm.py +++ b/zipline/algorithm.py @@ -116,7 +116,7 @@ from zipline.gens.sim_engine import ( from zipline.sources.benchmark_source import BenchmarkSource from zipline.zipline_warnings import ZiplineDeprecationWarning -DEFAULT_CAPITAL_BASE = float("1.0e5") +DEFAULT_CAPITAL_BASE = 1e5 log = logbook.Logger("ZiplineLog") diff --git a/zipline/assets/asset_writer.py b/zipline/assets/asset_writer.py index b1eda14c..dfc8566b 100644 --- a/zipline/assets/asset_writer.py +++ b/zipline/assets/asset_writer.py @@ -257,6 +257,95 @@ class AssetDBWriter(object): exchanges=None, root_symbols=None, chunk_size=DEFAULT_CHUNK_SIZE): + """Write asset metadata to a sqlite database. + + Parameters + ---------- + equities : pd.DataFrame, optional + The equity metadata. The columns for this dataframe are: + + symbol : str + The ticker symbol for this equity. + fuzzy_symbol : str, optional + The fuzzy symbol for this equity. This is the symbol + without any delimiting characters like '.' or '_'. + asset_name : str + The full name for this asset. + start_date : datetime + The date when this asset was created. + end_date : datetime, optional + The last date we have trade data for this asset. + first_traded : datetime, optional + The first date we have trade data for this asset. + auto_close_date : datetime, optional + The date on which to close any positions in this asset. + exchange : str, optional + The exchange where this asset is traded. + + The index of this dataframe should contain the sids. + futures : pd.Dataframe, optional + The future contract metadata. The columns for this dataframe are: + + symbol : str + The ticker symbol for this futures contract. + root_symbol : str + The root symbol, or the symbol with the expiration stripped + out. + asset_name : str + The full name for this asset. + start_date : datetime, optional + The date when this asset was created. + end_date : datetime, optional + The last date we have trade data for this asset. + first_traded : datetime, optional + The first date we have trade data for this asset. + exchange : str, optional + The exchange where this asset is traded. + notice_date : datetime + The date when the owner of the contract may be forced + to take physical delivery of the contract's asset. + expiration_date : datetime + The date when the contract expires. + auto_close_date : datetime + The date when the broker will automatically close any + positions in this contract. + tick_size : float + The minimum price movement of the contract. + multiplier: float + The amount of the underlying asset represented by this + contract. + exchanges : pd.Dataframe, optional + The exchanges where assets can be traded. The columns of this + dataframe are: + + exchange : str + The name of the exchange. + timezone : str + The timezone of the exchange. + root_symbols : pd.Dataframe, optional + The root symbols for the futures contracts. The columns for this + dataframe are: + + root_symbol : str + The root symbol name. + root_symbol_id : int + The unique id for this root symbol. + sector : string, optional + The sector of this root symbol. + description : string, optional + A short description of this root symbol. + exchange : str + The exchange where this root symbol is traded. + chunk_size : int, optional + The amount of rows to write to the SQLite table at once. + This defaults to the default number of bind params in sqlite. + If you have compiled sqlite3 with more bind or less params you may + want to pass that value here. + + See Also + -------- + zipline.assets.asset_finder + """ with self.engine.begin() as txn: # Create SQL tables if they do not exist. diff --git a/zipline/assets/assets.py b/zipline/assets/assets.py index ce498336..b671a754 100644 --- a/zipline/assets/assets.py +++ b/zipline/assets/assets.py @@ -94,7 +94,7 @@ class AssetFinder(object): See Also -------- - :class:`zipline.assets.asset_writer.AssetDBWriter` + :class:`zipline.assets.AssetDBWriter` """ # Token used as a substitute for pickling objects that contain a # reference to an AssetFinder. diff --git a/zipline/data/__init__.py b/zipline/data/__init__.py index 478d6261..f3dd9b36 100644 --- a/zipline/data/__init__.py +++ b/zipline/data/__init__.py @@ -1,8 +1,16 @@ from . import loader from .loader import ( - load_from_yahoo, load_bars_from_yahoo, load_prices_from_csv, - load_prices_from_csv_folder + load_from_yahoo, + load_bars_from_yahoo, + load_prices_from_csv, + load_prices_from_csv_folder, ) -__all__ = ['loader', 'load_from_yahoo', 'load_bars_from_yahoo', - 'load_prices_from_csv', 'load_prices_from_csv_folder'] + +__all__ = [ + 'load_bars_from_yahoo', + 'load_from_yahoo', + 'load_prices_from_csv', + 'load_prices_from_csv_folder', + 'loader', +] diff --git a/zipline/data/bundles/__init__.py b/zipline/data/bundles/__init__.py new file mode 100644 index 00000000..baca0f18 --- /dev/null +++ b/zipline/data/bundles/__init__.py @@ -0,0 +1,20 @@ +from .core import ( + bundles, + clean, + ingest, + load, + register, + unregister, +) +from .yahoo import yahoo_equities + + +__all__ = [ + 'bundles', + 'clean', + 'ingest', + 'load', + 'register', + 'unregister', + 'yahoo_equities', +] diff --git a/zipline/data/bundles/core.py b/zipline/data/bundles/core.py new file mode 100644 index 00000000..1efe024b --- /dev/null +++ b/zipline/data/bundles/core.py @@ -0,0 +1,469 @@ +from collections import namedtuple +import errno +import os +import shutil +import warnings + +import click +import pandas as pd +from toolz import curry, complement, compose + +from ..us_equity_pricing import ( + BcolzDailyBarReader, + BcolzDailyBarWriter, + SQLiteAdjustmentReader, + SQLiteAdjustmentWriter, +) +from ..minute_bars import ( + BcolzMinuteBarReader, + BcolzMinuteBarWriter, +) +from zipline.assets import AssetDBWriter, AssetFinder, ASSET_DB_VERSION +from zipline.utils.cache import ( + dataframe_cache, + working_file, + working_dir, +) +from zipline.utils.compat import mappingproxy +from zipline.utils.input_validation import ensure_timestamp, optionally +import zipline.utils.paths as pth +from zipline.utils.preprocess import preprocess +from zipline.utils.tradingcalendar import trading_days, open_and_closes + + +def asset_db_path(bundle_name, timestr, environ=None): + return pth.data_path( + [bundle_name, timestr, 'assets-%d.sqlite' % ASSET_DB_VERSION], + environ=environ, + ) + + +def minute_equity_path(bundle_name, timestr, environ=None): + return pth.data_path( + [bundle_name, timestr, 'minute_equities.bcolz'], + environ=environ, + ) + + +def daily_equity_path(bundle_name, timestr, environ=None): + return pth.data_path( + [bundle_name, timestr, 'daily_equities.bcolz'], + environ=environ, + ) + + +def adjustment_db_path(bundle_name, timestr, environ=None): + return pth.data_path( + [bundle_name, timestr, 'adjustments.sqlite'], + environ=environ, + ) + + +def cache_path(bundle_name, timestr, environ=None): + return pth.data_path( + [bundle_name, timestr, '.cache'], + environ=environ, + ) + + +_BundlePayload = namedtuple( + '_BundlePayload', + 'calendar opens closes minutes_per_day ingest', +) + + +class UnknownBundle(click.ClickException, LookupError): + """Raised if no bundle with the given name was registered. + """ + exit_code = 1 + + def __init__(self, name): + super(UnknownBundle, self).__init__( + 'No bundle registered with the name %r' % name, + ) + self.name = name + + def __str__(self): + return self.message + + +def _make_bundle_core(): + """Create a family of data bundle functions that read from the same + bundle mapping. + + Returns + ------- + bundles : mappingproxy + The mapping of bundles to bundle payloads. + register : callable + The function which registers new bundles in the ``bundles`` mapping. + unregister : callable + The function which deregisters bundles from the ``bundles`` mapping. + ingest_bundle : callable + The function which downloads and write data for a given data bundle. + """ + _bundles = {} # the registered bundles + # Expose _bundles through a proxy so that users cannot mutate this + # accidentally. Users may go through `register` to update this which will + # warn when trampling another bundle. + bundles = mappingproxy(_bundles) + + @curry + def register(name, + f, + calendar=trading_days, + opens=open_and_closes['market_open'], + closes=open_and_closes['market_close'], + minutes_per_day=390): + """Register a data bundle ingest function. + + Parameters + ---------- + name : str + The name of the bundle. + f : callable + The ingest function. This function will be passed: + + environ : mapping + The environment this is being run with. + asset_db_writer : AssetDBWriter + The asset db writer to write into. + minute_bar_writer : BcolzMinuteBarWriter + The minute bar writer to write into. + daily_bar_writer : BcolzDailyBarWriter + The daily bar writer to write into. + adjustment_writer : SQLiteAdjustmentWriter + The adjustment db writer to write into. + calendar : pd.DatetimeIndex + The trading calendar to ingest for. + cache : DataFrameCache + A mapping object to temporarily store dataframes. + This should be used to cache intermediates in case the load + fails. This will be automatically cleaned up after a + successful load. + show_progress : bool + Show the progress for the current load where possible. + calendar : pd.DatetimeIndex, optional + The exchange calendar to align the data to. This defaults to the + NYSE calendar. + market_open : pd.DatetimeIndex, optional + The minute when the market opens each day. This defaults to the + NYSE calendar. + market_close : pd.DatetimeIndex, optional + The minute when the market closes each day. This defaults to the + NYSE calendar. + minutes_per_day : int, optional + The number of minutes in each normal trading day. + + Notes + ----- + This function my be used as a decorator, for example: + + .. code-block:: python + + @register('quandl') + def quandl_ingest_function(...): + ... + + See Also + -------- + zipline.data.bundles.bundles + """ + if name in bundles: + warnings.warn( + 'Overwriting bundle with name %r' % name, + stacklevel=3, + ) + _bundles[name] = _BundlePayload( + calendar, + opens, + closes, + minutes_per_day, + f, + ) + return f + + def unregister(name): + """Unregister a bundle. + + Parameters + ---------- + name : str + The name of the bundle to unregister. + + Raises + ------ + UnknownBundle + Raised when no bundle has been registered with the given name. + + See Also + -------- + zipline.data.bundles.bundles + """ + try: + del _bundles[name] + except KeyError: + raise UnknownBundle(name) + + def ingest(name, + environ=os.environ, + timestamp=None, + show_progress=True): + """Ingest data for a given bundle. + + Parameters + ---------- + name : str + The name of the bundle. + environ : mapping, optional + The environment variables. By default this is os.environ. + timestamp : datetime, optional + The timestamp to use for the load. + By default this is the current time. + show_progress : bool, optional + Tell the ingest function to display the progress where possible. + """ + try: + bundle = bundles[name] + except KeyError: + raise UnknownBundle(name) + + if timestamp is None: + timestamp = pd.Timestamp.utcnow() + timestamp = timestamp.tz_convert('utc').tz_localize(None) + timestr = str(timestamp.value) + cachepath = cache_path(name, timestr, environ=environ) + pth.ensure_directory(cachepath) + + with dataframe_cache(cachepath, clean_on_failure=False) as cache, \ + working_dir( + daily_equity_path(name, timestr, environ=environ), + ) as daily_bars_dir, \ + working_dir( + minute_equity_path(name, timestr, environ=environ), + ) as minute_bars_dir, \ + working_file( + asset_db_path(name, timestr, environ=environ), + ) as asset_db_file, \ + working_file( + adjustment_db_path(name, timestr, environ=environ), + ) as adjustment_db_file: + # we use `cleanup_on_failure=False` so that we don't purge the + # cache directory if the load fails in the middle + daily_bar_writer = BcolzDailyBarWriter( + daily_bars_dir.name, + bundle.calendar, + ) + # Do an empty write to ensure that the daily ctables exist + # when we create the SQLiteAdjustmentWriter below. The + # SQLiteAdjustmentWriter needs to open the daily ctables so that + # it can compute the adjustment ratios for the dividends. + daily_bar_writer.write(()) + bundle.ingest( + environ, + AssetDBWriter(asset_db_file.name), + BcolzMinuteBarWriter( + bundle.calendar[0], + minute_bars_dir.name, + bundle.opens, + bundle.closes, + minutes_per_day=bundle.minutes_per_day, + ), + daily_bar_writer, + SQLiteAdjustmentWriter( + adjustment_db_file.name, + BcolzDailyBarReader(daily_bars_dir.name), + bundle.calendar, + overwrite=True, + ), + bundle.calendar, + cache, + show_progress, + ) + + return bundles, register, unregister, ingest + + +bundles, register, unregister, ingest = _make_bundle_core() + +BundleData = namedtuple( + 'BundleData', + 'asset_finder minute_bar_reader daily_bar_reader adjustment_reader', +) + + +def most_recent_data(bundle_name, timestamp, environ=None): + """Get the path to the most recent data after ``date``for the given bundle. + + Parameters + ---------- + bundle_name : str + The name of the bundle to lookup. + timestamp : datetime + The timestamp to begin searching on or before. + environ : dict, optional + An environment dict to forward to zipline_root. + """ + try: + candidates = os.listdir(pth.data_path([bundle_name], environ=environ)) + return pth.data_path( + [bundle_name, + max( + filter(complement(pth.hidden), candidates), + key=compose(pd.Timestamp, int), + )], + environ=environ, + ) + except ValueError: + raise ValueError( + 'no data for bundle %r on or before %s' % ( + bundle_name, + timestamp, + ), + ) + except OSError as e: + if e.errno != errno.ENOENT: + raise + raise UnknownBundle(bundle_name) + + +def load(name, environ=os.environ, timestamp=None): + """Loads a previously ingested bundle. + + Parameters + ---------- + name : str + The name of the bundle. + environ : mapping, optional + The environment variables. Defaults of os.environ. + timestamp : datetime, optional + The timestamp of the data to lookup. + Defaults to the current time. + + Returns + ------- + bundle_data : BundleData + The raw data readers for this bundle. + """ + if timestamp is None: + timestamp = pd.Timestamp.utcnow() + timestr = most_recent_data(name, timestamp, environ=environ) + return BundleData( + asset_finder=AssetFinder( + asset_db_path(name, timestr, environ=environ), + ), + minute_bar_reader=BcolzMinuteBarReader( + minute_equity_path(name, timestr, environ=environ), + ), + daily_bar_reader=BcolzDailyBarReader( + daily_equity_path(name, timestr, environ=environ), + ), + adjustment_reader=SQLiteAdjustmentReader( + adjustment_db_path(name, timestr, environ=environ), + ), + ) + + +class BadClean(click.ClickException, ValueError): + """Exception indicating that an invalid argument set was passed to + ``clean``. + + Parameters + ---------- + before, after, keep_last : any + The bad arguments to ``clean``. + + See Also + -------- + clean + """ + def __init__(self, before, after, keep_last): + super(BadClean, self).__init__( + 'Cannot pass a combination of `before` and `after` with' + '`keep_last`. Got: before=%r, after=%r, keep_n=%r\n' % ( + before, + after, + keep_last, + ), + ) + + def __str__(self): + return self.message + + +@preprocess( + before=optionally(ensure_timestamp), + after=optionally(ensure_timestamp), +) +def clean(name, before=None, after=None, keep_last=None, environ=os.environ): + """Clean up data that was created with ``ingest`` or + ``$ python -m zipline ingest`` + + Parameters + ---------- + name : str + The name of the bundle to remove data for. + before : datetime, optional + Remove data ingested before this date. + This argument is mutually exclusive with: keep_last + after : datetime, optional + Remove data ingested after this date. + This argument is mutually exclusive with: keep_last + keep_last : int, optional + Remove all but the last ``keep_last`` ingestions. + This argument is mutually exclusive with: + before + after + + Returns + ------- + cleaned : set[str] + The names of the runs that were removed. + + Raises + ------ + BadClean + Raised when ``before`` and or ``after`` are passed with ``keep_last``. + This is a subclass of ``ValueError``. + """ + try: + all_runs = sorted( + pd.Timestamp(f) + for f in os.listdir(pth.data_path([name], environ=environ)) + if not pth.hidden(f) + ) + except OSError as e: + if e.errno != errno.ENOENT: + raise + raise UnknownBundle(name) + + if (before is not None or after is not None) and keep_last is not None: + raise BadClean(before, after, keep_last) + + if keep_last is None: + def in_last_n(dt): + return False + else: + last_n_dts = set(all_runs[:keep_last]) + + def in_last_n(dt): + return dt in last_n_dts + + def should_clean(name): + dt = pd.Timestamp(name) + + return ( + ( + (before is not None and dt < before) or + (after is not None and dt > after) + ) and + not in_last_n(dt) + ) + + cleaned = set() + for run in all_runs: + if should_clean(run): + shutil.rmdir(run) + cleaned.add(run) + + return cleaned diff --git a/zipline/data/bundles/quandl.py b/zipline/data/bundles/quandl.py new file mode 100644 index 00000000..a0ca1906 --- /dev/null +++ b/zipline/data/bundles/quandl.py @@ -0,0 +1,298 @@ +""" +Module for building a complete daily dataset from Quandl's WIKI dataset. +""" +from itertools import count +from time import time, sleep + +from logbook import Logger +import pandas as pd +from six.moves.urllib.parse import urlencode + +from zipline.utils.cli import maybe_show_progress +from zipline.data import bundles + +log = Logger(__name__) +seconds_per_call = (pd.Timedelta('10 minutes') / 2000).total_seconds() + + +def _fetch_raw_metadata(api_key, cache, retries, environ): + """Generator that yields each page of data from the metadata endpoint + as a dataframe. + """ + for page_number in count(1): + key = 'metadata-page-%d' % page_number + try: + raw = cache[key] + except KeyError: + for _ in range(retries): + try: + raw = pd.read_csv( + format_metadata_url(api_key, page_number), + parse_dates=[ + 'oldest_available_date', + 'newest_available_date', + ], + usecols=[ + 'dataset_code', + 'name', + 'oldest_available_date', + 'newest_available_date', + ], + ) + break + except ValueError: + # when we are past the last page we will get a value + # error because there will be no columns + raw = pd.DataFrame([]) + break + except Exception: + pass + else: + raise ValueError( + 'Failed to download metadata page %d after %d' + ' attempts.' % (page_number, retries), + ) + + cache[key] = raw + + if raw.empty: + # use the empty dataframe to signal completion + break + yield raw + + +def fetch_symbol_metadata_frame(api_key, + cache, + retries=5, + environ=None, + show_progress=False): + """ + Download Quandl symbol metadata. + + Parameters + ---------- + api_key : str + The quandl api key to use. If this is None then no api key will be + sent. + cache : DataFrameCache + The cache to use for persisting the intermediate data. + retries : int, optional + The number of times to retry each request before failing. + environ : mapping[str -> str], optional + The environment to use to find the zipline home. By default this + is ``os.environ``. + show_progress : bool, optional + Show a progress bar for the download of this data. + + Returns + ------- + metadata_frame : pd.DataFrame + A dataframe with the following columns: + symbol: the asset's symbol + name: the full name of the asset + start_date: the first date of data for this asset + end_date: the last date of data for this asset + exchange: the exchange for the asset; this is always 'quandl' + The index of the dataframe will be used for symbol->sid mappings but + otherwise does not have specific meaning. + """ + raw_iter = _fetch_raw_metadata(api_key, cache, retries, environ) + + def item_show_func(_, _it=iter(count())): + 'Downloading page: %d' % next(_it) + + with maybe_show_progress(raw_iter, + show_progress, + item_show_func=item_show_func, + label='Downloading WIKI metadata: ') as blocks: + data = pd.concat(blocks, ignore_index=True).rename(columns={ + 'dataset_code': 'symbol', + 'name': 'asset_name', + 'oldest_available_date': 'start_date', + 'newest_available_date': 'end_date', + }).sort('symbol') + # cut out all the other stuff in the name column + # we need to escape the paren because it is actually splitting on a regex + data.asset_name = data.asset_name.str.split(r' \(', 1).str.get(0) + data['exchange'] = 'quandl' + return data + + +def format_metadata_url(api_key, page_number): + """Build the query RL for the quandl WIKI metadata. + """ + query_params = [ + ('per_page', '100'), + ('sort_by', 'id'), + ('page', str(page_number)), + ('database_code', 'WIKI'), + ] + if api_key is not None: + query_params = [('api_key', api_key)] + query_params + return ( + 'https://www.quandl.com/api/v3/datasets.csv?' + urlencode(query_params) + ) + + +def format_wiki_url(api_key, symbol, start_date, end_date): + """ + Build a query URL for a quandl WIKI dataset. + """ + query_params = [ + ('start_date', start_date.strftime('%Y-%m-%d')), + ('end_date', end_date.strftime('%Y-%m-%d')), + ('order', 'asc'), + ] + if api_key is not None: + query_params = [('api_key', api_key)] + query_params + + return ( + "https://www.quandl.com/api/v3/datasets/WIKI/" + "{symbol}.csv?{query}".format( + symbol=symbol, + query=urlencode(query_params), + ) + ) + + +def fetch_single_equity(api_key, + symbol, + start_date, + end_date, + retries=5): + """ + Download data for a single equity. + """ + for _ in range(retries): + try: + return pd.read_csv( + format_wiki_url(api_key, symbol, start_date, end_date), + parse_dates=['Date'], + index_col='Date', + usecols=[ + 'Open', + 'High', + 'Low', + 'Close', + 'Volume', + 'Date', + 'Ex-Dividend', + 'Split Ratio', + ], + na_values=['NA'], + ).rename(columns={ + 'Open': 'open', + 'High': 'high', + 'Low': 'low', + 'Close': 'close', + 'Volume': 'volume', + 'Date': 'date', + 'Ex-Dividend': 'ex_dividend', + 'Split Ratio': 'split_ratio', + }) + except Exception: + log.exception("Exception raised reading Quandl data. Retrying.") + else: + raise ValueError( + "Failed to download data for %r after %d attempts." % ( + symbol, retries + ) + ) + + +def _update_splits(splits, asset_id, raw_data): + split_ratios = raw_data.split_ratio + df = pd.DataFrame({'ratio': split_ratios[split_ratios != 1]}) + df.index.name = 'effective_date' + df.reset_index(inplace=True) + df['sid'] = asset_id + splits.append(df) + + +def _update_dividends(dividends, asset_id, raw_data): + divs = raw_data.ex_dividend + df = pd.DataFrame({'amount': divs[divs != 0]}) + df.index.name = 'ex_date' + df.reset_index(inplace=True) + df['sid'] = asset_id + # we do not have this data in the WIKI dataset + df['record_date'] = df['declared_date'] = df['pay_date'] = pd.NaT + dividends.append(df) + + +def gen_symbol_data(api_key, + cache, + symbol_map, + calendar, + splits, + dividends, + retries): + start_date = calendar[0] + end_date = calendar[-1] + for asset_id, symbol in symbol_map.iteritems(): + start_time = time() + try: + # see if we have this data cached. + raw_data = cache[symbol] + should_sleep = False + except KeyError: + # we need to fetch the data and then write it to our cache + raw_data = cache[symbol] = fetch_single_equity( + api_key, + symbol, + start_date=start_date, + end_date=end_date, + ) + should_sleep = True + + _update_splits(splits, asset_id, raw_data) + _update_dividends(dividends, asset_id, raw_data) + + raw_data = raw_data.reindex(calendar, copy=False).fillna(0.0) + yield asset_id, raw_data + + if should_sleep: + remaining = seconds_per_call - time() - start_time + if remaining > 0: + sleep(remaining) + + +@bundles.register('quandl') +def quandl_bundle(environ, + asset_db_writer, + minute_bar_writer, # unused + daily_bar_writer, + adjustment_writer, + calendar, + cache, + show_progress): + """Build a zipline data bundle from the Quandl WIKI dataset. + """ + api_key = environ.get('QUANDL_API_KEY') + metadata = fetch_symbol_metadata_frame( + api_key, + cache=cache, + show_progress=show_progress, + ) + symbol_map = metadata.symbol + + # data we will collect in `gen_symbol_data` + splits = [] + dividends = [] + + asset_db_writer.write(metadata) + daily_bar_writer.write( + gen_symbol_data( + api_key, + cache, + symbol_map, + calendar, + splits, + dividends, + environ.get('QUANDL_DOWNLOAD_ATTEMPTS', 5), + ), + ) + adjustment_writer.write( + splits=pd.concat(splits, ignore_index=True), + dividends=pd.concat(dividends, ignore_index=True), + ) diff --git a/zipline/data/bundles/yahoo.py b/zipline/data/bundles/yahoo.py new file mode 100644 index 00000000..12737c40 --- /dev/null +++ b/zipline/data/bundles/yahoo.py @@ -0,0 +1,164 @@ +import os + +import numpy as np +import pandas as pd +from pandas_datareader.data import DataReader +import requests + +from zipline.utils.cli import maybe_show_progress + + +def _cachpath(symbol, type_): + return '-'.join((symbol.replace(os.path.sep, '_'), type_)) + + +def yahoo_equities(symbols, start=None, end=None): + """Create a data bundle ingest function from a set of symbols loaded from + yahoo. + + Parameters + ---------- + symbols : iterable[str] + The ticker symbols to load data for. + start : datetime, optional + The start date to query for. By default this pulls the full history + for the calendar. + end : datetime, optional + The end date to query for. By default this pulls the full history + for the calendar. + + Returns + ------- + ingest : callable + The bundle ingest function for the given set of symbols. + + Examples + -------- + This code should be added to ~/.zipline/extension.py + + .. code-block:: python + + from zipline.data.bundles import yahoo_equities, register + + symbols = ( + 'AAPL', + 'IBM', + 'MSFT', + ) + register('my_bundle', yahoo_equities(symbols)) + + Notes + ----- + The sids for each symbol will be the index into the symbols sequence. + """ + # strict this in memory so that we can reiterate over it + symbols = tuple(symbols) + + def ingest(environ, + asset_db_writer, + minute_bar_writer, # unused + daily_bar_writer, + adjustment_writer, + calendar, + cache, + show_progress, + # pass these as defaults to make them 'nonlocal' in py2 + start=start, + end=end): + if start is None: + start = calendar[0] + if end is None: + end = None + + metadata = pd.DataFrame(np.empty(len(symbols), dtype=[ + ('start_date', 'datetime64[ns]'), + ('end_date', 'datetime64[ns]'), + ('symbol', 'object'), + ])) + + def _pricing_iter(): + sid = 0 + with maybe_show_progress( + symbols, + show_progress, + label='Downloading Yahoo pricing data: ') as it, \ + requests.Session() as session: + for symbol in it: + path = _cachpath(symbol, 'ohlcv') + try: + df = cache[path] + except KeyError: + df = cache[path] = DataReader( + symbol, + 'yahoo', + start, + end, + session=session, + ).sort_index() + + # the start date is the date of the first trade and + # the end date is the date of the last trade + metadata.iloc[sid] = df.index[0], df.index[-1], symbol + df.rename( + columns={ + 'Open': 'open', + 'High': 'high', + 'Low': 'low', + 'Close': 'close', + 'Volume': 'volume', + }, + inplace=True, + ) + yield sid, df + sid += 1 + + daily_bar_writer.write(_pricing_iter(), show_progress=True) + + symbol_map = pd.Series(metadata.symbol.index, metadata.symbol) + asset_db_writer.write(equities=metadata) + + adjustments = [] + with maybe_show_progress( + symbols, + show_progress, + label='Downloading Yahoo adjustment data: ') as it, \ + requests.Session() as session: + for symbol in it: + path = _cachpath(symbol, 'adjustment') + try: + df = cache[path] + except KeyError: + df = cache[path] = DataReader( + symbol, + 'yahoo-actions', + start, + end, + session=session, + ).sort_index() + + df['sid'] = symbol_map[symbol] + adjustments.append(df) + + adj_df = pd.concat(adjustments) + adj_df.index.name = 'date' + adj_df.reset_index(inplace=True) + + splits = adj_df[adj_df.action == 'SPLIT'] + splits = splits.rename( + columns={'value': 'ratio', 'date': 'effective_date'}, + ) + splits.drop('action', axis=1, inplace=True) + + dividends = adj_df[adj_df.action == 'DIVIDEND'] + dividends = dividends.rename( + columns={'value': 'amount', 'date': 'ex_date'}, + ) + dividends.drop('action', axis=1, inplace=True) + # we do not have this data in the yahoo dataset + dividends['record_date'] = pd.NaT + dividends['declared_date'] = pd.NaT + dividends['pay_date'] = pd.NaT + + adjustment_writer.write(splits=splits, dividends=dividends) + + return ingest diff --git a/zipline/data/data_portal.py b/zipline/data/data_portal.py index 457008f3..89c4bef4 100644 --- a/zipline/data/data_portal.py +++ b/zipline/data/data_portal.py @@ -163,8 +163,12 @@ class DailyHistoryAggregator(object): else: after_last = pd.Timestamp( last_visited_dt + self._one_min, tz='UTC') - window = self._minute_reader.unadjusted_window( - ['open'], after_last, dt, [asset])[0] + window = self._minute_reader.load_raw_arrays( + ['open'], + after_last, + dt, + [asset], + )[0] nonnan = window[~pd.isnull(window)] if len(nonnan): val = nonnan[0] @@ -174,8 +178,12 @@ class DailyHistoryAggregator(object): opens.append(val) continue except KeyError: - window = self._minute_reader.unadjusted_window( - ['open'], market_open, dt, [asset])[0] + window = self._minute_reader.load_raw_arrays( + ['open'], + market_open, + dt, + [asset], + )[0] nonnan = window[~pd.isnull(window)] if len(nonnan): val = nonnan[0] @@ -232,15 +240,23 @@ class DailyHistoryAggregator(object): else: after_last = pd.Timestamp( last_visited_dt + self._one_min, tz='UTC') - window = self._minute_reader.unadjusted_window( - ['high'], after_last, dt, [asset]) + window = self._minute_reader.load_raw_arrays( + ['high'], + after_last, + dt, + [asset], + )[0].T val = max(last_max, np.nanmax(window)) entries[asset] = (dt_value, val) highs.append(val) continue except KeyError: - window = self._minute_reader.unadjusted_window( - ['high'], market_open, dt, [asset]) + window = self._minute_reader.load_raw_arrays( + ['high'], + market_open, + dt, + [asset], + )[0].T val = np.nanmax(window) entries[asset] = (dt_value, val) highs.append(val) @@ -288,8 +304,12 @@ class DailyHistoryAggregator(object): else: after_last = pd.Timestamp( last_visited_dt + self._one_min, tz='UTC') - window = self._minute_reader.unadjusted_window( - ['low'], after_last, dt, [asset]) + window = self._minute_reader.load_raw_arrays( + ['low'], + after_last, + dt, + [asset], + )[0].T window_min = np.nanmin(window) if pd.isnull(window_min): val = last_min @@ -299,8 +319,12 @@ class DailyHistoryAggregator(object): lows.append(val) continue except KeyError: - window = self._minute_reader.unadjusted_window( - ['low'], market_open, dt, [asset]) + window = self._minute_reader.load_raw_arrays( + ['low'], + market_open, + dt, + [asset], + )[0].T val = np.nanmin(window) entries[asset] = (dt_value, val) lows.append(val) @@ -410,15 +434,23 @@ class DailyHistoryAggregator(object): else: after_last = pd.Timestamp( last_visited_dt + self._one_min, tz='UTC') - window = self._minute_reader.unadjusted_window( - ['volume'], after_last, dt, [asset]) + window = self._minute_reader.load_raw_arrays( + ['volume'], + after_last, + dt, + [asset], + )[0] val = np.nansum(window) + last_total entries[asset] = (dt_value, val) volumes.append(val) continue except KeyError: - window = self._minute_reader.unadjusted_window( - ['volume'], market_open, dt, [asset]) + window = self._minute_reader.load_raw_arrays( + ['volume'], + market_open, + dt, + [asset], + )[0] val = np.nansum(window) entries[asset] = (dt_value, val) volumes.append(val) diff --git a/zipline/data/loader.py b/zipline/data/loader.py index 56efed99..6b7bbf9b 100644 --- a/zipline/data/loader.py +++ b/zipline/data/loader.py @@ -16,22 +16,20 @@ import os from collections import OrderedDict import logbook - import pandas as pd from pandas.io.data import DataReader import pytz - from six import iteritems from six.moves.urllib_error import HTTPError -from . benchmarks import get_benchmark_returns +from .benchmarks import get_benchmark_returns from . import treasuries, treasuries_can -from .paths import ( +from ..utils.paths import ( cache_root, data_root, ) - -from zipline.utils.tradingcalendar import ( +from ..utils.deprecate import deprecated +from ..utils.tradingcalendar import ( trading_day as trading_day_nyse, trading_days as trading_days_nyse, ) @@ -413,6 +411,10 @@ def load_from_yahoo(indexes=None, return df +@deprecated( + 'load_bars_from_yahoo is deprecated, please register a' + ' yahoo_equities data bundle instead', +) def load_bars_from_yahoo(indexes=None, stocks=None, start=None, diff --git a/zipline/data/minute_bars.py b/zipline/data/minute_bars.py index 35693cfb..83531c96 100644 --- a/zipline/data/minute_bars.py +++ b/zipline/data/minute_bars.py @@ -92,6 +92,22 @@ def _sid_subdir_path(sid): class BcolzMinuteBarMetadata(object): + """ + Parameters + ---------- + first_trading_day : datetime-like + UTC midnight of the first day available in the dataset. + minute_index : pd.DatetimeIndex + The minutes which act as an index into the corresponding values + written into each sid's ctable. + market_opens : pd.DatetimeIndex + The market opens for each day in the data set. (Not yet required.) + market_closes : pd.DatetimeIndex + The market closes for each day in the data set. (Not yet required.) + ohlc_ratio : int + The factor by which the pricing data is multiplied so that the + float data can be stored as an integer. + """ METADATA_FILENAME = 'metadata.json' @@ -123,22 +139,6 @@ class BcolzMinuteBarMetadata(object): market_opens, market_closes, ohlc_ratio): - """ - Parameters: - ----------- - first_trading_day : datetime-like - UTC midnight of the first day available in the dataset. - minute_index : pd.DatetimeIndex - The minutes which act as an index into the corresponding values - written into each sid's ctable. - market_opens : pd.DatetimeIndex - The market opens for each day in the data set. (Not yet required.) - market_closes : pd.DatetimeIndex - The market closes for each day in the data set. (Not yet required.) - ohlc_ratio : int - The factor by which the pricing data is multiplied so that the - float data can be stored as an integer. - """ self.first_trading_day = first_trading_day self.market_opens = market_opens self.market_closes = market_closes @@ -177,6 +177,55 @@ class BcolzMinuteBarWriter(object): """ Class capable of writing minute OHLCV data to disk into bcolz format. + Parameters + ---------- + first_trading_day : datetime + The first trading day in the data set. + rootdir : string + Path to the root directory into which to write the metadata and + bcolz subdirectories. + market_opens : pd.Series + The market opens used as a starting point for each periodic span of + minutes in the index. + + The index of the series is expected to be a DatetimeIndex of the + UTC midnight of each trading day. + + The values are datetime64-like UTC market opens for each day in the + index. + market_closes : pd.Series + The market closes that correspond with the market opens, + + The index of the series is expected to be a DatetimeIndex of the + UTC midnight of each trading day. + + The values are datetime64-like UTC market opens for each day in the + index. + + The closes are written so that the reader can filter out non-market + minutes even though the tail end of early closes are written in + the data arrays to keep a regular shape. + minutes_per_day : int + The number of minutes per each period. Defaults to 390, the mode + of minutes in NYSE trading days. + ohlc_ratio : int, optional + The ratio by which to multiply the pricing data to convert the + floats from floats to an integer to fit within the np.uint32. + + The default is 1000 to support pricing data which comes in to the + thousands place. + expectedlen : int, optional + The expected length of the dataset, used when creating the initial + bcolz ctable. + + If the expectedlen is not used, the chunksize and corresponding + compression ratios are not ideal. + + Defaults to supporting 15 years of NYSE equity market data. + see: http://bcolz.blosc.org/opt-tips.html#informing-about-the-length-of-your-carrays # noqa + + Notes + ----- Writes a bcolz directory for each individual sid, all contained within a root directory which also contains metadata about the entire dataset. @@ -215,6 +264,10 @@ class BcolzMinuteBarWriter(object): The datetimes which correspond to each position are written in the metadata as integer nanoseconds since the epoch into the `minute_index` key. + + See Also + -------- + zipline.data.minute_bars.BcolzMinuteBarReader """ COL_NAMES = ('open', 'high', 'low', 'close', 'volume') @@ -226,61 +279,6 @@ class BcolzMinuteBarWriter(object): minutes_per_day, ohlc_ratio=OHLC_RATIO, expectedlen=DEFAULT_EXPECTEDLEN): - """ - Parameters: - ----------- - first_trading_day : datetime-like - The first trading day in the data set. - - rootdir : string - Path to the root directory into which to write the metadata and - bcolz subdirectories. - - market_opens : pd.Series - The market opens used as a starting point for each periodic span of - minutes in the index. - - The index of the series is expected to be a DatetimeIndex of the - UTC midnight of each trading day. - - The values are datetime64-like UTC market opens for each day in the - index. - - market_closes : pd.Series - The market closes that correspond with the market opens, - - The index of the series is expected to be a DatetimeIndex of the - UTC midnight of each trading day. - - The values are datetime64-like UTC market opens for each day in the - index. - - The closes are written so that the reader can filter out non-market - minutes even though the tail end of early closes are written in - the data arrays to keep a regular shape. - - minutes_per_day : int - The number of minutes per each period. Defaults to 390, the mode - of minutes in NYSE trading days. - - ohlc_ratio : int - The ratio by which to multiply the pricing data to convert the - floats from floats to an integer to fit within the np.uint32. - - The default is 1000 to support pricing data which comes in to the - thousands place. - - expectedlen : int - The expected length of the dataset, used when creating the initial - bcolz ctable. - - If the expectedlen is not used, the chunksize and corresponding - compression ratios are not ideal. - - Defaults to supporting 15 years of NYSE equity market data. - - see: http://bcolz.blosc.org/opt-tips.html#informing-about-the-length-of-your-carrays # noqa - """ self._rootdir = rootdir self._first_trading_day = first_trading_day self._market_opens = market_opens[ @@ -450,13 +448,15 @@ class BcolzMinuteBarWriter(object): data : iterable[(int, pd.DataFrame)] The data to write. Each element should be a tuple of sid, data where data has the following format: - columns : ('open', 'high', 'low', 'close', 'volume') - open : float64 - high : float64 - low : float64 - close : float64 - volume : float64|int64 - index : DatetimeIndex of market minutes. + columns : ('open', 'high', 'low', 'close', 'volume') + open : float64 + high : float64 + low : float64 + close : float64 + volume : float64|int64 + index : DatetimeIndex of market minutes. + A given sid may appear more than once in ``data``; however, + the dates must be strictly increasing. show_progress : bool, optional Whether or not to show a progress bar while writing. """ @@ -615,17 +615,21 @@ class BcolzMinuteBarWriter(object): class BcolzMinuteBarReader(object): + """ + Reader for data written by BcolzMinuteBarWriter + Parameters: + ----------- + rootdir : string + The root directory containing the metadata and asset bcolz + directories. + + See Also + -------- + zipline.data.minute_bars.BcolzMinuteBarWriter + """ def __init__(self, rootdir): - """ - Reader for data written by BcolzMinuteBarWriter - Parameters: - ----------- - rootdir : string - The root directory containing the metadata and asset bcolz - directories. - """ self._rootdir = rootdir metadata = self._get_metadata() @@ -856,7 +860,7 @@ class BcolzMinuteBarReader(object): US_EQUITIES_MINUTES_PER_DAY, ) - def unadjusted_window(self, fields, start_dt, end_dt, sids): + def load_raw_arrays(self, fields, start_dt, end_dt, sids): """ Parameters ---------- @@ -873,7 +877,7 @@ class BcolzMinuteBarReader(object): ------- list of np.ndarray A list with an entry per field of ndarrays with shape - (sids, minutes in range) with a dtype of float64, containing the + (minutes in range, sids) with a dtype of float64, containing the values for the respective field over start and end dt range. """ start_idx = self._find_position_of_minute(start_dt) @@ -890,7 +894,7 @@ class BcolzMinuteBarReader(object): length = excl_stop - excl_start + 1 num_minutes -= length - shape = (len(sids), num_minutes) + shape = num_minutes, len(sids) for field in fields: if field != 'volume': @@ -907,7 +911,9 @@ class BcolzMinuteBarReader(object): excl_start - start_idx:excl_stop - start_idx + 1] values = np.delete(values, excl_slice) where = values != 0 - out[i, where] = values[where] + # first slice down to len(where) because we might not have + # written data for all the minutes requested + out[:len(where), i][where] = values[where] if field != 'volume': out *= self._ohlc_inverse results.append(out) diff --git a/zipline/data/paths.py b/zipline/data/paths.py deleted file mode 100644 index 59b0ab98..00000000 --- a/zipline/data/paths.py +++ /dev/null @@ -1,91 +0,0 @@ -""" -Canonical path locations for zipline data. - -Paths are rooted at $ZIPLINE_ROOT if that environment variable is set. -Otherwise default to expanduser(~/.zipline) -""" -import os -from os.path import ( - expanduser, - join, -) - - -def zipline_root(environ=None): - """ - Get the root directory for all zipline-managed files. - - For testing purposes, this accepts a dictionary to interpret as the os - environment. - - Parameters - ---------- - environ : dict, optional - A dict to interpret as the os environment. - - Returns - ------- - root : string - Path to the zipline root dir. - """ - if environ is None: - environ = os.environ.copy() - - root = environ.get('ZIPLINE_ROOT', None) - if root is None: - root = expanduser('~/.zipline') - - return root - - -def zipline_root_path(path, environ=None): - """ - Get a path relative to the zipline root. - - Parameters - ---------- - path : str - The requested path. - environ : dict, optional - An environment dict to forward to zipline_root. - - Returns - ------- - newpath : str - The requested path joined with the zipline root. - """ - return join(zipline_root(environ=environ), path) - - -def data_root(environ=None): - """ - The root directory for zipline data files. - - Parameters - ---------- - environ : dict, optional - An environment dict to forward to zipline_root. - - Returns - ------- - data_root : str - The zipline data root. - """ - return zipline_root_path('data', environ=environ) - - -def cache_root(environ=None): - """ - The root directory for zipline cache files. - - Parameters - ---------- - environ : dict, optional - An environment dict to forward to zipline_root. - - Returns - ------- - cache_root : str - The zipline cache root. - """ - return zipline_root_path('cache', environ=environ) diff --git a/zipline/data/us_equity_loader.py b/zipline/data/us_equity_loader.py index f3392b63..5385b203 100644 --- a/zipline/data/us_equity_loader.py +++ b/zipline/data/us_equity_loader.py @@ -24,7 +24,6 @@ from pandas.tslib import normalize_date from six import with_metaclass -from zipline.pipeline.data.equity_pricing import USEquityPricing from zipline.lib._float64window import AdjustedArrayWindow as Float64Window from zipline.lib.adjustment import Float64Multiply from zipline.utils.cache import ExpiringCache @@ -300,9 +299,12 @@ class USEquityDailyHistoryLoader(USEquityHistoryLoader): return self._reader._calendar def _array(self, dts, assets, field): - col = getattr(USEquityPricing, field) return self._reader.load_raw_arrays( - [col], dts[0], dts[-1], assets)[0] + [field], + dts[0], + dts[-1], + assets, + )[0] class USEquityMinuteHistoryLoader(USEquityHistoryLoader): @@ -318,5 +320,9 @@ class USEquityMinuteHistoryLoader(USEquityHistoryLoader): end=self._reader.last_available_dt)] def _array(self, dts, assets, field): - return self._reader.unadjusted_window( - [field], dts[0], dts[-1], assets)[0].T + return self._reader.load_raw_arrays( + [field], + dts[0], + dts[-1], + assets, + )[0] diff --git a/zipline/data/us_equity_pricing.py b/zipline/data/us_equity_pricing.py index ba02dc15..fbfb1a06 100644 --- a/zipline/data/us_equity_pricing.py +++ b/zipline/data/us_equity_pricing.py @@ -22,7 +22,6 @@ import warnings from bcolz import ( carray, ctable, - open as open_ctable, ) from collections import namedtuple import logbook @@ -193,7 +192,7 @@ class BcolzDailyBarWriter(object): See Also -------- - BcolzDailyBarReader : Consumer of the data written by this class. + zipline.data.us_equity_pricing.BcolzDailyBarReader """ _csv_dtypes = { 'open': float64, @@ -365,6 +364,7 @@ class BcolzDailyBarWriter(object): full_table.attrs['last_row'] = last_row full_table.attrs['calendar_offset'] = calendar_offset full_table.attrs['calendar'] = calendar.asi8.tolist() + full_table.flush() return full_table @@ -451,11 +451,13 @@ class BcolzDailyBarReader(DailyBarReader): We use calendar_offset and calendar to orient loaded blocks within a range of queried dates. - """ - @preprocess(table=coerce_string(open_ctable, mode='r')) - def __init__(self, table, read_all_threshold=3000): - self._table = table + See Also + -------- + zipline.data.us_equity_pricing.BcolzDailyBarWriter + """ + def __init__(self, table, read_all_threshold=3000): + self._maybe_table_rootdir = table # Cache of fully read np.array for the carrays in the daily bar table. # raw_array does not use the same cache, but it could. # Need to test keeping the entire array in memory for the course of a @@ -464,6 +466,13 @@ class BcolzDailyBarReader(DailyBarReader): self.PRICE_ADJUSTMENT_FACTOR = 0.001 self._read_all_threshold = read_all_threshold + @lazyval + def _table(self): + maybe_table_rootdir = self._maybe_table_rootdir + if isinstance(maybe_table_rootdir, ctable): + return maybe_table_rootdir + return ctable(rootdir=maybe_table_rootdir, mode='r') + @lazyval def _calendar(self): return DatetimeIndex(self._table.attrs['calendar'], tz='UTC') @@ -565,7 +574,7 @@ class BcolzDailyBarReader(DailyBarReader): return _read_bcolz_data( self._table, (end_idx - start_idx + 1, len(assets)), - [column.name for column in columns], + list(columns), first_rows, last_rows, offsets, @@ -717,12 +726,12 @@ class PanelDailyBarReader(DailyBarReader): return self._calendar[-1] def load_raw_arrays(self, columns, start_date, end_date, assets): - col_names = [col.name for col in columns] + columns = list(columns) cal = self._calendar index = cal[cal.slice_indexer(start_date, end_date)] shape = (len(index), len(assets)) results = [] - for col in col_names: + for col in columns: outbuf = zeros(shape=shape) for i, asset in enumerate(assets): data = self.panel.loc[asset, start_date:end_date, col] @@ -792,7 +801,7 @@ class SQLiteAdjustmentWriter(object): See Also -------- - SQLiteAdjustmentReader + zipline.data.us_equity_pricing.SQLiteAdjustmentReader """ def __init__(self, @@ -862,6 +871,11 @@ class SQLiteAdjustmentWriter(object): SQLITE_ADJUSTMENT_TABLENAMES, ) ) + if not (frame is None or frame.empty): + frame = frame.copy() + frame['effective_date'] = frame['effective_date'].values.astype( + 'datetime64[s]', + ).astype('int64') return self._write( tablename, SQLITE_ADJUSTMENT_COLUMN_DTYPES, @@ -1016,82 +1030,72 @@ class SQLiteAdjustmentWriter(object): Parameters ---------- - splits : pandas.DataFrame - Dataframe containing split data. - mergers : pandas.DataFrame - DataFrame containing merger data. - dividends : pandas.DataFrame - DataFrame containing dividend data. + splits : pandas.DataFrame, optional + Dataframe containing split data. The format of this dataframe is: + effective_date : int + The date, represented as seconds since Unix epoch, on which + the adjustment should be applied. + ratio : float + A value to apply to all data earlier than the effective date. + For open, high, low, and close those values are multiplied by + the ratio. Volume is divided by this value. + sid : int + The asset id associated with this adjustment. + mergers : pandas.DataFrame, optional + DataFrame containing merger data. The format of this dataframe is: + effective_date : int + The date, represented as seconds since Unix epoch, on which + the adjustment should be applied. + ratio : float + A value to apply to all data earlier than the effective date. + For open, high, low, and close those values are multiplied by + the ratio. Volume is unaffected. + sid : int + The asset id associated with this adjustment. + dividends : pandas.DataFrame, optional + DataFrame containing dividend data. The format of the dataframe is: + sid : int + The asset id associated with this adjustment. + ex_date : datetime64 + The date on which an equity must be held to be eligible to + receive payment. + declared_date : datetime64 + The date on which the dividend is announced to the public. + pay_date : datetime64 + The date on which the dividend is distributed. + record_date : datetime64 + The date on which the stock ownership is checked to determine + distribution of dividends. + amount : float + The cash amount paid for each share. - Notes - ----- - DataFrame input (`splits`, `mergers`) should all have - the following columns: - - effective_date : int - The date, represented as seconds since Unix epoch, on which the - adjustment should be applied. - ratio : float - A value to apply to all data earlier than the effective date. - sid : int - The asset id associated with this adjustment. - - The ratio column is interpreted as follows: - - For all adjustment types, multiply price fields ('open', 'high', - 'low', and 'close') by the ratio. - - For **splits only**, **divide** volume by the adjustment ratio. - - DataFrame input, 'dividends' should have the following columns: - - sid : int - The asset id associated with this adjustment. - ex_date : datetime64 - The date on which an equity must be held to be eligible to receive - payment. - declared_date : datetime64 - The date on which the dividend is announced to the public. - pay_date : datetime64 - The date on which the dividend is distributed. - record_date : datetime64 - The date on which the stock ownership is checked to determine - distribution of dividends. - amount : float - The cash amount paid for each share. - - Dividend ratios are calculated as - 1.0 - (dividend_value / "close on day prior to dividend ex_date"). - - - DataFrame input, 'stock_dividends' should have the following columns: - - sid : int - The asset id associated with this adjustment. - ex_date : datetime64 - The date on which an equity must be held to be eligible to receive - payment. - declared_date : datetime64 - The date on which the dividend is announced to the public. - pay_date : datetime64 - The date on which the dividend is distributed. - record_date : datetime64 - The date on which the stock ownership is checked to determine - distribution of dividends. - payment_sid : int - The asset id of the shares that should be paid instead of cash. - ratio: float - The ratio of currently held shares in the held sid that should - be paid with new shares of the payment_sid. - - stock_dividends is optional. - - - Returns - ------- - None + Dividend ratios are calculated as: + ``1.0 - (dividend_value / "close on day prior to ex_date")`` + stock_dividends : pandas.DataFrame, optional + DataFrame containing stock dividend data. The format of the + dataframe is: + sid : int + The asset id associated with this adjustment. + ex_date : datetime64 + The date on which an equity must be held to be eligible to + receive payment. + declared_date : datetime64 + The date on which the dividend is announced to the public. + pay_date : datetime64 + The date on which the dividend is distributed. + record_date : datetime64 + The date on which the stock ownership is checked to determine + distribution of dividends. + payment_sid : int + The asset id of the shares that should be paid instead of + cash. + ratio : float + The ratio of currently held shares in the held sid that + should be paid with new shares of the payment_sid. See Also -------- - SQLiteAdjustmentReader : Consumer for the data written by this class + zipline.data.us_equity_pricing.SQLiteAdjustmentReader """ self.write_frame('splits', splits) self.write_frame('mergers', mergers) @@ -1168,6 +1172,10 @@ class SQLiteAdjustmentReader(object): ---------- conn : str or sqlite3.Connection Connection from which to load data. + + See Also + -------- + zipline.data.us_equity_pricing.SQLiteAdjustmentWriter """ @preprocess(conn=coerce_string(sqlite3.connect)) @@ -1177,7 +1185,7 @@ class SQLiteAdjustmentReader(object): def load_adjustments(self, columns, dates, assets): return load_adjustments_from_sqlite( self.conn, - [column.name for column in columns], + list(columns), dates, assets, ) diff --git a/zipline/examples/__init__.py b/zipline/examples/__init__.py new file mode 100644 index 00000000..62e5ea33 --- /dev/null +++ b/zipline/examples/__init__.py @@ -0,0 +1,17 @@ +from glob import glob +from importlib import import_module +import os + +for f in os.listdir(os.path.dirname(__file__)): + if not f.endswith('.py') or f == '__init__.py': + continue + modname = f[:-len('.py')] + globals()[modname] = import_module('.' + modname, package=__name__) + +del f +try: + del modname +except NameError: + pass + +del os, import_module, glob diff --git a/zipline/examples/buy_and_hold.py b/zipline/examples/buy_and_hold.py index 69af2159..fbd6b2f8 100644 --- a/zipline/examples/buy_and_hold.py +++ b/zipline/examples/buy_and_hold.py @@ -13,10 +13,7 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. -import pandas as pd -from zipline import TradingAlgorithm from zipline.api import order, symbol -from zipline.data.loader import load_bars_from_yahoo stocks = ['AAPL', 'MSFT'] @@ -33,18 +30,12 @@ def handle_data(context, data): context.has_ordered = True -if __name__ == '__main__': +def _test_args(): + """Extra arguments to use when zipline's automated tests run this example. + """ + import pandas as pd - # creating time interval - start = pd.Timestamp('2008-01-01', tz='UTC') - end = pd.Timestamp('2013-01-01', tz='UTC') - - # loading the data - input_data = load_bars_from_yahoo( - stocks=stocks, - start=start, - end=end, - ) - - algo = TradingAlgorithm(initialize=initialize, handle_data=handle_data) - results = algo.run(input_data) + return { + 'start': pd.Timestamp('2008', tz='utc'), + 'end': pd.Timestamp('2013', tz='utc'), + } diff --git a/zipline/examples/buyapple.conf b/zipline/examples/buyapple.conf deleted file mode 100644 index 42ce1cac..00000000 --- a/zipline/examples/buyapple.conf +++ /dev/null @@ -1,3 +0,0 @@ -[Defaults] -algofile=buyapple.py -symbols=AAPL diff --git a/zipline/examples/buyapple.py b/zipline/examples/buyapple.py old mode 100755 new mode 100644 index 2ea1f0af..df7d9486 --- a/zipline/examples/buyapple.py +++ b/zipline/examples/buyapple.py @@ -23,7 +23,7 @@ def initialize(context): def handle_data(context, data): order(symbol('AAPL'), 10) - record(AAPL=data.current(symbol('AAPL'), "price")) + record(AAPL=data.current(symbol('AAPL'), 'price')) # Note: this function can be removed if running @@ -43,24 +43,12 @@ def analyze(context=None, results=None): plt.show() -# Note: this if-block should be removed if running -# this algorithm on quantopian.com -if __name__ == '__main__': - from datetime import datetime - import pytz - from zipline.algorithm import TradingAlgorithm - from zipline.utils.factory import load_from_yahoo +def _test_args(): + """Extra arguments to use when zipline's automated tests run this example. + """ + import pandas as pd - # Set the simulation start and end dates - start = datetime(2014, 1, 1, 0, 0, 0, 0, pytz.utc) - end = datetime(2014, 11, 1, 0, 0, 0, 0, pytz.utc) - - # Load price data from yahoo. - data = load_from_yahoo(stocks=['AAPL'], indexes={}, start=start, - end=end) - - # Create and run the algorithm. - algo = TradingAlgorithm(initialize=initialize, handle_data=handle_data) - results = algo.run(data) - - analyze(results=results) + return { + 'start': pd.Timestamp('2014-01-01', tz='utc'), + 'end': pd.Timestamp('2014-11-01', tz='utc'), + } diff --git a/zipline/examples/dual_ema_talib.py b/zipline/examples/dual_ema_talib.py old mode 100755 new mode 100644 index 92b1b41b..cbdc9957 --- a/zipline/examples/dual_ema_talib.py +++ b/zipline/examples/dual_ema_talib.py @@ -98,25 +98,12 @@ def analyze(context=None, results=None): plt.show() -# Note: this if-block should be removed if running -# this algorithm on quantopian.com -if __name__ == '__main__': - from datetime import datetime - import pytz - from zipline.algorithm import TradingAlgorithm - from zipline.utils.factory import load_from_yahoo +def _test_args(): + """Extra arguments to use when zipline's automated tests run this example. + """ + import pandas as pd - # Set the simulation start and end dates. - start = datetime(2014, 1, 1, 0, 0, 0, 0, pytz.utc) - end = datetime(2014, 11, 1, 0, 0, 0, 0, pytz.utc) - - # Load price data from yahoo. - data = load_from_yahoo(stocks=['AAPL'], indexes={}, start=start, - end=end) - - # Create and run the algorithm. - algo = TradingAlgorithm(initialize=initialize, handle_data=handle_data) - results = algo.run(data).dropna() - - # Plot the portfolio and asset data. - analyze(results=results) + return { + 'start': pd.Timestamp('2014-01-01', tz='utc'), + 'end': pd.Timestamp('2014-11-01', tz='utc'), + } diff --git a/zipline/examples/dual_moving_average.conf b/zipline/examples/dual_moving_average.conf deleted file mode 100644 index dfe0a805..00000000 --- a/zipline/examples/dual_moving_average.conf +++ /dev/null @@ -1,5 +0,0 @@ -[Defaults] -algofile=dual_moving_average.py -symbols=AAPL -start=2000-1-1 -end=2014-1-1 diff --git a/zipline/examples/dual_moving_average.py b/zipline/examples/dual_moving_average.py old mode 100755 new mode 100644 index 55c877c4..529b497a --- a/zipline/examples/dual_moving_average.py +++ b/zipline/examples/dual_moving_average.py @@ -97,29 +97,12 @@ def analyze(context=None, results=None): plt.show() -# Note: this if-block should be removed if running -# this algorithm on quantopian.com -if __name__ == '__main__': - from datetime import datetime - import pytz - from zipline.algorithm import TradingAlgorithm - from zipline.utils.factory import load_bars_from_yahoo +def _test_args(): + """Extra arguments to use when zipline's automated tests run this example. + """ + import pandas as pd - # Set the simulation start and end dates. - start = datetime(2011, 1, 1, 0, 0, 0, 0, pytz.utc) - end = datetime(2013, 1, 1, 0, 0, 0, 0, pytz.utc) - - # Load price data from yahoo. - data = load_bars_from_yahoo( - stocks=['AAPL'], - indexes={}, - start=start, - end=end, - ) - - # Create and run the algorithm. - algo = TradingAlgorithm(initialize=initialize, handle_data=handle_data) - results = algo.run(data) - - # Plot the portfolio and asset data. - analyze(results=results) + return { + 'start': pd.Timestamp('2011', tz='utc'), + 'end': pd.Timestamp('2013', tz='utc'), + } diff --git a/zipline/examples/olmar.py b/zipline/examples/olmar.py index ca116e13..e041c778 100644 --- a/zipline/examples/olmar.py +++ b/zipline/examples/olmar.py @@ -1,11 +1,7 @@ import sys import logbook import numpy as np -from datetime import datetime -import pytz -from zipline.algorithm import TradingAlgorithm -from zipline.utils.factory import load_from_yahoo from zipline.finance import commission zipline_logging = logbook.NestedSetup([ @@ -161,20 +157,12 @@ def analyze(context=None, results=None): plt.show() -# Note: this if-block should be removed if running -# this algorithm on quantopian.com -if __name__ == '__main__': - # Set the simulation start and end dates. - start = datetime(2004, 1, 1, 0, 0, 0, 0, pytz.utc) - end = datetime(2008, 1, 1, 0, 0, 0, 0, pytz.utc) +def _test_args(): + """Extra arguments to use when zipline's automated tests run this example. + """ + import pandas as pd - # Load price data from yahoo. - data = load_from_yahoo(stocks=STOCKS, indexes={}, start=start, end=end) - data = data.dropna() - - # Create and run the algorithm. - olmar = TradingAlgorithm(handle_data=handle_data, initialize=initialize) - results = olmar.run(data) - - # Plot the portfolio data. - analyze(results=results) + return { + 'start': pd.Timestamp('2004', tz='utc'), + 'end': pd.Timestamp('2008', tz='utc'), + } diff --git a/zipline/pipeline/loaders/equity_pricing_loader.py b/zipline/pipeline/loaders/equity_pricing_loader.py index 64448903..60939da6 100644 --- a/zipline/pipeline/loaders/equity_pricing_loader.py +++ b/zipline/pipeline/loaders/equity_pricing_loader.py @@ -69,15 +69,15 @@ class USEquityPricingLoader(PipelineLoader): start_date, end_date = _shift_dates( self._calendar, dates[0], dates[-1], shift=1, ) - + colnames = [c.name for c in columns] raw_arrays = self.raw_price_loader.load_raw_arrays( - columns, + colnames, start_date, end_date, assets, ) adjustments = self.adjustments_loader.load_adjustments( - columns, + colnames, dates, assets, ) diff --git a/zipline/pipeline/loaders/synthetic.py b/zipline/pipeline/loaders/synthetic.py index 913dac82..f9957ae9 100644 --- a/zipline/pipeline/loaders/synthetic.py +++ b/zipline/pipeline/loaders/synthetic.py @@ -213,7 +213,7 @@ def asset_end(asset_info, asset): return ret -def make_daily_bar_data(asset_info, calendar): +def make_bar_data(asset_info, calendar): """ For a given asset/date/column combination, we generate a corresponding raw @@ -249,7 +249,7 @@ def make_daily_bar_data(asset_info, calendar): assert ( # Using .value here to avoid having to care about UTC-aware dates. PSEUDO_EPOCH.value < - calendar.min().value <= + calendar.normalize().min().value <= asset_info['start_date'].min().value ), "calendar.min(): %s\nasset_info['start_date'].min(): %s" % ( calendar.min(), @@ -266,13 +266,13 @@ def make_daily_bar_data(asset_info, calendar): """ # Get the dates for which this asset existed according to our asset # info. - dates = calendar[calendar.slice_indexer( + datetimes = calendar[calendar.slice_indexer( asset_start(asset_info, asset_id), asset_end(asset_info, asset_id), )] data = full( - (len(dates), len(US_EQUITY_PRICING_BCOLZ_COLUMNS)), + (len(datetimes), len(US_EQUITY_PRICING_BCOLZ_COLUMNS)), asset_id * 100 * 1000, dtype=uint32, ) @@ -281,15 +281,15 @@ def make_daily_bar_data(asset_info, calendar): data[:, :5] += arange(5, dtype=uint32) * 1000 # Add days since Jan 1 2001 for OHLCV columns. - data[:, :5] += (dates - PSEUDO_EPOCH).days[:, None].astype(uint32) + data[:, :5] += (datetimes - PSEUDO_EPOCH).days[:, None].astype(uint32) frame = DataFrame( data, - index=dates, + index=datetimes, columns=US_EQUITY_PRICING_BCOLZ_COLUMNS, ) - frame['day'] = nanos_to_seconds(dates.asi8) + frame['day'] = nanos_to_seconds(datetimes.asi8) frame['id'] = asset_id return frame @@ -297,7 +297,7 @@ def make_daily_bar_data(asset_info, calendar): yield asset, _raw_data_for_asset(asset) -def expected_daily_bar_value(asset_id, date, colname): +def expected_bar_value(asset_id, date, colname): """ Check that the raw value for an asset/date/column triple is as expected. @@ -310,7 +310,7 @@ def expected_daily_bar_value(asset_id, date, colname): return from_asset + from_colname + from_date -def expected_daily_bar_values_2d(dates, asset_info, colname): +def expected_bar_values_2d(dates, asset_info, colname): """ Return an 2D array containing cls.expected_value(asset_id, date, colname) for each date/asset pair in the inputs. @@ -336,7 +336,7 @@ def expected_daily_bar_values_2d(dates, asset_info, colname): # date. if not (start <= date <= end): continue - data[i, j] = expected_daily_bar_value(asset, date, colname) + data[i, j] = expected_bar_value(asset, date, colname) return data diff --git a/zipline/testing/__init__.py b/zipline/testing/__init__.py index 5106cac5..a415daa8 100644 --- a/zipline/testing/__init__.py +++ b/zipline/testing/__init__.py @@ -29,6 +29,7 @@ from .core import ( # noqa parameter_space, parameter_space, patch_os_environment, + patch_read_csv, permute_rows, powerset, product_upper_triangle, diff --git a/zipline/testing/core.py b/zipline/testing/core.py index 6d7a4a05..11430470 100644 --- a/zipline/testing/core.py +++ b/zipline/testing/core.py @@ -1178,6 +1178,7 @@ zipline_git_root = abspath( ) +@nottest def test_resource_path(*path_parts): return os.path.join(zipline_git_root, 'tests', 'resources', *path_parts) @@ -1308,3 +1309,37 @@ class tmp_bcolz_daily_bar_reader(_TmpBarReader): @staticmethod def _write(env, days, path, data): BcolzDailyBarWriter(path, days).write(data) + + +@contextmanager +def patch_read_csv(url_map, module=pd, strict=False): + """Patch pandas.read_csv to map lookups from url to another. + + Parameters + ---------- + url_map : mapping[str or file-like object -> str or file-like object] + The mapping to use to redirect read_csv calls. + module : module, optional + The module to patch ``read_csv`` on. By default this is ``pandas``. + This should be set to another module if ``read_csv`` is early-bound + like ``from pandas import read_csv`` instead of late-bound like: + ``import pandas as pd; pd.read_csv``. + strict : bool, optional + If true, then this will assert that ``read_csv`` is only called with + elements in the ``url_map``. + """ + read_csv = pd.read_csv + + def patched_read_csv(filepath_or_buffer, *args, **kwargs): + if filepath_or_buffer in url_map: + return read_csv(url_map[filepath_or_buffer], *args, **kwargs) + elif not strict: + return read_csv(filepath_or_buffer, *args, **kwargs) + else: + raise AssertionError( + 'attempted to call read_csv on %r which not in the url map' % + filepath_or_buffer, + ) + + with patch.object(module, 'read_csv', patched_read_csv): + yield diff --git a/zipline/testing/predicates.py b/zipline/testing/predicates.py index 6dd6f8d9..93f3610e 100644 --- a/zipline/testing/predicates.py +++ b/zipline/testing/predicates.py @@ -1,7 +1,89 @@ +from functools import partial +import inspect + +from nose.tools import ( # noqa + assert_almost_equal, + assert_almost_equals, + assert_dict_contains_subset, + assert_false, + assert_greater, + assert_greater_equal, + assert_in, + assert_is, + assert_is_instance, + assert_is_none, + assert_is_not, + assert_is_not_none, + assert_less, + assert_less_equal, + assert_multi_line_equal, + assert_not_almost_equal, + assert_not_almost_equals, + assert_not_equal, + assert_not_equals, + assert_not_in, + assert_not_is_instance, + assert_raises, + assert_raises_regexp, + assert_regexp_matches, + assert_sequence_equal, + assert_set_equal, + assert_true, + assert_tuple_equal, +) +import numpy as np +import pandas as pd +from pandas.util.testing import assert_frame_equal from six import iteritems, viewkeys, PY2 +from toolz import dissoc, keyfilter +import toolz.curried.operator as op from zipline.dispatch import dispatch +from zipline.lib.adjustment import Adjustment from zipline.utils.functional import dzip_exact +from zipline.utils.math_utils import tolerant_equals + + +def keywords(func): + """Get the argument names of a function + + >>> def f(x, y=2): + ... pass + + >>> keywords(f) + ['x', 'y'] + + Notes + ----- + Taken from odo.utils + """ + if isinstance(func, type): + return keywords(func.__init__) + return inspect.getargspec(func).args + + +def filter_kwargs(f, kwargs): + """Return a dict of valid kwargs for `f` from a subset of `kwargs` + + Examples + -------- + >>> def f(a, b=1, c=2): + ... return a + b + c + ... + >>> raw_kwargs = dict(a=1, b=3, d=4) + >>> f(**raw_kwargs) + Traceback (most recent call last): + ... + TypeError: f() got an unexpected keyword argument 'd' + >>> kwargs = filter_kwargs(f, raw_kwargs) + >>> f(**kwargs) + 6 + + Notes + ----- + Taken from odo.utils + """ + return keyfilter(op.contains(keywords(f)), kwargs) def _s(word, seq, suffix='s'): @@ -41,8 +123,26 @@ def _fmt_path(path): return 'path: _' + ''.join(path) +def _fmt_msg(msg): + """Format the message for final display. + + Parameters + ---------- + msg : str + The message to show to the user to provide additional context. + + returns + ------- + fmtd : str + The formatted message to put into the error message. + """ + if not msg: + return '' + return msg + '\n' + + @dispatch(object, object) -def assert_equal(result, expected, path=(), **kwargs): +def assert_equal(result, expected, path=(), msg='', **kwargs): """Assert that two objects are equal using the ``==`` operator. Parameters @@ -57,15 +157,42 @@ def assert_equal(result, expected, path=(), **kwargs): AssertionError Raised when ``result`` is not equal to ``expected``. """ - assert result == expected, '%s != %s\n%s' % ( + assert result == expected, '%s%s != %s\n%s' % ( + _fmt_msg(msg), result, expected, _fmt_path(path), ) +@assert_equal.register(float, float) +def assert_float_equal(result, + expected, + path=(), + msg='', + float_rtol=10e-7, + float_atol=10e-7, + float_equal_nan=True, + **kwargs): + assert tolerant_equals( + result, + expected, + rtol=float_rtol, + atol=float_atol, + equal_nan=float_equal_nan, + ), '%s%s != %s with rtol=%s and atol=%s%s\n%s' % ( + _fmt_msg(msg), + result, + expected, + float_rtol, + float_atol, + (' (with nan != nan)' if not float_equal_nan else ''), + _fmt_path(path), + ) + + @assert_equal.register(dict, dict) -def assert_dict_equal(result, expected, path=(), **kwargs): +def assert_dict_equal(result, expected, path=(), msg='', **kwargs): if path is None: path = () @@ -89,8 +216,8 @@ def assert_dict_equal(result, expected, path=(), **kwargs): in_expected, ) raise AssertionError( - 'dict keys do not match\n%s\n%s' % ( - msg, + '%sdict keys do not match\n%s' % ( + _fmt_msg(msg), _fmt_path(path + ('.%s()' % ('viewkeys' if PY2 else 'keys'),)), ), ) @@ -102,6 +229,7 @@ def assert_dict_equal(result, expected, path=(), **kwargs): resultv, expectedv, path=path + ('[%r]' % k,), + msg=msg, **kwargs ) except AssertionError as e: @@ -112,14 +240,16 @@ def assert_dict_equal(result, expected, path=(), **kwargs): @assert_equal.register(list, list) # noqa -def assert_list_equal(result, expected, path=(), **kwargs): +def assert_list_equal(result, expected, path=(), msg='', **kwargs): result_len = len(result) expected_len = len(expected) assert result_len == expected_len, ( - 'list lengths do not match: %d != %d\n%s' % - result_len, - expected_len, - _fmt_path(path), + '%slist lengths do not match: %d != %d\n%s' % ( + _fmt_msg(msg), + result_len, + expected_len, + _fmt_path(path), + ) ) for n, (resultv, expectedv) in enumerate(zip(result, expected)): @@ -127,6 +257,56 @@ def assert_list_equal(result, expected, path=(), **kwargs): resultv, expectedv, path=path + ('[%d]' % n,), + msg=msg, + **kwargs + ) + + +@assert_equal.register(np.ndarray, np.ndarray) +def assert_array_equal(result, + expected, + path=(), + msg='', + array_verbose=True, + array_decimal=None, + **kwargs): + f = ( + np.testing.assert_array_equal + if array_decimal is None else + partial(np.testing.assert_array_almost_equal, decimal=array_decimal) + ) + try: + f( + result, + expected, + verbose=array_verbose, + err_msg=msg, + ) + except AssertionError as e: + raise AssertionError('\n'.join((str(e), _fmt_path(path)))) + + +@assert_equal.register(pd.DataFrame, pd.DataFrame) +def assert_dataframe_equal(result, expected, path=(), msg='', **kwargs): + try: + assert_frame_equal( + result, + expected, + **filter_kwargs(assert_frame_equal, kwargs) + ) + except AssertionError as e: + raise AssertionError( + _fmt_msg(msg) + '\n'.join((str(e), _fmt_path(path))), + ) + + +@assert_equal.register(Adjustment, Adjustment) +def assert_adjustment_equal(result, expected, path=(), **kwargs): + for attr in ('first_row', 'last_row', 'first_col', 'last_col', 'value'): + assert_equal( + getattr(result, attr), + getattr(expected, attr), + path=path + ('.' + attr,), **kwargs ) @@ -137,4 +317,6 @@ try: except ImportError: pass else: - assert_equal.funcs.update(assert_dshape_equal.funcs) + assert_equal.funcs.update( + dissoc(assert_dshape_equal.funcs, (object, object)), + ) diff --git a/zipline/utils/__init__.py b/zipline/utils/__init__.py index fbdf09d2..e69de29b 100644 --- a/zipline/utils/__init__.py +++ b/zipline/utils/__init__.py @@ -1,18 +0,0 @@ -# -# Copyright 2014 Quantopian, Inc. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -from .cli import run_pipeline, parse_args, parse_cell_magic - -__all__ = ['run_pipeline', 'parse_args', 'parse_cell_magic'] diff --git a/zipline/utils/cache.py b/zipline/utils/cache.py index 657f17ed..93833b09 100644 --- a/zipline/utils/cache.py +++ b/zipline/utils/cache.py @@ -1,11 +1,22 @@ """ Caching utilities for zipline """ -from collections import namedtuple +from collections import namedtuple, MutableMapping +import errno +import os +import pickle +from shutil import rmtree, copyfile, copytree +from tempfile import mkdtemp, NamedTemporaryFile + +import pandas as pd + +from .context_tricks import nop_context +from .paths import ensure_directory class Expired(Exception): - pass + """Marks that a :class:`CachedObject` has expired. + """ class CachedObject(namedtuple("_CachedObject", "value expires")): @@ -20,11 +31,6 @@ class CachedObject(namedtuple("_CachedObject", "value expires")): Expiration date of `value`. The cache is considered invalid for dates **strictly greater** than `expires`. - Methods - ------- - get(self, dt) - Get the cached object. - Usage ----- >>> from pandas import Timestamp, Timedelta @@ -66,22 +72,11 @@ class ExpiringCache(object): Parameters ---------- - cache : dict-like + cache : dict-like, optional An instance of a dict-like object which needs to support at least: `__del__`, `__getitem__`, `__setitem__` If `None`, than a dict is used as a default. - Methods - ------- - get(self, key, dt) - Get the value of a cached object for the given `key` at `dt`, if the - CachedObject has expired then the object is removed from the cache, - and `KeyError` is raised. - - set(self, key, value, expiration_dt) - Add a new `value` to the cache at `dt` wrapped in a CachedObject which - expires at `expiration_dt`. - Usage ----- >>> from pandas import Timestamp, Timedelta @@ -104,6 +99,26 @@ class ExpiringCache(object): self._cache = {} def get(self, key, dt): + """Get the value of a cached object. + + Parameters + ---------- + key : any + The key to lookup. + dt : datetime + The time of the lookup. + + Returns + ------- + result : any + The value for ``key``. + + Raises + ------ + KeyError + Raised if the key is not in the cache or the value for the key + has expired. + """ try: return self._cache[key].unwrap(dt) except Expired: @@ -111,4 +126,208 @@ class ExpiringCache(object): raise KeyError(key) def set(self, key, value, expiration_dt): + """Adds a new key value pair to the cache. + + Parameters + ---------- + key : any + The key to use for the pair. + value : any + The value to store under the name ``key``. + expiration_dt : datetime + When should this mapping expire? The cache is considered invalid + for dates **strictly greater** than ``expiration_dt``. + """ self._cache[key] = CachedObject(value, expiration_dt) + + +class dataframe_cache(MutableMapping): + """A disk-backed cache for dataframes. + + ``dataframe_cache`` is a mutable mapping from string names to pandas + DataFrame objects. + This object may be used as a context manager to delete the cache directory + on exit. + + Parameters + ---------- + path : str, optional + The directory path to the cache. Files will be written as + ``path/``. + lock : Lock, optional + Thread lock for multithreaded/multiprocessed access to the cache. + If not provided no locking will be used. + clean_on_failure : bool, optional + Should the directory be cleaned up if an exception is raised in the + context manager. + serialize : {'msgpack', 'pickle:'}, optional + How should the data be serialized. If ``'pickle'`` is passed, an + optional pickle protocol can be passed like: ``'pickle:3'`` which says + to use pickle protocol 3. + + Notes + ----- + The syntax ``cache[:]`` will load all key:value pairs into memory as a + dictionary. + The cache uses a temporary file format that is subject to change between + versions of zipline. + """ + def __init__(self, + path=None, + lock=None, + clean_on_failure=True, + serialization='msgpack'): + self.path = path if path is not None else mkdtemp() + self.lock = lock if lock is not None else nop_context + self.clean_on_failure = clean_on_failure + + if serialization == 'msgpack': + self.serialize = pd.DataFrame.to_msgpack + self.deserialize = pd.read_msgpack + self._protocol = None + else: + s = serialization.split(':', 1) + if s[0] != 'pickle': + raise ValueError( + "'serialization' must be either 'msgpack' or 'pickle[:n]'", + ) + self._protocol = int(s[1]) if len(s) == 2 else None + + self.serialize = self._serialize_pickle + self.deserialize = self._deserialize_pickle + + ensure_directory(self.path) + + def _serialize_pickle(self, df, path): + with open(path, 'wb') as f: + pickle.dump(df, f, protocol=self._protocol) + + def _deserialize_pickle(self, path): + with open(path, 'rb') as f: + return pickle.load(f) + + def _keypath(self, key): + return os.path.join(self.path, key) + + def __enter__(self): + return self + + def __exit__(self, type_, value, tb): + if not (self.clean_on_failure or value is None): + # we are not cleaning up after a failure and there was an exception + return + + with self.lock: + rmtree(self.path) + + def __getitem__(self, key): + if key == slice(None): + return dict(self.items()) + + with self.lock: + try: + return self.deserialize(self._keypath(key)) + except UnboundLocalError: + # This is how pandas fails if the file doesn't exist! #pandas + raise KeyError(key) + + def __setitem__(self, key, value): + with self.lock: + self.serialize(value, self._keypath(key)) + + def __delitem__(self, key): + with self.lock: + try: + os.remove(self._keypath(key)) + except OSError as e: + if e.errno == errno.ENOENT: + # raise a keyerror if this directory did not exist + raise KeyError(key) + # reraise the actual oserror otherwise + raise + + def __iter__(self): + return iter(os.listdir(self.path)) + + def __len__(self): + return len(os.listdir(self.path)) + + def __repr__(self): + return '<%s: keys={%s}>' % ( + type(self).__name__, + ', '.join(map(repr, sorted(self))), + ) + + +class working_file(object): + """A context manager for managing a temporary file that will be moved + to a non-temporary location if no exceptions are raised in the context. + + Parameters + ---------- + final_path : str + The location to move the file when committing. + *args, **kwargs + Forwarded to NamedTemporaryFile. + + Notes + ----- + The file is moved on __exit__ if there are no exceptions. + ``working_file`` uses :func:`shutil.copyfile` to move the actual files, + meaning it has as strong of guarantees as :func:`shutil.copyfile`. + """ + def __init__(self, final_path, *args, **kwargs): + self._tmpfile = NamedTemporaryFile(*args, **kwargs) + self._final_path = final_path + + def _commit(self): + """Sync the temporary file to the final path. + """ + copyfile(self.name, self._final_path) + + def __getattr__(self, attr): + return getattr(self._tmpfile, attr) + + def __enter__(self): + self._tmpfile.__enter__() + return self + + def __exit__(self, *exc_info): + if exc_info[0] is None: + self._commit() + self._tmpfile.__exit__(*exc_info) + + +class working_dir(object): + """A context manager for managing a temporary directory that will be moved + to a non-temporary location if no exceptions are raised in the context. + + Parameters + ---------- + final_path : str + The location to move the file when committing. + *args, **kwargs + Forwarded to tmp_dir. + + Notes + ----- + The file is moved on __exit__ if there are no exceptions. + ``working_dir`` uses :func:`shutil.copytree` to move the actual files, + meaning it has as strong of guarantees as :func:`shutil.copytree`. + """ + def __init__(self, final_path, *args, **kwargs): + self.name = mkdtemp() + self._final_path = final_path + + def _commit(self): + """Sync the temporary directory to the final path. + """ + copytree(self.name, self._final_path) + + def __enter__(self): + return self + + def __exit__(self, *exc_info): + if exc_info[0] is None: + self._commit() + rmtree(self.name) diff --git a/zipline/utils/cli.py b/zipline/utils/cli.py index 4ff2138c..70e15d35 100644 --- a/zipline/utils/cli.py +++ b/zipline/utils/cli.py @@ -1,259 +1,8 @@ -# -# Copyright 2014 Quantopian, Inc. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -import sys -import os -import argparse -from copy import copy - import click -from six import print_ -from six.moves import configparser import pandas as pd -try: - from pygments import highlight - from pygments.lexers import PythonLexer - from pygments.formatters import TerminalFormatter - PYGMENTS = True -except: - PYGMENTS = False - -import zipline -from zipline.errors import NoSourceError, PipelineDateError from .context_tricks import CallbackManager -DEFAULTS = { - 'data_frequency': 'daily', - 'capital_base': '10e6', - 'source': 'yahoo', - 'symbols': 'AAPL', - 'metadata_index': 'symbol', - 'source_time_column': 'Date', -} - - -def parse_args(argv, ipython_mode=False): - """Parse list of arguments. - - If a config file is provided (via -c), it will read in the - supplied options and overwrite any global defaults. - - All other directly supplied arguments will overwrite the config - file settings. - - Arguments: - * argv : list of strings - List of arguments, e.g. ['-c', 'my.conf'] - * ipython_mode : bool - Whether to parse IPython specific arguments - like --local_namespace - - Notes: - Default settings can be found in zipline.utils.cli.DEFAULTS. - - """ - # Parse any conf_file specification - # We make this parser with add_help=False so that - # it doesn't parse -h and print help. - conf_parser = argparse.ArgumentParser( - # Don't mess with format of description - formatter_class=argparse.RawDescriptionHelpFormatter, - # Turn off help, so we print all options in response to -h - add_help=False - ) - conf_parser.add_argument("-c", "--conf_file", - help="Specify config file", - metavar="FILE") - args, remaining_argv = conf_parser.parse_known_args(argv) - - defaults = copy(DEFAULTS) - - if args.conf_file: - config = configparser.SafeConfigParser() - config.read([args.conf_file]) - defaults.update(dict(config.items("Defaults"))) - - # Parse rest of arguments - # Don't suppress add_help here so it will handle -h - parser = argparse.ArgumentParser( - # Inherit options from config_parser - description="Zipline version %s." % zipline.__version__, - parents=[conf_parser] - ) - - parser.set_defaults(**defaults) - - parser.add_argument('--algofile', '-f') - parser.add_argument('--data-frequency', - choices=('minute', 'daily')) - parser.add_argument('--start', '-s') - parser.add_argument('--end', '-e') - parser.add_argument('--capital_base') - parser.add_argument('--source', '-d', choices=('yahoo',)) - parser.add_argument('--source_time_column', '-t') - parser.add_argument('--symbols') - parser.add_argument('--output', '-o') - parser.add_argument('--metadata_path', '-m') - parser.add_argument('--metadata_index', '-x') - parser.add_argument('--print-algo', '-p', dest='print_algo', - action='store_true') - parser.add_argument('--no-print-algo', '-q', dest='print_algo', - action='store_false') - - if ipython_mode: - parser.add_argument('--local_namespace', action='store_true') - - args = parser.parse_args(remaining_argv) - - return(vars(args)) - - -def parse_cell_magic(line, cell): - """Parse IPython magic - """ - args_list = line.split(' ') - args = parse_args(args_list, ipython_mode=True) - - # Remove print_algo kwarg to overwrite below. - args.pop('print_algo') - - local_namespace = args.pop('local_namespace', False) - # By default, execute inside IPython namespace - if not local_namespace: - args['namespace'] = get_ipython().user_ns # flake8: noqa - - # If we are running inside NB, do not output to file but create a - # variable instead - output_var_name = args.pop('output', None) - - perf = run_pipeline(print_algo=False, algo_text=cell, **args) - - if output_var_name is not None: - get_ipython().user_ns[output_var_name] = perf # flake8: noqa - - -def run_pipeline(print_algo=True, **kwargs): - """Runs a full zipline pipeline given configuration keyword - arguments. - - 1. Load data (start and end dates can be provided a strings as - well as the source and symobls). - - 2. Instantiate algorithm (supply either algo_text or algofile - kwargs containing initialize() and handle_data() functions). If - algofile is supplied, will try to look for algofile_analyze.py and - append it. - - 3. Run algorithm (supply capital_base as float). - - 4. Return performance dataframe. - - :Arguments: - * print_algo : bool - Whether to print the algorithm to command line. Will use - pygments syntax coloring if pygments is found. - - """ - start = kwargs['start'] - end = kwargs['end'] - # Compare against None because strings/timestamps may have been given - if start is not None: - start = pd.Timestamp(start, tz='UTC') - if end is not None: - end = pd.Timestamp(end, tz='UTC') - - # Fail out if only one bound is provided - if ((start is None) or (end is None)) and (start != end): - raise PipelineDateError(start=start, end=end) - - # Check if start and end are provided, and if the sim_params need to read - # a start and end from the DataSource - if start is None: - overwrite_sim_params = True - else: - overwrite_sim_params = False - - symbols = kwargs['symbols'].split(',') - asset_identifier = kwargs['metadata_index'] - - # Pull asset metadata - asset_metadata = kwargs.get('asset_metadata', None) - asset_metadata_path = kwargs['metadata_path'] - # Read in a CSV file, if applicable - if asset_metadata_path is not None: - if os.path.isfile(asset_metadata_path): - asset_metadata = pd.read_csv(asset_metadata_path, - index_col=asset_identifier) - - source_arg = kwargs['source'] - source_time_column = kwargs['source_time_column'] - - if source_arg is None: - raise NoSourceError() - - elif source_arg == 'yahoo': - source = zipline.data.load_bars_from_yahoo( - stocks=symbols, start=start, end=end) - - elif os.path.isfile(source_arg): - source = zipline.data.load_prices_from_csv( - filepath=source_arg, - identifier_col=source_time_column - ) - - elif os.path.isdir(source_arg): - source = zipline.data.load_prices_from_csv_folder( - folderpath=source_arg, - identifier_col=source_time_column - ) - - else: - raise NotImplementedError( - 'Source %s not implemented.' % kwargs['source']) - - algo_text = kwargs.get('algo_text', None) - if algo_text is None: - # Expect algofile to be set - algo_fname = kwargs['algofile'] - with open(algo_fname, 'r') as fd: - algo_text = fd.read() - - if print_algo: - if PYGMENTS: - highlight(algo_text, PythonLexer(), TerminalFormatter(), - outfile=sys.stdout) - else: - print_(algo_text) - - algo = zipline.TradingAlgorithm(script=algo_text, - namespace=kwargs.get('namespace', {}), - capital_base=float(kwargs['capital_base']), - algo_filename=kwargs.get('algofile'), - equities_metadata=asset_metadata, - start=start, - end=end) - - perf = algo.run(source, overwrite_sim_params=overwrite_sim_params) - - output_fname = kwargs.get('output', None) - if output_fname is not None: - perf.to_pickle(output_fname) - - return perf - def maybe_show_progress(it, show_progress, **kwargs): """Optionally show a progress bar for the given iterator. @@ -274,12 +23,96 @@ def maybe_show_progress(it, show_progress, **kwargs): Examples -------- - with maybe_show_progress([1, 2, 3], True) as ns: - for n in ns: - ... + .. code-block:: python + + with maybe_show_progress([1, 2, 3], True) as ns: + for n in ns: + ... """ if show_progress: return click.progressbar(it, **kwargs) # context manager that just return `it` when we enter it return CallbackManager(lambda it=it: it) + + +class _DatetimeParam(click.ParamType): + def __init__(self, tz=None): + self.tz = tz + + def parser(self, value): + return pd.Timestamp(value, tz=self.tz) + + @property + def name(self): + return type(self).__name__.upper() + + def convert(self, value, param, ctx): + try: + return self.parser(value) + except ValueError: + self.fail( + '%s is not a valid %s' % (value, self.name.lower()), + param, + ctx, + ) + + +class Timestamp(_DatetimeParam): + """A click parameter that parses the value into pandas.Timestamp objects. + + Parameters + ---------- + tz : timezone-coercable, optional + The timezone to parse the string as. + By default the timezone will be infered from the string or naiive. + """ + + +class Date(_DatetimeParam): + """A click parameter that parses the value into datetime.date objects. + + Parameters + ---------- + tz : timezone-coercable, optional + The timezone to parse the string as. + By default the timezone will be infered from the string or naiive. + as_timestamp : bool, optional + If True, return the value as a pd.Timestamp object normalized to + midnight. + """ + def __init__(self, tz=None, as_timestamp=False): + super(Date, self).__init__(tz=tz) + self.as_timestamp = as_timestamp + + def parser(self, value): + ts = super(Date, self).parser(value) + return ts.normalize() if self.as_timestamp else ts.date() + + +class Time(_DatetimeParam): + """A click parameter that parses the value into timetime.time objects. + + Parameters + ---------- + tz : timezone-coercable, optional + The timezone to parse the string as. + By default the timezone will be infered from the string or naiive. + """ + def parser(self, value): + return super(Time, self).parser(value).time() + + +class Timedelta(_DatetimeParam): + """A click parameter that parses values into pd.Timedelta objects. + + Parameters + ---------- + unit : {'D', 'h', 'm', 's', 'ms', 'us', 'ns'}, optional + Denotes the unit of the input if the input is an integer. + """ + def __init__(self, unit='ns'): + self.unit = unit + + def parser(self, value): + return pd.Timedelta(value, unit=self.unit) diff --git a/zipline/utils/compat.py b/zipline/utils/compat.py index fff02e6b..62ed44ef 100644 --- a/zipline/utils/compat.py +++ b/zipline/utils/compat.py @@ -2,7 +2,6 @@ from six import PY2 if PY2: - from functools32 import lru_cache from ctypes import py_object, pythonapi mappingproxy = pythonapi.PyDictProxy_New @@ -10,10 +9,8 @@ if PY2: mappingproxy.restype = py_object else: - from functools import lru_cache from types import MappingProxyType as mappingproxy __all__ = [ - 'lru_cache', 'mappingproxy', ] diff --git a/zipline/utils/input_validation.py b/zipline/utils/input_validation.py index cf0696c1..66e8b9fe 100644 --- a/zipline/utils/input_validation.py +++ b/zipline/utils/input_validation.py @@ -16,6 +16,7 @@ from functools import partial, wraps from operator import attrgetter from numpy import dtype +import pandas as pd from pytz import timezone from six import iteritems, string_types, PY3 from toolz import valmap, complement, compose @@ -133,6 +134,35 @@ def ensure_timezone(func, argname, arg): ) +def ensure_timestamp(func, argname, arg): + """Argument preprocessor that converts the input into a pandas Timestamp + object. + + Usage + ----- + >>> from zipline.utils.preprocess import preprocess + >>> @preprocess(ts=ensure_timestamp) + ... def foo(ts): + ... return ts + >>> foo('2014-01-01') + Timestamp('2014-01-01 00:00:00') + """ + try: + return pd.Timestamp(arg) + except ValueError as e: + raise TypeError( + "{func}() couldn't convert argument " + "{argname}={arg!r} to a pandas Timestamp.\n" + "Original error was: {t}: {e}".format( + func=_qualified_name(func), + argname=argname, + arg=arg, + t=_qualified_name(type(e)), + e=e, + ), + ) + + def expect_dtypes(*_pos, **named): """ Preprocessing decorator that verifies inputs have expected numpy dtypes. diff --git a/zipline/utils/math_utils.py b/zipline/utils/math_utils.py index 590f407e..da99900d 100644 --- a/zipline/utils/math_utils.py +++ b/zipline/utils/math_utils.py @@ -12,11 +12,37 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. - import math +from numpy import isnan -def tolerant_equals(a, b, atol=10e-7, rtol=10e-7): + +def tolerant_equals(a, b, atol=10e-7, rtol=10e-7, equal_nan=False): + """Check if a and b are equal with some tolerance. + + Parameters + ---------- + a, b : float + The floats to check for equality. + atol : float, optional + The absolute tolerance. + rtol : float, optional + The relative tolerance. + equal_nan : bool, optional + Should NaN compare equal? + + See Also + -------- + numpy.isclose + + Notes + ----- + This function is just a scalar version of numpy.isclose for performance. + See the docstring of ``isclose`` for more information about ``atol`` and + ``rtol``. + """ + if equal_nan and isnan(a) and isnan(b): + return True return math.fabs(a - b) <= (atol + rtol * math.fabs(b)) diff --git a/zipline/utils/paths.py b/zipline/utils/paths.py new file mode 100644 index 00000000..103c116b --- /dev/null +++ b/zipline/utils/paths.py @@ -0,0 +1,223 @@ +""" +Canonical path locations for zipline data. + +Paths are rooted at $ZIPLINE_ROOT if that environment variable is set. +Otherwise default to expanduser(~/.zipline) +""" +from errno import EEXIST +import os +from os.path import exists, expanduser, join + +import pandas as pd + + +def hidden(path): + """Check if a path is hidden. + + Parameters + ---------- + path : str + A filepath. + """ + return path.startswith('.') + + +def ensure_directory(path): + """ + Ensure that a directory named "path" exists. + """ + try: + os.makedirs(path) + except OSError as exc: + if exc.errno == EEXIST and os.path.isdir(path): + return + raise + + +def ensure_directory_containing(path): + """ + Ensure that the directory containing `path` exists. + + This is just a convenience wrapper for doing:: + + ensure_directory(os.path.dirname(path)) + """ + ensure_directory(os.path.dirname(path)) + + +def last_modified_time(path): + """ + Get the last modified time of path as a Timestamp. + """ + return pd.Timestamp(os.path.getmtime(path), unit='s', tz='UTC') + + +def modified_since(path, dt): + """ + Check whether `path` was modified since `dt`. + + Returns False if path doesn't exist. + + Parameters + ---------- + path : str + Path to the file to be checked. + dt : pd.Timestamp + The date against which to compare last_modified_time(path). + + Returns + ------- + was_modified : bool + Will be ``False`` if path doesn't exists, or if its last modified date + is earlier than or equal to `dt` + """ + return exists(path) and last_modified_time(path) > dt + + +def zipline_root(environ=None): + """ + Get the root directory for all zipline-managed files. + + For testing purposes, this accepts a dictionary to interpret as the os + environment. + + Parameters + ---------- + environ : dict, optional + A dict to interpret as the os environment. + + Returns + ------- + root : string + Path to the zipline root dir. + """ + if environ is None: + environ = os.environ + + root = environ.get('ZIPLINE_ROOT', None) + if root is None: + root = expanduser('~/.zipline') + + return root + + +def zipline_path(paths, environ=None): + """ + Get a path relative to the zipline root. + + Parameters + ---------- + paths : list[str] + List of requested path pieces. + environ : dict, optional + An environment dict to forward to zipline_root. + + Returns + ------- + newpath : str + The requested path joined with the zipline root. + """ + return join(zipline_root(environ=environ), *paths) + + +def default_extension(environ=None): + """ + Get the path to the default zipline extension file. + + Parameters + ---------- + environ : dict, optional + An environment dict to forwart to zipline_root. + + Returns + ------- + default_extension_path : str + The file path to the default zipline extension file. + """ + return zipline_path(['extension.py'], environ=environ) + + +def data_root(environ=None): + """ + The root directory for zipline data files. + + Parameters + ---------- + environ : dict, optional + An environment dict to forward to zipline_root. + + Returns + ------- + data_root : str + The zipline data root. + """ + return zipline_path(['data'], environ=environ) + + +def ensure_data_root(environ=None): + """ + Ensure that the data root exists. + """ + ensure_directory(data_root(environ=environ)) + + +def data_path(paths, environ=None): + """ + Get a path relative to the zipline data directory. + + Parameters + ---------- + paths : iterable[str] + List of requested path pieces. + environ : dict, optional + An environment dict to forward to zipline_root. + + Returns + ------- + newpath : str + The requested path joined with the zipline data root. + """ + return zipline_path(['data'] + list(paths), environ=environ) + + +def cache_root(environ=None): + """ + The root directory for zipline cache files. + + Parameters + ---------- + environ : dict, optional + An environment dict to forward to zipline_root. + + Returns + ------- + cache_root : str + The zipline cache root. + """ + return zipline_path(['cache'], environ=environ) + + +def ensure_cache_root(environ=None): + """ + Ensure that the data root exists. + """ + ensure_directory(cache_root(environ=environ)) + + +def cache_path(paths, environ=None): + """ + Get a path relative to the zipline cache directory. + + Parameters + ---------- + paths : iterable[str] + List of requested path pieces. + environ : dict, optional + An environment dict to forward to zipline_root. + + Returns + ------- + newpath : str + The requested path joined with the zipline cache root. + """ + return zipline_path(['cache'] + list(paths), environ=environ) diff --git a/zipline/utils/run_algo.py b/zipline/utils/run_algo.py new file mode 100644 index 00000000..09dbd57e --- /dev/null +++ b/zipline/utils/run_algo.py @@ -0,0 +1,335 @@ +import os +import re +from runpy import run_path +import sys +import warnings + +import click +try: + from pygments import highlight + from pygments.lexers import PythonLexer + from pygments.formatters import TerminalFormatter + PYGMENTS = True +except: + PYGMENTS = False +from toolz import valfilter, concatv + +from zipline.algorithm import TradingAlgorithm +from zipline.data.bundles.core import load +from zipline.data.data_portal import DataPortal +from zipline.finance.trading import TradingEnvironment +import zipline.utils.paths as pth + + +class _RunAlgoError(click.ClickException, ValueError): + """Signal an error that should have a different message if invoked from + the cli. + + Parameters + ---------- + pyfunc_msg : str + The message that will be shown when called as a python function. + cmdline_msg : str + The message that will be shown on the command line. + """ + exit_code = 1 + + def __init__(self, pyfunc_msg, cmdline_msg): + super(_RunAlgoError, self).__init__(cmdline_msg) + self.pyfunc_msg = pyfunc_msg + + def __str__(self): + return self.pyfunc_msg + + +def _run(handle_data, + initialize, + before_trading_start, + analyze, + algofile, + algotext, + defines, + data_frequency, + capital_base, + data, + bundle, + bundle_timestamp, + start, + end, + output, + print_algo, + local_namespace, + environ): + """Run a backtest for the given algorithm. + + This is shared between the cli and :func:`zipline.run_algo`. + """ + if algotext is not None: + if local_namespace: + ip = get_ipython() # noqa + namespace = ip.user_ns + else: + namespace = {} + + for assign in defines: + try: + name, value = assign.split('=', 2) + except ValueError: + raise ValueError( + 'invalid define %r, should be of the form name=value' % + assign, + ) + try: + # evaluate in the same namespace so names may refer to + # eachother + namespace[name] = eval(value, namespace) + except Exception as e: + raise ValueError( + 'failed to execute definition for name %r: %s' % (name, e), + ) + elif defines: + raise _RunAlgoError( + 'cannot pass define without `algotext`', + "cannot pass '-D' / '--define' without '-t' / '--algotext'", + ) + else: + namespace = {} + if algofile is not None: + algotext = algofile.read() + + if print_algo: + if PYGMENTS: + highlight( + algotext, + PythonLexer(), + TerminalFormatter(), + outfile=sys.stdout, + ) + else: + click.echo(algotext) + + if bundle is not None: + bundle_data = load( + bundle, + environ, + bundle_timestamp, + ) + + prefix, connstr = re.split( + r'sqlite:///', + str(bundle_data.asset_finder.engine.url), + maxsplit=1, + ) + if prefix: + raise ValueError( + "invalid url %r, must begin with 'sqlite:///'" % + str(bundle_data.asset_finder.engine.url), + ) + env = TradingEnvironment(asset_db_path=connstr) + data = DataPortal( + env, + equity_minute_reader=bundle_data.minute_bar_reader, + equity_daily_reader=bundle_data.daily_bar_reader, + adjustment_reader=bundle_data.adjustment_reader, + ) + + perf = TradingAlgorithm( + namespace=namespace, + capital_base=capital_base, + start=start, + end=end, + env=env, + **{ + 'initialize': initialize, + 'handle_data': handle_data, + 'before_trading_start': before_trading_start, + 'analyze': analyze, + } if algotext is None else { + 'algo_filename': algofile, + 'script': algotext, + } + ).run( + data, + overwrite_sim_params=False, + ) + + if output == '-': + click.echo(str(perf)) + elif output != os.devnull: # make the zipline magic not write any data + perf.to_pickle(output) + + return perf + + +# All of the loaded extensions. We don't want to load an extension twice. +_loaded_extensions = set() + + +def load_extensions(default, extensions, strict, environ, reload=False): + """Load all of the given extensions. This should be called by run_algo + or the cli. + + Parameters + ---------- + default : bool + Load the default exension (~/.zipline/extension.py)? + extension : iterable[str] + The paths to the extensions to load. If the path ends in ``.py`` it is + treated as a script and executed. If it does not end in ``.py`` it is + treated as a module to be imported. + strict : bool + Should failure to load an extension raise. If this is false it will + still warn. + environ : mapping + The environment to use to find the default extension path. + reload : bool, optional + Reload any extensions that have already been loaded. + """ + if default: + default_extension_path = pth.default_extension(environ=environ) + open(default_extension_path, 'a+').close() # touch the file + # put the default extension first so other extensions can depend on + # the order they are loaded + extensions = concatv([default_extension_path], extensions) + + for ext in extensions: + if ext in _loaded_extensions and not reload: + continue + try: + # load all of the zipline extensionss + if ext.endswith('.py'): + run_path(ext, run_name='') + else: + __import__(ext) + except Exception as e: + if strict: + # if `strict` we should raise the actual exception and fail + raise + # without `strict` we should just log the failure + warnings.warn( + 'Failed to load extension: %r\n%s' % (ext, e), + stacklevel=2 + ) + else: + _loaded_extensions.add(ext) + + +def run_algorithm(start, + end, + initialize, + capital_base, + handle_data=None, + before_trading_start=None, + analyze=None, + data_frequency='daily', + data=None, + bundle=None, + bundle_timestamp=None, + default_extension=True, + extensions=(), + strict_extensions=True, + environ=os.environ): + """Run a trading algorithm. + + Parameters + ---------- + start : datetime + The start date of the backtest. + end : datetime + The end date of the backtest.. + initialize : callable[context -> None] + The initialize function to use for the algorithm. This is called once + at the very begining of the backtest and should be used to set up + any state needed by the algorithm. + capital_base : float + The starting capital for the backtest. + handle_data : callable[(context, BarData) -> None], optional + The handle_data function to use for the algorithm. This is called + every minute when ``data_frequency == 'minute'`` or every day + when ``data_frequency == 'daily'``. + before_trading_start : callable[(context, BarData) -> None], optional + The before_trading_start function for the algorithm. This is called + once before each trading day (after initialize on the first day). + analyze : callable[(context, pd.DataFrame) -> None], optional + The analyze function to use for the algorithm. This function is called + once at the end of the backtest and is passed the context and the + performance data. + data_frequency : {'daily', 'minute'}, optional + The data frequency to run the algorithm at. + data : pd.DataFrame, pd.Panel, or DataPortal, optional + The ohlcv data to run the backtest with. + This argument is mutually exclusive with: + ``bundle`` + ``bundle_timestamp`` + bundle : str, optional + The name of the data bundle to use to load the data to run the backtest + with. This defaults to 'quandl'. + This argument is mutually exclusive with ``data``. + bundle_timestamp : datetime, optional + The datetime to lookup the bundle data for. This defaults to the + current time. + This argument is mutually exclusive with ``data``. + default_extension : bool, optional + Should the default zipline extension be loaded. This is found at + ``$ZIPLINE_ROOT/extension.py`` + extensions : iterable[str], optional + The names of any other extensions to load. Each element may either be + a dotted module path like ``a.b.c`` or a path to a python file ending + in ``.py`` like ``a/b/c.py``. + strict_extensions : bool, optional + Should the run fail if any extensions fail to load. If this is false, + a warning will be raised instead. + environ : mapping[str -> str], optional + The os environment to use. Many extensions use this to get parameters. + This defaults to ``os.environ``. + + Returns + ------- + perf : pd.DataFrame + The daily performance of the algorithm. + + See Also + -------- + zipline.data.bundles.bundles : The available data bundles. + """ + load_extensions(default_extension, extensions, strict_extensions, environ) + + non_none_data = valfilter(bool, { + 'data': data, + 'bundle': bundle, + }) + if not non_none_data: + # if neither data nor bundle are passed use 'quandl' + bundle = 'quandl' + + if len(non_none_data) != 1: + raise ValueError( + 'must specify one of `data`, `data_portal`, or `bundle`,' + ' got: %r' % non_none_data, + ) + + if 'bundle' not in non_none_data and bundle_timestamp is not None: + raise ValueError( + 'cannot specify `bundle_timestamp` without passing `bundle`', + ) + + return _run( + handle_data=handle_data, + initialize=initialize, + before_trading_start=before_trading_start, + analyze=analyze, + algofile=None, + algotext=None, + defines=(), + data_frequency=data_frequency, + capital_base=capital_base, + data=data, + bundle=bundle, + bundle_timestamp=bundle_timestamp, + start=start, + end=end, + output=os.devnull, + print_algo=False, + local_namespace=False, + environ=environ, + )