diff --git a/docs/source/appendix.rst b/docs/source/appendix.rst index 27d28c2d..1ef2ded5 100644 --- a/docs/source/appendix.rst +++ b/docs/source/appendix.rst @@ -1,6 +1,11 @@ API Reference ------------- +Running a Backtest +~~~~~~~~~~~~~~~~~~ + +.. autofunction:: zipline.run_algorithm(...) + Algorithm API ~~~~~~~~~~~~~ @@ -85,29 +90,88 @@ Pipeline API Asset Metadata ~~~~~~~~~~~~~~ -.. autoclass:: zipline.assets.assets.Asset +.. autoclass:: zipline.assets.Asset :members: -.. autoclass:: zipline.assets.assets.Equity +.. autoclass:: zipline.assets.Equity :members: -.. autoclass:: zipline.assets.assets.Future +.. autoclass:: zipline.assets.Future :members: -.. autoclass:: zipline.assets.assets.AssetFinder - :members: - -.. autoclass:: zipline.assets.assets.AssetFinderCachedEquities - :members: - -.. autoclass:: zipline.assets.asset_writer.AssetDBWriter - :members: - -.. autoclass:: zipline.assets.assets.AssetConvertible +.. autoclass:: zipline.assets.AssetConvertible :members: Data API ~~~~~~~~ +Writers +``````` .. autoclass:: zipline.data.minute_bars.BcolzMinuteBarWriter :members: + +.. autoclass:: zipline.data.us_equity_pricing.BcolzDailyBarWriter + :members: + +.. autoclass:: zipline.data.us_equity_pricing.SQLiteAdjustmentWriter + :members: + +.. autoclass:: zipline.assets.AssetDBWriter + :members: + +Readers +``````` +.. autoclass:: zipline.data.minute_bars.BcolzMinuteBarReader + :members: + +.. autoclass:: zipline.data.us_equity_pricing.BcolzDailyBarReader + :members: + +.. autoclass:: zipline.data.us_equity_pricing.SQLiteAdjustmentReader + :members: + +.. autoclass:: zipline.assets.AssetFinder + :members: + +.. autoclass:: zipline.assets.AssetFinderCachedEquities + :members: + +Bundles +``````` +.. autofunction:: zipline.data.bundles.register + +.. autofunction:: zipline.data.bundles.ingest(name, environ=os.environ, date=None, show_progress=True) + +.. autofunction:: zipline.data.bundles.load(name, environ=os.environ, date=None) + +.. autofunction:: zipline.data.bundles.unregister + +.. data:: zipline.data.bundles.bundles + + The bundles that have been registered as a mapping from bundle name to bundle + data. This mapping is immutable and should only be updated through + :func:`~zipline.data.bundles.register` or + :func:`~zipline.data.bundles.unregister`. + +.. autofunction:: zipline.data.bundles.yahoo_equities + + +Utilities +~~~~~~~~~ + +Caching +``````` + +.. autoclass:: zipline.utils.cache.CachedObject + +.. autoclass:: zipline.utils.cache.ExpiringCache + +.. autoclass:: zipline.utils.cache.dataframe_cache + +.. autoclass:: zipline.utils.cache.working_file + +.. autoclass:: zipline.utils.cache.working_dir + +Command Line +```````````` +.. autofunction:: zipline.utils.cli.maybe_show_progress diff --git a/docs/source/beginner-tutorial.rst b/docs/source/beginner-tutorial.rst index 80a81205..3bda3cea 100644 --- a/docs/source/beginner-tutorial.rst +++ b/docs/source/beginner-tutorial.rst @@ -51,53 +51,50 @@ My first algorithm Lets take a look at a very simple algorithm from the ``examples`` directory, ``buyapple.py``: -.. code:: python +.. code-block:: python - !tail ../../zipline/examples/buyapple.py + from zipline.examples import buyapple + buyapple?? -.. parsed-literal:: +.. code-block:: python - # Load price data from yahoo. - data = load_from_yahoo(stocks=['AAPL'], indexes={}, start=start, - end=end) + from zipline.api import order, record, symbol - # Create and run the algorithm. - algo = TradingAlgorithm(initialize=initialize, handle_data=handle_data, - identifiers=['AAPL']) - results = algo.run(data) - analyze(results=results) + def initialize(context): + pass + + + def handle_data(context, data): + order(symbol('AAPL'), 10) + record(AAPL=data.current(symbol('AAPL'), 'price')) As you can see, we first have to import some functions we would like to use. All functions commonly used in your algorithm can be found in -``zipline.api``. Here we are using ``order()`` which takes two arguments --- a security object, and a number specifying how many stocks you would -like to order (if negative, ``order()`` will sell/short stocks). In this -case we want to order 10 shares of Apple at each iteration. For more -documentation on ``order()``, see the `Quantopian -docs `__. +``zipline.api``. Here we are using :func:`~zipline.api.order()` which takes two +arguments: a security object, and a number specifying how many stocks you would +like to order (if negative, :func:`~zipline.api.order()` will sell/short +stocks). In this case we want to order 10 shares of Apple at each iteration. For +more documentation on ``order()``, see the `Quantopian docs +`__. -You don't have to use the ``symbol()`` function and could just pass in -``AAPL`` directly but it is good practice as this way your code will be -Quantopian compatible. - -Finally, the ``record()`` function allows you to save the value of a -variable at each iteration. You provide it with a name for the variable +Finally, the :func:`~zipline.api.record` function allows you to save the value +of a variable at each iteration. You provide it with a name for the variable together with the variable itself: ``varname=var``. After the algorithm finished running you will have access to each variable value you tracked -with ``record()`` under the name you provided (we will see this further -below). You also see how we can access the current price data of the +with :func:`~zipline.api.record` under the name you provided (we will see this +further below). You also see how we can access the current price data of the AAPL stock in the ``data`` event frame (for more information see `here `__. Running the algorithm ~~~~~~~~~~~~~~~~~~~~~ -To now test this algorithm on financial data, ``zipline`` provides two -interfaces. A command-line interface and an ``IPython Notebook`` -interface. +To now test this algorithm on financial data, ``zipline`` provides three +interfaces: A command-line interface, ``IPython Notebook`` magic, and +:func:`~zipline.run_algorithm`. Command line interface ^^^^^^^^^^^^^^^^^^^^^^ @@ -106,60 +103,59 @@ After you installed zipline you should be able to execute the following from your command line (e.g. ``cmd.exe`` on Windows, or the Terminal app on OSX): -.. code:: python - - !run_algo.py --help +.. code-block:: bash + $ python -m zipline run --help .. parsed-literal:: - usage: run_algo.py [-h] [-c FILE] [--algofile ALGOFILE] [--data-frequency {minute,daily}] [--start START] [--end END] - [--capital_base CAPITAL_BASE] [--source {yahoo}] [--source_time_column SOURCE_TIME_COLUMN] [--symbols SYMBOLS] - [--output OUTPUT] [--metadata_path METADATA_PATH] [--metadata_index METADATA_INDEX] [--print-algo] [--no-print-algo] + Usage: __main__.py run [OPTIONS] - Zipline version 0.8.3. + Run a backtest for the given algorithm. - optional arguments: - -h, --help show this help message and exit - -c FILE, --conf_file FILE - Specify config file - --algofile ALGOFILE, -f ALGOFILE - --data-frequency {minute,daily} - --start START, -s START - --end END, -e END - --capital_base CAPITAL_BASE - --source {yahoo}, -d {yahoo} - --source_time_column SOURCE_TIME_COLUMN, -t SOURCE_TIME_COLUMN - --symbols SYMBOLS - --output OUTPUT, -o OUTPUT - --metadata_path METADATA_PATH, -m METADATA_PATH - --metadata_index METADATA_INDEX, -x METADATA_INDEX - --print-algo, -p - --no-print-algo, -q + Options: + -f, --algofile FILENAME The file that contains the algorithm to run. + -t, --algotext TEXT The algorithm script to run. + -D, --define TEXT Define a name to be bound in the namespace + before executing the algotext. For example + '-Dname=value'. The value may be any python + expression. These are evaluated in order so + they may refer to previously defined names. + --data-frequency [minute|daily] + The data frequency of the simulation. + [default: daily] + --capital-base FLOAT The starting capital for the simulation. + [default: 10000000.0] + -b, --bundle BUNDLE-NAME The data bundle to use for the simulation. + [default: quandl] + --bundle-timestamp TIMESTAMP The date to lookup data on or before. + [default: ] + -s, --start DATE The start date of the simulation. + -e, --end DATE The end date of the simulation. + -o, --output FILENAME The location to write the perf data. If this + is '-' the perf will be written to stdout. + [default: -] + --print-algo / --no-print-algo Print the algorithm to stdout. + --help Show this message and exit. - -Note that you have to omit the preceding '!' when you call -``run_algo.py``, this is only required by the IPython Notebook in which -this tutorial was written. - -As you can see there are a couple of flags that specify where to find -your algorithm (``-f``) as well as parameters specifying which stock -data to load from Yahoo! finance (``--symbols``) and the time-range -(``--start`` and ``--end``). Finally, you'll want to save the -performance metrics of your algorithm so that you can analyze how it -performed. This is done via the ``--output`` flag and will cause it to -write the performance ``DataFrame`` in the pickle Python file format. -Note that you can also define a configuration file with these parameters -that you can then conveniently pass to the ``-c`` option so that you -don't have to supply the command line args all the time (see the .conf -files in the examples directory). +As you can see there are a couple of flags that specify where to find your +algorithm (``-f``) as well as parameters specifying which data to use, +defaulting to the :ref:`quandl-data-bundle`. There are also arguments for the +date range to run the algorithm over (``--start`` and ``--end``). Finally, +you'll want to save the performance metrics of your algorithm so that you can +analyze how it performed. This is done via the ``--output`` flag and will cause +it to write the performance ``DataFrame`` in the pickle Python file format. +Note that you can also define a configuration file with these parameters that +you can then conveniently pass to the ``-c`` option so that you don't have to +supply the command line args all the time (see the .conf files in the examples +directory). Thus, to execute our algorithm from above and save the results to -``buyapple_out.pickle`` we would call ``run_algo.py`` as follows: +``buyapple_out.pickle`` we would call ``python -m zipline run`` as follows: -.. code:: python +.. code-block:: python - !run_algo.py -f ../../zipline/examples/buyapple.py --start 2000-1-1 --end 2014-1-1 --symbols AAPL -o buyapple_out.pickle + python -m zipline run -f ../../zipline/examples/buyapple.py --start 2000-1-1 --end 2014-1-1 --symbols AAPL -o buyapple_out.pickle .. parsed-literal:: @@ -170,9 +166,7 @@ Thus, to execute our algorithm from above and save the results to [2015-11-04 22:45:32.820401] INFO: Performance: last close: 2013-12-31 21:00:00+00:00 -``run_algo.py`` first outputs the algorithm contents. It then fetches -historical price and volume data of Apple from Yahoo! finance in the -desired time range, calls the ``initialize()`` function, and then +``run`` first calls the ``initialize()`` function, and then streams the historical stock price day-by-day through ``handle_data()``. After each call to ``handle_data()`` we instruct ``zipline`` to order 10 stocks of AAPL. After the call of the ``order()`` function, ``zipline`` @@ -187,31 +181,18 @@ slippage model that ``zipline`` uses, see the `Quantopian docs `__ for more information). -Note that there is also an ``analyze()`` function printed. -``run_algo.py`` will try and look for a file with the ending with -``_analyze.py`` and the same name of the algorithm (so -``buyapple_analyze.py``) or an ``analyze()`` function directly in the -script. If an ``analyze()`` function is found it will be called *after* -the simulation has finished and passed in the performance ``DataFrame``. -(The reason for allowing specification of an ``analyze()`` function in a -separate file is that this way ``buyapple.py`` remains a valid -Quantopian algorithm that you can copy&paste to the platform). - Lets take a quick look at the performance ``DataFrame``. For this, we use ``pandas`` from inside the IPython Notebook and print the first ten rows. Note that ``zipline`` makes heavy usage of ``pandas``, especially for data input and outputting so it's worth spending some time to learn it. -.. code:: python +.. code-block:: python import pandas as pd perf = pd.read_pickle('buyapple_out.pickle') # read in perf DataFrame perf.head() - - - .. raw:: html
@@ -378,7 +359,7 @@ and allows us to plot the price of apple. For example, we could easily examine now how our portfolio value changed over time compared to the AAPL stock price. -.. code:: python +.. code-block:: python %pylab inline figsize(12, 12) @@ -391,21 +372,14 @@ AAPL stock price. perf.AAPL.plot(ax=ax2) ax2.set_ylabel('AAPL stock price') - .. parsed-literal:: Populating the interactive namespace from numpy and matplotlib - - - .. parsed-literal:: - - - .. image:: tutorial_files/tutorial_11_2.png @@ -431,28 +405,21 @@ to run the algorithm from above with the same parameters we just have to execute the following cell after importing ``zipline`` to register the magic. -.. code:: python +.. code-block:: python - import zipline + %load_ext zipline -.. code:: python +.. code-block:: python - %%zipline --start 2000-1-1 --end 2014-1-1 --symbols AAPL -o perf_ipython + %%zipline --start 2000-1-1 --end 2014-1-1 --symbols AAPL + from zipline.api import symbol, order, record - from zipline.api import symbol, order, record - - def initialize(context): - pass - - def handle_data(context, data): - order(symbol('AAPL'), 10) - record(AAPL=data[symbol('AAPL')].price) - - -.. parsed-literal:: - - AAPL + def initialize(context): + pass + def handle_data(context, data): + order(symbol('AAPL'), 10) + record(AAPL=data[symbol('AAPL')].price) Note that we did not have to specify an input file as above since the magic will use the contents of the cell and look for your algorithm @@ -460,12 +427,9 @@ functions there. Also, instead of defining an output file we are specifying a variable name with ``-o`` that will be created in the name space and contain the performance ``DataFrame`` we looked at above. -.. code:: python - - perf_ipython.head() - - +.. code-block:: python + _.head() .. raw:: html @@ -624,58 +588,6 @@ space and contain the performance ``DataFrame`` we looked at above.
- -Manual (advanced) -~~~~~~~~~~~~~~~~~ - -If you are happy with either way above you can safely skip this passage. -To provide a closer look at how ``zipline`` actually works it is -instructive to see how we run an algorithm without any of the interfaces -demonstrated above which hide the actual ``zipline`` API. - -.. code:: python - - import pytz - from datetime import datetime - - from zipline.algorithm import TradingAlgorithm - from zipline.utils.factory import load_bars_from_yahoo - - # Load data manually from Yahoo! finance - start = datetime(2000, 1, 1, 0, 0, 0, 0, pytz.utc) - end = datetime(2012, 1, 1, 0, 0, 0, 0, pytz.utc) - data = load_bars_from_yahoo(stocks=['AAPL'], start=start, - end=end) - - # Define algorithm - def initialize(context): - pass - - def handle_data(context, data): - order(symbol('AAPL'), 10) - record(AAPL=data[symbol('AAPL')].price) - - # Create algorithm object passing in initialize and - # handle_data functions - algo_obj = TradingAlgorithm(initialize=initialize, - handle_data=handle_data) - - # Run algorithm - perf_manual = algo_obj.run(data) - - -.. parsed-literal:: - - AAPL - - -As you can see, we again define the functions as above but we manually -pass them to the ``TradingAlgorithm`` class which is the main -``zipline`` class for running algorithms. We also manually load the data -using ``load_bars_from_yahoo()`` and pass it to the -``TradingAlgorithm.run()`` method which kicks off the backtest -simulation. - Access to previous prices using ``history`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -706,81 +618,73 @@ you can directly use the ``history()`` function on Quantopian, in with ``add_history()`` and pass it the same arguments as the history function below. Lets look at the strategy which should make this clear: -.. code:: python +.. code-block:: python - %%zipline --start 2000-1-1 --end 2014-1-1 --symbols AAPL -o perf_dma + %%zipline --start 2000-1-1 --end 2014-1-1 -o perf_dma - from zipline.api import order_target, record, symbol, history, add_history - import numpy as np + from zipline.api import order_target, record, symbol, history, add_history + import numpy as np - def initialize(context): - # Register 2 histories that track daily prices, - # one with a 100 window and one with a 300 day window - add_history(100, '1d', 'price') - add_history(300, '1d', 'price') + def initialize(context): + # Register 2 histories that track daily prices, + # one with a 100 window and one with a 300 day window + add_history(100, '1d', 'price') + add_history(300, '1d', 'price') - context.i = 0 + context.i = 0 - def handle_data(context, data): - # Skip first 300 days to get full windows - context.i += 1 - if context.i < 300: - return + def handle_data(context, data): + # Skip first 300 days to get full windows + context.i += 1 + if context.i < 300: + return - # Compute averages - # history() has to be called with the same params - # from above and returns a pandas dataframe. - short_mavg = history(100, '1d', 'price').mean() - long_mavg = history(300, '1d', 'price').mean() + # Compute averages + # history() has to be called with the same params + # from above and returns a pandas dataframe. + short_mavg = history(100, '1d', 'price').mean() + long_mavg = history(300, '1d', 'price').mean() - # Trading logic - if short_mavg[0] > long_mavg[0]: - # order_target orders as many shares as needed to - # achieve the desired number of shares. - order_target(symbol('AAPL'), 100) - elif short_mavg[0] < long_mavg[0]: - order_target(symbol('AAPL'), 0) + # Trading logic + if short_mavg[0] > long_mavg[0]: + # order_target orders as many shares as needed to + # achieve the desired number of shares. + order_target(symbol('AAPL'), 100) + elif short_mavg[0] < long_mavg[0]: + order_target(symbol('AAPL'), 0) - # Save values for later inspection - record(AAPL=data[symbol('AAPL')].price, - short_mavg=short_mavg[0], - long_mavg=long_mavg[0]) + # Save values for later inspection + record(AAPL=data[symbol('AAPL')].price, + short_mavg=short_mavg[0], + long_mavg=long_mavg[0]) - def analyze(context, perf): - fig = plt.figure() - ax1 = fig.add_subplot(211) - perf.portfolio_value.plot(ax=ax1) - ax1.set_ylabel('portfolio value in $') - - ax2 = fig.add_subplot(212) - perf['AAPL'].plot(ax=ax2) - perf[['short_mavg', 'long_mavg']].plot(ax=ax2) - - perf_trans = perf.ix[[t != [] for t in perf.transactions]] - buys = perf_trans.ix[[t[0]['amount'] > 0 for t in perf_trans.transactions]] - sells = perf_trans.ix[ - [t[0]['amount'] < 0 for t in perf_trans.transactions]] - ax2.plot(buys.index, perf.short_mavg.ix[buys.index], - '^', markersize=10, color='m') - ax2.plot(sells.index, perf.short_mavg.ix[sells.index], - 'v', markersize=10, color='k') - ax2.set_ylabel('price in $') - plt.legend(loc=0) - plt.show() - - -.. parsed-literal:: - - AAPL + def analyze(context, perf): + fig = plt.figure() + ax1 = fig.add_subplot(211) + perf.portfolio_value.plot(ax=ax1) + ax1.set_ylabel('portfolio value in $') + ax2 = fig.add_subplot(212) + perf['AAPL'].plot(ax=ax2) + perf[['short_mavg', 'long_mavg']].plot(ax=ax2) + perf_trans = perf.ix[[t != [] for t in perf.transactions]] + buys = perf_trans.ix[[t[0]['amount'] > 0 for t in perf_trans.transactions]] + sells = perf_trans.ix[ + [t[0]['amount'] < 0 for t in perf_trans.transactions]] + ax2.plot(buys.index, perf.short_mavg.ix[buys.index], + '^', markersize=10, color='m') + ax2.plot(sells.index, perf.short_mavg.ix[sells.index], + 'v', markersize=10, color='k') + ax2.set_ylabel('price in $') + plt.legend(loc=0) + plt.show() .. image:: tutorial_files/tutorial_22_1.png - Here we are explicitly defining an ``analyze()`` function that gets automatically called once the backtest is done (this is not possible on Quantopian currently). diff --git a/docs/source/bundles.rst b/docs/source/bundles.rst new file mode 100644 index 00000000..5134ec7f --- /dev/null +++ b/docs/source/bundles.rst @@ -0,0 +1,277 @@ +Data Bundles +------------ + +A data bundle is a collection of pricing data, adjustment data, and an asset +database. Bundles allow us to preload all of the data we will need to run +backtests and store the data for future runs. + +Ingesting Data +~~~~~~~~~~~~~~ + +The first step to using a data bundle is to ingest the data. This will invoke +some custom bundle command and then write the data to a standard location that +zipline can find. By default this location is ``$ZIPLINE_ROOT/data/`` +where by default ``ZIPLINE_ROOT=~/.zipline``. This step may take some time as it +could involve downloading and processing a lot of data. This can be run with: + +.. code-block:: bash + + $ python -m zipline ingest + + +where ```` is the name of the bundle to ingest. + +Old Data +~~~~~~~~ + +When the ``ingest`` command is used it will write the new data to a subdirectory +of ``$ZIPLINE_ROOT/data/`` which is named with the current date. This +makes it possible to look at older data or even run backtests with this older +copy. This makers it easier to reproduce backtest results later. + +One drawback of saving all of this data by default is that the data directory +may grow quite large even if you do not want to use the data. To solve this +problem there is another command ``clean`` which will clear data bundles based +on some time constraints. + +For example: + +.. code-block:: bash + + # clean everything older than + $ python -m zipline clean --before + + # clean everything newer than + $ python -m zipline clean --after + + # keep everything in the range of [before, after] and delete the rest + $ python -m zipline clean --before --after + + # clean all but the last runs + $ python -m zipline clean --keep-last + + +Running Backtests with Data Bundles +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Now that the data has been ingested we can use it to run backtests with the +``run`` command. This can be specified with the ``--bundle`` option like: + +.. code-block:: bash + + $ python -m zipline run --bundle --algofile algo.py ... + + +We may also specify the date to use to look up the bundle data with the +``--bundle-date`` option. This will cause us to the the most recent bundle +ingestion that is less than or equal to the ``bundle-date``. This is how we can +run backtests with older data. The reason that this uses a less than or equal to +relationship is that we can specify the date that we ran an old backtest and get +the same data that would have been available to us on that date. The +``bundle-date`` defaults to the current day to use the most recent data. + +Default Data Bundles +~~~~~~~~~~~~~~~~~~~~ + +.. _quandl-data-bundle: + +Quandl WIKI Bundle +`````````````````` + +By default zipline comes with the ``quandl`` data bundle which uses quandl's +`WIKI dataset `_. The quandl data bundle +includes daily pricing data, splits, cash dividends, and asset metadata. This is +the bundle that ``run`` will use by default if no other bundle is specified. To +ingest this data bundle we recommend creating an account on quandl.com to get an +API key to be able to make more API requests per day. Once we have an API key we +may run: + +.. code-block:: bash + + $ QUANDL_API_KEY= python -m zipline ingest quandl + +though we may still run ``ingest`` as an anonymous quandl user (with no API +key). We may also set the ``QUANDL_DOWNLOAD_ATTEMPTS`` environment variable to +an integer which is the number of attempts that should be made to download data +from quandls servers. By default this will be 5, meaning that we will retry each +attempt 5 times. + +.. note:: + + ``QUANDL_DOWNLOAD_ATTEMPTS`` is not the total number of allowed failures, + just the number of allowed failures per request. The quandl loader will make + one request per 100 equities for the metadata followed by one request per + equity. + + +Yahoo Bundle Factories +`````````````````````` + +Zipline also ships with a factory function for creating a data bundle out of a +set of tickers from yahoo: :func:`~zipline.data.bundles.yahoo_equities`. +This makes it easy to pre-download and cache the data for a set of equities from +yahoo. This includes daily pricing data along with splits, cash dividends, and +inferred asset metadata. To create a bundle from a set of equities, add the +following to your ``~/.zipline/extensions.py`` file: + +.. code-block:: python + + from zipline.bundles import register, yahoo_equities + + # these are the tickers you would like data for + equities = { + 'AAPL', + 'MSFT', + 'GOOG', + } + register( + 'my-yahoo-equities-bundle', # name this whatever you like + yahoo_equities(equities), + ) + + +This may now be used like: + +.. code-block:: bash + + $ python -m zipline ingest my-yahoo-equities-bundle + $ python -m zipline run -f algo.py --bundle my-yahoo-equities-bundle + + +More than one yahoo equities bundle may be registered as long as they use +different names. + +Writing a New Bundle +~~~~~~~~~~~~~~~~~~~~ + +Data bundles exist to make it easy to use different data sources with +zipline. To add a new bundle, one must implement an ingest function. + +This function is responsible for loading the data into memory and passing it to +a set of writer objects provided by zipline to convert the data to zipline's +internal format. The ingest function may work by downloading data from a remote +location like the ``quandl`` bundle or yahoo bundles or it may just load files +that are already on the machine. The function is provided with writers that will +write the data to the correct location transactionally. If an ingestion fails +part way through the bundle will not be written in an incomplete state. + +The signature of the ingest function should be: + +.. code-block:: python + + ingest(environ, + asset_db_writer, + minute_bar_writer, + daily_bar_writer, + adjustment_writer, + calendar, + cache, + show_progress) + +``environ`` +``````````` + +``environ`` is a mapping representing the environment variables to use. This is +where any custom arguments needed for the ingestion should be passed, for +example: the ``quandl`` bundle uses the enviornment to pass the API key and the +download retry attempt count. + +``asset_db_writer`` +``````````````````` + +``asset_db_writer`` is an instance of :class:`~zipline.assets.AssetDBWriter`. +This is the writer for the asset metadata which provides the asset lifetimes and +the symbol to asset id (sid) mapping. This may also contain the asset name, +exchange and a few other columns. To write data, invoke +:meth:`~zipline.assets.AssetDBWriter.write` with dataframes for the various +pieces of metadata. More information about the format of the data exists in the +docs for write. + +``minute_bar_writer`` +````````````````````` + +``minute_bar_writer`` is an instance of +:class:`~zipline.data.minute_bars.BcolzMinuteBarWriter`. This writer is used to +convert data to zipline's internal bcolz format to later be read by a +:class:`~zipline.data.minute_bars.BcolzMinuteBarReader`. If minute data is +provided, users should call +:meth:`~zipline.data.minute_bars.BcolzMinuteBarWriter.write` with an iterable of +(sid, dataframe) tuples. The ``show_progress`` argument should also be forwarded +to this method. If the data source does not provide minute level data, then +there is no need to call the write method. It is also acceptable to pass an +empty iterator to :meth:`~zipline.data.minute_bars.BcolzMinuteBarWriter.write` +to signal that there is no minutely data. + +.. note:: + + The data passed to + :meth:`~zipline.data.minute_bars.BcolzMinuteBarWriter.write` may be a lazy + iterator or generator to avoid loading all of the minute data into memory at + a single time. A given sid may also appear multiple times in the data as long + as the dates are strictly increasing. + +``daily_bar_writer`` +```````````````````` + +``daily_bar_writer`` is an instance of +:class:`~zipline.data.us_equity_pricing.BcolzDailyBarWriter`. This writer is +used to convert data into zipline's internal bcolz format to later be read by a +:class:`~zipline.data.us_equity_pricing.BcolzDailyBarReader`. If daily data is +provided, users should call +:meth:`~zipline.data.minute_bars.BcolzDailyBarWriter.write` with an iterable of +(sid dataframe) tuples. The ``show_progress`` argument should also be forwarded +to this method. If the data shource does not provide daily data, then there is +no need to call the write method. It is also acceptable to pass an empty +iterable to :meth:`~zipline.data.minute_bars.BcolzMinuteBarWriter.write` to +signal that there is no daily data. If no daily data is provided but minute data +is provided, a daily rollup will happen to service daily history requests. + +.. note:: + + Like the ``minute_bar_writer``, the data passed to + :meth:`~zipline.data.minute_bars.BcolzMinuteBarWriter.write` may be a lazy + iterable or generator to avoid loading all of the data into memory at once. + Unlike the ``minute_bar_writer``, a sid may only appear once in the data + iterable. + +``adjustment_writer`` +````````````````````` + +``adjustment_writer`` is an instance of +:class:`~zipline.data.us_equity_pricing.SQLiteAdjustmentWriter`. This writer is +used to store splits, mergers, dividends, and stock dividends. The data should +be provided as dataframes and passed to +:meth:`~zipline.data.us_equity_pricing.SQLiteAdjustmentWriter.write`. Each of +these fields are optional, but the writer can accept as much of the data as you +have. + +``calendar`` +```````````` + +``calendar`` is a ``pandas.DatetimeIndex`` object holding all of the trading +days that the bundle should load data for. This is to help some bundles generate +queries for the days needed. + +``cache`` +````````` + +``cache`` is an instance of :class:`~zipline.utils.cache.dataframe_cache`. This +object is a mapping from strings to dataframes. This object is provided in case +an ingestion crashes part way through. The idea is that the ingest function +should check the cache for raw data, if it doesn't exist in the cache, it should +acquire it and then store it in the cache. Then it can parse and write the +data. The cache will be cleared only after a successful load, this prevents the +ingest function from needing to redownload all the data if there is some bug in +the parsing. If it is very fast to get the data, for example if it is coming +from another local file, then there is no need to use this cache. + +``show_progress`` +````````````````` + +``show_progress`` is a boolean indicating that the user would like to receive +feedback about the ingest function's progress fetching and writing the +data. Some examples for where to show how many files you have downloaded out of +the total needed, or how far into some data conversion the ingest function +is. One tool that may help with implementing ``show_progress`` for a loop is +:class:`~zipline.utils.cli.maybe_show_progress`. This argument should always be +forwarded to ``minute_bar_writer.write`` and ``daily_bar_writer.write``. diff --git a/docs/source/index.rst b/docs/source/index.rst index 9dad6958..b8f24239 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -5,6 +5,7 @@ install beginner-tutorial + bundles releases appendix release-process diff --git a/docs/source/whatsnew/1.0.0.txt b/docs/source/whatsnew/1.0.0.txt index 12faa317..3a11be8a 100644 --- a/docs/source/whatsnew/1.0.0.txt +++ b/docs/source/whatsnew/1.0.0.txt @@ -12,7 +12,51 @@ Development Highlights ~~~~~~~~~~ -None +New Entry Points (:issue:`xxxx`) +```````````````````````````````` + +In order to make it easier to use zipline we have updated the entry points for +a backtest. The three supported ways to run a backtest are now: + +1. :func:`zipline.run_algo` +2. ``$ python -m zipline run`` +3. ``%zipline`` (IPython magic) + +Data Bundles (:issue:`xxxx`) +```````````````````````````` + +1.0.0 introduces data bundles. Data bundles are groups of data that should be +preloaded and used to run backtests later. This allows users to not need to to +specify which tickers they are interested in each time they run an +algorithm. This also allows us to cache the data between runs. + +By default, the ``quandl`` bundle will be used which pulls data from quandl's +`WIKI dataset `_. New bundles may be +registered with :func:`zipline.data.bundles.register` like: + +.. code-block:: python + + @zipline.data.bundles.register('my-new-bundle') + def my_new_bundle_ingest(environ, + asset_db_writer, + minute_bar_writer, + daily_bar_writer, + adjustment_writer, + calendar, + cache, + show_progress): + ... + + +This function should retrieve the data it needs and then use the writers that +have been passed to write that data to disc in a location that zipline can find +later. + +This data can be used in backtests by passing the name as the ``-b / --bundle`` +argument to ``$ python -m zipline run`` or as the ``bundle`` argument to +:func:`zipline.run_algo`. + +For more information see `Data Bundles`_ for more information. Enhancements ~~~~~~~~~~~~ @@ -20,9 +64,10 @@ Enhancements * Made the data loading classes have more consistent interfaces. This includes the equity bar writers, adjustment writer, and asset db writer. The new interface is that the resource to be written to is passed at construction time - and the data to write is provided later to the `write` method as a - dataframe. This model allows us to pass these writer objects around as a - resource for other classes and functions to consume (:issue:`1109`). + and the data to write is provided later to the `write` method as + dataframes or some iterator of dataframes. This model allows us to pass these + writer objects around as a resource for other classes and functions to + consume (:issue:`1109` and :issue:`1149`). * Added masking to :class:`zipline.pipeline.CustomFactor`. Custom factors can now be passed a Filter upon instantiation. This tells the diff --git a/etc/requirements.txt b/etc/requirements.txt index a211f690..4875e914 100644 --- a/etc/requirements.txt +++ b/etc/requirements.txt @@ -15,6 +15,7 @@ numpy==1.9.2 # statsmodels in turn is required for some pandas packages scipy==0.15.1 pandas==0.16.1 +pandas-datareader==0.2.1 # Needed for parts of pandas.stats patsy==0.4.0 statsmodels==0.6.1 diff --git a/scripts/run_algo.py b/scripts/run_algo.py deleted file mode 100755 index de657a77..00000000 --- a/scripts/run_algo.py +++ /dev/null @@ -1,26 +0,0 @@ -#!/usr/bin/env python -# -# Copyright 2014 Quantopian, Inc. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -import logbook -import sys - -from zipline.utils import parse_args, run_pipeline - -if __name__ == "__main__": - logbook.StderrHandler().push_application() - parsed = parse_args(sys.argv[1:]) - run_pipeline(**parsed) - sys.exit(0) diff --git a/setup.py b/setup.py index c7e5594d..78063820 100644 --- a/setup.py +++ b/setup.py @@ -264,7 +264,6 @@ setup( author_email='opensource@quantopian.com', packages=find_packages('.', include=['zipline', 'zipline.*']), ext_modules=ext_modules, - scripts=['scripts/run_algo.py'], include_package_data=True, license='Apache 2.0', classifiers=[ diff --git a/tests/data/bundles/test_core.py b/tests/data/bundles/test_core.py new file mode 100644 index 00000000..7e61ac0d --- /dev/null +++ b/tests/data/bundles/test_core.py @@ -0,0 +1,218 @@ +import pandas as pd +from toolz import valmap +import toolz.curried.operator as op + +from zipline.assets.synthetic import make_simple_equity_info +from zipline.data.bundles import load +from zipline.data.bundles.core import _make_bundle_core +from zipline.lib.adjustment import Float64Multiply +from zipline.pipeline.loaders.synthetic import ( + make_bar_data, + expected_bar_values_2d, +) +from zipline.testing import ( + subtest, + tmp_dir, + str_to_seconds, + tmp_trading_env, +) +from zipline.testing.fixtures import ZiplineTestCase +from zipline.testing.predicates import ( + assert_equal, + assert_false, + assert_in, + assert_is, + assert_is_instance, +) +from zipline.utils.cache import dataframe_cache +from zipline.utils.functional import apply +from zipline.utils.tradingcalendar import trading_days + + +class BundleCoreTestCase(ZiplineTestCase): + def init_instance_fixtures(self): + super(BundleCoreTestCase, self).init_instance_fixtures() + (self.bundles, + self.register, + self.unregister, + self.ingest) = _make_bundle_core() + + def test_register_decorator(self): + @apply + @subtest(((c,) for c in 'abcde'), 'name') + def _(name): + @self.register(name) + def ingest(*args): + pass + + assert_in(name, self.bundles) + assert_is(self.bundles[name].ingest, ingest) + + self._check_bundles(set('abcde')) + + def test_register_call(self): + def ingest(*args): + pass + + @apply + @subtest(((c,) for c in 'abcde'), 'name') + def _(name): + self.register(name, ingest) + assert_in(name, self.bundles) + assert_is(self.bundles[name].ingest, ingest) + + assert_equal( + valmap(op.attrgetter('ingest'), self.bundles), + {k: ingest for k in 'abcde'}, + ) + self._check_bundles(set('abcde')) + + def _check_bundles(self, names): + assert_equal(set(self.bundles.keys()), names) + + for name in names: + self.unregister(name) + + assert_false(self.bundles) + + def test_ingest(self): + zipline_root = self.enter_instance_context(tmp_dir()).path + env = self.enter_instance_context(tmp_trading_env()) + + start = pd.Timestamp('2014-01-06', tz='utc') + end = pd.Timestamp('2014-01-10', tz='utc') + calendar = trading_days[trading_days.slice_indexer(start, end)] + minutes = env.minutes_for_days_in_range(calendar[0], calendar[-1]) + outer_environ = { + 'ZIPLINE_ROOT': zipline_root, + } + + sids = tuple(range(3)) + equities = make_simple_equity_info( + sids, + calendar[0], + calendar[-1], + ) + + daily_bar_data = make_bar_data(equities, calendar) + minute_bar_data = make_bar_data(equities, minutes) + first_split_ratio = 0.5 + second_split_ratio = 0.1 + splits = pd.DataFrame.from_records([ + { + 'effective_date': str_to_seconds('2014-01-08'), + 'ratio': first_split_ratio, + 'sid': 0, + }, + { + 'effective_date': str_to_seconds('2014-01-09'), + 'ratio': second_split_ratio, + 'sid': 1, + }, + ]) + + @self.register('bundle', + calendar=calendar, + opens=env.opens_in_range(calendar[0], calendar[-1]), + closes=env.closes_in_range(calendar[0], calendar[-1])) + def bundle_ingest(environ, + asset_db_writer, + minute_bar_writer, + daily_bar_writer, + adjustment_writer, + calendar, + cache, + show_progress): + assert_is(environ, outer_environ) + + asset_db_writer.write(equities=equities) + minute_bar_writer.write(minute_bar_data) + daily_bar_writer.write(daily_bar_data) + adjustment_writer.write(splits=splits) + + assert_is_instance(calendar, pd.DatetimeIndex) + assert_is_instance(cache, dataframe_cache) + assert_is_instance(show_progress, bool) + + self.ingest('bundle', environ=outer_environ) + bundle = load('bundle', environ=outer_environ) + + assert_equal(set(bundle.asset_finder.sids), set(sids)) + + columns = 'open', 'high', 'low', 'close', 'volume' + + actual = bundle.minute_bar_reader.load_raw_arrays( + columns, + minutes[0], + minutes[-1], + sids, + ) + + for actual_column, colname in zip(actual, columns): + assert_equal( + actual_column, + expected_bar_values_2d(minutes, equities, colname), + msg=colname, + ) + + actual = bundle.daily_bar_reader.load_raw_arrays( + columns, + calendar[0], + calendar[-1], + sids, + ) + for actual_column, colname in zip(actual, columns): + assert_equal( + actual_column, + expected_bar_values_2d(calendar, equities, colname), + msg=colname, + ) + adjustments_for_cols = bundle.adjustment_reader.load_adjustments( + columns, + calendar, + pd.Index(sids), + ) + for column, adjustments in zip(columns, adjustments_for_cols[:-1]): + # iterate over all the adjustments but `volume` + assert_equal( + adjustments, + { + 2: [Float64Multiply( + first_row=0, + last_row=2, + first_col=0, + last_col=0, + value=first_split_ratio, + )], + 3: [Float64Multiply( + first_row=0, + last_row=3, + first_col=1, + last_col=1, + value=second_split_ratio, + )], + }, + msg=column, + ) + + # check the volume, the value should be 1/ratio + assert_equal( + adjustments_for_cols[-1], + { + 2: [Float64Multiply( + first_row=0, + last_row=2, + first_col=0, + last_col=0, + value=1 / first_split_ratio, + )], + 3: [Float64Multiply( + first_row=0, + last_row=3, + first_col=1, + last_col=1, + value=1 / second_split_ratio, + )], + }, + msg='volume', + ) diff --git a/tests/data/bundles/test_quandl.py b/tests/data/bundles/test_quandl.py new file mode 100644 index 00000000..31627ae3 --- /dev/null +++ b/tests/data/bundles/test_quandl.py @@ -0,0 +1,243 @@ +from __future__ import division + +import numpy as np +import pandas as pd +from toolz import merge +import toolz.curried.operator as op + +from zipline.data.bundles import ingest, load, bundles +from zipline.data.bundles.quandl import ( + format_wiki_url, + format_metadata_url, +) +from zipline.lib.adjustment import Float64Multiply +from zipline.testing import ( + test_resource_path, + tmp_dir, + patch_read_csv, +) +from zipline.testing.fixtures import ZiplineTestCase +from zipline.testing.predicates import ( + assert_equal, +) +from zipline.utils.functional import apply + + +class QuandlBundleTestCase(ZiplineTestCase): + symbols = 'AAPL', 'BRK_A', 'MSFT', 'ZEN' + asset_start = pd.Timestamp('2014-01', tz='utc') + asset_end = pd.Timestamp('2015-01', tz='utc') + calendar = bundles['quandl'].calendar + start_date = calendar[0] + end_date = calendar[-1] + api_key = 'ayylmao' + columns = 'open', 'high', 'low', 'close', 'volume' + + def _expected_data(self, asset_finder): + sids = { + symbol: asset_finder.lookup_symbol( + symbol, + self.asset_start, + ).sid + for symbol in self.symbols + } + + def per_symbol(symbol): + df = pd.read_csv( + test_resource_path('quandl_samples', symbol + '.csv.gz'), + parse_dates=['Date'], + index_col='Date', + usecols=[ + 'Open', + 'High', + 'Low', + 'Close', + 'Volume', + 'Date', + 'Ex-Dividend', + 'Split Ratio', + ], + na_values=['NA'], + ).rename(columns={ + 'Open': 'open', + 'High': 'high', + 'Low': 'low', + 'Close': 'close', + 'Volume': 'volume', + 'Date': 'date', + 'Ex-Dividend': 'ex_dividend', + 'Split Ratio': 'split_ratio', + }) + df['sid'] = sids[symbol] + return df + + all_ = pd.concat(map(per_symbol, self.symbols)).set_index( + 'sid', + append=True, + ).unstack() + + # fancy list comprehension with statements + @list + @apply + def pricing(): + for column in self.columns: + vs = all_[column].values + if column == 'volume': + vs = np.nan_to_num(vs) + yield vs + + # the first index our written data will appear in the files on disk + start_idx = self.calendar.get_loc(self.asset_start, 'ffill') + 1 + + # convert an index into the raw dataframe into an index into the + # final data + i = op.add(start_idx) + + def expected_dividend_adjustment(idx, symbol): + sid = sids[symbol] + return ( + 1 - + all_.ix[idx, ('ex_dividend', sid)] / + all_.ix[idx - 1, ('close', sid)] + ) + + adjustments = [ + # ohlc + { + # dividends + i(24): [Float64Multiply( + first_row=0, + last_row=i(24), + first_col=sids['AAPL'], + last_col=sids['AAPL'], + value=expected_dividend_adjustment(24, 'AAPL'), + )], + i(87): [Float64Multiply( + first_row=0, + last_row=i(87), + first_col=sids['AAPL'], + last_col=sids['AAPL'], + value=expected_dividend_adjustment(87, 'AAPL'), + )], + i(150): [Float64Multiply( + first_row=0, + last_row=i(150), + first_col=sids['AAPL'], + last_col=sids['AAPL'], + value=expected_dividend_adjustment(150, 'AAPL'), + )], + i(214): [Float64Multiply( + first_row=0, + last_row=i(214), + first_col=sids['AAPL'], + last_col=sids['AAPL'], + value=expected_dividend_adjustment(214, 'AAPL'), + )], + + i(31): [Float64Multiply( + first_row=0, + last_row=i(31), + first_col=sids['MSFT'], + last_col=sids['MSFT'], + value=expected_dividend_adjustment(31, 'MSFT'), + )], + i(90): [Float64Multiply( + first_row=0, + last_row=i(90), + first_col=sids['MSFT'], + last_col=sids['MSFT'], + value=expected_dividend_adjustment(90, 'MSFT'), + )], + i(222): [Float64Multiply( + first_row=0, + last_row=i(222), + first_col=sids['MSFT'], + last_col=sids['MSFT'], + value=expected_dividend_adjustment(222, 'MSFT'), + )], + + # splits + i(108): [Float64Multiply( + first_row=0, + last_row=i(108), + first_col=sids['AAPL'], + last_col=sids['AAPL'], + value=7.0, + )], + }, + ] * (len(self.columns) - 1) + [ + # volume + { + i(108): [Float64Multiply( + first_row=0, + last_row=i(108), + first_col=sids['AAPL'], + last_col=sids['AAPL'], + value=1 / 7, + )], + } + ] + return pricing, adjustments + + def test_bundle(self): + url_map = merge( + { + format_wiki_url( + self.api_key, + symbol, + self.start_date, + self.end_date, + ): test_resource_path('quandl_samples', symbol + '.csv.gz') + for symbol in self.symbols + }, + { + format_metadata_url(self.api_key, n): test_resource_path( + 'quandl_samples', + 'metadata-%d.csv.gz' % n, + ) + for n in (1, 2) + }, + ) + zipline_root = self.enter_instance_context(tmp_dir()).path + environ = { + 'ZIPLINE_ROOT': zipline_root, + 'QUANDL_API_KEY': self.api_key, + } + + with patch_read_csv(url_map, strict=True): + ingest('quandl', environ=environ) + + bundle = load('quandl', environ=environ) + sids = 0, 1, 2, 3 + assert_equal(set(bundle.asset_finder.sids), set(sids)) + + for equity in bundle.asset_finder.retrieve_all(sids): + assert_equal(equity.start_date, self.asset_start, msg=equity) + assert_equal(equity.end_date, self.asset_end, msg=equity) + + cal = self.calendar + actual = bundle.daily_bar_reader.load_raw_arrays( + self.columns, + cal[cal.get_loc(self.asset_start, 'bfill')], + cal[cal.get_loc(self.asset_end, 'ffill')], + sids, + ) + expected_pricing, expected_adjustments = self._expected_data( + bundle.asset_finder, + ) + assert_equal(actual, expected_pricing, array_decimal=2) + + adjustments_for_cols = bundle.adjustment_reader.load_adjustments( + self.columns, + cal, + pd.Index(sids), + ) + + for column, adjustments, expected in zip(self.columns, + adjustments_for_cols, + expected_adjustments): + assert_equal( + adjustments, + expected, + msg=column, + ) diff --git a/tests/data/bundles/test_yahoo.py b/tests/data/bundles/test_yahoo.py new file mode 100644 index 00000000..03bf4ad9 --- /dev/null +++ b/tests/data/bundles/test_yahoo.py @@ -0,0 +1,201 @@ +import numpy as np +import pandas as pd +from six.moves.urllib.parse import urlparse, parse_qs +from toolz import flip, identity +from toolz.curried import merge_with, operator as op + +from zipline.data.bundles.core import _make_bundle_core +from zipline.data.bundles import yahoo_equities, load +from zipline.lib.adjustment import Float64Multiply +from zipline.testing import test_resource_path, tmp_dir, read_compressed +from zipline.testing.fixtures import WithResponses, ZiplineTestCase +from zipline.testing.predicates import assert_equal +from zipline.utils.tradingcalendar import trading_days + + +class YahooBundleTestCase(WithResponses, ZiplineTestCase): + symbols = 'AAPL', 'IBM', 'MSFT' + columns = 'open', 'high', 'low', 'close', 'volume' + asset_start = pd.Timestamp('2014-01-02', tz='utc') + asset_end = pd.Timestamp('2014-12-31', tz='utc') + calendar = trading_days[ + (trading_days >= asset_start) & + (trading_days <= asset_end) + ] + + @classmethod + def init_class_fixtures(cls): + super(YahooBundleTestCase, cls).init_class_fixtures() + (cls.bundles, + cls.register, + cls.unregister, + cls.ingest) = map(staticmethod, _make_bundle_core()) + + def _expected_data(self): + sids = 0, 1, 2 + modifier = { + 'low': 0, + 'open': 1, + 'close': 2, + 'high': 3, + 'volume': 0, + } + pricing = [ + np.hstack(( + np.arange(252, dtype='float64')[:, np.newaxis] + + 1 + + sid * 10000 + + modifier[column] * 1000 + for sid in sorted(sids) + )) + for column in self.columns + ] + + # There are two dividends and 1 split for each company. + + def dividend_adjustment(sid, which): + """The dividends occur at indices 252 // 4 and 3 * 252 / 4 + with a cash amount of sid + 1 / 10 and sid + 2 / 10 + """ + if which == 'first': + idx = 252 // 4 + else: + idx = 3 * 252 // 4 + + return { + idx: [Float64Multiply( + first_row=0, + last_row=idx, + first_col=sid, + last_col=sid, + value=float( + 1 - + ((sid + 1 + (which == 'second')) / 10) / + (idx - 1 + sid * 10000 + 2000) + ), + )], + } + + def split_adjustment(sid, volume): + """The splits occur at index 252 // 2 with a ratio of (sid + 1):1 + """ + idx = 252 // 2 + return { + idx: [Float64Multiply( + first_row=0, + last_row=idx, + first_col=sid, + last_col=sid, + value=(identity if volume else op.truediv(1))(sid + 2), + )], + } + + merge_adjustments = merge_with(flip(sum, [])) + + adjustments = [ + # ohlc + merge_adjustments( + *tuple(dividend_adjustment(sid, 'first') for sid in sids) + + tuple(dividend_adjustment(sid, 'second') for sid in sids) + + tuple(split_adjustment(sid, volume=False) for sid in sids) + ) + ] * (len(self.columns) - 1) + [ + # volume + merge_adjustments( + split_adjustment(sid, volume=True) for sid in sids + ), + ] + + return pricing, adjustments + + def test_bundle(self): + + def get_symbol_from_url(url): + params = parse_qs(urlparse(url).query) + symbol, = params['s'] + return symbol + + def pricing_callback(request): + headers = { + 'content-encoding': 'gzip', + 'content-type': 'text/csv', + } + path = test_resource_path( + 'yahoo_samples', + get_symbol_from_url(request.url) + '.csv.gz', + ) + with open(path, 'rb') as f: + return ( + 200, + headers, + f.read(), + ) + + for _ in range(3): + self.responses.add_callback( + self.responses.GET, + 'http://ichart.finance.yahoo.com/table.csv', + pricing_callback, + ) + + def adjustments_callback(request): + path = test_resource_path( + 'yahoo_samples', + get_symbol_from_url(request.url) + '.adjustments.gz', + ) + return 200, {}, read_compressed(path) + + for _ in range(3): + self.responses.add_callback( + self.responses.GET, + 'http://ichart.finance.yahoo.com/x', + adjustments_callback, + ) + + cal = self.calendar + self.register( + 'bundle', + yahoo_equities(self.symbols), + calendar=cal, + ) + + zipline_root = self.enter_instance_context(tmp_dir()).path + environ = { + 'ZIPLINE_ROOT': zipline_root, + } + + self.ingest('bundle', environ=environ) + bundle = load('bundle', environ=environ) + + sids = 0, 1, 2 + equities = bundle.asset_finder.retrieve_all(sids) + for equity, expected_symbol in zip(equities, self.symbols): + assert_equal(equity.symbol, expected_symbol) + + for equity in bundle.asset_finder.retrieve_all(sids): + assert_equal(equity.start_date, self.asset_start, msg=equity) + assert_equal(equity.end_date, self.asset_end, msg=equity) + + actual = bundle.daily_bar_reader.load_raw_arrays( + self.columns, + cal[cal.get_loc(self.asset_start, 'bfill')], + cal[cal.get_loc(self.asset_end, 'ffill')], + sids, + ) + expected_pricing, expected_adjustments = self._expected_data() + assert_equal(actual, expected_pricing, array_decimal=2) + + adjustments_for_cols = bundle.adjustment_reader.load_adjustments( + self.columns, + cal, + pd.Index(sids), + ) + + for column, adjustments, expected in zip(self.columns, + adjustments_for_cols, + expected_adjustments): + assert_equal( + adjustments, + expected, + msg=column, + ) diff --git a/tests/data/test_minute_bars.py b/tests/data/test_minute_bars.py index d99c0fc4..edddf060 100644 --- a/tests/data/test_minute_bars.py +++ b/tests/data/test_minute_bars.py @@ -24,6 +24,7 @@ from numpy import ( float64, full, nan, + transpose, zeros, ) from numpy.testing import assert_almost_equal, assert_array_equal @@ -100,7 +101,7 @@ class BcolzMinuteBarTestCase(TestCase): 'volume': [50.0] }, index=[minute]) - self.writer.write(sid, data) + self.writer.write_sid(sid, data) open_price = self.reader.get_value(sid, minute, 'open') @@ -135,7 +136,7 @@ class BcolzMinuteBarTestCase(TestCase): 'volume': [50.0, 51.0] }, index=[minute_0, minute_1]) - self.writer.write(sid, data) + self.writer.write_sid(sid, data) open_price = self.reader.get_value(sid, minute_0, 'open') @@ -190,7 +191,7 @@ class BcolzMinuteBarTestCase(TestCase): 'volume': [50.0] }, index=[minute]) - self.writer.write(sid, data) + self.writer.write_sid(sid, data) open_price = self.reader.get_value(sid, minute, 'open') @@ -224,7 +225,7 @@ class BcolzMinuteBarTestCase(TestCase): 'volume': [0] }, index=[minute]) - self.writer.write(sid, data) + self.writer.write_sid(sid, data) open_price = self.reader.get_value(sid, minute, 'open') @@ -267,7 +268,7 @@ class BcolzMinuteBarTestCase(TestCase): 'volume': [50.0, 51.0] }, index=minutes) - self.writer.write(sid, data) + self.writer.write_sid(sid, data) minute = minutes[0] @@ -325,10 +326,10 @@ class BcolzMinuteBarTestCase(TestCase): 'volume': [50.0] }, index=[minute]) - self.writer.write(sid, data) + self.writer.write_sid(sid, data) with self.assertRaises(BcolzMinuteOverlappingData): - self.writer.write(sid, data) + self.writer.write_sid(sid, data) def test_write_multiple_sids(self): """ @@ -361,7 +362,7 @@ class BcolzMinuteBarTestCase(TestCase): 'volume': [100.0] }, index=[minute]) - self.writer.write(sids[0], data) + self.writer.write_sid(sids[0], data) data = DataFrame( data={ @@ -372,7 +373,7 @@ class BcolzMinuteBarTestCase(TestCase): 'volume': [200.0] }, index=[minute]) - self.writer.write(sids[1], data) + self.writer.write_sid(sids[1], data) sid = sids[0] @@ -442,7 +443,7 @@ class BcolzMinuteBarTestCase(TestCase): 'volume': [100.0] }, index=[minute]) - self.writer.write(sid, data) + self.writer.write_sid(sid, data) open_price = self.reader.get_value(sid, minute, 'open') @@ -489,12 +490,13 @@ class BcolzMinuteBarTestCase(TestCase): 'volume': full(9, 0), }, index=[minutes]) - self.writer.write(sid, data) + self.writer.write_sid(sid, data) fields = ['open', 'high', 'low', 'close', 'volume'] - ohlcv_window = self.reader.unadjusted_window( - fields, minutes[0], minutes[-1], [sid]) + ohlcv_window = list(map(transpose, self.reader.load_raw_arrays( + fields, minutes[0], minutes[-1], [sid], + ))) for i, field in enumerate(fields): if field != 'volume': @@ -531,12 +533,13 @@ class BcolzMinuteBarTestCase(TestCase): 'volume': full(9, 0), }, index=[minutes]) - self.writer.write(sid, data) + self.writer.write_sid(sid, data) fields = ['open', 'high', 'low', 'close', 'volume'] - ohlcv_window = self.reader.unadjusted_window( - fields, minutes[0], minutes[-1], [sid]) + ohlcv_window = list(map(transpose, self.reader.load_raw_arrays( + fields, minutes[0], minutes[-1], [sid], + ))) for i, field in enumerate(fields): if field != 'volume': @@ -630,7 +633,7 @@ class BcolzMinuteBarTestCase(TestCase): 'volume': [1000, 0, 1001] }, index=minutes) - self.writer.write(sids[0], data_1) + self.writer.write_sid(sids[0], data_1) data_2 = DataFrame( data={ @@ -641,14 +644,15 @@ class BcolzMinuteBarTestCase(TestCase): 'volume': [2000, 0, 2001] }, index=minutes) - self.writer.write(sids[1], data_2) + self.writer.write_sid(sids[1], data_2) reader = BcolzMinuteBarReader(self.dest) columns = ['open', 'high', 'low', 'close', 'volume'] sids = [sids[0], sids[1]] - arrays = reader.unadjusted_window( - columns, minutes[0], minutes[-1], sids) + arrays = list(map(transpose, reader.load_raw_arrays( + columns, minutes[0], minutes[-1], sids, + ))) data = {sids[0]: data_1, sids[1]: data_2} @@ -681,7 +685,7 @@ class BcolzMinuteBarTestCase(TestCase): 'volume': [1000, 1001, 1002], }, index=minutes) - self.writer.write(sids[0], data_1) + self.writer.write_sid(sids[0], data_1) data_2 = DataFrame( data={ @@ -692,14 +696,15 @@ class BcolzMinuteBarTestCase(TestCase): 'volume': [2000, 2001, 2002], }, index=minutes) - self.writer.write(sids[1], data_2) + self.writer.write_sid(sids[1], data_2) reader = BcolzMinuteBarReader(self.dest) columns = ['open', 'high', 'low', 'close', 'volume'] sids = [sids[0], sids[1]] - arrays = reader.unadjusted_window( - columns, minutes[0], minutes[-1], sids) + arrays = list(map(transpose, reader.load_raw_arrays( + columns, minutes[0], minutes[-1], sids, + ))) data = {sids[0]: data_1, sids[1]: data_2} diff --git a/tests/data/test_us_equity_pricing.py b/tests/data/test_us_equity_pricing.py index 9391809b..261c2dc0 100644 --- a/tests/data/test_us_equity_pricing.py +++ b/tests/data/test_us_equity_pricing.py @@ -33,14 +33,13 @@ from zipline.data.us_equity_pricing import ( BcolzDailyBarReader, NoDataOnDate, ) -from zipline.pipeline.data import USEquityPricing from zipline.pipeline.loaders.synthetic import ( OHLCV, asset_start, asset_end, - expected_daily_bar_value, - expected_daily_bar_values_2d, - make_daily_bar_data, + expected_bar_value, + expected_bar_values_2d, + make_bar_data, ) from zipline.testing import seconds_to_timestamp from zipline.testing.fixtures import ( @@ -89,7 +88,7 @@ class BcolzDailyBarTestCase(WithBcolzDailyBarReader, ZiplineTestCase): @classmethod def make_daily_bar_data(cls): - return make_daily_bar_data( + return make_bar_data( EQUITY_INFO, cls.bcolz_daily_bar_days, ) @@ -129,7 +128,7 @@ class BcolzDailyBarTestCase(WithBcolzDailyBarReader, ZiplineTestCase): for asset_id in self.assets: for date in self.dates_for_asset(asset_id): self.assertEqual( - expected_daily_bar_value( + expected_bar_value( asset_id, date, column @@ -198,18 +197,18 @@ class BcolzDailyBarTestCase(WithBcolzDailyBarReader, ZiplineTestCase): for column, result in zip(columns, results): assert_array_equal( result, - expected_daily_bar_values_2d( + expected_bar_values_2d( dates, EQUITY_INFO, - column.name, + column, ) ) @parameterized.expand([ - ([USEquityPricing.open],), - ([USEquityPricing.close, USEquityPricing.volume],), - ([USEquityPricing.volume, USEquityPricing.high, USEquityPricing.low],), - (USEquityPricing.columns,), + (['open'],), + (['close', 'volume'],), + (['volume', 'high', 'low'],), + (['open', 'high', 'low', 'close', 'volume'],), ]) def test_read(self, columns): self._check_read_results( @@ -224,7 +223,7 @@ class BcolzDailyBarTestCase(WithBcolzDailyBarReader, ZiplineTestCase): Test loading with queries that starts on the first day of each asset's lifetime. """ - columns = [USEquityPricing.high, USEquityPricing.volume] + columns = ['high', 'volume'] for asset in self.assets: self._check_read_results( columns, @@ -238,7 +237,7 @@ class BcolzDailyBarTestCase(WithBcolzDailyBarReader, ZiplineTestCase): Test loading with queries that start on the last day of each asset's lifetime. """ - columns = [USEquityPricing.close, USEquityPricing.volume] + columns = ['close', 'volume'] for asset in self.assets: self._check_read_results( columns, @@ -252,7 +251,7 @@ class BcolzDailyBarTestCase(WithBcolzDailyBarReader, ZiplineTestCase): Test loading with queries that end on the first day of each asset's lifetime. """ - columns = [USEquityPricing.close, USEquityPricing.volume] + columns = ['close', 'volume'] for asset in self.assets: self._check_read_results( columns, @@ -266,7 +265,7 @@ class BcolzDailyBarTestCase(WithBcolzDailyBarReader, ZiplineTestCase): Test loading with queries that end on the last day of each asset's lifetime. """ - columns = [USEquityPricing.close, USEquityPricing.volume] + columns = ['close', 'volume'] for asset in self.assets: self._check_read_results( columns, diff --git a/tests/finance/test_slippage.py b/tests/finance/test_slippage.py index 67f8a37c..61720f69 100644 --- a/tests/finance/test_slippage.py +++ b/tests/finance/test_slippage.py @@ -58,18 +58,16 @@ class SlippageTestCase(WithSimParams, WithDataPortal, ZiplineTestCase): @classmethod def make_minute_bar_data(cls): - return { - 133: pd.DataFrame( - { - 'open': [3.0, 3.0, 3.5, 4.0, 3.5], - 'high': [3.15, 3.15, 3.15, 3.15, 3.15], - 'low': [2.85, 2.85, 2.85, 2.85, 2.85], - 'close': [3.0, 3.5, 4.0, 3.5, 3.0], - 'volume': [2000, 2000, 2000, 2000, 2000], - }, - index=cls.minutes, - ), - } + yield 133, pd.DataFrame( + { + 'open': [3.0, 3.0, 3.5, 4.0, 3.5], + 'high': [3.15, 3.15, 3.15, 3.15, 3.15], + 'low': [2.85, 2.85, 2.85, 2.85, 2.85], + 'close': [3.0, 3.5, 4.0, 3.5, 3.0], + 'volume': [2000, 2000, 2000, 2000, 2000], + }, + index=cls.minutes, + ) @classmethod def init_class_fixtures(cls): @@ -77,8 +75,8 @@ class SlippageTestCase(WithSimParams, WithDataPortal, ZiplineTestCase): cls.ASSET133 = cls.env.asset_finder.retrieve_asset(133) def test_volume_share_slippage(self): - assets = { - 133: pd.DataFrame( + assets = ( + (133, pd.DataFrame( { 'open': [3.00], 'high': [3.15], @@ -87,8 +85,8 @@ class SlippageTestCase(WithSimParams, WithDataPortal, ZiplineTestCase): 'volume': [200], }, index=[self.minutes[0]], - ), - } + )), + ) days = pd.date_range( start=normalize_date(self.minutes[0]), end=normalize_date(self.minutes[-1]) @@ -465,8 +463,8 @@ class SlippageTestCase(WithSimParams, WithDataPortal, ZiplineTestCase): data['sid'] = self.ASSET133 order = Order(**data) - assets = { - 133: pd.DataFrame( + assets = ( + (133, pd.DataFrame( { 'open': [event_data['open']], 'high': [event_data['high']], @@ -475,8 +473,8 @@ class SlippageTestCase(WithSimParams, WithDataPortal, ZiplineTestCase): 'volume': [event_data['volume']], }, index=[pd.Timestamp('2006-01-05 14:31', tz='UTC')], - ), - } + )), + ) days = pd.date_range( start=normalize_date(self.minutes[0]), end=normalize_date(self.minutes[-1]) diff --git a/tests/pipeline/test_engine.py b/tests/pipeline/test_engine.py index 8eba9dfb..f00d349a 100644 --- a/tests/pipeline/test_engine.py +++ b/tests/pipeline/test_engine.py @@ -40,8 +40,17 @@ from toolz import merge from zipline.assets.synthetic import make_rotating_equity_info from zipline.lib.adjustment import MULTIPLY -from zipline.pipeline import CustomFactor, Pipeline -from zipline.pipeline.data import Column, DataSet, USEquityPricing +from zipline.pipeline.loaders.synthetic import PrecomputedLoader +from zipline.pipeline import Pipeline +from zipline.pipeline.data import USEquityPricing, DataSet, Column +from zipline.pipeline.loaders.equity_pricing_loader import ( + USEquityPricingLoader, +) +from zipline.pipeline.factors import CustomFactor +from zipline.pipeline.loaders.synthetic import ( + make_bar_data, + expected_bar_values_2d, +) from zipline.pipeline.engine import SimplePipelineEngine from zipline.pipeline.factors import ( AverageDollarVolume, @@ -52,15 +61,7 @@ from zipline.pipeline.factors import ( MaxDrawdown, SimpleMovingAverage, ) -from zipline.pipeline.loaders.equity_pricing_loader import ( - USEquityPricingLoader, -) from zipline.pipeline.loaders.frame import DataFrameLoader -from zipline.pipeline.loaders.synthetic import ( - expected_daily_bar_values_2d, - make_daily_bar_data, - PrecomputedLoader, -) from zipline.pipeline.term import NotSpecified from zipline.testing import ( product_upper_triangle, @@ -925,7 +926,7 @@ class SyntheticBcolzTestCase(WithAdjustmentReader, @classmethod def make_daily_bar_data(cls): - return make_daily_bar_data( + return make_bar_data( cls.equity_info, cls.bcolz_daily_bar_days, ) @@ -999,7 +1000,7 @@ class SyntheticBcolzTestCase(WithAdjustmentReader, # computed results to be computed using values anchored on the # **previous** day's data. expected_raw = rolling_mean( - expected_daily_bar_values_2d( + expected_bar_values_2d( dates - self.env.trading_day, self.equity_info, 'close', diff --git a/tests/pipeline/test_us_equity_pricing_loader.py b/tests/pipeline/test_us_equity_pricing_loader.py index 12a44f95..1b319235 100644 --- a/tests/pipeline/test_us_equity_pricing_loader.py +++ b/tests/pipeline/test_us_equity_pricing_loader.py @@ -37,8 +37,8 @@ from toolz.curried.operator import getitem from zipline.lib.adjustment import Float64Multiply from zipline.pipeline.loaders.synthetic import ( NullAdjustmentReader, - make_daily_bar_data, - expected_daily_bar_values_2d, + make_bar_data, + expected_bar_values_2d, ) from zipline.pipeline.loaders.equity_pricing_loader import ( USEquityPricingLoader, @@ -282,7 +282,7 @@ class USEquityPricingLoaderTestCase(WithAdjustmentReader, @classmethod def make_daily_bar_data(cls): - return make_daily_bar_data( + return make_bar_data( EQUITY_INFO, cls.bcolz_daily_bar_days, ) @@ -364,7 +364,7 @@ class USEquityPricingLoaderTestCase(WithAdjustmentReader, ) adjustments = self.adjustment_reader.load_adjustments( - columns, + [c.name for c in columns], query_days, self.assets, ) @@ -410,7 +410,7 @@ class USEquityPricingLoaderTestCase(WithAdjustmentReader, ) adjustments = adjustment_reader.load_adjustments( - columns, + [c.name for c in columns], query_days, self.assets, ) @@ -429,12 +429,12 @@ class USEquityPricingLoaderTestCase(WithAdjustmentReader, ) closes, volumes = map(getitem(results), columns) - expected_baseline_closes = expected_daily_bar_values_2d( + expected_baseline_closes = expected_bar_values_2d( shifted_query_days, self.asset_info, 'close', ) - expected_baseline_volumes = expected_daily_bar_values_2d( + expected_baseline_volumes = expected_bar_values_2d( shifted_query_days, self.asset_info, 'volume', @@ -506,12 +506,12 @@ class USEquityPricingLoaderTestCase(WithAdjustmentReader, ) highs, volumes = map(getitem(results), columns) - expected_baseline_highs = expected_daily_bar_values_2d( + expected_baseline_highs = expected_bar_values_2d( shifted_query_days, self.asset_info, 'high', ) - expected_baseline_volumes = expected_daily_bar_values_2d( + expected_baseline_volumes = expected_bar_values_2d( shifted_query_days, self.asset_info, 'volume', diff --git a/tests/resources/example_data.tar.gz b/tests/resources/example_data.tar.gz new file mode 100644 index 00000000..7957345e Binary files /dev/null and b/tests/resources/example_data.tar.gz differ diff --git a/tests/resources/quandl_samples/AAPL.csv.gz b/tests/resources/quandl_samples/AAPL.csv.gz index 4c2c4e23..94a97af4 100644 Binary files a/tests/resources/quandl_samples/AAPL.csv.gz and b/tests/resources/quandl_samples/AAPL.csv.gz differ diff --git a/tests/resources/quandl_samples/BRK_A.csv.gz b/tests/resources/quandl_samples/BRK_A.csv.gz index 9a79fe1e..1893fabc 100644 Binary files a/tests/resources/quandl_samples/BRK_A.csv.gz and b/tests/resources/quandl_samples/BRK_A.csv.gz differ diff --git a/tests/resources/quandl_samples/MSFT.csv.gz b/tests/resources/quandl_samples/MSFT.csv.gz index b55a4ec2..f1e374f1 100644 Binary files a/tests/resources/quandl_samples/MSFT.csv.gz and b/tests/resources/quandl_samples/MSFT.csv.gz differ diff --git a/tests/resources/quandl_samples/ZEN.csv.gz b/tests/resources/quandl_samples/ZEN.csv.gz index 1f5f493b..b0d8ebb8 100644 Binary files a/tests/resources/quandl_samples/ZEN.csv.gz and b/tests/resources/quandl_samples/ZEN.csv.gz differ diff --git a/tests/resources/quandl_samples/metadata-1.csv.gz b/tests/resources/quandl_samples/metadata-1.csv.gz new file mode 100644 index 00000000..4bf475bc Binary files /dev/null and b/tests/resources/quandl_samples/metadata-1.csv.gz differ diff --git a/tests/resources/quandl_samples/metadata-2.csv.gz b/tests/resources/quandl_samples/metadata-2.csv.gz new file mode 100644 index 00000000..24a71e38 Binary files /dev/null and b/tests/resources/quandl_samples/metadata-2.csv.gz differ diff --git a/tests/resources/quandl_samples/rebuild_samples.py b/tests/resources/quandl_samples/rebuild_samples.py index 6135ff52..377d90ff 100644 --- a/tests/resources/quandl_samples/rebuild_samples.py +++ b/tests/resources/quandl_samples/rebuild_samples.py @@ -2,13 +2,14 @@ Script for rebuilding the samples for the Quandl tests. """ from __future__ import print_function +from operator import methodcaller import pandas as pd import requests from zipline.data.quandl import format_wiki_url -from zipline.utils.test_utils import test_resource_path, write_compressed +from zipline.testing import test_resource_path, write_compressed def zipfile_path(symbol): @@ -18,7 +19,14 @@ def zipfile_path(symbol): def main(): start_date = pd.Timestamp('2014') end_date = pd.Timestamp('2015') - symbols = ['AAPL', 'MSFT', 'BRK_A', 'ZEN'] + symbols = 'AAPL', 'MSFT', 'BRK_A', 'ZEN' + names = ( + 'Apple Inc.', + 'Microsoft Corporation', + 'Berkshire Hathaway Inc. Class A', + 'Zendesk Inc', + ) + print('Downloading equity data') for sym in symbols: url = format_wiki_url( api_key=None, @@ -26,14 +34,31 @@ def main(): start_date=start_date, end_date=end_date, ) - print("Fetching from %s" % url) + print('Fetching from %s' % url) response = requests.get(url) response.raise_for_status() path = zipfile_path(sym) - print("Writing compressed data to %s" % path) + print('Writing compressed data to %s' % path) write_compressed(path, response.content) + print('Writing mock metadata') + cols = b'dataset_code,name,oldest_available_date,newest_available_date\n' + metadata = cols + b'\n'.join( + b','.join(map(methodcaller('encode', 'ascii'), ( + symbol, + name, + str(start_date.date()), str(end_date.date())) + )) + for symbol, name in zip(symbols, names) + ) + path = zipfile_path('metadata-1') + print('Writing compressed data to %s' % path) + write_compressed(path, metadata) + path = zipfile_path('metadata-2') + print('Writing compressed data to %s' % path) + write_compressed(path, cols) + if __name__ == '__main__': main() diff --git a/tests/resources/rebuild_example_data b/tests/resources/rebuild_example_data new file mode 100755 index 00000000..957d72f7 --- /dev/null +++ b/tests/resources/rebuild_example_data @@ -0,0 +1,120 @@ +#!/usr/bin/env python +from code import InteractiveConsole +import readline # noqa +import shutil +import tarfile + +import click +import numpy as np +import pandas as pd + +from zipline import examples, run_algorithm +from zipline.testing import test_resource_path, tmp_dir +from zipline.utils.cache import dataframe_cache + +banner = """ +Please verify that the new perfomance is more correct than the old performance. + +To do this, please inspect `new` and `old` which are mappings from the name of +the example to the results. + +If you are sure that the new results are more correct, or that the difference +is acceptable, please call `correct()`. Otherwise, call `incorrect()`. + +Note +---- +Remember to run this with the other supported versions of pandas! +""" + + +def eof(*args, **kwargs): + raise EOFError() + + +@click.command() +@click.pass_context +def main(ctx): + """Rebuild the perf data for test_examples + """ + example_path = test_resource_path('example_data.tar.gz') + with tmp_dir() as d: + with tarfile.open(example_path) as tar: + tar.extractall(d.path) + + mods = ( + (e, getattr(examples, e)) + for e in dir(examples) + if not e.startswith('_') + ) + + new_perf_path = d.getpath( + 'example_data/new_perf/%s' % pd.__version__.replace('.', '-'), + ) + c = dataframe_cache( + new_perf_path, + serialization='pickle:2', + ) + with c: + for name, mod in mods: + c[name] = run_algorithm( + handle_data=mod.handle_data, + initialize=mod.initialize, + before_trading_start=getattr( + mod, 'before_trading_start', None, + ), + analyze=getattr(mod, 'analyze', None), + bundle='test', + environ={ + 'ZIPLINE_ROOT': d.getpath('example_data/root'), + }, + **mod._test_args() + ) + + correct_called = [False] + + console = None + + def _exit(*args, **kwargs): + console.raw_input = eof + + def correct(): + correct_called[0] = True + _exit() + + expected_perf_path = d.getpath( + 'example_data/expected_perf/%s' % + pd.__version__.replace('.', '-'), + ) + + # allow users to run some analysis to make sure that the new + # results check out + console = InteractiveConsole({ + 'correct': correct, + 'exit': _exit, + 'incorrect': _exit, + 'new': c, + 'np': np, + 'old': dataframe_cache( + expected_perf_path, + serialization='pickle', + ), + 'pd': pd, + }) + console.interact(banner) + + if not correct_called[0]: + ctx.fail( + '`correct()` was not called! This means that the new' + ' results will not be written', + ) + + # move the new results to the expected path + shutil.rmtree(expected_perf_path) + shutil.copytree(new_perf_path, expected_perf_path) + + with tarfile.open(example_path, 'w|gz') as tar: + tar.add(d.getpath('example_data'), 'example_data') + + +if __name__ == '__main__': + main() diff --git a/tests/resources/yahoo_samples/AAPL.adjustments.gz b/tests/resources/yahoo_samples/AAPL.adjustments.gz new file mode 100644 index 00000000..cfcc5d4e Binary files /dev/null and b/tests/resources/yahoo_samples/AAPL.adjustments.gz differ diff --git a/tests/resources/yahoo_samples/AAPL.csv.gz b/tests/resources/yahoo_samples/AAPL.csv.gz new file mode 100644 index 00000000..3ddf5813 Binary files /dev/null and b/tests/resources/yahoo_samples/AAPL.csv.gz differ diff --git a/tests/resources/yahoo_samples/IBM.adjustments.gz b/tests/resources/yahoo_samples/IBM.adjustments.gz new file mode 100644 index 00000000..98a5b74d Binary files /dev/null and b/tests/resources/yahoo_samples/IBM.adjustments.gz differ diff --git a/tests/resources/yahoo_samples/IBM.csv.gz b/tests/resources/yahoo_samples/IBM.csv.gz new file mode 100644 index 00000000..fbc42c69 Binary files /dev/null and b/tests/resources/yahoo_samples/IBM.csv.gz differ diff --git a/tests/resources/yahoo_samples/MSFT.adjustments.gz b/tests/resources/yahoo_samples/MSFT.adjustments.gz new file mode 100644 index 00000000..9367baf3 Binary files /dev/null and b/tests/resources/yahoo_samples/MSFT.adjustments.gz differ diff --git a/tests/resources/yahoo_samples/MSFT.csv.gz b/tests/resources/yahoo_samples/MSFT.csv.gz new file mode 100644 index 00000000..690fdbec Binary files /dev/null and b/tests/resources/yahoo_samples/MSFT.csv.gz differ diff --git a/tests/resources/yahoo_samples/rebuild_samples b/tests/resources/yahoo_samples/rebuild_samples new file mode 100644 index 00000000..7c45e4a6 --- /dev/null +++ b/tests/resources/yahoo_samples/rebuild_samples @@ -0,0 +1,79 @@ +#!/usr/bin/env python +""" +Script for rebuilding the samples for the Yahoo tests. +""" +from textwrap import dedent + +import numpy as np +import pandas as pd + +from zipline.testing import test_resource_path, write_compressed +from zipline.utils.tradingcalendar import trading_days + + +def zipfile_path(symbol, ext): + return test_resource_path('yahoo_samples', symbol + ext + '.gz') + + +def pricing_for_sid(sid): + modifier = { + 'Low': 0, + 'Open': 1, + 'Close': 2, + 'High': 3, + 'Volume': 0, + } + + def column(name): + return np.arange(252) + 1 + sid * 10000 + modifier[name] * 1000 + + return pd.DataFrame( + data={ + 'Date': trading_days[ + (trading_days >= pd.Timestamp('2014')) & + (trading_days < pd.Timestamp('2015')) + ], + 'Open': column('Open'), + 'High': column('High'), + 'Low': column('Low'), + 'Close': column('Close'), + 'Volume': column('Volume'), + 'Adj Close': 0, + }, + columns=[ + 'Date', 'Open', 'High', 'Low', 'Close', 'Volume', 'Adj Close', + ], + ).to_csv(index=False, date_format='%Y-%m-%d').encode('ascii') + + +def adjustments_for_sid(sid): + """This is not exactly a csv... thanks yahoo. + """ + return dedent( + """\ + Date,Dividends + DIVIDEND, 20140403,0.{p1}00000 + SPLIT, 20140703,{p2}:1 + DIVIDEND, 20141002,0.{p2}00000 + STARTDATE, 20140102 + ENDDATE, 20141231 + TOTALSIZE, 2 + """.format(p1=sid + 1, p2=sid + 2), + ).encode('ascii') + + +def main(): + symbols = 'AAPL', 'IBM', 'MSFT' + + for sid, symbol in enumerate(symbols): + write_compressed( + zipfile_path(symbol, '.csv'), + pricing_for_sid(sid), + ) + write_compressed( + zipfile_path(symbol, '.adjustments'), + adjustments_for_sid(sid), + ) + +if __name__ == '__main__': + main() diff --git a/tests/test_algorithm.py b/tests/test_algorithm.py index c74a346d..fbfda6ab 100644 --- a/tests/test_algorithm.py +++ b/tests/test_algorithm.py @@ -29,7 +29,6 @@ from testfixtures import TempDirectory import numpy as np import pandas as pd import pytz -from toolz import merge from zipline import TradingAlgorithm from zipline.api import FixedSlippage @@ -1039,25 +1038,20 @@ class TestBeforeTradingStart(WithDataPortal, index=asset_minutes, ) split_data.iloc[780:] = split_data.iloc[780:] / 2.0 - return merge( - { - sid: create_minute_df_for_asset( - cls.env, - cls.data_start, - cls.sim_params.period_end, - ) - for sid in (1, 8554) - }, - { - 2: create_minute_df_for_asset( - cls.env, - cls.data_start, - cls.sim_params.period_end, - 50, - ), - cls.SPLIT_ASSET_SID: split_data, - }, + for sid in (1, 8554): + yield sid, create_minute_df_for_asset( + cls.env, + cls.data_start, + cls.sim_params.period_end, + ) + + yield 2, create_minute_df_for_asset( + cls.env, + cls.data_start, + cls.sim_params.period_end, + 50, ) + yield cls.SPLIT_ASSET_SID, split_data @classmethod def make_splits_data(cls): @@ -2552,18 +2546,16 @@ class TestOrderCancelation(WithDataPortal, minutes_arr = np.arange(1, 1 + minutes_count) # normal test data, but volume is pinned at 1 share per minute - return { - 1: pd.DataFrame( - { - 'open': minutes_arr + 1, - 'high': minutes_arr + 2, - 'low': minutes_arr - 1, - 'close': minutes_arr, - 'volume': np.full(minutes_count, 1), - }, - index=asset_minutes, - ), - } + yield 1, pd.DataFrame( + { + 'open': minutes_arr + 1, + 'high': minutes_arr + 2, + 'low': minutes_arr - 1, + 'close': minutes_arr, + 'volume': np.full(minutes_count, 1), + }, + index=asset_minutes, + ) @classmethod def make_daily_bar_data(cls): diff --git a/tests/test_api_shim.py b/tests/test_api_shim.py index 09957090..5ff1c85c 100644 --- a/tests/test_api_shim.py +++ b/tests/test_api_shim.py @@ -122,14 +122,12 @@ class TestAPIShim(WithDataPortal, WithSimParams, ZiplineTestCase): @classmethod def make_minute_bar_data(cls): - return { - sid: create_minute_df_for_asset( + for sid in cls.sids: + yield sid, create_minute_df_for_asset( cls.env, cls.SIM_PARAMS_START, cls.SIM_PARAMS_END, ) - for sid in cls.sids - } @classmethod def make_daily_bar_data(cls): diff --git a/tests/test_bar_data.py b/tests/test_bar_data.py index d1dc2c2d..761530ff 100644 --- a/tests/test_bar_data.py +++ b/tests/test_bar_data.py @@ -15,7 +15,6 @@ from nose_parameterized import parameterized import numpy as np import pandas as pd -from toolz import merge from zipline._protocol import handle_non_market_minutes from zipline.protocol import BarData @@ -109,32 +108,26 @@ class TestMinuteBarData(WithBarDataChecks, # asset2 has trades every 10 minutes # split_asset trades every minute # illiquid_split_asset trades every 10 minutes - return merge( - { - sid: create_minute_df_for_asset( - cls.env, - cls.bcolz_minute_bar_days[0], - cls.bcolz_minute_bar_days[-1], - ) - for sid in (1, cls.SPLIT_ASSET_SID) - }, - { - sid: create_minute_df_for_asset( - cls.env, - cls.bcolz_minute_bar_days[0], - cls.bcolz_minute_bar_days[-1], - 10, - ) - for sid in (2, cls.ILLIQUID_SPLIT_ASSET_SID) - }, - { - cls.HILARIOUSLY_ILLIQUID_ASSET_SID: create_minute_df_for_asset( - cls.env, - cls.bcolz_minute_bar_days[0], - cls.bcolz_minute_bar_days[-1], - 50, - ) - }, + for sid in (1, cls.SPLIT_ASSET_SID): + yield sid, create_minute_df_for_asset( + cls.env, + cls.bcolz_minute_bar_days[0], + cls.bcolz_minute_bar_days[-1], + ) + + for sid in (2, cls.ILLIQUID_SPLIT_ASSET_SID): + yield sid, create_minute_df_for_asset( + cls.env, + cls.bcolz_minute_bar_days[0], + cls.bcolz_minute_bar_days[-1], + 10, + ) + + yield cls.HILARIOUSLY_ILLIQUID_ASSET_SID, create_minute_df_for_asset( + cls.env, + cls.bcolz_minute_bar_days[0], + cls.bcolz_minute_bar_days[-1], + 50, ) @classmethod diff --git a/tests/test_cli.py b/tests/test_cli.py deleted file mode 100644 index d7c187b2..00000000 --- a/tests/test_cli.py +++ /dev/null @@ -1,70 +0,0 @@ -# -# Copyright 2014 Quantopian, Inc. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -import os -from unittest import TestCase -from six import iteritems - -from zipline.utils import parse_args -from zipline.utils import cli - - -class TestParseArgs(TestCase): - def test_defaults(self): - args = parse_args([]) - for k, v in iteritems(cli.DEFAULTS): - self.assertEqual(v, args[k]) - - def write_conf_file(self): - conf_str = """ -[Defaults] -algofile=test.py -symbols=test_symbols -start=1990-1-1 - """ - - with open('test.conf', 'w') as fd: - fd.write(conf_str) - - def test_conf_file(self): - self.write_conf_file() - try: - args = parse_args(['-c', 'test.conf']) - - self.assertEqual(args['algofile'], 'test.py') - self.assertEqual(args['symbols'], 'test_symbols') - self.assertEqual(args['start'], '1990-1-1') - self.assertEqual(args['data_frequency'], - cli.DEFAULTS['data_frequency']) - finally: - os.remove('test.conf') - - def test_overwrite(self): - self.write_conf_file() - - try: - args = parse_args(['-c', 'test.conf', '--start', '1992-1-1', - '--algofile', 'test2.py']) - - # Overwritten values - self.assertEqual(args['algofile'], 'test2.py') - self.assertEqual(args['start'], '1992-1-1') - # Non-overwritten values - self.assertEqual(args['symbols'], 'test_symbols') - # Default values - self.assertEqual(args['data_frequency'], - cli.DEFAULTS['data_frequency']) - finally: - os.remove('test.conf') diff --git a/tests/test_examples.py b/tests/test_examples.py index 14080655..e074c9e4 100644 --- a/tests/test_examples.py +++ b/tests/test_examples.py @@ -15,15 +15,17 @@ # This code is based on a unittest written by John Salvatier: # https://github.com/pymc-devs/pymc/blob/pymc3/tests/test_examples.py +import tarfile -import glob import matplotlib from nose_parameterized import parameterized -import os -import runpy -from unittest import TestCase +import pandas as pd -from zipline.utils import parse_args, run_pipeline +from zipline import examples, run_algorithm +from zipline.testing import test_resource_path +from zipline.testing.fixtures import WithTmpDir, ZiplineTestCase +from zipline.testing.predicates import assert_equal +from zipline.utils.cache import dataframe_cache # Otherwise the next line sometimes complains about being run too late. _multiprocess_can_split_ = False @@ -31,22 +33,80 @@ _multiprocess_can_split_ = False matplotlib.use('Agg') -def example_dir(): - import zipline - d = os.path.dirname(zipline.__file__) - return os.path.join(os.path.abspath(d), 'examples') +class ExamplesTests(WithTmpDir, ZiplineTestCase): + # some columns contain values with unique ids that will not be the same + cols_to_check = [ + 'algo_volatility', + 'algorithm_period_return', + 'alpha', + 'benchmark_period_return', + 'benchmark_volatility', + 'beta', + 'capital_used', + 'ending_cash', + 'ending_exposure', + 'ending_value', + 'excess_return', + 'gross_leverage', + 'long_exposure', + 'long_value', + 'longs_count', + 'max_drawdown', + 'max_leverage', + 'net_leverage', + 'period_close', + 'period_label', + 'period_open', + 'pnl', + 'portfolio_value', + 'positions', + 'returns', + 'short_exposure', + 'short_value', + 'shorts_count', + 'sortino', + 'starting_cash', + 'starting_exposure', + 'starting_value', + 'trading_days', + 'treasury_period_return', + ] + @classmethod + def init_class_fixtures(cls): + super(ExamplesTests, cls).init_class_fixtures() -class ExamplesTests(TestCase): - # Test algorithms as if they are executed directly from the command line. - @parameterized.expand(((os.path.basename(f).replace('.', '_'), f) for f in - glob.glob(os.path.join(example_dir(), '*.py')))) - def test_example(self, name, example): - runpy.run_path(example, run_name='__main__') + with tarfile.open(test_resource_path('example_data.tar.gz')) as tar: + tar.extractall(cls.tmpdir.path) - # Test algorithm as if scripts/run_algo.py is being used. - def test_example_run_pipeline(self): - example = os.path.join(example_dir(), 'buyapple.py') - confs = ['-f', example, '--start', '2011-1-1', '--end', '2012-1-1'] - parsed_args = parse_args(confs) - run_pipeline(**parsed_args) + cls.expected_perf = dataframe_cache( + cls.tmpdir.getpath( + 'example_data/expected_perf/%s' % + pd.__version__.replace('.', '-'), + ), + serialization='pickle', + ) + + @parameterized.expand(e for e in dir(examples) if not e.startswith('_')) + def test_example(self, example): + mod = getattr(examples, example) + actual_perf = run_algorithm( + handle_data=mod.handle_data, + initialize=mod.initialize, + before_trading_start=getattr(mod, 'before_trading_start', None), + analyze=getattr(mod, 'analyze', None), + bundle='test', + environ={ + 'ZIPLINE_ROOT': self.tmpdir.getpath('example_data/root'), + }, + capital_base=1e7, + **mod._test_args() + ) + assert_equal( + actual_perf[self.cols_to_check], + self.expected_perf[example][self.cols_to_check], + # There is a difference in the datetime columns in pandas + # 0.16 and 0.17 because in 16 they are object and in 17 they are + # datetime[ns, UTC]. We will just ignore the dtypes for now. + check_dtype=False, + ) diff --git a/tests/test_finance.py b/tests/test_finance.py index 0f20f015..fc32934c 100644 --- a/tests/test_finance.py +++ b/tests/test_finance.py @@ -23,6 +23,7 @@ from nose.tools import timed import numpy as np import pandas as pd import pytz +from six import iteritems from six.moves import range from testfixtures import TempDirectory @@ -219,7 +220,7 @@ class FinanceTestCase(WithLogger, env, env.days_in_range(minutes[0], minutes[-1]), tempdir.path, - assets + iteritems(assets), ) equity_minute_reader = BcolzMinuteBarReader(tempdir.path) diff --git a/tests/test_history.py b/tests/test_history.py index 0aec4913..ae4c1c61 100644 --- a/tests/test_history.py +++ b/tests/test_history.py @@ -2,12 +2,12 @@ from textwrap import dedent from numbers import Real -import pandas as pd +from nose_parameterized import parameterized import numpy as np from numpy import nan from numpy.testing import assert_almost_equal - -from nose_parameterized import parameterized +import pandas as pd +from six import iteritems from zipline import TradingAlgorithm from zipline._protocol import handle_non_market_minutes @@ -473,7 +473,7 @@ class MinuteEquityHistoryTestCase(WithHistory, ZiplineTestCase): start_val=2, interval=10, ) - return data + return iteritems(data) def test_history_in_initialize(self): algo_text = dedent( @@ -986,24 +986,22 @@ class DailyEquityHistoryTestCase(WithHistory, ZiplineTestCase): def make_minute_bar_data(cls): asset1 = cls.asset_finder.retrieve_asset(1) asset2 = cls.asset_finder.retrieve_asset(2) - return { - asset1.sid: create_minute_df_for_asset( - cls.env, - asset1.start_date, - asset1.end_date, - start_val=2, - ), - asset2.sid: create_minute_df_for_asset( - cls.env, - asset2.start_date, - cls.env.previous_trading_day(asset2.end_date), - start_val=2, - minute_blacklist=[ - pd.Timestamp('2015-01-08 14:31', tz='UTC'), - pd.Timestamp('2015-01-08 21:00', tz='UTC'), - ], - ), - } + yield asset1.sid, create_minute_df_for_asset( + cls.env, + asset1.start_date, + asset1.end_date, + start_val=2, + ) + yield asset2.sid, create_minute_df_for_asset( + cls.env, + asset2.start_date, + cls.env.previous_trading_day(asset2.end_date), + start_val=2, + minute_blacklist=[ + pd.Timestamp('2015-01-08 14:31', tz='UTC'), + pd.Timestamp('2015-01-08 21:00', tz='UTC'), + ], + ) @classmethod def create_df_for_asset(cls, start_day, end_day, interval=1, @@ -1548,32 +1546,30 @@ class MinuteToDailyAggregationTestCase(WithBcolzMinuteBarReader, @classmethod def make_minute_bar_data(cls): - return { - # sid data is created so that at least one high is lower than a - # previous high, and the inverse for low - 1: pd.DataFrame( - { - 'open': [nan, 103.50, 102.50, 104.50, 101.50, nan], - 'high': [nan, 103.90, 102.90, 104.90, 101.90, nan], - 'low': [nan, 103.10, 102.10, 104.10, 101.10, nan], - 'close': [nan, 103.30, 102.30, 104.30, 101.30, nan], - 'volume': [0, 1003, 1002, 1004, 1001, 0] - }, - index=cls.minutes, - ), - # sid 2 is included to provide data on different bars than sid 1, - # as will as illiquidty mid-day - 2: pd.DataFrame( - { - 'open': [201.50, nan, 204.50, nan, 200.50, 202.50], - 'high': [201.90, nan, 204.90, nan, 200.90, 202.90], - 'low': [201.10, nan, 204.10, nan, 200.10, 202.10], - 'close': [201.30, nan, 203.50, nan, 200.30, 202.30], - 'volume': [2001, 0, 2004, 0, 2000, 2002], - }, - index=cls.minutes, - ), - } + # sid data is created so that at least one high is lower than a + # previous high, and the inverse for low + yield 1, pd.DataFrame( + { + 'open': [nan, 103.50, 102.50, 104.50, 101.50, nan], + 'high': [nan, 103.90, 102.90, 104.90, 101.90, nan], + 'low': [nan, 103.10, 102.10, 104.10, 101.10, nan], + 'close': [nan, 103.30, 102.30, 104.30, 101.30, nan], + 'volume': [0, 1003, 1002, 1004, 1001, 0] + }, + index=cls.minutes, + ) + # sid 2 is included to provide data on different bars than sid 1, + # as will as illiquidty mid-day + yield 2, pd.DataFrame( + { + 'open': [201.50, nan, 204.50, nan, 200.50, 202.50], + 'high': [201.90, nan, 204.90, nan, 200.90, 202.90], + 'low': [201.10, nan, 204.10, nan, 200.10, 202.10], + 'close': [201.30, nan, 203.50, nan, 200.30, 202.30], + 'volume': [2001, 0, 2004, 0, 2000, 2002], + }, + index=cls.minutes, + ) expected_values = { 1: pd.DataFrame( diff --git a/zipline/__init__.py b/zipline/__init__.py index 9166faa7..5dfe287b 100644 --- a/zipline/__init__.py +++ b/zipline/__init__.py @@ -20,6 +20,7 @@ from . import data from . import finance from . import gens from . import utils +from .utils.run_algo import run_algorithm from ._version import get_versions # These need to happen after the other imports. from . algorithm import TradingAlgorithm @@ -28,19 +29,18 @@ from . import api __version__ = get_versions()['version'] del get_versions -try: - ip = get_ipython() # flake8: noqa -except NameError: - pass -else: - ip.register_magic_function(utils.parse_cell_magic, "line_cell", "zipline") - del ip + +def load_ipython_extension(ipython): + from .__main__ import zipline_magic + ipython.register_magic_function(zipline_magic, 'line_cell', 'zipline') + __all__ = [ + 'TradingAlgorithm', + 'api', 'data', 'finance', 'gens', + 'run_algorithm', 'utils', - 'api', - 'TradingAlgorithm', ] diff --git a/zipline/__main__.py b/zipline/__main__.py new file mode 100644 index 00000000..df660d57 --- /dev/null +++ b/zipline/__main__.py @@ -0,0 +1,332 @@ +import datetime +import os +from functools import wraps + +import click +import logbook +import pandas as pd + +from zipline.data import bundles +from zipline.utils.cli import Date, Timestamp +from zipline.utils.run_algo import _run, load_extensions + +try: + __IPYTHON__ +except NameError: + __IPYTHON__ = False + + +@click.group() +@click.option( + '-e', + '--extension', + multiple=True, + help='File or module path to a zipline extension to load.', +) +@click.option( + '--strict-extensions/--non-strict-extensions', + is_flag=True, + help='If --strict-extensions is passed then zipline will not run if it' + ' cannot load all of the specified extensions. If this is not passed or' + ' --non-strict-extensions is passed then the failure will be logged but' + ' execution will continue.', +) +@click.option( + '--default-extension/--no-default-extension', + is_flag=True, + default=True, + help="Don't load the default zipline extension.py file in $ZIPLINE_HOME.", +) +def cli(extension, strict_extensions, default_extension): + """Top level zipline entry point. + """ + # install a logbook handler before performing any other operations + logbook.StderrHandler().push_application() + load_extensions( + default_extension, + extension, + strict_extensions, + os.environ, + ) + + +def extract_option_object(option): + """Convert a click.option call into a click.Option object. + + Parameters + ---------- + option : decorator + A click.option decorator. + + Returns + ------- + option_object : click.Option + The option object that this decorator will create. + """ + @option + def opt(): + pass + + return opt.__click_params__[0] + + +def ipython_only(option): + """Mark that an option should only be exposed in IPython. + + Parameters + ---------- + option : decorator + A click.option decorator. + + Returns + ------- + ipython_only_dec : decorator + A decorator that correctly applies the argument even when not + using IPython mode. + """ + if __IPYTHON__: + return option + + argname = extract_option_object(option).name + + def d(f): + @wraps(f) + def _(*args, **kwargs): + kwargs[argname] = None + return f(*args, **kwargs) + return _ + return d + + +@cli.command() +@click.option( + '-f', + '--algofile', + default=None, + type=click.File('r'), + help='The file that contains the algorithm to run.', +) +@click.option( + '-t', + '--algotext', + help='The algorithm script to run.', +) +@click.option( + '-D', + '--define', + multiple=True, + help="Define a name to be bound in the namespace before executing" + " the algotext. For example '-Dname=value'. The value may be any python" + " expression. These are evaluated in order so they may refer to previously" + " defined names.", +) +@click.option( + '--data-frequency', + type=click.Choice({'daily', 'minute'}), + default='daily', + show_default=True, + help='The data frequency of the simulation.', +) +@click.option( + '--capital-base', + type=float, + default=10e6, + show_default=True, + help='The starting capital for the simulation.', +) +@click.option( + '-b', + '--bundle', + default='quandl', + metavar='BUNDLE-NAME', + show_default=True, + help='The data bundle to use for the simulation.', +) +@click.option( + '--bundle-timestamp', + type=Timestamp(), + default=pd.Timestamp.utcnow(), + show_default=False, + help='The date to lookup data on or before.\n' + '[default: ]' +) +@click.option( + '-s', + '--start', + type=Date(tz='utc', as_timestamp=True), + help='The start date of the simulation.', +) +@click.option( + '-e', + '--end', + type=Date(tz='utc', as_timestamp=True), + help='The end date of the simulation.', +) +@click.option( + '-o', + '--output', + default='-', + metavar='FILENAME', + show_default=True, + help="The location to write the perf data. If this is '-' the perf will" + " be written to stdout.", +) +@click.option( + '--print-algo/--no-print-algo', + is_flag=True, + default=False, + help='Print the algorithm to stdout.', +) +@ipython_only(click.option( + '--local-namespace/--no-local-namespace', + is_flag=True, + default=None, + help='Should the algorithm methods be resolved in the local namespace.' +)) +@click.pass_context +def run(ctx, + algofile, + algotext, + define, + data_frequency, + capital_base, + bundle, + bundle_timestamp, + start, + end, + output, + print_algo, + local_namespace): + """Run a backtest for the given algorithm. + """ + # check that the start and end dates are passed correctly + if start is None and end is None: + # check both at the same time to avoid the case where a user + # does not pass either of these and then passes the first only + # to be told they need to pass the second argument also + ctx.fail( + "must specify dates with '-s' / '--start' and '-e' / '--end'", + ) + if start is None: + ctx.fail("must specify a start date with '-s' / '--start'") + if end is None: + ctx.fail("must specify an end date with '-s' / '--end'") + + if (algotext is not None) == (algofile is not None): + ctx.fail( + "must specify exactly one of '-f' / '--algofile' or" + " '-t' / '--algotext'", + ) + + perf = _run( + initialize=None, + handle_data=None, + before_trading_start=None, + analyze=None, + algofile=algofile, + algotext=algotext, + defines=define, + data_frequency=data_frequency, + capital_base=capital_base, + data=None, + bundle=bundle, + bundle_timestamp=bundle_timestamp, + start=start, + end=end, + output=output, + print_algo=print_algo, + local_namespace=local_namespace, + environ=os.environ, + ) + + if output == '-': + click.echo(str(perf)) + elif output != os.devnull: # make the zipline magic not write any data + perf.to_pickle(output) + + return perf + + +def zipline_magic(line, cell=None): + """The zipline IPython cell magic. + """ + try: + return run.main( + # put our overrides at the start of the parameter list so that + # users may pass values with higher precedence + [ + '--algotext', cell, + '--output', os.devnull, # don't write the results by default + ] + ([ + # these options are set when running in line magic mode + # set a non None algo text to use the ipython user_ns + '--algotext', '', + '--local-namespace', + ] if cell is None else []) + line.split(), + '%s%%zipline' % ((cell or '') and '%'), + # don't use system exit and propogate errors to the caller + standalone_mode=False, + ) + except SystemExit as e: + # https://github.com/mitsuhiko/click/pull/533 + # even in standalone_mode=False `--help` really wants to kill us ;_; + if e.code: + raise ValueError('main returned non-zero status code: %d' % e.code) + + +@cli.command() +@click.argument('BUNDLE-NAME') +@click.option( + '--show-progress/--no-show-progress', + is_flag=True, + default=True, + help='Print progress information to the terminal.' +) +def ingest(bundle_name, show_progress): + """Ingest the data for the given bundle. + """ + bundles.ingest( + bundle_name, + os.environ, + datetime.date.today(), + show_progress, + ) + + +@cli.command() +@click.argument('BUNDLE-NAME') +@click.option( + '-b', + '--before', + type=Timestamp(), + help='Clear all data before TIMESTAMP.' + ' This may not be passed with -k / --keep-last', +) +@click.option( + '-a', + '--after', + type=Timestamp(), + help='Clear all data after TIMESTAMP' + ' This may not be passed with -k / --keep-last', +) +@click.option( + '-k', + '--keep-last', + type=int, + metavar='N', + help='Clear all but the last N downloads.' + ' This may not be passed with -b / --before or -a / --after', +) +def clean(bundle_name, before, after, keep_last): + """Clean up data downloaded with the ingest command. + """ + bundles.clean( + bundle_name, + before, + after, + keep_last, + ) + + +if __name__ == '__main__': + cli() diff --git a/zipline/algorithm.py b/zipline/algorithm.py index ed0f124d..496a6a56 100644 --- a/zipline/algorithm.py +++ b/zipline/algorithm.py @@ -116,7 +116,7 @@ from zipline.gens.sim_engine import ( from zipline.sources.benchmark_source import BenchmarkSource from zipline.zipline_warnings import ZiplineDeprecationWarning -DEFAULT_CAPITAL_BASE = float("1.0e5") +DEFAULT_CAPITAL_BASE = 1e5 log = logbook.Logger("ZiplineLog") diff --git a/zipline/assets/asset_writer.py b/zipline/assets/asset_writer.py index b1eda14c..dfc8566b 100644 --- a/zipline/assets/asset_writer.py +++ b/zipline/assets/asset_writer.py @@ -257,6 +257,95 @@ class AssetDBWriter(object): exchanges=None, root_symbols=None, chunk_size=DEFAULT_CHUNK_SIZE): + """Write asset metadata to a sqlite database. + + Parameters + ---------- + equities : pd.DataFrame, optional + The equity metadata. The columns for this dataframe are: + + symbol : str + The ticker symbol for this equity. + fuzzy_symbol : str, optional + The fuzzy symbol for this equity. This is the symbol + without any delimiting characters like '.' or '_'. + asset_name : str + The full name for this asset. + start_date : datetime + The date when this asset was created. + end_date : datetime, optional + The last date we have trade data for this asset. + first_traded : datetime, optional + The first date we have trade data for this asset. + auto_close_date : datetime, optional + The date on which to close any positions in this asset. + exchange : str, optional + The exchange where this asset is traded. + + The index of this dataframe should contain the sids. + futures : pd.Dataframe, optional + The future contract metadata. The columns for this dataframe are: + + symbol : str + The ticker symbol for this futures contract. + root_symbol : str + The root symbol, or the symbol with the expiration stripped + out. + asset_name : str + The full name for this asset. + start_date : datetime, optional + The date when this asset was created. + end_date : datetime, optional + The last date we have trade data for this asset. + first_traded : datetime, optional + The first date we have trade data for this asset. + exchange : str, optional + The exchange where this asset is traded. + notice_date : datetime + The date when the owner of the contract may be forced + to take physical delivery of the contract's asset. + expiration_date : datetime + The date when the contract expires. + auto_close_date : datetime + The date when the broker will automatically close any + positions in this contract. + tick_size : float + The minimum price movement of the contract. + multiplier: float + The amount of the underlying asset represented by this + contract. + exchanges : pd.Dataframe, optional + The exchanges where assets can be traded. The columns of this + dataframe are: + + exchange : str + The name of the exchange. + timezone : str + The timezone of the exchange. + root_symbols : pd.Dataframe, optional + The root symbols for the futures contracts. The columns for this + dataframe are: + + root_symbol : str + The root symbol name. + root_symbol_id : int + The unique id for this root symbol. + sector : string, optional + The sector of this root symbol. + description : string, optional + A short description of this root symbol. + exchange : str + The exchange where this root symbol is traded. + chunk_size : int, optional + The amount of rows to write to the SQLite table at once. + This defaults to the default number of bind params in sqlite. + If you have compiled sqlite3 with more bind or less params you may + want to pass that value here. + + See Also + -------- + zipline.assets.asset_finder + """ with self.engine.begin() as txn: # Create SQL tables if they do not exist. diff --git a/zipline/assets/assets.py b/zipline/assets/assets.py index ce498336..b671a754 100644 --- a/zipline/assets/assets.py +++ b/zipline/assets/assets.py @@ -94,7 +94,7 @@ class AssetFinder(object): See Also -------- - :class:`zipline.assets.asset_writer.AssetDBWriter` + :class:`zipline.assets.AssetDBWriter` """ # Token used as a substitute for pickling objects that contain a # reference to an AssetFinder. diff --git a/zipline/data/__init__.py b/zipline/data/__init__.py index 478d6261..f3dd9b36 100644 --- a/zipline/data/__init__.py +++ b/zipline/data/__init__.py @@ -1,8 +1,16 @@ from . import loader from .loader import ( - load_from_yahoo, load_bars_from_yahoo, load_prices_from_csv, - load_prices_from_csv_folder + load_from_yahoo, + load_bars_from_yahoo, + load_prices_from_csv, + load_prices_from_csv_folder, ) -__all__ = ['loader', 'load_from_yahoo', 'load_bars_from_yahoo', - 'load_prices_from_csv', 'load_prices_from_csv_folder'] + +__all__ = [ + 'load_bars_from_yahoo', + 'load_from_yahoo', + 'load_prices_from_csv', + 'load_prices_from_csv_folder', + 'loader', +] diff --git a/zipline/data/bundles/__init__.py b/zipline/data/bundles/__init__.py new file mode 100644 index 00000000..baca0f18 --- /dev/null +++ b/zipline/data/bundles/__init__.py @@ -0,0 +1,20 @@ +from .core import ( + bundles, + clean, + ingest, + load, + register, + unregister, +) +from .yahoo import yahoo_equities + + +__all__ = [ + 'bundles', + 'clean', + 'ingest', + 'load', + 'register', + 'unregister', + 'yahoo_equities', +] diff --git a/zipline/data/bundles/core.py b/zipline/data/bundles/core.py new file mode 100644 index 00000000..1efe024b --- /dev/null +++ b/zipline/data/bundles/core.py @@ -0,0 +1,469 @@ +from collections import namedtuple +import errno +import os +import shutil +import warnings + +import click +import pandas as pd +from toolz import curry, complement, compose + +from ..us_equity_pricing import ( + BcolzDailyBarReader, + BcolzDailyBarWriter, + SQLiteAdjustmentReader, + SQLiteAdjustmentWriter, +) +from ..minute_bars import ( + BcolzMinuteBarReader, + BcolzMinuteBarWriter, +) +from zipline.assets import AssetDBWriter, AssetFinder, ASSET_DB_VERSION +from zipline.utils.cache import ( + dataframe_cache, + working_file, + working_dir, +) +from zipline.utils.compat import mappingproxy +from zipline.utils.input_validation import ensure_timestamp, optionally +import zipline.utils.paths as pth +from zipline.utils.preprocess import preprocess +from zipline.utils.tradingcalendar import trading_days, open_and_closes + + +def asset_db_path(bundle_name, timestr, environ=None): + return pth.data_path( + [bundle_name, timestr, 'assets-%d.sqlite' % ASSET_DB_VERSION], + environ=environ, + ) + + +def minute_equity_path(bundle_name, timestr, environ=None): + return pth.data_path( + [bundle_name, timestr, 'minute_equities.bcolz'], + environ=environ, + ) + + +def daily_equity_path(bundle_name, timestr, environ=None): + return pth.data_path( + [bundle_name, timestr, 'daily_equities.bcolz'], + environ=environ, + ) + + +def adjustment_db_path(bundle_name, timestr, environ=None): + return pth.data_path( + [bundle_name, timestr, 'adjustments.sqlite'], + environ=environ, + ) + + +def cache_path(bundle_name, timestr, environ=None): + return pth.data_path( + [bundle_name, timestr, '.cache'], + environ=environ, + ) + + +_BundlePayload = namedtuple( + '_BundlePayload', + 'calendar opens closes minutes_per_day ingest', +) + + +class UnknownBundle(click.ClickException, LookupError): + """Raised if no bundle with the given name was registered. + """ + exit_code = 1 + + def __init__(self, name): + super(UnknownBundle, self).__init__( + 'No bundle registered with the name %r' % name, + ) + self.name = name + + def __str__(self): + return self.message + + +def _make_bundle_core(): + """Create a family of data bundle functions that read from the same + bundle mapping. + + Returns + ------- + bundles : mappingproxy + The mapping of bundles to bundle payloads. + register : callable + The function which registers new bundles in the ``bundles`` mapping. + unregister : callable + The function which deregisters bundles from the ``bundles`` mapping. + ingest_bundle : callable + The function which downloads and write data for a given data bundle. + """ + _bundles = {} # the registered bundles + # Expose _bundles through a proxy so that users cannot mutate this + # accidentally. Users may go through `register` to update this which will + # warn when trampling another bundle. + bundles = mappingproxy(_bundles) + + @curry + def register(name, + f, + calendar=trading_days, + opens=open_and_closes['market_open'], + closes=open_and_closes['market_close'], + minutes_per_day=390): + """Register a data bundle ingest function. + + Parameters + ---------- + name : str + The name of the bundle. + f : callable + The ingest function. This function will be passed: + + environ : mapping + The environment this is being run with. + asset_db_writer : AssetDBWriter + The asset db writer to write into. + minute_bar_writer : BcolzMinuteBarWriter + The minute bar writer to write into. + daily_bar_writer : BcolzDailyBarWriter + The daily bar writer to write into. + adjustment_writer : SQLiteAdjustmentWriter + The adjustment db writer to write into. + calendar : pd.DatetimeIndex + The trading calendar to ingest for. + cache : DataFrameCache + A mapping object to temporarily store dataframes. + This should be used to cache intermediates in case the load + fails. This will be automatically cleaned up after a + successful load. + show_progress : bool + Show the progress for the current load where possible. + calendar : pd.DatetimeIndex, optional + The exchange calendar to align the data to. This defaults to the + NYSE calendar. + market_open : pd.DatetimeIndex, optional + The minute when the market opens each day. This defaults to the + NYSE calendar. + market_close : pd.DatetimeIndex, optional + The minute when the market closes each day. This defaults to the + NYSE calendar. + minutes_per_day : int, optional + The number of minutes in each normal trading day. + + Notes + ----- + This function my be used as a decorator, for example: + + .. code-block:: python + + @register('quandl') + def quandl_ingest_function(...): + ... + + See Also + -------- + zipline.data.bundles.bundles + """ + if name in bundles: + warnings.warn( + 'Overwriting bundle with name %r' % name, + stacklevel=3, + ) + _bundles[name] = _BundlePayload( + calendar, + opens, + closes, + minutes_per_day, + f, + ) + return f + + def unregister(name): + """Unregister a bundle. + + Parameters + ---------- + name : str + The name of the bundle to unregister. + + Raises + ------ + UnknownBundle + Raised when no bundle has been registered with the given name. + + See Also + -------- + zipline.data.bundles.bundles + """ + try: + del _bundles[name] + except KeyError: + raise UnknownBundle(name) + + def ingest(name, + environ=os.environ, + timestamp=None, + show_progress=True): + """Ingest data for a given bundle. + + Parameters + ---------- + name : str + The name of the bundle. + environ : mapping, optional + The environment variables. By default this is os.environ. + timestamp : datetime, optional + The timestamp to use for the load. + By default this is the current time. + show_progress : bool, optional + Tell the ingest function to display the progress where possible. + """ + try: + bundle = bundles[name] + except KeyError: + raise UnknownBundle(name) + + if timestamp is None: + timestamp = pd.Timestamp.utcnow() + timestamp = timestamp.tz_convert('utc').tz_localize(None) + timestr = str(timestamp.value) + cachepath = cache_path(name, timestr, environ=environ) + pth.ensure_directory(cachepath) + + with dataframe_cache(cachepath, clean_on_failure=False) as cache, \ + working_dir( + daily_equity_path(name, timestr, environ=environ), + ) as daily_bars_dir, \ + working_dir( + minute_equity_path(name, timestr, environ=environ), + ) as minute_bars_dir, \ + working_file( + asset_db_path(name, timestr, environ=environ), + ) as asset_db_file, \ + working_file( + adjustment_db_path(name, timestr, environ=environ), + ) as adjustment_db_file: + # we use `cleanup_on_failure=False` so that we don't purge the + # cache directory if the load fails in the middle + daily_bar_writer = BcolzDailyBarWriter( + daily_bars_dir.name, + bundle.calendar, + ) + # Do an empty write to ensure that the daily ctables exist + # when we create the SQLiteAdjustmentWriter below. The + # SQLiteAdjustmentWriter needs to open the daily ctables so that + # it can compute the adjustment ratios for the dividends. + daily_bar_writer.write(()) + bundle.ingest( + environ, + AssetDBWriter(asset_db_file.name), + BcolzMinuteBarWriter( + bundle.calendar[0], + minute_bars_dir.name, + bundle.opens, + bundle.closes, + minutes_per_day=bundle.minutes_per_day, + ), + daily_bar_writer, + SQLiteAdjustmentWriter( + adjustment_db_file.name, + BcolzDailyBarReader(daily_bars_dir.name), + bundle.calendar, + overwrite=True, + ), + bundle.calendar, + cache, + show_progress, + ) + + return bundles, register, unregister, ingest + + +bundles, register, unregister, ingest = _make_bundle_core() + +BundleData = namedtuple( + 'BundleData', + 'asset_finder minute_bar_reader daily_bar_reader adjustment_reader', +) + + +def most_recent_data(bundle_name, timestamp, environ=None): + """Get the path to the most recent data after ``date``for the given bundle. + + Parameters + ---------- + bundle_name : str + The name of the bundle to lookup. + timestamp : datetime + The timestamp to begin searching on or before. + environ : dict, optional + An environment dict to forward to zipline_root. + """ + try: + candidates = os.listdir(pth.data_path([bundle_name], environ=environ)) + return pth.data_path( + [bundle_name, + max( + filter(complement(pth.hidden), candidates), + key=compose(pd.Timestamp, int), + )], + environ=environ, + ) + except ValueError: + raise ValueError( + 'no data for bundle %r on or before %s' % ( + bundle_name, + timestamp, + ), + ) + except OSError as e: + if e.errno != errno.ENOENT: + raise + raise UnknownBundle(bundle_name) + + +def load(name, environ=os.environ, timestamp=None): + """Loads a previously ingested bundle. + + Parameters + ---------- + name : str + The name of the bundle. + environ : mapping, optional + The environment variables. Defaults of os.environ. + timestamp : datetime, optional + The timestamp of the data to lookup. + Defaults to the current time. + + Returns + ------- + bundle_data : BundleData + The raw data readers for this bundle. + """ + if timestamp is None: + timestamp = pd.Timestamp.utcnow() + timestr = most_recent_data(name, timestamp, environ=environ) + return BundleData( + asset_finder=AssetFinder( + asset_db_path(name, timestr, environ=environ), + ), + minute_bar_reader=BcolzMinuteBarReader( + minute_equity_path(name, timestr, environ=environ), + ), + daily_bar_reader=BcolzDailyBarReader( + daily_equity_path(name, timestr, environ=environ), + ), + adjustment_reader=SQLiteAdjustmentReader( + adjustment_db_path(name, timestr, environ=environ), + ), + ) + + +class BadClean(click.ClickException, ValueError): + """Exception indicating that an invalid argument set was passed to + ``clean``. + + Parameters + ---------- + before, after, keep_last : any + The bad arguments to ``clean``. + + See Also + -------- + clean + """ + def __init__(self, before, after, keep_last): + super(BadClean, self).__init__( + 'Cannot pass a combination of `before` and `after` with' + '`keep_last`. Got: before=%r, after=%r, keep_n=%r\n' % ( + before, + after, + keep_last, + ), + ) + + def __str__(self): + return self.message + + +@preprocess( + before=optionally(ensure_timestamp), + after=optionally(ensure_timestamp), +) +def clean(name, before=None, after=None, keep_last=None, environ=os.environ): + """Clean up data that was created with ``ingest`` or + ``$ python -m zipline ingest`` + + Parameters + ---------- + name : str + The name of the bundle to remove data for. + before : datetime, optional + Remove data ingested before this date. + This argument is mutually exclusive with: keep_last + after : datetime, optional + Remove data ingested after this date. + This argument is mutually exclusive with: keep_last + keep_last : int, optional + Remove all but the last ``keep_last`` ingestions. + This argument is mutually exclusive with: + before + after + + Returns + ------- + cleaned : set[str] + The names of the runs that were removed. + + Raises + ------ + BadClean + Raised when ``before`` and or ``after`` are passed with ``keep_last``. + This is a subclass of ``ValueError``. + """ + try: + all_runs = sorted( + pd.Timestamp(f) + for f in os.listdir(pth.data_path([name], environ=environ)) + if not pth.hidden(f) + ) + except OSError as e: + if e.errno != errno.ENOENT: + raise + raise UnknownBundle(name) + + if (before is not None or after is not None) and keep_last is not None: + raise BadClean(before, after, keep_last) + + if keep_last is None: + def in_last_n(dt): + return False + else: + last_n_dts = set(all_runs[:keep_last]) + + def in_last_n(dt): + return dt in last_n_dts + + def should_clean(name): + dt = pd.Timestamp(name) + + return ( + ( + (before is not None and dt < before) or + (after is not None and dt > after) + ) and + not in_last_n(dt) + ) + + cleaned = set() + for run in all_runs: + if should_clean(run): + shutil.rmdir(run) + cleaned.add(run) + + return cleaned diff --git a/zipline/data/bundles/quandl.py b/zipline/data/bundles/quandl.py new file mode 100644 index 00000000..a0ca1906 --- /dev/null +++ b/zipline/data/bundles/quandl.py @@ -0,0 +1,298 @@ +""" +Module for building a complete daily dataset from Quandl's WIKI dataset. +""" +from itertools import count +from time import time, sleep + +from logbook import Logger +import pandas as pd +from six.moves.urllib.parse import urlencode + +from zipline.utils.cli import maybe_show_progress +from zipline.data import bundles + +log = Logger(__name__) +seconds_per_call = (pd.Timedelta('10 minutes') / 2000).total_seconds() + + +def _fetch_raw_metadata(api_key, cache, retries, environ): + """Generator that yields each page of data from the metadata endpoint + as a dataframe. + """ + for page_number in count(1): + key = 'metadata-page-%d' % page_number + try: + raw = cache[key] + except KeyError: + for _ in range(retries): + try: + raw = pd.read_csv( + format_metadata_url(api_key, page_number), + parse_dates=[ + 'oldest_available_date', + 'newest_available_date', + ], + usecols=[ + 'dataset_code', + 'name', + 'oldest_available_date', + 'newest_available_date', + ], + ) + break + except ValueError: + # when we are past the last page we will get a value + # error because there will be no columns + raw = pd.DataFrame([]) + break + except Exception: + pass + else: + raise ValueError( + 'Failed to download metadata page %d after %d' + ' attempts.' % (page_number, retries), + ) + + cache[key] = raw + + if raw.empty: + # use the empty dataframe to signal completion + break + yield raw + + +def fetch_symbol_metadata_frame(api_key, + cache, + retries=5, + environ=None, + show_progress=False): + """ + Download Quandl symbol metadata. + + Parameters + ---------- + api_key : str + The quandl api key to use. If this is None then no api key will be + sent. + cache : DataFrameCache + The cache to use for persisting the intermediate data. + retries : int, optional + The number of times to retry each request before failing. + environ : mapping[str -> str], optional + The environment to use to find the zipline home. By default this + is ``os.environ``. + show_progress : bool, optional + Show a progress bar for the download of this data. + + Returns + ------- + metadata_frame : pd.DataFrame + A dataframe with the following columns: + symbol: the asset's symbol + name: the full name of the asset + start_date: the first date of data for this asset + end_date: the last date of data for this asset + exchange: the exchange for the asset; this is always 'quandl' + The index of the dataframe will be used for symbol->sid mappings but + otherwise does not have specific meaning. + """ + raw_iter = _fetch_raw_metadata(api_key, cache, retries, environ) + + def item_show_func(_, _it=iter(count())): + 'Downloading page: %d' % next(_it) + + with maybe_show_progress(raw_iter, + show_progress, + item_show_func=item_show_func, + label='Downloading WIKI metadata: ') as blocks: + data = pd.concat(blocks, ignore_index=True).rename(columns={ + 'dataset_code': 'symbol', + 'name': 'asset_name', + 'oldest_available_date': 'start_date', + 'newest_available_date': 'end_date', + }).sort('symbol') + # cut out all the other stuff in the name column + # we need to escape the paren because it is actually splitting on a regex + data.asset_name = data.asset_name.str.split(r' \(', 1).str.get(0) + data['exchange'] = 'quandl' + return data + + +def format_metadata_url(api_key, page_number): + """Build the query RL for the quandl WIKI metadata. + """ + query_params = [ + ('per_page', '100'), + ('sort_by', 'id'), + ('page', str(page_number)), + ('database_code', 'WIKI'), + ] + if api_key is not None: + query_params = [('api_key', api_key)] + query_params + return ( + 'https://www.quandl.com/api/v3/datasets.csv?' + urlencode(query_params) + ) + + +def format_wiki_url(api_key, symbol, start_date, end_date): + """ + Build a query URL for a quandl WIKI dataset. + """ + query_params = [ + ('start_date', start_date.strftime('%Y-%m-%d')), + ('end_date', end_date.strftime('%Y-%m-%d')), + ('order', 'asc'), + ] + if api_key is not None: + query_params = [('api_key', api_key)] + query_params + + return ( + "https://www.quandl.com/api/v3/datasets/WIKI/" + "{symbol}.csv?{query}".format( + symbol=symbol, + query=urlencode(query_params), + ) + ) + + +def fetch_single_equity(api_key, + symbol, + start_date, + end_date, + retries=5): + """ + Download data for a single equity. + """ + for _ in range(retries): + try: + return pd.read_csv( + format_wiki_url(api_key, symbol, start_date, end_date), + parse_dates=['Date'], + index_col='Date', + usecols=[ + 'Open', + 'High', + 'Low', + 'Close', + 'Volume', + 'Date', + 'Ex-Dividend', + 'Split Ratio', + ], + na_values=['NA'], + ).rename(columns={ + 'Open': 'open', + 'High': 'high', + 'Low': 'low', + 'Close': 'close', + 'Volume': 'volume', + 'Date': 'date', + 'Ex-Dividend': 'ex_dividend', + 'Split Ratio': 'split_ratio', + }) + except Exception: + log.exception("Exception raised reading Quandl data. Retrying.") + else: + raise ValueError( + "Failed to download data for %r after %d attempts." % ( + symbol, retries + ) + ) + + +def _update_splits(splits, asset_id, raw_data): + split_ratios = raw_data.split_ratio + df = pd.DataFrame({'ratio': split_ratios[split_ratios != 1]}) + df.index.name = 'effective_date' + df.reset_index(inplace=True) + df['sid'] = asset_id + splits.append(df) + + +def _update_dividends(dividends, asset_id, raw_data): + divs = raw_data.ex_dividend + df = pd.DataFrame({'amount': divs[divs != 0]}) + df.index.name = 'ex_date' + df.reset_index(inplace=True) + df['sid'] = asset_id + # we do not have this data in the WIKI dataset + df['record_date'] = df['declared_date'] = df['pay_date'] = pd.NaT + dividends.append(df) + + +def gen_symbol_data(api_key, + cache, + symbol_map, + calendar, + splits, + dividends, + retries): + start_date = calendar[0] + end_date = calendar[-1] + for asset_id, symbol in symbol_map.iteritems(): + start_time = time() + try: + # see if we have this data cached. + raw_data = cache[symbol] + should_sleep = False + except KeyError: + # we need to fetch the data and then write it to our cache + raw_data = cache[symbol] = fetch_single_equity( + api_key, + symbol, + start_date=start_date, + end_date=end_date, + ) + should_sleep = True + + _update_splits(splits, asset_id, raw_data) + _update_dividends(dividends, asset_id, raw_data) + + raw_data = raw_data.reindex(calendar, copy=False).fillna(0.0) + yield asset_id, raw_data + + if should_sleep: + remaining = seconds_per_call - time() - start_time + if remaining > 0: + sleep(remaining) + + +@bundles.register('quandl') +def quandl_bundle(environ, + asset_db_writer, + minute_bar_writer, # unused + daily_bar_writer, + adjustment_writer, + calendar, + cache, + show_progress): + """Build a zipline data bundle from the Quandl WIKI dataset. + """ + api_key = environ.get('QUANDL_API_KEY') + metadata = fetch_symbol_metadata_frame( + api_key, + cache=cache, + show_progress=show_progress, + ) + symbol_map = metadata.symbol + + # data we will collect in `gen_symbol_data` + splits = [] + dividends = [] + + asset_db_writer.write(metadata) + daily_bar_writer.write( + gen_symbol_data( + api_key, + cache, + symbol_map, + calendar, + splits, + dividends, + environ.get('QUANDL_DOWNLOAD_ATTEMPTS', 5), + ), + ) + adjustment_writer.write( + splits=pd.concat(splits, ignore_index=True), + dividends=pd.concat(dividends, ignore_index=True), + ) diff --git a/zipline/data/bundles/yahoo.py b/zipline/data/bundles/yahoo.py new file mode 100644 index 00000000..12737c40 --- /dev/null +++ b/zipline/data/bundles/yahoo.py @@ -0,0 +1,164 @@ +import os + +import numpy as np +import pandas as pd +from pandas_datareader.data import DataReader +import requests + +from zipline.utils.cli import maybe_show_progress + + +def _cachpath(symbol, type_): + return '-'.join((symbol.replace(os.path.sep, '_'), type_)) + + +def yahoo_equities(symbols, start=None, end=None): + """Create a data bundle ingest function from a set of symbols loaded from + yahoo. + + Parameters + ---------- + symbols : iterable[str] + The ticker symbols to load data for. + start : datetime, optional + The start date to query for. By default this pulls the full history + for the calendar. + end : datetime, optional + The end date to query for. By default this pulls the full history + for the calendar. + + Returns + ------- + ingest : callable + The bundle ingest function for the given set of symbols. + + Examples + -------- + This code should be added to ~/.zipline/extension.py + + .. code-block:: python + + from zipline.data.bundles import yahoo_equities, register + + symbols = ( + 'AAPL', + 'IBM', + 'MSFT', + ) + register('my_bundle', yahoo_equities(symbols)) + + Notes + ----- + The sids for each symbol will be the index into the symbols sequence. + """ + # strict this in memory so that we can reiterate over it + symbols = tuple(symbols) + + def ingest(environ, + asset_db_writer, + minute_bar_writer, # unused + daily_bar_writer, + adjustment_writer, + calendar, + cache, + show_progress, + # pass these as defaults to make them 'nonlocal' in py2 + start=start, + end=end): + if start is None: + start = calendar[0] + if end is None: + end = None + + metadata = pd.DataFrame(np.empty(len(symbols), dtype=[ + ('start_date', 'datetime64[ns]'), + ('end_date', 'datetime64[ns]'), + ('symbol', 'object'), + ])) + + def _pricing_iter(): + sid = 0 + with maybe_show_progress( + symbols, + show_progress, + label='Downloading Yahoo pricing data: ') as it, \ + requests.Session() as session: + for symbol in it: + path = _cachpath(symbol, 'ohlcv') + try: + df = cache[path] + except KeyError: + df = cache[path] = DataReader( + symbol, + 'yahoo', + start, + end, + session=session, + ).sort_index() + + # the start date is the date of the first trade and + # the end date is the date of the last trade + metadata.iloc[sid] = df.index[0], df.index[-1], symbol + df.rename( + columns={ + 'Open': 'open', + 'High': 'high', + 'Low': 'low', + 'Close': 'close', + 'Volume': 'volume', + }, + inplace=True, + ) + yield sid, df + sid += 1 + + daily_bar_writer.write(_pricing_iter(), show_progress=True) + + symbol_map = pd.Series(metadata.symbol.index, metadata.symbol) + asset_db_writer.write(equities=metadata) + + adjustments = [] + with maybe_show_progress( + symbols, + show_progress, + label='Downloading Yahoo adjustment data: ') as it, \ + requests.Session() as session: + for symbol in it: + path = _cachpath(symbol, 'adjustment') + try: + df = cache[path] + except KeyError: + df = cache[path] = DataReader( + symbol, + 'yahoo-actions', + start, + end, + session=session, + ).sort_index() + + df['sid'] = symbol_map[symbol] + adjustments.append(df) + + adj_df = pd.concat(adjustments) + adj_df.index.name = 'date' + adj_df.reset_index(inplace=True) + + splits = adj_df[adj_df.action == 'SPLIT'] + splits = splits.rename( + columns={'value': 'ratio', 'date': 'effective_date'}, + ) + splits.drop('action', axis=1, inplace=True) + + dividends = adj_df[adj_df.action == 'DIVIDEND'] + dividends = dividends.rename( + columns={'value': 'amount', 'date': 'ex_date'}, + ) + dividends.drop('action', axis=1, inplace=True) + # we do not have this data in the yahoo dataset + dividends['record_date'] = pd.NaT + dividends['declared_date'] = pd.NaT + dividends['pay_date'] = pd.NaT + + adjustment_writer.write(splits=splits, dividends=dividends) + + return ingest diff --git a/zipline/data/data_portal.py b/zipline/data/data_portal.py index 457008f3..89c4bef4 100644 --- a/zipline/data/data_portal.py +++ b/zipline/data/data_portal.py @@ -163,8 +163,12 @@ class DailyHistoryAggregator(object): else: after_last = pd.Timestamp( last_visited_dt + self._one_min, tz='UTC') - window = self._minute_reader.unadjusted_window( - ['open'], after_last, dt, [asset])[0] + window = self._minute_reader.load_raw_arrays( + ['open'], + after_last, + dt, + [asset], + )[0] nonnan = window[~pd.isnull(window)] if len(nonnan): val = nonnan[0] @@ -174,8 +178,12 @@ class DailyHistoryAggregator(object): opens.append(val) continue except KeyError: - window = self._minute_reader.unadjusted_window( - ['open'], market_open, dt, [asset])[0] + window = self._minute_reader.load_raw_arrays( + ['open'], + market_open, + dt, + [asset], + )[0] nonnan = window[~pd.isnull(window)] if len(nonnan): val = nonnan[0] @@ -232,15 +240,23 @@ class DailyHistoryAggregator(object): else: after_last = pd.Timestamp( last_visited_dt + self._one_min, tz='UTC') - window = self._minute_reader.unadjusted_window( - ['high'], after_last, dt, [asset]) + window = self._minute_reader.load_raw_arrays( + ['high'], + after_last, + dt, + [asset], + )[0].T val = max(last_max, np.nanmax(window)) entries[asset] = (dt_value, val) highs.append(val) continue except KeyError: - window = self._minute_reader.unadjusted_window( - ['high'], market_open, dt, [asset]) + window = self._minute_reader.load_raw_arrays( + ['high'], + market_open, + dt, + [asset], + )[0].T val = np.nanmax(window) entries[asset] = (dt_value, val) highs.append(val) @@ -288,8 +304,12 @@ class DailyHistoryAggregator(object): else: after_last = pd.Timestamp( last_visited_dt + self._one_min, tz='UTC') - window = self._minute_reader.unadjusted_window( - ['low'], after_last, dt, [asset]) + window = self._minute_reader.load_raw_arrays( + ['low'], + after_last, + dt, + [asset], + )[0].T window_min = np.nanmin(window) if pd.isnull(window_min): val = last_min @@ -299,8 +319,12 @@ class DailyHistoryAggregator(object): lows.append(val) continue except KeyError: - window = self._minute_reader.unadjusted_window( - ['low'], market_open, dt, [asset]) + window = self._minute_reader.load_raw_arrays( + ['low'], + market_open, + dt, + [asset], + )[0].T val = np.nanmin(window) entries[asset] = (dt_value, val) lows.append(val) @@ -410,15 +434,23 @@ class DailyHistoryAggregator(object): else: after_last = pd.Timestamp( last_visited_dt + self._one_min, tz='UTC') - window = self._minute_reader.unadjusted_window( - ['volume'], after_last, dt, [asset]) + window = self._minute_reader.load_raw_arrays( + ['volume'], + after_last, + dt, + [asset], + )[0] val = np.nansum(window) + last_total entries[asset] = (dt_value, val) volumes.append(val) continue except KeyError: - window = self._minute_reader.unadjusted_window( - ['volume'], market_open, dt, [asset]) + window = self._minute_reader.load_raw_arrays( + ['volume'], + market_open, + dt, + [asset], + )[0] val = np.nansum(window) entries[asset] = (dt_value, val) volumes.append(val) diff --git a/zipline/data/loader.py b/zipline/data/loader.py index 56efed99..6b7bbf9b 100644 --- a/zipline/data/loader.py +++ b/zipline/data/loader.py @@ -16,22 +16,20 @@ import os from collections import OrderedDict import logbook - import pandas as pd from pandas.io.data import DataReader import pytz - from six import iteritems from six.moves.urllib_error import HTTPError -from . benchmarks import get_benchmark_returns +from .benchmarks import get_benchmark_returns from . import treasuries, treasuries_can -from .paths import ( +from ..utils.paths import ( cache_root, data_root, ) - -from zipline.utils.tradingcalendar import ( +from ..utils.deprecate import deprecated +from ..utils.tradingcalendar import ( trading_day as trading_day_nyse, trading_days as trading_days_nyse, ) @@ -413,6 +411,10 @@ def load_from_yahoo(indexes=None, return df +@deprecated( + 'load_bars_from_yahoo is deprecated, please register a' + ' yahoo_equities data bundle instead', +) def load_bars_from_yahoo(indexes=None, stocks=None, start=None, diff --git a/zipline/data/minute_bars.py b/zipline/data/minute_bars.py index dca46e10..83531c96 100644 --- a/zipline/data/minute_bars.py +++ b/zipline/data/minute_bars.py @@ -29,6 +29,7 @@ from zipline.data._minute_bar_internal import ( ) from zipline.gens.sim_engine import NANOS_IN_MINUTE +from zipline.utils.cli import maybe_show_progress from zipline.utils.memoize import lazyval US_EQUITIES_MINUTES_PER_DAY = 390 @@ -91,6 +92,22 @@ def _sid_subdir_path(sid): class BcolzMinuteBarMetadata(object): + """ + Parameters + ---------- + first_trading_day : datetime-like + UTC midnight of the first day available in the dataset. + minute_index : pd.DatetimeIndex + The minutes which act as an index into the corresponding values + written into each sid's ctable. + market_opens : pd.DatetimeIndex + The market opens for each day in the data set. (Not yet required.) + market_closes : pd.DatetimeIndex + The market closes for each day in the data set. (Not yet required.) + ohlc_ratio : int + The factor by which the pricing data is multiplied so that the + float data can be stored as an integer. + """ METADATA_FILENAME = 'metadata.json' @@ -122,22 +139,6 @@ class BcolzMinuteBarMetadata(object): market_opens, market_closes, ohlc_ratio): - """ - Parameters: - ----------- - first_trading_day : datetime-like - UTC midnight of the first day available in the dataset. - minute_index : pd.DatetimeIndex - The minutes which act as an index into the corresponding values - written into each sid's ctable. - market_opens : pd.DatetimeIndex - The market opens for each day in the data set. (Not yet required.) - market_closes : pd.DatetimeIndex - The market closes for each day in the data set. (Not yet required.) - ohlc_ratio : int - The factor by which the pricing data is multiplied so that the - float data can be stored as an integer. - """ self.first_trading_day = first_trading_day self.market_opens = market_opens self.market_closes = market_closes @@ -176,6 +177,55 @@ class BcolzMinuteBarWriter(object): """ Class capable of writing minute OHLCV data to disk into bcolz format. + Parameters + ---------- + first_trading_day : datetime + The first trading day in the data set. + rootdir : string + Path to the root directory into which to write the metadata and + bcolz subdirectories. + market_opens : pd.Series + The market opens used as a starting point for each periodic span of + minutes in the index. + + The index of the series is expected to be a DatetimeIndex of the + UTC midnight of each trading day. + + The values are datetime64-like UTC market opens for each day in the + index. + market_closes : pd.Series + The market closes that correspond with the market opens, + + The index of the series is expected to be a DatetimeIndex of the + UTC midnight of each trading day. + + The values are datetime64-like UTC market opens for each day in the + index. + + The closes are written so that the reader can filter out non-market + minutes even though the tail end of early closes are written in + the data arrays to keep a regular shape. + minutes_per_day : int + The number of minutes per each period. Defaults to 390, the mode + of minutes in NYSE trading days. + ohlc_ratio : int, optional + The ratio by which to multiply the pricing data to convert the + floats from floats to an integer to fit within the np.uint32. + + The default is 1000 to support pricing data which comes in to the + thousands place. + expectedlen : int, optional + The expected length of the dataset, used when creating the initial + bcolz ctable. + + If the expectedlen is not used, the chunksize and corresponding + compression ratios are not ideal. + + Defaults to supporting 15 years of NYSE equity market data. + see: http://bcolz.blosc.org/opt-tips.html#informing-about-the-length-of-your-carrays # noqa + + Notes + ----- Writes a bcolz directory for each individual sid, all contained within a root directory which also contains metadata about the entire dataset. @@ -214,6 +264,10 @@ class BcolzMinuteBarWriter(object): The datetimes which correspond to each position are written in the metadata as integer nanoseconds since the epoch into the `minute_index` key. + + See Also + -------- + zipline.data.minute_bars.BcolzMinuteBarReader """ COL_NAMES = ('open', 'high', 'low', 'close', 'volume') @@ -225,61 +279,6 @@ class BcolzMinuteBarWriter(object): minutes_per_day, ohlc_ratio=OHLC_RATIO, expectedlen=DEFAULT_EXPECTEDLEN): - """ - Parameters: - ----------- - first_trading_day : datetime-like - The first trading day in the data set. - - rootdir : string - Path to the root directory into which to write the metadata and - bcolz subdirectories. - - market_opens : pd.Series - The market opens used as a starting point for each periodic span of - minutes in the index. - - The index of the series is expected to be a DatetimeIndex of the - UTC midnight of each trading day. - - The values are datetime64-like UTC market opens for each day in the - index. - - market_closes : pd.Series - The market closes that correspond with the market opens, - - The index of the series is expected to be a DatetimeIndex of the - UTC midnight of each trading day. - - The values are datetime64-like UTC market opens for each day in the - index. - - The closes are written so that the reader can filter out non-market - minutes even though the tail end of early closes are written in - the data arrays to keep a regular shape. - - minutes_per_day : int - The number of minutes per each period. Defaults to 390, the mode - of minutes in NYSE trading days. - - ohlc_ratio : int - The ratio by which to multiply the pricing data to convert the - floats from floats to an integer to fit within the np.uint32. - - The default is 1000 to support pricing data which comes in to the - thousands place. - - expectedlen : int - The expected length of the dataset, used when creating the initial - bcolz ctable. - - If the expectedlen is not used, the chunksize and corresponding - compression ratios are not ideal. - - Defaults to supporting 15 years of NYSE equity market data. - - see: http://bcolz.blosc.org/opt-tips.html#informing-about-the-length-of-your-carrays # noqa - """ self._rootdir = rootdir self._first_trading_day = first_trading_day self._market_opens = market_opens[ @@ -441,7 +440,38 @@ class BcolzMinuteBarWriter(object): assert new_last_date == date, "new_last_date={0} != date={1}".format( new_last_date, date) - def write(self, sid, df): + def write(self, data, show_progress=False): + """Write a stream of minute data. + + Parameters + ---------- + data : iterable[(int, pd.DataFrame)] + The data to write. Each element should be a tuple of sid, data + where data has the following format: + columns : ('open', 'high', 'low', 'close', 'volume') + open : float64 + high : float64 + low : float64 + close : float64 + volume : float64|int64 + index : DatetimeIndex of market minutes. + A given sid may appear more than once in ``data``; however, + the dates must be strictly increasing. + show_progress : bool, optional + Whether or not to show a progress bar while writing. + """ + ctx = maybe_show_progress( + data, + show_progress=show_progress, + item_show_func=lambda e: e if e is None else str(e[0]), + label="Merging minute equity files:", + ) + write_sid = self.write_sid + with ctx as it: + for e in it: + write_sid(*e) + + def write_sid(self, sid, df): """ Write the OHLCV data for the given sid. If there is no bcolz ctable yet created for the sid, create it. @@ -585,17 +615,21 @@ class BcolzMinuteBarWriter(object): class BcolzMinuteBarReader(object): + """ + Reader for data written by BcolzMinuteBarWriter + Parameters: + ----------- + rootdir : string + The root directory containing the metadata and asset bcolz + directories. + + See Also + -------- + zipline.data.minute_bars.BcolzMinuteBarWriter + """ def __init__(self, rootdir): - """ - Reader for data written by BcolzMinuteBarWriter - Parameters: - ----------- - rootdir : string - The root directory containing the metadata and asset bcolz - directories. - """ self._rootdir = rootdir metadata = self._get_metadata() @@ -826,7 +860,7 @@ class BcolzMinuteBarReader(object): US_EQUITIES_MINUTES_PER_DAY, ) - def unadjusted_window(self, fields, start_dt, end_dt, sids): + def load_raw_arrays(self, fields, start_dt, end_dt, sids): """ Parameters ---------- @@ -843,7 +877,7 @@ class BcolzMinuteBarReader(object): ------- list of np.ndarray A list with an entry per field of ndarrays with shape - (sids, minutes in range) with a dtype of float64, containing the + (minutes in range, sids) with a dtype of float64, containing the values for the respective field over start and end dt range. """ start_idx = self._find_position_of_minute(start_dt) @@ -860,7 +894,7 @@ class BcolzMinuteBarReader(object): length = excl_stop - excl_start + 1 num_minutes -= length - shape = (len(sids), num_minutes) + shape = num_minutes, len(sids) for field in fields: if field != 'volume': @@ -877,7 +911,9 @@ class BcolzMinuteBarReader(object): excl_start - start_idx:excl_stop - start_idx + 1] values = np.delete(values, excl_slice) where = values != 0 - out[i, where] = values[where] + # first slice down to len(where) because we might not have + # written data for all the minutes requested + out[:len(where), i][where] = values[where] if field != 'volume': out *= self._ohlc_inverse results.append(out) diff --git a/zipline/data/paths.py b/zipline/data/paths.py deleted file mode 100644 index 59b0ab98..00000000 --- a/zipline/data/paths.py +++ /dev/null @@ -1,91 +0,0 @@ -""" -Canonical path locations for zipline data. - -Paths are rooted at $ZIPLINE_ROOT if that environment variable is set. -Otherwise default to expanduser(~/.zipline) -""" -import os -from os.path import ( - expanduser, - join, -) - - -def zipline_root(environ=None): - """ - Get the root directory for all zipline-managed files. - - For testing purposes, this accepts a dictionary to interpret as the os - environment. - - Parameters - ---------- - environ : dict, optional - A dict to interpret as the os environment. - - Returns - ------- - root : string - Path to the zipline root dir. - """ - if environ is None: - environ = os.environ.copy() - - root = environ.get('ZIPLINE_ROOT', None) - if root is None: - root = expanduser('~/.zipline') - - return root - - -def zipline_root_path(path, environ=None): - """ - Get a path relative to the zipline root. - - Parameters - ---------- - path : str - The requested path. - environ : dict, optional - An environment dict to forward to zipline_root. - - Returns - ------- - newpath : str - The requested path joined with the zipline root. - """ - return join(zipline_root(environ=environ), path) - - -def data_root(environ=None): - """ - The root directory for zipline data files. - - Parameters - ---------- - environ : dict, optional - An environment dict to forward to zipline_root. - - Returns - ------- - data_root : str - The zipline data root. - """ - return zipline_root_path('data', environ=environ) - - -def cache_root(environ=None): - """ - The root directory for zipline cache files. - - Parameters - ---------- - environ : dict, optional - An environment dict to forward to zipline_root. - - Returns - ------- - cache_root : str - The zipline cache root. - """ - return zipline_root_path('cache', environ=environ) diff --git a/zipline/data/us_equity_loader.py b/zipline/data/us_equity_loader.py index f3392b63..5385b203 100644 --- a/zipline/data/us_equity_loader.py +++ b/zipline/data/us_equity_loader.py @@ -24,7 +24,6 @@ from pandas.tslib import normalize_date from six import with_metaclass -from zipline.pipeline.data.equity_pricing import USEquityPricing from zipline.lib._float64window import AdjustedArrayWindow as Float64Window from zipline.lib.adjustment import Float64Multiply from zipline.utils.cache import ExpiringCache @@ -300,9 +299,12 @@ class USEquityDailyHistoryLoader(USEquityHistoryLoader): return self._reader._calendar def _array(self, dts, assets, field): - col = getattr(USEquityPricing, field) return self._reader.load_raw_arrays( - [col], dts[0], dts[-1], assets)[0] + [field], + dts[0], + dts[-1], + assets, + )[0] class USEquityMinuteHistoryLoader(USEquityHistoryLoader): @@ -318,5 +320,9 @@ class USEquityMinuteHistoryLoader(USEquityHistoryLoader): end=self._reader.last_available_dt)] def _array(self, dts, assets, field): - return self._reader.unadjusted_window( - [field], dts[0], dts[-1], assets)[0].T + return self._reader.load_raw_arrays( + [field], + dts[0], + dts[-1], + assets, + )[0] diff --git a/zipline/data/us_equity_pricing.py b/zipline/data/us_equity_pricing.py index d54e209b..fbfb1a06 100644 --- a/zipline/data/us_equity_pricing.py +++ b/zipline/data/us_equity_pricing.py @@ -22,7 +22,6 @@ import warnings from bcolz import ( carray, ctable, - open as open_ctable, ) from collections import namedtuple import logbook @@ -193,7 +192,7 @@ class BcolzDailyBarWriter(object): See Also -------- - BcolzDailyBarReader : Consumer of the data written by this class. + zipline.data.us_equity_pricing.BcolzDailyBarReader """ _csv_dtypes = { 'open': float64, @@ -209,7 +208,7 @@ class BcolzDailyBarWriter(object): @property def progress_bar_message(self): - return "Merging asset files:" + return "Merging daily equity files:" def progress_bar_item_show_func(self, value): return value if value is None else str(value[0]) @@ -229,9 +228,9 @@ class BcolzDailyBarWriter(object): The assets that should be in ``data``. If this is provided we will check ``data`` against the assets and provide better progress information. - show_progress : bool + show_progress : bool, optional Whether or not to show a progress bar while writing. - invalid_data_behavior : {'warn', 'raise', 'ignore'} + invalid_data_behavior : {'warn', 'raise', 'ignore'}, optional What to do when data is encountered that is outside the range of a uint32. @@ -365,6 +364,7 @@ class BcolzDailyBarWriter(object): full_table.attrs['last_row'] = last_row full_table.attrs['calendar_offset'] = calendar_offset full_table.attrs['calendar'] = calendar.asi8.tolist() + full_table.flush() return full_table @@ -451,11 +451,13 @@ class BcolzDailyBarReader(DailyBarReader): We use calendar_offset and calendar to orient loaded blocks within a range of queried dates. - """ - @preprocess(table=coerce_string(open_ctable, mode='r')) - def __init__(self, table, read_all_threshold=3000): - self._table = table + See Also + -------- + zipline.data.us_equity_pricing.BcolzDailyBarWriter + """ + def __init__(self, table, read_all_threshold=3000): + self._maybe_table_rootdir = table # Cache of fully read np.array for the carrays in the daily bar table. # raw_array does not use the same cache, but it could. # Need to test keeping the entire array in memory for the course of a @@ -464,6 +466,13 @@ class BcolzDailyBarReader(DailyBarReader): self.PRICE_ADJUSTMENT_FACTOR = 0.001 self._read_all_threshold = read_all_threshold + @lazyval + def _table(self): + maybe_table_rootdir = self._maybe_table_rootdir + if isinstance(maybe_table_rootdir, ctable): + return maybe_table_rootdir + return ctable(rootdir=maybe_table_rootdir, mode='r') + @lazyval def _calendar(self): return DatetimeIndex(self._table.attrs['calendar'], tz='UTC') @@ -565,7 +574,7 @@ class BcolzDailyBarReader(DailyBarReader): return _read_bcolz_data( self._table, (end_idx - start_idx + 1, len(assets)), - [column.name for column in columns], + list(columns), first_rows, last_rows, offsets, @@ -717,12 +726,12 @@ class PanelDailyBarReader(DailyBarReader): return self._calendar[-1] def load_raw_arrays(self, columns, start_date, end_date, assets): - col_names = [col.name for col in columns] + columns = list(columns) cal = self._calendar index = cal[cal.slice_indexer(start_date, end_date)] shape = (len(index), len(assets)) results = [] - for col in col_names: + for col in columns: outbuf = zeros(shape=shape) for i, asset in enumerate(assets): data = self.panel.loc[asset, start_date:end_date, col] @@ -792,7 +801,7 @@ class SQLiteAdjustmentWriter(object): See Also -------- - SQLiteAdjustmentReader + zipline.data.us_equity_pricing.SQLiteAdjustmentReader """ def __init__(self, @@ -862,6 +871,11 @@ class SQLiteAdjustmentWriter(object): SQLITE_ADJUSTMENT_TABLENAMES, ) ) + if not (frame is None or frame.empty): + frame = frame.copy() + frame['effective_date'] = frame['effective_date'].values.astype( + 'datetime64[s]', + ).astype('int64') return self._write( tablename, SQLITE_ADJUSTMENT_COLUMN_DTYPES, @@ -1016,82 +1030,72 @@ class SQLiteAdjustmentWriter(object): Parameters ---------- - splits : pandas.DataFrame - Dataframe containing split data. - mergers : pandas.DataFrame - DataFrame containing merger data. - dividends : pandas.DataFrame - DataFrame containing dividend data. + splits : pandas.DataFrame, optional + Dataframe containing split data. The format of this dataframe is: + effective_date : int + The date, represented as seconds since Unix epoch, on which + the adjustment should be applied. + ratio : float + A value to apply to all data earlier than the effective date. + For open, high, low, and close those values are multiplied by + the ratio. Volume is divided by this value. + sid : int + The asset id associated with this adjustment. + mergers : pandas.DataFrame, optional + DataFrame containing merger data. The format of this dataframe is: + effective_date : int + The date, represented as seconds since Unix epoch, on which + the adjustment should be applied. + ratio : float + A value to apply to all data earlier than the effective date. + For open, high, low, and close those values are multiplied by + the ratio. Volume is unaffected. + sid : int + The asset id associated with this adjustment. + dividends : pandas.DataFrame, optional + DataFrame containing dividend data. The format of the dataframe is: + sid : int + The asset id associated with this adjustment. + ex_date : datetime64 + The date on which an equity must be held to be eligible to + receive payment. + declared_date : datetime64 + The date on which the dividend is announced to the public. + pay_date : datetime64 + The date on which the dividend is distributed. + record_date : datetime64 + The date on which the stock ownership is checked to determine + distribution of dividends. + amount : float + The cash amount paid for each share. - Notes - ----- - DataFrame input (`splits`, `mergers`) should all have - the following columns: - - effective_date : int - The date, represented as seconds since Unix epoch, on which the - adjustment should be applied. - ratio : float - A value to apply to all data earlier than the effective date. - sid : int - The asset id associated with this adjustment. - - The ratio column is interpreted as follows: - - For all adjustment types, multiply price fields ('open', 'high', - 'low', and 'close') by the ratio. - - For **splits only**, **divide** volume by the adjustment ratio. - - DataFrame input, 'dividends' should have the following columns: - - sid : int - The asset id associated with this adjustment. - ex_date : datetime64 - The date on which an equity must be held to be eligible to receive - payment. - declared_date : datetime64 - The date on which the dividend is announced to the public. - pay_date : datetime64 - The date on which the dividend is distributed. - record_date : datetime64 - The date on which the stock ownership is checked to determine - distribution of dividends. - amount : float - The cash amount paid for each share. - - Dividend ratios are calculated as - 1.0 - (dividend_value / "close on day prior to dividend ex_date"). - - - DataFrame input, 'stock_dividends' should have the following columns: - - sid : int - The asset id associated with this adjustment. - ex_date : datetime64 - The date on which an equity must be held to be eligible to receive - payment. - declared_date : datetime64 - The date on which the dividend is announced to the public. - pay_date : datetime64 - The date on which the dividend is distributed. - record_date : datetime64 - The date on which the stock ownership is checked to determine - distribution of dividends. - payment_sid : int - The asset id of the shares that should be paid instead of cash. - ratio: float - The ratio of currently held shares in the held sid that should - be paid with new shares of the payment_sid. - - stock_dividends is optional. - - - Returns - ------- - None + Dividend ratios are calculated as: + ``1.0 - (dividend_value / "close on day prior to ex_date")`` + stock_dividends : pandas.DataFrame, optional + DataFrame containing stock dividend data. The format of the + dataframe is: + sid : int + The asset id associated with this adjustment. + ex_date : datetime64 + The date on which an equity must be held to be eligible to + receive payment. + declared_date : datetime64 + The date on which the dividend is announced to the public. + pay_date : datetime64 + The date on which the dividend is distributed. + record_date : datetime64 + The date on which the stock ownership is checked to determine + distribution of dividends. + payment_sid : int + The asset id of the shares that should be paid instead of + cash. + ratio : float + The ratio of currently held shares in the held sid that + should be paid with new shares of the payment_sid. See Also -------- - SQLiteAdjustmentReader : Consumer for the data written by this class + zipline.data.us_equity_pricing.SQLiteAdjustmentReader """ self.write_frame('splits', splits) self.write_frame('mergers', mergers) @@ -1168,6 +1172,10 @@ class SQLiteAdjustmentReader(object): ---------- conn : str or sqlite3.Connection Connection from which to load data. + + See Also + -------- + zipline.data.us_equity_pricing.SQLiteAdjustmentWriter """ @preprocess(conn=coerce_string(sqlite3.connect)) @@ -1177,7 +1185,7 @@ class SQLiteAdjustmentReader(object): def load_adjustments(self, columns, dates, assets): return load_adjustments_from_sqlite( self.conn, - [column.name for column in columns], + list(columns), dates, assets, ) diff --git a/zipline/examples/__init__.py b/zipline/examples/__init__.py new file mode 100644 index 00000000..62e5ea33 --- /dev/null +++ b/zipline/examples/__init__.py @@ -0,0 +1,17 @@ +from glob import glob +from importlib import import_module +import os + +for f in os.listdir(os.path.dirname(__file__)): + if not f.endswith('.py') or f == '__init__.py': + continue + modname = f[:-len('.py')] + globals()[modname] = import_module('.' + modname, package=__name__) + +del f +try: + del modname +except NameError: + pass + +del os, import_module, glob diff --git a/zipline/examples/buy_and_hold.py b/zipline/examples/buy_and_hold.py index 69af2159..fbd6b2f8 100644 --- a/zipline/examples/buy_and_hold.py +++ b/zipline/examples/buy_and_hold.py @@ -13,10 +13,7 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. -import pandas as pd -from zipline import TradingAlgorithm from zipline.api import order, symbol -from zipline.data.loader import load_bars_from_yahoo stocks = ['AAPL', 'MSFT'] @@ -33,18 +30,12 @@ def handle_data(context, data): context.has_ordered = True -if __name__ == '__main__': +def _test_args(): + """Extra arguments to use when zipline's automated tests run this example. + """ + import pandas as pd - # creating time interval - start = pd.Timestamp('2008-01-01', tz='UTC') - end = pd.Timestamp('2013-01-01', tz='UTC') - - # loading the data - input_data = load_bars_from_yahoo( - stocks=stocks, - start=start, - end=end, - ) - - algo = TradingAlgorithm(initialize=initialize, handle_data=handle_data) - results = algo.run(input_data) + return { + 'start': pd.Timestamp('2008', tz='utc'), + 'end': pd.Timestamp('2013', tz='utc'), + } diff --git a/zipline/examples/buyapple.conf b/zipline/examples/buyapple.conf deleted file mode 100644 index 42ce1cac..00000000 --- a/zipline/examples/buyapple.conf +++ /dev/null @@ -1,3 +0,0 @@ -[Defaults] -algofile=buyapple.py -symbols=AAPL diff --git a/zipline/examples/buyapple.py b/zipline/examples/buyapple.py old mode 100755 new mode 100644 index 2ea1f0af..df7d9486 --- a/zipline/examples/buyapple.py +++ b/zipline/examples/buyapple.py @@ -23,7 +23,7 @@ def initialize(context): def handle_data(context, data): order(symbol('AAPL'), 10) - record(AAPL=data.current(symbol('AAPL'), "price")) + record(AAPL=data.current(symbol('AAPL'), 'price')) # Note: this function can be removed if running @@ -43,24 +43,12 @@ def analyze(context=None, results=None): plt.show() -# Note: this if-block should be removed if running -# this algorithm on quantopian.com -if __name__ == '__main__': - from datetime import datetime - import pytz - from zipline.algorithm import TradingAlgorithm - from zipline.utils.factory import load_from_yahoo +def _test_args(): + """Extra arguments to use when zipline's automated tests run this example. + """ + import pandas as pd - # Set the simulation start and end dates - start = datetime(2014, 1, 1, 0, 0, 0, 0, pytz.utc) - end = datetime(2014, 11, 1, 0, 0, 0, 0, pytz.utc) - - # Load price data from yahoo. - data = load_from_yahoo(stocks=['AAPL'], indexes={}, start=start, - end=end) - - # Create and run the algorithm. - algo = TradingAlgorithm(initialize=initialize, handle_data=handle_data) - results = algo.run(data) - - analyze(results=results) + return { + 'start': pd.Timestamp('2014-01-01', tz='utc'), + 'end': pd.Timestamp('2014-11-01', tz='utc'), + } diff --git a/zipline/examples/dual_ema_talib.py b/zipline/examples/dual_ema_talib.py old mode 100755 new mode 100644 index 92b1b41b..cbdc9957 --- a/zipline/examples/dual_ema_talib.py +++ b/zipline/examples/dual_ema_talib.py @@ -98,25 +98,12 @@ def analyze(context=None, results=None): plt.show() -# Note: this if-block should be removed if running -# this algorithm on quantopian.com -if __name__ == '__main__': - from datetime import datetime - import pytz - from zipline.algorithm import TradingAlgorithm - from zipline.utils.factory import load_from_yahoo +def _test_args(): + """Extra arguments to use when zipline's automated tests run this example. + """ + import pandas as pd - # Set the simulation start and end dates. - start = datetime(2014, 1, 1, 0, 0, 0, 0, pytz.utc) - end = datetime(2014, 11, 1, 0, 0, 0, 0, pytz.utc) - - # Load price data from yahoo. - data = load_from_yahoo(stocks=['AAPL'], indexes={}, start=start, - end=end) - - # Create and run the algorithm. - algo = TradingAlgorithm(initialize=initialize, handle_data=handle_data) - results = algo.run(data).dropna() - - # Plot the portfolio and asset data. - analyze(results=results) + return { + 'start': pd.Timestamp('2014-01-01', tz='utc'), + 'end': pd.Timestamp('2014-11-01', tz='utc'), + } diff --git a/zipline/examples/dual_moving_average.conf b/zipline/examples/dual_moving_average.conf deleted file mode 100644 index dfe0a805..00000000 --- a/zipline/examples/dual_moving_average.conf +++ /dev/null @@ -1,5 +0,0 @@ -[Defaults] -algofile=dual_moving_average.py -symbols=AAPL -start=2000-1-1 -end=2014-1-1 diff --git a/zipline/examples/dual_moving_average.py b/zipline/examples/dual_moving_average.py old mode 100755 new mode 100644 index 55c877c4..529b497a --- a/zipline/examples/dual_moving_average.py +++ b/zipline/examples/dual_moving_average.py @@ -97,29 +97,12 @@ def analyze(context=None, results=None): plt.show() -# Note: this if-block should be removed if running -# this algorithm on quantopian.com -if __name__ == '__main__': - from datetime import datetime - import pytz - from zipline.algorithm import TradingAlgorithm - from zipline.utils.factory import load_bars_from_yahoo +def _test_args(): + """Extra arguments to use when zipline's automated tests run this example. + """ + import pandas as pd - # Set the simulation start and end dates. - start = datetime(2011, 1, 1, 0, 0, 0, 0, pytz.utc) - end = datetime(2013, 1, 1, 0, 0, 0, 0, pytz.utc) - - # Load price data from yahoo. - data = load_bars_from_yahoo( - stocks=['AAPL'], - indexes={}, - start=start, - end=end, - ) - - # Create and run the algorithm. - algo = TradingAlgorithm(initialize=initialize, handle_data=handle_data) - results = algo.run(data) - - # Plot the portfolio and asset data. - analyze(results=results) + return { + 'start': pd.Timestamp('2011', tz='utc'), + 'end': pd.Timestamp('2013', tz='utc'), + } diff --git a/zipline/examples/olmar.py b/zipline/examples/olmar.py index ca116e13..e041c778 100644 --- a/zipline/examples/olmar.py +++ b/zipline/examples/olmar.py @@ -1,11 +1,7 @@ import sys import logbook import numpy as np -from datetime import datetime -import pytz -from zipline.algorithm import TradingAlgorithm -from zipline.utils.factory import load_from_yahoo from zipline.finance import commission zipline_logging = logbook.NestedSetup([ @@ -161,20 +157,12 @@ def analyze(context=None, results=None): plt.show() -# Note: this if-block should be removed if running -# this algorithm on quantopian.com -if __name__ == '__main__': - # Set the simulation start and end dates. - start = datetime(2004, 1, 1, 0, 0, 0, 0, pytz.utc) - end = datetime(2008, 1, 1, 0, 0, 0, 0, pytz.utc) +def _test_args(): + """Extra arguments to use when zipline's automated tests run this example. + """ + import pandas as pd - # Load price data from yahoo. - data = load_from_yahoo(stocks=STOCKS, indexes={}, start=start, end=end) - data = data.dropna() - - # Create and run the algorithm. - olmar = TradingAlgorithm(handle_data=handle_data, initialize=initialize) - results = olmar.run(data) - - # Plot the portfolio data. - analyze(results=results) + return { + 'start': pd.Timestamp('2004', tz='utc'), + 'end': pd.Timestamp('2008', tz='utc'), + } diff --git a/zipline/pipeline/loaders/equity_pricing_loader.py b/zipline/pipeline/loaders/equity_pricing_loader.py index 64448903..60939da6 100644 --- a/zipline/pipeline/loaders/equity_pricing_loader.py +++ b/zipline/pipeline/loaders/equity_pricing_loader.py @@ -69,15 +69,15 @@ class USEquityPricingLoader(PipelineLoader): start_date, end_date = _shift_dates( self._calendar, dates[0], dates[-1], shift=1, ) - + colnames = [c.name for c in columns] raw_arrays = self.raw_price_loader.load_raw_arrays( - columns, + colnames, start_date, end_date, assets, ) adjustments = self.adjustments_loader.load_adjustments( - columns, + colnames, dates, assets, ) diff --git a/zipline/pipeline/loaders/synthetic.py b/zipline/pipeline/loaders/synthetic.py index 913dac82..f9957ae9 100644 --- a/zipline/pipeline/loaders/synthetic.py +++ b/zipline/pipeline/loaders/synthetic.py @@ -213,7 +213,7 @@ def asset_end(asset_info, asset): return ret -def make_daily_bar_data(asset_info, calendar): +def make_bar_data(asset_info, calendar): """ For a given asset/date/column combination, we generate a corresponding raw @@ -249,7 +249,7 @@ def make_daily_bar_data(asset_info, calendar): assert ( # Using .value here to avoid having to care about UTC-aware dates. PSEUDO_EPOCH.value < - calendar.min().value <= + calendar.normalize().min().value <= asset_info['start_date'].min().value ), "calendar.min(): %s\nasset_info['start_date'].min(): %s" % ( calendar.min(), @@ -266,13 +266,13 @@ def make_daily_bar_data(asset_info, calendar): """ # Get the dates for which this asset existed according to our asset # info. - dates = calendar[calendar.slice_indexer( + datetimes = calendar[calendar.slice_indexer( asset_start(asset_info, asset_id), asset_end(asset_info, asset_id), )] data = full( - (len(dates), len(US_EQUITY_PRICING_BCOLZ_COLUMNS)), + (len(datetimes), len(US_EQUITY_PRICING_BCOLZ_COLUMNS)), asset_id * 100 * 1000, dtype=uint32, ) @@ -281,15 +281,15 @@ def make_daily_bar_data(asset_info, calendar): data[:, :5] += arange(5, dtype=uint32) * 1000 # Add days since Jan 1 2001 for OHLCV columns. - data[:, :5] += (dates - PSEUDO_EPOCH).days[:, None].astype(uint32) + data[:, :5] += (datetimes - PSEUDO_EPOCH).days[:, None].astype(uint32) frame = DataFrame( data, - index=dates, + index=datetimes, columns=US_EQUITY_PRICING_BCOLZ_COLUMNS, ) - frame['day'] = nanos_to_seconds(dates.asi8) + frame['day'] = nanos_to_seconds(datetimes.asi8) frame['id'] = asset_id return frame @@ -297,7 +297,7 @@ def make_daily_bar_data(asset_info, calendar): yield asset, _raw_data_for_asset(asset) -def expected_daily_bar_value(asset_id, date, colname): +def expected_bar_value(asset_id, date, colname): """ Check that the raw value for an asset/date/column triple is as expected. @@ -310,7 +310,7 @@ def expected_daily_bar_value(asset_id, date, colname): return from_asset + from_colname + from_date -def expected_daily_bar_values_2d(dates, asset_info, colname): +def expected_bar_values_2d(dates, asset_info, colname): """ Return an 2D array containing cls.expected_value(asset_id, date, colname) for each date/asset pair in the inputs. @@ -336,7 +336,7 @@ def expected_daily_bar_values_2d(dates, asset_info, colname): # date. if not (start <= date <= end): continue - data[i, j] = expected_daily_bar_value(asset, date, colname) + data[i, j] = expected_bar_value(asset, date, colname) return data diff --git a/zipline/testing/__init__.py b/zipline/testing/__init__.py index 5106cac5..a415daa8 100644 --- a/zipline/testing/__init__.py +++ b/zipline/testing/__init__.py @@ -29,6 +29,7 @@ from .core import ( # noqa parameter_space, parameter_space, patch_os_environment, + patch_read_csv, permute_rows, powerset, product_upper_triangle, diff --git a/zipline/testing/core.py b/zipline/testing/core.py index a3dcae6e..11430470 100644 --- a/zipline/testing/core.py +++ b/zipline/testing/core.py @@ -349,7 +349,7 @@ def make_trade_data_for_asset_info(dates, ) if writer: - writer.write(sid, df) + writer.write_sid(sid, df) trade_data[sid] = df @@ -424,8 +424,8 @@ def write_minute_data(env, tempdir, minutes, sids): def create_minute_bar_data(minutes, sids): length = len(minutes) - return { - sid: pd.DataFrame( + for sid_idx, sid in enumerate(sids): + yield sid, pd.DataFrame( { 'open': np.arange(length) + 10 + sid_idx, 'high': np.arange(length) + 15 + sid_idx, @@ -435,8 +435,6 @@ def create_minute_bar_data(minutes, sids): }, index=minutes, ) - for sid_idx, sid in enumerate(sids) - } def create_daily_bar_data(trading_days, sids): @@ -492,20 +490,17 @@ def create_data_portal(env, tempdir, sim_params, sids, adjustment_reader=None): ) -def write_bcolz_minute_data(env, days, path, df_dict): +def write_bcolz_minute_data(env, days, path, data): market_opens = env.open_and_closes.market_open.loc[days] market_closes = env.open_and_closes.market_close.loc[days] - writer = BcolzMinuteBarWriter( + BcolzMinuteBarWriter( days[0], path, market_opens, market_closes, US_EQUITIES_MINUTES_PER_DAY - ) - - for sid, df in iteritems(df_dict): - writer.write(sid, df) + ).write(data) def create_minute_df_for_asset(env, @@ -1183,6 +1178,7 @@ zipline_git_root = abspath( ) +@nottest def test_resource_path(*path_parts): return os.path.join(zipline_git_root, 'tests', 'resources', *path_parts) @@ -1313,3 +1309,37 @@ class tmp_bcolz_daily_bar_reader(_TmpBarReader): @staticmethod def _write(env, days, path, data): BcolzDailyBarWriter(path, days).write(data) + + +@contextmanager +def patch_read_csv(url_map, module=pd, strict=False): + """Patch pandas.read_csv to map lookups from url to another. + + Parameters + ---------- + url_map : mapping[str or file-like object -> str or file-like object] + The mapping to use to redirect read_csv calls. + module : module, optional + The module to patch ``read_csv`` on. By default this is ``pandas``. + This should be set to another module if ``read_csv`` is early-bound + like ``from pandas import read_csv`` instead of late-bound like: + ``import pandas as pd; pd.read_csv``. + strict : bool, optional + If true, then this will assert that ``read_csv`` is only called with + elements in the ``url_map``. + """ + read_csv = pd.read_csv + + def patched_read_csv(filepath_or_buffer, *args, **kwargs): + if filepath_or_buffer in url_map: + return read_csv(url_map[filepath_or_buffer], *args, **kwargs) + elif not strict: + return read_csv(filepath_or_buffer, *args, **kwargs) + else: + raise AssertionError( + 'attempted to call read_csv on %r which not in the url map' % + filepath_or_buffer, + ) + + with patch.object(module, 'read_csv', patched_read_csv): + yield diff --git a/zipline/testing/fixtures.py b/zipline/testing/fixtures.py index d1547489..ff82042c 100644 --- a/zipline/testing/fixtures.py +++ b/zipline/testing/fixtures.py @@ -6,7 +6,7 @@ from contextlib2 import ExitStack from logbook import NullHandler, Logger from nose_parameterized import parameterized from pandas.util.testing import assert_series_equal -from six import with_metaclass, iteritems +from six import with_metaclass from toolz import flip import numpy as np import pandas as pd @@ -681,7 +681,7 @@ class WithBcolzMinuteBarReader(WithTradingEnvironment, WithTmpDir): Methods ------- - make_minute_bar_data() -> dict[int -> pd.DataFrame] + make_minute_bar_data() -> iterable[(int, pd.DataFrame)] A class method that returns a dict mapping sid to dataframe which will be written to the bcolz files that the class's ``BcolzMinuteBarReader`` will read from. By default this creates @@ -734,9 +734,7 @@ class WithBcolzMinuteBarReader(WithTradingEnvironment, WithTmpDir): cls.env.open_and_closes.market_close.loc[days], US_EQUITIES_MINUTES_PER_DAY ) - cls.bcolz_minute_bar_data = cls.make_minute_bar_data() - for sid, df in iteritems(cls.bcolz_minute_bar_data): - writer.write(sid, df) + writer.write(cls.make_minute_bar_data()) cls.bcolz_minute_bar_reader = BcolzMinuteBarReader(p) diff --git a/zipline/testing/predicates.py b/zipline/testing/predicates.py index 6dd6f8d9..93f3610e 100644 --- a/zipline/testing/predicates.py +++ b/zipline/testing/predicates.py @@ -1,7 +1,89 @@ +from functools import partial +import inspect + +from nose.tools import ( # noqa + assert_almost_equal, + assert_almost_equals, + assert_dict_contains_subset, + assert_false, + assert_greater, + assert_greater_equal, + assert_in, + assert_is, + assert_is_instance, + assert_is_none, + assert_is_not, + assert_is_not_none, + assert_less, + assert_less_equal, + assert_multi_line_equal, + assert_not_almost_equal, + assert_not_almost_equals, + assert_not_equal, + assert_not_equals, + assert_not_in, + assert_not_is_instance, + assert_raises, + assert_raises_regexp, + assert_regexp_matches, + assert_sequence_equal, + assert_set_equal, + assert_true, + assert_tuple_equal, +) +import numpy as np +import pandas as pd +from pandas.util.testing import assert_frame_equal from six import iteritems, viewkeys, PY2 +from toolz import dissoc, keyfilter +import toolz.curried.operator as op from zipline.dispatch import dispatch +from zipline.lib.adjustment import Adjustment from zipline.utils.functional import dzip_exact +from zipline.utils.math_utils import tolerant_equals + + +def keywords(func): + """Get the argument names of a function + + >>> def f(x, y=2): + ... pass + + >>> keywords(f) + ['x', 'y'] + + Notes + ----- + Taken from odo.utils + """ + if isinstance(func, type): + return keywords(func.__init__) + return inspect.getargspec(func).args + + +def filter_kwargs(f, kwargs): + """Return a dict of valid kwargs for `f` from a subset of `kwargs` + + Examples + -------- + >>> def f(a, b=1, c=2): + ... return a + b + c + ... + >>> raw_kwargs = dict(a=1, b=3, d=4) + >>> f(**raw_kwargs) + Traceback (most recent call last): + ... + TypeError: f() got an unexpected keyword argument 'd' + >>> kwargs = filter_kwargs(f, raw_kwargs) + >>> f(**kwargs) + 6 + + Notes + ----- + Taken from odo.utils + """ + return keyfilter(op.contains(keywords(f)), kwargs) def _s(word, seq, suffix='s'): @@ -41,8 +123,26 @@ def _fmt_path(path): return 'path: _' + ''.join(path) +def _fmt_msg(msg): + """Format the message for final display. + + Parameters + ---------- + msg : str + The message to show to the user to provide additional context. + + returns + ------- + fmtd : str + The formatted message to put into the error message. + """ + if not msg: + return '' + return msg + '\n' + + @dispatch(object, object) -def assert_equal(result, expected, path=(), **kwargs): +def assert_equal(result, expected, path=(), msg='', **kwargs): """Assert that two objects are equal using the ``==`` operator. Parameters @@ -57,15 +157,42 @@ def assert_equal(result, expected, path=(), **kwargs): AssertionError Raised when ``result`` is not equal to ``expected``. """ - assert result == expected, '%s != %s\n%s' % ( + assert result == expected, '%s%s != %s\n%s' % ( + _fmt_msg(msg), result, expected, _fmt_path(path), ) +@assert_equal.register(float, float) +def assert_float_equal(result, + expected, + path=(), + msg='', + float_rtol=10e-7, + float_atol=10e-7, + float_equal_nan=True, + **kwargs): + assert tolerant_equals( + result, + expected, + rtol=float_rtol, + atol=float_atol, + equal_nan=float_equal_nan, + ), '%s%s != %s with rtol=%s and atol=%s%s\n%s' % ( + _fmt_msg(msg), + result, + expected, + float_rtol, + float_atol, + (' (with nan != nan)' if not float_equal_nan else ''), + _fmt_path(path), + ) + + @assert_equal.register(dict, dict) -def assert_dict_equal(result, expected, path=(), **kwargs): +def assert_dict_equal(result, expected, path=(), msg='', **kwargs): if path is None: path = () @@ -89,8 +216,8 @@ def assert_dict_equal(result, expected, path=(), **kwargs): in_expected, ) raise AssertionError( - 'dict keys do not match\n%s\n%s' % ( - msg, + '%sdict keys do not match\n%s' % ( + _fmt_msg(msg), _fmt_path(path + ('.%s()' % ('viewkeys' if PY2 else 'keys'),)), ), ) @@ -102,6 +229,7 @@ def assert_dict_equal(result, expected, path=(), **kwargs): resultv, expectedv, path=path + ('[%r]' % k,), + msg=msg, **kwargs ) except AssertionError as e: @@ -112,14 +240,16 @@ def assert_dict_equal(result, expected, path=(), **kwargs): @assert_equal.register(list, list) # noqa -def assert_list_equal(result, expected, path=(), **kwargs): +def assert_list_equal(result, expected, path=(), msg='', **kwargs): result_len = len(result) expected_len = len(expected) assert result_len == expected_len, ( - 'list lengths do not match: %d != %d\n%s' % - result_len, - expected_len, - _fmt_path(path), + '%slist lengths do not match: %d != %d\n%s' % ( + _fmt_msg(msg), + result_len, + expected_len, + _fmt_path(path), + ) ) for n, (resultv, expectedv) in enumerate(zip(result, expected)): @@ -127,6 +257,56 @@ def assert_list_equal(result, expected, path=(), **kwargs): resultv, expectedv, path=path + ('[%d]' % n,), + msg=msg, + **kwargs + ) + + +@assert_equal.register(np.ndarray, np.ndarray) +def assert_array_equal(result, + expected, + path=(), + msg='', + array_verbose=True, + array_decimal=None, + **kwargs): + f = ( + np.testing.assert_array_equal + if array_decimal is None else + partial(np.testing.assert_array_almost_equal, decimal=array_decimal) + ) + try: + f( + result, + expected, + verbose=array_verbose, + err_msg=msg, + ) + except AssertionError as e: + raise AssertionError('\n'.join((str(e), _fmt_path(path)))) + + +@assert_equal.register(pd.DataFrame, pd.DataFrame) +def assert_dataframe_equal(result, expected, path=(), msg='', **kwargs): + try: + assert_frame_equal( + result, + expected, + **filter_kwargs(assert_frame_equal, kwargs) + ) + except AssertionError as e: + raise AssertionError( + _fmt_msg(msg) + '\n'.join((str(e), _fmt_path(path))), + ) + + +@assert_equal.register(Adjustment, Adjustment) +def assert_adjustment_equal(result, expected, path=(), **kwargs): + for attr in ('first_row', 'last_row', 'first_col', 'last_col', 'value'): + assert_equal( + getattr(result, attr), + getattr(expected, attr), + path=path + ('.' + attr,), **kwargs ) @@ -137,4 +317,6 @@ try: except ImportError: pass else: - assert_equal.funcs.update(assert_dshape_equal.funcs) + assert_equal.funcs.update( + dissoc(assert_dshape_equal.funcs, (object, object)), + ) diff --git a/zipline/utils/__init__.py b/zipline/utils/__init__.py index fbdf09d2..e69de29b 100644 --- a/zipline/utils/__init__.py +++ b/zipline/utils/__init__.py @@ -1,18 +0,0 @@ -# -# Copyright 2014 Quantopian, Inc. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -from .cli import run_pipeline, parse_args, parse_cell_magic - -__all__ = ['run_pipeline', 'parse_args', 'parse_cell_magic'] diff --git a/zipline/utils/cache.py b/zipline/utils/cache.py index 657f17ed..93833b09 100644 --- a/zipline/utils/cache.py +++ b/zipline/utils/cache.py @@ -1,11 +1,22 @@ """ Caching utilities for zipline """ -from collections import namedtuple +from collections import namedtuple, MutableMapping +import errno +import os +import pickle +from shutil import rmtree, copyfile, copytree +from tempfile import mkdtemp, NamedTemporaryFile + +import pandas as pd + +from .context_tricks import nop_context +from .paths import ensure_directory class Expired(Exception): - pass + """Marks that a :class:`CachedObject` has expired. + """ class CachedObject(namedtuple("_CachedObject", "value expires")): @@ -20,11 +31,6 @@ class CachedObject(namedtuple("_CachedObject", "value expires")): Expiration date of `value`. The cache is considered invalid for dates **strictly greater** than `expires`. - Methods - ------- - get(self, dt) - Get the cached object. - Usage ----- >>> from pandas import Timestamp, Timedelta @@ -66,22 +72,11 @@ class ExpiringCache(object): Parameters ---------- - cache : dict-like + cache : dict-like, optional An instance of a dict-like object which needs to support at least: `__del__`, `__getitem__`, `__setitem__` If `None`, than a dict is used as a default. - Methods - ------- - get(self, key, dt) - Get the value of a cached object for the given `key` at `dt`, if the - CachedObject has expired then the object is removed from the cache, - and `KeyError` is raised. - - set(self, key, value, expiration_dt) - Add a new `value` to the cache at `dt` wrapped in a CachedObject which - expires at `expiration_dt`. - Usage ----- >>> from pandas import Timestamp, Timedelta @@ -104,6 +99,26 @@ class ExpiringCache(object): self._cache = {} def get(self, key, dt): + """Get the value of a cached object. + + Parameters + ---------- + key : any + The key to lookup. + dt : datetime + The time of the lookup. + + Returns + ------- + result : any + The value for ``key``. + + Raises + ------ + KeyError + Raised if the key is not in the cache or the value for the key + has expired. + """ try: return self._cache[key].unwrap(dt) except Expired: @@ -111,4 +126,208 @@ class ExpiringCache(object): raise KeyError(key) def set(self, key, value, expiration_dt): + """Adds a new key value pair to the cache. + + Parameters + ---------- + key : any + The key to use for the pair. + value : any + The value to store under the name ``key``. + expiration_dt : datetime + When should this mapping expire? The cache is considered invalid + for dates **strictly greater** than ``expiration_dt``. + """ self._cache[key] = CachedObject(value, expiration_dt) + + +class dataframe_cache(MutableMapping): + """A disk-backed cache for dataframes. + + ``dataframe_cache`` is a mutable mapping from string names to pandas + DataFrame objects. + This object may be used as a context manager to delete the cache directory + on exit. + + Parameters + ---------- + path : str, optional + The directory path to the cache. Files will be written as + ``path/``. + lock : Lock, optional + Thread lock for multithreaded/multiprocessed access to the cache. + If not provided no locking will be used. + clean_on_failure : bool, optional + Should the directory be cleaned up if an exception is raised in the + context manager. + serialize : {'msgpack', 'pickle:'}, optional + How should the data be serialized. If ``'pickle'`` is passed, an + optional pickle protocol can be passed like: ``'pickle:3'`` which says + to use pickle protocol 3. + + Notes + ----- + The syntax ``cache[:]`` will load all key:value pairs into memory as a + dictionary. + The cache uses a temporary file format that is subject to change between + versions of zipline. + """ + def __init__(self, + path=None, + lock=None, + clean_on_failure=True, + serialization='msgpack'): + self.path = path if path is not None else mkdtemp() + self.lock = lock if lock is not None else nop_context + self.clean_on_failure = clean_on_failure + + if serialization == 'msgpack': + self.serialize = pd.DataFrame.to_msgpack + self.deserialize = pd.read_msgpack + self._protocol = None + else: + s = serialization.split(':', 1) + if s[0] != 'pickle': + raise ValueError( + "'serialization' must be either 'msgpack' or 'pickle[:n]'", + ) + self._protocol = int(s[1]) if len(s) == 2 else None + + self.serialize = self._serialize_pickle + self.deserialize = self._deserialize_pickle + + ensure_directory(self.path) + + def _serialize_pickle(self, df, path): + with open(path, 'wb') as f: + pickle.dump(df, f, protocol=self._protocol) + + def _deserialize_pickle(self, path): + with open(path, 'rb') as f: + return pickle.load(f) + + def _keypath(self, key): + return os.path.join(self.path, key) + + def __enter__(self): + return self + + def __exit__(self, type_, value, tb): + if not (self.clean_on_failure or value is None): + # we are not cleaning up after a failure and there was an exception + return + + with self.lock: + rmtree(self.path) + + def __getitem__(self, key): + if key == slice(None): + return dict(self.items()) + + with self.lock: + try: + return self.deserialize(self._keypath(key)) + except UnboundLocalError: + # This is how pandas fails if the file doesn't exist! #pandas + raise KeyError(key) + + def __setitem__(self, key, value): + with self.lock: + self.serialize(value, self._keypath(key)) + + def __delitem__(self, key): + with self.lock: + try: + os.remove(self._keypath(key)) + except OSError as e: + if e.errno == errno.ENOENT: + # raise a keyerror if this directory did not exist + raise KeyError(key) + # reraise the actual oserror otherwise + raise + + def __iter__(self): + return iter(os.listdir(self.path)) + + def __len__(self): + return len(os.listdir(self.path)) + + def __repr__(self): + return '<%s: keys={%s}>' % ( + type(self).__name__, + ', '.join(map(repr, sorted(self))), + ) + + +class working_file(object): + """A context manager for managing a temporary file that will be moved + to a non-temporary location if no exceptions are raised in the context. + + Parameters + ---------- + final_path : str + The location to move the file when committing. + *args, **kwargs + Forwarded to NamedTemporaryFile. + + Notes + ----- + The file is moved on __exit__ if there are no exceptions. + ``working_file`` uses :func:`shutil.copyfile` to move the actual files, + meaning it has as strong of guarantees as :func:`shutil.copyfile`. + """ + def __init__(self, final_path, *args, **kwargs): + self._tmpfile = NamedTemporaryFile(*args, **kwargs) + self._final_path = final_path + + def _commit(self): + """Sync the temporary file to the final path. + """ + copyfile(self.name, self._final_path) + + def __getattr__(self, attr): + return getattr(self._tmpfile, attr) + + def __enter__(self): + self._tmpfile.__enter__() + return self + + def __exit__(self, *exc_info): + if exc_info[0] is None: + self._commit() + self._tmpfile.__exit__(*exc_info) + + +class working_dir(object): + """A context manager for managing a temporary directory that will be moved + to a non-temporary location if no exceptions are raised in the context. + + Parameters + ---------- + final_path : str + The location to move the file when committing. + *args, **kwargs + Forwarded to tmp_dir. + + Notes + ----- + The file is moved on __exit__ if there are no exceptions. + ``working_dir`` uses :func:`shutil.copytree` to move the actual files, + meaning it has as strong of guarantees as :func:`shutil.copytree`. + """ + def __init__(self, final_path, *args, **kwargs): + self.name = mkdtemp() + self._final_path = final_path + + def _commit(self): + """Sync the temporary directory to the final path. + """ + copytree(self.name, self._final_path) + + def __enter__(self): + return self + + def __exit__(self, *exc_info): + if exc_info[0] is None: + self._commit() + rmtree(self.name) diff --git a/zipline/utils/cli.py b/zipline/utils/cli.py index 4ff2138c..70e15d35 100644 --- a/zipline/utils/cli.py +++ b/zipline/utils/cli.py @@ -1,259 +1,8 @@ -# -# Copyright 2014 Quantopian, Inc. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -import sys -import os -import argparse -from copy import copy - import click -from six import print_ -from six.moves import configparser import pandas as pd -try: - from pygments import highlight - from pygments.lexers import PythonLexer - from pygments.formatters import TerminalFormatter - PYGMENTS = True -except: - PYGMENTS = False - -import zipline -from zipline.errors import NoSourceError, PipelineDateError from .context_tricks import CallbackManager -DEFAULTS = { - 'data_frequency': 'daily', - 'capital_base': '10e6', - 'source': 'yahoo', - 'symbols': 'AAPL', - 'metadata_index': 'symbol', - 'source_time_column': 'Date', -} - - -def parse_args(argv, ipython_mode=False): - """Parse list of arguments. - - If a config file is provided (via -c), it will read in the - supplied options and overwrite any global defaults. - - All other directly supplied arguments will overwrite the config - file settings. - - Arguments: - * argv : list of strings - List of arguments, e.g. ['-c', 'my.conf'] - * ipython_mode : bool - Whether to parse IPython specific arguments - like --local_namespace - - Notes: - Default settings can be found in zipline.utils.cli.DEFAULTS. - - """ - # Parse any conf_file specification - # We make this parser with add_help=False so that - # it doesn't parse -h and print help. - conf_parser = argparse.ArgumentParser( - # Don't mess with format of description - formatter_class=argparse.RawDescriptionHelpFormatter, - # Turn off help, so we print all options in response to -h - add_help=False - ) - conf_parser.add_argument("-c", "--conf_file", - help="Specify config file", - metavar="FILE") - args, remaining_argv = conf_parser.parse_known_args(argv) - - defaults = copy(DEFAULTS) - - if args.conf_file: - config = configparser.SafeConfigParser() - config.read([args.conf_file]) - defaults.update(dict(config.items("Defaults"))) - - # Parse rest of arguments - # Don't suppress add_help here so it will handle -h - parser = argparse.ArgumentParser( - # Inherit options from config_parser - description="Zipline version %s." % zipline.__version__, - parents=[conf_parser] - ) - - parser.set_defaults(**defaults) - - parser.add_argument('--algofile', '-f') - parser.add_argument('--data-frequency', - choices=('minute', 'daily')) - parser.add_argument('--start', '-s') - parser.add_argument('--end', '-e') - parser.add_argument('--capital_base') - parser.add_argument('--source', '-d', choices=('yahoo',)) - parser.add_argument('--source_time_column', '-t') - parser.add_argument('--symbols') - parser.add_argument('--output', '-o') - parser.add_argument('--metadata_path', '-m') - parser.add_argument('--metadata_index', '-x') - parser.add_argument('--print-algo', '-p', dest='print_algo', - action='store_true') - parser.add_argument('--no-print-algo', '-q', dest='print_algo', - action='store_false') - - if ipython_mode: - parser.add_argument('--local_namespace', action='store_true') - - args = parser.parse_args(remaining_argv) - - return(vars(args)) - - -def parse_cell_magic(line, cell): - """Parse IPython magic - """ - args_list = line.split(' ') - args = parse_args(args_list, ipython_mode=True) - - # Remove print_algo kwarg to overwrite below. - args.pop('print_algo') - - local_namespace = args.pop('local_namespace', False) - # By default, execute inside IPython namespace - if not local_namespace: - args['namespace'] = get_ipython().user_ns # flake8: noqa - - # If we are running inside NB, do not output to file but create a - # variable instead - output_var_name = args.pop('output', None) - - perf = run_pipeline(print_algo=False, algo_text=cell, **args) - - if output_var_name is not None: - get_ipython().user_ns[output_var_name] = perf # flake8: noqa - - -def run_pipeline(print_algo=True, **kwargs): - """Runs a full zipline pipeline given configuration keyword - arguments. - - 1. Load data (start and end dates can be provided a strings as - well as the source and symobls). - - 2. Instantiate algorithm (supply either algo_text or algofile - kwargs containing initialize() and handle_data() functions). If - algofile is supplied, will try to look for algofile_analyze.py and - append it. - - 3. Run algorithm (supply capital_base as float). - - 4. Return performance dataframe. - - :Arguments: - * print_algo : bool - Whether to print the algorithm to command line. Will use - pygments syntax coloring if pygments is found. - - """ - start = kwargs['start'] - end = kwargs['end'] - # Compare against None because strings/timestamps may have been given - if start is not None: - start = pd.Timestamp(start, tz='UTC') - if end is not None: - end = pd.Timestamp(end, tz='UTC') - - # Fail out if only one bound is provided - if ((start is None) or (end is None)) and (start != end): - raise PipelineDateError(start=start, end=end) - - # Check if start and end are provided, and if the sim_params need to read - # a start and end from the DataSource - if start is None: - overwrite_sim_params = True - else: - overwrite_sim_params = False - - symbols = kwargs['symbols'].split(',') - asset_identifier = kwargs['metadata_index'] - - # Pull asset metadata - asset_metadata = kwargs.get('asset_metadata', None) - asset_metadata_path = kwargs['metadata_path'] - # Read in a CSV file, if applicable - if asset_metadata_path is not None: - if os.path.isfile(asset_metadata_path): - asset_metadata = pd.read_csv(asset_metadata_path, - index_col=asset_identifier) - - source_arg = kwargs['source'] - source_time_column = kwargs['source_time_column'] - - if source_arg is None: - raise NoSourceError() - - elif source_arg == 'yahoo': - source = zipline.data.load_bars_from_yahoo( - stocks=symbols, start=start, end=end) - - elif os.path.isfile(source_arg): - source = zipline.data.load_prices_from_csv( - filepath=source_arg, - identifier_col=source_time_column - ) - - elif os.path.isdir(source_arg): - source = zipline.data.load_prices_from_csv_folder( - folderpath=source_arg, - identifier_col=source_time_column - ) - - else: - raise NotImplementedError( - 'Source %s not implemented.' % kwargs['source']) - - algo_text = kwargs.get('algo_text', None) - if algo_text is None: - # Expect algofile to be set - algo_fname = kwargs['algofile'] - with open(algo_fname, 'r') as fd: - algo_text = fd.read() - - if print_algo: - if PYGMENTS: - highlight(algo_text, PythonLexer(), TerminalFormatter(), - outfile=sys.stdout) - else: - print_(algo_text) - - algo = zipline.TradingAlgorithm(script=algo_text, - namespace=kwargs.get('namespace', {}), - capital_base=float(kwargs['capital_base']), - algo_filename=kwargs.get('algofile'), - equities_metadata=asset_metadata, - start=start, - end=end) - - perf = algo.run(source, overwrite_sim_params=overwrite_sim_params) - - output_fname = kwargs.get('output', None) - if output_fname is not None: - perf.to_pickle(output_fname) - - return perf - def maybe_show_progress(it, show_progress, **kwargs): """Optionally show a progress bar for the given iterator. @@ -274,12 +23,96 @@ def maybe_show_progress(it, show_progress, **kwargs): Examples -------- - with maybe_show_progress([1, 2, 3], True) as ns: - for n in ns: - ... + .. code-block:: python + + with maybe_show_progress([1, 2, 3], True) as ns: + for n in ns: + ... """ if show_progress: return click.progressbar(it, **kwargs) # context manager that just return `it` when we enter it return CallbackManager(lambda it=it: it) + + +class _DatetimeParam(click.ParamType): + def __init__(self, tz=None): + self.tz = tz + + def parser(self, value): + return pd.Timestamp(value, tz=self.tz) + + @property + def name(self): + return type(self).__name__.upper() + + def convert(self, value, param, ctx): + try: + return self.parser(value) + except ValueError: + self.fail( + '%s is not a valid %s' % (value, self.name.lower()), + param, + ctx, + ) + + +class Timestamp(_DatetimeParam): + """A click parameter that parses the value into pandas.Timestamp objects. + + Parameters + ---------- + tz : timezone-coercable, optional + The timezone to parse the string as. + By default the timezone will be infered from the string or naiive. + """ + + +class Date(_DatetimeParam): + """A click parameter that parses the value into datetime.date objects. + + Parameters + ---------- + tz : timezone-coercable, optional + The timezone to parse the string as. + By default the timezone will be infered from the string or naiive. + as_timestamp : bool, optional + If True, return the value as a pd.Timestamp object normalized to + midnight. + """ + def __init__(self, tz=None, as_timestamp=False): + super(Date, self).__init__(tz=tz) + self.as_timestamp = as_timestamp + + def parser(self, value): + ts = super(Date, self).parser(value) + return ts.normalize() if self.as_timestamp else ts.date() + + +class Time(_DatetimeParam): + """A click parameter that parses the value into timetime.time objects. + + Parameters + ---------- + tz : timezone-coercable, optional + The timezone to parse the string as. + By default the timezone will be infered from the string or naiive. + """ + def parser(self, value): + return super(Time, self).parser(value).time() + + +class Timedelta(_DatetimeParam): + """A click parameter that parses values into pd.Timedelta objects. + + Parameters + ---------- + unit : {'D', 'h', 'm', 's', 'ms', 'us', 'ns'}, optional + Denotes the unit of the input if the input is an integer. + """ + def __init__(self, unit='ns'): + self.unit = unit + + def parser(self, value): + return pd.Timedelta(value, unit=self.unit) diff --git a/zipline/utils/compat.py b/zipline/utils/compat.py index fff02e6b..62ed44ef 100644 --- a/zipline/utils/compat.py +++ b/zipline/utils/compat.py @@ -2,7 +2,6 @@ from six import PY2 if PY2: - from functools32 import lru_cache from ctypes import py_object, pythonapi mappingproxy = pythonapi.PyDictProxy_New @@ -10,10 +9,8 @@ if PY2: mappingproxy.restype = py_object else: - from functools import lru_cache from types import MappingProxyType as mappingproxy __all__ = [ - 'lru_cache', 'mappingproxy', ] diff --git a/zipline/utils/input_validation.py b/zipline/utils/input_validation.py index cf0696c1..66e8b9fe 100644 --- a/zipline/utils/input_validation.py +++ b/zipline/utils/input_validation.py @@ -16,6 +16,7 @@ from functools import partial, wraps from operator import attrgetter from numpy import dtype +import pandas as pd from pytz import timezone from six import iteritems, string_types, PY3 from toolz import valmap, complement, compose @@ -133,6 +134,35 @@ def ensure_timezone(func, argname, arg): ) +def ensure_timestamp(func, argname, arg): + """Argument preprocessor that converts the input into a pandas Timestamp + object. + + Usage + ----- + >>> from zipline.utils.preprocess import preprocess + >>> @preprocess(ts=ensure_timestamp) + ... def foo(ts): + ... return ts + >>> foo('2014-01-01') + Timestamp('2014-01-01 00:00:00') + """ + try: + return pd.Timestamp(arg) + except ValueError as e: + raise TypeError( + "{func}() couldn't convert argument " + "{argname}={arg!r} to a pandas Timestamp.\n" + "Original error was: {t}: {e}".format( + func=_qualified_name(func), + argname=argname, + arg=arg, + t=_qualified_name(type(e)), + e=e, + ), + ) + + def expect_dtypes(*_pos, **named): """ Preprocessing decorator that verifies inputs have expected numpy dtypes. diff --git a/zipline/utils/math_utils.py b/zipline/utils/math_utils.py index 590f407e..da99900d 100644 --- a/zipline/utils/math_utils.py +++ b/zipline/utils/math_utils.py @@ -12,11 +12,37 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. - import math +from numpy import isnan -def tolerant_equals(a, b, atol=10e-7, rtol=10e-7): + +def tolerant_equals(a, b, atol=10e-7, rtol=10e-7, equal_nan=False): + """Check if a and b are equal with some tolerance. + + Parameters + ---------- + a, b : float + The floats to check for equality. + atol : float, optional + The absolute tolerance. + rtol : float, optional + The relative tolerance. + equal_nan : bool, optional + Should NaN compare equal? + + See Also + -------- + numpy.isclose + + Notes + ----- + This function is just a scalar version of numpy.isclose for performance. + See the docstring of ``isclose`` for more information about ``atol`` and + ``rtol``. + """ + if equal_nan and isnan(a) and isnan(b): + return True return math.fabs(a - b) <= (atol + rtol * math.fabs(b)) diff --git a/zipline/utils/paths.py b/zipline/utils/paths.py new file mode 100644 index 00000000..103c116b --- /dev/null +++ b/zipline/utils/paths.py @@ -0,0 +1,223 @@ +""" +Canonical path locations for zipline data. + +Paths are rooted at $ZIPLINE_ROOT if that environment variable is set. +Otherwise default to expanduser(~/.zipline) +""" +from errno import EEXIST +import os +from os.path import exists, expanduser, join + +import pandas as pd + + +def hidden(path): + """Check if a path is hidden. + + Parameters + ---------- + path : str + A filepath. + """ + return path.startswith('.') + + +def ensure_directory(path): + """ + Ensure that a directory named "path" exists. + """ + try: + os.makedirs(path) + except OSError as exc: + if exc.errno == EEXIST and os.path.isdir(path): + return + raise + + +def ensure_directory_containing(path): + """ + Ensure that the directory containing `path` exists. + + This is just a convenience wrapper for doing:: + + ensure_directory(os.path.dirname(path)) + """ + ensure_directory(os.path.dirname(path)) + + +def last_modified_time(path): + """ + Get the last modified time of path as a Timestamp. + """ + return pd.Timestamp(os.path.getmtime(path), unit='s', tz='UTC') + + +def modified_since(path, dt): + """ + Check whether `path` was modified since `dt`. + + Returns False if path doesn't exist. + + Parameters + ---------- + path : str + Path to the file to be checked. + dt : pd.Timestamp + The date against which to compare last_modified_time(path). + + Returns + ------- + was_modified : bool + Will be ``False`` if path doesn't exists, or if its last modified date + is earlier than or equal to `dt` + """ + return exists(path) and last_modified_time(path) > dt + + +def zipline_root(environ=None): + """ + Get the root directory for all zipline-managed files. + + For testing purposes, this accepts a dictionary to interpret as the os + environment. + + Parameters + ---------- + environ : dict, optional + A dict to interpret as the os environment. + + Returns + ------- + root : string + Path to the zipline root dir. + """ + if environ is None: + environ = os.environ + + root = environ.get('ZIPLINE_ROOT', None) + if root is None: + root = expanduser('~/.zipline') + + return root + + +def zipline_path(paths, environ=None): + """ + Get a path relative to the zipline root. + + Parameters + ---------- + paths : list[str] + List of requested path pieces. + environ : dict, optional + An environment dict to forward to zipline_root. + + Returns + ------- + newpath : str + The requested path joined with the zipline root. + """ + return join(zipline_root(environ=environ), *paths) + + +def default_extension(environ=None): + """ + Get the path to the default zipline extension file. + + Parameters + ---------- + environ : dict, optional + An environment dict to forwart to zipline_root. + + Returns + ------- + default_extension_path : str + The file path to the default zipline extension file. + """ + return zipline_path(['extension.py'], environ=environ) + + +def data_root(environ=None): + """ + The root directory for zipline data files. + + Parameters + ---------- + environ : dict, optional + An environment dict to forward to zipline_root. + + Returns + ------- + data_root : str + The zipline data root. + """ + return zipline_path(['data'], environ=environ) + + +def ensure_data_root(environ=None): + """ + Ensure that the data root exists. + """ + ensure_directory(data_root(environ=environ)) + + +def data_path(paths, environ=None): + """ + Get a path relative to the zipline data directory. + + Parameters + ---------- + paths : iterable[str] + List of requested path pieces. + environ : dict, optional + An environment dict to forward to zipline_root. + + Returns + ------- + newpath : str + The requested path joined with the zipline data root. + """ + return zipline_path(['data'] + list(paths), environ=environ) + + +def cache_root(environ=None): + """ + The root directory for zipline cache files. + + Parameters + ---------- + environ : dict, optional + An environment dict to forward to zipline_root. + + Returns + ------- + cache_root : str + The zipline cache root. + """ + return zipline_path(['cache'], environ=environ) + + +def ensure_cache_root(environ=None): + """ + Ensure that the data root exists. + """ + ensure_directory(cache_root(environ=environ)) + + +def cache_path(paths, environ=None): + """ + Get a path relative to the zipline cache directory. + + Parameters + ---------- + paths : iterable[str] + List of requested path pieces. + environ : dict, optional + An environment dict to forward to zipline_root. + + Returns + ------- + newpath : str + The requested path joined with the zipline cache root. + """ + return zipline_path(['cache'] + list(paths), environ=environ) diff --git a/zipline/utils/run_algo.py b/zipline/utils/run_algo.py new file mode 100644 index 00000000..09dbd57e --- /dev/null +++ b/zipline/utils/run_algo.py @@ -0,0 +1,335 @@ +import os +import re +from runpy import run_path +import sys +import warnings + +import click +try: + from pygments import highlight + from pygments.lexers import PythonLexer + from pygments.formatters import TerminalFormatter + PYGMENTS = True +except: + PYGMENTS = False +from toolz import valfilter, concatv + +from zipline.algorithm import TradingAlgorithm +from zipline.data.bundles.core import load +from zipline.data.data_portal import DataPortal +from zipline.finance.trading import TradingEnvironment +import zipline.utils.paths as pth + + +class _RunAlgoError(click.ClickException, ValueError): + """Signal an error that should have a different message if invoked from + the cli. + + Parameters + ---------- + pyfunc_msg : str + The message that will be shown when called as a python function. + cmdline_msg : str + The message that will be shown on the command line. + """ + exit_code = 1 + + def __init__(self, pyfunc_msg, cmdline_msg): + super(_RunAlgoError, self).__init__(cmdline_msg) + self.pyfunc_msg = pyfunc_msg + + def __str__(self): + return self.pyfunc_msg + + +def _run(handle_data, + initialize, + before_trading_start, + analyze, + algofile, + algotext, + defines, + data_frequency, + capital_base, + data, + bundle, + bundle_timestamp, + start, + end, + output, + print_algo, + local_namespace, + environ): + """Run a backtest for the given algorithm. + + This is shared between the cli and :func:`zipline.run_algo`. + """ + if algotext is not None: + if local_namespace: + ip = get_ipython() # noqa + namespace = ip.user_ns + else: + namespace = {} + + for assign in defines: + try: + name, value = assign.split('=', 2) + except ValueError: + raise ValueError( + 'invalid define %r, should be of the form name=value' % + assign, + ) + try: + # evaluate in the same namespace so names may refer to + # eachother + namespace[name] = eval(value, namespace) + except Exception as e: + raise ValueError( + 'failed to execute definition for name %r: %s' % (name, e), + ) + elif defines: + raise _RunAlgoError( + 'cannot pass define without `algotext`', + "cannot pass '-D' / '--define' without '-t' / '--algotext'", + ) + else: + namespace = {} + if algofile is not None: + algotext = algofile.read() + + if print_algo: + if PYGMENTS: + highlight( + algotext, + PythonLexer(), + TerminalFormatter(), + outfile=sys.stdout, + ) + else: + click.echo(algotext) + + if bundle is not None: + bundle_data = load( + bundle, + environ, + bundle_timestamp, + ) + + prefix, connstr = re.split( + r'sqlite:///', + str(bundle_data.asset_finder.engine.url), + maxsplit=1, + ) + if prefix: + raise ValueError( + "invalid url %r, must begin with 'sqlite:///'" % + str(bundle_data.asset_finder.engine.url), + ) + env = TradingEnvironment(asset_db_path=connstr) + data = DataPortal( + env, + equity_minute_reader=bundle_data.minute_bar_reader, + equity_daily_reader=bundle_data.daily_bar_reader, + adjustment_reader=bundle_data.adjustment_reader, + ) + + perf = TradingAlgorithm( + namespace=namespace, + capital_base=capital_base, + start=start, + end=end, + env=env, + **{ + 'initialize': initialize, + 'handle_data': handle_data, + 'before_trading_start': before_trading_start, + 'analyze': analyze, + } if algotext is None else { + 'algo_filename': algofile, + 'script': algotext, + } + ).run( + data, + overwrite_sim_params=False, + ) + + if output == '-': + click.echo(str(perf)) + elif output != os.devnull: # make the zipline magic not write any data + perf.to_pickle(output) + + return perf + + +# All of the loaded extensions. We don't want to load an extension twice. +_loaded_extensions = set() + + +def load_extensions(default, extensions, strict, environ, reload=False): + """Load all of the given extensions. This should be called by run_algo + or the cli. + + Parameters + ---------- + default : bool + Load the default exension (~/.zipline/extension.py)? + extension : iterable[str] + The paths to the extensions to load. If the path ends in ``.py`` it is + treated as a script and executed. If it does not end in ``.py`` it is + treated as a module to be imported. + strict : bool + Should failure to load an extension raise. If this is false it will + still warn. + environ : mapping + The environment to use to find the default extension path. + reload : bool, optional + Reload any extensions that have already been loaded. + """ + if default: + default_extension_path = pth.default_extension(environ=environ) + open(default_extension_path, 'a+').close() # touch the file + # put the default extension first so other extensions can depend on + # the order they are loaded + extensions = concatv([default_extension_path], extensions) + + for ext in extensions: + if ext in _loaded_extensions and not reload: + continue + try: + # load all of the zipline extensionss + if ext.endswith('.py'): + run_path(ext, run_name='') + else: + __import__(ext) + except Exception as e: + if strict: + # if `strict` we should raise the actual exception and fail + raise + # without `strict` we should just log the failure + warnings.warn( + 'Failed to load extension: %r\n%s' % (ext, e), + stacklevel=2 + ) + else: + _loaded_extensions.add(ext) + + +def run_algorithm(start, + end, + initialize, + capital_base, + handle_data=None, + before_trading_start=None, + analyze=None, + data_frequency='daily', + data=None, + bundle=None, + bundle_timestamp=None, + default_extension=True, + extensions=(), + strict_extensions=True, + environ=os.environ): + """Run a trading algorithm. + + Parameters + ---------- + start : datetime + The start date of the backtest. + end : datetime + The end date of the backtest.. + initialize : callable[context -> None] + The initialize function to use for the algorithm. This is called once + at the very begining of the backtest and should be used to set up + any state needed by the algorithm. + capital_base : float + The starting capital for the backtest. + handle_data : callable[(context, BarData) -> None], optional + The handle_data function to use for the algorithm. This is called + every minute when ``data_frequency == 'minute'`` or every day + when ``data_frequency == 'daily'``. + before_trading_start : callable[(context, BarData) -> None], optional + The before_trading_start function for the algorithm. This is called + once before each trading day (after initialize on the first day). + analyze : callable[(context, pd.DataFrame) -> None], optional + The analyze function to use for the algorithm. This function is called + once at the end of the backtest and is passed the context and the + performance data. + data_frequency : {'daily', 'minute'}, optional + The data frequency to run the algorithm at. + data : pd.DataFrame, pd.Panel, or DataPortal, optional + The ohlcv data to run the backtest with. + This argument is mutually exclusive with: + ``bundle`` + ``bundle_timestamp`` + bundle : str, optional + The name of the data bundle to use to load the data to run the backtest + with. This defaults to 'quandl'. + This argument is mutually exclusive with ``data``. + bundle_timestamp : datetime, optional + The datetime to lookup the bundle data for. This defaults to the + current time. + This argument is mutually exclusive with ``data``. + default_extension : bool, optional + Should the default zipline extension be loaded. This is found at + ``$ZIPLINE_ROOT/extension.py`` + extensions : iterable[str], optional + The names of any other extensions to load. Each element may either be + a dotted module path like ``a.b.c`` or a path to a python file ending + in ``.py`` like ``a/b/c.py``. + strict_extensions : bool, optional + Should the run fail if any extensions fail to load. If this is false, + a warning will be raised instead. + environ : mapping[str -> str], optional + The os environment to use. Many extensions use this to get parameters. + This defaults to ``os.environ``. + + Returns + ------- + perf : pd.DataFrame + The daily performance of the algorithm. + + See Also + -------- + zipline.data.bundles.bundles : The available data bundles. + """ + load_extensions(default_extension, extensions, strict_extensions, environ) + + non_none_data = valfilter(bool, { + 'data': data, + 'bundle': bundle, + }) + if not non_none_data: + # if neither data nor bundle are passed use 'quandl' + bundle = 'quandl' + + if len(non_none_data) != 1: + raise ValueError( + 'must specify one of `data`, `data_portal`, or `bundle`,' + ' got: %r' % non_none_data, + ) + + if 'bundle' not in non_none_data and bundle_timestamp is not None: + raise ValueError( + 'cannot specify `bundle_timestamp` without passing `bundle`', + ) + + return _run( + handle_data=handle_data, + initialize=initialize, + before_trading_start=before_trading_start, + analyze=analyze, + algofile=None, + algotext=None, + defines=(), + data_frequency=data_frequency, + capital_base=capital_base, + data=data, + bundle=bundle, + bundle_timestamp=bundle_timestamp, + start=start, + end=end, + output=os.devnull, + print_algo=False, + local_namespace=False, + environ=environ, + )