From a26802efd25e020d78d4b0a1c61c0ca570337612 Mon Sep 17 00:00:00 2001 From: Joe Jevnik Date: Thu, 5 May 2016 17:06:49 -0400 Subject: [PATCH] DOC: Update docs for bundles and fix the whatsnew --- docs/source/bundles.rst | 149 +++++++++++++++++++++------------ docs/source/whatsnew/1.0.0.txt | 23 ++--- 2 files changed, 107 insertions(+), 65 deletions(-) diff --git a/docs/source/bundles.rst b/docs/source/bundles.rst index dede1031..f6d5a32b 100644 --- a/docs/source/bundles.rst +++ b/docs/source/bundles.rst @@ -1,3 +1,5 @@ +.. _data-bundles: + Data Bundles ------------ @@ -5,71 +7,107 @@ A data bundle is a collection of pricing data, adjustment data, and an asset database. Bundles allow us to preload all of the data we will need to run backtests and store the data for future runs. -Ingesting Data -~~~~~~~~~~~~~~ +.. _bundles-command: -The first step to using a data bundle is to ingest the data. This will invoke -some custom bundle command and then write the data to a standard location that -zipline can find. By default this location is ``$ZIPLINE_ROOT/data/`` -where by default ``ZIPLINE_ROOT=~/.zipline``. This step may take some time as it -could involve downloading and processing a lot of data. This can be run with: +Discovering Available Bundles +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Zipline comes with a few bundles by default as well as the ability to register +new bundles. To see which bundles we have have available, we may run the +``bundles`` command, for example: .. code-block:: bash - $ python -m zipline ingest [-b ] + $ zipline bundles + my-custom-bundle 2016-05-05 20:35:19.809398 + my-custom-bundle 2016-05-05 20:34:53.654082 + my-custom-bundle 2016-05-05 20:34:48.401767 + quandl + quantopian-quandl 2016-05-05 20:06:40.894956 + +The output here shows that there are 3 bundles available: + +- ``my-custom-bundle`` (added by the user) +- ``quandl`` (provided by zipline) +- ``quantopian-quandl`` (provided by zipline) + +The dates and times next to the name show the times when the data for this +bundle was ingested. We have run three different ingestions for +``my-custom-bundle``. We have never ingested any data for the ``quandl`` bundle +so it just shows ```` instead. Finally, there is only one +ingestion for ``quantopian-quandl``. + +Ingesting Data +~~~~~~~~~~~~~~ + +The first step to using a data bundle is to ingest the data. The ingestion +process will invoke some custom bundle command and then write the data to a +standard location that zipline can find. By default the location where ingested +data will be written is ``$ZIPLINE_ROOT/data/`` where by default +``ZIPLINE_ROOT=~/.zipline``. The ingestion step may take some time as it could +involve downloading and processing a lot of data. This can be run with: + +.. code-block:: bash + + $ zipline ingest [-b ] where ```` is the name of the bundle to ingest, defaulting to -:ref:`quantipian-quandl `. +:ref:`quantopian-quandl `. Old Data ~~~~~~~~ When the ``ingest`` command is used it will write the new data to a subdirectory of ``$ZIPLINE_ROOT/data/`` which is named with the current date. This -makes it possible to look at older data or even run backtests with this older -copy. This makers it easier to reproduce backtest results later. +makes it possible to look at older data or even run backtests with the older +copies. Running a backtest with an old ingestion makes it easier to reproduce +backtest results later. -One drawback of saving all of this data by default is that the data directory -may grow quite large even if you do not want to use the data. To solve this -problem there is another command ``clean`` which will clear data bundles based -on some time constraints. +One drawback of saving all of the data by default is that the data directory +may grow quite large even if you do not want to use the data. As shown earlier, +we can list all of the ingestions with the :ref:`bundles command +`. To solve the problem of leaking old data there is another +command: ``clean``, which will clear data bundles based on some time +constraints. For example: .. code-block:: bash # clean everything older than - $ python -m zipline clean [-b ] --before + $ zipline clean [-b ] --before # clean everything newer than - $ python -m zipline clean [-b ] --after + $ zipline clean [-b ] --after # keep everything in the range of [before, after] and delete the rest - $ python -m zipline clean [-b ] --before --after + $ zipline clean [-b ] --before --after # clean all but the last runs - $ python -m zipline clean [-b ] --keep-last + $ zipline clean [-b ] --keep-last Running Backtests with Data Bundles ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Now that the data has been ingested we can use it to run backtests with the -``run`` command. This can be specified with the ``--bundle`` option like: +``run`` command. The bundle to use can be specified with the ``--bundle`` option +like: .. code-block:: bash - $ python -m zipline run --bundle --algofile algo.py ... + $ zipline run --bundle --algofile algo.py ... We may also specify the date to use to look up the bundle data with the -``--bundle-date`` option. This will cause us to the the most recent bundle -ingestion that is less than or equal to the ``bundle-date``. This is how we can -run backtests with older data. The reason that this uses a less than or equal to -relationship is that we can specify the date that we ran an old backtest and get -the same data that would have been available to us on that date. The -``bundle-date`` defaults to the current day to use the most recent data. +``--bundle-date`` option. Setting the ``--bundle-date`` will cause run to use +the most recent bundle ingestion that is less than or equal to the +``bundle-date``. This is how we can run backtests with older data. The reason +that ``-bundle-date`` uses a less than or equal to relationship is that we can +specify the date that we ran an old backtest and get the same data that would +have been available to us on that date. The ``bundle-date`` defaults to the +current day to use the most recent data. Default Data Bundles ~~~~~~~~~~~~~~~~~~~~ @@ -78,22 +116,23 @@ Default Data Bundles Quandl WIKI Bundle `````````````````` + By default zipline comes with the ``quandl`` data bundle which uses quandl's `WIKI dataset `_. The quandl data bundle includes daily pricing data, splits, cash dividends, and asset metadata. To -ingest this data bundle we recommend creating an account on quandl.com to get an -API key to be able to make more API requests per day. Once we have an API key we -may run: +ingest the ``quandl`` data bundle we recommend creating an account on quandl.com +to get an API key to be able to make more API requests per day. Once we have an +API key we may run: .. code-block:: bash - $ QUANDL_API_KEY= python -m zipline ingest quandl + $ QUANDL_API_KEY= zipline ingest quandl though we may still run ``ingest`` as an anonymous quandl user (with no API key). We may also set the ``QUANDL_DOWNLOAD_ATTEMPTS`` environment variable to an integer which is the number of attempts that should be made to download data -from quandls servers. By default this will be 5, meaning that we will retry each -attempt 5 times. +from quandls servers. By default ``QUANDL_DOWNLOAD_ATTEMPTS`` will be 5, meaning +that we will retry each attempt 5 times. .. note:: @@ -117,10 +156,11 @@ Yahoo Bundle Factories Zipline also ships with a factory function for creating a data bundle out of a set of tickers from yahoo: :func:`~zipline.data.bundles.yahoo_equities`. -This makes it easy to pre-download and cache the data for a set of equities from -yahoo. This includes daily pricing data along with splits, cash dividends, and -inferred asset metadata. To create a bundle from a set of equities, add the -following to your ``~/.zipline/extensions.py`` file: +:func:`~zipline.data.bundles.yahoo_equities` makes it easy to pre-download and +cache the data for a set of equities from yahoo. The yahoo bundles include daily +pricing data along with splits, cash dividends, and inferred asset metadata. To +create a bundle from a set of equities, add the following to your +``~/.zipline/extensions.py`` file: .. code-block:: python @@ -142,8 +182,8 @@ This may now be used like: .. code-block:: bash - $ python -m zipline ingest my-yahoo-equities-bundle - $ python -m zipline run -f algo.py --bundle my-yahoo-equities-bundle + $ zipline ingest my-yahoo-equities-bundle + $ zipline run -f algo.py --bundle my-yahoo-equities-bundle More than one yahoo equities bundle may be registered as long as they use @@ -153,15 +193,16 @@ Writing a New Bundle ~~~~~~~~~~~~~~~~~~~~ Data bundles exist to make it easy to use different data sources with -zipline. To add a new bundle, one must implement an ingest function. +zipline. To add a new bundle, one must implement an ``ingest`` function. -This function is responsible for loading the data into memory and passing it to -a set of writer objects provided by zipline to convert the data to zipline's -internal format. The ingest function may work by downloading data from a remote -location like the ``quandl`` bundle or yahoo bundles or it may just load files -that are already on the machine. The function is provided with writers that will -write the data to the correct location transactionally. If an ingestion fails -part way through the bundle will not be written in an incomplete state. +The ``ingest`` function is responsible for loading the data into memory and +passing it to a set of writer objects provided by zipline to convert the data to +zipline's internal format. The ingest function may work by downloading data from +a remote location like the ``quandl`` bundle or yahoo bundles or it may just +load files that are already on the machine. The function is provided with +writers that will write the data to the correct location transactionally. If an +ingestion fails part way through the bundle will not be written in an incomplete +state. The signature of the ingest function should be: @@ -258,8 +299,8 @@ have. ```````````` ``calendar`` is a ``pandas.DatetimeIndex`` object holding all of the trading -days that the bundle should load data for. This is to help some bundles generate -queries for the days needed. +days that the bundle should load data for. The calendar is provided to help some +bundles generate queries for the days needed. ``cache`` ````````` @@ -290,8 +331,8 @@ forwarded to ``minute_bar_writer.write`` and ``daily_bar_writer.write``. `````````````` ``output_dir`` is a string representing the file path where all the data will be -written. This will be some subdirectory of ``$ZIPLINE_ROOT`` and will contain -the time of the start of the current ingestion. This can be used to directly -move resources here if for some reason your ingest function can produce it's own -outputs without the writers. For example, the ``quantopian:quandl`` bundle uses -this to directly untar the bundle into the ``output_dir``. +written. ``output_dir`` will be some subdirectory of ``$ZIPLINE_ROOT`` and will +contain the time of the start of the current ingestion. This can be used to +directly move resources here if for some reason your ingest function can produce +it's own outputs without the writers. For example, the ``quantopian:quandl`` +bundle uses this to directly untar the bundle into the ``output_dir``. diff --git a/docs/source/whatsnew/1.0.0.txt b/docs/source/whatsnew/1.0.0.txt index 3a11be8a..29750a1e 100644 --- a/docs/source/whatsnew/1.0.0.txt +++ b/docs/source/whatsnew/1.0.0.txt @@ -12,27 +12,28 @@ Development Highlights ~~~~~~~~~~ -New Entry Points (:issue:`xxxx`) -```````````````````````````````` +New Entry Points (:issue:`1173` and :issue:`1178`) +`````````````````````````````````````````````````` In order to make it easier to use zipline we have updated the entry points for a backtest. The three supported ways to run a backtest are now: 1. :func:`zipline.run_algo` -2. ``$ python -m zipline run`` +2. ``$ zipline run`` 3. ``%zipline`` (IPython magic) -Data Bundles (:issue:`xxxx`) -```````````````````````````` +Data Bundles (:issue:`1173` and :issue:`1178`) +`````````````````````````````````````````````` 1.0.0 introduces data bundles. Data bundles are groups of data that should be preloaded and used to run backtests later. This allows users to not need to to specify which tickers they are interested in each time they run an algorithm. This also allows us to cache the data between runs. -By default, the ``quandl`` bundle will be used which pulls data from quandl's -`WIKI dataset `_. New bundles may be -registered with :func:`zipline.data.bundles.register` like: +By default, the ``quantopian-quandl`` bundle will be used which pulls data from +Quantopian's mirror of the quandl `WIKI dataset +`_. New bundles may be registered with +:func:`zipline.data.bundles.register` like: .. code-block:: python @@ -53,10 +54,10 @@ have been passed to write that data to disc in a location that zipline can find later. This data can be used in backtests by passing the name as the ``-b / --bundle`` -argument to ``$ python -m zipline run`` or as the ``bundle`` argument to -:func:`zipline.run_algo`. +argument to ``$ zipline run`` or as the ``bundle`` argument to +:func:`zipline.run_algorithm`. -For more information see `Data Bundles`_ for more information. +For more information see :ref:`data-bundles` for more information. Enhancements ~~~~~~~~~~~~