wassname/catalyst
1
0
Fork 0
mirror of https://github.com/wassname/catalyst.git synced 2026-07-17 11:25:55 +08:00
Code Issues Packages Projects Releases Wiki Activity

DOC: Update docs for bundles and fix the whatsnew

Browse Source
This commit is contained in:
Joe Jevnik
2016-05-05 18:22:14 -04:00
parent d819721d96
commit a26802efd2
2 changed files with 107 additions and 65 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/source/bundles.rst
+95 -54
View File
@@ -1,3 +1,5 @@
.. _data-bundles:
Data Bundles
------------
@@ -5,71 +7,107 @@ A data bundle is a collection of pricing data, adjustment data, and an asset
database. Bundles allow us to preload all of the data we will need to run
backtests and store the data for future runs.
Ingesting Data
~~~~~~~~~~~~~~
.. _bundles-command:
The first step to using a data bundle is to ingest the data. This will invoke
some custom bundle command and then write the data to a standard location that
zipline can find. By default this location is ``$ZIPLINE_ROOT/data/<bundle>``
where by default ``ZIPLINE_ROOT=~/.zipline``. This step may take some time as it
could involve downloading and processing a lot of data. This can be run with:
Discovering Available Bundles
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Zipline comes with a few bundles by default as well as the ability to register
new bundles. To see which bundles we have have available, we may run the
``bundles`` command, for example:
.. code-block:: bash
$ python -m zipline ingest [-b <bundle>]
$ zipline bundles
my-custom-bundle 2016-05-05 20:35:19.809398
my-custom-bundle 2016-05-05 20:34:53.654082
my-custom-bundle 2016-05-05 20:34:48.401767
quandl <no ingestions>
quantopian-quandl 2016-05-05 20:06:40.894956
The output here shows that there are 3 bundles available:
- ``my-custom-bundle`` (added by the user)
- ``quandl`` (provided by zipline)
- ``quantopian-quandl`` (provided by zipline)
The dates and times next to the name show the times when the data for this
bundle was ingested. We have run three different ingestions for
``my-custom-bundle``. We have never ingested any data for the ``quandl`` bundle
so it just shows ``<no ingestions>`` instead. Finally, there is only one
ingestion for ``quantopian-quandl``.
Ingesting Data
~~~~~~~~~~~~~~
The first step to using a data bundle is to ingest the data. The ingestion
process will invoke some custom bundle command and then write the data to a
standard location that zipline can find. By default the location where ingested
data will be written is ``$ZIPLINE_ROOT/data/<bundle>`` where by default
``ZIPLINE_ROOT=~/.zipline``. The ingestion step may take some time as it could
involve downloading and processing a lot of data. This can be run with:
.. code-block:: bash
$ zipline ingest [-b <bundle>]
where ``<bundle>`` is the name of the bundle to ingest, defaulting to
:ref:`quantipian-quandl <quantopian-quandl-mirror>`.
:ref:`quantopian-quandl <quantopian-quandl-mirror>`.
Old Data
~~~~~~~~
When the ``ingest`` command is used it will write the new data to a subdirectory
of ``$ZIPLINE_ROOT/data/<bundle>`` which is named with the current date. This
makes it possible to look at older data or even run backtests with this older
copy. This makers it easier to reproduce backtest results later.
makes it possible to look at older data or even run backtests with the older
copies. Running a backtest with an old ingestion makes it easier to reproduce
backtest results later.
One drawback of saving all of this data by default is that the data directory
may grow quite large even if you do not want to use the data. To solve this
problem there is another command ``clean`` which will clear data bundles based
on some time constraints.
One drawback of saving all of the data by default is that the data directory
may grow quite large even if you do not want to use the data. As shown earlier,
we can list all of the ingestions with the :ref:`bundles command
<bundles-command>`. To solve the problem of leaking old data there is another
command: ``clean``, which will clear data bundles based on some time
constraints.
For example:
.. code-block:: bash
# clean everything older than <date>
$ python -m zipline clean [-b <bundle>] --before <date>
$ zipline clean [-b <bundle>] --before <date>
# clean everything newer than <date>
$ python -m zipline clean [-b <bundle>] --after <date>
$ zipline clean [-b <bundle>] --after <date>
# keep everything in the range of [before, after] and delete the rest
$ python -m zipline clean [-b <bundle>] --before <date> --after <after>
$ zipline clean [-b <bundle>] --before <date> --after <after>
# clean all but the last <int> runs
$ python -m zipline clean [-b <bundle>] --keep-last <int>
$ zipline clean [-b <bundle>] --keep-last <int>
Running Backtests with Data Bundles
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Now that the data has been ingested we can use it to run backtests with the
``run`` command. This can be specified with the ``--bundle`` option like:
``run`` command. The bundle to use can be specified with the ``--bundle`` option
like:
.. code-block:: bash
$ python -m zipline run --bundle <bundle> --algofile algo.py ...
$ zipline run --bundle <bundle> --algofile algo.py ...
We may also specify the date to use to look up the bundle data with the
``--bundle-date`` option. This will cause us to the the most recent bundle
ingestion that is less than or equal to the ``bundle-date``. This is how we can
run backtests with older data. The reason that this uses a less than or equal to
relationship is that we can specify the date that we ran an old backtest and get
the same data that would have been available to us on that date. The
``bundle-date`` defaults to the current day to use the most recent data.
``--bundle-date`` option. Setting the ``--bundle-date`` will cause run to use
the most recent bundle ingestion that is less than or equal to the
``bundle-date``. This is how we can run backtests with older data. The reason
that ``-bundle-date`` uses a less than or equal to relationship is that we can
specify the date that we ran an old backtest and get the same data that would
have been available to us on that date. The ``bundle-date`` defaults to the
current day to use the most recent data.
Default Data Bundles
~~~~~~~~~~~~~~~~~~~~
@@ -78,22 +116,23 @@ Default Data Bundles
Quandl WIKI Bundle
``````````````````
By default zipline comes with the ``quandl`` data bundle which uses quandl's
`WIKI dataset <https://www.quandl.com/data/WIKI>`_. The quandl data bundle
includes daily pricing data, splits, cash dividends, and asset metadata. To
ingest this data bundle we recommend creating an account on quandl.com to get an
API key to be able to make more API requests per day. Once we have an API key we
may run:
ingest the ``quandl`` data bundle we recommend creating an account on quandl.com
to get an API key to be able to make more API requests per day. Once we have an
API key we may run:
.. code-block:: bash
$ QUANDL_API_KEY=<api-key> python -m zipline ingest quandl
$ QUANDL_API_KEY=<api-key> zipline ingest quandl
though we may still run ``ingest`` as an anonymous quandl user (with no API
key). We may also set the ``QUANDL_DOWNLOAD_ATTEMPTS`` environment variable to
an integer which is the number of attempts that should be made to download data
from quandls servers. By default this will be 5, meaning that we will retry each
attempt 5 times.
from quandls servers. By default ``QUANDL_DOWNLOAD_ATTEMPTS`` will be 5, meaning
that we will retry each attempt 5 times.
.. note::
@@ -117,10 +156,11 @@ Yahoo Bundle Factories
Zipline also ships with a factory function for creating a data bundle out of a
set of tickers from yahoo: :func:`~zipline.data.bundles.yahoo_equities`.
This makes it easy to pre-download and cache the data for a set of equities from
yahoo. This includes daily pricing data along with splits, cash dividends, and
inferred asset metadata. To create a bundle from a set of equities, add the
following to your ``~/.zipline/extensions.py`` file:
:func:`~zipline.data.bundles.yahoo_equities` makes it easy to pre-download and
cache the data for a set of equities from yahoo. The yahoo bundles include daily
pricing data along with splits, cash dividends, and inferred asset metadata. To
create a bundle from a set of equities, add the following to your
``~/.zipline/extensions.py`` file:
.. code-block:: python
@@ -142,8 +182,8 @@ This may now be used like:
.. code-block:: bash
$ python -m zipline ingest my-yahoo-equities-bundle
$ python -m zipline run -f algo.py --bundle my-yahoo-equities-bundle
$ zipline ingest my-yahoo-equities-bundle
$ zipline run -f algo.py --bundle my-yahoo-equities-bundle
More than one yahoo equities bundle may be registered as long as they use
@@ -153,15 +193,16 @@ Writing a New Bundle
~~~~~~~~~~~~~~~~~~~~
Data bundles exist to make it easy to use different data sources with
zipline. To add a new bundle, one must implement an ingest function.
zipline. To add a new bundle, one must implement an ``ingest`` function.
This function is responsible for loading the data into memory and passing it to
a set of writer objects provided by zipline to convert the data to zipline's
internal format. The ingest function may work by downloading data from a remote
location like the ``quandl`` bundle or yahoo bundles or it may just load files
that are already on the machine. The function is provided with writers that will
write the data to the correct location transactionally. If an ingestion fails
part way through the bundle will not be written in an incomplete state.
The ``ingest`` function is responsible for loading the data into memory and
passing it to a set of writer objects provided by zipline to convert the data to
zipline's internal format. The ingest function may work by downloading data from
a remote location like the ``quandl`` bundle or yahoo bundles or it may just
load files that are already on the machine. The function is provided with
writers that will write the data to the correct location transactionally. If an
ingestion fails part way through the bundle will not be written in an incomplete
state.
The signature of the ingest function should be:
@@ -258,8 +299,8 @@ have.
````````````
``calendar`` is a ``pandas.DatetimeIndex`` object holding all of the trading
days that the bundle should load data for. This is to help some bundles generate
queries for the days needed.
days that the bundle should load data for. The calendar is provided to help some
bundles generate queries for the days needed.
``cache``
`````````
@@ -290,8 +331,8 @@ forwarded to ``minute_bar_writer.write`` and ``daily_bar_writer.write``.
``````````````
``output_dir`` is a string representing the file path where all the data will be
written. This will be some subdirectory of ``$ZIPLINE_ROOT`` and will contain
the time of the start of the current ingestion. This can be used to directly
move resources here if for some reason your ingest function can produce it's own
outputs without the writers. For example, the ``quantopian:quandl`` bundle uses
this to directly untar the bundle into the ``output_dir``.
written. ``output_dir`` will be some subdirectory of ``$ZIPLINE_ROOT`` and will
contain the time of the start of the current ingestion. This can be used to
directly move resources here if for some reason your ingest function can produce
it's own outputs without the writers. For example, the ``quantopian:quandl``
bundle uses this to directly untar the bundle into the ``output_dir``.
docs/source/whatsnew/1.0.0.txt
+12 -11
View File
@@ -12,27 +12,28 @@ Development
Highlights
~~~~~~~~~~
New Entry Points (:issue:`xxxx`)
````````````````````````````````
New Entry Points (:issue:`1173` and :issue:`1178`)
``````````````````````````````````````````````````
In order to make it easier to use zipline we have updated the entry points for
a backtest. The three supported ways to run a backtest are now:
1. :func:`zipline.run_algo`
2. ``$ python -m zipline run``
2. ``$ zipline run``
3. ``%zipline`` (IPython magic)
Data Bundles (:issue:`xxxx`)
````````````````````````````
Data Bundles (:issue:`1173` and :issue:`1178`)
``````````````````````````````````````````````
1.0.0 introduces data bundles. Data bundles are groups of data that should be
preloaded and used to run backtests later. This allows users to not need to to
specify which tickers they are interested in each time they run an
algorithm. This also allows us to cache the data between runs.
By default, the ``quandl`` bundle will be used which pulls data from quandl's
`WIKI dataset <https://www.quandl.com/data/WIKI>`_. New bundles may be
registered with :func:`zipline.data.bundles.register` like:
By default, the ``quantopian-quandl`` bundle will be used which pulls data from
Quantopian's mirror of the quandl `WIKI dataset
<https://www.quandl.com/data/WIKI>`_. New bundles may be registered with
:func:`zipline.data.bundles.register` like:
.. code-block:: python
@@ -53,10 +54,10 @@ have been passed to write that data to disc in a location that zipline can find
later.
This data can be used in backtests by passing the name as the ``-b / --bundle``
argument to ``$ python -m zipline run`` or as the ``bundle`` argument to
:func:`zipline.run_algo`.
argument to ``$ zipline run`` or as the ``bundle`` argument to
:func:`zipline.run_algorithm`.
For more information see `Data Bundles`_ for more information.
For more information see :ref:`data-bundles` for more information.
Enhancements
~~~~~~~~~~~~