Adds the data bundle concept which makes it easy for users to register
loading functions to build out minute and daily data along with an
assets db and adjustments db. By default we have provided a `quandl`
bundle which pulls from the public domain WIKI dataset. Users may
register new bundles by decorating an ingest function with
`zipline.data.bundles.register(<name>)`. This also provides a
`yahoo_equities` function for creating an ingestion function that will
load a static set of assets from yahoo.
The cli is now structured as a couple of subcommands and has been
changed to `python -m zipline`. The old behavior of `run_algo.py` has
been moved to the `run` subcommand. This is almost entirely the same
except that it now takes the name of the data bundle to use, defaulting
to `quandl`.
The next subcommand is `ingest` which takes the name of
a data bundle to ingest. This will run the loading machinery and write
the data to a specified location that `run` can find.
There is also a `clean` subcommand which deletes the data that was
written with `ingest`.
Extensions have also been added to zipline. This is an experimental
feature where users can provide an extra set of python files to run at
the start of the process. These can be used to configure aspects of
zipline. Right now the only thing that is supported in an extension file
is the registration of a new data bundle.
We were trying to use the previous day in before_trading_start because
we were looking for the previous market minute, then normalizing it. That's
no longer the case, as we want to use today's date for fetcher lookups
in before_trading_start.
Also refactored a bit how dataportal determines if a query should be
routed to the fetcher data structures.
The cache in data portal was added before the change to using a
CachedObject to wrap the window_blocks in the USEquityHistoryLoader.
Removing this extra layer saves some cycles.
Does not fix current memory investigaton (since only one sids/dts pair
per column was cached in `_equity_daily_reader_array_data` at a time),
but removing should make it more clear where needed references are being
held.
The minute history loader caching was incorrectly mimicking the daily
history loader caching.
Where caching the adjusted array on the last dt helps an access pattern
of repeated calling history windows on the same day (which has an end_dt
of the previous day), with minute windows the end dt is always moving
forward, so the cached values are seldom used. (Would only be used if
`history` was called with same parameters twice on the same simulation time.)
Moved from the `lazy-mainline` branch,
https://github.com/quantopian/zipline/pull/858
The intent of this patch to provide the basic class and readers
interfaces, developed on that branch, so that the use of creating the
object and opening paths etc. can be tested internally.
Additional changes beyond the lazy-mainline branch, addition of future
minute reader, and daily bar reader.
Also allow an argument of the future_daily_reader, though no such reader
yet exists.
It may be that future and equity readers share an interface, and a
further improvement would be providing an abstract base class.
co-author: @jbredeche <jean@quantopian.com>
Joe Jevnik