In the data portal, remove methods that make a distinction between
future and equity asset type. Instead rely on the pricing reader
dispatching.
In support of incoming work which will upsample equity history arrays to
the larger future calendar.
Also, remove perf tracker tests which were using an equity
reader/writer, to be added back in later.
Also, move `DailyHistoryAggregator` to `resample` module, so that tools
for converting from minute to session bars are collocated.
This patch is in preparation of adding a daily bar reader which
resamples minute data, which will be located in the `resample` module
and share the test cases and expected results in `test_resample`.
Use the future asset equity pricing reader, instead of reading directly
from the bcolz table. Required since the format for writing the future
data now uses the minute bar reader/writer pair.
Add test cases to `test_data_portal` asserting both equity and future
`get_spot_value` results.
Also, add direct coverage of last_traded_dt in the `test_data_portal`
module.
Prepares for adding test coverage of `get_last_traded_dt` for `Future` assets.
Instead of having separate ExchangeCalendar and TradingSchedule objects, we
now just have TradingCalendar. The TradingCalendar keeps track of each
session (defined as a contiguous set of minutes between an open and a close).
It's also responsible for handling the grouping logic of any given minute
to its containing session, or the next/previous session if it's not a market
minute for the given calendar.
In preparation of adding futures, add equity to the names of both the
classes and methods for writing bcolz data. Futures data will use a
different minutes per day with a separate reader. This change will allow
both equity and futures fixtures to be side by side.
Also, break out the method which generates the dataframes and trading
days member into fixtures (`EquityMinuteBarData` and
`EquityDailyBarData`) on which the `*BarReader` fixture depends. This
fixture is separated out to enable reader/writers in different formats
to use the same data setup. (There is internal code which needs to write
minute and daily bar data in a database format.)
Querying for the price field of an equity actually looks at the close
field, so we should do the same for futures. Otherwise `data.can_trade`
and `data.current` of 'price' fail for futures.
Since the first trading day is now passed directly to the DataPortal on
init, there's no need for a method that does this. Moves all the
additional logic/assignments into the init. Also corrects an issue where
we would never create certain attributes if self._first_trading_day was
None.
Adds the ability to specify the first trading day for a data portal in a
test case when using the WithDataPortal fixture.
Fix behavior in minute mode history with frequency `1d`, where on the
day immediately following an adjustment action, the overnight adjustment
would not apply. (However the adjustment would be applied after a 1 day
lag.)
The root cause of the bug was that the history data for minute mode when
using `1d` stitches together a sliding window of the daily data for
previous and the current minute. That daily data sliding window and
corresponding adjustments was being read as if the data was being viewed
from on the last day of the window; however in this case the data is
being viewed from the day after the window has completed. The difference
in view points requires the adjustments to popped and applied by the
adjusted array one index earlier. The fix uses the `extra_slot` value as
signifier on whether the data is being viewed on the following day and
then accordingly adjusts the index of the mulitpy object.
Also, change the split and merger test data ratios to have different values,
to ensure that different adjustment values are applied; as opposed to
doubling up on just one of the values.
Instead of inferring it from the minute/daily writer, we now require the
first trading day to be passed explicitly, so the creator of the
DataPortal controls what is used as the first trading day.
Adds the data bundle concept which makes it easy for users to register
loading functions to build out minute and daily data along with an
assets db and adjustments db. By default we have provided a `quandl`
bundle which pulls from the public domain WIKI dataset. Users may
register new bundles by decorating an ingest function with
`zipline.data.bundles.register(<name>)`. This also provides a
`yahoo_equities` function for creating an ingestion function that will
load a static set of assets from yahoo.
The cli is now structured as a couple of subcommands and has been
changed to `python -m zipline`. The old behavior of `run_algo.py` has
been moved to the `run` subcommand. This is almost entirely the same
except that it now takes the name of the data bundle to use, defaulting
to `quandl`.
The next subcommand is `ingest` which takes the name of
a data bundle to ingest. This will run the loading machinery and write
the data to a specified location that `run` can find.
There is also a `clean` subcommand which deletes the data that was
written with `ingest`.
Extensions have also been added to zipline. This is an experimental
feature where users can provide an extra set of python files to run at
the start of the process. These can be used to configure aspects of
zipline. Right now the only thing that is supported in an extension file
is the registration of a new data bundle.
We were trying to use the previous day in before_trading_start because
we were looking for the previous market minute, then normalizing it. That's
no longer the case, as we want to use today's date for fetcher lookups
in before_trading_start.
Also refactored a bit how dataportal determines if a query should be
routed to the fetcher data structures.
The cache in data portal was added before the change to using a
CachedObject to wrap the window_blocks in the USEquityHistoryLoader.
Removing this extra layer saves some cycles.
Does not fix current memory investigaton (since only one sids/dts pair
per column was cached in `_equity_daily_reader_array_data` at a time),
but removing should make it more clear where needed references are being
held.
The minute history loader caching was incorrectly mimicking the daily
history loader caching.
Where caching the adjusted array on the last dt helps an access pattern
of repeated calling history windows on the same day (which has an end_dt
of the previous day), with minute windows the end dt is always moving
forward, so the cached values are seldom used. (Would only be used if
`history` was called with same parameters twice on the same simulation time.)
Moved from the `lazy-mainline` branch,
https://github.com/quantopian/zipline/pull/858
The intent of this patch to provide the basic class and readers
interfaces, developed on that branch, so that the use of creating the
object and opening paths etc. can be tested internally.
Additional changes beyond the lazy-mainline branch, addition of future
minute reader, and daily bar reader.
Also allow an argument of the future_daily_reader, though no such reader
yet exists.
It may be that future and equity readers share an interface, and a
further improvement would be providing an abstract base class.
co-author: @jbredeche <jean@quantopian.com>
