To support using a `DataPortal` and `HistoryLoader` in a notebook, allow
the prefetch length to be configurable, so that it can be set to 0.
Unlike backtesting where the prefetch is useful for repeated history
windows viewed from datetimes which are monotonically increasing by a
small amount, the notebook usage of history windows needs only to
retrieve the exact data needed for the window specified.
This patch also fixes some boundary conditions related to rolls and
adjustments which were uncovered by querying for the adjustments with an
end date near the end of the window.
Rename _get_daily_window_for_sids to _get_daily_window_data.
Rename _get_minute_window_for_assets to _get_minute_window_data.
Rename _get_daily_data to get_daily_spot_value.
Add roll style which takes the volume of the contracts into account.
If the volume moves from the front to the back before the auto close
date, the roll is put at that session.
Also, factors out some of the common logic shared with calendar based rolls.
Match the behavior of the minute bar reader, now that the session and
minute bar readers share a common interface.
isnull is slightly slower than checking against -1; however, n cases
where we check against illiquid trades in a tight loop, volume is
checked which is not using nan. The change here should be marginal with
regards to performance.
Add `.adj('mul')` and `.adj('add')` methods on ContinuousFuture, which
when used with `history`, will calculate and apply adjustments so that
the values are adjusted to account for discounts and premiums during
rolls.
Example usage in an algo:
```
from zipline.api import continuous_future
def initialize(context):
context.cl_add = continuous_future('CL', offset=0, roll='calendar').adj('add')
context.cl_mul = continuous_future('CL', offset=0, roll='calendar').adj('mul')
context.cl = continuous_future('CL', offset=0, roll='calendar')
schedule_function(print_history)
def print_history(context, data):
frame = data.history([context.cl, context.cl_add, context.cl_mul],
['price', 'sid'],
20,
'1d')
print 'unadjusted'
print frame.loc[:, :, context.cl]
print 'adjusted add'
print frame.loc[:, :, context.cl_add]
print 'adjusted mul'
print frame.loc[:, :, context.cl_mul]
```
Enable unadjusted history for continuous futures.
The history array is filled by the values for the underlying contracts,
where the contract used changes based on rolls.
e.g., if a `1d` history window was over the range
`2016-01-20` -> `2016-02-29` with contracts with a suffix of `F16` that
rolls at the beginning of the session on `2016-01-26`, `G16` on
`2016-02-26`, and `H16` on `2016-03-26`. The `2016-01-20` ->
`2016-01-25` portion would use the values for `F16', the `2016-01-26` ->
`2016-02-25` portion would use `G16` and the `2016-02-26` ->
`2016-02-29` portion would use `H16`.
Using the same contracts as above, a `1m` history window over the range
(using a timezone of US/Eastern) `2016-01-25 4:00PM` -> `2016-01-25
7:00PM` would fill the `4:00PM` -> `6:00PM` portion with data for `F16`
and the `6:01PM` -> `7:00PM` portion with data for `G16`, since the
beginning of the `2016-01-26` session is `2016-01-25 6:01PM`.
Supports `1d` and `1m`.
Also adds the `sid` field to `history` to assist in showing the active
contract at each dt in the window.
Add `chain`field to current, as well as supporting methods in DataPortal
and OrderedContracts.
Enables the following example:
```
from zipline.api import continuous_future
def initialize(context):
context.primary_cl = continuous_future('CL', offset=0, roll='calendar')
schedule_function(print_current_chain)
def print_current_chain(context, data):
chain = data.current_chain(context.primary_cl)
print 'datetime={0}'.format(get_datetime())
print 'primary={0}'.format(chain[0])
print 'secondary={0}'.format(chain[1])
print 'tertiary={0}'.format(chain[2])
```
```
datetime=2015-12-23 14:31:00+00:00
primary=Future(1058201602 [CLG16])
secondary=Future(1058201603 [CLH16])
tertiary=Future(1058201604 [CLJ16])
```
Also:
- make return types of OrderedContracts methods compatible across
architectures. (Noticed while adding `active_chain` method.)
- Add year suffix to future contract names in test data.
Add the ability for an algorithm to request the current contract for a
future chain via `data.current`.
e.g.:
```
data.current(ContinuousFuture('CL', offset=0, roll='calendar'),
'contract')
```
This allows optionally setting the last available dts in the DataPortal
explicitly. If these args aren't provided, we fall back to inferring
these from the underlying readers, which was the previous behavior.
- Fixes a warning on indexing with a float that ultimately came from
pd.Timedelta.total_seconds(). Adds ``timedelta_to_integral_seconds``
and ``timedelta_to_integral_minutes()`` functions and replaces various
usages of ``int(delta.total_seconds())`` with them.
- Fixes a warnings triggered in ``_create_daily_stats`` from
passing tz-aware datetimes to np.datetime64.
`1d` history calls were failing on key errors when using the
`us_futures` calendar, because of timestamps occuring before a midnight
would present the wrong midnight (i.e. the midnight before the session,
instead of the following midnight, which is the label for the current
session.)
Tests will follow when bringing up coverage on resample and data portal
modules.
Combine the equity and future readers into asset dispatch readers, so
that simulations that use both asset types can access data for each.
This patch enables `history` for future assets in algorithms; however,
it does not add extra coverage in the `test_data_portal` or `test_history`
to cover future assets. Those tests will follow, however putting this in
separately since it shows that the wrapping of the readers in the asset
dispatch reader does not break existing equity strategies.
The daily/session bar reader's `spot_price` took the same parameters and
returned the same kind of output as the minute bar reader's `get_value`.
Standardize on one method to make a common interface, which may be
formally factored out in a later patch; to help enable writing reader
implementations or mixins which can be agnostic to the bar frequency.
In the data portal, remove methods that make a distinction between
future and equity asset type. Instead rely on the pricing reader
dispatching.
In support of incoming work which will upsample equity history arrays to
the larger future calendar.
Also, remove perf tracker tests which were using an equity
reader/writer, to be added back in later.
Also, move `DailyHistoryAggregator` to `resample` module, so that tools
for converting from minute to session bars are collocated.
This patch is in preparation of adding a daily bar reader which
resamples minute data, which will be located in the `resample` module
and share the test cases and expected results in `test_resample`.
Use the future asset equity pricing reader, instead of reading directly
from the bcolz table. Required since the format for writing the future
data now uses the minute bar reader/writer pair.
Add test cases to `test_data_portal` asserting both equity and future
`get_spot_value` results.
Also, add direct coverage of last_traded_dt in the `test_data_portal`
module.
Prepares for adding test coverage of `get_last_traded_dt` for `Future` assets.
Instead of having separate ExchangeCalendar and TradingSchedule objects, we
now just have TradingCalendar. The TradingCalendar keeps track of each
session (defined as a contiguous set of minutes between an open and a close).
It's also responsible for handling the grouping logic of any given minute
to its containing session, or the next/previous session if it's not a market
minute for the given calendar.
In preparation of adding futures, add equity to the names of both the
classes and methods for writing bcolz data. Futures data will use a
different minutes per day with a separate reader. This change will allow
both equity and futures fixtures to be side by side.
Also, break out the method which generates the dataframes and trading
days member into fixtures (`EquityMinuteBarData` and
`EquityDailyBarData`) on which the `*BarReader` fixture depends. This
fixture is separated out to enable reader/writers in different formats
to use the same data setup. (There is internal code which needs to write
minute and daily bar data in a database format.)
Querying for the price field of an equity actually looks at the close
field, so we should do the same for futures. Otherwise `data.can_trade`
and `data.current` of 'price' fail for futures.
Since the first trading day is now passed directly to the DataPortal on
init, there's no need for a method that does this. Moves all the
additional logic/assignments into the init. Also corrects an issue where
we would never create certain attributes if self._first_trading_day was
None.
Adds the ability to specify the first trading day for a data portal in a
test case when using the WithDataPortal fixture.
