If a KeyError occurred in the adjustment logic, the exception would be
swallowed by the try block, which was intended to just check whether or
not there was an adjustment reader adjusted.
Discovered when some logic in a futures adjustment reader were failing
because of a mismatch of minute and session labels, which resulted in no
adjustments during windows when there should have been.
The minute to session sampling reading was creating two DataFrame
objects, the first to hold the minute data, and then a second returned
by the `DataFrame.groupby` to sample down to sessions.
Instead use the arrays returned by the minute readers `load_raw_arrays`
and implement sampling logic which takes advantage that the minutes
being passed start with the first minute of the first session and end
with the last minute of the last session.
On my machine this takes the tests in `test/test_continuous_futures`
from ~4.0 to about ~0.1 seconds.
Add `.adj('mul')` and `.adj('add')` methods on ContinuousFuture, which
when used with `history`, will calculate and apply adjustments so that
the values are adjusted to account for discounts and premiums during
rolls.
Example usage in an algo:
```
from zipline.api import continuous_future
def initialize(context):
context.cl_add = continuous_future('CL', offset=0, roll='calendar').adj('add')
context.cl_mul = continuous_future('CL', offset=0, roll='calendar').adj('mul')
context.cl = continuous_future('CL', offset=0, roll='calendar')
schedule_function(print_history)
def print_history(context, data):
frame = data.history([context.cl, context.cl_add, context.cl_mul],
['price', 'sid'],
20,
'1d')
print 'unadjusted'
print frame.loc[:, :, context.cl]
print 'adjusted add'
print frame.loc[:, :, context.cl_add]
print 'adjusted mul'
print frame.loc[:, :, context.cl_mul]
```
Start making the equity adjustments calculations for the history loader
conform to the same method signature as `load_adjustments` provided by
`SQLiteAdjustmentReader, so that an `AdjustmentReader` interface can
begin to take form.
This prepares for creating a `DispatchAdjustmentReader` which will route
adjustment calculations for equities to the
`HistoryCompatibleUSEquityAdjustmentReader` and continuous futures to a
not yet implemented adjustment reader. All of these readers will share
the `load_adjustments` method.
Add a perspective offset to `AdjustedArrayWindow` and `AdjustedArray`,
so that `HistoryLoader` does not need to twiddle with offsets to support
viewing the data from the bar after end of the window, (Which is the
case when a '1d' history window is retrieved in minute mode, which is
explained in the docstring for `HistoryLoader.history`)
Presently, this simplifies the logic in
`HistoryLoader._get_adjustments_in_range`, and other incoming
AdjustmentReader's, (e.g. the roll based adjustment reader for continous
futures.) This patch should also make it easier for history and pipeline
to converge on a singular `load_adjustments` method.
Enable unadjusted history for continuous futures.
The history array is filled by the values for the underlying contracts,
where the contract used changes based on rolls.
e.g., if a `1d` history window was over the range
`2016-01-20` -> `2016-02-29` with contracts with a suffix of `F16` that
rolls at the beginning of the session on `2016-01-26`, `G16` on
`2016-02-26`, and `H16` on `2016-03-26`. The `2016-01-20` ->
`2016-01-25` portion would use the values for `F16', the `2016-01-26` ->
`2016-02-25` portion would use `G16` and the `2016-02-26` ->
`2016-02-29` portion would use `H16`.
Using the same contracts as above, a `1m` history window over the range
(using a timezone of US/Eastern) `2016-01-25 4:00PM` -> `2016-01-25
7:00PM` would fill the `4:00PM` -> `6:00PM` portion with data for `F16`
and the `6:01PM` -> `7:00PM` portion with data for `G16`, since the
beginning of the `2016-01-26` session is `2016-01-25 6:01PM`.
Supports `1d` and `1m`.
Also adds the `sid` field to `history` to assist in showing the active
contract at each dt in the window.
Add `chain`field to current, as well as supporting methods in DataPortal
and OrderedContracts.
Enables the following example:
```
from zipline.api import continuous_future
def initialize(context):
context.primary_cl = continuous_future('CL', offset=0, roll='calendar')
schedule_function(print_current_chain)
def print_current_chain(context, data):
chain = data.current_chain(context.primary_cl)
print 'datetime={0}'.format(get_datetime())
print 'primary={0}'.format(chain[0])
print 'secondary={0}'.format(chain[1])
print 'tertiary={0}'.format(chain[2])
```
```
datetime=2015-12-23 14:31:00+00:00
primary=Future(1058201602 [CLG16])
secondary=Future(1058201603 [CLH16])
tertiary=Future(1058201604 [CLJ16])
```
Also:
- make return types of OrderedContracts methods compatible across
architectures. (Noticed while adding `active_chain` method.)
- Add year suffix to future contract names in test data.
Add the ability for an algorithm to request the current contract for a
future chain via `data.current`.
e.g.:
```
data.current(ContinuousFuture('CL', offset=0, roll='calendar'),
'contract')
```
This allows optionally setting the last available dts in the DataPortal
explicitly. If these args aren't provided, we fall back to inferring
these from the underlying readers, which was the previous behavior.
When dispatching to sub readers in dispatch reader, pass along the asset
object, instead of extracting the sid.
The in development reader for continuous futures values besides `sid`
are needed from the `ContinuousFuture` object.
`data.loader.ensure_benchmark_data()` was trying to use data after an exception was raised loading it. The code was logging and swallowing exceptions; this re-raises.
With the addition of the truncate function, there are cases where we'll
want to construct a BcolzMinuteBarWriter to call truncate, without
gathering all the metadata. This commit adds a write_metadata arg to its
init, which is True by default. If False is specified, no metadata is
written.
Requires adding logic to truncate to update end_session in metadata to
the truncate date.
- Fixes a warning on indexing with a float that ultimately came from
pd.Timedelta.total_seconds(). Adds ``timedelta_to_integral_seconds``
and ``timedelta_to_integral_minutes()`` functions and replaces various
usages of ``int(delta.total_seconds())`` with them.
- Fixes a warnings triggered in ``_create_daily_stats`` from
passing tz-aware datetimes to np.datetime64.
This reverts commit 86c7635b45, reversing
changes made to c77f2b92df.
Some real world cases hit errors with this change, due to the new offset
logic attempting to create Adjustments with invalid parameters.
Will identify exact conditions that cause this error and add as a test
case before remerging.
Instead of `HistoryLoader` containing separate adjustment calculation
logic, use `SQLiteAdjustmentReader.load_adjustments`.
This change required the addition of two offset parameters to
`load_adjustments` since the perspective on the data from within
`schedule_function` is skewed from how Pipeline looks at historical
data.
This is working towards creating an `AdjustmentReader` abc which
`SQLiteAdjustmentReader` and a upcoming continuous future adjustment
reader will share.
Remove module scope invocations of `get_calendar('NYSE')`, which cuts
zipline import time in half on my machine. This make the zipline CLI
noticeably more responsive, and it reduces memory consumed at import
time from 130MB to 90MB.
Before:
$ time python -c 'import zipline'
real 0m1.262s
user 0m1.128s
sys 0m0.120s
After:
$ time python -c 'import zipline'
real 0m0.676s
user 0m0.536s
sys 0m0.132s
test_resample now fully covers the resample module.
Fix a bug exposed by increased coverage, where daily aggregation on
`high` would return `nan` for an asset instead of 1) during the
course of day `1d` history was called on non-consecutive minutes and 2)
either, a) the value for the previously inspected dt was `nan` or b)
there were only `nan`s between the previous and current dt.
`low` had a similar bug which was only triggered if the value for the
previously inspected dt was `nan`.
Increase coverage on `ReindexSessionBarReader` so that all methods which
are considered part of the interface are covered by `test_resample`.
Fix bug in `get_value`, exposed by increased coverage, where the
`NoDataOnDate` exception was bubbling from the bcolz reader all the way
up when a session which was a holidy on the underlying reader was passed
to the reindex reader. (The reindex reader should return nan/0 in that
case.)
Also, move location of data index exceptions so that they are agnostic
to bcolz/us_equity_pricing; since the exception is now used by the
resample module to fix aforementioned bug.
Remove special handling for the last session of an asset, which was
moving the last traded back a session.
If the asset has data on a session, `get_last_traded_dt` should always
return that session if it is the parameter to the method.
`1d` history calls were failing on key errors when using the
`us_futures` calendar, because of timestamps occuring before a midnight
would present the wrong midnight (i.e. the midnight before the session,
instead of the following midnight, which is the label for the current
session.)
Tests will follow when bringing up coverage on resample and data portal
modules.
Combine the equity and future readers into asset dispatch readers, so
that simulations that use both asset types can access data for each.
This patch enables `history` for future assets in algorithms; however,
it does not add extra coverage in the `test_data_portal` or `test_history`
to cover future assets. Those tests will follow, however putting this in
separately since it shows that the wrapping of the readers in the asset
dispatch reader does not break existing equity strategies.
Add `AssetDispatchSessionBarReader` and corresponding minute and session
bar version of that reader.
This reader routes requests to the appropriate reader based on the asset
type of the requested sids.
`load_raw_array` in the dispatch reader batches the sid by asset type
and then interleaves the results in the out arrays, so that the arrays
data corresponds with sids in the order that sids are passed to the
method, to meet the expected behavior of `load_raw_arrays`.
The dispatch redaer is intended for use by the data portal when using
both future and equities. The dispatch reader will also be passed to the
to the `HistoryLoader`s contained within the data portal, where the
batched `load_raw_arrays` will be used.
Also, BUG:
- Fix the return of `MinuteResampleSessionBarReader.load_raw_arrays` to
match all other readers.
- Use the input dt for the `MinuteResampleSessionBarReader.load_raw_arrays`
as a session label, instead of a minute dt, since it is a session bar
reader.
(Both of these bugs where discovered when using the resample reader for
future data in the dispatch tests.)
Working towards history results which contain mixed asset types, add
a reader which makes `load_raw_arrays` return results indexed on the
session/minute ranges specified by the specified `trading_calendar`
instead of the calendar of the backing reader.
This reader will be used to make Equity readers align with Future
readers. It is intended for use as part of another reader (which will
dispatch queries based on asset type and then recombined results) which
will be passed to the `[Minute|Session]HistoryLoaders in the data portal.
For scaling up pricing data before writing to bcolz, the writer now
accepts a dict mapping each sid to the ratio to use. It still accepts a
single ratio as default_ohlc_ratio, which is used as a fallback if no
mapping exists for a given sid. The default is OHLC_RATIO (1000).
This allows better handling of futures pricing data, where the required
precision across root symbols is not consistent.
The daily/session bar reader's `spot_price` took the same parameters and
returned the same kind of output as the minute bar reader's `get_value`.
Standardize on one method to make a common interface, which may be
formally factored out in a later patch; to help enable writing reader
implementations or mixins which can be agnostic to the bar frequency.
Eddie Hebert