* ENH: Adds support for supplementary asset mappings
- Adds a supplementary_mappings table to the assets.db, to hold point-
in-time mappings of sids to arbitrary categories of values, e.g.
alternative identifiers. This bumps ASSET_DB_VERSION.
- Adds supplementary_map and supplementary_map_by_sid to AssetFinder,
caches of the underlying table that are fully populated on first
access, which map the supplementary values to sids and vice versa,
respectively.
- Adds lookup_by_supplementary_mapping method, which fronts
supplementary_map to query for the asset last known to have held a
value at a given dt.
- Add get_supplementary_field method, which fronts
supplementary_map_by_sid to query for the last known value held by an
asset at a given dt.
For futures that behave like GC, use the latest roll as the back contract when
walking backwards over the window, so that when the front contract is skipped
because it never has more volume between its auto close date and the previous
auto close date, the back contract which did have volume is still used when
making comparisons to construct the chain.
Instead of maintaining a separate index into the sessions index, use the `.freq`
member of the sessions index for decrementing to the current session and finding
the previous session.
To support contracts such as `PL` which should roll from F->J->N->V, add the
ability to pass a predicate function to the ordered contract chain contstrution
which returns `True` if the contract is allowed in the chain.
Convert the end minute to the its session label before calling `_active_contract`,
otherwise the volume roll finder's attempt to use the session bar reader fails
due to a non-session label Timestamp.
Fix multiple errors when attempting to generate rolls for futures which do not
roll month to month, e.g. the Eurodollar.
These errors were caused by logic that always incremented from contract to
contract by delivery month, with errors when the next contract was not part of
the quarterly roll chain and thus had not yet begun trading even though the
previous contract had autoclosed. Instead, filter out these contracts and only
allow contracts that have begun trading before the previous contract's autoclose.
This is in lieu of a more explicit specification of quarterly rolls.
Instead of requiring the roll finder to juggle the indices into the ordered
contracts, use a doubly linked list where the nodes element is the contract
with members pointing to the previous and next contracts in the chain.
Besides improving legibility in the roll finder code, this change is on the path
to adding a predicate to exclude contracts from the chain, e.g. contracts in ED
which are not in the roll schedule.
Change test results for primary chain, since new implementaton does not stop at
contract in which has not yet started when constructing the chain.
Fix common error condition which was triggered whenever the session at the end
of the prefetched history window was a session where the back contract was
active. When the back contract was the active contract, the next contract for
consideration was the front contract at the end of the window, which
definitionally always has an autoclose after the end of the window.
Instead, just start seeking backwards from the end of the window.
Also prevent lookahead bias in volume rolls, which was caused by the using the
volume for a session to determine whether that session had rolled. Information
that would not have been available at the beginning of the session.
This change makes the volume rolls overly conservative, and may be improved by
looking at vectors of the preceding volume and making the roll off of momentum.
The end date of the last contract with a sufficient start date was being
used for the continuous future overall end date; however the end date of
that contract (which is the last day for which there is data for the
contract) is not necessarily the greatest end date out of all contracts.
It is possible for the furthest out contract to have some, but very
few, trades before it is more actively traded. Which would give it a
start date within in the range of the simulation, but an end date is
earlier than the other contracts which are active during the simulation.
This bug would result in `nan`s when getting the current price because
of the `end_date` check in `get_spot_value`. When the current simulation
time was greater than the `end_date` of the last contract the condition
which guards against attempting to get data for an instrument past its
end date would return a `nan`, even when the current underlying contract
did have data for that date.
Use max end date of all contracts instead of the last one, to ensure
that the continuous future last date is always great enough to allow
access to all contracts with in the chain.
Also, use min start date to accurately mirror the end date behavior.
Register equities, futures, and continuous futures to an `abc` which
signifies that the type is associable with data, and thus can be used in
a history context.
May want to use this in `check_parameters` for `BarData` methods, but
work would need to be done to make sure the error message still displays
the registered types.
Allow `ContinuousFuture` when checking for a single "asset".
This could be further improved by:
- Defininng a tuple of the `Asset`-like types OR making
`ContinuousFuture` and `Asset` share a common type (whether that is
`ContinuousFuture` inheriting from `Asset` or making `Asset` and
`ContinuousFuture` share a common type.)
- Make a `history` test which uses `BarData` + `ContinuousFuture`,
instead of just using the history loader directly from tests.
To support using a `DataPortal` and `HistoryLoader` in a notebook, allow
the prefetch length to be configurable, so that it can be set to 0.
Unlike backtesting where the prefetch is useful for repeated history
windows viewed from datetimes which are monotonically increasing by a
small amount, the notebook usage of history windows needs only to
retrieve the exact data needed for the window specified.
This patch also fixes some boundary conditions related to rolls and
adjustments which were uncovered by querying for the adjustments with an
end date near the end of the window.
Add roll style which takes the volume of the contracts into account.
If the volume moves from the front to the back before the auto close
date, the roll is put at that session.
Also, factors out some of the common logic shared with calendar based rolls.
Add `.adj('mul')` and `.adj('add')` methods on ContinuousFuture, which
when used with `history`, will calculate and apply adjustments so that
the values are adjusted to account for discounts and premiums during
rolls.
Example usage in an algo:
```
from zipline.api import continuous_future
def initialize(context):
context.cl_add = continuous_future('CL', offset=0, roll='calendar').adj('add')
context.cl_mul = continuous_future('CL', offset=0, roll='calendar').adj('mul')
context.cl = continuous_future('CL', offset=0, roll='calendar')
schedule_function(print_history)
def print_history(context, data):
frame = data.history([context.cl, context.cl_add, context.cl_mul],
['price', 'sid'],
20,
'1d')
print 'unadjusted'
print frame.loc[:, :, context.cl]
print 'adjusted add'
print frame.loc[:, :, context.cl_add]
print 'adjusted mul'
print frame.loc[:, :, context.cl_mul]
```
Enable unadjusted history for continuous futures.
The history array is filled by the values for the underlying contracts,
where the contract used changes based on rolls.
e.g., if a `1d` history window was over the range
`2016-01-20` -> `2016-02-29` with contracts with a suffix of `F16` that
rolls at the beginning of the session on `2016-01-26`, `G16` on
`2016-02-26`, and `H16` on `2016-03-26`. The `2016-01-20` ->
`2016-01-25` portion would use the values for `F16', the `2016-01-26` ->
`2016-02-25` portion would use `G16` and the `2016-02-26` ->
`2016-02-29` portion would use `H16`.
Using the same contracts as above, a `1m` history window over the range
(using a timezone of US/Eastern) `2016-01-25 4:00PM` -> `2016-01-25
7:00PM` would fill the `4:00PM` -> `6:00PM` portion with data for `F16`
and the `6:01PM` -> `7:00PM` portion with data for `G16`, since the
beginning of the `2016-01-26` session is `2016-01-25 6:01PM`.
Supports `1d` and `1m`.
Also adds the `sid` field to `history` to assist in showing the active
contract at each dt in the window.
Add `chain`field to current, as well as supporting methods in DataPortal
and OrderedContracts.
Enables the following example:
```
from zipline.api import continuous_future
def initialize(context):
context.primary_cl = continuous_future('CL', offset=0, roll='calendar')
schedule_function(print_current_chain)
def print_current_chain(context, data):
chain = data.current_chain(context.primary_cl)
print 'datetime={0}'.format(get_datetime())
print 'primary={0}'.format(chain[0])
print 'secondary={0}'.format(chain[1])
print 'tertiary={0}'.format(chain[2])
```
```
datetime=2015-12-23 14:31:00+00:00
primary=Future(1058201602 [CLG16])
secondary=Future(1058201603 [CLH16])
tertiary=Future(1058201604 [CLJ16])
```
Also:
- make return types of OrderedContracts methods compatible across
architectures. (Noticed while adding `active_chain` method.)
- Add year suffix to future contract names in test data.
Add the ability for an algorithm to request the current contract for a
future chain via `data.current`.
e.g.:
```
data.current(ContinuousFuture('CL', offset=0, roll='calendar'),
'contract')
```
`future_chain` will be replaced by the as yet to be implemented method,
`data.current_chain`
Also removing `FutureChain` which will be replaced by another version
which only supports indexing and iteration.
No longer auto-updates its internal as-of date, instead requires an explicit
as-of date from the consumer.
Take a static list of contracts (instead of needing an assetfinder).
Instead of the as_of method, the user-facing API now lets you pass in an
offset, which is defined as an integral number of sessions.
* BUG: Fixes asset writer to the select the latest asset to hold a sid
When constructing the asset_info dataframe, we were previously taking
the first symbol/sid pair to include, when we should be taking the most
recent.
* Ensure groups are sorted by increasing end_date
* Updates test_lookup_symbol_change_ticker to also cover asset_name
Scott Sanderson