Should be no functional change.
By making the raise on `if not isinstance` instead of doing a continue on `if
isinstance` (with a raise at the end of the loop if no 'good' conditions were
met'), the function should be more amenable to adding an additional validity
check, after the type check passes.
This is on the path to adding an additional validity checks parameter to
`check_parameters`, e.g. adding an 'is positive' check.
- Instead of maintaining a separate `j` value, set the bounds of the range so
that `i` is the values emitted by the range.
- Change `close_loc` to `prev_close_loc` since the market close location is used
to ensure that the data index stops at the market open if the entire day is
nans.
- Change the setting of `loc` to be done before the loop which check for nans,
instead of setting to the previous close loc at the end of the loop.
This prepares for a separate fix to prevent out of bounds access when the first
session has nans for all minutes.
When opening with a new `end_session`, i.e. opening for append, write the new
end session to the metadata.
Fixes an issue where the calendar on minute bar readers did not include the
recently appended day, causing reads on the last values to fail.
According, update append test to read a value, instead of checking table length.
Remove need for a consumer that is editing an existing minute bars directory to
reread the values which should not change from the metadata.
Add a test to the append on new day and truncate, which would be the common
usage of this method.
This format is intended for storing data for all sids of an asset type,
e.g. equities or futures for a session. bcolz is not used to avoid the overhead
of creating the directories and files for each asset (which numbers around ~8000
for active equities) can be removed since the update is meant to be read at
once, instead of supporting the random access pattern needed by the simulation.
This patch only adds the reader/writer pair, with the management of finding the
paths to delta files and the application of the updates to the bcolz write left
to internal loader code.
Also, the update reader interface is intentionally constrained to the data for
an entire session to allow for an implementation that allows for mid-session updates.
For futures that behave like GC, use the latest roll as the back contract when
walking backwards over the window, so that when the front contract is skipped
because it never has more volume between its auto close date and the previous
auto close date, the back contract which did have volume is still used when
making comparisons to construct the chain.
Instead of maintaining a separate index into the sessions index, use the `.freq`
member of the sessions index for decrementing to the current session and finding
the previous session.
When following the release guide, installing from testpypi using the
`-i` flag failed on my, and at least one other's, development machines.
The cause of the failure appears to be that pip would look for packages,
such as `LogBook` or `pandas` on `testpypi`. However many dependencies
do not have versions that meet our version criteria. (e.g. pandas does
not have a version between 0.16.0 and 0.18.0 on testpypi.)
Instead, use `--extra-index-url` so that other packages can use `pypi`
as a fallback server, instead of being limited to `testpypi`.
To support contracts such as `PL` which should roll from F->J->N->V, add the
ability to pass a predicate function to the ordered contract chain contstrution
which returns `True` if the contract is allowed in the chain.
Convert the end minute to the its session label before calling `_active_contract`,
otherwise the volume roll finder's attempt to use the session bar reader fails
due to a non-session label Timestamp.
Fix multiple errors when attempting to generate rolls for futures which do not
roll month to month, e.g. the Eurodollar.
These errors were caused by logic that always incremented from contract to
contract by delivery month, with errors when the next contract was not part of
the quarterly roll chain and thus had not yet begun trading even though the
previous contract had autoclosed. Instead, filter out these contracts and only
allow contracts that have begun trading before the previous contract's autoclose.
This is in lieu of a more explicit specification of quarterly rolls.
Instead of requiring the roll finder to juggle the indices into the ordered
contracts, use a doubly linked list where the nodes element is the contract
with members pointing to the previous and next contracts in the chain.
Besides improving legibility in the roll finder code, this change is on the path
to adding a predicate to exclude contracts from the chain, e.g. contracts in ED
which are not in the roll schedule.
Change test results for primary chain, since new implementaton does not stop at
contract in which has not yet started when constructing the chain.
Fix common error condition which was triggered whenever the session at the end
of the prefetched history window was a session where the back contract was
active. When the back contract was the active contract, the next contract for
consideration was the front contract at the end of the window, which
definitionally always has an autoclose after the end of the window.
Instead, just start seeking backwards from the end of the window.
Also prevent lookahead bias in volume rolls, which was caused by the using the
volume for a session to determine whether that session had rolled. Information
that would not have been available at the beginning of the session.
This change makes the volume rolls overly conservative, and may be improved by
looking at vectors of the preceding volume and making the roll off of momentum.
Protect a case where data is written with a non-zero volume, but a 0/nan for the
OHLC values. The slippage model was relying on a non-zero volume implying that
there was a valid trade price for the corresponding bar. When there was a mismatch,
a transaction with a nan value was created, which would in turn propagate the
nan into portfolio value, which would then cause errors when the portfolio value
was used to size orders during rebalancing.
When data is fixed, can remove.
(Also may want to add behavior to minute bar writer to ensure that 0 volumes
always have corresponding nan ohlc.)
When the following conditions occur,
- a `nan` occurred after a half day (e.g. on the Monday after
Thanksgiving, where the Friday would be a half day.)
-data was written to the span between the early close and where the market close
would have been if it were not an early close session
- a `nan` also occured on the last minute of the early market session.
the exisitng implementation would incorrectly return a `nan` when requesting a
forward filled price.
The steps that caused this error were.
1. Request for `'price'` on the market open of the day after the early close.
2. `nan` is found for that minute
3. `get_last_traded_dt` is called, and finds a volume that occurs after the
early close. e.g. `18:47` when the market close was `18:00`.
4. The minute position for `18:47` is used, when calling
`find_positon_of_minute`, since that value is after the `market_close` the
minute is set to the position of `18:00`` due to the delta logic in
5. Since there is also no data in at `18:00`, a `nan` is returned, even though
there were valid minutes earlier in the session. e.g. a non-zero volume at
`16:47` should have been used, but was not.
Fix by checking the current minute against the minute close when searching for
the last traded minute. If the minute is greater than the market close for the
corresponding day, continue the search until the minute position is within the
trading session.
This could also be fixed by enforcing that only zeros can be written between an
early close and the minute where the close would have been, but this fix allows
the reader to work with existing data.
The rolls are already calculated and assigned to `rolls_by_asset` earlier in the
`load_raw_arrays` method, so remove the duplication.
The change should not affect results.
The use of `slice_indexer` on all market minutes was taking about 110ms on my
development machine.
This change to getting the start and end indices changes the entire `_calendar`
method to take 10ms on the same machine.
Noticed while creating a `HistoryLoader` in a notebook context.
The end date of the last contract with a sufficient start date was being
used for the continuous future overall end date; however the end date of
that contract (which is the last day for which there is data for the
contract) is not necessarily the greatest end date out of all contracts.
It is possible for the furthest out contract to have some, but very
few, trades before it is more actively traded. Which would give it a
start date within in the range of the simulation, but an end date is
earlier than the other contracts which are active during the simulation.
This bug would result in `nan`s when getting the current price because
of the `end_date` check in `get_spot_value`. When the current simulation
time was greater than the `end_date` of the last contract the condition
which guards against attempting to get data for an instrument past its
end date would return a `nan`, even when the current underlying contract
did have data for that date.
Use max end date of all contracts instead of the last one, to ensure
that the continuous future last date is always great enough to allow
access to all contracts with in the chain.
Also, use min start date to accurately mirror the end date behavior.
Use `roll_style` not `roll`.
Also, add test case to cover using the session bar reader `get_value`,
by adding a test which uses `close`, since only `contract` was being
exercised, which does not exercise the session daily bar reader.
Register equities, futures, and continuous futures to an `abc` which
signifies that the type is associable with data, and thus can be used in
a history context.
May want to use this in `check_parameters` for `BarData` methods, but
work would need to be done to make sure the error message still displays
the registered types.
In preparation for using `DataPortal` in notebooks, remove restriction on
the `HistoryLoader` to dates that are monotonically increasing. Notebook
usage of the `DataPortal` is more useful when the end of the history
window can be arbitrary dates without having to restart the notebook kernel.
Due to the implementation of the prefetch and caching logic, the end
date of history calls could previously only increase. e.g. `2016-11-01`,
`2016-11-02`, `2016-11-03`. This pattern was sufficient for backtesting
and live simulations, since the current time of the algorithm only ever increases.
With this change, which resets the underlying sliding window when the
last fetched idx is greater than the
Now calls to history in the same process with end dates such
`2016-11-01`, `2016-10-31`, `2015-11-02` should work.
Allow `ContinuousFuture` when checking for a single "asset".
This could be further improved by:
- Defininng a tuple of the `Asset`-like types OR making
`ContinuousFuture` and `Asset` share a common type (whether that is
`ContinuousFuture` inheriting from `Asset` or making `Asset` and
`ContinuousFuture` share a common type.)
- Make a `history` test which uses `BarData` + `ContinuousFuture`,
instead of just using the history loader directly from tests.
To support using a `DataPortal` and `HistoryLoader` in a notebook, allow
the prefetch length to be configurable, so that it can be set to 0.
Unlike backtesting where the prefetch is useful for repeated history
windows viewed from datetimes which are monotonically increasing by a
small amount, the notebook usage of history windows needs only to
retrieve the exact data needed for the window specified.
This patch also fixes some boundary conditions related to rolls and
adjustments which were uncovered by querying for the adjustments with an
end date near the end of the window.
Instead of using the difference between the session close of the front
contract before the roll and and the open of back contract on the
beginning of the roll, use the close of both at the end of the session
before the roll.
The closes of the session prior to roll is in lieu of settlement data.
There have been cases where the requested start or end date is not in
the history calendar.
Add the beginning and of the calendar to the KeyError to give more
detail to figure out root cause.
Add roll style which takes the volume of the contracts into account.
If the volume moves from the front to the back before the auto close
date, the roll is put at that session.
Also, factors out some of the common logic shared with calendar based rolls.
Match the behavior of the minute bar reader, now that the session and
minute bar readers share a common interface.
isnull is slightly slower than checking against -1; however, n cases
where we check against illiquid trades in a tight loop, volume is
checked which is not using nan. The change here should be marginal with
regards to performance.
Move dates queried near beginning of test data so that the range of data
covered does not extend beyond the beginning of the range.
i.e. the windows were covering 2016-01-25, which had no test data generated.
(Does not matter for the calendar based rolls, but is needed for volume
based rolls.)
Also, make room for having the first roll to be a day before the first auto
close by moving the first contracts auto close date back a day.
In preparation for testing volume rolls.
The last traded dt provided from the session bar reader which resamples
from minutes should provide a dt that is a session label, not one that
is at the minute frequency.
If a KeyError occurred in the adjustment logic, the exception would be
swallowed by the try block, which was intended to just check whether or
not there was an adjustment reader adjusted.
Discovered when some logic in a futures adjustment reader were failing
because of a mismatch of minute and session labels, which resulted in no
adjustments during windows when there should have been.
The minute to session sampling reading was creating two DataFrame
objects, the first to hold the minute data, and then a second returned
by the `DataFrame.groupby` to sample down to sessions.
Instead use the arrays returned by the minute readers `load_raw_arrays`
and implement sampling logic which takes advantage that the minutes
being passed start with the first minute of the first session and end
with the last minute of the last session.
On my machine this takes the tests in `test/test_continuous_futures`
from ~4.0 to about ~0.1 seconds.
Add `.adj('mul')` and `.adj('add')` methods on ContinuousFuture, which
when used with `history`, will calculate and apply adjustments so that
the values are adjusted to account for discounts and premiums during
rolls.
Example usage in an algo:
```
from zipline.api import continuous_future
def initialize(context):
context.cl_add = continuous_future('CL', offset=0, roll='calendar').adj('add')
context.cl_mul = continuous_future('CL', offset=0, roll='calendar').adj('mul')
context.cl = continuous_future('CL', offset=0, roll='calendar')
schedule_function(print_history)
def print_history(context, data):
frame = data.history([context.cl, context.cl_add, context.cl_mul],
['price', 'sid'],
20,
'1d')
print 'unadjusted'
print frame.loc[:, :, context.cl]
print 'adjusted add'
print frame.loc[:, :, context.cl_add]
print 'adjusted mul'
print frame.loc[:, :, context.cl_mul]
```
Include minutes (in addition to the days) in the price encoding for
continuous futures tests.
Need for different values minute to minute arose when working on tests
for adjusted values.
Start making the equity adjustments calculations for the history loader
conform to the same method signature as `load_adjustments` provided by
`SQLiteAdjustmentReader, so that an `AdjustmentReader` interface can
begin to take form.
This prepares for creating a `DispatchAdjustmentReader` which will route
adjustment calculations for equities to the
`HistoryCompatibleUSEquityAdjustmentReader` and continuous futures to a
not yet implemented adjustment reader. All of these readers will share
the `load_adjustments` method.
Limit the perspective offset to 1. There is a possibility that if a
consumer of the AdjustedArrayWindow does not fetch adjustments between
the end of the data window and the vantage points beyond the end of the
window.
Until that case has a solution, e.g. having the consumer of the
AdjustedArrayWindow include the perspective offset when calculating the
query for adjustments, limit the offsets to 1.
Add a perspective offset to `AdjustedArrayWindow` and `AdjustedArray`,
so that `HistoryLoader` does not need to twiddle with offsets to support
viewing the data from the bar after end of the window, (Which is the
case when a '1d' history window is retrieved in minute mode, which is
explained in the docstring for `HistoryLoader.history`)
Presently, this simplifies the logic in
`HistoryLoader._get_adjustments_in_range`, and other incoming
AdjustmentReader's, (e.g. the roll based adjustment reader for continous
futures.) This patch should also make it easier for history and pipeline
to converge on a singular `load_adjustments` method.
Enable unadjusted history for continuous futures.
The history array is filled by the values for the underlying contracts,
where the contract used changes based on rolls.
e.g., if a `1d` history window was over the range
`2016-01-20` -> `2016-02-29` with contracts with a suffix of `F16` that
rolls at the beginning of the session on `2016-01-26`, `G16` on
`2016-02-26`, and `H16` on `2016-03-26`. The `2016-01-20` ->
`2016-01-25` portion would use the values for `F16', the `2016-01-26` ->
`2016-02-25` portion would use `G16` and the `2016-02-26` ->
`2016-02-29` portion would use `H16`.
Using the same contracts as above, a `1m` history window over the range
(using a timezone of US/Eastern) `2016-01-25 4:00PM` -> `2016-01-25
7:00PM` would fill the `4:00PM` -> `6:00PM` portion with data for `F16`
and the `6:01PM` -> `7:00PM` portion with data for `G16`, since the
beginning of the `2016-01-26` session is `2016-01-25 6:01PM`.
Supports `1d` and `1m`.
Also adds the `sid` field to `history` to assist in showing the active
contract at each dt in the window.
Add `chain`field to current, as well as supporting methods in DataPortal
and OrderedContracts.
Enables the following example:
```
from zipline.api import continuous_future
def initialize(context):
context.primary_cl = continuous_future('CL', offset=0, roll='calendar')
schedule_function(print_current_chain)
def print_current_chain(context, data):
chain = data.current_chain(context.primary_cl)
print 'datetime={0}'.format(get_datetime())
print 'primary={0}'.format(chain[0])
print 'secondary={0}'.format(chain[1])
print 'tertiary={0}'.format(chain[2])
```
```
datetime=2015-12-23 14:31:00+00:00
primary=Future(1058201602 [CLG16])
secondary=Future(1058201603 [CLH16])
tertiary=Future(1058201604 [CLJ16])
```
Also:
- make return types of OrderedContracts methods compatible across
architectures. (Noticed while adding `active_chain` method.)
- Add year suffix to future contract names in test data.
Add the ability for an algorithm to request the current contract for a
future chain via `data.current`.
e.g.:
```
data.current(ContinuousFuture('CL', offset=0, roll='calendar'),
'contract')
```
When dispatching to sub readers in dispatch reader, pass along the asset
object, instead of extracting the sid.
The in development reader for continuous futures values besides `sid`
are needed from the `ContinuousFuture` object.
`future_chain` will be replaced by the as yet to be implemented method,
`data.current_chain`
Also removing `FutureChain` which will be replaced by another version
which only supports indexing and iteration.
This reverts commit 86c7635b45, reversing
changes made to c77f2b92df.
Some real world cases hit errors with this change, due to the new offset
logic attempting to create Adjustments with invalid parameters.
Will identify exact conditions that cause this error and add as a test
case before remerging.
Instead of `HistoryLoader` containing separate adjustment calculation
logic, use `SQLiteAdjustmentReader.load_adjustments`.
This change required the addition of two offset parameters to
`load_adjustments` since the perspective on the data from within
`schedule_function` is skewed from how Pipeline looks at historical
data.
This is working towards creating an `AdjustmentReader` abc which
`SQLiteAdjustmentReader` and a upcoming continuous future adjustment
reader will share.
test_resample now fully covers the resample module.
Fix a bug exposed by increased coverage, where daily aggregation on
`high` would return `nan` for an asset instead of 1) during the
course of day `1d` history was called on non-consecutive minutes and 2)
either, a) the value for the previously inspected dt was `nan` or b)
there were only `nan`s between the previous and current dt.
`low` had a similar bug which was only triggered if the value for the
previously inspected dt was `nan`.
Increase coverage on `ReindexSessionBarReader` so that all methods which
are considered part of the interface are covered by `test_resample`.
Fix bug in `get_value`, exposed by increased coverage, where the
`NoDataOnDate` exception was bubbling from the bcolz reader all the way
up when a session which was a holidy on the underlying reader was passed
to the reindex reader. (The reindex reader should return nan/0 in that
case.)
Also, move location of data index exceptions so that they are agnostic
to bcolz/us_equity_pricing; since the exception is now used by the
resample module to fix aforementioned bug.
Remove special handling for the last session of an asset, which was
moving the last traded back a session.
If the asset has data on a session, `get_last_traded_dt` should always
return that session if it is the parameter to the method.
Add direct coverage on last_available_dt.
Also move reader creation into the instance fixture.
This patch attempted to add coverage on `get_last_traded_dt`, but in doing
so, revealed a bug in `BcolzDailyBarReader.get_last_traded_dt` when
requesting the last trading session of an asset.
When that is fixed, the skip can be removed.
Add a test to directly cover the first_trading_day method via the
`test_resample` suite. (The lack of coverage was exposed when testing
against real data.)
Also, refactor resample bar tests so that session bar reader is set up
in instance fixture.
`1d` history calls were failing on key errors when using the
`us_futures` calendar, because of timestamps occuring before a midnight
would present the wrong midnight (i.e. the midnight before the session,
instead of the following midnight, which is the label for the current
session.)
Tests will follow when bringing up coverage on resample and data portal
modules.
Combine the equity and future readers into asset dispatch readers, so
that simulations that use both asset types can access data for each.
This patch enables `history` for future assets in algorithms; however,
it does not add extra coverage in the `test_data_portal` or `test_history`
to cover future assets. Those tests will follow, however putting this in
separately since it shows that the wrapping of the readers in the asset
dispatch reader does not break existing equity strategies.
Add `AssetDispatchSessionBarReader` and corresponding minute and session
bar version of that reader.
This reader routes requests to the appropriate reader based on the asset
type of the requested sids.
`load_raw_array` in the dispatch reader batches the sid by asset type
and then interleaves the results in the out arrays, so that the arrays
data corresponds with sids in the order that sids are passed to the
method, to meet the expected behavior of `load_raw_arrays`.
The dispatch redaer is intended for use by the data portal when using
both future and equities. The dispatch reader will also be passed to the
to the `HistoryLoader`s contained within the data portal, where the
batched `load_raw_arrays` will be used.
Also, BUG:
- Fix the return of `MinuteResampleSessionBarReader.load_raw_arrays` to
match all other readers.
- Use the input dt for the `MinuteResampleSessionBarReader.load_raw_arrays`
as a session label, instead of a minute dt, since it is a session bar
reader.
(Both of these bugs where discovered when using the resample reader for
future data in the dispatch tests.)
Working towards history results which contain mixed asset types, add
a reader which makes `load_raw_arrays` return results indexed on the
session/minute ranges specified by the specified `trading_calendar`
instead of the calendar of the backing reader.
This reader will be used to make Equity readers align with Future
readers. It is intended for use as part of another reader (which will
dispatch queries based on asset type and then recombined results) which
will be passed to the `[Minute|Session]HistoryLoaders in the data portal.
The daily/session bar reader's `spot_price` took the same parameters and
returned the same kind of output as the minute bar reader's `get_value`.
Standardize on one method to make a common interface, which may be
formally factored out in a later patch; to help enable writing reader
implementations or mixins which can be agnostic to the bar frequency.
Use the equity calendar to write equity data, even when the simulation
calendar has been set to a different calendar.
Discovered when writing a test that used a calendar for future asset
data, but also wrote equity data.
In the data portal, remove methods that make a distinction between
future and equity asset type. Instead rely on the pricing reader
dispatching.
In support of incoming work which will upsample equity history arrays to
the larger future calendar.
Also, remove perf tracker tests which were using an equity
reader/writer, to be added back in later.
Implement a `SessionBarReader` which uses a minute bar reader as a
backing source, resampling the minute bars into the box around the
corresponding session data.
Also, add future/CME test cases to resample suite.
Replace `DailyBarReader` with `SessionBarReader`.
This was intended to go with the patch that added the `SessionBarReader`
abstract base class.
Also, added `trading_calendar` property decorator.
In preparation for making a resampling session bar reader, make an
abstract base class with the methods currently used by consumers of the
reader; which should assist in making a drop-in replacement of the daily
bar reader.
While pulling out the interface, it does appear that `spot_price` and
the minute bar reader's `get_value` are the same method but by different
names, showing that there may be room for having both `MinuteBarReader`
and `SessionBarReader` sharing a common `BarReader` interface.
Also, move `DailyHistoryAggregator` to `resample` module, so that tools
for converting from minute to session bars are collocated.
This patch is in preparation of adding a daily bar reader which
resamples minute data, which will be located in the `resample` module
and share the test cases and expected results in `test_resample`.
Use the future asset equity pricing reader, instead of reading directly
from the bcolz table. Required since the format for writing the future
data now uses the minute bar reader/writer pair.
Add test cases to `test_data_portal` asserting both equity and future
`get_spot_value` results.
Also, add direct coverage of last_traded_dt in the `test_data_portal`
module.
Prepares for adding test coverage of `get_last_traded_dt` for `Future` assets.
Change the mock minute data to no longer use an increasing arange, so
that a days worth of minute data can be summed and fit inside of a
uint32.
This change was required because of working on new test data that looked
like [0, 100, 200, 0, ] which was resulting in a daily rollup of 0 data,
when the coverage needed a non-0 value.
Also, factor out the resampling function, with an eye on a making it
easier to convert from minute bars to daily bars during ingest/load
processes.
When adding fixtures for futures data, there will be a need for multiple
calendars in the fixture ecosystem. e.g. a test that includes both
equities and futures would need an overall calendar which encompasses
both equities and futures; however, the test data for equities should
still still be limited to the bounds set by the NYSE calendar.
Make the fixtures that setup trading calendars and values dervied from
the trading calendar (e.g. trading sessions) accept an iterable of
calendars which need to be created, then populate those values into a
dict keyed by the calendar name.
Change `WithNYSETradingDays` to include sessions in the name,
since we are moving to session as the name for the 'day' unit.
Provide `trading_days` which is really "NYSE trading sessions` on
`WithTradingSessions` for backwards compatibility.
Eddie Hebert