Previously, a dataframe passed into BcolzDailyBarWriter.write that was
missing an expected session between its first and last sessions would be
written incorrectly. Upon converting the dataframe to a ctable, the
values for all days following the gap would be shifted backwards, and
nans would be shifted in at the end.
This commit handles the issue by asserting that the number of rows in
the input table matches the number of sessions in the calendar between
the table's first and last sessions.
Also fixes a test that was mistakenly using minutes_in_range where it
should have been using sessions_in_range (uncovered by this change).
Instead of recursively calling `DailyHistoryAggregator.closes` until we
find a non-nan close, we can instead call `load_raw_arrays` once, and
find the value from the returned array.
When opening with a new `end_session`, i.e. opening for append, write the new
end session to the metadata.
Fixes an issue where the calendar on minute bar readers did not include the
recently appended day, causing reads on the last values to fail.
According, update append test to read a value, instead of checking table length.
Remove need for a consumer that is editing an existing minute bars directory to
reread the values which should not change from the metadata.
Add a test to the append on new day and truncate, which would be the common
usage of this method.
This format is intended for storing data for all sids of an asset type,
e.g. equities or futures for a session. bcolz is not used to avoid the overhead
of creating the directories and files for each asset (which numbers around ~8000
for active equities) can be removed since the update is meant to be read at
once, instead of supporting the random access pattern needed by the simulation.
This patch only adds the reader/writer pair, with the management of finding the
paths to delta files and the application of the updates to the bcolz write left
to internal loader code.
Also, the update reader interface is intentionally constrained to the data for
an entire session to allow for an implementation that allows for mid-session updates.
When the following conditions occur,
- a `nan` occurred after a half day (e.g. on the Monday after
Thanksgiving, where the Friday would be a half day.)
-data was written to the span between the early close and where the market close
would have been if it were not an early close session
- a `nan` also occured on the last minute of the early market session.
the exisitng implementation would incorrectly return a `nan` when requesting a
forward filled price.
The steps that caused this error were.
1. Request for `'price'` on the market open of the day after the early close.
2. `nan` is found for that minute
3. `get_last_traded_dt` is called, and finds a volume that occurs after the
early close. e.g. `18:47` when the market close was `18:00`.
4. The minute position for `18:47` is used, when calling
`find_positon_of_minute`, since that value is after the `market_close` the
minute is set to the position of `18:00`` due to the delta logic in
5. Since there is also no data in at `18:00`, a `nan` is returned, even though
there were valid minutes earlier in the session. e.g. a non-zero volume at
`16:47` should have been used, but was not.
Fix by checking the current minute against the minute close when searching for
the last traded minute. If the minute is greater than the market close for the
corresponding day, continue the search until the minute position is within the
trading session.
This could also be fixed by enforcing that only zeros can be written between an
early close and the minute where the close would have been, but this fix allows
the reader to work with existing data.
Match the behavior of the minute bar reader, now that the session and
minute bar readers share a common interface.
isnull is slightly slower than checking against -1; however, n cases
where we check against illiquid trades in a tight loop, volume is
checked which is not using nan. The change here should be marginal with
regards to performance.
The last traded dt provided from the session bar reader which resamples
from minutes should provide a dt that is a session label, not one that
is at the minute frequency.
The minute to session sampling reading was creating two DataFrame
objects, the first to hold the minute data, and then a second returned
by the `DataFrame.groupby` to sample down to sessions.
Instead use the arrays returned by the minute readers `load_raw_arrays`
and implement sampling logic which takes advantage that the minutes
being passed start with the first minute of the first session and end
with the last minute of the last session.
On my machine this takes the tests in `test/test_continuous_futures`
from ~4.0 to about ~0.1 seconds.
With the addition of the truncate function, there are cases where we'll
want to construct a BcolzMinuteBarWriter to call truncate, without
gathering all the metadata. This commit adds a write_metadata arg to its
init, which is True by default. If False is specified, no metadata is
written.
Requires adding logic to truncate to update end_session in metadata to
the truncate date.
Remove module scope invocations of `get_calendar('NYSE')`, which cuts
zipline import time in half on my machine. This make the zipline CLI
noticeably more responsive, and it reduces memory consumed at import
time from 130MB to 90MB.
Before:
$ time python -c 'import zipline'
real 0m1.262s
user 0m1.128s
sys 0m0.120s
After:
$ time python -c 'import zipline'
real 0m0.676s
user 0m0.536s
sys 0m0.132s
test_resample now fully covers the resample module.
Fix a bug exposed by increased coverage, where daily aggregation on
`high` would return `nan` for an asset instead of 1) during the
course of day `1d` history was called on non-consecutive minutes and 2)
either, a) the value for the previously inspected dt was `nan` or b)
there were only `nan`s between the previous and current dt.
`low` had a similar bug which was only triggered if the value for the
previously inspected dt was `nan`.
Increase coverage on `ReindexSessionBarReader` so that all methods which
are considered part of the interface are covered by `test_resample`.
Fix bug in `get_value`, exposed by increased coverage, where the
`NoDataOnDate` exception was bubbling from the bcolz reader all the way
up when a session which was a holidy on the underlying reader was passed
to the reindex reader. (The reindex reader should return nan/0 in that
case.)
Also, move location of data index exceptions so that they are agnostic
to bcolz/us_equity_pricing; since the exception is now used by the
resample module to fix aforementioned bug.
Remove special handling for the last session of an asset, which was
moving the last traded back a session.
If the asset has data on a session, `get_last_traded_dt` should always
return that session if it is the parameter to the method.
Add direct coverage on last_available_dt.
Also move reader creation into the instance fixture.
This patch attempted to add coverage on `get_last_traded_dt`, but in doing
so, revealed a bug in `BcolzDailyBarReader.get_last_traded_dt` when
requesting the last trading session of an asset.
When that is fixed, the skip can be removed.
Add a test to directly cover the first_trading_day method via the
`test_resample` suite. (The lack of coverage was exposed when testing
against real data.)
Also, refactor resample bar tests so that session bar reader is set up
in instance fixture.
Add `AssetDispatchSessionBarReader` and corresponding minute and session
bar version of that reader.
This reader routes requests to the appropriate reader based on the asset
type of the requested sids.
`load_raw_array` in the dispatch reader batches the sid by asset type
and then interleaves the results in the out arrays, so that the arrays
data corresponds with sids in the order that sids are passed to the
method, to meet the expected behavior of `load_raw_arrays`.
The dispatch redaer is intended for use by the data portal when using
both future and equities. The dispatch reader will also be passed to the
to the `HistoryLoader`s contained within the data portal, where the
batched `load_raw_arrays` will be used.
Also, BUG:
- Fix the return of `MinuteResampleSessionBarReader.load_raw_arrays` to
match all other readers.
- Use the input dt for the `MinuteResampleSessionBarReader.load_raw_arrays`
as a session label, instead of a minute dt, since it is a session bar
reader.
(Both of these bugs where discovered when using the resample reader for
future data in the dispatch tests.)
Working towards history results which contain mixed asset types, add
a reader which makes `load_raw_arrays` return results indexed on the
session/minute ranges specified by the specified `trading_calendar`
instead of the calendar of the backing reader.
This reader will be used to make Equity readers align with Future
readers. It is intended for use as part of another reader (which will
dispatch queries based on asset type and then recombined results) which
will be passed to the `[Minute|Session]HistoryLoaders in the data portal.
The daily/session bar reader's `spot_price` took the same parameters and
returned the same kind of output as the minute bar reader's `get_value`.
Standardize on one method to make a common interface, which may be
formally factored out in a later patch; to help enable writing reader
implementations or mixins which can be agnostic to the bar frequency.
We were mistakenly using the minute_per_day field.
We now expose from the metadata object the version from which the
metadata was read. This allows a new test that verifies the version is
read correctly.
* First pass.
* Improvements and fixes
- Update usages of BcolzMinuteBarWriter
- Updates with rebuilt example data
- Expose calendar from BcolzMinuteBarMetadata instead of calendar_name
- Keep market_opens and market_closes in metadata for compatibility
* Store start_session and end_session in minute bcolz metadata
- start_session replaces first_trading_day
- Add end_session to limit to correct days
* For last_available_dt, get last close from calendar to maintain tz
* Bumps version and handles earlier versionson read
* Rebuilt example data on python 3
* Indicate metadata fields that are deprecated
Implement a `SessionBarReader` which uses a minute bar reader as a
backing source, resampling the minute bars into the box around the
corresponding session data.
Also, add future/CME test cases to resample suite.
* MAINT: Use TradingCalendar objects for bundles
Instead of trading days, opens, and closes, register now takes a
TradingCalendar object, along with a start_session and end_session. The
ingest function is now passed these values instead as well.
* Accept calendar name in addition to the actual object
* Updates bundles documentation for changes
* Fix typo in docs
* Use class formatting
* Force start_session and end_session within the bounds of the calendar
* Use UTC timestamps in test_core
* Document Trading Calendar API in appendix.rst
Also, move `DailyHistoryAggregator` to `resample` module, so that tools
for converting from minute to session bars are collocated.
This patch is in preparation of adding a daily bar reader which
resamples minute data, which will be located in the `resample` module
and share the test cases and expected results in `test_resample`.
When adding fixtures for futures data, there will be a need for multiple
calendars in the fixture ecosystem. e.g. a test that includes both
equities and futures would need an overall calendar which encompasses
both equities and futures; however, the test data for equities should
still still be limited to the bounds set by the NYSE calendar.
Make the fixtures that setup trading calendars and values dervied from
the trading calendar (e.g. trading sessions) accept an iterable of
calendars which need to be created, then populate those values into a
dict keyed by the calendar name.
Change `WithNYSETradingDays` to include sessions in the name,
since we are moving to session as the name for the 'day' unit.
Provide `trading_days` which is really "NYSE trading sessions` on
`WithTradingSessions` for backwards compatibility.
Changes the overlap behavior so that it is an error to write data which
would have two companies holding the same ticker. Other than one test
around which company would win in that case, all the other tests are
passing. That single test has been changed to check the write-time
error.
is backwards-compatible with the previous format.
In USEquityLoader, use dailyreader's trading_calendar.
This is backwards compatible and will fall back to the NYSE calendar if
the reader doesn’t have a calendar specified.
Instead of having separate ExchangeCalendar and TradingSchedule objects, we
now just have TradingCalendar. The TradingCalendar keeps track of each
session (defined as a contiguous set of minutes between an open and a close).
It's also responsible for handling the grouping logic of any given minute
to its containing session, or the next/previous session if it's not a market
minute for the given calendar.
Conner Fromknecht