This allows us to remove the check for whether the provided dt had a
time of midnight, which was a flimsy way to infer if the data frequency
was 'daily'. Besides the explicit check being preferable, this method
was broken on the futures calendar, since midnight is a valid market
minute.
Previously, a dataframe passed into BcolzDailyBarWriter.write that was
missing an expected session between its first and last sessions would be
written incorrectly. Upon converting the dataframe to a ctable, the
values for all days following the gap would be shifted backwards, and
nans would be shifted in at the end.
This commit handles the issue by asserting that the number of rows in
the input table matches the number of sessions in the calendar between
the table's first and last sessions.
Also fixes a test that was mistakenly using minutes_in_range where it
should have been using sessions_in_range (uncovered by this change).
Optimize session close lookups in MinuteResampleSessionBarReader:
- Adds `session_closes_in_range` method (along with
`session_opens_in_range`) to TradingCalendar to allow vectorized
retrieval of all values in a range of sessions.
- Improves code path for resampling a single session's worth of data (as
is the case when calling `get_value`), since we don't actually need to
look up the close minute.
Instead of recursively calling `DailyHistoryAggregator.closes` until we
find a non-nan close, we can instead call `load_raw_arrays` once, and
find the value from the returned array.
- Instead of maintaining a separate `j` value, set the bounds of the range so
that `i` is the values emitted by the range.
- Change `close_loc` to `prev_close_loc` since the market close location is used
to ensure that the data index stops at the market open if the entire day is
nans.
- Change the setting of `loc` to be done before the loop which check for nans,
instead of setting to the previous close loc at the end of the loop.
This prepares for a separate fix to prevent out of bounds access when the first
session has nans for all minutes.
Fixes a bug where we'd fail to raise an error if the start/end of a
history window call don't aren't in the loader's calendar.
We were started dropping this error after a previous change swapped out
calls to `index.get_loc` with calls to `index.searchsorted` to avoid
creating hash tables in pandas.
When opening with a new `end_session`, i.e. opening for append, write the new
end session to the metadata.
Fixes an issue where the calendar on minute bar readers did not include the
recently appended day, causing reads on the last values to fail.
According, update append test to read a value, instead of checking table length.
Remove need for a consumer that is editing an existing minute bars directory to
reread the values which should not change from the metadata.
Add a test to the append on new day and truncate, which would be the common
usage of this method.
From pep-0008:
```
Always use a def statement instead of an assignment statement that binds a
lambda expression directly to an identifier.
Yes:
def f(x): return 2*x
No:
f = lambda x: 2*x
The first form means that the name of the resulting function object is
specifically 'f' instead of the generic '<lambda>'. This is more useful for
tracebacks and string representations in general. The use of the assignment
statement eliminates the sole benefit a lambda expression can offer over an
explicit def statement (i.e. that it can be embedded inside a larger expression)
```
This format is intended for storing data for all sids of an asset type,
e.g. equities or futures for a session. bcolz is not used to avoid the overhead
of creating the directories and files for each asset (which numbers around ~8000
for active equities) can be removed since the update is meant to be read at
once, instead of supporting the random access pattern needed by the simulation.
This patch only adds the reader/writer pair, with the management of finding the
paths to delta files and the application of the updates to the bcolz write left
to internal loader code.
Also, the update reader interface is intentionally constrained to the data for
an entire session to allow for an implementation that allows for mid-session updates.
Previously, if input to the BcolzMinuteBarWriter had the first bar on a
non-trading minute, the next trading session would be considered the
"first day" in the input. Now, we consider the previous trading session
the "first day".
The intention is to correctly associate minutes after official trading
hours on half days with session that closed early, not the following
session (a future improvement here would be to not accept minutes
outside trading hours).
Otherwise, we either raise an exception or filter out all unsafe values.
This addresses an issue where the BcolzMinuteBarWriter would scale up
values to convert to uint32, but the resulting values were too large,
and would be mangled.
Based on the approach we take in the BcolzDailyBarWriter.
When the following conditions occur,
- a `nan` occurred after a half day (e.g. on the Monday after
Thanksgiving, where the Friday would be a half day.)
-data was written to the span between the early close and where the market close
would have been if it were not an early close session
- a `nan` also occured on the last minute of the early market session.
the exisitng implementation would incorrectly return a `nan` when requesting a
forward filled price.
The steps that caused this error were.
1. Request for `'price'` on the market open of the day after the early close.
2. `nan` is found for that minute
3. `get_last_traded_dt` is called, and finds a volume that occurs after the
early close. e.g. `18:47` when the market close was `18:00`.
4. The minute position for `18:47` is used, when calling
`find_positon_of_minute`, since that value is after the `market_close` the
minute is set to the position of `18:00`` due to the delta logic in
5. Since there is also no data in at `18:00`, a `nan` is returned, even though
there were valid minutes earlier in the session. e.g. a non-zero volume at
`16:47` should have been used, but was not.
Fix by checking the current minute against the minute close when searching for
the last traded minute. If the minute is greater than the market close for the
corresponding day, continue the search until the minute position is within the
trading session.
This could also be fixed by enforcing that only zeros can be written between an
early close and the minute where the close would have been, but this fix allows
the reader to work with existing data.
The rolls are already calculated and assigned to `rolls_by_asset` earlier in the
`load_raw_arrays` method, so remove the duplication.
The change should not affect results.
The use of `slice_indexer` on all market minutes was taking about 110ms on my
development machine.
This change to getting the start and end indices changes the entire `_calendar`
method to take 10ms on the same machine.
Noticed while creating a `HistoryLoader` in a notebook context.
Use `roll_style` not `roll`.
Also, add test case to cover using the session bar reader `get_value`,
by adding a test which uses `close`, since only `contract` was being
exercised, which does not exercise the session daily bar reader.
In preparation for using `DataPortal` in notebooks, remove restriction on
the `HistoryLoader` to dates that are monotonically increasing. Notebook
usage of the `DataPortal` is more useful when the end of the history
window can be arbitrary dates without having to restart the notebook kernel.
Due to the implementation of the prefetch and caching logic, the end
date of history calls could previously only increase. e.g. `2016-11-01`,
`2016-11-02`, `2016-11-03`. This pattern was sufficient for backtesting
and live simulations, since the current time of the algorithm only ever increases.
With this change, which resets the underlying sliding window when the
last fetched idx is greater than the
Now calls to history in the same process with end dates such
`2016-11-01`, `2016-10-31`, `2015-11-02` should work.
Ana Ruelas