This modificaiton to the estimates loader allows the caller to pass
in an equity pricing loader which can then be used to get split data
for sids. That split data is then used to do point-in-time adjustments
of estimates data.
TST: add test for multiple estimates columns
TST: add test for multiple datasets requesting different columns
TST: add blaze versions for all next/previous tests
Protect a case where data is written with a non-zero volume, but a 0/nan for the
OHLC values. The slippage model was relying on a non-zero volume implying that
there was a valid trade price for the corresponding bar. When there was a mismatch,
a transaction with a nan value was created, which would in turn propagate the
nan into portfolio value, which would then cause errors when the portfolio value
was used to size orders during rebalancing.
When data is fixed, can remove.
(Also may want to add behavior to minute bar writer to ensure that 0 volumes
always have corresponding nan ohlc.)
When the following conditions occur,
- a `nan` occurred after a half day (e.g. on the Monday after
Thanksgiving, where the Friday would be a half day.)
-data was written to the span between the early close and where the market close
would have been if it were not an early close session
- a `nan` also occured on the last minute of the early market session.
the exisitng implementation would incorrectly return a `nan` when requesting a
forward filled price.
The steps that caused this error were.
1. Request for `'price'` on the market open of the day after the early close.
2. `nan` is found for that minute
3. `get_last_traded_dt` is called, and finds a volume that occurs after the
early close. e.g. `18:47` when the market close was `18:00`.
4. The minute position for `18:47` is used, when calling
`find_positon_of_minute`, since that value is after the `market_close` the
minute is set to the position of `18:00`` due to the delta logic in
5. Since there is also no data in at `18:00`, a `nan` is returned, even though
there were valid minutes earlier in the session. e.g. a non-zero volume at
`16:47` should have been used, but was not.
Fix by checking the current minute against the minute close when searching for
the last traded minute. If the minute is greater than the market close for the
corresponding day, continue the search until the minute position is within the
trading session.
This could also be fixed by enforcing that only zeros can be written between an
early close and the minute where the close would have been, but this fix allows
the reader to work with existing data.
The rolls are already calculated and assigned to `rolls_by_asset` earlier in the
`load_raw_arrays` method, so remove the duplication.
The change should not affect results.
The use of `slice_indexer` on all market minutes was taking about 110ms on my
development machine.
This change to getting the start and end indices changes the entire `_calendar`
method to take 10ms on the same machine.
Noticed while creating a `HistoryLoader` in a notebook context.
The end date of the last contract with a sufficient start date was being
used for the continuous future overall end date; however the end date of
that contract (which is the last day for which there is data for the
contract) is not necessarily the greatest end date out of all contracts.
It is possible for the furthest out contract to have some, but very
few, trades before it is more actively traded. Which would give it a
start date within in the range of the simulation, but an end date is
earlier than the other contracts which are active during the simulation.
This bug would result in `nan`s when getting the current price because
of the `end_date` check in `get_spot_value`. When the current simulation
time was greater than the `end_date` of the last contract the condition
which guards against attempting to get data for an instrument past its
end date would return a `nan`, even when the current underlying contract
did have data for that date.
Use max end date of all contracts instead of the last one, to ensure
that the continuous future last date is always great enough to allow
access to all contracts with in the chain.
Also, use min start date to accurately mirror the end date behavior.
Use `roll_style` not `roll`.
Also, add test case to cover using the session bar reader `get_value`,
by adding a test which uses `close`, since only `contract` was being
exercised, which does not exercise the session daily bar reader.
Register equities, futures, and continuous futures to an `abc` which
signifies that the type is associable with data, and thus can be used in
a history context.
May want to use this in `check_parameters` for `BarData` methods, but
work would need to be done to make sure the error message still displays
the registered types.
In preparation for using `DataPortal` in notebooks, remove restriction on
the `HistoryLoader` to dates that are monotonically increasing. Notebook
usage of the `DataPortal` is more useful when the end of the history
window can be arbitrary dates without having to restart the notebook kernel.
Due to the implementation of the prefetch and caching logic, the end
date of history calls could previously only increase. e.g. `2016-11-01`,
`2016-11-02`, `2016-11-03`. This pattern was sufficient for backtesting
and live simulations, since the current time of the algorithm only ever increases.
With this change, which resets the underlying sliding window when the
last fetched idx is greater than the
Now calls to history in the same process with end dates such
`2016-11-01`, `2016-10-31`, `2015-11-02` should work.
Allow `ContinuousFuture` when checking for a single "asset".
This could be further improved by:
- Defininng a tuple of the `Asset`-like types OR making
`ContinuousFuture` and `Asset` share a common type (whether that is
`ContinuousFuture` inheriting from `Asset` or making `Asset` and
`ContinuousFuture` share a common type.)
- Make a `history` test which uses `BarData` + `ContinuousFuture`,
instead of just using the history loader directly from tests.
To support using a `DataPortal` and `HistoryLoader` in a notebook, allow
the prefetch length to be configurable, so that it can be set to 0.
Unlike backtesting where the prefetch is useful for repeated history
windows viewed from datetimes which are monotonically increasing by a
small amount, the notebook usage of history windows needs only to
retrieve the exact data needed for the window specified.
This patch also fixes some boundary conditions related to rolls and
adjustments which were uncovered by querying for the adjustments with an
end date near the end of the window.
Maya Tydykov