This format is intended for storing data for all sids of an asset type,
e.g. equities or futures for a session. bcolz is not used to avoid the overhead
of creating the directories and files for each asset (which numbers around ~8000
for active equities) can be removed since the update is meant to be read at
once, instead of supporting the random access pattern needed by the simulation.
This patch only adds the reader/writer pair, with the management of finding the
paths to delta files and the application of the updates to the bcolz write left
to internal loader code.
Also, the update reader interface is intentionally constrained to the data for
an entire session to allow for an implementation that allows for mid-session updates.
* ENH: Adds support for supplementary asset mappings
- Adds a supplementary_mappings table to the assets.db, to hold point-
in-time mappings of sids to arbitrary categories of values, e.g.
alternative identifiers. This bumps ASSET_DB_VERSION.
- Adds supplementary_map and supplementary_map_by_sid to AssetFinder,
caches of the underlying table that are fully populated on first
access, which map the supplementary values to sids and vice versa,
respectively.
- Adds lookup_by_supplementary_mapping method, which fronts
supplementary_map to query for the asset last known to have held a
value at a given dt.
- Add get_supplementary_field method, which fronts
supplementary_map_by_sid to query for the last known value held by an
asset at a given dt.
For futures that behave like GC, use the latest roll as the back contract when
walking backwards over the window, so that when the front contract is skipped
because it never has more volume between its auto close date and the previous
auto close date, the back contract which did have volume is still used when
making comparisons to construct the chain.
Instead of maintaining a separate index into the sessions index, use the `.freq`
member of the sessions index for decrementing to the current session and finding
the previous session.
To support contracts such as `PL` which should roll from F->J->N->V, add the
ability to pass a predicate function to the ordered contract chain contstrution
which returns `True` if the contract is allowed in the chain.
Convert the end minute to the its session label before calling `_active_contract`,
otherwise the volume roll finder's attempt to use the session bar reader fails
due to a non-session label Timestamp.
Fix multiple errors when attempting to generate rolls for futures which do not
roll month to month, e.g. the Eurodollar.
These errors were caused by logic that always incremented from contract to
contract by delivery month, with errors when the next contract was not part of
the quarterly roll chain and thus had not yet begun trading even though the
previous contract had autoclosed. Instead, filter out these contracts and only
allow contracts that have begun trading before the previous contract's autoclose.
This is in lieu of a more explicit specification of quarterly rolls.
Instead of requiring the roll finder to juggle the indices into the ordered
contracts, use a doubly linked list where the nodes element is the contract
with members pointing to the previous and next contracts in the chain.
Besides improving legibility in the roll finder code, this change is on the path
to adding a predicate to exclude contracts from the chain, e.g. contracts in ED
which are not in the roll schedule.
Change test results for primary chain, since new implementaton does not stop at
contract in which has not yet started when constructing the chain.
- Use `expect_bounded` to check inputs.
- Add tests for expected failures from `MACDSignal`.
- Use `float64` instead of `float` in a few places. This prevents
diverging behavior on 32-bit systems.
- Docstring edits.
Previously, if input to the BcolzMinuteBarWriter had the first bar on a
non-trading minute, the next trading session would be considered the
"first day" in the input. Now, we consider the previous trading session
the "first day".
The intention is to correctly associate minutes after official trading
hours on half days with session that closed early, not the following
session (a future improvement here would be to not accept minutes
outside trading hours).
Fix common error condition which was triggered whenever the session at the end
of the prefetched history window was a session where the back contract was
active. When the back contract was the active contract, the next contract for
consideration was the front contract at the end of the window, which
definitionally always has an autoclose after the end of the window.
Instead, just start seeking backwards from the end of the window.
Also prevent lookahead bias in volume rolls, which was caused by the using the
volume for a session to determine whether that session had rolled. Information
that would not have been available at the beginning of the session.
This change makes the volume rolls overly conservative, and may be improved by
looking at vectors of the preceding volume and making the roll off of momentum.
Any DataFrame that's had `.loc` or `.iloc `called on it participates in
a cycle, which means they're not immediately garbage collected when they
go out of scope. This matters for pipeline results because they consume
multiple megabytes per column, which means that a pipeline result with
many columns can hold take up over 100MB. By manually breaking
DataFrame cycles, we can ensure that we never hold multiple pipeline
results in memory at once.
Otherwise, we either raise an exception or filter out all unsafe values.
This addresses an issue where the BcolzMinuteBarWriter would scale up
values to convert to uint32, but the resulting values were too large,
and would be mangled.
Based on the approach we take in the BcolzDailyBarWriter.
This modificaiton to the estimates loader allows the caller to pass
in an equity pricing loader which can then be used to get split data
for sids. That split data is then used to do point-in-time adjustments
of estimates data.
TST: add test for multiple estimates columns
TST: add test for multiple datasets requesting different columns
TST: add blaze versions for all next/previous tests
Protect a case where data is written with a non-zero volume, but a 0/nan for the
OHLC values. The slippage model was relying on a non-zero volume implying that
there was a valid trade price for the corresponding bar. When there was a mismatch,
a transaction with a nan value was created, which would in turn propagate the
nan into portfolio value, which would then cause errors when the portfolio value
was used to size orders during rebalancing.
When data is fixed, can remove.
(Also may want to add behavior to minute bar writer to ensure that 0 volumes
always have corresponding nan ohlc.)
When the following conditions occur,
- a `nan` occurred after a half day (e.g. on the Monday after
Thanksgiving, where the Friday would be a half day.)
-data was written to the span between the early close and where the market close
would have been if it were not an early close session
- a `nan` also occured on the last minute of the early market session.
the exisitng implementation would incorrectly return a `nan` when requesting a
forward filled price.
The steps that caused this error were.
1. Request for `'price'` on the market open of the day after the early close.
2. `nan` is found for that minute
3. `get_last_traded_dt` is called, and finds a volume that occurs after the
early close. e.g. `18:47` when the market close was `18:00`.
4. The minute position for `18:47` is used, when calling
`find_positon_of_minute`, since that value is after the `market_close` the
minute is set to the position of `18:00`` due to the delta logic in
5. Since there is also no data in at `18:00`, a `nan` is returned, even though
there were valid minutes earlier in the session. e.g. a non-zero volume at
`16:47` should have been used, but was not.
Fix by checking the current minute against the minute close when searching for
the last traded minute. If the minute is greater than the market close for the
corresponding day, continue the search until the minute position is within the
trading session.
This could also be fixed by enforcing that only zeros can be written between an
early close and the minute where the close would have been, but this fix allows
the reader to work with existing data.
Eddie Hebert