Overhaul the core HistoryContainer logic to be more robust to changing
universes.
Major Changes
-------------
* Remove `return_frame` cache. The original purpose of using
return_frames was to avoid having to create new DataFrames on each
iteration of handle_data, but we ended up having to copy the return
frames anyway because user code could mutate the frames in place.
Removing the return_frames reduces unnecessary copying, and reduces
the logic of `get_history` to just forward-filling and concatenating
two DataFrames.
* Use a `MultiIndex`ed DataFrame to represent
`last_known_prior_values`. This makes lookups faster and greatly
simplifies the logic of adding and dropping sids.
* HistoryContainer no longer attempts to determine its universe based on
the contents of its internal buffers. The TradingAlgorithm
controlling the container is now responsible for explicitly calling
`add_sids` or `drop_sids` when securities enter or leave the
algorithm's universe. These methods, along with the internal
`_realign` method, provide a clean interface for changing the universe
of securities managed by the container.
* Refactor index mutation logic in `RollingPanel` into a
`MutableIndexRollingPanel` subclass. Maintenance of the old behavior
is regrettably necessary to support `BatchTransform`.
* Refactor shared logic from `roll` and `get_history` into a single
`aggregate_ohlcv_panel` method that's responsible for collapsing an
OHLCV buffer into a frame.
In support of source that emits a subclass of Event which defines some
fields as properties instead of doubling the value in the
`Event.__dict__`
Use hasattr instead of the overridden __contains__ method of the Event
class, so that when non-algorithm facing code checks for field existence,
properties count.
Intentionally not touching the `__contains__` in Event, to avoid
changing, at the moment, any algo behavior that relies on the
`__contains__` behavior's use of `__dict__`
This commit adds support for arbitrary objects in addition to NaN
and infinity values. The object well be returned in string format
as part of the error message.
Previously order was not checking for nan values sent as
limit or stop prices. It will now raise a runtime exception
in the event that an attempt to order with a nan price is made.
We guarantee that `TradingAlgorithm.updated_portfolio` is called at least once
between each perf message emission. This change moves that guaranteed call
into `AlgorithmSimulator.get_message` and adds a comment to make it clear why
updated_portfolio is being called.
Removes unused method `raw_positions` and `raw_orders` from `TradingAlgorithm`.
These methods existed to help support features in the Quantopian codebase,
which no longer use the existing interface.
Fixes a crash in various transforms when providing CUSTOM events whose fields
don't match the fields required for the transform.
This is fixed by requiring all `EventWindow` subclasses to supply a `fields`
property, which returns a list of strings that are required keys for any event
that can be processed by the window. Any CUSTOM events the don't supply the
required fields for a transform window are ignored by that window.
Previously the last sale price was not correctly being set on
positions when the transaction arrived before the trade event.
The last sale price was defaulted to zero and never updated. This resulted
in one holding stocks that were bough >>0 and now had value 0 from
the perspective of returns. The returns would display correctly again
when the next trade of that security happened. For most securities trading is
frequent enough that there's no issue, but for some illiquid ones it took
hours to fix itself.
Updated test_perf_tracking:TestPerformanceTracker.test_minute_tracker
This test was based on assuming that last_sale_price was zero,
allowing the sharpe ratio to be calculated. The sharpe ratio can no longer
be calculated for this specific tested scenario and the test has been changed
accordingly.
Scott Sanderson