Limited use of `pandas` data structures in both `HistoryContainer` and
`RollingPanel`. Where possible, methods were amended to return raw
`ndarrays` with the indexing logic done separately. This allows us to
cut down the number of times pandas objects are created both as returns
and intermediate values. The separation of indexing from data access
allowed us to minimize the times we’d make use of pandas indexes.
This required that that certain methods like `NDFrame.ffill` be replaced
with versions that work with `ndarrays`. Some of this was done via
straight numpy methods and others by access pandas internal
machinery. Outside of allowing us to use faster ndarrays, many of these
function provided speedups over their pandas counterparts as we didn’t
require the extra features like handling multiple dtypes. i.e. np.isnan
is faster than pd.isnull, but only works with certain dtypes.
contstruction.
BarData can be falsey. in create_buffer_panel, the intention of the
check against bar_data was to see if it was passed at all, not if it was
truthey. In order to make that check more explicit, the check now
asserts that bar_data is not None.
When a history minute digest panel was constructed at runtime, there
would be 1 minute missing between the current minute at iloc[-1] and the
next prior minute at iloc[-2].
getting filled with the wrong datetimes and causing errors.
Updates the logic for addressing missing datetimes and adds unit tests
for the 2 main cases (no missing datetimes, and some missing datetimes).
Previously, all specs had to be pre-allocated by using the 'add_history'
function. This is now no longer required and instead serves as a hint to
the HistoryContainer to pre-allocate the space for the given spec.
History can grow by increasing the length for a frequency, adding a
frequency, or adding a field. It can grow with any combination of
these.
HistoryContainer now is aware of the data_frequency of the algorithm,
and no longer uses the daily_at_midnight flag; instead, this is the
default behavior.
Uses a numpy array instead of a dict of dicts when initializing history
container.
In testing this reduced the total time spent in HistoryContainer.update
by 66%.
BEFORE COMMIT:
Thu Oct 16 22:30:46 2014 results/cprofile/unoptimized
185223320 function calls (182210491 primitive calls) in 401.351
seconds
Ordered by: cumulative time
List reduced from 2398 to 27 due to restriction <'update'>
ncalls tottime percall cumtime percall filename:lineno(function)
8580 0.461 0.000 160.571 0.019
qexec/zipline/history/history_container.py:388(update)
AFTER COMMIT:
Thu Oct 16 22:12:28 2014 results/cprofile/optimized
143177181 function calls (140164352 primitive calls) in 272.403
seconds
Ordered by: cumulative time
List reduced from 2395 to 27 due to restriction <'update'>
ncalls tottime percall cumtime percall filename:lineno(function)
8580 0.086 0.000 47.294 0.006 qexec/zipline/history/history_container.py:388(update)
Overhaul the core HistoryContainer logic to be more robust to changing
universes.
Major Changes
-------------
* Remove `return_frame` cache. The original purpose of using
return_frames was to avoid having to create new DataFrames on each
iteration of handle_data, but we ended up having to copy the return
frames anyway because user code could mutate the frames in place.
Removing the return_frames reduces unnecessary copying, and reduces
the logic of `get_history` to just forward-filling and concatenating
two DataFrames.
* Use a `MultiIndex`ed DataFrame to represent
`last_known_prior_values`. This makes lookups faster and greatly
simplifies the logic of adding and dropping sids.
* HistoryContainer no longer attempts to determine its universe based on
the contents of its internal buffers. The TradingAlgorithm
controlling the container is now responsible for explicitly calling
`add_sids` or `drop_sids` when securities enter or leave the
algorithm's universe. These methods, along with the internal
`_realign` method, provide a clean interface for changing the universe
of securities managed by the container.
* Refactor index mutation logic in `RollingPanel` into a
`MutableIndexRollingPanel` subclass. Maintenance of the old behavior
is regrettably necessary to support `BatchTransform`.
* Refactor shared logic from `roll` and `get_history` into a single
`aggregate_ohlcv_panel` method that's responsible for collapsing an
OHLCV buffer into a frame.
We can't be sure that security identifiers can be converted
to int. In fact, most of the times they are strings. This
adds an identity function that can be overwritten if such
a conversion is necessary (as on Quantopian).
Fixes an issue that caused a crash if a user assigned a new column into a
returned history DataFrame. This occurred because we re-use DataFrames between
history() calls, so the new column caused an index-size mismatch on the next
attempted calculation.
Ideally we would fix this by in-place dropping the columns, but that isn't
supported in pandas 0.12.0, so instead we just return a complete copy of the
frame. We should re-evaluate this implementation when we're on a more modern
pandas version.
The bug occurred because there is a special case in the initial window setup
code for handling the case where only a length-1 history is used for a given
frequency. Previously, the code was incorrectly calculating the period end
using a hard-coded expression for the end of the day (the correct behavior for
a length-1 '1d' history), and then using the frequency object to calculate the
period start for the window. In the case of length 1 '1m' data, this resulted
in an initial window whose start and end was the last minute of the day rather
than the first minute of the day. For non-price fields, this error doesn't
matter, because the window is only used for rolling digests (which doesn't
happen when there's only a length-1 history), and for the forward-filling logic
(which only happens on price fields). For a length-1 '1m' price, however, the
incorrect window causes us to attempt to forward-fill an empty panel, resulting
in an IndexError when we do an iloc[0] on a length-0 axis.
Adds a new argument, `freq_filter`, to `HistoryContainer.update_digest_panels`,
which can be used to supply a predicate determining which digest panels are
updated by the call. This is useful for supporting initialization/backfilling
of multi-frequency panels in subclasses.
Updates `HistoryContainer.roll` to handle cases where no data is present for
the period being rolled.
We now only forward-fill the `price` field when `ffill` is specified.
Overhauls `HistoryContainer` in prep for support of more than one frequency.
Major changes:
- Methods/variables referring to "day" have been renamed/generalized.
- `current_day_panel` became `buffer_panel`, which is now a `RollingPanel`
- `prior_day_panel` became a dictionary mapping `Frequency` objects to
"digest panels", which are instances of `RollingPanel`.
- Hard-coded daily rollover replaced with a notion of a "current window" for
each unique frequency managed by the panel.
- When the end of the current window is reached for a given frequency, we
compute an aggregate bar (code refers to this as a "digest"), which is
appended to a panel associated with that frequency.
- Window rollover dates are managed by a pair of dictionaries,
`cur_window_starts` and `cur_window_closes`. The `Frequency` class is
responsible for computing window bounds based on the open/close of the
previous window.
- Semantic change to the `open_price` field: `open_price` now always
contains the price of the first trade occurring in the given window.
Previously it contained the price of the first minute in the window,
returning NaN it the security happened not to trade in the first minute.
jfkirk