Previously, all specs had to be pre-allocated by using the 'add_history'
function. This is now no longer required and instead serves as a hint to
the HistoryContainer to pre-allocate the space for the given spec.
History can grow by increasing the length for a frequency, adding a
frequency, or adding a field. It can grow with any combination of
these.
HistoryContainer now is aware of the data_frequency of the algorithm,
and no longer uses the daily_at_midnight flag; instead, this is the
default behavior.
Uses a numpy array instead of a dict of dicts when initializing history
container.
In testing this reduced the total time spent in HistoryContainer.update
by 66%.
BEFORE COMMIT:
Thu Oct 16 22:30:46 2014 results/cprofile/unoptimized
185223320 function calls (182210491 primitive calls) in 401.351
seconds
Ordered by: cumulative time
List reduced from 2398 to 27 due to restriction <'update'>
ncalls tottime percall cumtime percall filename:lineno(function)
8580 0.461 0.000 160.571 0.019
qexec/zipline/history/history_container.py:388(update)
AFTER COMMIT:
Thu Oct 16 22:12:28 2014 results/cprofile/optimized
143177181 function calls (140164352 primitive calls) in 272.403
seconds
Ordered by: cumulative time
List reduced from 2395 to 27 due to restriction <'update'>
ncalls tottime percall cumtime percall filename:lineno(function)
8580 0.086 0.000 47.294 0.006 qexec/zipline/history/history_container.py:388(update)
- NotHalfDay only worked at midnight
- week_(start|end) were actually month_(start|end)
- Removes check_args from api.
- Default offset of 30mins for market_(open|close)
zipline.utils.events is imported.
Changes the class level attribute `env` on EventRule to a property so
that the environment is only looked up at when needed.
schedule_function takes a date rule, a time rule, and a function and
will call the function, passing context and data only when the two rules
fire. This allows for code that is conditional to the datetime of the
algo.
This is implemented internally with `Event` objects which are pairings
of `EventRule`s and callbacks.
handle_data becomes a special event with a rule that always fires. This
makes the logic for handling events more complete and compact.
Overhaul the core HistoryContainer logic to be more robust to changing
universes.
Major Changes
-------------
* Remove `return_frame` cache. The original purpose of using
return_frames was to avoid having to create new DataFrames on each
iteration of handle_data, but we ended up having to copy the return
frames anyway because user code could mutate the frames in place.
Removing the return_frames reduces unnecessary copying, and reduces
the logic of `get_history` to just forward-filling and concatenating
two DataFrames.
* Use a `MultiIndex`ed DataFrame to represent
`last_known_prior_values`. This makes lookups faster and greatly
simplifies the logic of adding and dropping sids.
* HistoryContainer no longer attempts to determine its universe based on
the contents of its internal buffers. The TradingAlgorithm
controlling the container is now responsible for explicitly calling
`add_sids` or `drop_sids` when securities enter or leave the
algorithm's universe. These methods, along with the internal
`_realign` method, provide a clean interface for changing the universe
of securities managed by the container.
* Refactor index mutation logic in `RollingPanel` into a
`MutableIndexRollingPanel` subclass. Maintenance of the old behavior
is regrettably necessary to support `BatchTransform`.
* Refactor shared logic from `roll` and `get_history` into a single
`aggregate_ohlcv_panel` method that's responsible for collapsing an
OHLCV buffer into a frame.
In support of source that emits a subclass of Event which defines some
fields as properties instead of doubling the value in the
`Event.__dict__`
Use hasattr instead of the overridden __contains__ method of the Event
class, so that when non-algorithm facing code checks for field existence,
properties count.
Intentionally not touching the `__contains__` in Event, to avoid
changing, at the moment, any algo behavior that relies on the
`__contains__` behavior's use of `__dict__`
Joe Jevnik