Move the responsibility of wrapping the main simulation loop in the
ZiplineAPI context from the algorithm modules generator setup to the
main trade simulation loop, so that different methods of invoking loop
do not need to duplicate how the context is set.
To make it easier for internal implementation of handle_data to
transition off of calling the ZiplineAPI every bar, to only invoking the
context once per simulation.
Make the ordering in which processing of event types both explicit and
independent of the sort ordering of the incoming sources.
The overhead of creating the list per snapshot and the iterators appears
to be marginal in the minute data case when tested locally.
This patch is intended as part of the path towards making the trade
simulation loop not depend on consuming and tracking every trade event.
The timing of where last_sale_date was needed to be changed was proving
difficult to adapt in the previous model.
Should also allow the removal of sorting of the various source streams.
By having both the trade simulation main loop route events to "process"
methods based on event type and the process methods also checking event
type, there was some duplicated effort in doing that comparison many
times.
A particular case where this was noted in profiling was for the
`process_event` function which was checking if the type was not a trade
and returning early, when in a larger universe of stocks the value
returned False 99% of the time.
Instead provide separate process functions specific to each type,
e.g. e.g. `process_trade` and `process_transaction` and route traffic to
those functions in tradesimulation.
For a universe of 160 stocks on both no-op algo and an algo that rebuys
its universe every day, saw about a 10% increase locally.
Also:
- Add process_benchmark to blotter since internal subclass relies on
logic on benchmark, this allows the internal process_trade to be a
`pass`.
- Add warning on unrecoginzed event types.
Remove pieces that are no longer used now that the simple transforms are
wrappers around history via the SIDData object.
Move window length related pieces into batch_transform, since the rest
of the utils module is no longer used.
Reduce overhead of the attribute access of grabbing the process_event
and process_trade from both the algorithm object and the perf_tracekr or
blotter, by assigning those functions to a variable once per snapshot.
schedule_function takes a date rule, a time rule, and a function and
will call the function, passing context and data only when the two rules
fire. This allows for code that is conditional to the datetime of the
algo.
This is implemented internally with `Event` objects which are pairings
of `EventRule`s and callbacks.
handle_data becomes a special event with a rule that always fires. This
makes the logic for handling events more complete and compact.
We guarantee that `TradingAlgorithm.updated_portfolio` is called at least once
between each perf message emission. This change moves that guaranteed call
into `AlgorithmSimulator.get_message` and adds a comment to make it clear why
updated_portfolio is being called.
The function that handles a market close for daily frequency changed from
`handle_market_close` to `handle_market_close_daily`.
The function that is called at on the closing minute each day when running
minutely changed from `handle_intraday_close` to
`handle_intraday_market_close`.
Breaks out the main snapshot processing loop into its own function, and does
some minor variable renaming-shuffling.
Adds `TradingAlgorithm.on_dt_changed`, a function to be called when the
simulation dt changes, prior to processing any events.
There should be no difference in behavior as a result of this change.
In situations where the performance tracker has been reset or patched
to handle state juggling with warming up live data, the `market_close`
member of the performance tracker could end up out of sync with the
current algo time as determined by the
The symptom was dividends never triggering, because the end of day
checks would not match the current time.
Fix by having the tradesimulation loop be responsible, in minute/minute
mode, for advancing the market close and passing that value to the
performance tracker, instead of having the market close advanced by
the performance tracker as well.
Adding a copy of the Event's dt field as datetime via the
`alias_dt` generator, so that the API was forgiving and allowed
both datetime and dt on a SIDData object, was creating noticeable
overhead, even on an noop algorithms.
Instead of incurring the cost of copying the datetime value and
assigning it to the Event object on every event that is passed
through the system, add a property to SIDData which acts as an
alias `datetime` to `dt`.
Eventually support for `data['foo'].datetime` may be removed,
and could be considered deprecated.
To prevent cases where custom data types had unaligned timestamps,
only call handle_data when market data passes through.
Custom data that comes before market data will still update
the data bar. But the handling of that data will only be done
when there is actionable market data.
The next day calculation was causing an error when a minute
emission algorithm reached the end of available data.
Instead of a generic exception when available data is reached,
raise and catch a named exception so that the tradesimulation loop
can skip over, since the next market close is not needed at the end.
When running with minutely emissions the simulator would report to the
user that it simulated 'n - 1' days (where n is the number of days
specified in the simulation params). Now the correct number of trading
days are reported as being simulated.
Instead of porting these cases of type checking, remove them instead.
Slightly more Python-ic to be more generous in what is allowed, and
the conversion to make these compatible with Python 3 are more trouble
than they are worth.
Use the six module to import functions and types that are
consistent between Python 2 and 3, so that one code base can
support both versions.
- Use integer types instead of int and long.
- Use string_types instead of basestring.
- Account for iteritems, itervalues, iterkeys.
- Use six.moves for filter and zip, reduce
- Use compatible bytes for md5 hasher.
- xrange and range
So that each reference to `.portfolio` in the algoscript,
cache the value of the portfolio, and mark the need for a new
value at the end of each dt in the tradesimulation loop.
The defaultdict behavior was allowing both algo code and
TradingAlgorithm wrappers to add unintended keys.
Remove use of defaultdict in favor of a dictionary that explicitly
adds the values in tradesimulation, otherwise allow a KeyError
if the bar is indexed with a sid that doesn't exist.
Also, when iterating over the keys in the data bar, only return
those keys that have pricing data.
This reverts commit e3a9ca27b1, reversing
changes made to 3d8bdeb429.
Conflicts:
zipline/gens/tradesimulation.py
The aforementioned change needs a revert because it caused a 'doubling'
of orders, since the portfolio is not updated until after handle_data
is called a second time after an order has been processed.
The flexibility of fill_delay is still desired, but remove for now,
favoring reverting back to existing behavior over trying ot fix the
fill_delay logic.
Instead of searching through the open orders to find the ones
that match the current transactions, now that simulate returns
the pair of transaction and order for which that transaction
was created for, that order can be used where we previously
searched for a modified order.
This should be a runtime improvement since, but not yet verified
via thorough profiling.
instead of just limiting it with max_date. This allows for an
additional "live" day appended to the end, fixing various uses
of the trading_day_map, and replacing the isolated check in
get_next_close.
Now that the tradesimulation loop has changed to use benchmarks
as a 'clock', the logic for setting the current time can be grouped
together at the beginning of each iteration instead of the date
and snapshot grouping.
Also, can remove the snapshot_dt and use simulation_dt instead
of having two variables that were keeping track of the same value.
Also, it is no longer needed to peek into the data to get the first
simulation_dt now that simulation_dt is set at the beginning of each
loop iteration.
The datetime was only being set while updating the universe with the
bar's trade events.
Now that benchmarks are used as a clock, it is possible to have
benchmarks without having trade bars during that dt, so the datetime
should be updated via benchmark as well.
Scott Sanderson