The function that handles a market close for daily frequency changed from
`handle_market_close` to `handle_market_close_daily`.
The function that is called at on the closing minute each day when running
minutely changed from `handle_intraday_close` to
`handle_intraday_market_close`.
Breaks out the main snapshot processing loop into its own function, and does
some minor variable renaming-shuffling.
Adds `TradingAlgorithm.on_dt_changed`, a function to be called when the
simulation dt changes, prior to processing any events.
There should be no difference in behavior as a result of this change.
In situations where the performance tracker has been reset or patched
to handle state juggling with warming up live data, the `market_close`
member of the performance tracker could end up out of sync with the
current algo time as determined by the
The symptom was dividends never triggering, because the end of day
checks would not match the current time.
Fix by having the tradesimulation loop be responsible, in minute/minute
mode, for advancing the market close and passing that value to the
performance tracker, instead of having the market close advanced by
the performance tracker as well.
Adding a copy of the Event's dt field as datetime via the
`alias_dt` generator, so that the API was forgiving and allowed
both datetime and dt on a SIDData object, was creating noticeable
overhead, even on an noop algorithms.
Instead of incurring the cost of copying the datetime value and
assigning it to the Event object on every event that is passed
through the system, add a property to SIDData which acts as an
alias `datetime` to `dt`.
Eventually support for `data['foo'].datetime` may be removed,
and could be considered deprecated.
To prevent cases where custom data types had unaligned timestamps,
only call handle_data when market data passes through.
Custom data that comes before market data will still update
the data bar. But the handling of that data will only be done
when there is actionable market data.
The next day calculation was causing an error when a minute
emission algorithm reached the end of available data.
Instead of a generic exception when available data is reached,
raise and catch a named exception so that the tradesimulation loop
can skip over, since the next market close is not needed at the end.
When running with minutely emissions the simulator would report to the
user that it simulated 'n - 1' days (where n is the number of days
specified in the simulation params). Now the correct number of trading
days are reported as being simulated.
Instead of porting these cases of type checking, remove them instead.
Slightly more Python-ic to be more generous in what is allowed, and
the conversion to make these compatible with Python 3 are more trouble
than they are worth.
Use the six module to import functions and types that are
consistent between Python 2 and 3, so that one code base can
support both versions.
- Use integer types instead of int and long.
- Use string_types instead of basestring.
- Account for iteritems, itervalues, iterkeys.
- Use six.moves for filter and zip, reduce
- Use compatible bytes for md5 hasher.
- xrange and range
So that each reference to `.portfolio` in the algoscript,
cache the value of the portfolio, and mark the need for a new
value at the end of each dt in the tradesimulation loop.
The defaultdict behavior was allowing both algo code and
TradingAlgorithm wrappers to add unintended keys.
Remove use of defaultdict in favor of a dictionary that explicitly
adds the values in tradesimulation, otherwise allow a KeyError
if the bar is indexed with a sid that doesn't exist.
Also, when iterating over the keys in the data bar, only return
those keys that have pricing data.
This reverts commit e3a9ca27b1, reversing
changes made to 3d8bdeb429.
Conflicts:
zipline/gens/tradesimulation.py
The aforementioned change needs a revert because it caused a 'doubling'
of orders, since the portfolio is not updated until after handle_data
is called a second time after an order has been processed.
The flexibility of fill_delay is still desired, but remove for now,
favoring reverting back to existing behavior over trying ot fix the
fill_delay logic.
Instead of searching through the open orders to find the ones
that match the current transactions, now that simulate returns
the pair of transaction and order for which that transaction
was created for, that order can be used where we previously
searched for a modified order.
This should be a runtime improvement since, but not yet verified
via thorough profiling.
instead of just limiting it with max_date. This allows for an
additional "live" day appended to the end, fixing various uses
of the trading_day_map, and replacing the isolated check in
get_next_close.
Now that the tradesimulation loop has changed to use benchmarks
as a 'clock', the logic for setting the current time can be grouped
together at the beginning of each iteration instead of the date
and snapshot grouping.
Also, can remove the snapshot_dt and use simulation_dt instead
of having two variables that were keeping track of the same value.
Also, it is no longer needed to peek into the data to get the first
simulation_dt now that simulation_dt is set at the beginning of each
loop iteration.
The datetime was only being set while updating the universe with the
bar's trade events.
Now that benchmarks are used as a clock, it is possible to have
benchmarks without having trade bars during that dt, so the datetime
should be updated via benchmark as well.
Critical that trade events be last, so that the perf tracker's
position information be updated with both the transaction and
the trade for last_sale.
Without, the first transaction would be recorded with a last_sale
of 0.
With @fawce
Work towards running an algorithm against 'live' data, which can't
be bound to the available benchmarks and treasuries, since the
benchmarks and treasury curves for that day won't be published
until that night.
Instead of having the performance tracker as part of the
tradesimulation class, hold on to it inside of the algorithm
object, so that the perf_tracker is more easily accessed for
reset behavior, etc.
- moved Order and Blotter to zipline.finance.blotter
- moved order method from AlgoSimulator to Blotter
- eliminated the set_order method in algorithm
- moved blotter to the algorithm
Refactor PerformanceTracker, Blotter, and AlgorithmSimulator to
work with handling the end of a bar at the AlgorithmSimulator level
instead of within PerformanceTracker.
- PerforamnceTracker and Blotter are longer generators,
both provide functions to process events instead.
- AlgorithmSimulator calls each from within the loop running
over the data generator.
- Change test_perf_tracker utility to be compatible with change
away from PerformanceTracker as a generator.
Has the effect of:
- Fixing the timing of order emission.
- Allow minutely emission of benchmarks, which was prevented
by the extra grouping previously caused by Blotter.
Minutely emission also depends on work for streaming benchmarks
through performance and risk at a minute granularity.
To fix the grouping of events so that (dt, events) ordering
is preserved, the tracking of order states needs to change
in the following way.
Change how order keeps track of dates:
- Change order's dt field to reflect modified date.
- Add a created field.
Change how performance keeps track of orders by:
- Map dt to transactions
- Map dt to orders
- Map order ids to keep track of updated orders.
The emission of order updates from the blotter were incorrect,
and subsequently, performance.
Previously, only the first action of the order was emitted,
fix so that all status updates are emitted.
The use of np.allclose introduced a severe performance penalty,
caused by the creation of two `np.array`s for each check.
Instead create and use a similar check which maintains tolerance
to floating point rounding, but operates only on scalars.
- Add transaction and order types
- Move TransactionSimulator from trading.py to tradesimulation.py
(only used by other members of the tradesimulation module)
- Make Transaction an independent event, like dividend
- Add Blotter class.
- Flatten the transaction events to be independent of trade bar events
- Make orders into events that reach performance (need to add
handling)
- Issue IDs to orders and tracking each transaction's order id.
- Make volume share slippage fill orders independently, rather than
aggregating them into a single transaction.
- Perf tracker holds orders, serializes them with transactions.
- Order state defined and maintained by order class.
- Minutely emission of orders based on last_modified date.
Also, fix double emission of performance results with the last minute.
Change the perf tracker unit tests so that it doesn't rely on an
'extra' event triggering emission.
Unlike daily, minute emission now emits at the end of the bar in
the PerformanceTracker.transform instead of waiting for the next event.
Scott Sanderson