Instead of having the performance tracker as part of the
tradesimulation class, hold on to it inside of the algorithm
object, so that the perf_tracker is more easily accessed for
reset behavior, etc.
- moved Order and Blotter to zipline.finance.blotter
- moved order method from AlgoSimulator to Blotter
- eliminated the set_order method in algorithm
- moved blotter to the algorithm
Refactor PerformanceTracker, Blotter, and AlgorithmSimulator to
work with handling the end of a bar at the AlgorithmSimulator level
instead of within PerformanceTracker.
- PerforamnceTracker and Blotter are longer generators,
both provide functions to process events instead.
- AlgorithmSimulator calls each from within the loop running
over the data generator.
- Change test_perf_tracker utility to be compatible with change
away from PerformanceTracker as a generator.
Has the effect of:
- Fixing the timing of order emission.
- Allow minutely emission of benchmarks, which was prevented
by the extra grouping previously caused by Blotter.
Minutely emission also depends on work for streaming benchmarks
through performance and risk at a minute granularity.
To fix the grouping of events so that (dt, events) ordering
is preserved, the tracking of order states needs to change
in the following way.
Change how order keeps track of dates:
- Change order's dt field to reflect modified date.
- Add a created field.
Change how performance keeps track of orders by:
- Map dt to transactions
- Map dt to orders
- Map order ids to keep track of updated orders.
The emission of order updates from the blotter were incorrect,
and subsequently, performance.
Previously, only the first action of the order was emitted,
fix so that all status updates are emitted.
The use of np.allclose introduced a severe performance penalty,
caused by the creation of two `np.array`s for each check.
Instead create and use a similar check which maintains tolerance
to floating point rounding, but operates only on scalars.
- Add transaction and order types
- Move TransactionSimulator from trading.py to tradesimulation.py
(only used by other members of the tradesimulation module)
- Make Transaction an independent event, like dividend
- Add Blotter class.
- Flatten the transaction events to be independent of trade bar events
- Make orders into events that reach performance (need to add
handling)
- Issue IDs to orders and tracking each transaction's order id.
- Make volume share slippage fill orders independently, rather than
aggregating them into a single transaction.
- Perf tracker holds orders, serializes them with transactions.
- Order state defined and maintained by order class.
- Minutely emission of orders based on last_modified date.
Also, fix double emission of performance results with the last minute.
Change the perf tracker unit tests so that it doesn't rely on an
'extra' event triggering emission.
Unlike daily, minute emission now emits at the end of the bar in
the PerformanceTracker.transform instead of waiting for the next event.
During minute emissions, it is still helpful to have a final daily
performance result, analogous to what would be the final packet in
a daily emitted backtest, so that all transactions, etc. are contained
in one place.
The indexing into performance results during the simulation loop fails
when emitting minutely since 'daily_perf' only exists on daily performance
results, not the minutely results.
Fix by making the key used to index into performance results depend
on the emission rate.
This function is unused, and on the general path of moving testing
functions out of gens.utils
# Please enter the commit message for your changes. Lines starting
- perf modified to let non-performance related events flow through.
- changes to support streaming non-trading data through batch transforms
and for mixing in sids with just custom data.
- allowing CUSTOM events to flow through to transforms.
- Added logic to maintain pre-specified sid filter.
Instead of a loosely defined object for Order, explicitly
defines the parameters and corresponding members.
Clearing the way for adding more members to the Order object.
Algorithm returns and the risk calculations that depend on them now include
cash dividends. This commit does _not_ provide an API for user algorithms to
access dividends.
PerformanceTracker expects the dividend data to arrive as events, similar to
the way that Trades arrive. Dividends are expected to have adjusted payment
amounts that are inline with adjusted trades.
PerformanceTracker maintains state of all the unpaid dividends in the position
objects held in PerformancePeriod. Dividend objects contain all the relevant
dates (declared, ex, payment) as well as net and gross amounts. Dividends are
removed from the list as they are paid. Cash flow is not incremented until the
payment day. This creates the possibility of a dividend being owed but not
paid or realized before the end of a test. For example, a dividend with an
ex_date of today may have a pay date 2 weeks in the future. Right now the
algorithm does not receive any credit for unpaid dividends.
Tests cover buying/selling around the ex_date and payment_date, and checking
that the performance calculated is as expected.
Takes the value set for a variable on handle_data and records it,
e.g.:
```
def initialize(self):
self.incr = 0
self.record_variables(['incr'])
def handle_data(self, data):
self.incr += 1
```
Would record a variable of `incr`.
Emits the recorded variables as part of the daily performance.
This batch combins work from:
Thomas Wiecki <thomas.wiecki@gmail.com> (@twiecki)
fawce <fawce@quantopian.com> (@fawce)
Uses heapq.merge to sort input from mulitple sources instead of
our own sort module.
From profiling heapq.merge is more efficient than our own efforts.
update_universe is a bottleneck on large data sets.
A large portion of that bottleneck is the call to getitem while
looping over the keys, so using update while passing along the internal
__dict__
Seeing about a 40% improvement.
The delta was ensuring that the backtester wouldn't exceed the
delta of a bar if it were being run against live data.
However, this extra overhead of getting the current time on each
side of the handle_data adds a penalty in pure backtest mode.
Also, it makes the backtest results potentially non-repeatable,
since it is sensitive to current conditions on a box for processing
time.
Favoring having the timeout handled by whatever is running the
zipline algorithm.
Previously, on days that were trading days, but there with no
event data to process for that day, performance metrics were
not emitted, since the handling was based on having an event
trigger the daily performance metric.
Handled by grouping together performance messages, on market open,
for all days since the last market close.
Also, changes perf_tracker unit test to simulate missing data.
Taken from @richafrank's branch handling the same case.
Two reasons for removal:
- On the path of removing most non-postconditional asserts.
Since the asserts on every message is incurring a
non-insignificant penalty on large datasets.
- Since the assert was invoked as a function, the 'right side'
of the assert statement, i.e. the error message was being invoked
as a function, discovered since the __repr__ of the message was
high on the bottleneck list.
The main bottle neck here was using `len`.
A boolean check is a sufficient test for more items in the queue.
Also, uses all instead of several functions.
When run over large amounts of data the use of ndict's gets and sets
become a large bottleneck, around 1/5th of the CPU time is spent
in ndict's __setattr__, __getattr__, etc.
By switching to an object for an event,
we reduce the penalty significantly.
Removes asserts that check for event being an ndict, as well as those
that assume a certain behavior of the __contains__ method for events.
Moved grouping by date earlier in the pipeline of generators,
prior to any date-dependent state getting involved. Grouping
pulls from the pipeline until the start of the next group,
which is in the next day. The effect of grouping after
slippage but before handle_data is that slippage and the algo
are out of sync by a transaction.
