Algorithm returns and the risk calculations that depend on them now include
cash dividends. This commit does _not_ provide an API for user algorithms to
access dividends.
PerformanceTracker expects the dividend data to arrive as events, similar to
the way that Trades arrive. Dividends are expected to have adjusted payment
amounts that are inline with adjusted trades.
PerformanceTracker maintains state of all the unpaid dividends in the position
objects held in PerformancePeriod. Dividend objects contain all the relevant
dates (declared, ex, payment) as well as net and gross amounts. Dividends are
removed from the list as they are paid. Cash flow is not incremented until the
payment day. This creates the possibility of a dividend being owed but not
paid or realized before the end of a test. For example, a dividend with an
ex_date of today may have a pay date 2 weeks in the future. Right now the
algorithm does not receive any credit for unpaid dividends.
Tests cover buying/selling around the ex_date and payment_date, and checking
that the performance calculated is as expected.
Takes the value set for a variable on handle_data and records it,
e.g.:
```
def initialize(self):
self.incr = 0
self.record_variables(['incr'])
def handle_data(self, data):
self.incr += 1
```
Would record a variable of `incr`.
Emits the recorded variables as part of the daily performance.
This batch combins work from:
Thomas Wiecki <thomas.wiecki@gmail.com> (@twiecki)
fawce <fawce@quantopian.com> (@fawce)
Uses heapq.merge to sort input from mulitple sources instead of
our own sort module.
From profiling heapq.merge is more efficient than our own efforts.
update_universe is a bottleneck on large data sets.
A large portion of that bottleneck is the call to getitem while
looping over the keys, so using update while passing along the internal
__dict__
Seeing about a 40% improvement.
The delta was ensuring that the backtester wouldn't exceed the
delta of a bar if it were being run against live data.
However, this extra overhead of getting the current time on each
side of the handle_data adds a penalty in pure backtest mode.
Also, it makes the backtest results potentially non-repeatable,
since it is sensitive to current conditions on a box for processing
time.
Favoring having the timeout handled by whatever is running the
zipline algorithm.
Previously, on days that were trading days, but there with no
event data to process for that day, performance metrics were
not emitted, since the handling was based on having an event
trigger the daily performance metric.
Handled by grouping together performance messages, on market open,
for all days since the last market close.
Also, changes perf_tracker unit test to simulate missing data.
Taken from @richafrank's branch handling the same case.
Two reasons for removal:
- On the path of removing most non-postconditional asserts.
Since the asserts on every message is incurring a
non-insignificant penalty on large datasets.
- Since the assert was invoked as a function, the 'right side'
of the assert statement, i.e. the error message was being invoked
as a function, discovered since the __repr__ of the message was
high on the bottleneck list.
The main bottle neck here was using `len`.
A boolean check is a sufficient test for more items in the queue.
Also, uses all instead of several functions.
When run over large amounts of data the use of ndict's gets and sets
become a large bottleneck, around 1/5th of the CPU time is spent
in ndict's __setattr__, __getattr__, etc.
By switching to an object for an event,
we reduce the penalty significantly.
Removes asserts that check for event being an ndict, as well as those
that assume a certain behavior of the __contains__ method for events.
Moved grouping by date earlier in the pipeline of generators,
prior to any date-dependent state getting involved. Grouping
pulls from the pipeline until the start of the next group,
which is in the next day. The effect of grouping after
slippage but before handle_data is that slippage and the algo
are out of sync by a transaction.
I wrote this a little while ago as I noticed that a lot of time is spent
computing risk statistics. This is done over the complete history over
and over again while this could be done just by using the previously
computed value (iteratively).
We didn't go forward back then because for minute trade data the
difference was not significant enough. However, now with zipline
standalone I think most people will use daily (because that's
what's available) and it makes a huge difference
(speed-up of a couple of 100%).
Unfortunately, we can't just replace the existing one with an
iterative as for the final cumulative stats the batch is still
better. So that's not as nice, but the performance increase is
big enough for me to issue this PR (zipline is actually painfully
slow with daily data).
There is a unittest that compares that both produce exactly
the same outputs.
Speed measurements (for 500 trading days, daily source):
with iterative:
real 26.617 user 12.909 sys 6.112 pcpu 71.46
prior:
real 44.176 user 31.030 sys 11.381 pcpu 96.00
Mostly whitespace, line width and other spacing changes.
Also, removes use of deprecated has_key in favor of `in`
Going forward new patches should pass running `flake8` before
submission.
fawce