Since the position amount and price ndarrays are one dimensional
and use real numbers, we do not need the overhead of the extra
case handling provided by numpy.vdot, which comes at a cost of
performance.
With thanks to @jlowin, for pointing out the better fit of numpy.dot.
Gets almost 100x speed up over iterating over the values and
summing up the values in Python.
Farms out the work to numpy and atlas by using the vector dot
product of the amounts and last sale prices.
Adds some wiring of keeping track of an index into the numpy arrays
for each position, so that value can be overwritten as events update
those amounts and sale prices.
Instead of doing the rollover by creating a new PerformancePeriod,
introduces a `rollover` method that resets the values that need
to be fresh in a new period, and moves the ending values to starting
values, and leaves positions intact.
This isn't a major runtime improvement in of itself, but it does
allow us to more easily keep track of position values from period
to period, which other improvements will use.
Instead of creating a new ndict for each position on every event,
we change the values in the object that held the previous position.
The creation of new objects on each event was incurring too much
overhead.
Changes the position type returned by performance module.
For improved speed, changes from ndict to a simple Python object,
since the cost of setting ndict values is too expensive for the
number of times that positions are returned.
Also, changes the containing type of the positions to be dictionary
with the __missing__ overloaded, instead of the ndict that had that
behavior, to reduce the penalty of using ndicts.
The creation of a new portfolio ndict on each call of handle_data
was creating a very high performance overhead.
Instead, we use the same the portfolio object for each event,
and replace the values contained within.
Gains some performance by using a 'regular' object instead of
an ndict.
Also, directly sets up the values that we return, instead of going in
between with __core_dict and then removing values.
In it's entirety performanc.as_portfolio is the current
highest bottleneck, working on reducing time spent in that function.
Previously, on days that were trading days, but there with no
event data to process for that day, performance metrics were
not emitted, since the handling was based on having an event
trigger the daily performance metric.
Handled by grouping together performance messages, on market open,
for all days since the last market close.
Also, changes perf_tracker unit test to simulate missing data.
Taken from @richafrank's branch handling the same case.
When run over large amounts of data the use of ndict's gets and sets
become a large bottleneck, around 1/5th of the CPU time is spent
in ndict's __setattr__, __getattr__, etc.
By switching to an object for an event,
we reduce the penalty significantly.
Removes asserts that check for event being an ndict, as well as those
that assume a certain behavior of the __contains__ method for events.
Moved grouping by date earlier in the pipeline of generators,
prior to any date-dependent state getting involved. Grouping
pulls from the pipeline until the start of the next group,
which is in the next day. The effect of grouping after
slippage but before handle_data is that slippage and the algo
are out of sync by a transaction.
We were only incrementing the risk report by one day, and never
checking to see if that day we incremented into was a trading day
or not.
We now increment by day until we are on a trading day.
With an assist from @twiecki on:
Adapted test_risk_compare_batch_iterative to work with fixed
iterative risk class.
Starting down the path of making the portfolio completely read-only
with respect to the handle_data in algo.
The portfolio should only be changed during the course of running
the algorithm by the simulator.
This doesn't do a 100% protection, i.e. an algo could use _portfolio,
or the set_attr property, but hoping this helps guides algo writing
to treat the portfolio as read-only.
I wrote this a little while ago as I noticed that a lot of time is spent
computing risk statistics. This is done over the complete history over
and over again while this could be done just by using the previously
computed value (iteratively).
We didn't go forward back then because for minute trade data the
difference was not significant enough. However, now with zipline
standalone I think most people will use daily (because that's
what's available) and it makes a huge difference
(speed-up of a couple of 100%).
Unfortunately, we can't just replace the existing one with an
iterative as for the final cumulative stats the batch is still
better. So that's not as nice, but the performance increase is
big enough for me to issue this PR (zipline is actually painfully
slow with daily data).
There is a unittest that compares that both produce exactly
the same outputs.
Speed measurements (for 500 trading days, daily source):
with iterative:
real 26.617 user 12.909 sys 6.112 pcpu 71.46
prior:
real 44.176 user 31.030 sys 11.381 pcpu 96.00
Removes TRANSFORM_TYPE from protocol, since it is unused.
Also, removes use of ndict as a member of protocol, since it's
import there was for the TRANSFORM_TYPE. Changed to
utils.protocol_utils instead.
Mostly whitespace, line width and other spacing changes.
Also, removes use of deprecated has_key in favor of `in`
Going forward new patches should pass running `flake8` before
submission.
Eddie Hebert