- added LSE reference rrules calendar (thanks to Edward Johns)
- added tests to verify LSE environment matches rrule calendar
- added a test to verify global environment behavior can be set.
- moved DailyReturn class to trading to eliminate circularity from
risk <-> trading.
- updated TradingEnvironment to be a context manager. This allows users
to run algorithms in individually isolated environments in one python
process. This is useful for managing multiple algorithms in a single
ipython notebook.
- added comments to explain behavior and useage of the global environment
Global state for the financial simulation environment is accessed through the
zipline.finance.trading module, which now contains a module variable:
environment.
Parameters are passed into an algorithm as a keyword argument, sim_params.
SimulationParameters creates a trading day index for the test period that
can be used to find trading days, calculate distance between trading days,
and other common operations. The sim params index is just selected from the
global state.
================
Details:
- adding delorean to the requirements.
- made index symbol a parameter for loading the benchmark data. changed
messagepack storage to be symbol specific.
- ported risk, performance, algorithm, transforms, batch transforms
and associated tests to use simulation parameters and global environment
- factory and sim factory use global state and sim params
- factory method parameter names now reflect the class expected
With this patch, on the close of markets we "fast forward" to midnight of the
next trading day and calculate the dividend payments. This patch assumes that
the dividend dates are all at midnight UTC.
Algorithm returns and the risk calculations that depend on them now include
cash dividends. This commit does _not_ provide an API for user algorithms to
access dividends.
PerformanceTracker expects the dividend data to arrive as events, similar to
the way that Trades arrive. Dividends are expected to have adjusted payment
amounts that are inline with adjusted trades.
PerformanceTracker maintains state of all the unpaid dividends in the position
objects held in PerformancePeriod. Dividend objects contain all the relevant
dates (declared, ex, payment) as well as net and gross amounts. Dividends are
removed from the list as they are paid. Cash flow is not incremented until the
payment day. This creates the possibility of a dividend being owed but not
paid or realized before the end of a test. For example, a dividend with an
ex_date of today may have a pay date 2 weeks in the future. Right now the
algorithm does not receive any credit for unpaid dividends.
Tests cover buying/selling around the ex_date and payment_date, and checking
that the performance calculated is as expected.
Since the position amount and price ndarrays are one dimensional
and use real numbers, we do not need the overhead of the extra
case handling provided by numpy.vdot, which comes at a cost of
performance.
With thanks to @jlowin, for pointing out the better fit of numpy.dot.
Gets almost 100x speed up over iterating over the values and
summing up the values in Python.
Farms out the work to numpy and atlas by using the vector dot
product of the amounts and last sale prices.
Adds some wiring of keeping track of an index into the numpy arrays
for each position, so that value can be overwritten as events update
those amounts and sale prices.
Instead of doing the rollover by creating a new PerformancePeriod,
introduces a `rollover` method that resets the values that need
to be fresh in a new period, and moves the ending values to starting
values, and leaves positions intact.
This isn't a major runtime improvement in of itself, but it does
allow us to more easily keep track of position values from period
to period, which other improvements will use.
Instead of creating a new ndict for each position on every event,
we change the values in the object that held the previous position.
The creation of new objects on each event was incurring too much
overhead.
Changes the position type returned by performance module.
For improved speed, changes from ndict to a simple Python object,
since the cost of setting ndict values is too expensive for the
number of times that positions are returned.
Also, changes the containing type of the positions to be dictionary
with the __missing__ overloaded, instead of the ndict that had that
behavior, to reduce the penalty of using ndicts.
The creation of a new portfolio ndict on each call of handle_data
was creating a very high performance overhead.
Instead, we use the same the portfolio object for each event,
and replace the values contained within.
Gains some performance by using a 'regular' object instead of
an ndict.
Also, directly sets up the values that we return, instead of going in
between with __core_dict and then removing values.
In it's entirety performanc.as_portfolio is the current
highest bottleneck, working on reducing time spent in that function.
Previously, the list was generated, but only used to calculate
the number of days in the environment.
With exposing this list, working towards a path where the simulation
uses the trading days to determine when to handle market closes.
Previously, on days that were trading days, but there with no
event data to process for that day, performance metrics were
not emitted, since the handling was based on having an event
trigger the daily performance metric.
Handled by grouping together performance messages, on market open,
for all days since the last market close.
Also, changes perf_tracker unit test to simulate missing data.
Taken from @richafrank's branch handling the same case.
When run over large amounts of data the use of ndict's gets and sets
become a large bottleneck, around 1/5th of the CPU time is spent
in ndict's __setattr__, __getattr__, etc.
By switching to an object for an event,
we reduce the penalty significantly.
Removes asserts that check for event being an ndict, as well as those
that assume a certain behavior of the __contains__ method for events.
To handle, for instance, Columbus Day (Oct 10),
on which there is no treasury data.
We're only forward-filling data now, and
no longer searching both back and forward in time.
Updated the search for treasury data when there is none for the
test end date.
It could be that the end date is not a trading day, or we could
just be missing treasury data. In either case, we try to recover
more gracefully now, by searching as far as possible and maybe
logging a warning.
Similarly, if there is no benchmark data for the test end date,
look for the next trading day. If we really have no data,
blow up with our own explicit exception, instead of overflowing
in our search for dates in the future.
Moved grouping by date earlier in the pipeline of generators,
prior to any date-dependent state getting involved. Grouping
pulls from the pipeline until the start of the next group,
which is in the next day. The effect of grouping after
slippage but before handle_data is that slippage and the algo
are out of sync by a transaction.
We were only incrementing the risk report by one day, and never
checking to see if that day we incremented into was a trading day
or not.
We now increment by day until we are on a trading day.
With an assist from @twiecki on:
Adapted test_risk_compare_batch_iterative to work with fixed
iterative risk class.
Starting down the path of making the portfolio completely read-only
with respect to the handle_data in algo.
The portfolio should only be changed during the course of running
the algorithm by the simulator.
This doesn't do a 100% protection, i.e. an algo could use _portfolio,
or the set_attr property, but hoping this helps guides algo writing
to treat the portfolio as read-only.
The latest flake8 release in now 1.5, which pulls in pep8: 1.3.4a0
The upgrade pep8 has changes to what it picks up as lint.
Making code base compatible, so that new devs can install pep8
from PyPI and not have friction over the version difference.
Currently using these ignores in the config file:
```
[pep8]
ignore = E124,E125,E126
```
Ignoring these since they are difficult to squash while maintaining
an 80 char line length, and appear spurious.
Should address later.
Updates Travis config, README, and pip requirements to reflect change.
I wrote this a little while ago as I noticed that a lot of time is spent
computing risk statistics. This is done over the complete history over
and over again while this could be done just by using the previously
computed value (iteratively).
We didn't go forward back then because for minute trade data the
difference was not significant enough. However, now with zipline
standalone I think most people will use daily (because that's
what's available) and it makes a huge difference
(speed-up of a couple of 100%).
Unfortunately, we can't just replace the existing one with an
iterative as for the final cumulative stats the batch is still
better. So that's not as nice, but the performance increase is
big enough for me to issue this PR (zipline is actually painfully
slow with daily data).
There is a unittest that compares that both produce exactly
the same outputs.
Speed measurements (for 500 trading days, daily source):
with iterative:
real 26.617 user 12.909 sys 6.112 pcpu 71.46
prior:
real 44.176 user 31.030 sys 11.381 pcpu 96.00
Removes TRANSFORM_TYPE from protocol, since it is unused.
Also, removes use of ndict as a member of protocol, since it's
import there was for the TRANSFORM_TYPE. Changed to
utils.protocol_utils instead.
Changes our filter so that instead of just checking for the current
day, we ensure that orders are before or on the current event time.
This adds a delay, (defaulting to one minute), to the order so that we
avoid filling an order exactly when it is placed.
So that we don't replace the orders variable with the list comp.
No functional change, but easier to compare the original and the
results of the list comp when debugging.
Expiration is something that way may want to have in the future,
but this current is implementation is dropping orders that
aren't meant to be expired. So removing expiration, so that all
expected orders are executed.
fawce