Fixes a bug where doing a boolean comparison between a non-numexpr
Filter and a NumExprFilter would fail because we don't implement
`__rand__` and `__ror__`.
So that consumers can write empty days worth of data, without needing
to construct a DataFrame with zero data force a write.
The internal loader uses `last_date_in_output_for_sid` to signify that
data has been attempted to be retrieved for all dates up until that, so
that when resuming a job those retrieval of data for those dates are not
re-attempted.
Also, used to make the write logic cleaneer, by making it only
necessary to create an array large enough for the given df.
Fix a bug where creating a sid bcolz file when the containing directory
was already occupied by a sid caused an OSError on attempt of creating
the directory because it already existed.
e.g. if there were two sids, `1` and `2`. The paths would be
`00/00/000001.bcolz` and `00/00/000002.bcolz` which share the same
directory `00/00`.
Fixed by checking for directory existence before calling `makedirs`.
Add test coverage which exercises writing of sids that are siblings in
the sid directory structure.
Implement a writer for minute data into a format comprised of multiple
ctables, one for each individual asset, with a common 'index' shared by
all ctables where a given a dt maps to the same array index for all
equities and fields.
This format is pulled from the lazy-mainline/Q2.0 branch, with some
changes to the interface.
Add basic retrieval of values at a given dt to reader. Not yet used by
Zipline simulations, but added to support unit tests.
Also, rename stubbed out us_equity_minutes to minute_bars, since the
writer can be agnostic to asset type.
loaders
This allows people to set their cutoff time to the time they will
actually execute 'before_trading_start'. Currently this is just passed
to the constructor of the loader; however, I would like to make this
managed by the algorithm simulation runner. This would help keep all of
the loaders in sync and lock 'before_trading_start's execution to the
time the data is queried for.
In preparation for the incoming changes which no longer push every bar
through the tradesimulation, remove the adjustment of the period's cash on
every pricing change of a held futures asset.
Instead hold the last sale price for each held future either:
- At the end of each peformance period update the last sale prices of
all held futures, so that the pnl for the next period uses values
derived from the cash difference between the end of the two periods.
- When a transaction is processed for the Future, so that the correct
amount is applied to each cash adjustment. (i.e. the cash adjustment
is reset on every change of amount of the Future being held, so that
multiple size and prices do not need to be tracked for the same asset.)
Also, remove now unused dict of payout calculation modifier, since new
calculation reads the value directly off of the asset.
Remove update_last_sale test, since the method no longer returns a cash
value.
Scott Sanderson