Fixes a bug where doing a boolean comparison between a non-numexpr
Filter and a NumExprFilter would fail because we don't implement
`__rand__` and `__ror__`.
So that consumers can write empty days worth of data, without needing
to construct a DataFrame with zero data force a write.
The internal loader uses `last_date_in_output_for_sid` to signify that
data has been attempted to be retrieved for all dates up until that, so
that when resuming a job those retrieval of data for those dates are not
re-attempted.
Also, used to make the write logic cleaneer, by making it only
necessary to create an array large enough for the given df.
Fix a bug where creating a sid bcolz file when the containing directory
was already occupied by a sid caused an OSError on attempt of creating
the directory because it already existed.
e.g. if there were two sids, `1` and `2`. The paths would be
`00/00/000001.bcolz` and `00/00/000002.bcolz` which share the same
directory `00/00`.
Fixed by checking for directory existence before calling `makedirs`.
Add test coverage which exercises writing of sids that are siblings in
the sid directory structure.
Implement a writer for minute data into a format comprised of multiple
ctables, one for each individual asset, with a common 'index' shared by
all ctables where a given a dt maps to the same array index for all
equities and fields.
This format is pulled from the lazy-mainline/Q2.0 branch, with some
changes to the interface.
Add basic retrieval of values at a given dt to reader. Not yet used by
Zipline simulations, but added to support unit tests.
Also, rename stubbed out us_equity_minutes to minute_bars, since the
writer can be agnostic to asset type.
loaders
This allows people to set their cutoff time to the time they will
actually execute 'before_trading_start'. Currently this is just passed
to the constructor of the loader; however, I would like to make this
managed by the algorithm simulation runner. This would help keep all of
the loaders in sync and lock 'before_trading_start's execution to the
time the data is queried for.
Joe Jevnik