The BcolzDailyBarReader was optimized for the pipeline case of reading
all assets at once.
Now that the reader is also used to support daily history the case of
reading a data for a small number of assets is more common, particularly
in algorithms that use the history API which have a high rotation of
assets (e.g. an algorithm which pipeline uses to set the active
universe)
Remove the bottleneck in reading a small number of assets by
conditionally reading the slice for each asset from the carray, instead
of reading the data for all equities and then indexing into that full
array. On a certain number of assets, it is still better to read all the
data at once. On the Quantopian dataset, which holds data for 20000
about for the last 10 years of equity data (where not all equities trade
over the full range), stored in 118 blosc blp files per column, the
tipping point where the 'read all' mode wins out between 3000-4000
assets.
That number was tested by trying to exercise a worst case scenario where
the equities were spread out evenly across the blp files, by stepping
along a sorted list of assets that were alive over a query range which
spanned 70 trading days.
```
size = 3000
sids = [assets[i] for i in range(0, len(assets), len(assets) /
size)][:size]
```
Also, add parameter to WithBcolzDailyBarReader fixture which allows the
test to specify what the threshold count for reading all data should be,
so that the test_us_equity_pricing can be forced into either mode to
make sure that both branches in logic are covered by all test cases.
On local dev machine this patch improves the read time of `load_raw_array`
for one asset from 100 ms to 96.5 µs. (10^5 improvement.) With reading
only asset per call a being an observed common case when populating the
non-cached values in USEquityHistoryLoader.
We were trying to use the previous day in before_trading_start because
we were looking for the previous market minute, then normalizing it. That's
no longer the case, as we want to use today's date for fetcher lookups
in before_trading_start.
Also refactored a bit how dataportal determines if a query should be
routed to the fetcher data structures.
Changes BcolzDailyBarWriter to not be an abc, data is passed as an
iterator of (sid, dataframe) pairs to the write method.
Changes the AssetsDBWriter to be a single class which accepts an engine
at construction time and has a `write` method for writing dataframes for
the various tables. We no longer support writing the various other data
types, callers should coerce their data into a dataframe themselves. See
zipline.assets.synthetic for some helpers to do this.
Adds many new fixtures and updates some existing fixtures to use the new
ones:
WithDefaultDateBounds
A fixture that provides the suite a START_DATE and END_DATE. This is
meant to make it easy for other fixtures to synchronize their date
ranges without depending on eachother in strange ways. For example,
WithBcolzMinuteBarReader and WithBcolzDailyBarReader by default should
both have data for the same dates, so they may use depend on
WithDefaultDates without forcing a dependency between them.
WithTmpDir, WithInstanceTmpDir
Provides the suite or individual test case a temporary directory.
WithBcolzDailyBarReader
Provides the suite a BcolzDailyBarReader which reads from bcolz data
written to a temporary directory. The data will be read from
dataframes and then converted to bcolz files with
BcolzDailyBarWriter.write
WithBcolzDailyBarReaderFromCSVs
Provides the suite a BcolzDailyBarReader which reads from bcolz data
written to a temporary directory. The data will be read from a
collection of CSV files and then converted into the bcolz data through
BcolzDailyBarWriter.write_csvs
WithBcolzMinuteBarReader
Provides the suite a BcolzMinuteBarReader which reads from bcolz data
written to a temporary directory. The data will be read from
dataframes and then converted to bcolz files with
BcolzMinuteBarWriter.write
WithAdjustmentReader
Provides the suite a SQLiteAdjustmentReader which reads from an in
memory sqlite database. The data will be read from dataframes and then
converted into sqlite with SQLiteAdjustmentWriter.write
WithDataPortal
Provides each test case a DataPortal object with data from temporary
resources.
Fix a bug where if history were called with assets `[1, 2]` and then
subsequently, `[2, 1]`, the loader would return the cached array in
order for `[1, 2]`.
Instead cache an AdjustedArray for each asset, then when a history
window is requested, check if each asset has a sufficient cache, and if
not then read values for the assets which are missing or need to be
refreshed.
An added benefit of this change is that if a subsequent call to history
has a smaller number of assets than the previous, no new data needs to
be read from disk. e.g. a call with assets `[1, 2, 3]` and then `[1, 2]`
would use the cached values for `1` and `2` from the first call.
Conversely, if the second call has more assets, then only the data for
the new assets needs to be retrieved. e.g. a history with `[1, 2]`, then
`[1, 2, 3]` would only need (assuming `1` and `2` have not expired) to
retrieve data for `3`. Unfortunately, the benefit here is not great
because `load_raw_arrays` is optimized for reading many assets, and
pulls the entire daily bar dataset into memory. This change makes tuning
`load_raw_arrays` so that faster reads (e.g. by slicing from the carray
for each asset, instead of pulling all data into a numpy array), when
only a few assets are requested, more beneficial than it would have been
previously.
Add a cap of 5 sliding windows (one per OHCLV column) to the history
loader's cache of sliding windos.
This prevents unbounded growth on algorithms that call history with a
highly varied list of equities.
To follow is splitting the cache up by column and by sid, so that the
loader does not re-prefetch sids which have already been read with
sufficient data; however this patch is enough to fix the issue where an
algo with high rotation can add up a megabyte per day of memory on
algorithms which rotate on a 5% dollar volume pipeline. With this cap
those algorithms have more plateaus with regard to memory consumption.
This patch requires new dependency of `cachetools` library.
Instead of using the `remember_last` memoization on all calls to
`_find_position_of_minute`, add an instance local cache which is only
used by the `get_value` call. The `get_value` call is very hot, so any
extra overhead (e.g. creating the WeakArgs on every invocation) becomes
costly. The current usage `get_value` also has the property that it is
called with monotonically increasing, but with a high repeat count on
each value. (A further improvement could making a `get_value` which
supports being used by many sids, for use by the update portfolio
positions.)
The caching is not done at the `_find_position_of_minute_level` because
`unadjusted_window` always uses two positions on the tape (start and end
of range) which would cause the entries and removal into the cache which
would be invalidated both between the calls of start and end, and next
call of the function.
The argument was only needed for mapping the positions which need to be
removed on adjusted windows. The start and end position of each range
can be derived from the early closes' positions and the market open,
respectively.
Remove to reduce moving parts.
The cache in data portal was added before the change to using a
CachedObject to wrap the window_blocks in the USEquityHistoryLoader.
Removing this extra layer saves some cycles.
Does not fix current memory investigaton (since only one sids/dts pair
per column was cached in `_equity_daily_reader_array_data` at a time),
but removing should make it more clear where needed references are being
held.
The minute history loader caching was incorrectly mimicking the daily
history loader caching.
Where caching the adjusted array on the last dt helps an access pattern
of repeated calling history windows on the same day (which has an end_dt
of the previous day), with minute windows the end dt is always moving
forward, so the cached values are seldom used. (Would only be used if
`history` was called with same parameters twice on the same simulation time.)
The intervals are returned as a set, so order is not guaranteed,
which becomes exposed when reading windows which span multiple years.
The deletion of values from the regular sized minute array assumes that
intervals can be reversed to delete the array from the back.
When the dts and length of cols are mismatched the writer behaves in
unintended ways. e.g. in a case where a consumer passed dts which had
minutes with no trades removed, but regular (market minute for day)
sized arrays for the data with `0`'s on minutes without trades, the non
trade minutes from cols are written to slots in the output where a trade
is intended.
Protect against this misuse by checking that all lengths are equal when
using the `write_cols` method.
Make a separate `_write_cols` method for use by both `write_cols` and
`write`, since the `write` method which takes a DataFrame has the
matched input length enforced by the DataFrame.
Write arrays representing corresponding market opens and market closes,
which will eventually replace the `minute_index` field.
The market closes are being added for incoming work on another branch
which will use the market closes to generate a list of non-market
minutes to filter out when returning data from `unadjusted_window`.
Add a method to minute bar reader which returns the OHLCV for all
requested fields for a list assets over the specified start and end
minutes.
Initial usage is intended for use by a loader which consumes minute bar
data to resample into daily bars, but may also be used when aggregating
minute data during '1d' history calls in Q2.0.
This iteration does not include including of early closes.
So that consumers can write empty days worth of data, without needing
to construct a DataFrame with zero data force a write.
The internal loader uses `last_date_in_output_for_sid` to signify that
data has been attempted to be retrieved for all dates up until that, so
that when resuming a job those retrieval of data for those dates are not
re-attempted.
Also, used to make the write logic cleaneer, by making it only
necessary to create an array large enough for the given df.
Use the preexisting metadata method when instantiating the minute bar
reader.
An internal sublcass uses the `_get_metadata` method to setup data for
directories that have not used the new writer/reader interface.
(i.e. allows for reader creation when the metadata.json file does not
exist.)
Fix a bug where creating a sid bcolz file when the containing directory
was already occupied by a sid caused an OSError on attempt of creating
the directory because it already existed.
e.g. if there were two sids, `1` and `2`. The paths would be
`00/00/000001.bcolz` and `00/00/000002.bcolz` which share the same
directory `00/00`.
Fixed by checking for directory existence before calling `makedirs`.
Add test coverage which exercises writing of sids that are siblings in
the sid directory structure.
Implement a writer for minute data into a format comprised of multiple
ctables, one for each individual asset, with a common 'index' shared by
all ctables where a given a dt maps to the same array index for all
equities and fields.
This format is pulled from the lazy-mainline/Q2.0 branch, with some
changes to the interface.
Add basic retrieval of values at a given dt to reader. Not yet used by
Zipline simulations, but added to support unit tests.
Also, rename stubbed out us_equity_minutes to minute_bars, since the
writer can be agnostic to asset type.
Moved from the `lazy-mainline` branch,
https://github.com/quantopian/zipline/pull/858
The intent of this patch to provide the basic class and readers
interfaces, developed on that branch, so that the use of creating the
object and opening paths etc. can be tested internally.
Additional changes beyond the lazy-mainline branch, addition of future
minute reader, and daily bar reader.
Also allow an argument of the future_daily_reader, though no such reader
yet exists.
It may be that future and equity readers share an interface, and a
further improvement would be providing an abstract base class.
co-author: @jbredeche <jean@quantopian.com>
Return -1 when there is a zero value for a spot price.
Intended for use by the incoming data portal changes. When the data
portal will see a -1 value, the portal will seek back a trading day
until a non-negative value is returned.
Volumes were incorrectly having the thousands factor applied, however
the volume is written as is (without the factor, since it volume is an
int, not float value.)
Fix by adding a special case for volume which returns the price as is.
Rather than repeatedly try and fail to download data that's not yet
available, only try to download again if we haven't successfully
downloaded in the last hour.
Jean Bredeche