Minutely data can now be appended to bcolz files even when
minutes in the same day have already been written. For example,
previously attempting to write data for the minute 2016-05-11 16:30
would raise an exception if any OHLCV data for 2016-05-11 had been
written to the same file.
Trying to overwrite existing minutes still raises a
BcolzMinuteOverlappingData exception.
Note that previously all sids' bcolz files ended at the same time.
This is no longer necessarily the case. The last record in each
sid's bcolz file now corresponds to the latest minute for which
OHLCV data is provided to the writer.
Previously, we have assumed that the `amounts` and `last_sale_prices`
lists have the same order as the `value_multipliers`. This is not
correct, since to populate the `amounts` and `last_sale_prices` lists
we iterate over a `dict` (self.positions). The order of this `dict`
can change in arbitrary ways when it is updated, which occurs when
we call `update_positions`. Our `value_multipliers` however are stored
in an `OrderedDict`, meaning the order of existing key/value pairs
is not changed when they are updated.
To address this issue, we make sure that `self.positions` subclasses
`OrderedDict`.
Previously we were not accounting for cases where we would invoke
next_market_minute() with a time on a trading day *before* the
market open, or previous_market_minute() with a time on a trading
day *after* the market close.
Previously we have capitalized input strings at different levels in
our code: in the user-facing API methods and in the asset finder.
This commit moves input string capitalization exclusively to the API
method to which the string was supplied. Specifically, the string is
capitalized by a preprocess API method decorator. The preprocess
decorator passes the input string to the newly defined
ensure_upper_case() method, which returns a TypeError if the argument
supplied is not a string.
ensure_upper_case() is defined in a new file, zipline/utils/input_validation.py.
The existing expect_types() method is also moved there.
Various tests in tests/test_assets.py are modified to account for the
fact that the asset finder method lookup_symol() no longer capitalizes
its supplied argument.
_convert_asset_str_fields was added to address the fact that the
unicode returned by SQLAlchemy was causing test failures, in
particular test_repr and test_root_symbols failed because of the
leading 'u' character in the unicode of python-2.x. Later it was
pointed out that this method would behave differently in python 2
and 3, in particular the str() method would return bytes in python
2 and unicode in python 3. After investigating it was found that
the _convert_asset_str_fields was no longer necessary. It is unclear
why the original tests were failing. It is not due to SQLAlchemy
versions, as tests pass for version >1 and <1.
Bug fixes include:
- Do not call float on tuples
- Correct indentation
- Do not call value() on int
- Use fuzzy correctly
- Use symbol for lookup
- Ensure we raise Error
The write_data methods invokes the relevant AssetDBWriter subclass
to write data to the database. update_asset_finder is no longer
a relevant method since the AssetFinder is strictly a reader class.
The AssetDBWriter class and its subclasses will
ultimately be responsible for creating the SQLite
database tables and writing data to these tables.
In the longer term AssetDBWriter and AssetFinder will
be decoupled, sharing only an SQLite connection.
However, for backward compatibility reasons this has
not yet been fully implemented.
Modify tests since AssetFinder no longer has a
metadata_cache attribute.
Previously symbols were resolved to sids based on the end of
simulation date. This commit allows the user to specify the
date for which resolution will take place using a new
set_symbol_lookup_date() API method.
If the user does not use this method the lookup date will
default back to the simulation end date.
Stewart Douglas