Adds tests asserting that we resolve conflicts in accordance with the
following rules when we have multiple assets holding the same symbol at
the same time:
If multiple SIDs exist for symbol S at time T, return the candidate
SID whose start_date is highest. (200 cases)
If multiple SIDs exist for symbol S at time T, the best candidate
SIDs share the highest start_date, return the SID with the highest
end_date. (34 cases)
It is the opinion of the author (ssanderson) that we should consider
this malformed input and fail here. But this is the current indended
behavior of the code, and I accidentally broke it while refactoring.
These will serve as regression tests until the time comes that we
decide to enforce this as an error.
See https://github.com/quantopian/zipline/issues/837 for more
details.
Previously we have capitalized input strings at different levels in
our code: in the user-facing API methods and in the asset finder.
This commit moves input string capitalization exclusively to the API
method to which the string was supplied. Specifically, the string is
capitalized by a preprocess API method decorator. The preprocess
decorator passes the input string to the newly defined
ensure_upper_case() method, which returns a TypeError if the argument
supplied is not a string.
ensure_upper_case() is defined in a new file, zipline/utils/input_validation.py.
The existing expect_types() method is also moved there.
Various tests in tests/test_assets.py are modified to account for the
fact that the asset finder method lookup_symol() no longer capitalizes
its supplied argument.
Improves the query for futures contract to use the date that comes first
in time (between notice_date and expiration_date) to determine cotnract
validity. If one of these is missing, we'll use the other.
Also modifies the query to order the resulting contracts by their
expiration_date if available, and to use their notice_date if not.
lookup_symbol_resolve_multiple was identical to lookup_symbol, except that lookup_symbol performed upper-casing of the input string and lookup_symbol would return Nones. Now, lookup_symbol has a kwarg 'default_None=True' and all symbols are upper-cased on insertion and request.
- Fixes an error where Modeling API data known as of the close of `day
N` would be shown to algorithms during `before_trading_start` as of
the close of the same day. Algorithms should now only receive data
during `before_trading_start/handle_data` that was known as of the
simulation time at which the function would be called.
- All Term instances now have a `mask` attribute that must be a `Filter`
or an instance of `AssetExists()`. `mask` can be used to specify that
a Factor should be computed in a manner that ignores the values that
were not `True` in the mask.
- Changed the interface for `FFCLoader.load_adjusted_array` and
`Term._compute` from `(columns, mask)`, with mask as a DataFrame, to
`(columns, dates, assets, mask)`, where mask is a numpy array. This
is primarily to avoid having to reconstruct extra DataFrames when
using masks produced by non `AssetExists` filters.
- Adds `BoundColumn.latest`, which gives the most-recently-known value
of a column.
This commit removes the ability to reference a shared TradingEnvironment through the zipline.finance.trading module. In place, the classes that require a TradingEnvironment, or its child AssetFinder, contain their own references to those objects.
This commit also adds serialization utilities that allow for the pickling/unpickling of objects without unintentionally their TradingEnvironments or AssetFinders.
The AssetDBWriter class and its subclasses will
ultimately be responsible for creating the SQLite
database tables and writing data to these tables.
In the longer term AssetDBWriter and AssetFinder will
be decoupled, sharing only an SQLite connection.
However, for backward compatibility reasons this has
not yet been fully implemented.
Modify tests since AssetFinder no longer has a
metadata_cache attribute.
If lookup_future_chain was provided with an as_of_date or knowledge date that was pandas.NaT, the query we were forming wasn't what we want. Instead, as_of_date, if not NaT, is used for knowledge_date, and if both are NaT, no date filtering is done in the query.
This patch lays the groundwork for a compute engine designed to
facilitate construction of factor-based universe screening and portfolio
allocation. It contains:
A new module, `zipline.modelling`, containing entities that can be used
to express computations as dependency graphs. Each node in such a graph
is an instance of the base `Term` class, defined in
`zipline.modelling.term`. Dependency graphs are executed by instances
of `FFCEngine`, defined in `zipline.modelling.engine`.
A new module, `zipline.data.ffc`, containing loaders and dataset
definitions for inputs to the modelling API.
New `TradingAlgorithm` api methods: `add_factor`, and `add_filter`.
These methods can only be called from `initialize`, and are used to
inform the algorithm that each day it should compute the given terms.
Computed factor results are made available through a new attribute of
the `data` object in `before_trading_start` and `handle_data`. Computed
filter results control which assets are available in the factor matrix
on each day.
Attack the startup bottleneck of creating the asset finders caches for a
large universe, which was between 1-2 seconds on development and
production machines.
Instead, allow the AssetFinder to be passed a sqlite3 file that has
already been populated and then hydrate asset objects only when an
equity is referenced for the first time.
To create aforementioned sqlite3, create an AssetFinder with an db_path
and `create_table` set to True. If `create_table` is set to False, the
prepopulated data in the sqlite file found at db_path will be used.
Default behavior is to use an in memory database.
Behavior that changes:
- Fuzzy lookup now only works on one character, that character needs to be
specified at write/metadata consumption time, since the fuzzy lookup key
is created by dropping the character from each symbol.
- Overwriting partially written metadata is no longer
supported. i.e. some unit tests allowed for inserting just the identifier,
and then later updating the symbol, end_date, etc.
Instead of building an upsert behavior at this time, this patch
changes the unit tests so that the data for each asset is only
inserted once.
Other notes:
- populate_cache is now removed, since there is no longer a two step
process of inserting metadata and then realizing that metadata into
assets. _spawn_asset is rolled into insert_metadata, so that a call to
insert_metadata both converts the metadata and makes it available in
the data store.
Since most brokers will cease accepting trades by the notice date, contracts should not be considered valid after the notice date. This commit adjusts the lookup_future_chain method to consider all contracts with notice dates on or following the current date invalid.
The asset finder retrieved from the test environment is empty, so the
test does not end up testing anything, since the test cases loop over
the empty list of sids in the asset finder.
Remove to possibly be added back in and re-implemented after a larger
refactoring of the module.
Removes unused future lookup methods and consolidates everything into lookup_future_chain. Since the FutureChain object will have to hold a root symbol and dates, it should be responsible for cleaning the user input, so this is removed from the lookup method.
Adds knowledge date to future lookups. This makes our definition of valid contracts more flexible. We know about a contract if it starts trading by the knowledge date, and a contract is expired if it expires by the as_of_date.
Also fixes a bug with computing future chains, where contracts were not included in the chain on their expiration date.
Richard Frank