Our DataFrame index resolution logic relies on failed lookups **not**
being cached, but not caching failed lookups is a nontrivial performance
hit when repeatedly looking up sids. The "solution" here is to clear
the caches after writing in new assets.
The real fix for this is either:
1. Don't construct an AssetFinder until we have the datasource in hand
in run(), or
2. Don't symbol-map the user's input source if it's a DataFrame.
Instead we should make our data loaders pre-map the data.
_analyze Method in algorithm.py
Added one line to check for the keyword argument 'analyze' and set the
the _analyze method when a TradingAlgorithm object is initialized
within a script.
Previously we have capitalized input strings at different levels in
our code: in the user-facing API methods and in the asset finder.
This commit moves input string capitalization exclusively to the API
method to which the string was supplied. Specifically, the string is
capitalized by a preprocess API method decorator. The preprocess
decorator passes the input string to the newly defined
ensure_upper_case() method, which returns a TypeError if the argument
supplied is not a string.
ensure_upper_case() is defined in a new file, zipline/utils/input_validation.py.
The existing expect_types() method is also moved there.
Various tests in tests/test_assets.py are modified to account for the
fact that the asset finder method lookup_symol() no longer capitalizes
its supplied argument.
- Adds `zipline.pipeline.Pipeline`, a new user-facing class for managing
pipelines of Modeling API expressions.
- Adds `attach_pipeline` and `drain_pipeline` as API methods
- Removes `add_factor` and `add_filter` as API methods. These have been
replaced two new methods on `Pipeline`: `add`, and `apply_screen`.
- Adding a `Filter` as a column no longer implicitly truncates rows from
the Modelling API output. It simply causes a new column, of dtype
`bool` to show up in the output. Removal of rows is now handled by the
new `apply_screen` method of `Pipeline`.
- Refactors the existing Modeling API tests to reflect the new APIs.
lookup_symbol_resolve_multiple was identical to lookup_symbol, except that lookup_symbol performed upper-casing of the input string and lookup_symbol would return Nones. Now, lookup_symbol has a kwarg 'default_None=True' and all symbols are upper-cased on insertion and request.
- Fixes an error where Modeling API data known as of the close of `day
N` would be shown to algorithms during `before_trading_start` as of
the close of the same day. Algorithms should now only receive data
during `before_trading_start/handle_data` that was known as of the
simulation time at which the function would be called.
- All Term instances now have a `mask` attribute that must be a `Filter`
or an instance of `AssetExists()`. `mask` can be used to specify that
a Factor should be computed in a manner that ignores the values that
were not `True` in the mask.
- Changed the interface for `FFCLoader.load_adjusted_array` and
`Term._compute` from `(columns, mask)`, with mask as a DataFrame, to
`(columns, dates, assets, mask)`, where mask is a numpy array. This
is primarily to avoid having to reconstruct extra DataFrames when
using masks produced by non `AssetExists` filters.
- Adds `BoundColumn.latest`, which gives the most-recently-known value
of a column.
The initialize method of TradingAlgorithm no longer accepts and
silently ignores args and kwargs, but instead forwards them
to the user-defined function referenced by self._initialize.
To avoid passing unexpected arguments to self._initialize, the
following additional adjustments are made:
- pop 'namespace' from the kwargs supplied to TradingAlgorithm
rather than simply get()ing it
- do not pass an AssetFinder to the TradingAlgorithm in
test_modelling_algo.py, as this has been deprecated and will
cause self._initialize to fail
This commit removes the ability to reference a shared TradingEnvironment through the zipline.finance.trading module. In place, the classes that require a TradingEnvironment, or its child AssetFinder, contain their own references to those objects.
This commit also adds serialization utilities that allow for the pickling/unpickling of objects without unintentionally their TradingEnvironments or AssetFinders.
The write_data methods invokes the relevant AssetDBWriter subclass
to write data to the database. update_asset_finder is no longer
a relevant method since the AssetFinder is strictly a reader class.
Previously symbols were resolved to sids based on the end of
simulation date. This commit allows the user to specify the
date for which resolution will take place using a new
set_symbol_lookup_date() API method.
If the user does not use this method the lookup date will
default back to the simulation end date.
`period_end` can be outside the range of data for which we have dates.
`last_close` properly gets pulled back to the last date for which we
actually have data.
We should consider whether or not we need to be storing period_end at
all.
This patch lays the groundwork for a compute engine designed to
facilitate construction of factor-based universe screening and portfolio
allocation. It contains:
A new module, `zipline.modelling`, containing entities that can be used
to express computations as dependency graphs. Each node in such a graph
is an instance of the base `Term` class, defined in
`zipline.modelling.term`. Dependency graphs are executed by instances
of `FFCEngine`, defined in `zipline.modelling.engine`.
A new module, `zipline.data.ffc`, containing loaders and dataset
definitions for inputs to the modelling API.
New `TradingAlgorithm` api methods: `add_factor`, and `add_filter`.
These methods can only be called from `initialize`, and are used to
inform the algorithm that each day it should compute the given terms.
Computed factor results are made available through a new attribute of
the `data` object in `before_trading_start` and `handle_data`. Computed
filter results control which assets are available in the factor matrix
on each day.
The datetime attribute of TradingAlgorithm was initially None, so calling get_datetime in initialize was causing an unhandled exception. This commit addresses that issue by calling on_dt_changed when initializing the TradingAlgorithm, to force all datetimes to be in line with the period_start.
This commit modifies the DataFrameSource and DataPanelSource to accept only Int64Indexes on the incoming data and moves the burden of mapping user identifiers to TradingAlgorithm.run().
Move the responsibility of wrapping the main simulation loop in the
ZiplineAPI context from the algorithm modules generator setup to the
main trade simulation loop, so that different methods of invoking loop
do not need to duplicate how the context is set.
To make it easier for internal implementation of handle_data to
transition off of calling the ZiplineAPI every bar, to only invoking the
context once per simulation.
Joe Jevnik