- Fixes an error where Modeling API data known as of the close of `day
N` would be shown to algorithms during `before_trading_start` as of
the close of the same day. Algorithms should now only receive data
during `before_trading_start/handle_data` that was known as of the
simulation time at which the function would be called.
- All Term instances now have a `mask` attribute that must be a `Filter`
or an instance of `AssetExists()`. `mask` can be used to specify that
a Factor should be computed in a manner that ignores the values that
were not `True` in the mask.
- Changed the interface for `FFCLoader.load_adjusted_array` and
`Term._compute` from `(columns, mask)`, with mask as a DataFrame, to
`(columns, dates, assets, mask)`, where mask is a numpy array. This
is primarily to avoid having to reconstruct extra DataFrames when
using masks produced by non `AssetExists` filters.
- Adds `BoundColumn.latest`, which gives the most-recently-known value
of a column.
All terms just implement `_compute` now. (We reserve `compute` for the
public API of `CustomFactor`.)
Also removed `TestingTermMixin` and its subclasses in favor of just
using `CustomFactor.`
This commit removes the ability to reference a shared TradingEnvironment through the zipline.finance.trading module. In place, the classes that require a TradingEnvironment, or its child AssetFinder, contain their own references to those objects.
This commit also adds serialization utilities that allow for the pickling/unpickling of objects without unintentionally their TradingEnvironments or AssetFinders.
The write_data methods invokes the relevant AssetDBWriter subclass
to write data to the database. update_asset_finder is no longer
a relevant method since the AssetFinder is strictly a reader class.
- Add an `ascending=True` keyword to `rank()`.
- Add `top(N)` and `bottom(N)` methods to Factor. These return Filters
that pass the top and bottom N elements each day.
- Add a slightly faster path for rank(method='ordinal'). I had
originally thought the fast path was 2-3x faster because I had my
benchmark data axes flipped. The actual speedup is only 5-10%, which
means it probably wasn't worth the effort to Cythonize...but we have a
slightly faster version now so we might as well use it.
- Refactor test_filter and test_factor to make it easier to implement
and test transformations on factors. These tests now subclass
BaseFFCTestCase, which provides facilities for passing a dict of terms
and an "initial_workspace", the values for which are used by
SimpleFFCEngine rather than needing to manually manage the inputs and
outputs of each term.
- Previously it was returning a DataFrame because of how we applied an &
with a DataFrame mask. The error was masked by the fact that
`np.assert_array_equal` coerces inputs to arrays before comparing.
- Added `zp.utils.test_utils.check_arrays`, which checks type equality
before calling `np.assert_array_equal`.
This patch lays the groundwork for a compute engine designed to
facilitate construction of factor-based universe screening and portfolio
allocation. It contains:
A new module, `zipline.modelling`, containing entities that can be used
to express computations as dependency graphs. Each node in such a graph
is an instance of the base `Term` class, defined in
`zipline.modelling.term`. Dependency graphs are executed by instances
of `FFCEngine`, defined in `zipline.modelling.engine`.
A new module, `zipline.data.ffc`, containing loaders and dataset
definitions for inputs to the modelling API.
New `TradingAlgorithm` api methods: `add_factor`, and `add_filter`.
These methods can only be called from `initialize`, and are used to
inform the algorithm that each day it should compute the given terms.
Computed factor results are made available through a new attribute of
the `data` object in `before_trading_start` and `handle_data`. Computed
filter results control which assets are available in the factor matrix
on each day.
Test sources are now defined by the sim_params period_start and period_end, rather than by the period_start and a defined 'count' of bars. This allows us to consider the sim_params.period_end as the canonical definition of the end of a simulation.
The correct thing to look at to figure out where the root of the
zipline tree is, is `zipline.__file__`, not `zipline.__path__`. The
latter could contain multiple directories in it, and is not intended
to be `os.path.join()`ed as the previous code was doing.
Rather than drop files temporarily into the master security lists
directory during unit tests, create temporary directories for the
tests. This avoids issues when the tests are being run at the same
time as other code that uses the real security lists data.
Remove pieces that are no longer used now that the simple transforms are
wrappers around history via the SIDData object.
Move window length related pieces into batch_transform, since the rest
of the utils module is no longer used.
Previously the class SerializeableZiplineObject was used to
house basic __setstate__ and __getstate__ methods. It wasn't
really doing much that was helpful, so it is now gone.
Some unit tests for test_tradingcalendar failed on 2015-03-01, because the
addition of 365 days put the end date at 2016-02-29; when the replaces
the year on that date it fails because there is no 2017-02-29.
Instead use relativedelta with a year argument which accounts for leap
years.
Fixes the following test failure:
```
======================================================================
ERROR: test_day_after_thanksgiving (tests.test_tradingcalendar.TestTradingCalendar)
----------------------------------------------------------------------
Traceback (most recent call last):
File "./tests/test_tradingcalendar.py", line 211, in test_day_after_thanksgiving
tradingcalendar.end.replace(year=tradingcalendar.end.year + 1)
File "tslib.pyx", line 297, in pandas.tslib.Timestamp.replace (pandas/tslib.c:7325)
ValueError: day is out of range for month
----------------------------------------------------------------------
Ran 1 test in 0.001s
```
Limited use of `pandas` data structures in both `HistoryContainer` and
`RollingPanel`. Where possible, methods were amended to return raw
`ndarrays` with the indexing logic done separately. This allows us to
cut down the number of times pandas objects are created both as returns
and intermediate values. The separation of indexing from data access
allowed us to minimize the times we’d make use of pandas indexes.
This required that that certain methods like `NDFrame.ffill` be replaced
with versions that work with `ndarrays`. Some of this was done via
straight numpy methods and others by access pandas internal
machinery. Outside of allowing us to use faster ndarrays, many of these
function provided speedups over their pandas counterparts as we didn’t
require the extra features like handling multiple dtypes. i.e. np.isnan
is faster than pd.isnull, but only works with certain dtypes.
and added a new test case
was not iterating over lookup date directory names, and
therefore mising all by one list of stocks.
discovered because of differing sort orders between
my local machine, other devs, and travis ci.
Scott Sanderson