This patch lays the groundwork for a compute engine designed to
facilitate construction of factor-based universe screening and portfolio
allocation. It contains:
A new module, `zipline.modelling`, containing entities that can be used
to express computations as dependency graphs. Each node in such a graph
is an instance of the base `Term` class, defined in
`zipline.modelling.term`. Dependency graphs are executed by instances
of `FFCEngine`, defined in `zipline.modelling.engine`.
A new module, `zipline.data.ffc`, containing loaders and dataset
definitions for inputs to the modelling API.
New `TradingAlgorithm` api methods: `add_factor`, and `add_filter`.
These methods can only be called from `initialize`, and are used to
inform the algorithm that each day it should compute the given terms.
Computed factor results are made available through a new attribute of
the `data` object in `before_trading_start` and `handle_data`. Computed
filter results control which assets are available in the factor matrix
on each day.
On Ubuntu (assume this is true for all posix) tickers containing a slash char ("CRD/A", "BRK/A", both valid tickers with yahoo api accessible timeseries) lead to a path error in loader.py line 286.
Python 2 and 3 throw different exception types when a file does
not exist.
Catch both exception types to trigger the download, so that the
loader works under both Python versions.
The compatibility between the two versions was made easier by
letting pandas handle the heavy lifting, so pass filenames to the
pandas serialization methods, instead of dealing doing the file
handling and reading/writing within the data module.
Use the six module to import functions and types that are
consistent between Python 2 and 3, so that one code base can
support both versions.
- Use integer types instead of int and long.
- Use string_types instead of basestring.
- Account for iteritems, itervalues, iterkeys.
- Use six.moves for filter and zip, reduce
- Use compatible bytes for md5 hasher.
- xrange and range
- Use `print()` function for all print calls
- Fix strip and format calls that were on the outside of the
print function for some reason.
(Which were breaking in Python 3 because of print returning None.)
- Remove commented out print calls.
Check for whether or not the index's timezone is UTC or not before
attempting to localize, since an already localized index throws an
error when tz_localize is called.
Remove the lists of DailyReturn objects in favor of using pd.Series
to store the return values.
Should make it easier to inspect the values when stepping through,
make the windowing of data to a certain range more facile by using,
and have some performance increases due to removing object creation
and member access.
The dump and update of curves were both using the entire history.
So instead of having the update use a different code path, always
use dump and overwrite.
Both unit tests and repeated runs while developing an algorithm
can benefit from having a local copy of the Yahoo data, instead
of doing a network call each time.
Store the web request results as a csv file in a cache directory,
named by symbol and date range.
Instead of writing our own serialization using msgpack, leverage
the csv serialization provided by pandas.
Also, lessens the need for msgpack and functions in date_utils.
The loader module printed some warning messages, these could
be changed to use a logger, but for now convert to use the print
function for compatibility with Python 3.
Python 3 requires using dot syntax for relative imports,
otherwise the import is treated as an absolute import, i.e.
an import of a module from outside of the project.
By using dot syntax now, imports should be compatible with both
Python 2.7 and Python 3.
On ranges with missing data from Yahoo, e.g.:
On 2013-04-2 the date range of April 2013-03-29 failed because
of the first day in the range being Good Friday, and the API not
yet updating for the Monday after.
Handle the 404 that is found by raising and warning that no
benchmark data was found, but continuing on.
If the trading_days end date is not greater than the date being
tested, (this can happen if the algorithm's end date is set to a date
that is before the latest date available saved in the msgpacks),
then trying to get the location will fail, instead searchsorted
will get the lastest date available in the trading day map to use
as a test date.
other details:
- also fixed grammatical errors in loader's status messages.
- converting the treasury curves to an ordered dict.
- moved to using a lambda for clarity as per @ehebert
- initializing calendar end dates to be midnight of current date in
- US/Eastern. Yahoo data isn't available until midnight eastern.
- added LSE reference rrules calendar (thanks to Edward Johns)
- added tests to verify LSE environment matches rrule calendar
- added a test to verify global environment behavior can be set.
- moved DailyReturn class to trading to eliminate circularity from
risk <-> trading.
- updated TradingEnvironment to be a context manager. This allows users
to run algorithms in individually isolated environments in one python
process. This is useful for managing multiple algorithms in a single
ipython notebook.
- added comments to explain behavior and useage of the global environment
Global state for the financial simulation environment is accessed through the
zipline.finance.trading module, which now contains a module variable:
environment.
Parameters are passed into an algorithm as a keyword argument, sim_params.
SimulationParameters creates a trading day index for the test period that
can be used to find trading days, calculate distance between trading days,
and other common operations. The sim params index is just selected from the
global state.
================
Details:
- adding delorean to the requirements.
- made index symbol a parameter for loading the benchmark data. changed
messagepack storage to be symbol specific.
- ported risk, performance, algorithm, transforms, batch transforms
and associated tests to use simulation parameters and global environment
- factory and sim factory use global state and sim params
- factory method parameter names now reflect the class expected
So that the zipline library can be used when installed to a
write-protected location, e.g. the global site-packages,
moving the download files to a directory in the user's path,
which should be writeable.
For now, choosing a ~/.zipline/data location.
The latest flake8 release in now 1.5, which pulls in pep8: 1.3.4a0
The upgrade pep8 has changes to what it picks up as lint.
Making code base compatible, so that new devs can install pep8
from PyPI and not have friction over the version difference.
Currently using these ignores in the config file:
```
[pep8]
ignore = E124,E125,E126
```
Ignoring these since they are difficult to squash while maintaining
an 80 char line length, and appear spurious.
Should address later.
Updates Travis config, README, and pip requirements to reflect change.
Hopefully, this helps ease ramp up time for developing against
market data, without us distributing the data.
We do a check for the data when attempting to read the msgpack
files, if they don't exist the loader makes a web request and
retrieves and serializes the data for the user.
Provides a loader for:
- curves from data.treasury.gov
- benchmarks from Yahoo! Finance
Adds dependency of requests library in dev requirements.
Scott Sanderson