The write_data methods invokes the relevant AssetDBWriter subclass
to write data to the database. update_asset_finder is no longer
a relevant method since the AssetFinder is strictly a reader class.
This patch lays the groundwork for a compute engine designed to
facilitate construction of factor-based universe screening and portfolio
allocation. It contains:
A new module, `zipline.modelling`, containing entities that can be used
to express computations as dependency graphs. Each node in such a graph
is an instance of the base `Term` class, defined in
`zipline.modelling.term`. Dependency graphs are executed by instances
of `FFCEngine`, defined in `zipline.modelling.engine`.
A new module, `zipline.data.ffc`, containing loaders and dataset
definitions for inputs to the modelling API.
New `TradingAlgorithm` api methods: `add_factor`, and `add_filter`.
These methods can only be called from `initialize`, and are used to
inform the algorithm that each day it should compute the given terms.
Computed factor results are made available through a new attribute of
the `data` object in `before_trading_start` and `handle_data`. Computed
filter results control which assets are available in the factor matrix
on each day.
Attack the startup bottleneck of creating the asset finders caches for a
large universe, which was between 1-2 seconds on development and
production machines.
Instead, allow the AssetFinder to be passed a sqlite3 file that has
already been populated and then hydrate asset objects only when an
equity is referenced for the first time.
To create aforementioned sqlite3, create an AssetFinder with an db_path
and `create_table` set to True. If `create_table` is set to False, the
prepopulated data in the sqlite file found at db_path will be used.
Default behavior is to use an in memory database.
Behavior that changes:
- Fuzzy lookup now only works on one character, that character needs to be
specified at write/metadata consumption time, since the fuzzy lookup key
is created by dropping the character from each symbol.
- Overwriting partially written metadata is no longer
supported. i.e. some unit tests allowed for inserting just the identifier,
and then later updating the symbol, end_date, etc.
Instead of building an upsert behavior at this time, this patch
changes the unit tests so that the data for each asset is only
inserted once.
Other notes:
- populate_cache is now removed, since there is no longer a two step
process of inserting metadata and then realizing that metadata into
assets. _spawn_asset is rolled into insert_metadata, so that a call to
insert_metadata both converts the metadata and makes it available in
the data store.
- AssetFinder no longer accepts an unused trading_calendar.
- AssetFinder correctly accepts a DataFrame as input.
- Tests for AssetFinder no longer rely on a global trading environment.
Previously, all specs had to be pre-allocated by using the 'add_history'
function. This is now no longer required and instead serves as a hint to
the HistoryContainer to pre-allocate the space for the given spec.
History can grow by increasing the length for a frequency, adding a
frequency, or adding a field. It can grow with any combination of
these.
HistoryContainer now is aware of the data_frequency of the algorithm,
and no longer uses the daily_at_midnight flag; instead, this is the
default behavior.
schedule_function takes a date rule, a time rule, and a function and
will call the function, passing context and data only when the two rules
fire. This allows for code that is conditional to the datetime of the
algo.
This is implemented internally with `Event` objects which are pairings
of `EventRule`s and callbacks.
handle_data becomes a special event with a rule that always fires. This
makes the logic for handling events more complete and compact.
There were sevaral places you could supply sim_params
in TradingAlgorithm (__init__, run). This got confusing
as its not clear who updated what and which one was the
correct one to use at each time.
Then there were to ways to define data_frequency, one in
__init__() and one in the sim_params which also added code
complexity.
This refactor makes it explicit that sim_params are to be
passed to __init__() only. Moreover, data_frequency is
only stored in sim_params. For backwards compatibility,
it can still be supplied separately but will link to
the one in sim_params.
For example, you could create new sim params via:
sim_params = create_simulation_parameters(data_frequency='minute')
algo = MyAlgo(sim_params)
algo.run(data)
In addition, perf_tracker only gets initialized in one place:
_create_generator() which should also make the various ways
of running an algorithm more deterministic.
This also fixes a bug with SimulationParameters where
you could not change the period_start. Unfortunately, the
current implementation still requieres an implicit call to
update the internal variables.
Replace usage of .ix in TradingEnvironment with .loc when we know that we're
using an index key.
DataFrame.ix can be used with either integer or key-based indices, and as such
it incurs an overhead for figuring out which you meant.
Adds a suite of new functions for querying data from the trading calendar.
These include:
`previous_trading_day`
`minutes_for_days_in_range` (minutely version of `days_in_range`)
`previous_open_and_close` (inverse of `next_open_and_close`)
`next_market_minute`
`previous_market_minute`
`open_close_window` (get a range of opens/closes with slicing semantics)
`market_minute_window` (get a range of minutes with slicing semantics)
Also refactors `test_finance` to move `TradingEnvironment` tests into their own
TestCase.
Adds a classmethod, `instance` on `TradingEnvironment` that returns
`zipline.finance.trading.environment`, instantiating it if necessary.
This makes it possible to initialize the default environment instance in a
less-roundabout way than creating a `SimulationParameters` object.
TradingEnvironment class uses env_trading_calendar for trading days,
but the default trading calendar for open_and_close data, which causes
errors later, because of misalignment of trading days.
The issue can be resolved by using env_trading_calendar for
open_and_closes as well
Instead of the benchmarks' index, use the trading calendar to
populate the environment's trading days.
Remove `extra_date` field, since unlike the benchmarks list,
the trading calendar can generate future dates, so dates for
current day trading do not need to be appended.
Motivations:
- The source for the open and close/early close calendar and the
trading day calendar is now the same, which should help prevent
potential issues due to misalignment.
- Allows configurations where the benchmark is provided as a
generator based data source to need to supply a second benchmark
list just to populate dates.
The next day calculation was causing an error when a minute
emission algorithm reached the end of available data.
Instead of a generic exception when available data is reached,
raise and catch a named exception so that the tradesimulation loop
can skip over, since the next market close is not needed at the end.
The market_open_and_close method was a performance bottleneck,
since it was creating new dates on each query for open and close.
Instead use the pre-rendered frame of open and closes values
from the trading environment.
Remove the lists of DailyReturn objects in favor of using pd.Series
to store the return values.
Should make it easier to inspect the values when stepping through,
make the windowing of data to a certain range more facile by using,
and have some performance increases due to removing object creation
and member access.
This reverts commit 17b8980fb9.
Backing out rigidness of market and close, while sorting out how
to handle events that are not on a day in the trading calendar.
Instead of creating the market open and close mid-simulation,
calculate upfront the values for market open and close in a
DataFrame, so that they values can be looked up by date, as
viewed as series while investigating data issues.
One downside of this implementation is that the entire history
has open and close values calculated, even though the simulation
may only be a subset of the trade data on record.
Should consider moving the `times` property and other methods
that care about the start and end date of a simulation to
SimulationParameters or another like object.
Instead of using all calendar days between start and end in test
sources, use the trading calendar for test sources.
Needed for an incoming refactoring of market open and close,
where the opens and closes are indexed by market days.
Instead of sliding to the next trading day because of the behavior
of `searchsorted`, if dt argument is not a trading day use it as a
max value for corresponding date of the index.
Fixes a bug where if the end of the quarter is calculated with
disregard to trading days, get_index would return the first day
of the next quarter, instead of the last trading day of the intended
quarter.
Instead of using a pandas Series of with dictionaries as the
values treasury curves, use a DataFrame which more naturally fits
the data type of a having a timeseries with mulitple values.
Should allow easier slicing/manipulation of the treasury curves,
e.g. getting 10 year curves would now be:
```
treasury_curves['10year']
```
Looking forwards to Python 3 capability, maintaing code that uses
iterkeys would require some overhead of checking what Python version
is available etc., instead use a DatetimeIndex so there is no need
to use key iterators, reverse, etc. on an OrderedDict, as the
DatetimeIndex allows 0 and -1 index, as well as the needed fast `in`
functionality.
The "use_environment" decorator is too side-effectful (e.g.,
connecting to Yahoo! Finance or another data source) to be used as a
decorator to a function that gets evaluated during module load. This
causes problems, e.g., if Zipline is being used in a gevent
environment, when the trading environment created by the decorator
argument tries to use greenlets when gevent hasn't been fully
initialized.
Since the decorator is nothing more than a context-manager wrapper,
this commit removes the decorator and replaces its use with contexts,
i.e., "with" statements.
instead of just limiting it with max_date. This allows for an
additional "live" day appended to the end, fixing various uses
of the trading_day_map, and replacing the isolated check in
get_next_close.
Working towards performance and risk logic being aware of
data frequency, as different handling of order of events based
on the data frequency is needed.
Both risk and performance now calculate performance since inception
(cumulative) and since the open. Both periods are updated intraday
and both are reported.
Batch risk for periods starting after the end of the treasury curve
history now use most recent curve.
Work towards running an algorithm against 'live' data, which can't
be bound to the available benchmarks and treasuries, since the
benchmarks and treasury curves for that day won't be published
until that night.
Stewart Douglas