Release 1.0.0 ------------- :Release: 1.0.0 :Date: May 19, 2016 Highlights ~~~~~~~~~~ Zipline 1.0 Rewrite (:issue:`1105`) ``````````````````````````````````` We have rewritten a lot of Zipline and its basic concepts in order to improve runtime performance. At the same time, we've introduced several new APIs. At a high level, earlier versions of Zipline simulations pulled from a multiplexed stream of data sources, which were merged via heapq. This stream was fed to the main simulation loop, driving the clock forward. This strong dependency on reading all the data made it difficult to optimize simulation performance because there was no connection between the amount of data we fetched and the amount of data actually used by the algorithm. Now, we only fetch data when the algorithm needs it. A new class, :class:`~zipline.data.data_portal.DataPortal`, dispatches data requests to various data sources and returns the requested values. This makes the runtime of a simulation scale much more closely with the complexity of the algorithm, rather than with the number of assets provided by the data sources. Instead of the data stream driving the clock, now simulations iterate through a pre-calculated set of day or minute timestamps. The timestamps are emitted by :class:`~zipline.gens.sim_engine.MinuteSimulationClock` and :class:`~zipline.gens.sim_engine.DailySimulationClock`, and consumed by the main loop in :meth:`~zipline.gens.tradesimulation.AlgorithmSimulator.transform`. We've retired the ``data[sid(N)]`` and ``history`` APIs, replacing them with several methods on the :class:`~zipline.protocol.BarData` object: :meth:`~zipline.protocol.BarData.current`, :meth:`~zipline.protocol.BarData.history`, :meth:`~zipline.protocol.BarData.can_trade`, and :meth:`~zipline.protocol.BarData.is_stale`. Old APIs will continue to work for now, but will issue deprecation warnings. You can now pass in an adjustments source to the :class:`~zipline.data.data_portal.DataPortal`, and we will apply adjustments to the pricing data when looking backwards at data. Prices and volumes for execution and presented to the algorithm in data.current are the as-traded value of the asset. New Entry Points (:issue:`1173` and :issue:`1178`) `````````````````````````````````````````````````` In order to make it easier to use zipline we have updated the entry points for a backtest. The three supported ways to run a backtest are now: 1. :func:`zipline.run_algo` 2. ``$ zipline run`` 3. ``%zipline`` (IPython magic) Data Bundles (:issue:`1173` and :issue:`1178`) `````````````````````````````````````````````` 1.0.0 introduces data bundles. Data bundles are groups of data that should be preloaded and used to run backtests later. This allows users to not need to to specify which tickers they are interested in each time they run an algorithm. This also allows us to cache the data between runs. By default, the ``quantopian-quandl`` bundle will be used which pulls data from Quantopian's mirror of the quandl `WIKI dataset `_. New bundles may be registered with :func:`zipline.data.bundles.register` like: .. code-block:: python @zipline.data.bundles.register('my-new-bundle') def my_new_bundle_ingest(environ, asset_db_writer, minute_bar_writer, daily_bar_writer, adjustment_writer, calendar, cache, show_progress): ... This function should retrieve the data it needs and then use the writers that have been passed to write that data to disc in a location that zipline can find later. This data can be used in backtests by passing the name as the ``-b / --bundle`` argument to ``$ zipline run`` or as the ``bundle`` argument to :func:`zipline.run_algorithm`. For more information see :ref:`data-bundles` for more information. String Support in Pipeline (:issue:`1174`) `````````````````````````````````````````` Added support for string data in Pipeline. :class:`zipline.pipeline.data.Column` now accepts ``object`` as a dtype, which signifies that loaders for that column should emit windowed iterators over the experimental new :class:`~zipline.lib.labelarray.LabelArray` class. Several new :class:`~zipline.pipeline.Classifier` methods have also been added for constructing :class:`~zipline.pipeline.Filter` instances based on string operations. The new methods are: - :meth:`~zipline.pipeline.Classifier.element_of` - :meth:`~zipline.pipeline.Classifier.startswith` - :meth:`~zipline.pipeline.Classifier.endswith` - :meth:`~zipline.pipeline.Classifier.has_substring` - :meth:`~zipline.pipeline.Classifier.matches` ``element_of`` is defined for all classifiers. The remaining methods are only defined for string-dtype classifiers. Enhancements ~~~~~~~~~~~~ * Made the data loading classes have more consistent interfaces. This includes the equity bar writers, adjustment writer, and asset db writer. The new interface is that the resource to be written to is passed at construction time and the data to write is provided later to the `write` method as dataframes or some iterator of dataframes. This model allows us to pass these writer objects around as a resource for other classes and functions to consume (:issue:`1109` and :issue:`1149`). * Added masking to :class:`zipline.pipeline.CustomFactor`. Custom factors can now be passed a Filter upon instantiation. This tells the factor to only compute over stocks for which the filter returns True, rather than always computing over the entire universe of stocks. (:issue:`1095`) * Added :class:`zipline.utils.cache.ExpiringCache`. A cache which wraps entries in a :class:`zipline.utils.cache.CachedObject`, which manages expiration of entries based on the `dt` supplied to the `get` method. (:issue:`1130`) * Implemented :class:`zipline.pipeline.factors.RecarrayField`, a new pipeline term designed to be the output type of a CustomFactor with multiple outputs. (:issue:`1119`) * Added optional `outputs` parameter to :class:`zipline.pipeline.CustomFactor`. Custom factors are now capable of computing and returning multiple outputs, each of which are themselves a Factor. (:issue:`1119`) * Added support for string-dtype pipeline columns. Loaders for thse columns should produce instances of :class:`zipline.lib.labelarray.LabelArray` when traversed. :meth:`~zipline.pipeline.data.BoundColumn.latest` on string columns produces a string-dtype :class:`zipline.pipeline.Classifier`. (:issue:`1174`) * Added several methods for converting Classifiers into Filters. The new methods are: - :meth:`~zipline.pipeline.Classifier.element_of` - :meth:`~zipline.pipeline.Classifier.startswith` - :meth:`~zipline.pipeline.Classifier.endswith` - :meth:`~zipline.pipeline.Classifier.has_substring` - :meth:`~zipline.pipeline.Classifier.matches` ``element_of`` is defined for all classifiers. The remaining methods are only defined for strings. (:issue:`1174`) * Added :class:`~zipline.pipeline.factors.BollingerBands` factor. This factor implements the Bollinger Bands technical indicator: https://en.wikipedia.org/wiki/Bollinger_Bands (:issue:`1199`). * Fetcher has been moved from Quantopian internal code into Zipline (:issue:`1105`). * Added new built-in factors, :class:`~zipline.pipeline.factors.RollingPearsonOfReturns`, :class:`~zipline.pipeline.factors.RollingSpearmanOfReturns` and :class:`~zipline.pipeline.factors.RollingLinearRegressionOfReturns` (:issue:`1154`) Experimental Features ~~~~~~~~~~~~~~~~~~~~~ .. warning:: Experimental features are subject to change. * Added a new :class:`zipline.lib.labelarray.LabelArray` class for efficiently representing and computing on string data with numpy. This class is conceptually similar to :class:`pandas.Categorical`, in that it represents string arrays as arrays of indices into a (smaller) array of unique string values. (:issue:`1174`) Bug Fixes ~~~~~~~~~ None Performance ~~~~~~~~~~~ None Maintenance and Refactorings ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ None Build ~~~~~ None Documentation ~~~~~~~~~~~~~ * Updated documentation for the API methods (:issue:`1188`). * Updated release process to mention that docs should be built with python 3 (:issue:`1188`). Miscellaneous ~~~~~~~~~~~~~ * Zipline now provides a `stub file `_ for the ``zipline.api`` module. This module is normally dynamically created so the stub file provides some static information for utilities that can consume it, for example PyCharm (:issue:`1208`).