From 3f969b7913b1b4bc2f6c88c692a47bbc1cdff75a Mon Sep 17 00:00:00 2001 From: Tobias Brandt Date: Wed, 14 Nov 2012 12:02:03 +0200 Subject: [PATCH 01/10] DOC: Made the Welcome page a little more welcoming and moved high-level notes out into a separate page. --- docs/index.rst | 71 ++++++++++++++++++----------------------------- docs/overview.rst | 47 +++++++++++++++++++++++++++++++ 2 files changed, 74 insertions(+), 44 deletions(-) create mode 100644 docs/overview.rst diff --git a/docs/index.rst b/docs/index.rst index 44172299..05ae6948 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -3,56 +3,41 @@ You can adapt this file completely to your liking, but it should at least contain the root `toctree` directive. -Contents: +.. module:: zipline + +******************************************* +Zipline: The Open Source Backtesting Engine +******************************************* + +Python is quickly becoming the glue language which holds together data science +and related fields like quantitative finance. Zipline is a new, BSD-licensed +quantitative trading system which allows easy backtesting of investment +algorithms on historical data. The system is fundamentally event-driven and a +close approximation of how live-trading systems operate. Moreover, Zipline +comes "batteries included" as many common statistics like +moving average and linear regression can be readily accessed from within a +user-written algorithm. Input of historical data and output of performance +statistics is based on Pandas DataFrames to integrate nicely into the existing +Python eco-system. Furthermore, statistic and machine learning libraries like +matplotlib, scipy, statsmodels, and sklearn support development, analysis and +visualization of state-of-the-art trading systems. + +Zipline is currently used in production as the backtesting engine powering +_ free, community-centered platform that allows +development and real-time backtesting of trading algorithms in the web browser. +Zipline was released at PyData NYC'12. + +Contents +======== .. toctree:: :maxdepth: 4 notes.rst + overview.rst modules.rst messaging.rst -Zipline -======= - -Zipline runs backtests using asynchronous components and zeromq messaging for communication and coordination. - -Simulator is the heart of Zipline, and the primary access point for creating, launching, and tracking simulations. You can find it in :py:class:`~zipline.core.Simulator` - -Simulator Sub-Components -======================== - -Each simulation contains numerous subcomponents, each operating asynchronously from all others, and communicating -via zeromq. - -DataSources --------------------- - -A DataSource represents a historical event record, which will be played back during simulation. A simulation may have one or more DataSources, which will be combined in DataFeed. Generally, datasources read records from a persistent store (db, csv file, remote service), format the messages for downstream simulation components, and send them to a PUSH socket. See the base class for all datasources :py:class:`~zipline.messaging.DataSource` and the module holding all datasources :py:mod:`zipline.sources` - -DataFeed --------------------- - -All simulations start with a collection of :py:class:`~zipline.messaging.DataSource`, which need to be fed to an algorithm. Each :py:class:`~zipline.sources.DataSource`can contain events of differing content (trades, quotes, corporate event) and frequency (quarterly, intraday). To simplify the process of managing the data sources, :py:class:`~zipline.core.DataFeed` can receive events from multiple :py:class:`DataSources ` and combine them into a serial chronological stream. - -Transforms --------------------- - -Often, an algorithm will require a running calculation on top of a :py:class:`~zipline.messaging.DataSource`, or on the consolidated feed. A simple example is a technical indicator or a moving average. Transforms can be described in :py:class:`~zipline.core.Simulator`'s configuration. Subclass :py:class:`~zipline.transforms.core.Transform` to add your own Transform. Transforms must hold their own state between events, and serialize their current values into messages. - - -Data Alignment --------------------- - -Like Datasources, Transforms have differing frequencies and results. Simulator manages the results of parallel transforms and **aligns** transform results with the raw DataFeed. Client algorithms simply receive a map of data, which includes the current event and all the transformed values. - -Time Compression --------------------- - - -Review the unit test coverage_. - - Indices and tables ================== @@ -60,5 +45,3 @@ Indices and tables * :ref:`genindex` * :ref:`modindex` * :ref:`search` - -.. _coverage: cover/index.html diff --git a/docs/overview.rst b/docs/overview.rst new file mode 100644 index 00000000..5e48d686 --- /dev/null +++ b/docs/overview.rst @@ -0,0 +1,47 @@ +******************************************* +Overview +******************************************* + +Simulations +=========== + +Zipline runs backtests using asynchronous components and zeromq messaging for communication and coordination. + +Simulator is the heart of Zipline, and the primary access point for creating, launching, and tracking simulations. You can find it in :py:class:`~zipline.core.Simulator` + +Simulator Sub-Components +======================== + +Each simulation contains numerous subcomponents, each operating asynchronously from all others, and communicating +via zeromq. + +DataSources +-------------------- + +A DataSource represents a historical event record, which will be played back during simulation. A simulation may have one or more DataSources, which will be combined in DataFeed. Generally, datasources read records from a persistent store (db, csv file, remote service), format the messages for downstream simulation components, and send them to a PUSH socket. See the base class for all datasources :py:class:`~zipline.messaging.DataSource` and the module holding all datasources :py:mod:`zipline.sources` + +DataFeed +-------------------- + +All simulations start with a collection of :py:class:`~zipline.messaging.DataSource`, which need to be fed to an algorithm. Each :py:class:`~zipline.sources.DataSource`can contain events of differing content (trades, quotes, corporate event) and frequency (quarterly, intraday). To simplify the process of managing the data sources, :py:class:`~zipline.core.DataFeed` can receive events from multiple :py:class:`DataSources ` and combine them into a serial chronological stream. + +Transforms +-------------------- + +Often, an algorithm will require a running calculation on top of a :py:class:`~zipline.messaging.DataSource`, or on the consolidated feed. A simple example is a technical indicator or a moving average. Transforms can be described in :py:class:`~zipline.core.Simulator`'s configuration. Subclass :py:class:`~zipline.transforms.core.Transform` to add your own Transform. Transforms must hold their own state between events, and serialize their current values into messages. + + +Data Alignment +-------------------- + +Like Datasources, Transforms have differing frequencies and results. Simulator manages the results of parallel transforms and **aligns** transform results with the raw DataFeed. Client algorithms simply receive a map of data, which includes the current event and all the transformed values. + +Time Compression +-------------------- + + +Review the unit test coverage_. + + + +.. _coverage: cover/index.html From 92fa9bf41d6c6564b0d747fdee1268004f3201d1 Mon Sep 17 00:00:00 2001 From: Tobias Brandt Date: Wed, 14 Nov 2012 12:12:42 +0200 Subject: [PATCH 02/10] DOC: Renamed the notes section the installation section. Fixed some formatting in the DataSources paragraph. --- docs/index.rst | 2 +- docs/{notes.rst => installation.rst} | 29 +++++++++++++++++++++------- 2 files changed, 23 insertions(+), 8 deletions(-) rename docs/{notes.rst => installation.rst} (82%) diff --git a/docs/index.rst b/docs/index.rst index 05ae6948..829c7aef 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -33,7 +33,7 @@ Contents .. toctree:: :maxdepth: 4 - notes.rst + installation.rst overview.rst modules.rst messaging.rst diff --git a/docs/notes.rst b/docs/installation.rst similarity index 82% rename from docs/notes.rst rename to docs/installation.rst index bf2d2237..358f4b35 100644 --- a/docs/notes.rst +++ b/docs/installation.rst @@ -1,3 +1,7 @@ +************ +Installation +************ + System Setup ============== @@ -78,11 +82,22 @@ Scientific python on the Mac can be a bit confusing because of the many independ Data Sources ============= -The Backtest can handle multiple concurrent data sources. QBT will start a subprocess to run each datasource, and merge all events from all sources into a single serial feed, ordered by date. -Data sources have events with very different frequencies. For example, liquid stocks will trade many times per minute, while illiquid stocks may trade just once a day. In order to serialize events from all sources into a single feed, qbt loads events from all sources into memory, then sorts. The communication happens like this: -1. QBT requests the next event from each data source, ignoring date (i.e. just next in sequence for all) -2. Using the earliest date from all the events from all sources, QBT then asks for "next after " from all sources. -3. All datasources send all events in their history from before , moving their internal pointer forward to the next unsent event. -4. QBT merges all events in memory -5. goto 1! +The Backtest can handle multiple concurrent data sources. QBT will start a +subprocess to run each datasource, and merge all events from all sources into a +single serial feed, ordered by date. + +Data sources have events with very different frequencies. For example, liquid +stocks will trade many times per minute, while illiquid stocks may trade just +once a day. In order to serialize events from all sources into a single feed, +qbt loads events from all sources into memory, then sorts. The communication +happens like this: + +1. QBT requests the next event from each data source, ignoring date (i.e. + just next in sequence for all) +2. Using the earliest date from all the events from all sources, QBT then + asks for "next after " from all sources. +3. All datasources send all events in their history from before , + moving their internal pointer forward to the next unsent event. +4. QBT merges all events in memory +5. goto 1! From 2f325ee2b43365b04b47b48b2bb44b4dd97d6e86 Mon Sep 17 00:00:00 2001 From: Tobias Brandt Date: Wed, 14 Nov 2012 12:17:12 +0200 Subject: [PATCH 03/10] DOC: Added chapter headings for Messaging and Extensions to make the TOC more readable. --- docs/extensions.rst | 4 ++++ docs/index.rst | 1 + docs/messaging.rst | 6 +++++- 3 files changed, 10 insertions(+), 1 deletion(-) diff --git a/docs/extensions.rst b/docs/extensions.rst index 50185706..639a58c1 100644 --- a/docs/extensions.rst +++ b/docs/extensions.rst @@ -1,3 +1,7 @@ +********** +Extensions +********** + .. highlight:: cython Philosophy diff --git a/docs/index.rst b/docs/index.rst index 829c7aef..fa37478e 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -37,6 +37,7 @@ Contents overview.rst modules.rst messaging.rst + extensions.rst Indices and tables diff --git a/docs/messaging.rst b/docs/messaging.rst index 5ade0428..f460cc8d 100644 --- a/docs/messaging.rst +++ b/docs/messaging.rst @@ -1,3 +1,7 @@ +********* +Messaging +********* + qbt runs backtests using multiple processes and zeromq messaging for communication and coordination. Backtest is the primary process. It maintains both server and client sockets: @@ -23,4 +27,4 @@ zmq sockets for backtest clients: - event feed, ZMQ.RESP. Port = port_start + 6 - backtest will bind to this socket and respond to requests from client for more data. Response data will be the queue of events that transpired since the last request. - \ No newline at end of file + From 03f582701f5810532b6f5b03c8593b7210be2bbb Mon Sep 17 00:00:00 2001 From: Tobias Brandt Date: Wed, 14 Nov 2012 12:20:15 +0200 Subject: [PATCH 04/10] DOC: Changed the modules heading. --- docs/modules.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/modules.rst b/docs/modules.rst index 199e3784..1e0777eb 100644 --- a/docs/modules.rst +++ b/docs/modules.rst @@ -1,5 +1,6 @@ -zipline -==== +*********************** +Packages and Modules +*********************** .. toctree:: :maxdepth: 4 From b8380499bc6c80dca7806c799463c06d0a39a306 Mon Sep 17 00:00:00 2001 From: Tobias Brandt Date: Wed, 14 Nov 2012 12:40:03 +0200 Subject: [PATCH 05/10] DOC: Added Quantopian Manifesto --- docs/index.rst | 3 +- docs/manifesto.rst | 140 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 142 insertions(+), 1 deletion(-) create mode 100644 docs/manifesto.rst diff --git a/docs/index.rst b/docs/index.rst index fa37478e..9df0fb79 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -23,7 +23,7 @@ matplotlib, scipy, statsmodels, and sklearn support development, analysis and visualization of state-of-the-art trading systems. Zipline is currently used in production as the backtesting engine powering -_ free, community-centered platform that allows +``_, a free, community-centered platform that allows development and real-time backtesting of trading algorithms in the web browser. Zipline was released at PyData NYC'12. @@ -33,6 +33,7 @@ Contents .. toctree:: :maxdepth: 4 + manifesto.rst installation.rst overview.rst modules.rst diff --git a/docs/manifesto.rst b/docs/manifesto.rst new file mode 100644 index 00000000..dbcf190b --- /dev/null +++ b/docs/manifesto.rst @@ -0,0 +1,140 @@ +******************** +Quantopian Manifesto +******************** + +Wall Street's culture was born in an age of information scarcity. +Hoarding information and keeping secrets were the norm. The world has changed. +Today's world is defined by information that wants to be free. The new +scarcity is people: people with the talent and drive to wring insight from all +of that data. + +Quantopian's mission is to attract the world's algorithmic and +financial talent. We want to attract today's quants, and we want to +attract talent that hasn't yet had the opportunity to be a quant. We +want to bring this talent together, provide them with the tools that they +require, and help them build a community. First and foremost, our community is +rooted in openness and sharing. Members share code, know-how, and data. +Quantopian sets the tone by providing open-sourced code, discussing our +techniques, and supplying the historical data needed for algorithmic investing. + +By educating more people about statistical arbitrage and data mining for +finance, we aim to dispense with the secrecy and raise the state of the art. +Rather than hoard data, we relentlessly push data to our community. We want to +diversify the data that can be mined, and permit our members to explore as much +as they like. Our members' success in analyzing and investing will help +us draw more data and more members to our community. Every individual's +success will also help other Quantopians. + +The Evolution of Algorithmic Finance +==================================== + +Charting +-------- + +Algorithmic finance originated as chart reading. Chartists would look for +certain patterns in price history charts. The patterns were always graced with +artfully chosen names like 'head and shoulders,' 'spinning top', or 'morning +star'. Chart reading looks a lot like palm reading, and for the skeptics among +us the similarities don't end with appearances. Still, chart reading is an +attempt to infer the balance of buying and selling appetites in the markets +from a stock's history. Viewed that way, chart reading pursues the noble goal +of prediction. Charting is so common that certain events can trigger market +responses, possibly because so many participants infer the same meaning from a +stock's price chart. + +Technical Analysis +------------------ + +Analysis grew more sophisticated as chartists gave way to +computer scientists writing algorithms. These algorithms have more scientific +sounding names like Moving Averages, Volume Weighted Moving Averages, Bollinger +Bands, Relative Strength Indicators, and Pearson's Correlation +Coefficient. Building technical analysis algorithms looks a lot like modern +statistics, and the optimists among us would say the similarities run deep. +Technical analysts take algorithmic approaches to the same concept: inferring +future behavior from trailing data. In addition to greater sophistication, +technical analysts can also test their algorithms over historic data. Imperfect +to be sure, but a giant leap from staring at a chart. + +Reasonable people can disagree about the 'correctness' of +inferring future events from past behavior. Rather than dwell on that question, +we choose to point out a different limitation of both charting and modern +technical analysis: **both interpret the movement of a single stock in +isolation**. This limitation is both a blessing and a curse. + +On the one hand, there is little room for sophisticated statistics or machine +learning when you have just a single time series for both your signal and your +prediction target. + +On the other, technical analysis can still be intuitive, which makes it easier +to get acquainted with the idea of automated trading. Often there is a mental +leap for people to make from understanding the interpretation of a price series +to issuing orders. Because the signals are easy to understand, technical +analysis makes for a good initial learning experience to explore risk and +performance evaluation as well as order management: the price going above its +30 day moving average is something you can visualize. So, you can focus your +attention on the financial and trading aspects of the problem. + +Statistical Arbitrage +--------------------- + +Statistical Arbitrage is the grandchild of chart reading. +Like technical analysis it relies on algorithms and statistics, but it departs +in one very significant way: 'stat arb' looks for relationships +among many stocks. The challenge with stat arb is twofold: + +* visualizing the relationships can be quite difficult, since the relationships + can have high dimensionality +* the data processing load is quite high - a simple linear regression for all + stocks results in 32 million individual regressions. Assuming a 10-day + window, that can be 320 million individual calculations. To prepare, + backtest, and trade a stat arb strategy required both familiarity with the + mechanics of trading, knowledge of statistics, and a strong computer science + background. + +As stat arb matured, the competition to find stat arb strategies that work +became a two part race: + +1. execute the trades faster +2. find new ways to identify relationships within + market data + +We think the pursuit of faster trades reached diminishing returns when the +market hit sub-millisecond trade execution. We think that the resulting high +level of liquidity is a good thing, but we agree with Thomas Petterffy that +`pursuing even faster trades +`_ +"has absolutely no social value". + +Finding new relationships in the market data is possible and more important now +than ever. In the summer of 2007, there was a sudden meltdown in quantitative +trading firms. Subsequent analysis points to quants crowding into the same +arbitrage bets, and an unforeseen fund liquidation driving all the quants to +unwind those bets concurrently. We believe finding new relationships should +permit investments with lower correlation and lower risks. + +Algorithmic Investing and the Future +==================================== + +A revolution in market understanding happens next. We want Quantopian to enable +more quants than all of Wall Street combined. We want quants, new and old, to +explore and share new ways to view the market. We want to clear away the +obstacles that have so far kept all but a few from doing algorithmic investing +by: + +* simulating with clean, high-quality market data for free +* easy access to markets through trusted brokers +* providing a robust, flexible open-source backtester to permit evaluation and + iteration of algorithms +* supporting a community that fosters the exchange of knowledge, ideas, code + solutions, and data sources + +The community will find new ways to identify market opportunities. It may take +the form of new, non-market data sources, like news feeds or Twitter. It may be +new algorithmic techniques. Most likely, it will be something we +haven't heard of yet: your idea. The one you keep coming back to. The +idea you couldn't test without data. The idea that needs backtesting, +and iteration, and encouragement from other quants. + +Do you want to unleash your idea? This is your chance. `Come hack Wall Street +`_. From 630ef8d4b91941e45d807ba48184775430e5ceff Mon Sep 17 00:00:00 2001 From: Tobias Brandt Date: Wed, 14 Nov 2012 12:44:00 +0200 Subject: [PATCH 06/10] DOC: Removed the messaging documentation since it seems obsolete. --- docs/index.rst | 1 - docs/messaging.rst | 30 ------------------------------ 2 files changed, 31 deletions(-) delete mode 100644 docs/messaging.rst diff --git a/docs/index.rst b/docs/index.rst index 9df0fb79..8195b494 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -37,7 +37,6 @@ Contents installation.rst overview.rst modules.rst - messaging.rst extensions.rst diff --git a/docs/messaging.rst b/docs/messaging.rst deleted file mode 100644 index f460cc8d..00000000 --- a/docs/messaging.rst +++ /dev/null @@ -1,30 +0,0 @@ -********* -Messaging -********* - -qbt runs backtests using multiple processes and zeromq messaging for communication and coordination. - -Backtest is the primary process. It maintains both server and client sockets: -zmq sockets for internal processing:: - - - data sink, ZMQ.REQ. Port = port_start + 1 - - backtest will connect to socket, and then spawn one process per datasource, passing the data sink url as a startup arg. Each - datasource process will bind to the socket, and start processing - - backtest is responsible for merging the data events from all sources into a serialized stream and relaying it to the - aggregators, merging agg results, and transmitting consolidated stream to event feed. - - agg source, ZMQ.PUSH. Port = port_start + 2 - - agg sink, ZMQ.PULL. Port = port_start + 3 - - control source, ZMQ.PUB. Port = port_start + 4 - - all child processes must subscribe to this socket. Control commands: - - START -- begin processing - - TIME -- current simulated time in backtest - - KILL -- exit immediately - -zmq sockets for backtest clients: -================================= - - orders sink, ZMQ.RESP. Port = port_start + 5 - - backtest will connect (can you bind?) to this socket and await orders from the client. Order data will be processed against the streaming datafeed. - - event feed, ZMQ.RESP. Port = port_start + 6 - - backtest will bind to this socket and respond to requests from client for more data. Response data will be the queue of events that - transpired since the last request. - From e955012fceda7a6d1599b8c28b98aba2282d3763 Mon Sep 17 00:00:00 2001 From: Tobias Brandt Date: Wed, 14 Nov 2012 12:54:04 +0200 Subject: [PATCH 07/10] DOC: Replaced installation instructions with the ones from pypi. --- docs/installation.rst | 38 +++++++++++++++++++++++--------------- 1 file changed, 23 insertions(+), 15 deletions(-) diff --git a/docs/installation.rst b/docs/installation.rst index 358f4b35..0ece920c 100644 --- a/docs/installation.rst +++ b/docs/installation.rst @@ -2,23 +2,31 @@ Installation ************ -System Setup -============== - -You need to have zeromq installed - http://www.zeromq.org/intro:get-the-software. +Since zipline is pure-python code it should be very easy to install +and set up with pip: -Running -------- +:: -Initial `virtualenv` setup:: + pip install zipline - $ mkvirtualenv zipline - $ workon zipline - #go get coffee, this will compile a heap of C/C++ code - $ ./etc/ordered_pip.sh requirements_sci.txt - $ ./etc/ordered_pip.sh requirements.txt - #optionally - $ ./etc/ordered_pip.sh requirements_dev.txt +If there are problems installing the dependencies or zipline we +recommend installing these packages via some other means. For Windows, +the `Enthought Python Distribution +`_ +includes most of the necessary dependencies. On OSX, the `Scipy Superpack +`_ works very well. + +Dependencies +------------ + +* Python (>= 2.7.2) +* numpy (>= 1.6.0) +* pandas (>= 0.9.0) +* pytz +* msgpack-python +* iso8601 +* Logbook +* blist Develop @@ -55,7 +63,7 @@ For building distributable egg:: Tooling hints ================ -QBT relies heavily on scientific python components (numpy, scikit, pandas, matplotlib, ipython, etc). Tooling up can be a pain, and it often involves managing a configuration including your OS, c/c++/fortran compilers, python version, and versions of numerous modules. I've found the following tools absolutely indispensable: +:mod:`zipline` relies heavily on scientific python components (numpy, scikit, pandas, matplotlib, ipython, etc). Tooling up can be a pain, and it often involves managing a configuration including your OS, c/c++/fortran compilers, python version, and versions of numerous modules. I've found the following tools absolutely indispensable: - some kind of package manager for your platform. package managers generally give you a way to search, install, uninstall, and check currently installed packages. They also do a great job of managing dependencies. - linux: yum/apt-get From 51838dbdd3bbb9d62ecc5b1a475dbd6457c80774 Mon Sep 17 00:00:00 2001 From: Tobias Brandt Date: Wed, 14 Nov 2012 13:18:55 +0200 Subject: [PATCH 08/10] DOC: I took more documentation from the pypi README page and added it to the Sphinx documentation. --- docs/contributing.rst | 40 +++++++++++++++++++++++++++++++ docs/index.rst | 34 +++++++++++++++++++------- docs/quickstart.rst | 55 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 121 insertions(+), 8 deletions(-) create mode 100644 docs/contributing.rst create mode 100644 docs/quickstart.rst diff --git a/docs/contributing.rst b/docs/contributing.rst new file mode 100644 index 00000000..d60d5af0 --- /dev/null +++ b/docs/contributing.rst @@ -0,0 +1,40 @@ +*************************** +Contributing to the project +*************************** + +Style Guide +=========== + +To ensure that changes and patches are focused on behavior changes, +the zipline codebase adheres to PEP-8, +``_. + +The maintainers check the code using the flake8 script, +``_, which is included in the +requirements_dev.txt. + +Before submitting patches or pull requests, please ensure that your +changes pass + +:: + + flake8 --ignore=E124,E125,E126 zipline tests + +Discussion and Help +=================== + +Discussion of the project is held at the Google Group, +``_, +``_. + +Source +====== + +The source for Zipline is hosted at +``_. + +Contact +======= + +For other questions, please contact ``_. + diff --git a/docs/index.rst b/docs/index.rst index 8195b494..9151c79c 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -5,9 +5,9 @@ .. module:: zipline -******************************************* -Zipline: The Open Source Backtesting Engine -******************************************* +**************************************************** +Zipline: Financial Backtester for Trading Algorithms +**************************************************** Python is quickly becoming the glue language which holds together data science and related fields like quantitative finance. Zipline is a new, BSD-licensed @@ -22,10 +22,27 @@ Python eco-system. Furthermore, statistic and machine learning libraries like matplotlib, scipy, statsmodels, and sklearn support development, analysis and visualization of state-of-the-art trading systems. -Zipline is currently used in production as the backtesting engine powering -``_, a free, community-centered platform that allows -development and real-time backtesting of trading algorithms in the web browser. -Zipline was released at PyData NYC'12. +Zipline is currently used in production as the backtesting engine +powering `quantopian.com `_ -- a free, community-centered +platform that allows development and real-time backtesting of trading +algorithms in the web browser. + +Features +======== + +* Ease of use: Zipline tries to get out of your way so that you can focus on + algorithm development. See below for a code example. + +* Zipline comes "batteries included" as many common statistics like moving + average and linear regression can be readily accessed from within a + user-written algorithm. + +* Input of historical data and output of performance statistics is based on + Pandas DataFrames to integrate nicely into the existing Python eco-system. + +* Statistic and machine learning libraries like matplotlib, scipy, statsmodels, + and sklearn support development, analysis and visualization of + state-of-the-art trading systems. Contents ======== @@ -35,11 +52,12 @@ Contents manifesto.rst installation.rst + quickstart.rst + contributing.rst overview.rst modules.rst extensions.rst - Indices and tables ================== diff --git a/docs/quickstart.rst b/docs/quickstart.rst new file mode 100644 index 00000000..01e77c55 --- /dev/null +++ b/docs/quickstart.rst @@ -0,0 +1,55 @@ +********** +Quickstart +********** + +Dual-Moving Average Example +=========================== + +The following code implements a simple dual moving average algorithm +and tests it on data extracted from yahoo finance. + +.. code:: python + + from zipline.algorithm import TradingAlgorithm + from zipline.transforms import MovingAverage + from zipline.utils.factory import load_from_yahoo + + class DualMovingAverage(TradingAlgorithm): + """Dual Moving Average algorithm. + """ + def initialize(self, short_window=200, long_window=400): + # Add 2 mavg transforms, one with a long window, one + # with a short window. + self.add_transform(MovingAverage, 'short_mavg', ['price'], + market_aware=True, + days=short_window) + + self.add_transform(MovingAverage, 'long_mavg', ['price'], + market_aware=True, + days=long_window) + + # To keep track of whether we invested in the stock or not + self.invested = False + + self.short_mavg = [] + self.long_mavg = [] + + + def handle_data(self, data): + if (data['AAPL'].short_mavg['price'] > data['AAPL'].long_mavg['price']) and not self.invested: + self.order('AAPL', 100) + self.invested = True + elif (data['AAPL'].short_mavg['price'] < data['AAPL'].long_mavg['price']) and self.invested: + self.order('AAPL', -100) + self.invested = False + + # Save mavgs for later analysis. + self.short_mavg.append(data['AAPL'].short_mavg['price']) + self.long_mavg.append(data['AAPL'].long_mavg['price']) + + data = load_from_yahoo() + dma = DualMovingAverage() + results = dma.run(data) + +You can find other examples in the zipline/examples directory. + From 220cc557dfae498bbdca76cf0130f3bc0242ad5c Mon Sep 17 00:00:00 2001 From: Tobias Brandt Date: Wed, 14 Nov 2012 13:43:03 +0200 Subject: [PATCH 09/10] DOC: Added a note on Time Compression and reformatted the .rst file. --- docs/overview.rst | 46 ++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 38 insertions(+), 8 deletions(-) diff --git a/docs/overview.rst b/docs/overview.rst index 5e48d686..c6bd676f 100644 --- a/docs/overview.rst +++ b/docs/overview.rst @@ -5,40 +5,70 @@ Overview Simulations =========== -Zipline runs backtests using asynchronous components and zeromq messaging for communication and coordination. +:mod:`zipline` runs backtests using asynchronous components and zeromq messaging for +communication and coordination. -Simulator is the heart of Zipline, and the primary access point for creating, launching, and tracking simulations. You can find it in :py:class:`~zipline.core.Simulator` +Simulator is the heart of :mod:zipline, and the primary access point for creating, +launching, and tracking simulations. You can find it in +:py:class:`~zipline.core.Simulator` Simulator Sub-Components ======================== -Each simulation contains numerous subcomponents, each operating asynchronously from all others, and communicating -via zeromq. +Each simulation contains numerous subcomponents, each operating asynchronously +from all others, and communicating via zeromq. DataSources -------------------- -A DataSource represents a historical event record, which will be played back during simulation. A simulation may have one or more DataSources, which will be combined in DataFeed. Generally, datasources read records from a persistent store (db, csv file, remote service), format the messages for downstream simulation components, and send them to a PUSH socket. See the base class for all datasources :py:class:`~zipline.messaging.DataSource` and the module holding all datasources :py:mod:`zipline.sources` +A DataSource represents a historical event record, which will be played back +during simulation. A simulation may have one or more DataSources, which will be +combined in DataFeed. Generally, datasources read records from a persistent +store (db, csv file, remote service), format the messages for downstream +simulation components, and send them to a PUSH socket. See the base class for +all datasources :py:class:`~zipline.messaging.DataSource` and the module +holding all datasources :py:mod:`zipline.sources` DataFeed -------------------- -All simulations start with a collection of :py:class:`~zipline.messaging.DataSource`, which need to be fed to an algorithm. Each :py:class:`~zipline.sources.DataSource`can contain events of differing content (trades, quotes, corporate event) and frequency (quarterly, intraday). To simplify the process of managing the data sources, :py:class:`~zipline.core.DataFeed` can receive events from multiple :py:class:`DataSources ` and combine them into a serial chronological stream. +All simulations start with a collection of +:py:class:`~zipline.messaging.DataSource`, which need to be fed to an +algorithm. Each :py:class:`~zipline.sources.DataSource`can contain events of +differing content (trades, quotes, corporate event) and frequency (quarterly, +intraday). To simplify the process of managing the data sources, +:py:class:`~zipline.core.DataFeed` can receive events from multiple +:py:class:`DataSources ` and combine them into a +serial chronological stream. Transforms -------------------- -Often, an algorithm will require a running calculation on top of a :py:class:`~zipline.messaging.DataSource`, or on the consolidated feed. A simple example is a technical indicator or a moving average. Transforms can be described in :py:class:`~zipline.core.Simulator`'s configuration. Subclass :py:class:`~zipline.transforms.core.Transform` to add your own Transform. Transforms must hold their own state between events, and serialize their current values into messages. +Often, an algorithm will require a running calculation on top of a +:py:class:`~zipline.messaging.DataSource`, or on the consolidated feed. A +simple example is a technical indicator or a moving average. Transforms can be +described in :py:class:`~zipline.core.Simulator`'s configuration. Subclass +:py:class:`~zipline.transforms.core.Transform` to add your own Transform. +Transforms must hold their own state between events, and serialize their +current values into messages. Data Alignment -------------------- -Like Datasources, Transforms have differing frequencies and results. Simulator manages the results of parallel transforms and **aligns** transform results with the raw DataFeed. Client algorithms simply receive a map of data, which includes the current event and all the transformed values. +Like Datasources, Transforms have differing frequencies and results. Simulator +manages the results of parallel transforms and **aligns** transform results +with the raw DataFeed. Client algorithms simply receive a map of data, which +includes the current event and all the transformed values. Time Compression -------------------- +According to `this post +`_ on the Quantopian +forums, time periods during which none of the selected SIDs were traded are +skipped. + Review the unit test coverage_. From b10bfc98a13ee904049ab0d6774636cbcdb1dba6 Mon Sep 17 00:00:00 2001 From: Tobias Brandt Date: Wed, 14 Nov 2012 14:29:50 +0200 Subject: [PATCH 10/10] DOC: Added autodoc generating links for all submodules --- docs/overview.rst | 4 +-- docs/zipline.data.rst | 40 ++++++++++++++++++++++++++ docs/zipline.finance.rst | 25 +++++++++++++++-- docs/zipline.gens.rst | 40 ++++++++++++++++++++++++++ docs/zipline.rst | 20 +++++++------ docs/zipline.transforms.rst | 47 +++++++++++++++++++++++++++++++ docs/zipline.utils.rst | 56 +++++++++++++++++++++++++++++++++++++ 7 files changed, 220 insertions(+), 12 deletions(-) create mode 100644 docs/zipline.data.rst create mode 100644 docs/zipline.gens.rst create mode 100644 docs/zipline.transforms.rst create mode 100644 docs/zipline.utils.rst diff --git a/docs/overview.rst b/docs/overview.rst index c6bd676f..9d7d3ad5 100644 --- a/docs/overview.rst +++ b/docs/overview.rst @@ -8,9 +8,9 @@ Simulations :mod:`zipline` runs backtests using asynchronous components and zeromq messaging for communication and coordination. -Simulator is the heart of :mod:zipline, and the primary access point for creating, +:class:`.algorithm.TradingAlgorithm` is the heart of :mod:`zipline`, and the primary access point for creating, launching, and tracking simulations. You can find it in -:py:class:`~zipline.core.Simulator` +:py:class:`~zipline.algorithm.TradingAlgorithm` Simulator Sub-Components ======================== diff --git a/docs/zipline.data.rst b/docs/zipline.data.rst new file mode 100644 index 00000000..c7269916 --- /dev/null +++ b/docs/zipline.data.rst @@ -0,0 +1,40 @@ +:mod:`zipline.data` subpackage +=============================== + +.. automodule:: zipline.data.__init__ + :members: + :undoc-members: + :show-inheritance: + +:mod:`benchmarks` Module +------------------------- + +.. automodule:: zipline.data.benchmarks + :members: + :undoc-members: + :show-inheritance: + +:mod:`loader` Module +-------------------- + +.. automodule:: zipline.data.loader + :members: + :undoc-members: + :show-inheritance: + +:mod:`loader_utils` Module +-------------------------- + +.. automodule:: zipline.data.loader_utils + :members: + :undoc-members: + :show-inheritance: + +:mod:`treasuries` Module +------------------------ + +.. automodule:: zipline.data.treasuries + :members: + :undoc-members: + :show-inheritance: + diff --git a/docs/zipline.finance.rst b/docs/zipline.finance.rst index e99189f6..33b8fe59 100644 --- a/docs/zipline.finance.rst +++ b/docs/zipline.finance.rst @@ -1,5 +1,18 @@ -finance Package -=============== +:mod:`zipline.finance` subpackage +================================== + +.. automodule:: zipline.finance.__init__ + :members: + :undoc-members: + :show-inheritance: + +:mod:`commission` Module +------------------------- + +.. automodule:: zipline.finance.commission + :members: + :undoc-members: + :show-inheritance: :mod:`performance` Module ------------------------- @@ -17,6 +30,14 @@ finance Package :undoc-members: :show-inheritance: +:mod:`slippage` Module +------------------------- + +.. automodule:: zipline.finance.slippage + :members: + :undoc-members: + :show-inheritance: + :mod:`trading` Module --------------------- diff --git a/docs/zipline.gens.rst b/docs/zipline.gens.rst new file mode 100644 index 00000000..b6d344aa --- /dev/null +++ b/docs/zipline.gens.rst @@ -0,0 +1,40 @@ +:mod:`zipline.gens` subpackage +============================== + +.. automodule:: zipline.gens.__init__ + :members: + :undoc-members: + :show-inheritance: + +:mod:`composites` Module +------------------------- + +.. automodule:: zipline.gens.composites + :members: + :undoc-members: + :show-inheritance: + +:mod:`sort` Module +------------------ + +.. automodule:: zipline.gens.sort + :members: + :undoc-members: + :show-inheritance: + +:mod:`tradesimulation` Module +------------------------------ + +.. automodule:: zipline.gens.tradesimulation + :members: + :undoc-members: + :show-inheritance: + +:mod:`utils` Module +--------------------- + +.. automodule:: zipline.gens.utils + :members: + :undoc-members: + :show-inheritance: + diff --git a/docs/zipline.rst b/docs/zipline.rst index cf294c88..87be42b1 100644 --- a/docs/zipline.rst +++ b/docs/zipline.rst @@ -1,18 +1,23 @@ -zipline Package -=============== - :mod:`zipline` Package ----------------------- +======================= .. automodule:: zipline.__init__ :members: :undoc-members: :show-inheritance: -:mod:`protocol` Module +:mod:`algorithm` Module +------------------------- + +.. automodule:: zipline.algorithm + :members: + :undoc-members: + :show-inheritance: + +:mod:`sources` Module ---------------------- -.. automodule:: zipline.protocol +.. automodule:: zipline.sources :members: :undoc-members: :show-inheritance: @@ -38,10 +43,9 @@ Subpackages .. toctree:: - zipline.core zipline.data zipline.finance zipline.gens - zipline.optimize + zipline.transforms zipline.utils diff --git a/docs/zipline.transforms.rst b/docs/zipline.transforms.rst new file mode 100644 index 00000000..84e9648d --- /dev/null +++ b/docs/zipline.transforms.rst @@ -0,0 +1,47 @@ +:mod:`zipline.transforms` subpackage +===================================== + +.. automodule:: zipline.transforms.__init__ + :members: + :undoc-members: + :show-inheritance: + +:mod:`mavg` Module +------------------------- + +.. automodule:: zipline.transforms.mavg + :members: + :undoc-members: + :show-inheritance: + +:mod:`returns` Module +------------------------- + +.. automodule:: zipline.transforms.returns + :members: + :undoc-members: + :show-inheritance: + +:mod:`stddev` Module +------------------------- + +.. automodule:: zipline.transforms.stddev + :members: + :undoc-members: + :show-inheritance: + +:mod:`utils` Module +------------------------- + +.. automodule:: zipline.transforms.utils + :members: + :undoc-members: + :show-inheritance: + +:mod:`vwap` Module +------------------------- + +.. automodule:: zipline.transforms.vwap + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/zipline.utils.rst b/docs/zipline.utils.rst new file mode 100644 index 00000000..e7851fe0 --- /dev/null +++ b/docs/zipline.utils.rst @@ -0,0 +1,56 @@ +:mod:`zipline.utils` subpackage +=============================== + +.. automodule:: zipline.utils.__init__ + :members: + :undoc-members: + :show-inheritance: + +:mod:`date_utils` Module +-------------------------- + +.. automodule:: zipline.utils.date_utils + :members: + :undoc-members: + :show-inheritance: + +:mod:`factory` Module +--------------------- + +.. automodule:: zipline.utils.factory + :members: + :undoc-members: + :show-inheritance: + +:mod:`protocol_units` Module +---------------------------- + +.. automodule:: zipline.utils.protocol_utils + :members: + :undoc-members: + :show-inheritance: + +:mod:`simfactory` Module +-------------------------- + +.. automodule:: zipline.utils.simfactory + :members: + :undoc-members: + :show-inheritance: + +:mod:`test_utils` Module +------------------------ + +.. automodule:: zipline.utils.test_utils + :members: + :undoc-members: + :show-inheritance: + +:mod:`tradingcalendar` Module +------------------------------ + +.. automodule:: zipline.utils.tradingcalendar + :members: + :undoc-members: + :show-inheritance: +