wassname/catalyst
1
0
Fork 0
mirror of https://github.com/wassname/catalyst.git synced 2026-06-27 20:04:30 +08:00
Code Issues Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'origin/concurrent-exchanges' into concurrent-exchanges

Browse Source
This commit is contained in:
fredfortier
2017-10-20 13:17:10 -04:00
parent f72074876d fadd4abe5a
commit 18e19bb1ae
7 changed files with 525 additions and 586 deletions
Download Patch File Download Diff File Expand all files Collapse all files
catalyst/__main__.py
+1 -1
View File
@@ -498,7 +498,7 @@ def ingest_exchange(exchange_name, data_frequency, start, end,
exchange = get_exchange(exchange_name)
exchange_bundle = ExchangeBundle(exchange)
click.echo('ingesting exchange bundle {}'.format(exchange_name))
click.echo('Ingesting exchange bundle {}...'.format(exchange_name))
exchange_bundle.ingest(
data_frequency=data_frequency,
include_symbols=include_symbols,
catalyst/examples/buy_btc_simple.py
+8
View File
@@ -0,0 +1,8 @@
from catalyst.api import order, record, symbol
def initialize(context):
context.asset = symbol('btc_usd')
def handle_data(context, data):
order(asset, 1)
record(btc=data.current(context.asset, 'price'))
docs/source/beginner-tutorial.rst
+191 -569
View File
@@ -1,608 +1,281 @@
Zipline Beginner Tutorial
-------------------------
Catalyst Beginner Tutorial
--------------------------
Basics
~~~~~~
Zipline is an open-source algorithmic trading simulator written in
Python.
Catalyst is an open-source algorithmic trading simulator for crypto
assets written in Python.
The source can be found at: https://github.com/quantopian/zipline
The source can be found at: https://github.com/enigmampc/catalyst
Some benefits include:
- Support for several of the top crypto-exchanges by trading volume.
- Realistic: slippage, transaction costs, order delays.
- Stream-based: Process each event individually, avoids look-ahead
bias.
- Batteries included: Common transforms (moving average) as well as
common risk calculations (Sharpe).
- Developed and continuously updated by
`Quantopian <https://www.quantopian.com>`__ which provides an
easy-to-use web-interface to Zipline, 10 years of minute-resolution
historical US stock data, and live-trading capabilities. This
tutorial is directed at users wishing to use Zipline without using
Quantopian. If you instead want to get started on Quantopian, see
`here <https://www.quantopian.com/faq#get-started>`__.
`Enigma MPC <https://www.enigma.co>`__ which is building the Enigma
data marketplace protocol as well as Catalyst, the first application
that will run on our protocol. Powered by our financial data
marketplace, Catalyst empowers users to share and curate data and
build profitable, data-driven investment strategies.
This tutorial assumes that you have zipline correctly installed, see the
`installation
instructions <https://github.com/quantopian/zipline#installation>`__ if
you haven't set up zipline yet.
This tutorial assumes that you have Catalyst correctly installed, see the
:doc:`installation instructions <install>` if you haven't set up
Catalyst yet.
Every ``zipline`` algorithm consists of two functions you have to
Every ``catalyst`` algorithm consists of at least two functions you have to
define:
* ``initialize(context)``
* ``handle_data(context, data)``
Before the start of the algorithm, ``zipline`` calls the
Before the start of the algorithm, ``catalyst`` calls the
``initialize()`` function and passes in a ``context`` variable.
``context`` is a persistent namespace for you to store variables you
need to access from one algorithm iteration to the next.
After the algorithm has been initialized, ``zipline`` calls the
After the algorithm has been initialized, ``catalyst`` calls the
``handle_data()`` function once for each event. At every call, it passes
the same ``context`` variable and an event-frame called ``data``
containing the current trading bar with open, high, low, and close
(OHLC) prices as well as volume for each stock in your universe. For
more information on these functions, see the `relevant part of the
Quantopian docs <https://www.quantopian.com/help#api-toplevel>`__.
(OHLC) prices as well as volume for each crypto asset in your universe.
.. For more information on these functions, see the `relevant part of the
.. Quantopian docs <https://www.quantopian.com/help#api-toplevel>`.
My first algorithm
~~~~~~~~~~~~~~~~~~
Lets take a look at a very simple algorithm from the ``examples``
directory, ``buyapple.py``:
directory, ``buy_btc.py``:
.. code-block:: python
from zipline.examples import buyapple
buyapple??
.. code-block:: python
from zipline.api import order, record, symbol
from catalyst.api import order, record, symbol
def initialize(context):
pass
context.asset = symbol('btc_usd')
def handle_data(context, data):
order(symbol('AAPL'), 10)
record(AAPL=data.current(symbol('AAPL'), 'price'))
order(context.asset, 1)
record(btc = data.current(context.asset, 'price'))
As you can see, we first have to import some functions we would like to
use. All functions commonly used in your algorithm can be found in
``zipline.api``. Here we are using :func:`~zipline.api.order()` which takes two
arguments: a security object, and a number specifying how many stocks you would
like to order (if negative, :func:`~zipline.api.order()` will sell/short
stocks). In this case we want to order 10 shares of Apple at each iteration. For
more documentation on ``order()``, see the `Quantopian docs
<https://www.quantopian.com/help#api-order>`__.
``catalyst.api``. Here we are using :func:`~catalyst.api.order()` which takes two
arguments: a cryptoasset object, and a number specifying how many assets you would
like to order (if negative, :func:`~catalyst.api.order()` will sell/short
assets). In this case we want to order 1 bitcoin at each iteration.
Finally, the :func:`~zipline.api.record` function allows you to save the value
.. For more documentation on ``order()``, see the `Quantopian docs
.. <https://www.quantopian.com/help#api-order>`__.
Finally, the :func:`~catalyst.api.record` function allows you to save the value
of a variable at each iteration. You provide it with a name for the variable
together with the variable itself: ``varname=var``. After the algorithm
finished running you will have access to each variable value you tracked
with :func:`~zipline.api.record` under the name you provided (we will see this
further below). You also see how we can access the current price data of the
AAPL stock in the ``data`` event frame (for more information see
`here <https://www.quantopian.com/help#api-event-properties>`__.
with :func:`~catalyst.api.record` under the name you provided (we will see this
further below). You also see how we can access the current price data of
a bitcoin in the ``data`` event frame.
.. (for more information see `here <https://www.quantopian.com/help#api-event-properties>`__.
Running the algorithm
~~~~~~~~~~~~~~~~~~~~~
To now test this algorithm on financial data, ``zipline`` provides three
interfaces: A command-line interface, ``IPython Notebook`` magic, and
:func:`~zipline.run_algorithm`.
To can now test this algorithm on crypto data, ``catalyst`` provides three
interfaces:
Ingesting Data
- A command-line interface,
- ``IPython Notebook`` magic,
- and :func:`~catalyst.run_algorithm`.
Ingesting data
^^^^^^^^^^^^^^
If you haven't ingested the data, run:
.. code-block:: bash
In previous versions of Catalyst you needed to manually ingest data before running
your algorithm to make it available at runtime. Starting with version 0.3, the
algorithm will automagically ingest the data it needs the first time that encounters
a data request for data that it doesn't have.
$ zipline ingest [-b <bundle>]
Still, we believe it is important for you to have a high-level understanding
of how data is managed:
where ``<bundle>`` is the name of the bundle to ingest, defaulting to
:ref:`quantopian-quandl <quantopian-quandl-mirror>`.
- Pricing data is split and packaged into ``bundles``: chunks of data organized
as time series that are kept up to date daily on Enigma's servers. Catalyst
downloads the bundles that needs at any given time, and reconstructs the whole
dataset in your hard drive.
you can check out the :ref:`ingesting data <ingesting-data>` section for
more detail.
- Pricing data is provided in ``daily`` and ``minute`` resolution. Those are different
bundle datasets, and are managed separately.
- Bundles are exchange-specific, as the pricing data is specific to the trades that
happen in each exchange. You can optionally specify which exchange you want pricing
data from.
- Catalyst keeps track of all the downloaded bundles, so that it only has to download
them once, and will do incremental updates as needed.
- When running in ``live trading`` mode, Catalyst will first look for historical
pricing data in the locally stored bundles. If there is anything missing, Catalyst will
hit the exchange for the most recent data, and merge it with the local bundle to make
it available for future iterations.
If you want to learn more, check out the :ref:`ingesting data <ingesting-data>` section
for more detail.
Command line interface
^^^^^^^^^^^^^^^^^^^^^^
After you installed zipline you should be able to execute the following
After you installed Catalyst you should be able to execute the following
from your command line (e.g. ``cmd.exe`` on Windows, or the Terminal app
on OSX):
on OSX). Displaying here a simplified output for eductional purposes:
.. code-block:: bash
$ zipline run --help
$ catalyst --help
.. parsed-literal::
Usage: zipline run [OPTIONS]
Usage: catalyst [OPTIONS] COMMAND [ARGS]...
Run a backtest for the given algorithm.
Top level catalyst entry point.
Options:
--version Show the version and exit.
--help Show this message and exit.
Commands:
ingest-exchange Ingest data for the given exchange.
live Trade live with the given algorithm.
run Run a backtest for the given algorithm.
There are three main modes you can run on Catalyst. The first being ``ingest-exchange``
for data ingestion, which we have summarized in the previous section. The second
is ``live`` to use your algorithm to trade live against a given exchange, and the
third mode ``run`` is to backtest your algorithm before trading live with it.
Let's start with backtesting, so run this other command to learn more about
the available options:
.. code-block:: bash
$ catalyst run --help
.. parsed-literal::
Usage: catalyst run [OPTIONS]
Run a backtest for the given algorithm.
Options:
-f, --algofile FILENAME The file that contains the algorithm to run.
-t, --algotext TEXT The algorithm script to run.
-D, --define TEXT Define a name to be bound in the namespace
before executing the algotext. For example
'-Dname=value'. The value may be any python
expression. These are evaluated in order so
they may refer to previously defined names.
--data-frequency [daily|minute]
The data frequency of the simulation.
[default: daily]
--capital-base FLOAT The starting capital for the simulation.
[default: 10000000.0]
-b, --bundle BUNDLE-NAME The data bundle to use for the simulation.
[default: poloniex]
--bundle-timestamp TIMESTAMP The date to lookup data on or before.
[default: <current-time>]
-s, --start DATE The start date of the simulation.
-e, --end DATE The end date of the simulation.
-o, --output FILENAME The location to write the perf data. If this
is '-' the perf will be written to stdout.
[default: -]
--print-algo / --no-print-algo Print the algorithm to stdout.
-x, --exchange-name [poloniex|bitfinex|bittrex]
The name of the targeted exchange
(supported: bitfinex, bittrex, poloniex).
-n, --algo-namespace TEXT A label assigned to the algorithm for data
storage purposes.
-c, --base-currency TEXT The base currency used to calculate
statistics (e.g. usd, btc, eth).
--help Show this message and exit.
Options:
-f, --algofile FILENAME The file that contains the algorithm to run.
-t, --algotext TEXT The algorithm script to run.
-D, --define TEXT Define a name to be bound in the namespace
before executing the algotext. For example
'-Dname=value'. The value may be any python
expression. These are evaluated in order so
they may refer to previously defined names.
--data-frequency [minute|daily]
The data frequency of the simulation.
[default: daily]
--capital-base FLOAT The starting capital for the simulation.
[default: 10000000.0]
-b, --bundle BUNDLE-NAME The data bundle to use for the simulation.
[default: quantopian-quandl]
--bundle-timestamp TIMESTAMP The date to lookup data on or before.
[default: <current-time>]
-s, --start DATE The start date of the simulation.
-e, --end DATE The end date of the simulation.
-o, --output FILENAME The location to write the perf data. If this
is '-' the perf will be written to stdout.
[default: -]
--print-algo / --no-print-algo Print the algorithm to stdout.
--help Show this message and exit.
As you can see there are a couple of flags that specify where to find your
algorithm (``-f``) as well as parameters specifying which data to use,
defaulting to the :ref:`quantopian-quandl-mirror`. There are also arguments for
the date range to run the algorithm over (``--start`` and ``--end``). Finally,
you'll want to save the performance metrics of your algorithm so that you can
analyze how it performed. This is done via the ``--output`` flag and will cause
it to write the performance ``DataFrame`` in the pickle Python file format.
Note that you can also define a configuration file with these parameters that
you can then conveniently pass to the ``-c`` option so that you don't have to
supply the command line args all the time (see the .conf files in the examples
directory).
algorithm (``-f``) as well as a parameter to specify which exchange to use.
There are also arguments for the date range to run the algorithm over
(``--start`` and ``--end``). Finally, you'll want to save the performance
metrics of your algorithm so that you can analyze how it performed. This is
done via the ``--output`` flag and will cause it to write the performance
``DataFrame`` in the pickle Python file format. Note that you can also define
a configuration file with these parameters that you can then conveniently pass
to the ``-c`` option so that you don't have to supply the command line args
all the time (see the .conf files in the examples directory).
Thus, to execute our algorithm from above and save the results to
``buyapple_out.pickle`` we would call ``zipline run`` as follows:
``buy_btc_simple_out.pickle`` we would call ``catalyst run`` as follows:
.. code-block:: python
zipline run -f ../../zipline/examples/buyapple.py --start 2000-1-1 --end 2014-1-1 -o buyapple_out.pickle
catalyst run -f buy_btc_simple.py -x bitfinex --start 2016-1-1 --end 2016-9-29 -o buy_simple_btc_out.pickle
.. parsed-literal::
..
.. parsed-literal
AAPL
[2015-11-04 22:45:32.820166] INFO: Performance: Simulated 3521 trading days out of 3521.
[2015-11-04 22:45:32.820314] INFO: Performance: first open: 2000-01-03 14:31:00+00:00
[2015-11-04 22:45:32.820401] INFO: Performance: last close: 2013-12-31 21:00:00+00:00
.. AAPL
.. [2015-11-04 22:45:32.820166] INFO: Performance: Simulated 3521 trading days out of 3521.
.. [2015-11-04 22:45:32.820314] INFO: Performance: first open: 2000-01-03 14:31:00+00:00
.. [2015-11-04 22:45:32.820401] INFO: Performance: last close: 2013-12-31 21:00:00+00:00
``run`` first calls the ``initialize()`` function, and then
streams the historical stock price day-by-day through ``handle_data()``.
After each call to ``handle_data()`` we instruct ``zipline`` to order 10
stocks of AAPL. After the call of the ``order()`` function, ``zipline``
streams the historical asset price day-by-day through ``handle_data()``.
After each call to ``handle_data()`` we instruct ``catalyst`` to order 1
bitcoin. After the call of the ``order()`` function, ``catalyst``
enters the ordered stock and amount in the order book. After the
``handle_data()`` function has finished, ``zipline`` looks for any open
``handle_data()`` function has finished, ``catalyst`` looks for any open
orders and tries to fill them. If the trading volume is high enough for
this stock, the order is executed after adding the commission and
this asset, the order is executed after adding the commission and
applying the slippage model which models the influence of your order on
the stock price, so your algorithm will be charged more than just the
stock price \* 10. (Note, that you can also change the commission and
slippage model that ``zipline`` uses, see the `Quantopian
docs <https://www.quantopian.com/help#ide-slippage>`__ for more
information).
asset price. (Note, that you can also change the commission and
slippage model that ``catalyst`` uses).
Lets take a quick look at the performance ``DataFrame``. For this, we
.. see the `Quantopian docs <https://www.quantopian.com/help#ide-slippage>`__
.. for more information).
Let's take a quick look at the performance ``DataFrame``. For this, we
use ``pandas`` from inside the IPython Notebook and print the first ten
rows. Note that ``zipline`` makes heavy usage of ``pandas``, especially
for data input and outputting so it's worth spending some time to learn
it.
rows. Note that ``catalyst`` makes heavy usage of
`pandas <http://pandas.pydata.org/>`_, especially for data input and
outputting so it's worth spending some time to learn it.
.. code-block:: python
import pandas as pd
perf = pd.read_pickle('buyapple_out.pickle') # read in perf DataFrame
perf = pd.read_pickle('buy_btc_simple_out.pickle') # read in perf DataFrame
perf.head()
.. raw:: html
<div style="max-height:1000px;max-width:1500px;overflow:auto;">
<table border="1" class="dataframe">
<thead>
<tr style="text-align: right;">
<th></th>
<th>AAPL</th>
<th>algo_volatility</th>
<th>algorithm_period_return</th>
<th>alpha</th>
<th>benchmark_period_return</th>
<th>benchmark_volatility</th>
<th>beta</th>
<th>capital_used</th>
<th>ending_cash</th>
<th>ending_exposure</th>
<th>...</th>
<th>short_exposure</th>
<th>short_value</th>
<th>shorts_count</th>
<th>sortino</th>
<th>starting_cash</th>
<th>starting_exposure</th>
<th>starting_value</th>
<th>trading_days</th>
<th>transactions</th>
<th>treasury_period_return</th>
</tr>
</thead>
<tbody>
<tr>
<th>2000-01-03 21:00:00</th>
<td>3.738314</td>
<td>0.000000e+00</td>
<td>0.000000e+00</td>
<td>-0.065800</td>
<td>-0.009549</td>
<td>0.000000</td>
<td>0.000000</td>
<td>0.00000</td>
<td>10000000.00000</td>
<td>0.00000</td>
<td>...</td>
<td>0</td>
<td>0</td>
<td>0</td>
<td>0.000000</td>
<td>10000000.00000</td>
<td>0.00000</td>
<td>0.00000</td>
<td>1</td>
<td>[]</td>
<td>0.0658</td>
</tr>
<tr>
<th>2000-01-04 21:00:00</th>
<td>3.423135</td>
<td>3.367492e-07</td>
<td>-3.000000e-08</td>
<td>-0.064897</td>
<td>-0.047528</td>
<td>0.323229</td>
<td>0.000001</td>
<td>-34.53135</td>
<td>9999965.46865</td>
<td>34.23135</td>
<td>...</td>
<td>0</td>
<td>0</td>
<td>0</td>
<td>0.000000</td>
<td>10000000.00000</td>
<td>0.00000</td>
<td>0.00000</td>
<td>2</td>
<td>[{u'order_id': u'513357725cb64a539e3dd02b47da7...</td>
<td>0.0649</td>
</tr>
<tr>
<th>2000-01-05 21:00:00</th>
<td>3.473229</td>
<td>4.001918e-07</td>
<td>-9.906000e-09</td>
<td>-0.066196</td>
<td>-0.045697</td>
<td>0.329321</td>
<td>0.000001</td>
<td>-35.03229</td>
<td>9999930.43636</td>
<td>69.46458</td>
<td>...</td>
<td>0</td>
<td>0</td>
<td>0</td>
<td>0.000000</td>
<td>9999965.46865</td>
<td>34.23135</td>
<td>34.23135</td>
<td>3</td>
<td>[{u'order_id': u'd7d4ad03cfec4d578c0d817dc3829...</td>
<td>0.0662</td>
</tr>
<tr>
<th>2000-01-06 21:00:00</th>
<td>3.172661</td>
<td>4.993979e-06</td>
<td>-6.410420e-07</td>
<td>-0.065758</td>
<td>-0.044785</td>
<td>0.298325</td>
<td>-0.000006</td>
<td>-32.02661</td>
<td>9999898.40975</td>
<td>95.17983</td>
<td>...</td>
<td>0</td>
<td>0</td>
<td>0</td>
<td>-12731.780516</td>
<td>9999930.43636</td>
<td>69.46458</td>
<td>69.46458</td>
<td>4</td>
<td>[{u'order_id': u'1fbf5e9bfd7c4d9cb2e8383e1085e...</td>
<td>0.0657</td>
</tr>
<tr>
<th>2000-01-07 21:00:00</th>
<td>3.322945</td>
<td>5.977002e-06</td>
<td>-2.201900e-07</td>
<td>-0.065206</td>
<td>-0.018908</td>
<td>0.375301</td>
<td>0.000005</td>
<td>-33.52945</td>
<td>9999864.88030</td>
<td>132.91780</td>
<td>...</td>
<td>0</td>
<td>0</td>
<td>0</td>
<td>-12629.274583</td>
<td>9999898.40975</td>
<td>95.17983</td>
<td>95.17983</td>
<td>5</td>
<td>[{u'order_id': u'9ea6b142ff09466b9113331a37437...</td>
<td>0.0652</td>
</tr>
</tbody>
</table>
<p>5 rows × 39 columns</p>
</div>
As you can see, there is a row for each trading day, starting on the
first business day of 2000. In the columns you can find various
There is a row for each trading day, starting on the first day of our
simulation Jan 1st, 2016. In the columns you can find various
information about the state of your algorithm. The very first column
``AAPL`` was placed there by the ``record()`` function mentioned earlier
and allows us to plot the price of apple. For example, we could easily
``btc`` was placed there by the ``record()`` function mentioned earlier
and allows us to plot the price of bitcoin. For example, we could easily
examine now how our portfolio value changed over time compared to the
AAPL stock price.
bitcoin price.
.. code-block:: python
%pylab inline
figsize(12, 12)
import matplotlib.pyplot as plt
ax1 = plt.subplot(211)
perf.portfolio_value.plot(ax=ax1)
ax1.set_ylabel('portfolio value')
ax2 = plt.subplot(212, sharex=ax1)
perf.AAPL.plot(ax=ax2)
ax2.set_ylabel('AAPL stock price')
.. parsed-literal::
Populating the interactive namespace from numpy and matplotlib
.. parsed-literal::
<matplotlib.text.Text at 0x7ff5c6147f90>
.. image:: tutorial_files/tutorial_11_2.png
As you can see, our algorithm performance as assessed by the
``portfolio_value`` closely matches that of the AAPL stock price. This
is not surprising as our algorithm only bought AAPL every chance it got.
IPython Notebook
~~~~~~~~~~~~~~~~
The `IPython Notebook <http://ipython.org/notebook.html>`__ is a very
powerful browser-based interface to a Python interpreter (this tutorial
was written in it). As it is already the de-facto interface for most
quantitative researchers ``zipline`` provides an easy way to run your
algorithm inside the Notebook without requiring you to use the CLI.
To use it you have to write your algorithm in a cell and let ``zipline``
know that it is supposed to run this algorithm. This is done via the
``%%zipline`` IPython magic command that is available after you
``import zipline`` from within the IPython Notebook. This magic takes
the same arguments as the command line interface described above. Thus
to run the algorithm from above with the same parameters we just have to
execute the following cell after importing ``zipline`` to register the
magic.
.. code-block:: python
%load_ext zipline
.. code-block:: python
%%zipline --start 2000-1-1 --end 2014-1-1
from zipline.api import symbol, order, record
def initialize(context):
pass
def handle_data(context, data):
order(symbol('AAPL'), 10)
record(AAPL=data[symbol('AAPL')].price)
Note that we did not have to specify an input file as above since the
magic will use the contents of the cell and look for your algorithm
functions there. Also, instead of defining an output file we are
specifying a variable name with ``-o`` that will be created in the name
space and contain the performance ``DataFrame`` we looked at above.
.. code-block:: python
_.head()
.. raw:: html
<div style="max-height:1000px;max-width:1500px;overflow:auto;">
<table border="1" class="dataframe">
<thead>
<tr style="text-align: right;">
<th></th>
<th>AAPL</th>
<th>algo_volatility</th>
<th>algorithm_period_return</th>
<th>alpha</th>
<th>benchmark_period_return</th>
<th>benchmark_volatility</th>
<th>beta</th>
<th>capital_used</th>
<th>ending_cash</th>
<th>ending_exposure</th>
<th>...</th>
<th>short_exposure</th>
<th>short_value</th>
<th>shorts_count</th>
<th>sortino</th>
<th>starting_cash</th>
<th>starting_exposure</th>
<th>starting_value</th>
<th>trading_days</th>
<th>transactions</th>
<th>treasury_period_return</th>
</tr>
</thead>
<tbody>
<tr>
<th>2000-01-03 21:00:00</th>
<td>3.738314</td>
<td>0.000000e+00</td>
<td>0.000000e+00</td>
<td>-0.065800</td>
<td>-0.009549</td>
<td>0.000000</td>
<td>0.000000</td>
<td>0.00000</td>
<td>10000000.00000</td>
<td>0.00000</td>
<td>...</td>
<td>0</td>
<td>0</td>
<td>0</td>
<td>0.000000</td>
<td>10000000.00000</td>
<td>0.00000</td>
<td>0.00000</td>
<td>1</td>
<td>[]</td>
<td>0.0658</td>
</tr>
<tr>
<th>2000-01-04 21:00:00</th>
<td>3.423135</td>
<td>3.367492e-07</td>
<td>-3.000000e-08</td>
<td>-0.064897</td>
<td>-0.047528</td>
<td>0.323229</td>
<td>0.000001</td>
<td>-34.53135</td>
<td>9999965.46865</td>
<td>34.23135</td>
<td>...</td>
<td>0</td>
<td>0</td>
<td>0</td>
<td>0.000000</td>
<td>10000000.00000</td>
<td>0.00000</td>
<td>0.00000</td>
<td>2</td>
<td>[{u'commission': 0.3, u'amount': 10, u'sid': 0...</td>
<td>0.0649</td>
</tr>
<tr>
<th>2000-01-05 21:00:00</th>
<td>3.473229</td>
<td>4.001918e-07</td>
<td>-9.906000e-09</td>
<td>-0.066196</td>
<td>-0.045697</td>
<td>0.329321</td>
<td>0.000001</td>
<td>-35.03229</td>
<td>9999930.43636</td>
<td>69.46458</td>
<td>...</td>
<td>0</td>
<td>0</td>
<td>0</td>
<td>0.000000</td>
<td>9999965.46865</td>
<td>34.23135</td>
<td>34.23135</td>
<td>3</td>
<td>[{u'commission': 0.3, u'amount': 10, u'sid': 0...</td>
<td>0.0662</td>
</tr>
<tr>
<th>2000-01-06 21:00:00</th>
<td>3.172661</td>
<td>4.993979e-06</td>
<td>-6.410420e-07</td>
<td>-0.065758</td>
<td>-0.044785</td>
<td>0.298325</td>
<td>-0.000006</td>
<td>-32.02661</td>
<td>9999898.40975</td>
<td>95.17983</td>
<td>...</td>
<td>0</td>
<td>0</td>
<td>0</td>
<td>-12731.780516</td>
<td>9999930.43636</td>
<td>69.46458</td>
<td>69.46458</td>
<td>4</td>
<td>[{u'commission': 0.3, u'amount': 10, u'sid': 0...</td>
<td>0.0657</td>
</tr>
<tr>
<th>2000-01-07 21:00:00</th>
<td>3.322945</td>
<td>5.977002e-06</td>
<td>-2.201900e-07</td>
<td>-0.065206</td>
<td>-0.018908</td>
<td>0.375301</td>
<td>0.000005</td>
<td>-33.52945</td>
<td>9999864.88030</td>
<td>132.91780</td>
<td>...</td>
<td>0</td>
<td>0</td>
<td>0</td>
<td>-12629.274583</td>
<td>9999898.40975</td>
<td>95.17983</td>
<td>95.17983</td>
<td>5</td>
<td>[{u'commission': 0.3, u'amount': 10, u'sid': 0...</td>
<td>0.0652</td>
</tr>
</tbody>
</table>
<p>5 rows × 39 columns</p>
</div>
Our algorithm performance as assessed by the
``portfolio_value`` closely matches that of the bitcoin price. This
is not surprising as our algorithm only bought bitcoin every chance it got.
Access to previous prices using ``history``
@@ -627,22 +300,16 @@ we need a new concept: History
``data.history()`` is a convenience function that keeps a rolling window of
data for you. The first argument is the number of bars you want to
collect, the second argument is the unit (either ``'1d'`` for ``'1m'``
but note that you need to have minute-level data for using ``1m``). For
a more detailed description ``history()``'s features, see the
`Quantopian docs <https://www.quantopian.com/help#ide-history>`__.
Let's look at the strategy which should make this clear:
but note that you need to have minute-level data for using ``1m``). This is
a function we use in the ``handle_data()`` section:
.. code-block:: python
%%zipline --start 2000-1-1 --end 2012-1-1 -o dma.pickle
from catalyst.api import order, record, symbol
from zipline.api import order_target, record, symbol
def initialize(context):
def initialize(context):
context.i = 0
context.asset = symbol('AAPL')
context.asset = symbol('btc_usd')
def handle_data(context, data):
# Skip first 300 days to get full windows
@@ -665,67 +332,22 @@ Let's look at the strategy which should make this clear:
order_target(context.asset, 0)
# Save values for later inspection
record(AAPL=data.current(context.asset, 'price'),
record(btc=data.current(context.asset, 'price'),
short_mavg=short_mavg,
long_mavg=long_mavg)
def analyze(context, perf):
fig = plt.figure()
ax1 = fig.add_subplot(211)
perf.portfolio_value.plot(ax=ax1)
ax1.set_ylabel('portfolio value in $')
ax2 = fig.add_subplot(212)
perf['AAPL'].plot(ax=ax2)
perf[['short_mavg', 'long_mavg']].plot(ax=ax2)
perf_trans = perf.ix[[t != [] for t in perf.transactions]]
buys = perf_trans.ix[[t[0]['amount'] > 0 for t in perf_trans.transactions]]
sells = perf_trans.ix[
[t[0]['amount'] < 0 for t in perf_trans.transactions]]
ax2.plot(buys.index, perf.short_mavg.ix[buys.index],
'^', markersize=10, color='m')
ax2.plot(sells.index, perf.short_mavg.ix[sells.index],
'v', markersize=10, color='k')
ax2.set_ylabel('price in $')
plt.legend(loc=0)
plt.show()
.. image:: tutorial_files/tutorial_22_1.png
Here we are explicitly defining an ``analyze()`` function that gets
automatically called once the backtest is done (this is not possible on
Quantopian currently).
Although it might not be directly apparent, the power of ``history()``
(pun intended) can not be under-estimated as most algorithms make use of
prior market developments in one form or another. You could easily
devise a strategy that trains a classifier with
`scikit-learn <http://scikit-learn.org/stable/>`__ which tries to
predict future market movements based on past prices (note, that most of
the ``scikit-learn`` functions require ``numpy.ndarray``\ s rather than
``pandas.DataFrame``\ s, so you can simply pass the underlying
``ndarray`` of a ``DataFrame`` via ``.values``).
We also used the ``order_target()`` function above. This and other
functions like it can make order management and portfolio rebalancing
much easier. See the `Quantopian documentation on order
functions <https://www.quantopian.com/help#api-order-methods>`__ fore
more details.
Conclusions
~~~~~~~~~~~
We hope that this tutorial gave you a little insight into the
architecture, API, and features of ``zipline``. For next steps, check
architecture, API, and features of ``catalyst``. For next steps, check
out some of the
`examples <https://github.com/quantopian/zipline/tree/master/zipline/examples>`__.
`examples <https://github.com/enigmampc/catalyst/tree/master/catalyst/examples>`__.
The natural next step would be too look into the
`buy_and_hodl <https://github.com/enigmampc/catalyst/blob/master/catalyst/examples/buy_and_hodl.py>`_
example, which is a more elaborated and realistic version of the ``buy_btc_simple`` example presented in this tutorial.
Feel free to ask questions on `our mailing
list <https://groups.google.com/forum/#!forum/zipline>`__, report
problems on our `GitHub issue
tracker <https://github.com/quantopian/zipline/issues?state=open>`__,
`get
involved <https://github.com/quantopian/zipline/wiki/Contribution-Requests>`__,
and `checkout Quantopian <https://quantopian.com>`__.
Feel free to ask questions on the ``#catalyst_dev`` channel of our
`Discord group <https://discord.gg/SJK32GY>`__ and report
problems on our `GitHub issue tracker <https://github.com/enigmampc/catalyst/issues>`__.
docs/source/index.rst
+11 -6
View File
@@ -1,12 +1,17 @@
.. include:: ../../README.rst
.. include:: welcome.rst
|
|
Table of Contents
-----------------
.. toctree::
:maxdepth: 1
install
beginner-tutorial
bundles
development-guidelines
appendix
release-process
releases
naming-convention
.. bundles
.. development-guidelines
.. appendix
.. release-process
.. releases
docs/source/install.rst
+220 -10
View File
@@ -13,7 +13,7 @@ There are two reasons for the additional complexity:
In order to build the C extensions, ``pip`` needs access to the CPython
header files for your Python installation.
2. Zipline depends on `numpy <http://www.numpy.org/>`_, the core library for
2. Catalyst depends on `numpy <http://www.numpy.org/>`_, the core library for
numerical array computing in Python. Numpy depends on having the `LAPACK
<http://www.netlib.org/lapack>`_ linear algebra routines available.
@@ -41,7 +41,15 @@ version:
$ virtualenv catalyst-venv
$ source ./catalyst-venv/bin/activate
$ pip install enigma-catalyst
$ pip install enigma-
Though not required by Catalyst directly, our example algorithms use matplotlib
to visually display the results of the trading algorithms. If you wish to run
any examples or use matplotlib during development, it can be installed using:
.. code-block:: bash
$ pip install matplotlib
GNU/Linux
~~~~~~~~~
@@ -71,8 +79,8 @@ On `Arch Linux`_, you can acquire the additional dependencies via ``pacman``:
..
.. There are also AUR packages available for installing `Python 3.4
.. <https://aur.archlinux.org/packages/python34/>`_ (Arch's default python is now
.. 3.5, but Zipline only currently supports 3.4), and `ta-lib
.. <https://aur.archlinux.org/packages/ta-lib/>`_, an optional Zipline dependency.
.. 3.5, but Catalyst only currently supports 3.4), and `ta-lib
.. <https://aur.archlinux.org/packages/ta-lib/>`_, an optional Catalyst dependency.
.. Python 2 is also installable via:
..
@@ -96,12 +104,132 @@ following brew packages:
$ brew install freetype pkg-config gcc openssl
OSX + virtualenv + matplotlib
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
A note about using matplotlib in virtual enviroments on OSX: it may be necessary to run
.. code-block:: bash
echo "backend: TkAgg" > ~/.matplotlib/matplotlibrc
in order to override the default ``macosx`` backend for your system, which may not
be accessible from inside the virtual environment. This will allow Catalyst to open
matplotlib charts from within a virtual environment, which is useful for displaying
the performance of your backtests. To learn more about matplotlib backends, please refer to the
`matplotlib backend documentation <https://matplotlib.org/faq/usage_faq.html#what-is-a-backend>`_.
Windows
~~~~~~~
For windows, the easiest and best supported way to install zipline is to use
In Windows, you will need the `Microsoft Visual C++ Compiler for Python 2.7
<https://www.microsoft.com/en-us/download/details.aspx?id=44266>`_. This package
contains the compiler and the set of system headers necessary for producing
binary wheels for Python 2.7 packages. If it's not already in your system, download
it and install it before proceeding to the next step.
For windows, the easiest and best supported way to install Catalyst is to use
:ref:`Conda <conda>`.
Amazon Linux AMI
~~~~~~~~~~~~~~~~
The packages ``pip`` and ``setuptools`` that come shipped by default are very outdated.
Thus, you first need to run:
.. code-block:: bash
pip install --upgrade pip setuptools
The default installation is also missing the C and C++ compilers, which you install by:
.. code-block:: bash
sudo yum install gcc gcc-c++
Then you should follow the regular installation instructions outlined at the beginning
of this page.
Troubleshooting ``pip`` Install
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
**Issue**:
Package enigma-catalyst cannot be found
**Solution**:
Make sure you have the most up-to-date version of pip installed, by running:
.. code-block:: bash
pip install --upgrade pip
On Windows, the recommended command is:
.. code-block:: bash
python -m pip install --upgrade pip
----
**Issue**:
Package enigma-catalyst cannot still be found, even after upgrading pip (see above), with an error similar to:
.. code-block:: bash
Downloading/unpacking enigma-catalyst
Could not find a version that satisfies the requirement enigma-catalyst (from versions: 0.1.dev9, 0.2.dev2, 0.1.dev4, 0.1.dev5, 0.1.dev3, 0.2.dev1, 0.1.dev8, 0.1.dev6)
Cleaning up...
No distributions matching the version for enigma-catalyst
**Solution**:
In some systems (this error has been reported in Ubuntu), pip is configured to only find stable versions by default. Since Catalyst is in alpha version, pip cannot find a matching version that satisfies the installation requirements. The solution is to include the `--pre` flag to include pre-release and development versions:
.. code-block:: bash
pip install --pre enigma-catalyst
----
**Issue**:
Package enigma-catalyst fails to install because of outdated setuptools
**Solution**:
Upgrade to the most up-to-date setuptools package by running:
.. code-block:: bash
pip install --upgrade pip setuptools
----
**Issue**:
Missing required packages
**Solution**:
Download `requirements.txt
<https://github.com/enigmampc/catalyst/blob/master/etc/requirements.txt>`_
(click on the *Raw* button and Right click -> Save As...) and use it to
install all the required dependencies by running:
.. code-block:: bash
pip install -r requirements.txt
----
**Issue**:
Installation fails with error: ``fatal error: Python.h: No such file or directory``
**Solution**:
Some systems (this issue has been reported in Ubuntu) require `python-dev` for the proper build and installation of package dependencies. The solution is to install python-dev, which is independent of the virtual environment. In Ubuntu, you would need to run:
.. code-block:: bash
sudo apt-get install python-dev
.. _conda:
Installing with ``conda``
@@ -118,14 +246,96 @@ without requiring the use of a second tool to acquire Catalyst's non-Python
dependencies.
For instructions on how to install ``conda``, see the `Conda Installation
Documentation <http://conda.pydata.org/docs/download.html>`_
Documentation <http://conda.pydata.org/docs/download.html>`_. Alternatively, you
can install MiniConda, which is a smaller footprint (fewer packages and smaller
size) than its big brother Anaconda, but it still contains all the main packages
needed. To install MiniConda, you can follow these steps:
Once conda has been set up you can install Catalyst from our ``Quantopian``
channel:
1. Download `MiniConda <https://conda.io/miniconda.html>`_. Select Python 2.7 for
your Operating System.
2. Install MiniConda. See the `Installation Instructions <https://conda.io/docs/user-guide/install/index.html>`_
if you need help.
3. Ensure the correct installation by running ``conda list`` in a Terminal window,
which should print the list of packages installed with Conda.
.. code-block:: bash
Once either Conda or MiniConda has been set up you can install Catalyst:
1. Download the file `python2.7-environment.yml <https://github.com/enigmampc/catalyst/blob/master/etc/python2.7-environment.yml>`_.
2. Open a Terminal window and enter [``cd/dir``] into the directory where you saved
the above ``python2.7-environment.yml`` file.
3. Install using this file. This step can take about 5-10 minutes to install.
.. code-block:: bash
conda env create -f python2.7-environment.yml
4. Activate the environment (which you need to do every time you start a new session
to run Catalyst):
**Linux or OSX:**
.. code-block:: bash
source activate catalyst
**Windows:**
.. code-block:: bash
activate catalyst
Congratulations! You now have Catalyst installed.
Troubleshooting ``conda`` Install
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If the command ``conda env create -f python2.7-environment.yml`` in step 3 above failed
for any reason, you can try setting up the environment manually with the following steps:
1. Create the environment:
.. code-block:: bash
conda create --name catalyst python=2.7 scipy
2. Activate the environment:
**Linux or OSX:**
.. code-block:: bash
source activate catalyst
**Windows:**
.. code-block:: bash
activate catalyst
3. Install the Catalyst inside the environment:
.. code-block:: bash
pip install enigma-catalyst matplotlib
Getting Help
------------
If after following the instructions above, and going through the *Troubleshooting* sections,
you still experience problems installing Catalyst, you can seek additional help through the
following channels:
- Join our `Discord community <https://discord.gg/SJK32GY>`_, and head over the #catalyst_dev
channel where many other users (as well as the project developers) hang out, and can assist
you with your particular issue. The more descriptive and the more information you can provide,
the easiest will be for others to help you out.
- Report the problem you are experiencing on our
`GitHub repository <https://github.com/enigmampc/catalyst/issues>`_ following the guidelines
provided therein. Before you do so, take a moment to browse through all `previous reported issues
<https://github.com/enigmampc/catalyst/issues?utf8=%E2%9C%93&q=is%3Aissue>`_ in the likely case
that someone else experienced that same issue before, and you get a hint on how to solve it.
conda install -c Quantopian zipline
.. _`Debian-derived`: https://www.debian.org/misc/children-distros
.. _`RHEL-derived`: https://en.wikipedia.org/wiki/Red_Hat_Enterprise_Linux_derivatives
docs/source/naming-convention.rst
+66
View File
@@ -0,0 +1,66 @@
Naming Convention
=================
Catalyst introduces a standardized naming convention for all asset pairs
trading on any exchange in the following form:
**{market_currency}_{base_currency}**
Where {market_currency} is the asset to be traded using {base_currency} as
the reference, both written in lowercase and separated with an underscore.
This standardization is needed to overcome the lack of consistency in the
naming of assets across different exchanges, and making it easier to the user
to refer to the asset pairs that you want to trade.
Catalyst maintains a `Market Coverage Overview <https://www.enigma.co/catalyst/status>`_
where you can check the mapping between Catalyst naming pairs and that of each
exchange. Catalyst will always expect in all its functions that you will refer to
the asset pairs by using the Catalyst naming convention.
If at any point, you input the wrong name for an asset pair, you will get an error
of that pair not found in the given exchange, and a list of pairs available on that exchange:
.. code-block:: bash
$ catalyst ingest-exchange -x poloniex -i btc_usd
.. parsed-literal::
Ingesting exchange bundle poloniex...
Error traceback: /Volumes/Data/Users/victoris/Desktop/Enigma/user-install/catalyst-dev/catalyst/exchange/exchange.py (line 175)
SymbolNotFoundOnExchange: Symbol btc_usd not found on exchange Poloniex.
Choose from: ['rep_usdt', 'gno_btc', 'xvc_btc', 'pink_btc', 'sys_btc',
'emc2_btc', 'rads_btc', 'note_btc', 'maid_btc', 'bch_btc', 'gnt_btc',
'bcn_btc', 'rep_btc', 'bcy_btc', 'cvc_btc', 'nxt_xmr', 'zec_usdt',
'fct_btc', 'gas_btc', 'pot_btc', 'eth_usdt', 'btc_usdt', 'lbc_btc',
'dcr_btc', 'etc_usdt', 'omg_eth', 'amp_btc', 'xpm_btc', 'nxt_btc',
'vtc_btc', 'steem_eth', 'blk_xmr', 'pasc_btc', 'zec_xmr', 'grc_btc',
'nxc_btc', 'btcd_btc', 'ltc_btc', 'dash_btc', 'naut_btc', 'zec_eth',
'zec_btc', 'burst_btc', 'zrx_eth', 'bela_btc', 'steem_btc', 'etc_btc',
'eth_btc', 'huc_btc', 'strat_btc', 'lsk_btc', 'exp_btc', 'clam_btc',
'rep_eth', 'dash_xmr', 'cvc_eth', 'bch_usdt', 'zrx_btc', 'dash_usdt',
'blk_btc', 'xrp_btc', 'nxt_usdt', 'neos_btc', 'omg_btc', 'bts_btc',
'doge_btc', 'gnt_eth', 'sbd_btc', 'gno_eth', 'xcp_btc', 'ltc_usdt',
'btm_btc', 'xmr_usdt', 'lsk_eth', 'omni_btc', 'nav_btc', 'fldc_btc',
'ppc_btc', 'xbc_btc', 'dgb_btc', 'sc_btc', 'btcd_xmr', 'vrc_btc',
'ric_btc', 'str_btc', 'maid_xmr', 'xmr_btc', 'sjcx_btc', 'via_btc',
'xem_btc', 'nmc_btc', 'etc_eth', 'ltc_xmr', 'ardr_btc', 'gas_eth',
'flo_btc', 'xrp_usdt', 'game_btc', 'bch_eth', 'bcn_xmr', 'str_usdt']
In the example above, exchange Poloniex does not use USD, but uses instead the
USDT cryptocurrency asset that is issued on the Bitcoin blockchain via the Omni
Layer Protocol. Each USDT unit is backed by a U.S Dollar held in the reserves of
Tether Limited. USDT can be transferred, stored, and spent, just like bitcoins
or any other cryptocurrency. Given its 1:1 mapping to the USD, is a viable alternative.
.. code-block:: bash
$ catalyst ingest-exchange -x poloniex -i btc_usdt
.. parsed-literal::
Ingesting exchange bundle poloniex...
[====================================] Fetching poloniex daily candles: : 100%
docs/source/welcome.rst
+28
View File
@@ -0,0 +1,28 @@
.. image:: https://s3.amazonaws.com/enigmaco-docs/enigma-catalyst.jpg
|
Catalyst is a data-driven crypto investment platform. It supports both
backtesting and live-trading in a number of different crypto-exchanges.
Catalyst empowers users to share and curate data and build profitable,
data-driven investment strategies.
Features
========
- Ease of use: Catalyst tries to get out of your way so that you can
focus on algorithm development. See
`examples of trading strategies <https://github.com/enigmampc/catalyst/tree/master/catalyst/examples>`_
provided.
- Support for several of the top crypto-exchanges by trading volume:
`Bitfinex <https://www.bitfinex.com>`_, `Bittrex <http://www.bittrex.com>`_,
and `Poloniex <https://www.poloniex.com>`_.
- Secure: You and only you have access to each exchange API keys for your accounts.
- Input of historical pricing data of all crypto-assets by exchange,
with daily and minute resolution. See
`Catalyst Market Coverage Overview <https://www.enigma.co/catalyst/status>`_.
- Backtesting and live-trading functionality, with a seamless transition
between the two modes.
- Output of performance statistics are based on Pandas DataFrames to
integrate nicely into the existing PyData eco-system.
- Statistic and machine learning libraries like matplotlib, scipy,
statsmodels, and sklearn support development, analysis, and
visualization of state-of-the-art trading systems.