Merge remote-tracking branch 'origin/develop' into develop

This commit is contained in:
fredfortier
2017-10-30 21:18:03 -04:00
9 changed files with 235 additions and 270 deletions
-10
View File
@@ -1,10 +0,0 @@
from catalyst.api import order, record, symbol
def initialize(context):
context.asset = symbol('btc_usd')
def handle_data(context, data):
order(context.asset, 1)
record(btc=data.current(context.asset, 'price'))
+21
View File
@@ -1,3 +1,24 @@
'''
This is a very simple example referenced in the beginner's tutorial:
https://enigmampc.github.io/catalyst/beginner-tutorial.html
Run this example, by executing the following from your terminal:
catalyst run -f buy_btc_simple.py -x bitfinex --start 2016-1-1 --end 2017-9-30 -o buy_btc_simple_out.pickle
If you want to run this code using another exchange, make sure that
the asset is available on that exchange. For example, if you were to run
it for exchange Poloniex, you would need to edit the following line:
context.asset = symbol('btc_usdt') # note 'usdt' instead of 'usd'
and specify exchange poloniex as follows:
catalyst run -f buy_btc_simple.py -x poloniex --start 2016-1-1 --end 2017-9-30 -o buy_btc_simple_out.pickle
To see which assets are available on each exchange, visit:
https://www.enigma.co/catalyst/status
'''
from catalyst.api import order, record, symbol
def initialize(context):
-105
View File
@@ -1,105 +0,0 @@
<h1>Live Trading</h1>
This document explains how to get started with live trading.
<h2>Supported Exchanges</h2>
Catalyst can trade against these exchanges:
* Bitfinex, id=`bitfinex`
* Bittrex, id=`bittrex`
<h3>Authentication</h3>
Most exchanges require key/token combination for authentication. By
convention, Catalyst uses an "auth.json" file to hold this data.
This example illustrates the convention using the Bitfinex exchange.
Here is how to generate key and secret values for bitfinex:
https://docs.bitfinex.com/v1/docs/api-access. Most exchanges follow
a similar process.
The auth.json file:
```json
{
"name": "bitfinex",
"key": "my-key",
"secret": "my-secret"
}
```
The file goes here:
```
~/.catalyst/data/exchanges/bitfinex/auth.json
```
Note that the 'bitfinex' directory corresponds to the id of the Bitfinex
exchange as defined in the "Supported Exchanges" section above.
Attempting to run an algorithm where the targeted exchange is missing
its "auth.json" file will create the directory structure but result
in an error.
<h3>Currency Symbols</h3>
Catalyst introduces a universal convention to reference
trading pairs and individual currencies. This
is required to ensure that the `symbol()` api predictably
returns the correct asset regardless of the targeted exchange.
Exchanges tend to use their own convention to represent currencies
(e.g. XBT and BTC both represent Bitcoin on different exchanges).
Trading pairs are also inconsistent. For example, Bitfinex
puts the market currency before the base currency without a
separator, Bittrex puts the base currency first and uses a dash
seperator.
Here is the Catalyst convention:
*[Market Currency]_[Base Currency]* all lowercase.
Currency symbols (e.g. btc, eth, ltc) follow the Bittrex convention.
Here are some examples:
```python
# With Bitfinex
bitcoin_usd_asset = symbol('btc_usd')
ethereum_bitcoin_asset = symbol('eth_btc')
# With Bittrex
ethereum_bitcoin_asset = symbol('eth_btc')
neo_ethereum_asset = symbol('neo_eth)
```
Note that the trading pairs are always referenced in the same manner.
However, not all trading pairs are available on all exchanges. An
error will occur if the specified trading pair is not trading
on the exchange.
<h2>Trading an Algorithm</h2>
There is no special convention to follow when writing an
algorithm for live trading. The same algorithm should work in
backtest and live execution mode without modification.
What differs are the arguments provided to the catalyst client or
`run_algorithm()` interface. Here is example:
```python
run_algorithm(
initialize=initialize,
handle_data=handle_data,
analyze=analyze,
exchange_name='bitfinex',
live=True,
algo_namespace='my_algo_trading_xrp',
base_currency='btc'
)
```
Here is the breakdown of the new arguments:
* live: Boolean flag which enables live trading.
* exchange_name: The name of the targeted exchange
(supported values: *bitfinex*, *bittrex*).
* algo_namespace: A arbitrary label assigned to your algorithm for
data storage purposes.
* base_currency: The base currency used to calculate the
statistics of your algorithm. Currently, the base currency of all
trading pairs of your algorithm must match this value.
Here is a complete algorithm for reference:
[Buy Low and Sell High](../catalyst/examples/buy_low_sell_high_live.py)
+16 -7
View File
@@ -255,7 +255,7 @@ slippage model that ``catalyst`` uses).
Let's take a quick look at the performance ``DataFrame``. For this, we
use ``pandas`` from inside the IPython Notebook and print the first ten
rows. and print the first ten rows. Note that ``catalyst`` makes heavy usage of
rows. Note that ``catalyst`` makes heavy usage of
`pandas <http://pandas.pydata.org/>`_, especially for data input and
outputting so it's worth spending some time to learn it.
@@ -429,6 +429,10 @@ and allows us to plot the price of bitcoin. For example, we could easily
examine now how our portfolio value changed over time compared to the
bitcoin price.
.. code-block:: python
%load_ext catalyst
.. code-block:: python
%pylab inline
@@ -483,8 +487,13 @@ but note that you need to have minute-level data for using ``1m``). This is
a function we use in the ``handle_data()`` section:
.. code-block:: python
%load_ext catalyst
.. code-block:: python
%%catalyst --start 2016-4-1 --end 2017-9-30 -x bitfinex
%%catalyst --start 2016-1-1 --end 2017-9-30 -x bitfinex -o dma.pickle
from catalyst.api import order, record, symbol, order_target
def initialize(context):
@@ -492,16 +501,16 @@ a function we use in the ``handle_data()`` section:
context.asset = symbol('btc_usd')
def handle_data(context, data):
# Skip first 300 days to get full windows
# Skip first 150 days to get full windows
context.i += 1
if context.i < 300:
if context.i < 150:
return
# Compute averages
# data.history() has to be called with the same params
# from above and returns a pandas dataframe.
short_mavg = data.history(context.asset, 'price', bar_count=100, frequency="1d").mean()
long_mavg = data.history(context.asset, 'price', bar_count=300, frequency="1d").mean()
short_mavg = data.history(context.asset, 'price', bar_count=50, frequency="1d").mean()
long_mavg = data.history(context.asset, 'price', bar_count=150, frequency="1d").mean()
# Trading logic
if short_mavg > long_mavg:
@@ -518,7 +527,7 @@ a function we use in the ``handle_data()`` section:
def analyze(context, perf):
import matplotlib.pyplot as plt
fig = plt.figure()
fig = plt.figure(figsize=(12,12))
ax1 = fig.add_subplot(211)
perf.portfolio_value.plot(ax=ax1)
ax1.set_ylabel('portfolio value in $')
+30 -105
View File
@@ -1,21 +1,17 @@
Development Guidelines
======================
This page is intended for developers of Zipline, people who want to contribute to the Zipline codebase or documentation, or people who want to install from source and make local changes to their copy of Zipline.
This page is intended for developers of Catalyst, people who want to contribute to the Catalyst codebase or documentation, or people who want to install from source and make local changes to their copy of Catalyst.
All contributions, bug reports, bug fixes, documentation improvements, enhancements and ideas are welcome. We `track issues`__ on `GitHub`__ and also have a `mailing list`__ where you can ask questions.
__ https://github.com/quantopian/zipline/issues
__ https://github.com/
__ https://groups.google.com/forum/#!forum/zipline
All contributions, bug reports, bug fixes, documentation improvements, enhancements and ideas are welcome. We `track issues <https://github.com/enigmampc/catalyst/issues>`_ on `GitHub <https://github.com/enigmampc/catalyst>`_ and also have a `discord group <https://discord.gg/SJK32GY>`_ where you can ask questions.
Creating a Development Environment
----------------------------------
First, you'll need to clone Zipline by running:
First, you'll need to clone Catalyst by running:
.. code-block:: bash
$ git clone git@github.com:your-github-username/zipline.git
$ git clone git@github.com:enigmampc/catalyst.git
Then check out to a new branch where you can make your changes:
@@ -23,15 +19,13 @@ Then check out to a new branch where you can make your changes:
$ git checkout -b some-short-descriptive-name
If you don't already have them, you'll need some C library dependencies. You can follow the `install guide`__ to get the appropriate dependencies.
__ install.html
If you don't already have them, you'll need some C library dependencies. You can follow the `install guide <install.html>`_ to get the appropriate dependencies.
The following section assumes you already have virtualenvwrapper and pip installed on your system. Suggested installation of Python library dependencies used for development:
.. code-block:: bash
$ mkvirtualenv zipline
$ mkvirtualenv catalyst
$ ./etc/ordered_pip.sh ./etc/requirements.txt
$ pip install -r ./etc/requirements_dev.txt
$ pip install -r ./etc/requirements_blaze.txt
@@ -42,104 +36,39 @@ Finally, you can build the C extensions by running:
$ python setup.py build_ext --inplace
To finish, make sure `tests`__ pass.
.. To finish, make sure `tests`__ pass.
__ #style-guide-running-tests
.. __ #style-guide-running-tests
If you get an error running nosetests after setting up a fresh virtualenv, please try running
.. If you get an error running nosetests after setting up a fresh virtualenv, please try running
.. code-block:: bash
.. code-block
# where zipline is the name of your virtualenv
$ deactivate zipline
$ workon zipline
.. # where zipline is the name of your virtualenv
.. $ deactivate zipline
.. $ workon zipline
Development with Docker
.. Development with Docker
.. -----------------------
..If you want to work with zipline using a `Docker`__ container, you'll need to build the ``Dockerfile`` in the Zipline root directory, and then build ``Dockerfile-dev``. Instructions for building both containers can be found in ``Dockerfile`` and ``Dockerfile-dev``, respectively.
.. __ https://docs.docker.com/get-started/
Git Branching Structure
-----------------------
If you want to work with zipline using a `Docker`__ container, you'll need to build the ``Dockerfile`` in the Zipline root directory, and then build ``Dockerfile-dev``. Instructions for building both containers can be found in ``Dockerfile`` and ``Dockerfile-dev``, respectively.
If you want to contribute to the codebase of Catalyst, familiarize yourself with our branching structure, a fairly standardized one for that matter, that follows what is documented in the following article: `A successful Git branching model <http://nvie.com/posts/a-successful-git-branching-model/>`_. To contribute, create your local branch and submit a Pull Request (PR) to the **develop** branch.
__ https://docs.docker.com/get-started/
.. image:: https://camo.githubusercontent.com/9bde6fb64a9542a572e0e2017cbb58d9d2c440ac/687474703a2f2f6e7669652e636f6d2f696d672f6769742d6d6f64656c4032782e706e67
Style Guide & Running Tests
---------------------------
We use `flake8`__ for checking style requirements and `nosetests`__ to run Zipline tests. Our `continuous integration`__ tools will run these commands.
__ http://flake8.pycqa.org/en/latest/
__ http://nose.readthedocs.io/en/latest/
__ https://en.wikipedia.org/wiki/Continuous_integration
Before submitting patches or pull requests, please ensure that your changes pass when running:
.. code-block:: bash
$ flake8 zipline tests
In order to run tests locally, you'll need `TA-lib`__, which you can install on Linux by running:
__ https://mrjbq7.github.io/ta-lib/install.html
.. code-block:: bash
$ wget http://prdownloads.sourceforge.net/ta-lib/ta-lib-0.4.0-src.tar.gz
$ tar -xvzf ta-lib-0.4.0-src.tar.gz
$ cd ta-lib/
$ ./configure --prefix=/usr
$ make
$ sudo make install
And for ``TA-lib`` on OS X you can just run:
.. code-block:: bash
$ brew install ta-lib
Then run ``pip install`` TA-lib:
.. code-block:: bash
$ pip install -r ./etc/requirements_talib.txt
You should now be free to run tests:
.. code-block:: bash
$ nosetests
Continuous Integration
----------------------
We use `Travis CI`__ for Linux-64 bit builds and `AppVeyor`__ for Windows-64 bit builds.
.. note::
We do not currently have CI for OSX-64 bit builds. 32-bit builds may work but are not included in our integration tests.
__ https://travis-ci.org/quantopian/zipline
__ https://ci.appveyor.com/project/quantopian/zipline
Packaging
---------
To learn about how we build Zipline conda packages, you can read `this`__ section in our release process notes.
__ release-process.html#uploading-conda-packages
Contributing to the Docs
------------------------
If you'd like to contribute to the documentation on zipline.io, you can navigate to ``docs/source/`` where each `reStructuredText`__ (``.rst``) file is a separate section there. To add a section, create a new file called ``some-descriptive-name.rst`` and add ``some-descriptive-name`` to ``appendix.rst``. To edit a section, simply open up one of the existing files, make your changes, and save them.
__ https://en.wikipedia.org/wiki/ReStructuredText
We use `Sphinx`__ to generate documentation for Zipline, which you will need to install by running:
__ http://www.sphinx-doc.org/en/stable/
If you'd like to contribute to the documentation on enigmampc.github.io, you can navigate to ``docs/source/`` where each `reStructuredText <https://en.wikipedia.org/wiki/ReStructuredText>`_ file is a separate section there. To add a section, create a new file called ``some-descriptive-name.rst`` and add ``some-descriptive-name`` to ``index.rst``. To edit a section, simply open up one of the existing files, make your changes, and save them.
We use `Sphinx <http://www.sphinx-doc.org/en/stable/>`_ to generate documentation for Catalyst, which you will need to install by running:
.. code-block:: bash
@@ -149,7 +78,7 @@ To build and view the docs locally, run:
.. code-block:: bash
# assuming you're in the Zipline root directory
# assuming you're in the Catalyst root directory
$ cd docs
$ make html
$ {BROWSER} build/html/index.html
@@ -162,7 +91,7 @@ Standard prefixes to start a commit message:
.. code-block:: text
BLD: change related to building Zipline
BLD: change related to building Catalyst
BUG: bug fix
DEP: deprecate something, or remove a deprecated object
DEV: development tool or utility
@@ -172,15 +101,13 @@ Standard prefixes to start a commit message:
REV: revert an earlier commit
STY: style fix (whitespace, PEP8, flake8, etc)
TST: addition or modification of tests
REL: related to releasing Zipline
REL: related to releasing Catalyst
PERF: performance enhancements
Some commit style guidelines:
Commit lines should be no longer than `72 characters`__. The first line of the commit should include one of the above prefixes. There should be an empty line between the commit subject and the body of the commit. In general, the message should be in the imperative tense. Best practice is to include not only what the change is, but why the change was made.
__ https://git-scm.com/book/en/v2/Distributed-Git-Contributing-to-a-Project
Commit lines should be no longer than `72 characters <https://git-scm.com/book/en/v2/Distributed-Git-Contributing-to-a-Project>`_. The first line of the commit should include one of the above prefixes. There should be an empty line between the commit subject and the body of the commit. In general, the message should be in the imperative tense. Best practice is to include not only what the change is, but why the change was made.
**Example:**
@@ -203,8 +130,6 @@ __ https://git-scm.com/book/en/v2/Distributed-Git-Contributing-to-a-Project
Formatting Docstrings
---------------------
When adding or editing docstrings for classes, functions, etc, we use `numpy`__ as the canonical reference.
__ https://github.com/numpy/numpy/blob/master/doc/HOWTO_DOCUMENT.rst.txt
When adding or editing docstrings for classes, functions, etc, we use `numpy <https://github.com/numpy/numpy/blob/master/doc/HOWTO_DOCUMENT.rst.txt>`_ as the canonical reference.
+2
View File
@@ -9,8 +9,10 @@ Table of Contents
install
beginner-tutorial
live-trading
naming-convention
videos
development-guidelines
.. bundles
.. development-guidelines
.. appendix
+32 -1
View File
@@ -39,9 +39,10 @@ version:
.. code-block:: bash
$ pip install virtualenv
$ virtualenv catalyst-venv
$ source ./catalyst-venv/bin/activate
$ pip install enigma-
$ pip install enigma-catalyst
Though not required by Catalyst directly, our example algorithms use matplotlib
to visually display the results of the trading algorithms. If you wish to run
@@ -132,6 +133,36 @@ it and install it before proceeding to the next step.
For windows, the easiest and best supported way to install Catalyst is to use
:ref:`Conda <conda>`.
Some problems we have encountered installing the **Visual C++ Compiler** mentioned above
are as follows:
- **The system administrator has set policies to prevent this installation**.
In some systems, there is a default *Windows Software Restriction* policy that
prevents the installation of some software packages like this one. You'll have
to change the Registry to circumvent this:
- Click ``Start``, and search for ``regedit`` and launch the ``Registry Editor``
- Navigate to the following folder:
``HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\Installer``
- If there is an entry for ``DisableMSI``, set the Value data to 0.
- If there is no such entry, click on the ``Edit`` menu -> ``New`` -> ``DWORD (32-bit) Value``
and enter ``DisableMSI`` as the Name (and by default you get 0 as the Value Data)
|
- **The installer has encountered an unexpected error installing this package.
This may indicate a problem with this package. The error code is 2503.**
We have observed this when trying to install a package without enough administrator
permissions. Even when you are logged in as an Administrator, you have to explictily
install this package with administrator privileges:
- Click ``Start`` and find ``CMD`` or ``Command Prompt``
- Right click on it and choose ``Run as administrator``
- ``cd`` into the folder where you downloaded ``VCForPython27.msi``
- Run ``msiexec /i VCForPython27.msi``
Amazon Linux AMI
~~~~~~~~~~~~~~~~
+118
View File
@@ -0,0 +1,118 @@
Live Trading
============
This document explains how to get started with live trading.
Supported Exchanges
^^^^^^^^^^^^^^^^^^^
Catalyst can trade against these exchanges:
- Bitfinex, id= ``bitfinex``
- Bittrex, id= ``bittrex``
- Poloniex, id= ``poloniex``
Authentication
^^^^^^^^^^^^^^
Most exchanges require token key/secret combination for authentication. By
convention, Catalyst uses an ``auth.json`` file to hold this data.
This example illustrates the convention using the *Bitfinex* exchange.
Here is how to generate key and secret values for the Bitfinex exchange:
https://docs.bitfinex.com/v1/docs/api-access. Most exchanges follow
a similar process.
The auth.json file:
.. code-block:: json
{
"name": "bitfinex",
"key": "my-key",
"secret": "my-secret"
}
The file goes here: ``~/.catalyst/data/exchanges/bitfinex/auth.json``
Note that the `bitfinex` part in the directory above corresponds to the id of the Bitfinex
exchange as defined in the "Supported Exchanges" section above.
Attempting to run an algorithm where the targeted exchange is missing
its ``auth.json`` file will create the directory structure and create an empty
auth.json file, but will result in an error.
Currency Symbols
^^^^^^^^^^^^^^^^
Catalyst introduces a universal convention to reference
trading pairs and individual currencies. This
is required to ensure that the ``symbol()`` api predictably
returns the correct asset regardless of the targeted exchange.
Exchanges tend to use their own convention to represent currencies
(e.g. XBT and BTC both represent Bitcoin on different exchanges).
Trading pairs are also inconsistent. For example, Bitfinex
puts the market currency before the base currency without a
separator, Bittrex puts the base currency first and uses a dash
seperator.
Here is the Catalyst convention:
*[Market Currency]_[Base Currency]* all lowercase.
Currency symbols (e.g. btc, eth, ltc) follow the Bittrex convention.
Here are some examples:
.. code-block:: json
# With Bitfinex
bitcoin_usd_asset = symbol('btc_usd')
ethereum_bitcoin_asset = symbol('eth_btc')
# With Bittrex
ethereum_bitcoin_asset = symbol('eth_btc')
neo_ethereum_asset = symbol('neo_eth)
Note that the trading pairs are always referenced in the same manner.
However, not all trading pairs are available on all exchanges. An
error will occur if the specified trading pair is not trading
on the exchange. To check which currency pairs are available on each
of the supported exchanges, see `Catalyst Market Coverage <https://www.enigma.co/catalyst/status`_.
Trading an Algorithm
^^^^^^^^^^^^^^^^^^^^
There is no special convention to follow when writing an
algorithm for live trading. The same algorithm should work in
backtest and live execution mode without modification.
What differs are the arguments provided to the catalyst client or
`run_algorithm()` interface. Here is the same example in both interfaces:
.. code-block:: bash
catalyst live -f my_algo_code -x bitfinex -c btc -n my_algo_name
.. code-block:: python
run_algorithm(
initialize=initialize,
handle_data=handle_data,
analyze=analyze,
exchange_name='bitfinex',
live=True,
algo_namespace='my_algo_name',
base_currency='btc'
)
Here is the breakdown of the new arguments:
- ``live``: Boolean flag which enables live trading.
- ``exchange_name``: The name of the targeted exchange
(supported values: *bitfinex*, *bittrex*).
- ``algo_namespace``: A arbitrary label assigned to your algorithm for
data storage purposes.
- ``base_currency``: The base currency used to calculate the
statistics of your algorithm. Currently, the base currency of all
trading pairs of your algorithm must match this value.
Here is a complete algorithm for reference:
`Buy Low and Sell High <https://github.com/enigmampc/catalyst/blob/master/catalyst/examples/buy_low_sell_high_live.py>`_
+16 -42
View File
@@ -1,30 +1,22 @@
name: catalyst
channels:
- statiskit
- defaults
dependencies:
- certifi=2016.2.28=py27_0
- coverage=4.4.1=py27_0
- nose=1.3.7=py27_1
- openssl=1.0.2l=0
- path.py=10.3.1=py27_0
- mkl=2017.0.3
- numpy=1.13.1=py27_0
- openssl=1.0.2l
- pip=9.0.1=py27_1
- python=2.7.13=0
- pyyaml=3.12=py27_0
- readline=6.2=2
- setuptools=36.4.0=py27_0
- six=1.10.0=py27_0
- sqlite=3.13.0=0
- tk=8.5.18=0
- python=2.7.13
- scipy=0.19.1=np113py27_0
- setuptools=36.4.0=py27_1
- sqlite=3.13.0
- tk=8.5.18
- wheel=0.29.0=py27_0
- yaml=0.1.6=0
- zlib=1.2.11=0
- libdev=1.0.0=py27_0
- python-dev=1.0.0=py27_0
- python-scons=3.0.0=py27_0
- zlib=1.2.11
- pip:
- alembic==0.9.5
- backports.shutil-get-terminal-size==1.0.0
- alembic==0.9.6
- backports.functools-lru-cache==1.4
- bcolz==0.12.1
- bottleneck==1.2.1
- chardet==3.0.4
@@ -32,36 +24,22 @@ dependencies:
- contextlib2==0.5.5
- cycler==0.10.0
- cyordereddict==1.0.0
- cython==0.26.1
- cython==0.27.1
- decorator==4.1.2
- empyrical==0.2.1
- enigma-catalyst>=0.2.dev2
- enum34==1.1.6
- functools32==3.2.3.post2
- idna==2.6
- intervaltree==2.1.0
- ipdb==0.10.3
- ipdbplugin==1.4.5
- ipython==5.5.0
- ipython-genutils==0.2.0
- logbook==1.1.0
- lru-dict==1.1.6
- mako==1.0.7
- markupsafe==1.0
- matplotlib==2.0.2
- matplotlib==2.1.0
- multipledispatch==0.4.9
- networkx==1.11
- networkx==2.0
- numexpr==2.6.4
- numpy==1.13.1
- pandas==0.19.2
- pandas-datareader==0.5.0
- pathlib2==2.3.0
- patsy==0.4.1
- pexpect==4.2.1
- pickleshare==0.7.4
- prompt-toolkit==1.0.15
- ptyprocess==0.5.2
- pygments==2.2.0
- pyparsing==2.2.0
- python-dateutil==2.6.1
- python-editor==1.0.3
@@ -69,16 +47,12 @@ dependencies:
- requests==2.18.4
- requests-file==1.4.2
- requests-ftp==0.3.1
- scandir==1.5
- scipy==0.19.1
- scons==3.0.0a20170821
- simplegeneric==0.8.1
- six==1.11.0
- sortedcontainers==1.5.7
- sqlalchemy==1.1.14
- statsmodels==0.8.0
- subprocess32==3.2.7
- tables==3.4.2
- toolz==0.8.2
- traitlets==4.3.2
- urllib3==1.22
- wcwidth==0.1.7
- enigma-catalyst>=0.3