catalyst/docs/source/example-algos.rst

|
Example Algorithms
==================

This section documents a number of example algorithms to complement the 
beginner tutorial, and show how other trading algorithms can be implemented 
using Catalyst.

Overview
~~~~~~~~

- :ref:`Buy BTC Simple<buy_btc_simple>`: The simplest algorithm that introduces
  the ``initialize()`` and ``handle_data()`` functions, and is used in the 
  :doc:`beginner tutorial<beginner-tutorial>` to show how to run catalyst 
  for the first time.

- :ref:`Buy and Hodl <buy_and_hodl>`: A very straightforward *buy and hold* that 
  makes one single buy at the very beginning. Introduces the notions of 
  ``cash``, management of outstanding ``orders``, and ``order_target_value`` 
  to place orders. It also introduces the ``analyze()`` function to visualize 
  the performance of our strategy using the external library ``matplotlib``.

- :ref:`Dual Moving Average Crossover<dual_moving_average>`: A classic momentum 
  strategy used in the second part of the 
  `beginner tutorial <beginner-tutorial.html#history>`_ to introduce the 
  ``data.history()`` function. It makes a heavy use of ``matplotlib`` library 
  in the ``analyze()`` function to chart the performance of the algorithm.

- :ref:`Mean Reversion Algorithm <mean_reversion>`: Another simple momentum 
  strategy that is used in our 
  `two-part video tutorial <videos.html#backtesting-a-strategy>`_ to show how 
  to get started in backtesting and live trading with Catalyst.


.. _buy_btc_simple:

Buy BTC Simple Algorithm
~~~~~~~~~~~~~~~~~~~~~~~~

Source code: `examples/buy_btc_simple.py <https://github.com/enigmampc/catalyst/blob/master/catalyst/examples/buy_btc_simple.py>`_

.. code-block:: python

  '''
    Run this example, by executing the following from your terminal:
      catalyst ingest-exchange -x bitfinex -f daily -i btc_usdt
      catalyst run -f buy_btc_simple.py -x bitfinex --start 2016-1-1 --end 2017-9-30 -o buy_btc_simple_out.pickle

    If you want to run this code using another exchange, make sure that 
    the asset is available on that exchange. For example, if you were to run 
    it for exchange Poloniex, you would need to edit the following line:

      context.asset = symbol('btc_usdt')     # note 'usdt' instead of 'usd'

    and specify exchange poloniex as follows:
    catalyst ingest-exchange -x poloniex -f daily -i btc_usdt
    catalyst run -f buy_btc_simple.py -x poloniex --start 2016-1-1 --end 2017-9-30 -o buy_btc_simple_out.pickle

    To see which assets are available on each exchange, visit:
    https://www.enigma.co/catalyst/status
  '''

  from catalyst.api import order, record, symbol

  def initialize(context):
      context.asset = symbol('btc_usd')

  def handle_data(context, data):
      order(context.asset, 1)
      record(btc = data.current(context.asset, 'price'))

This simple algorithm does not produce any output nor displays any chart.


.. _buy_and_hodl:

Buy and Hodl Algorithm
~~~~~~~~~~~~~~~~~~~~~~

Source code: `examples/buy_and_hodl.py <https://github.com/enigmampc/catalyst/blob/master/catalyst/examples/buy_and_hodl.py>`_

First ingest the historical pricing data needed to run this algorithm:

.. code-block:: bash

   catalyst ingest-exchange -x bitfinex -f daily -i btc_usd

Then, you can run the code below with the following command:

.. code-block:: bash

   catalyst run -f buy_and_hodl.py --start 2015-3-1 --end 2017-10-31 --capital-base 100000 -x bitfinex -c btc -o bah.pickle

or using the same parameters specified in the run_algorithm() function at the 
end of the file:

.. code-block:: bash

  python buy_and_hodl.py


This command will run the trading algorithm in the specified time range and 
plot the resulting performance using the matplotlib library. You can choose any 
date interval with the ``--start`` and ``--end`` parameters, but bear in mind 
that 2015-3-1 is the earliest date that Catalyst supports (if you choose an 
earlier date, you'll get an error), and the most recent date you can choose is 
one day prior to the current date. 


.. code-block:: python

  #!/usr/bin/env python
  #
  # Copyright 2017 Enigma MPC, Inc.
  # Copyright 2015 Quantopian, Inc.
  #
  # Licensed under the Apache License, Version 2.0 (the "License");
  # you may not use this file except in compliance with the License.
  # You may obtain a copy of the License at
  #
  #     http://www.apache.org/licenses/LICENSE-2.0
  #
  # Unless required by applicable law or agreed to in writing, software
  # distributed under the License is distributed on an "AS IS" BASIS,
  # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  # See the License for the specific language governing permissions and
  # limitations under the License.
  import pandas as pd
  import matplotlib.pyplot as plt

  from catalyst import run_algorithm
  from catalyst.api import (order_target_value, symbol, record,
      cancel_order, get_open_orders, )


  def initialize(context):
      context.ASSET_NAME = 'btc_usd'
      context.TARGET_HODL_RATIO = 0.8
      context.RESERVE_RATIO = 1.0 - context.TARGET_HODL_RATIO

      context.is_buying = True
      context.asset = symbol(context.ASSET_NAME)

      context.i = 0


  def handle_data(context, data):
      context.i += 1

      starting_cash = context.portfolio.starting_cash
      target_hodl_value = context.TARGET_HODL_RATIO * starting_cash
      reserve_value = context.RESERVE_RATIO * starting_cash

      # Cancel any outstanding orders
      orders = get_open_orders(context.asset) or []
      for order in orders:
          cancel_order(order)

      # Stop buying after passing the reserve threshold
      cash = context.portfolio.cash
      if cash <= reserve_value:
          context.is_buying = False

      # Retrieve current asset price from pricing data
      price = data.current(context.asset, 'price')

      # Check if still buying and could (approximately) afford another purchase
      if context.is_buying and cash > price:
          print('buying')
          # Place order to make position in asset equal to target_hodl_value
          order_target_value(
              context.asset,
              target_hodl_value,
              limit_price=price * 1.1,
              stop_price=price * 0.9,
          )

      record(
          price=price,
          volume=data.current(context.asset, 'volume'),
          cash=cash,
          starting_cash=context.portfolio.starting_cash,
          leverage=context.account.leverage,
      )


  def analyze(context=None, results=None):

      # Plot the portfolio and asset data.
      ax1 = plt.subplot(611)
      results[['portfolio_value']].plot(ax=ax1)
      ax1.set_ylabel('Portfolio Value (USD)')

      ax2 = plt.subplot(612, sharex=ax1)
      ax2.set_ylabel('{asset} (USD)'.format(asset=context.ASSET_NAME))
      results[['price']].plot(ax=ax2)

      trans = results.ix[[t != [] for t in results.transactions]]
      buys = trans.ix[
          [t[0]['amount'] > 0 for t in trans.transactions]
      ]
      ax2.scatter(
          buys.index.to_pydatetime(),
          results.price[buys.index],
          marker='^',
          s=100,
          c='g',
          label=''
      )

      ax3 = plt.subplot(613, sharex=ax1)
      results[['leverage', 'alpha', 'beta']].plot(ax=ax3)
      ax3.set_ylabel('Leverage ')

      ax4 = plt.subplot(614, sharex=ax1)
      results[['starting_cash', 'cash']].plot(ax=ax4)
      ax4.set_ylabel('Cash (USD)')

      results[[
          'treasury',
          'algorithm',
          'benchmark',
      ]] = results[[
          'treasury_period_return',
          'algorithm_period_return',
          'benchmark_period_return',
      ]]

      ax5 = plt.subplot(615, sharex=ax1)
      results[[
          'treasury',
          'algorithm',
          'benchmark',
      ]].plot(ax=ax5)
      ax5.set_ylabel('Percent Change')

      ax6 = plt.subplot(616, sharex=ax1)
      results[['volume']].plot(ax=ax6)
      ax6.set_ylabel('Volume (mCoins/5min)')

      plt.legend(loc=3)

      # Show the plot.
      plt.gcf().set_size_inches(18, 8)
      plt.show()


  if __name__ == '__main__':
      run_algorithm(
          capital_base=10000,
          data_frequency='daily',
          initialize=initialize,
          handle_data=handle_data,
          analyze=analyze,
          exchange_name='bitfinex',
          algo_namespace='buy_and_hodl',
          base_currency='usd',
          start=pd.to_datetime('2015-03-01', utc=True),
          end=pd.to_datetime('2017-10-31', utc=True),
      )

.. image:: https://s3.amazonaws.com/enigmaco-docs/github.io/example_buy_and_hodl.png

.. _dual_moving_average:

Dual Moving Average Crossover
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Source Code: `examples/dual_moving_average.py <https://github.com/enigmampc/catalyst/blob/master/catalyst/examples/dual_moving_average.py>`_

This strategy is covered in detail in the last part of 
`this tutorial <beginner-tutorial.html#history>`_.

.. code-block:: python

  import numpy as np
  import pandas as pd
  from logbook import Logger
  import matplotlib.pyplot as plt

  from catalyst import run_algorithm
  from catalyst.api import (order, record, symbol, order_target_percent,
          get_open_orders)
  from catalyst.exchange.stats_utils import extract_transactions

  NAMESPACE = 'dual_moving_average'
  log = Logger(NAMESPACE)

  def initialize(context):
      context.i = 0
      context.asset = symbol('ltc_usd')
      context.base_price = None


  def handle_data(context, data):
      # define the windows for the moving averages
      short_window = 50
      long_window = 200

      # Skip as many bars as long_window to properly compute the average
      context.i += 1
      if context.i < long_window:
         return

      # Compute moving averages calling data.history() for each
      # moving average with the appropriate parameters. We choose to use
      # minute bars for this simulation -> freq="1m"
      # Returns a pandas dataframe.
      short_mavg = data.history(context.asset, 'price',
                          bar_count=short_window, frequency="1m").mean()
      long_mavg = data.history(context.asset, 'price',
                          bar_count=long_window, frequency="1m").mean()

      # Let's keep the price of our asset in a more handy variable
      price = data.current(context.asset, 'price')

      # If base_price is not set, we use the current value. This is the
      # price at the first bar which we reference to calculate price_change.
      if context.base_price is None:
          context.base_price = price
      price_change = (price - context.base_price) / context.base_price

      # Save values for later inspection
      record(price=price,
             cash=context.portfolio.cash,
             price_change=price_change,
             short_mavg=short_mavg,
             long_mavg=long_mavg)

      # Since we are using limit orders, some orders may not execute immediately
      # we wait until all orders are executed before considering more trades.
      orders = get_open_orders(context.asset)
      if len(orders) > 0:
          return

      # Exit if we cannot trade
      if not data.can_trade(context.asset):
          return

      # We check what's our position on our portfolio and trade accordingly
      pos_amount = context.portfolio.positions[context.asset].amount

      # Trading logic
      if short_mavg > long_mavg and pos_amount == 0:
         # we buy 100% of our portfolio for this asset
         order_target_percent(context.asset, 1)
      elif short_mavg < long_mavg and pos_amount > 0:
         # we sell all our positions for this asset
         order_target_percent(context.asset, 0)


  def analyze(context, perf):

      # Get the base_currency that was passed as a parameter to the simulation
      base_currency = context.exchanges.values()[0].base_currency.upper()

      # First chart: Plot portfolio value using base_currency
      ax1 = plt.subplot(411)
      perf.loc[:, ['portfolio_value']].plot(ax=ax1)
      ax1.legend_.remove()
      ax1.set_ylabel('Portfolio Value\n({})'.format(base_currency))
      start, end = ax1.get_ylim()
      ax1.yaxis.set_ticks(np.arange(start, end, (end-start)/5))

      # Second chart: Plot asset price, moving averages and buys/sells
      ax2 = plt.subplot(412, sharex=ax1)
      perf.loc[:, ['price','short_mavg','long_mavg']].plot(ax=ax2, label='Price')
      ax2.legend_.remove()
      ax2.set_ylabel('{asset}\n({base})'.format(
          asset = context.asset.symbol,
          base = base_currency
          ))
      start, end = ax2.get_ylim()
      ax2.yaxis.set_ticks(np.arange(start, end, (end-start)/5))

      transaction_df = extract_transactions(perf)
      if not transaction_df.empty:
          buy_df = transaction_df[transaction_df['amount'] > 0]
          sell_df = transaction_df[transaction_df['amount'] < 0]
          ax2.scatter(
              buy_df.index.to_pydatetime(),
              perf.loc[buy_df.index, 'price'],
              marker='^',
              s=100,
              c='green',
              label=''
          )
          ax2.scatter(
              sell_df.index.to_pydatetime(),
              perf.loc[sell_df.index, 'price'],
              marker='v',
              s=100,
              c='red',
              label=''
          )

      # Third chart: Compare percentage change between our portfolio
      # and the price of the asset
      ax3 = plt.subplot(413, sharex=ax1)
      perf.loc[:, ['algorithm_period_return', 'price_change']].plot(ax=ax3)
      ax3.legend_.remove()
      ax3.set_ylabel('Percent Change')
      start, end = ax3.get_ylim()
      ax3.yaxis.set_ticks(np.arange(start, end, (end-start)/5))

      # Fourth chart: Plot our cash
      ax4 = plt.subplot(414, sharex=ax1)
      perf.cash.plot(ax=ax4)
      ax4.set_ylabel('Cash\n({})'.format(base_currency))
      start, end = ax4.get_ylim()
      ax4.yaxis.set_ticks(np.arange(0, end, end/5))

      plt.show()


  if __name__ == '__main__':
      run_algorithm(
              capital_base=1000,
              data_frequency='minute',
              initialize=initialize,
              handle_data=handle_data,
              analyze=analyze,
              exchange_name='bitfinex',
              algo_namespace=NAMESPACE,
              base_currency='usd',
              start=pd.to_datetime('2017-9-22', utc=True),
              end=pd.to_datetime('2017-9-23', utc=True),
          )

.. image:: https://s3.amazonaws.com/enigmaco-docs/github.io/tutorial_dual_moving_average.png


.. _mean_reversion:

Mean Reversion Algorithm
~~~~~~~~~~~~~~~~~~~~~~~~

Source code: `examples/mean_reversion_simple.py <https://github.com/enigmampc/catalyst/blob/master/catalyst/examples/mean_reversion_simple.py>`_

This algorithm is based on a simple momentum strategy. When the cryptoasset goes
up quickly, we're going to buy; when it goes down quickly, we're going to sell. 
Hopefully, we'll ride the waves.

We are choosing to backtest this trading algorithm with the ``neo_usd`` currency 
pairon the ``Bitfinex`` exchange. Thus, first ingest the historical pricing data
that we need, with minute resolution:

.. code-block:: bash

   catalyst ingest-exchange -x bitfinex -f minute -i neo_usd

To run this algorithm, we are opting for the Python interpreter, instead of the 
command line (CLI). All of the parameters for the simulation are specified in 
lines 218-245, so in order to run the algorithm we just type:

.. code-block:: bash

   python mean_reversion_simple.py

.. code-block:: python

  import os
  import tempfile
  import time

  import numpy as np
  import pandas as pd
  import talib
  from logbook import Logger

  from catalyst import run_algorithm
  from catalyst.api import symbol, record, order_target_percent, get_open_orders
  from catalyst.exchange.stats_utils import extract_transactions
  # We give a name to the algorithm which Catalyst will use to persist its state.
  # In this example, Catalyst will create the `.catalyst/data/live_algos`
  # directory. If we stop and start the algorithm, Catalyst will resume its
  # state using the files included in the folder.
  from catalyst.utils.paths import ensure_directory

  NAMESPACE = 'mean_reversion_simple'
  log = Logger(NAMESPACE)


  # To run an algorithm in Catalyst, you need two functions: initialize and
  # handle_data.

  def initialize(context):
      # This initialize function sets any data or variables that you'll use in
      # your algorithm.  For instance, you'll want to define the trading pair (or
      # trading pairs) you want to backtest.  You'll also want to define any
      # parameters or values you're going to use.

      # In our example, we're looking at Neo in USD.
      context.neo_eth = symbol('neo_usd')
      context.base_price = None
      context.current_day = None

      context.RSI_OVERSOLD = 30
      context.RSI_OVERBOUGHT = 80
      context.CANDLE_SIZE = '15T'

      context.start_time = time.time()


  def handle_data(context, data):
      # This handle_data function is where the real work is done.  Our data is
      # minute-level tick data, and each minute is called a frame.  This function
      # runs on each frame of the data.

      # We flag the first period of each day.
      # Since cryptocurrencies trade 24/7 the `before_trading_starts` handle
      # would only execute once. This method works with minute and daily
      # frequencies.
      today = data.current_dt.floor('1D')
      if today != context.current_day:
          context.traded_today = False
          context.current_day = today

      # We're computing the volume-weighted-average-price of the security
      # defined above, in the context.neo_eth variable.  For this example, we're 
      # using three bars on the 15 min bars.

      # The frequency attribute determine the bar size. We use this convention
      # for the frequency alias:
      # http://pandas.pydata.org/pandas-docs/stable/timeseries.html#offset-aliases
      prices = data.history(
          context.neo_eth,
          fields='close',
          bar_count=50,
          frequency=context.CANDLE_SIZE
      )

      # Ta-lib calculates various technical indicator based on price and
      # volume arrays.

      # In this example, we are comp
      rsi = talib.RSI(prices.values, timeperiod=14)

      # We need a variable for the current price of the security to compare to
      # the average. Since we are requesting two fields, data.current()
      # returns a DataFrame with
      current = data.current(context.neo_eth, fields=['close', 'volume'])
      price = current['close']

      # If base_price is not set, we use the current value. This is the
      # price at the first bar which we reference to calculate price_change.
      if context.base_price is None:
          context.base_price = price

      price_change = (price - context.base_price) / context.base_price
      cash = context.portfolio.cash

      # Now that we've collected all current data for this frame, we use
      # the record() method to save it. This data will be available as
      # a parameter of the analyze() function for further analysis.
      record(
          price=price,
          volume=current['volume'],
          price_change=price_change,
          rsi=rsi[-1],
          cash=cash
      )

      # We are trying to avoid over-trading by limiting our trades to
      # one per day.
      if context.traded_today:
          return

      # Since we are using limit orders, some orders may not execute immediately
      # we wait until all orders are executed before considering more trades.
      orders = get_open_orders(context.neo_eth)
      if len(orders) > 0:
          return

      # Exit if we cannot trade
      if not data.can_trade(context.neo_eth):
          return

      # Another powerful built-in feature of the Catalyst backtester is the
      # portfolio object.  The portfolio object tracks your positions, cash,
      # cost basis of specific holdings, and more.  In this line, we calculate
      # how long or short our position is at this minute.   
      pos_amount = context.portfolio.positions[context.neo_eth].amount

      if rsi[-1] <= context.RSI_OVERSOLD and pos_amount == 0:
          log.info(
              '{}: buying - price: {}, rsi: {}'.format(
                  data.current_dt, price, rsi[-1]
              )
          )
          # Set a style for limit orders,
          limit_price = price * 1.005
          order_target_percent(
              context.neo_eth, 1, limit_price=limit_price
          )
          context.traded_today = True

      elif rsi[-1] >= context.RSI_OVERBOUGHT and pos_amount > 0:
          log.info(
              '{}: selling - price: {}, rsi: {}'.format(
                  data.current_dt, price, rsi[-1]
              )
          )
          limit_price = price * 0.995
          order_target_percent(
              context.neo_eth, 0, limit_price=limit_price
          )
          context.traded_today = True


  def analyze(context=None, perf=None):
      end = time.time()
      log.info('elapsed time: {}'.format(end - context.start_time))

      import matplotlib.pyplot as plt
      # The base currency of the algo exchange
      base_currency = context.exchanges.values()[0].base_currency.upper()

      # Plot the portfolio value over time.
      ax1 = plt.subplot(611)
      perf.loc[:, 'portfolio_value'].plot(ax=ax1)
      ax1.set_ylabel('Portfolio\nValue\n({})'.format(base_currency))

      # Plot the price increase or decrease over time.
      ax2 = plt.subplot(612, sharex=ax1)
      perf.loc[:, 'price'].plot(ax=ax2, label='Price')

      ax2.set_ylabel('{asset}\n({base})'.format(
          asset=context.neo_eth.symbol, base=base_currency
      ))

      transaction_df = extract_transactions(perf)
      if not transaction_df.empty:
          buy_df = transaction_df[transaction_df['amount'] > 0]
          sell_df = transaction_df[transaction_df['amount'] < 0]
          ax2.scatter(
              buy_df.index.to_pydatetime(),
              perf.loc[buy_df.index.floor('1 min'), 'price'],
              marker='^',
              s=100,
              c='green',
              label=''
          )
          ax2.scatter(
              sell_df.index.to_pydatetime(),
              perf.loc[sell_df.index.floor('1 min'), 'price'],
              marker='v',
              s=100,
              c='red',
              label=''
          )

      ax4 = plt.subplot(613, sharex=ax1)
      perf.loc[:, 'cash'].plot(
          ax=ax4, label='Base Currency ({})'.format(base_currency)
      )
      ax4.set_ylabel('Cash\n({})'.format(base_currency))

      perf['algorithm'] = perf.loc[:, 'algorithm_period_return']

      ax5 = plt.subplot(614, sharex=ax1)
      perf.loc[:, ['algorithm', 'price_change']].plot(ax=ax5)
      ax5.set_ylabel('Percent\nChange')

      ax6 = plt.subplot(615, sharex=ax1)
      perf.loc[:, 'rsi'].plot(ax=ax6, label='RSI')
      ax6.set_ylabel('RSI')
      ax6.axhline(context.RSI_OVERBOUGHT, color='darkgoldenrod')
      ax6.axhline(context.RSI_OVERSOLD, color='darkgoldenrod')

      if not transaction_df.empty:
          ax6.scatter(
              buy_df.index.to_pydatetime(),
              perf.loc[buy_df.index.floor('1 min'), 'rsi'],
              marker='^',
              s=100,
              c='green',
              label=''
          )
          ax6.scatter(
              sell_df.index.to_pydatetime(),
              perf.loc[sell_df.index.floor('1 min'), 'rsi'],
              marker='v',
              s=100,
              c='red',
              label=''
          )
      plt.legend(loc=3)
      start, end = ax6.get_ylim()
      ax6.yaxis.set_ticks(np.arange(0, end, end/5))

      # Show the plot.
      plt.gcf().set_size_inches(18, 8)
      plt.show()
      pass


  if __name__ == '__main__':
      # The execution mode: backtest or live
      MODE = 'backtest'

      if MODE == 'backtest':
          folder = os.path.join(
              tempfile.gettempdir(), 'catalyst', NAMESPACE
          )
          ensure_directory(folder)

          timestr = time.strftime('%Y%m%d-%H%M%S')
          out = os.path.join(folder, '{}.p'.format(timestr))
          # catalyst run -f catalyst/examples/mean_reversion_simple.py -x bitfinex -s 2017-10-1 -e 2017-11-10 -c usdt -n mean-reversion --data-frequency minute --capital-base 10000
          run_algorithm(
              capital_base=10000,
              data_frequency='minute',
              initialize=initialize,
              handle_data=handle_data,
              analyze=analyze,
              exchange_name='bitfinex',
              algo_namespace=NAMESPACE,
              base_currency='usd',
              start=pd.to_datetime('2017-10-01', utc=True),
              end=pd.to_datetime('2017-11-10', utc=True),
              output=out
          )
          log.info('saved perf stats: {}'.format(out))

      elif MODE == 'live':
          run_algorithm(
              capital_base=0.5,
              initialize=initialize,
              handle_data=handle_data,
              analyze=analyze,
              exchange_name='bittrex',
              live=True,
              algo_namespace=NAMESPACE,
              base_currency='usd',
              live_graph=False
          )

.. image:: https://s3.amazonaws.com/enigmaco-docs/github.io/example_mean_reversion_simple.png

Notice the difference in performance between the charts above and those seen on 
`this video tutorial <https://youtu.be/JOBRwst9jUY>`_ at 
minute 8:10. The buy and sell orders are triggered at the same exact times, but
the differences result from a more realistic slippage model 
implemented after the video was recorded, which executes the orders at slighlty
different prices, but resulting in significant changes in performance of our 
strategy.