merged from develop

Example: Simple Universe

.gitignore

@@ -78,3 +78,7 @@ zipline.iml
 ./data
 
 TAGS
+
+python2
+python3
+scratch

README.rst

@@ -1,196 +1,72 @@
-========
-Catalyst
-========
+.. image:: https://s3.amazonaws.com/enigmaco-docs/enigma-catalyst.jpg
+    :target: https://enigmampc.github.io/catalyst
+    :align: center
+    :alt: Enigma | Catalyst
 
+|version tag|
 |version status|
+|discord|
+|twitter|
+
+|
 
 Catalyst is an algorithmic trading library for crypto-assets written in Python.
-It allows trading strategies to be easily expressed and backtested against historical data, providing analytics and insights regarding a particular strategy's performance.
-Catalyst will be expanded to support live-trading of crypto-assets in the coming months.
-Please visit `<enigma.co>`_ to learn about Catalyst, or refer to the 
-`whitepaper <https://www.enigma.co/enigma_catalyst.pdf>`_ for further technical details.
+It allows trading strategies to be easily expressed and backtested against 
+historical data (with daily and minute resolution), providing analytics and 
+insights regarding a particular strategy's performance. Catalyst also supports
+live-trading of crypto-assets starting with three exchanges (Bitfinex, Bittrex, 
+and Poloniex) with more being added over time. Catalyst empowers users to share 
+and curate data and build profitable, data-driven investment strategies. Please 
+visit `enigma.co <https://www.enigma.co>`_ to learn more about Catalyst, or 
+refer to the `whitepaper <https://www.enigma.co/enigma_catalyst.pdf>`_ for 
+further technical details.
 
-Catalyst builds on top of the well-established `Zipline <https://github.com/quantopian/zipline>`_ project.
-We did our best to minimize structural changes to the general API to maximize compatibility with existing trading algorithms, developer knowledge, and tutorials.
-For now, please refer to the `Zipline API Docs <http://zipline.io>`_ as a general reference and bring any other questions you have to our #dev channel on `Slack <https://join.slack.com/enigmacatalyst/shared_invite/MTkzMjQ0MTg1NTczLTE0OTY3MjE3MDEtZGZmMTI5YzI3ZA>`_.
+Catalyst builds on top of the well-established 
+`Zipline <https://github.com/quantopian/zipline>`_ project. We did our best to 
+minimize structural changes to the general API to maximize compatibility with 
+existing trading algorithms, developer knowledge, and tutorials. Join us on 
+`Discord <https://discord.gg/SJK32GY>`_ where we have a *#catalyst_dev* channel 
+for questions around Catalyst, algorithmic trading and technical support.
 
-Our primary contributions include the:
+Overview
+========
 
-- Introduction of an open trading calendar that permits simulation to allow trades on weekends, holidays, and outside of normal business hours.
-- Curation of OHLCV data bundle from `Poloniex's API <https://poloniex.com/support/api/>`_, which contains data in five-minute intervals as early as 2/19/2015.
-- Support for backtesting of daily trading strategies, support for five-minute backtesting is in development.
-- Addition of Bitcoin price (USDT_BTC) as a benchmark asset for comparing performance.
+-  Ease of use: Catalyst tries to get out of your way so that you can 
+   focus on algorithm development. See 
+   `examples of trading strategies <https://github.com/enigmampc/catalyst/tree/master/catalyst/examples>`_ 
+   provided.
+-  Support for several of the top crypto-exchanges by trading volume:
+   `Bitfinex <https://www.bitfinex.com>`_, `Bittrex <http://www.bittrex.com>`_,
+   and `Poloniex <https://www.poloniex.com>`_. 
+-  Secure: You and only you have access to each exchange API keys for your accounts.
+-  Input of historical pricing data of all crypto-assets by exchange, 
+   with daily and minute resolution. See 
+   `Catalyst Market Coverage Overview <https://www.enigma.co/catalyst/status>`_.
+-  Backtesting and live-trading functionality, with a seamless transition
+   between the two modes.
+-  Output of performance statistics are based on Pandas DataFrames to 
+   integrate nicely into the existing PyData eco-system.
+-  Statistic and machine learning libraries like matplotlib, scipy, 
+   statsmodels, and sklearn support development, analysis, and 
+   visualization of state-of-the-art trading systems.
+-  Addition of Bitcoin price (btc_usdt) as a benchmark for comparing 
+   performance across trading algorithms.
 
-Interested in getting involved?
-`Join us on Slack! <https://join.slack.com/enigmacatalyst/shared_invite/MTkzMjQ0MTg1NTczLTE0OTY3MjE3MDEtZGZmMTI5YzI3ZA>`_
+Go to our `Documentation Website <https://enigmampc.github.io/catalyst/>`_.
 
 
-Installation
-============
-
-At the moment, Catalyst has some fairly specific and strict depedency requirements.
-We recommend the use of Python virtual environments if you wish to simplify the installation process, or otherwise isolate Catalyst's dependencies from your other projects.
-If you don't have ``virtualenv`` installed, see our later section on Virtual Environments.
-
-.. code-block:: bash
-
-    $ virtualenv catalyst-venv
-    $ source ./catalyst-venv/bin/activate
-    $ pip install enigma-catalyst
-
-**Note:** A successful installation will require several minutes in order to compile dependencies that expose C APIs.
-
-Dependencies
-------------
-
-Catalyst's depedencies can be found in the ``etc/requirements.txt`` file.
-If you need to install them outside of a typical ``pip install``, this is done using:
-
-.. code-block:: bash
-
-    $ pip install -r etc/requirements.txt
-
-Though not required by Catalyst directly, our example algorithms use matplotlib to visually display backtest results.
-If you wish to run any examples or use matplotlib during development, it can be installed using:
-
-.. code-block:: bash
-
-    $ pip install matplotlib
-
-**Note:** If you plan to use matplotlib and virtualenv on Mac OS X, see our later section for additional setup instructions.
-
-Getting Started
-===============
-
-The following code implements a simple buy and hold algorithm. The full source can be found in ``catalyst/examples/buy_and_hodl.py``.
-
-.. code:: python
-
-    import numpy as np
-    
-    from catalyst.api import (
-        order_target_value,
-        symbol,
-        record,
-        cancel_order,
-        get_open_orders,
-    )
-    
-    ASSET = 'USDT_BTC'
-    
-    TARGET_HODL_RATIO = 0.8
-    RESERVE_RATIO = 1.0 - TARGET_HODL_RATIO
-
-    def initialize(context):
-        context.is_buying = True
-        context.asset = symbol(ASSET)
-
-    def handle_data(context, data):
-        cash = context.portfolio.cash
-        target_hodl_value = TARGET_HODL_RATIO * context.portfolio.starting_cash
-        reserve_value = RESERVE_RATIO * context.portfolio.starting_cash
-        
-        # Cancel any outstanding orders from the previous day
-        orders = get_open_orders(context.asset) or []
-        for order in orders:
-            cancel_order(order)
-        
-        # Stop buying after passing reserve threshold
-        if cash <= reserve_value:
-            context.is_buying = False
-        
-        # Retrieve current price from pricing data
-        price = data[context.asset].price
-        
-        # Check if still buying and could (approximately) afford another purchase                    
-        if context.is_buying and cash > price:
-            # Place order to make position in asset equal to target_hodl_value
-            order_target_value(
-                context.asset,
-                target_hodl_value,
-                limit_price=1.1 * price,
-                stop_price=0.9 * price,
-            )
-        
-        # Record any state for later analysis
-        record(
-            price=price,
-            cash=context.portfolio.cash,
-            leverage=context.account.leverage,
-        )
 
 
-You can then run this algorithm using the Catalyst CLI. From the command
-line, run:
-
-.. code:: bash
-
-    $ catalyst ingest
-    $ catalyst run -f buy_and_hodl.py --start 2015-3-1 --end 2017-6-28 --capital-base 100000 -o bah.pickle
-
-This will download the crypto-asset price data from a poloniex bundle
-curated by Enigma in the specified time range and stream it through
-the algorithm and plot the resulting performance using matplotlib.
-
-You can find other examples in the ``catalyst/examples`` directory.
-
-Limitations
------------
-
-This project is currently in a pre-alpha state and has some limitations we'd like to address:
-
-- *Minimum Denomination:* The smallest tradable unit in Catalyst is equal to 1/1000th of a full coin. We plan to enable more granular increments, but have capped it at 1/1000th for the time being.
-- *Supported Assets:* Currently the poloniex bundle comes prepopulated with data for all 90 registered trading pairs. However, due to limitations in how portfolios are currently modeled, we recommend sticking to ``USDT_*`` trading pairs. USDT is an independent currency listed on Poloniex whose price is pegged to the US dollar. Currently, this list includes: ``USDT_BTC``, ``USDT_DASH``, ``USDT_ETC``, ``USDT_ETH``, ``USDT_LTC``, ``USDT_NXT``, ``USDT_REP``, ``USDT_STR``, ``USDT_XMR``, ``USDT_XRP``, and ``USDT_ZEC``. We plan to add support for basing your portfolio in arbitrary currencies and provide native support for modeling ForEx trades in the near future!
-
-Virtual Environments
-====================
-
-Here we will provide a brief tutorial for installing ``virtualenv`` and its basic usage.
-For more information regarding ``virtualenv``, please refer to this `virtualenv guide <http://python-guide-pt-br.readthedocs.io/en/latest/dev/virtualenvs/>`_.
-
-The ``virtualenv`` command can be installed using:
-
-.. code-block:: bash
-
-    $ pip install virtualenv
-
-To create a new virtual environment, choose a directory, e.g. ``/path/to/venv-dir``, where project-specific packages and files will be stored.  The environment is created by running:
-
-.. code-block:: bash
-
-    $ virtualenv /path/to/venv-dir
-
-To enter an environment, run the ``bin/activate`` script located in ``/path/to/venv-dir`` using:
-
-.. code-block:: bash
-
-    $ source /path/to/venv-dir/bin/activate
-
-Exiting an environment is accomplished using ``deactivate``, and removing it entirely is done by deleting ``/path/to/venv-dir``.
-
-OS X + virtualenv + matplotlib
--------------------------------------
-
-A note about using matplotlib in virtual enviroments on OS X: it may be necessary to run
-
-.. code-block:: python
-
-    echo "backend: TkAgg" > ~/.matplotlib/matplotlibrc
-
-in order to override the default ``macosx`` backend for your system, which may not be accessible from inside the virtual environment.
-This will allow Catalyst to open matplotlib charts from within a virtual environment, which is useful for displaying the performance of your backtests.  To learn more about matplotlib backends, please refer to the
-`matplotlib backend documentation <https://matplotlib.org/faq/usage_faq.html#what-is-a-backend>`_.
-
-Disclaimer
-==========
-
-Keep in mind that this project is still under active development, and is not recommended for production use in its current state.
-We are deeply committed to improving the overall user experience, reliability, and feature-set offered by Catalyst.
-If you have any suggestions, feedback, or general improvements regarding any of these topics, please let us know!
-
-Hello World,
-
-The Enigma Team
+.. |version tag| image:: https://img.shields.io/pypi/v/enigma-catalyst.svg
+   :target: https://pypi.python.org/pypi/enigma-catalyst
 
 .. |version status| image:: https://img.shields.io/pypi/pyversions/enigma-catalyst.svg
    :target: https://pypi.python.org/pypi/enigma-catalyst
+
+.. |discord| image:: https://img.shields.io/badge/discord-join%20chat-green.svg
+   :target: https://discordapp.com/invite/SJK32GY
+
+.. |twitter| image:: https://img.shields.io/twitter/follow/enigmampc.svg?style=social&label=Follow&style=flat-square
+   :target: https://twitter.com/enigmampc
+
+

catalyst/__init__.py

@@ -29,11 +29,14 @@ from ._version import get_versions
 from . algorithm import TradingAlgorithm
 from . import api
 
+from catalyst.utils.calendars.calendar_utils import global_calendar_dispatcher
+
+__version__ = get_versions()['version']
+del get_versions
 
 # PERF: Fire a warning if calendars were instantiated during catalyst import.
 # Having calendars doesn't break anything per-se, but it makes catalyst imports
 # noticeably slower, which becomes particularly noticeable in the Zipline CLI.
-from catalyst.utils.calendars.calendar_utils import global_calendar_dispatcher
 if global_calendar_dispatcher._calendars:
     import warnings
     warnings.warn(
@@ -44,10 +47,6 @@ if global_calendar_dispatcher._calendars:
 del global_calendar_dispatcher
 
 
-__version__ = get_versions()['version']
-del get_versions
-
-
 def load_ipython_extension(ipython):
     from .__main__ import catalyst_magic
     ipython.register_magic_function(catalyst_magic, 'line_cell', 'catalyst')
@@ -69,7 +68,6 @@ if os.name == 'nt':
     _()
     del _
 
-
 __all__ = [
     'TradingAlgorithm',
     'api',
@@ -80,7 +78,3 @@ __all__ = [
     'run_algorithm',
     'utils',
 ]
-
-from ._version import get_versions
-__version__ = get_versions()['version']
-del get_versions

catalyst/__main__.py

@@ -8,6 +8,8 @@ import pandas as pd
 from six import text_type
 
 from catalyst.data import bundles as bundles_module
+from catalyst.exchange.exchange_bundle import ExchangeBundle
+from catalyst.exchange.exchange_utils import delete_algo_folder
 from catalyst.utils.cli import Date, Timestamp
 from catalyst.utils.run_algo import _run, load_extensions
 
@@ -27,17 +29,19 @@ except NameError:
 @click.option(
     '--strict-extensions/--non-strict-extensions',
     is_flag=True,
-    help='If --strict-extensions is passed then catalyst will not run if it'
-    ' cannot load all of the specified extensions. If this is not passed or'
-    ' --non-strict-extensions is passed then the failure will be logged but'
-    ' execution will continue.',
+    help='If --strict-extensions is passed then catalyst will not run '
+         'if it cannot load all of the specified extensions. If this is '
+         'not passed or --non-strict-extensions is passed then the '
+         'failure will be logged but execution will continue.',
 )
 @click.option(
     '--default-extension/--no-default-extension',
     is_flag=True,
     default=True,
-    help="Don't load the default catalyst extension.py file in $ZIPLINE_HOME.",
+    help="Don't load the default catalyst extension.py file "
+         "in $CATALYST_HOME.",
 )
+@click.version_option()
 def main(extension, strict_extensions, default_extension):
     """Top level catalyst entry point.
     """
@@ -64,6 +68,7 @@ def extract_option_object(option):
     option_object : click.Option
         The option object that this decorator will create.
     """
+
     @option
     def opt():
         pass
@@ -95,7 +100,9 @@ def ipython_only(option):
         def _(*args, **kwargs):
             kwargs[argname] = None
             return f(*args, **kwargs)
+
         return _
+
     return d
 
 
@@ -117,13 +124,13 @@ def ipython_only(option):
     '--define',
     multiple=True,
     help="Define a name to be bound in the namespace before executing"
-    " the algotext. For example '-Dname=value'. The value may be any python"
-    " expression. These are evaluated in order so they may refer to previously"
-    " defined names.",
+         " the algotext. For example '-Dname=value'. The value may be"
+         " any python expression. These are evaluated in order so they"
+         " may refer to previously defined names.",
 )
 @click.option(
     '--data-frequency',
-    type=click.Choice({'daily', '5-minute', 'minute'}),
+    type=click.Choice({'daily', 'minute'}),
     default='daily',
     show_default=True,
     help='The data frequency of the simulation.',
@@ -131,7 +138,6 @@ def ipython_only(option):
 @click.option(
     '--capital-base',
     type=float,
-    default=10e6,
     show_default=True,
     help='The starting capital for the simulation.',
 )
@@ -149,7 +155,7 @@ def ipython_only(option):
     default=pd.Timestamp.utcnow(),
     show_default=False,
     help='The date to lookup data on or before.\n'
-    '[default: <current-time>]'
+         '[default: <current-time>]'
 )
 @click.option(
     '-s',
@@ -169,8 +175,8 @@ def ipython_only(option):
     default='-',
     metavar='FILENAME',
     show_default=True,
-    help="The location to write the perf data. If this is '-' the perf will"
-    " be written to stdout.",
+    help="The location to write the perf data. If this is '-' the perf"
+         " will be written to stdout.",
 )
 @click.option(
     '--print-algo/--no-print-algo',
@@ -184,6 +190,22 @@ def ipython_only(option):
     default=None,
     help='Should the algorithm methods be resolved in the local namespace.'
 ))
+@click.option(
+    '-x',
+    '--exchange-name',
+    help='The name of the targeted exchange.',
+)
+@click.option(
+    '-n',
+    '--algo-namespace',
+    help='A label assigned to the algorithm for data storage purposes.'
+)
+@click.option(
+    '-c',
+    '--base-currency',
+    help='The base currency used to calculate statistics '
+         '(e.g. usd, btc, eth).',
+)
 @click.pass_context
 def run(ctx,
         algofile,
@@ -197,21 +219,12 @@ def run(ctx,
         end,
         output,
         print_algo,
-        local_namespace):
+        local_namespace,
+        exchange_name,
+        algo_namespace,
+        base_currency):
     """Run a backtest for the given algorithm.
     """
-    # check that the start and end dates are passed correctly
-    if start is None and end is None:
-        # check both at the same time to avoid the case where a user
-        # does not pass either of these and then passes the first only
-        # to be told they need to pass the second argument also
-        ctx.fail(
-            "must specify dates with '-s' / '--start' and '-e' / '--end'",
-        )
-    if start is None:
-        ctx.fail("must specify a start date with '-s' / '--start'")
-    if end is None:
-        ctx.fail("must specify an end date with '-e' / '--end'")
 
     if (algotext is not None) == (algofile is not None):
         ctx.fail(
@@ -219,6 +232,33 @@ def run(ctx,
             " '-t' / '--algotext'",
         )
 
+    # check that the start and end dates are passed correctly
+    if start is None and end is None:
+        # check both at the same time to avoid the case where a user
+        # does not pass either of these and then passes the first only
+        # to be told they need to pass the second argument also
+        ctx.fail(
+            "must specify dates with '-s' / '--start' and '-e' / '--end'"
+            " in backtest mode",
+        )
+    if start is None:
+        ctx.fail("must specify a start date with '-s' / '--start'"
+                 " in backtest mode")
+    if end is None:
+        ctx.fail("must specify an end date with '-e' / '--end'"
+                 " in backtest mode")
+
+    if exchange_name is None:
+        ctx.fail("must specify an exchange name '-x'")
+
+    if base_currency is None:
+        ctx.fail("must specify a base currency with '-c' in backtest mode")
+
+    if capital_base is None:
+        ctx.fail("must specify a capital base with '--capital-base'")
+
+    click.echo('Running in backtesting mode.')
+
     perf = _run(
         initialize=None,
         handle_data=None,
@@ -238,6 +278,13 @@ def run(ctx,
         print_algo=print_algo,
         local_namespace=local_namespace,
         environ=os.environ,
+        live=False,
+        exchange=exchange_name,
+        algo_namespace=algo_namespace,
+        base_currency=base_currency,
+        live_graph=False,
+        simulate_orders=True,
+        stats_output=None,
     )
 
     if output == '-':
@@ -281,15 +328,309 @@ def catalyst_magic(line, cell=None):
             raise ValueError('main returned non-zero status code: %d' % e.code)
 
 
+@main.command()
+@click.option(
+    '-f',
+    '--algofile',
+    default=None,
+    type=click.File('r'),
+    help='The file that contains the algorithm to run.',
+)
+@click.option(
+    '--capital-base',
+    type=float,
+    show_default=True,
+    help='The amount of capital (in base_currency) allocated to trading.',
+)
+@click.option(
+    '-t',
+    '--algotext',
+    help='The algorithm script to run.',
+)
+@click.option(
+    '-D',
+    '--define',
+    multiple=True,
+    help="Define a name to be bound in the namespace before executing"
+         " the algotext. For example '-Dname=value'. The value may be"
+         " any python expression. These are evaluated in order so they"
+         " may refer to previously defined names.",
+)
+@click.option(
+    '-o',
+    '--output',
+    default='-',
+    metavar='FILENAME',
+    show_default=True,
+    help="The location to write the perf data. If this is '-' the perf will"
+         " be written to stdout.",
+)
+@click.option(
+    '--print-algo/--no-print-algo',
+    is_flag=True,
+    default=False,
+    help='Print the algorithm to stdout.',
+)
+@ipython_only(click.option(
+    '--local-namespace/--no-local-namespace',
+    is_flag=True,
+    default=None,
+    help='Should the algorithm methods be resolved in the local namespace.'
+))
+@click.option(
+    '-x',
+    '--exchange-name',
+    help='The name of the targeted exchange.',
+)
+@click.option(
+    '-n',
+    '--algo-namespace',
+    help='A label assigned to the algorithm for data storage purposes.'
+)
+@click.option(
+    '-c',
+    '--base-currency',
+    help='The base currency used to calculate statistics '
+         '(e.g. usd, btc, eth).',
+)
+@click.option(
+    '--live-graph/--no-live-graph',
+    is_flag=True,
+    default=False,
+    help='Display live graph.',
+)
+@click.option(
+    '--simulate-orders/--no-simulate-orders',
+    is_flag=True,
+    default=True,
+    help='Simulating orders enable the paper trading mode. No orders will be '
+         'sent to the exchange unless set to false.',
+)
+@click.pass_context
+def live(ctx,
+         algofile,
+         capital_base,
+         algotext,
+         define,
+         output,
+         print_algo,
+         local_namespace,
+         exchange_name,
+         algo_namespace,
+         base_currency,
+         live_graph,
+         simulate_orders):
+    """Trade live with the given algorithm.
+    """
+    if (algotext is not None) == (algofile is not None):
+        ctx.fail(
+            "must specify exactly one of '-f' / '--algofile' or"
+            " '-t' / '--algotext'",
+        )
+
+    if exchange_name is None:
+        ctx.fail("must specify an exchange name '-x'")
+
+    if algo_namespace is None:
+        ctx.fail("must specify an algorithm name '-n' in live execution mode")
+
+    if base_currency is None:
+        ctx.fail("must specify a base currency '-c' in live execution mode")
+
+    if capital_base is None:
+        ctx.fail("must specify a capital base with '--capital-base'")
+
+    if simulate_orders:
+        click.echo('Running in paper trading mode.')
+
+    else:
+        click.echo('Running in live trading mode.')
+
+    perf = _run(
+        initialize=None,
+        handle_data=None,
+        before_trading_start=None,
+        analyze=None,
+        algofile=algofile,
+        algotext=algotext,
+        defines=define,
+        data_frequency=None,
+        capital_base=capital_base,
+        data=None,
+        bundle=None,
+        bundle_timestamp=None,
+        start=None,
+        end=None,
+        output=output,
+        print_algo=print_algo,
+        local_namespace=local_namespace,
+        environ=os.environ,
+        live=True,
+        exchange=exchange_name,
+        algo_namespace=algo_namespace,
+        base_currency=base_currency,
+        live_graph=live_graph,
+        simulate_orders=simulate_orders,
+        stats_output=None,
+    )
+
+    if output == '-':
+        click.echo(str(perf))
+    elif output != os.devnull:  # make the catalyst magic not write any data
+        perf.to_pickle(output)
+
+    return perf
+
+
+@main.command(name='ingest-exchange')
+@click.option(
+    '-x',
+    '--exchange-name',
+    help='The name of the exchange bundle to ingest.',
+)
+@click.option(
+    '-f',
+    '--data-frequency',
+    type=click.Choice({'daily', 'minute', 'daily,minute', 'minute,daily'}),
+    default='daily',
+    show_default=True,
+    help='The data frequency of the desired OHLCV bars.',
+)
+@click.option(
+    '-s',
+    '--start',
+    default=None,
+    type=Date(tz='utc', as_timestamp=True),
+    help='The start date of the data range. (default: one year from end date)',
+)
+@click.option(
+    '-e',
+    '--end',
+    default=None,
+    type=Date(tz='utc', as_timestamp=True),
+    help='The end date of the data range. (default: today)',
+)
+@click.option(
+    '-i',
+    '--include-symbols',
+    default=None,
+    help='A list of symbols to ingest (optional comma separated list)',
+)
+@click.option(
+    '--exclude-symbols',
+    default=None,
+    help='A list of symbols to exclude from the ingestion '
+         '(optional comma separated list)',
+)
+@click.option(
+    '--csv',
+    default=None,
+    help='The path of a CSV file containing the data. If specified, start, '
+         'end, include-symbols and exclude-symbols will be ignored. Instead,'
+         'all data in the file will be ingested.',
+)
+@click.option(
+    '--show-progress/--no-show-progress',
+    default=True,
+    help='Print progress information to the terminal.'
+)
+@click.option(
+    '--verbose/--no-verbose`',
+    default=False,
+    help='Show a progress indicator for every currency pair.'
+)
+@click.option(
+    '--validate/--no-validate`',
+    default=False,
+    help='Report potential anomalies found in data bundles.'
+)
+@click.pass_context
+def ingest_exchange(ctx, exchange_name, data_frequency, start, end,
+                    include_symbols, exclude_symbols, csv, show_progress,
+                    verbose, validate):
+    """
+    Ingest data for the given exchange.
+    """
+
+    if exchange_name is None:
+        ctx.fail("must specify an exchange name '-x'")
+
+    exchange_bundle = ExchangeBundle(exchange_name)
+
+    click.echo('Ingesting exchange bundle {}...'.format(exchange_name))
+    exchange_bundle.ingest(
+        data_frequency=data_frequency,
+        include_symbols=include_symbols,
+        exclude_symbols=exclude_symbols,
+        start=start,
+        end=end,
+        show_progress=show_progress,
+        show_breakdown=verbose,
+        show_report=validate,
+        csv=csv
+    )
+
+
+@main.command(name='clean-algo')
+@click.option(
+    '-n',
+    '--algo-namespace',
+    help='The label of the algorithm to for which to clean the state.'
+)
+@click.pass_context
+def clean_algo(ctx, algo_namespace):
+    click.echo(
+        'Cleaning algo state: {}'.format(algo_namespace)
+    )
+    delete_algo_folder(algo_namespace)
+    click.echo('Done')
+
+
+@main.command(name='clean-exchange')
+@click.option(
+    '-x',
+    '--exchange-name',
+    help='The name of the exchange bundle to ingest.',
+)
+@click.option(
+    '-f',
+    '--data-frequency',
+    type=click.Choice({'daily', 'minute'}),
+    default=None,
+    help='The bundle data frequency to remove. If not specified, it will '
+         'remove both daily and minute bundles.',
+)
+@click.pass_context
+def clean_exchange(ctx, exchange_name, data_frequency):
+    """Clean up bundles from 'ingest-exchange'.
+    """
+
+    if exchange_name is None:
+        ctx.fail("must specify an exchange name '-x'")
+
+    exchange_bundle = ExchangeBundle(exchange_name)
+
+    click.echo('Cleaning exchange bundle {}...'.format(exchange_name))
+    exchange_bundle.clean(
+        data_frequency=data_frequency,
+    )
+    click.echo('Done')
+
+
 @main.command()
 @click.option(
     '-b',
     '--bundle',
-    default='poloniex',
     metavar='BUNDLE-NAME',
-    show_default=True,
+    default=None,
+    show_default=False,
     help='The data bundle to ingest.',
 )
+@click.option(
+    '-x',
+    '--exchange-name',
+    help='The name of the exchange bundle to ingest.',
+)
 @click.option(
     '-c',
     '--compile-locally',
@@ -308,9 +649,12 @@ def catalyst_magic(line, cell=None):
     default=True,
     help='Print progress information to the terminal.'
 )
-def ingest(bundle, compile_locally, assets_version, show_progress):
+@click.pass_context
+def ingest(ctx, bundle, exchange_name, compile_locally, assets_version,
+           show_progress):
     """Ingest the data for the given bundle.
     """
+
     bundles_module.ingest(
         bundle,
         os.environ,
@@ -330,19 +674,26 @@ def ingest(bundle, compile_locally, assets_version, show_progress):
     show_default=True,
     help='The data bundle to clean.',
 )
+@click.option(
+    '-x',
+    '--exchange_name',
+    metavar='EXCHANGE-NAME',
+    show_default=True,
+    help='The exchange bundle name to clean.',
+)
 @click.option(
     '-e',
     '--before',
     type=Timestamp(),
     help='Clear all data before TIMESTAMP.'
-    ' This may not be passed with -k / --keep-last',
+         ' This may not be passed with -k / --keep-last',
 )
 @click.option(
     '-a',
     '--after',
     type=Timestamp(),
     help='Clear all data after TIMESTAMP'
-    ' This may not be passed with -k / --keep-last',
+         ' This may not be passed with -k / --keep-last',
 )
 @click.option(
     '-k',
@@ -350,10 +701,10 @@ def ingest(bundle, compile_locally, assets_version, show_progress):
     type=int,
     metavar='N',
     help='Clear all but the last N downloads.'
-    ' This may not be passed with -e / --before or -a / --after',
+         ' This may not be passed with -e / --before or -a / --after',
 )
 def clean(bundle, before, after, keep_last):
-    """Clean up data downloaded with the ingest command.
+    """Clean up bundles from 'ingest'.
     """
     bundles_module.clean(
         bundle,

catalyst/algorithm.py

@@ -124,7 +124,7 @@ from catalyst.utils.events import (
 from catalyst.utils.factory import create_simulation_parameters
 from catalyst.utils.math_utils import (
     tolerant_equals,
-    round_if_near_integer,
+    round_nearest
 )
 from catalyst.utils.pandas_utils import clear_dataframe_indexer_caches
 from catalyst.utils.preprocess import preprocess
@@ -133,15 +133,13 @@ from catalyst.utils.security_list import SecurityList
 import catalyst.protocol
 from catalyst.sources.requests_csv import PandasRequestsCSV
 
-from catalyst.gens.sim_engine import (
-    MinuteSimulationClock,
-    FiveMinuteSimulationClock,
-)
+from catalyst.gens.sim_engine import MinuteSimulationClock
 from catalyst.sources.benchmark_source import BenchmarkSource
 from catalyst.catalyst_warnings import ZiplineDeprecationWarning
 
+from catalyst.constants import LOG_LEVEL
 
-log = logbook.Logger("ZiplineLog")
+log = logbook.Logger("CatalystLog", level=LOG_LEVEL)
 
 
 class TradingAlgorithm(object):
@@ -173,7 +171,7 @@ class TradingAlgorithm(object):
     algo_filename : str, optional
         The filename for the algoscript. This will be used in exception
         tracebacks. default: '<string>'.
-    data_frequency : {'daily', '5-minute', 'minute'}, optional
+    data_frequency : {'daily', 'minute'}, optional
         The duration of the bars.
     instant_fill : bool, optional
         Whether to fill orders immediately or on next bar. default: False
@@ -226,7 +224,7 @@ class TradingAlgorithm(object):
             script : str
                 Algoscript that contains initialize and
                 handle_data function definition.
-            data_frequency : {'daily', '5-minute', 'minute'}
+            data_frequency : {'daily', 'minute'}
                The duration of the bars.
             capital_base : float <default: 1.0e5>
                How much capital to start with.
@@ -434,8 +432,6 @@ class TradingAlgorithm(object):
         if get_loader is not None:
             if data_frequency == 'daily':
                 all_dates = self.trading_calendar.all_sessions
-            elif data_frequency == '5-minute':
-                all_dates = self.trading_calendar.all_five_minutes
             elif data_frequency == 'minute':
                 all_dates = self.trading_calendar.all_minutes
             else:
@@ -444,9 +440,6 @@ class TradingAlgorithm(object):
                     'data frequency: {}'.format(data_frequency)
                 )
 
-            print 'first_dates:', all_dates[:10]
-            print 'last_dates:', all_dates[:-10]
-
             self.engine = SimplePipelineEngine(
                 get_loader,
                 all_dates,
@@ -470,7 +463,7 @@ class TradingAlgorithm(object):
         self._in_before_trading_start = True
 
         with handle_non_market_minutes(data) if \
-                self.data_frequency in ('minute', '5-minute') else ExitStack():
+                self.data_frequency == 'minute' else ExitStack():
             self._before_trading_start(self, data)
 
         self._in_before_trading_start = False
@@ -526,11 +519,10 @@ class TradingAlgorithm(object):
         market_closes = trading_o_and_c['market_close']
         minutely_emission = False
 
-        if self.sim_params.data_frequency in set(('minute', '5-minute')):
+        if self.sim_params.data_frequency == 'minute':
             market_opens = trading_o_and_c['market_open']
 
-            minutely_emission = self.sim_params.emission_rate in \
-                set(('minute', '5-minute'))
+            minutely_emission = self.sim_params.emission_rate == 'minute'
         else:
             # in daily mode, we want to have one bar per session, timestamped
             # as the last minute of the session.
@@ -554,15 +546,6 @@ class TradingAlgorithm(object):
             'UTC',
         )
 
-        if self.sim_params.data_frequency == '5-minute':
-            return FiveMinuteSimulationClock(
-                self.sim_params.sessions,
-                execution_opens,
-                execution_closes,
-                before_trading_start_minutes,
-                minute_emission=minutely_emission,
-            )
-
         return MinuteSimulationClock(
             self.sim_params.sessions,
             execution_opens,
@@ -694,8 +677,6 @@ class TradingAlgorithm(object):
                     time_count = times.nunique()
                     if time_count == 1:
                         self.sim_params.data_frequency = 'daily'
-                    elif time_count == 288:
-                        self.sim_params.data_frequency = '5-minute'
                     else:
                         self.sim_params.data_frequency = 'minute'
 
@@ -717,8 +698,6 @@ class TradingAlgorithm(object):
 
                 if self.sim_params.data_frequency == 'daily':
                     equity_reader_arg = 'equity_daily_reader'
-                elif self.sim_params.data_frequency == '5-minute':
-                    equity_daily_reader = 'equity_5_minute_reader'
                 elif self.sim_params.data_frequency == 'minute':
                     equity_reader_arg = 'equity_minute_reader'
                 equity_reader = PanelBarReader(
@@ -962,9 +941,9 @@ class TradingAlgorithm(object):
                   The arena from the simulation parameters. This will normally
                   be ``'backtest'`` but some systems may use this distinguish
                   live trading from backtesting.
-              data_frequency : {'daily', '5-minute', 'minute'}
+              data_frequency : {'daily', 'minute'}
                   data_frequency tells the algorithm if it is running with
-                  daily, minute, or five-minute mode.
+                  daily or minute mode.
               start : datetime
                   The start date for the simulation.
               end : datetime
@@ -1138,19 +1117,12 @@ class TradingAlgorithm(object):
                           'date_rule. You should use keyword argument '
                           'time_rule= when calling schedule_function without '
                           'specifying a date_rule', stacklevel=3)
-        
-        freq = self.sim_params.data_frequency
 
         date_rule = date_rule or date_rules.every_day()
-        if freq is 'daily':
-            # ignore time rule in daily mode
-            time_rule = time_rules.every_minute()
-        else:
-            # use provided time rule or default to every minute or 5 minutes
-            # based on desired data frequency.
-            time_rule = time_rule or (time_rules.every_5_minutes()
-                                      if freq is '5-minute' else
-                                      time_rules.every_minute())
+        time_rule = ((time_rule or time_rules.every_minute())
+                     if self.sim_params.data_frequency == 'minute' else
+                     # If we are in daily mode the time_rule is ignored.
+                     time_rules.every_minute())
 
         # Check the type of the algorithm's schedule before pulling calendar
         # Note that the ExchangeTradingSchedule is currently the only
@@ -1491,7 +1463,7 @@ class TradingAlgorithm(object):
 
     def _calculate_order(self, asset, amount,
                          limit_price=None, stop_price=None, style=None):
-        amount = self.round_order(amount)
+        amount = self.round_order(amount, asset)
 
         # Raises a ZiplineError if invalid parameters are detected.
         self.validate_order_params(asset,
@@ -1508,16 +1480,12 @@ class TradingAlgorithm(object):
         return amount, style
 
     @staticmethod
-    def round_order(amount):
+    def round_order(amount, asset):
         """
-        Convert number of shares to an integer.
-
-        By default, truncates to the integer share count that's either within
-        .0001 of amount or closer to zero.
-
-        E.g. 3.9999 -> 4.0; 5.5 -> 5.0; -5.5 -> -5.0
+        Converts the number of shares to the smallest tradable lot size for
+        the asset being ordered.
         """
-        return int(round_if_near_integer(amount))
+        return round_nearest(amount, asset.min_trade_size)
 
     def validate_order_params(self,
                               asset,
@@ -1825,7 +1793,7 @@ class TradingAlgorithm(object):
 
     @data_frequency.setter
     def data_frequency(self, value):
-        assert value in ('daily', '5-minute', 'minute')
+        assert value in ('daily', 'minute')
         self.sim_params.data_frequency = value
 
     @api_method

catalyst/assets/_assets.pyx

@@ -17,34 +17,37 @@
 """
 Cythonized Asset object.
 """
+import hashlib
+
 cimport cython
 from cpython.number cimport PyNumber_Index
 from cpython.object cimport (
-    Py_EQ,
-    Py_NE,
-    Py_GE,
-    Py_LE,
-    Py_GT,
-    Py_LT,
+Py_EQ,
+Py_NE,
+Py_GE,
+Py_LE,
+Py_GT,
+Py_LT,
 )
 from cpython cimport bool
 
+import pandas as pd
+from datetime import timedelta
 import numpy as np
 from numpy cimport int64_t
 import warnings
 cimport numpy as np
 
+from catalyst.exchange.exchange_utils import get_sid
 from catalyst.utils.calendars import get_calendar
-
+from catalyst.exchange.exchange_errors import InvalidSymbolError, SidHashError
 
 # IMPORTANT NOTE: You must change this template if you change
 # Asset.__reduce__, or else we'll attempt to unpickle an old version of this
 # class
 CACHE_FILE_TEMPLATE = '/tmp/.%s-%s.v7.cache'
 
-
 cdef class Asset:
-
     cdef readonly int sid
     # Cached hash of self.sid
     cdef int sid_hash
@@ -59,6 +62,7 @@ cdef class Asset:
 
     cdef readonly object exchange
     cdef readonly object exchange_full
+    cdef readonly object min_trade_size
 
     _kwargnames = frozenset({
         'sid',
@@ -70,18 +74,20 @@ cdef class Asset:
         'auto_close_date',
         'exchange',
         'exchange_full',
+        'min_trade_size',
     })
 
     def __init__(self,
-                 int sid, # sid is required
-                 object exchange, # exchange is required
+                 int sid,  # sid is required
+                 object exchange,  # exchange is required
                  object symbol="",
                  object asset_name="",
                  object start_date=None,
                  object end_date=None,
                  object first_traded=None,
                  object auto_close_date=None,
-                 object exchange_full=None):
+                 object exchange_full=None,
+                 object min_trade_size=None):
 
         self.sid = sid
         self.sid_hash = hash(sid)
@@ -94,6 +100,7 @@ cdef class Asset:
         self.end_date = end_date
         self.first_traded = first_traded
         self.auto_close_date = auto_close_date
+        self.min_trade_size = min_trade_size
 
     def __int__(self):
         return self.sid
@@ -148,7 +155,8 @@ cdef class Asset:
 
     def __repr__(self):
         attrs = ('symbol', 'asset_name', 'exchange',
-                 'start_date', 'end_date', 'first_traded', 'auto_close_date')
+                 'start_date', 'end_date', 'first_traded', 'auto_close_date',
+                 'min_trade_size')
         tuples = ((attr, repr(getattr(self, attr, None)))
                   for attr in attrs)
         strings = ('%s=%s' % (t[0], t[1]) for t in tuples)
@@ -170,7 +178,8 @@ cdef class Asset:
                                  self.end_date,
                                  self.first_traded,
                                  self.auto_close_date,
-                                 self.exchange_full))
+                                 self.exchange_full,
+                                 self.min_trade_size))
 
     cpdef to_dict(self):
         """
@@ -186,6 +195,7 @@ cdef class Asset:
             'auto_close_date': self.auto_close_date,
             'exchange': self.exchange,
             'exchange_full': self.exchange_full,
+            'min_trade_size': self.min_trade_size
         }
 
     @classmethod
@@ -230,13 +240,11 @@ cdef class Asset:
         calendar = get_calendar(self.exchange)
         return calendar.is_open_on_minute(dt_minute)
 
-
 cdef class Equity(Asset):
-
     def __repr__(self):
         attrs = ('symbol', 'asset_name', 'exchange',
                  'start_date', 'end_date', 'first_traded', 'auto_close_date',
-                 'exchange_full')
+                 'exchange_full', 'min_trade_size')
         tuples = ((attr, repr(getattr(self, attr, None)))
                   for attr in attrs)
         strings = ('%s=%s' % (t[0], t[1]) for t in tuples)
@@ -250,8 +258,8 @@ cdef class Equity(Asset):
         """
         def __get__(self):
             warnings.warn("The security_start_date property will soon be "
-            "retired. Please use the start_date property instead.",
-            DeprecationWarning)
+                          "retired. Please use the start_date property instead.",
+                          DeprecationWarning)
             return self.start_date
 
     property security_end_date:
@@ -261,8 +269,8 @@ cdef class Equity(Asset):
         """
         def __get__(self):
             warnings.warn("The security_end_date property will soon be "
-            "retired. Please use the end_date property instead.",
-            DeprecationWarning)
+                          "retired. Please use the end_date property instead.",
+                          DeprecationWarning)
             return self.end_date
 
     property security_name:
@@ -272,13 +280,11 @@ cdef class Equity(Asset):
         """
         def __get__(self):
             warnings.warn("The security_name property will soon be "
-            "retired. Please use the asset_name property instead.",
-            DeprecationWarning)
+                          "retired. Please use the asset_name property instead.",
+                          DeprecationWarning)
             return self.asset_name
 
-
 cdef class Future(Asset):
-
     cdef readonly object root_symbol
     cdef readonly object notice_date
     cdef readonly object expiration_date
@@ -303,8 +309,8 @@ cdef class Future(Asset):
     })
 
     def __init__(self,
-                 int sid, # sid is required
-                 object exchange, # exchange is required
+                 int sid,  # sid is required
+                 object exchange,  # exchange is required
                  object symbol="",
                  object root_symbol="",
                  object asset_name="",
@@ -388,6 +394,258 @@ cdef class Future(Asset):
         super_dict['multiplier'] = self.multiplier
         return super_dict
 
+cdef class TradingPair(Asset):
+    cdef readonly float leverage
+    cdef readonly object quote_currency
+    cdef readonly object base_currency
+    cdef readonly object end_daily
+    cdef readonly object end_minute
+    cdef readonly object exchange_symbol
+    cdef readonly float maker
+    cdef readonly float taker
+    cdef readonly int trading_state
+    cdef readonly object data_source
+    cdef readonly float max_trade_size
+    cdef readonly float lot
+    cdef readonly int decimals
+
+    _kwargnames = frozenset({
+        'sid',
+        'symbol',
+        'asset_name',
+        'start_date',
+        'end_date',
+        'first_traded',
+        'auto_close_date',
+        'exchange',
+        'exchange_full',
+        'leverage',
+        'quote_currency',
+        'base_currency',
+        'end_daily',
+        'end_minute',
+        'exchange_symbol',
+        'min_trade_size',
+        'max_trade_size',
+        'lot',
+        'maker',
+        'taker',
+        'trading_state',
+        'data_source',
+        'decimals'
+    })
+    def __init__(self,
+                 object symbol,
+                 object exchange,
+                 object start_date=None,
+                 object asset_name=None,
+                 int sid=0,
+                 float leverage=1.0,
+                 object end_daily=None,
+                 object end_minute=None,
+                 object end_date=None,
+                 object exchange_symbol=None,
+                 object first_traded=None,
+                 object auto_close_date=None,
+                 object exchange_full=None,
+                 float min_trade_size=0.0001,
+                 float max_trade_size=1000000,
+                 float maker=0.0015,
+                 float taker=0.0025,
+                 float lot=0,
+                 int decimals = 8,
+                 int trading_state=0,
+                 object data_source='catalyst'):
+        """
+        Replicates the Asset constructor with some built-in conventions
+        and adds properties for leverage and fees.
+
+        Symbol
+        ------
+        Catalyst defines its own set of "universal" symbols to reference
+        trading pairs across exchanges. This is required because exchanges
+        are not adhering to a universal symbolism. For example, Bitfinex
+        uses the BTC symbol for Bitcon while Kraken uses XBT. In addition,
+        pairs are sometimes presented differently. For example, Bitfinex
+        puts the market currency before the base currency without a
+        separator, Bittrex puts the base currency first and uses a dash
+        seperator.
+
+        Here is the Catalyst convention: [Market Currency]_[Base Currency]
+        For example: btc_usd, eth_btc, neo_eth, ltc_eur.
+
+        The symbol for each currency (e.g. btc, eth, ltc) is generally
+        aligned with the Bittrex exchange.
+
+        Sid
+        ---
+        The sid of each asset is calculated based on a numeric hash of the
+        universal symbol. This simple approach avoids maintaining a mapping
+        of sids.
+
+        Leverage
+        --------
+        In contrast with equities, crypto exchanges generally assign
+        leverage values to specific trading pairs. Pairs with the
+        highest volume and market cap generally benefit from high leverage.
+        New currencies from ICO generally cannot be leveraged.
+
+        Leverage allows you to open a larger position with a smaller amount
+        of funds. For example, if you open a $5,000 position in BTC/USD
+        with 5:1 leverage, only one-fifth of this amount, or $1000, will be
+        tied to the position from your balance. Your remaining balance will
+        be available for opening more positions. If you open this same
+        position with 2:1 leverage, $2,500 of your balance will be tied to
+        the position. If you open with 1:1 leverage, $5,000 of your balance
+        will be tied to the position.
+
+        Fees
+        ----
+        Exchanges generally charge a taker (taking from the order book) or
+        maker (adding to the order book) fee.
+
+        :param symbol:
+        :param exchange:
+        :param start_date:
+        :param asset_name:
+        :param sid:
+        :param leverage:
+        :param end_daily
+        :param end_minute
+        :param end_date:
+        :param exchange_symbol:
+        :param first_traded:
+        :param auto_close_date:
+        :param exchange_full:
+        :param min_trade_size:
+        :param max_trade_size:
+        :param maker:
+        :param taker:
+        :param data_source
+        :param decimals
+        :param lot
+        """
+
+        symbol = symbol.lower()
+        try:
+            self.base_currency, self.quote_currency = symbol.split('_')
+        except Exception as e:
+            raise InvalidSymbolError(symbol=symbol, error=e)
+
+        if sid == 0 or sid is None:
+            try:
+                sid = get_sid(symbol)
+            except Exception as e:
+                raise SidHashError(symbol=symbol)
+
+        if asset_name is None:
+            asset_name = ' / '.join(symbol.split('_')).upper()
+
+        if start_date is None:
+            start_date = pd.to_datetime('2009-1-1', utc=True)
+
+        if end_date is None:
+            end_date = pd.Timestamp.utcnow() + timedelta(days=365)
+
+        if lot == 0 and min_trade_size > 0:
+            lot = min_trade_size
+
+        super().__init__(
+            sid,
+            exchange,
+            symbol=symbol,
+            asset_name=asset_name,
+            start_date=start_date,
+            end_date=end_date,
+            first_traded=first_traded,
+            auto_close_date=auto_close_date,
+            exchange_full=exchange_full,
+            min_trade_size=min_trade_size,
+        )
+
+        self.maker = maker
+        self.taker = taker
+        self.leverage = leverage
+        self.end_daily = end_daily
+        self.end_minute = end_minute
+        self.exchange_symbol = exchange_symbol
+        self.trading_state = trading_state
+        self.data_source = data_source
+        self.max_trade_size = max_trade_size
+        self.lot = lot
+        self.decimals = decimals
+
+    def __repr__(self):
+        return 'Trading Pair {symbol}({sid}) Exchange: {exchange}, ' \
+               'Introduced On: {start_date}, ' \
+               'Base Currency: {base_currency}, ' \
+               'Quote Currency: {quote_currency}, ' \
+               'Exchange Leverage: {leverage}, ' \
+               'Minimum Trade Size: {min_trade_size} ' \
+               'Last daily ingestion: {end_daily} ' \
+               'Last minutely ingestion: {end_minute}'.format(
+            symbol=self.symbol,
+            sid=self.sid,
+            exchange=self.exchange,
+            start_date=self.start_date,
+            quote_currency=self.quote_currency,
+            base_currency=self.base_currency,
+            leverage=self.leverage,
+            min_trade_size=self.min_trade_size,
+            end_daily=self.end_daily,
+            end_minute=self.end_minute
+        )
+
+    cpdef to_dict(self):
+        """
+        Convert to a python dict.
+        """
+        #TODO: missing fields
+        super_dict = super(TradingPair, self).to_dict()
+        super_dict['end_daily'] = self.end_daily
+        super_dict['end_minute'] = self.end_minute
+        super_dict['leverage'] = self.leverage
+        super_dict['min_trade_size'] = self.min_trade_size
+        return super_dict
+
+    def is_exchange_open(self, dt_minute):
+        """
+        Parameters
+        ----------
+        dt_minute: pd.Timestamp (UTC, tz-aware)
+            The minute to check.
+
+        Returns
+        -------
+        boolean: whether the asset's exchange is open at the given minute.
+        """
+        #TODO: make more dymanic to catch holds
+        return True
+
+    cpdef __reduce__(self):
+        """
+        Function used by pickle to determine how to serialize/deserialize this
+        class.  Should return a tuple whose first element is self.__class__,
+        and whose second element is a tuple of all the attributes that should
+        be serialized/deserialized during pickling.
+        """
+        #TODO: make sure that all fields set there
+        return (self.__class__, (self.symbol,
+                                 self.exchange,
+                                 self.start_date,
+                                 self.asset_name,
+                                 self.sid,
+                                 self.leverage,
+                                 self.end_date,
+                                 self.first_traded,
+                                 self.auto_close_date,
+                                 self.exchange_full,
+                                 self.min_trade_size,
+                                 self.max_trade_size,
+                                 self.lot,
+                                 self.decimals,
+                                 self.taker,
+                                 self.maker))
 
 def make_asset_array(int size, Asset asset):
     cdef np.ndarray out = np.empty([size], dtype=object)

catalyst/assets/asset_db_schema.py

@@ -39,7 +39,8 @@ equities = sa.Table(
     sa.Column('first_traded', sa.Integer),
     sa.Column('auto_close_date', sa.Integer),
     sa.Column('exchange', sa.Text),
-    sa.Column('exchange_full', sa.Text)
+    sa.Column('exchange_full', sa.Text),
+    sa.Column('min_trade_size', sa.Float)
 )
 
 equity_symbol_mappings = sa.Table(

catalyst/assets/asset_writer.py

@@ -73,6 +73,7 @@ _equities_defaults = {
     'exchange': None,
     # optional, something like "New York Stock Exchange"
     'exchange_full': None,
+    'min_trade_size': 1
 }
 
 # Default values for the futures DataFrame
@@ -390,6 +391,8 @@ class AssetDBWriter(object):
                   The date on which to close any positions in this asset.
               exchange : str
                   The exchange where this asset is traded.
+              min_trade_size: float, optional
+                  The minimum denomination this asset can be traded.
 
             The index of this dataframe should contain the sids.
         futures : pd.DataFrame, optional

catalyst/assets/assets.py

@@ -76,7 +76,9 @@ from catalyst.utils.numpy_utils import as_column
 from catalyst.utils.preprocess import preprocess
 from catalyst.utils.sqlite_utils import group_into_chunks, coerce_string_to_eng
 
-log = Logger('assets.py')
+from catalyst.constants import LOG_LEVEL
+
+log = Logger('assets.py', level=LOG_LEVEL)
 
 # A set of fields that need to be converted to strings before building an
 # Asset to avoid unicode fields

catalyst/constants.py

@@ -0,0 +1,18 @@
+# -*- coding: utf-8 -*-
+
+import os
+import logbook
+
+''' You can override the LOG level from your environment.
+    For example, if you want to see the DEBUG messages, run:
+    $ export CATALYST_LOG_LEVEL=10
+'''
+LOG_LEVEL = int(os.environ.get('CATALYST_LOG_LEVEL', logbook.INFO))
+
+SYMBOLS_URL = 'https://s3.amazonaws.com/enigmaco/catalyst-exchanges/' \
+              '{exchange}/symbols.json'
+
+DATE_TIME_FORMAT = '%Y-%m-%d %H:%M'
+DATE_FORMAT = '%Y-%m-%d'
+
+AUTO_INGEST = False

catalyst/curate/poloniex.py

@@ -1,35 +1,47 @@
-import json, time, csv
-from datetime import datetime
-import pandas as pd
 import os
 import time
+import shutil
+import json
+import csv
+from datetime import datetime
+
+import pandas as pd
 import requests
 import logbook
 
-DT_START        = time.mktime(datetime(2010, 1, 1, 0, 0).timetuple())
-CSV_OUT_FOLDER  = '/var/tmp/catalyst/data/poloniex/'
-CONN_RETRIES    = 2
+from catalyst.exchange.exchange_utils import get_exchange_symbols_filename
+
+
+DT_START = int(time.mktime(datetime(2010, 1, 1, 0, 0).timetuple()))
+DT_END = pd.to_datetime('today').value // 10 ** 9
+CSV_OUT_FOLDER = os.environ.get('CSV_OUT_FOLDER', '/efs/exchanges/poloniex/')
+CONN_RETRIES = 2
 
 logbook.StderrHandler().push_application()
 log = logbook.Logger(__name__)
 
-class PoloniexCurator(object):
-    """
-    OHLCV data feed generator for crypto data. Based on Poloniex market data
-    """
 
-    _api_path       = 'https://poloniex.com/public?'
-    currency_pairs  = []
+class PoloniexCurator(object):
+    '''
+    OHLCV data feed generator for crypto data. Based on Poloniex market data
+    '''
+
+    _api_path = 'https://poloniex.com/public?'
+    currency_pairs = []
 
     def __init__(self):
         if not os.path.exists(CSV_OUT_FOLDER):
             try:
                 os.makedirs(CSV_OUT_FOLDER)
             except Exception as e:
-                log.error('Failed to create data folder: %s' % CSV_OUT_FOLDER)
+                log.error('Failed to create data folder: {}'.format(
+                            CSV_OUT_FOLDER))
                 log.exception(e)
 
     def get_currency_pairs(self):
+        '''
+        Retrieves and returns all currency pairs from the exchange
+        '''
         url = self._api_path + 'command=returnTicker'
 
         try:
@@ -40,105 +52,330 @@ class PoloniexCurator(object):
             return None
 
         data = response.json()
-        self.currency_pairs  = []
+        self.currency_pairs = []
         for ticker in data:
             self.currency_pairs.append(ticker)
         self.currency_pairs.sort()
 
-        log.debug('Currency pairs retrieved successfully: %d' % (len(self.currency_pairs)))
+        log.debug('Currency pairs retrieved successfully: {}'.format(
+                    len(self.currency_pairs)
+                    ))
 
-    def _get_start_date(self, csv_fn):
-        ''' Function returns latest appended date, if the file has been previously written
-            the last line is an empty one, so we have to read the second to last line 
+    def _retrieve_tradeID_date(self, row):
+        '''
+        Helper function that reads tradeID and date fields from CSV readline
+        '''
+        tId = int(row.split(',')[0])
+        d = pd.to_datetime(row.split(',')[1],
+                           infer_datetime_format=True).value // 10 ** 9
+        return tId, d
+
+    def retrieve_trade_history(self, currencyPair, start=DT_START,
+                               end=DT_END, temp=None):
+        '''
+        Retrieves TradeHistory from exchange for a given currencyPair
+        between start and end dates. If no start date is provided, uses
+        a system-wide one (beginning of time for cryptotrading).
+        If no end date is provided, 'now' is used.
+
+        Stores results in CSV file on disk.
+
+        This function is called recursively to work around the
+        limitations imposed by the provider API.
+        '''
+        csv_fn = CSV_OUT_FOLDER + 'crypto_trades-' + currencyPair + '.csv'
+
+        '''
+        Check what data we already have on disk, reading first and last
+        lines from file. Data is stored on file from NEWEST to OLDEST.
         '''
         try:
-            with open(csv_fn, 'ab+') as f: 
-                f.seek(0, os.SEEK_END)              # First check file is not zero size
-                if(f.tell() > 2):
-                    f.seek(-2, os.SEEK_END)         # Jump to the second last byte.
+            with open(csv_fn, 'ab+') as f:
+                f.seek(0, os.SEEK_END)
+                if(f.tell() > 2):                   # Check file size is not 0
+                    f.seek(0)                       # Go to start to read
+                    last_tradeID, end_file = self._retrieve_tradeID_date(
+                                                    f.readline())
+                    f.seek(-2, os.SEEK_END)         # Jump to the 2nd last byte
                     while f.read(1) != b"\n":       # Until EOL is found...
-                        f.seek(-2, os.SEEK_CUR)     # ...jump back the read byte plus one more.
-                    lastrow = f.readline() 
-                    return int(lastrow.split(',')[0]) + 300
+                        # ...jump back the read byte plus one more.
+                        f.seek(-2, os.SEEK_CUR)
+                    first_tradeID, start_file = self._retrieve_tradeID_date(
+                                                    f.readline())
+
+                    if(end_file + 3600 * 6 > DT_END
+                        and (first_tradeID == 1
+                             or (currencyPair == 'BTC_HUC'
+                                 and first_tradeID == 2)
+                             or (currencyPair == 'BTC_RIC'
+                                 and first_tradeID == 2)
+                             or (currencyPair == 'BTC_XCP'
+                                 and first_tradeID == 2)
+                             or (currencyPair == 'BTC_NAV'
+                                 and first_tradeID == 4569)
+                             or (currencyPair == 'BTC_POT'
+                                 and first_tradeID == 23511))):
+                        return
 
         except Exception as e:
-            log.error('Error opening file: %s' % csv_fn)
+            log.error('Error opening file: {}'.format(csv_fn))
             log.exception(e)
 
-        return DT_START
+        '''
+        Poloniex API limits querying TradeHistory to intervals smaller
+        than 1 month, so we make sure that start date is never more than
+        1 month apart from end date
+        '''
+        if(end - start > 2419200):   # 60s/min * 60min/hr * 24hr/day * 28days
+            newstart = end - 2419200
+        else:
+            newstart = start
 
-    def get_data(self, currencyPair, start, end=9999999999, period=300):
-        url = self._api_path + 'command=returnChartData&currencyPair=' + currencyPair + '&start=' + str(start) + '&end=' + str(end) + '&period=' + str(period)
+        log.debug('{}: Retrieving from {} to {}\t {} - {}'.format(
+                    currencyPair, str(newstart), str(end),
+                    time.ctime(newstart), time.ctime(end)))
 
-        try:
-            response = requests.get(url)
-        except Exception as e:
-            log.error('Failed to retrieve candlestick chart data for %s' % currencyPair)
-            log.exception(e)
+        url = '{path}command=returnTradeHistory&currencyPair={pair}' \
+              '&start={start}&end={end}'.format(
+                    path=self._api_path,
+                    pair=currencyPair,
+                    start=str(newstart),
+                    end=str(end)
+                )
+
+        attempts = 0
+        success = 0
+        while attempts < CONN_RETRIES:
+            try:
+                response = requests.get(url)
+            except Exception as e:
+                log.error('Failed to retrieve trade history data'
+                          'for {}'.format(currencyPair))
+                log.exception(e)
+                attempts += 1
+            else:
+                try:
+                    if(isinstance(response.json(), dict)
+                       and response.json()['error']):
+                        log.error('Failed to to retrieve trade history data '
+                                  'for {}: {}'.format(
+                                    currencyPair,
+                                    response.json()['error']
+                                    ))
+                        attempts += 1
+                except Exception as e:
+                    log.exception(e)
+                    attempts += 1
+                else:
+                    success = 1
+                    break
+
+        if not success:
             return None
 
-        return response.json()
+        '''
+        If we get to transactionId == 1, and we already have that on
+        disk, we got to the end of TradeHistory for this coin.
+        '''
+        if('first_tradeID' in locals()
+                and response.json()[-1]['tradeID'] == first_tradeID):
+            return
 
-    '''
-    Pulls latest data for a single pair
-    '''
-    def append_data_single_pair(self, currencyPair, repeat=0):
-        log.debug('Getting data for %s' % currencyPair)
-        csv_fn = CSV_OUT_FOLDER + 'crypto_prices-' + currencyPair + '.csv'
-        start  = self._get_start_date(csv_fn)
-        # Only fetch data if more than 5min have passed since last fetch
-        if (time.time() > start):
-            data   = self.get_data(currencyPair, start)
-            if data is not None:
-                try: 
-                    with open(csv_fn, 'ab') as csvfile:
-                        csvwriter = csv.writer(csvfile)
-                        for item in data:
-                            if item['date'] == 0:
-                                continue
-                            csvwriter.writerow([
-                                item['date'],
-                                item['open'],
-                                item['high'],
-                                item['low'],
-                                item['close'],
-                                item['volume'],
-                            ])
-                except Exception as e:
-                    log.error('Error opening %s' % csv_fn)
-                    log.exception(e)
-            elif (repeat < CONN_RETRIES):
-                    log.debug('Retrying: attemt %d' % (repeat+1) )
-                    self.append_data_single_pair(currencyPair, repeat + 1)    
+        '''
+            There are primarily two scenarios:
+                a) There is newer data available that we need to add at
+                   the beginning of the file. We'll retrieve all what we
+                   need until we get to what we already have, writing it
+                   to a temporary file; and we will write that at the
+                   beginning of our existing file.
+                b) We are going back in time, appending at the end of
+                   our existing TradeHistory until the first transaction
+                   for this currencyPair
+        '''
+        try:
+            if('end_file' in locals() and end_file + 3600 < end):
+                if (temp is None):
+                    temp = os.tmpfile()
+                tempcsv = csv.writer(temp)
+                for item in response.json():
+                    if(item['tradeID'] <= last_tradeID):
+                        continue
+                    tempcsv.writerow([
+                        item['tradeID'],
+                        item['date'],
+                        item['type'],
+                        item['rate'],
+                        item['amount'],
+                        item['total'],
+                        item['globalTradeID'],
+                    ])
+                if(response.json()[-1]['tradeID'] > last_tradeID):
+                    end = pd.to_datetime(response.json()[-1]['date'],
+                                         infer_datetime_format=True
+                                         ).value // 10**9
+                    self.retrieve_trade_history(currencyPair, start,
+                                                end, temp=temp)
+                else:
+                    with open(csv_fn, 'rb+') as f:
+                        shutil.copyfileobj(f, temp)
+                        f.seek(0)
+                        temp.seek(0)
+                        shutil.copyfileobj(temp, f)
+                    temp.close()
+                    end = start_file
+            else:
+                with open(csv_fn, 'ab') as csvfile:
+                    csvwriter = csv.writer(csvfile)
+                    for item in response.json():
+                        if('first_tradeID' in locals()
+                                and item['tradeID'] >= first_tradeID):
+                            continue
+                        csvwriter.writerow([
+                            item['tradeID'],
+                            item['date'],
+                            item['type'],
+                            item['rate'],
+                            item['amount'],
+                            item['total'],
+                            item['globalTradeID']
+                        ])
+                end = pd.to_datetime(response.json()[-1]['date'],
+                                     infer_datetime_format=True).value//10**9
 
-    '''
-    Pulls latest data for all currency pairs
-    '''
-    def append_data(self):
-        for currencyPair in self.currency_pairs:
-            self.append_data_single_pair(currencyPair)
-            # Rate limit is 6 calls per second, sleep 1sec/6 to be safe
-            time.sleep(0.17)
+        except Exception as e:
+            log.error('Error opening {}'.format(csv_fn))
+            log.exception(e)
 
-    '''
-    Returns a data frame for all pairs, or for the requests currency pair.
-    Makes sure data is up to date
-    '''
-    def to_dataframe(self, start, end, currencyPair=None):
-        csv_fn = CSV_OUT_FOLDER + 'crypto_prices-' + currencyPair + '.csv'
-        last_date = self._get_start_date(csv_fn)
-        if last_date + 300 < end or not os.path.exists(csv_fn):
-            # get latest data
-            self.append_data_single_pair(currencyPair)
+        '''
+            If we got here, we aren't done yet. Call recursively with
+            'end' times that go sequentially back in time.
+        '''
+        self.retrieve_trade_history(currencyPair, start, end)
 
-        # CSV holds the latest snapshot
-        df = pd.read_csv(csv_fn, names=['date', 'open', 'high', 'low', 'close', 'volume'])
-        df['date']=pd.to_datetime(df['date'],unit='s')
+    def generate_ohlcv(self, df):
+        '''
+        Generates OHLCV dataframe from a dataframe containing all TradeHistory
+        by resampling with 1-minute period
+        '''
+        df.set_index('date', inplace=True)             # Index by date
+        vol = df['total'].to_frame('volume')           # set Vol aside
+        df.drop('total', axis=1, inplace=True)         # Drop volume data
+        ohlc = df.resample('T').ohlc()                 # Resample OHLC 1min
+        ohlc.cols = ohlc.cols.map(lambda t: t[1])      # Raname cols
+        closes = ohlc['close'].fillna(method='pad')    # Pad fwd missing close
+        ohlc = ohlc.apply(lambda x: x.fillna(closes))  # Fill NA w/ last close
+        vol = vol.resample('T').sum().fillna(0)        # Add volumes by bin
+        ohlcv = pd.concat([ohlc, vol], axis=1)         # Concat OHLC + Vol
+        return ohlcv
+
+    def write_ohlcv_file(self, currencyPair):
+        '''
+        Generates OHLCV data file with 1minute bars from TradeHistory on disk
+        '''
+        csv_trades = CSV_OUT_FOLDER + 'crypto_trades-' + currencyPair + '.csv'
+        csv_1min = CSV_OUT_FOLDER + 'crypto_1min-' + currencyPair + '.csv'
+        if(os.path.getmtime(csv_1min) > time.time() - 7200):
+            log.debug(currencyPair+': 1min data file already up to date. '
+                      'Delete the file if you want to rebuild it.')
+        else:
+            df = pd.read_csv(csv_trades,
+                             names=['tradeID',
+                                    'date',
+                                    'type',
+                                    'rate',
+                                    'amount',
+                                    'total',
+                                    'globalTradeID'],
+                             dtype={'tradeID': int,
+                                    'date': str,
+                                    'type': str,
+                                    'rate': float,
+                                    'amount': float,
+                                    'total': float,
+                                    'globalTradeID': int}
+                             )
+            df.drop(['tradeID', 'type', 'amount', 'globalTradeID'],
+                    axis=1, inplace=True)
+            df['date'] = pd.to_datetime(df['date'], infer_datetime_format=True)
+            ohlcv = self.generate_ohlcv(df)
+            try:
+                with open(csv_1min, 'w') as csvfile:
+                    csvwriter = csv.writer(csvfile)
+                    for item in ohlcv.itertuples():
+                        if item.Index == 0:
+                            continue
+                        csvwriter.writerow([
+                            item.Index.value // 10 ** 9,
+                            item.open,
+                            item.high,
+                            item.low,
+                            item.close,
+                            item.volume,
+                        ])
+            except Exception as e:
+                log.error('Error opening {}'.format(csv_1min))
+                log.exception(e)
+            log.debug('{}: Generated 1min OHLCV data.'.format(currencyPair))
+
+    def onemin_to_dataframe(self, currencyPair, start, end):
+        '''
+        Returns a data frame for a given currencyPair from data on disk
+        '''
+        csv_fn = CSV_OUT_FOLDER + 'crypto_1min-' + currencyPair + '.csv'
+        df = pd.read_csv(csv_fn, names=['date',
+                                        'open',
+                                        'high',
+                                        'low',
+                                        'close',
+                                        'volume'])
+        df['date'] = pd.to_datetime(df['date'], unit='s')
         df.set_index('date', inplace=True)
+        return df[start:end]
+
+    def generate_symbols_json(self, filename=None):
+        '''
+        Generates a symbols.json file with corresponding start_date
+        for each currencyPair
+        '''
+        symbol_map = {}
+
+        if(filename is None):
+            filename = get_exchange_symbols_filename('poloniex')
+
+        with open(filename, 'w') as symbols:
+            for currencyPair in self.currency_pairs:
+                start = None
+                csv_fn = '{}crypto_trades-{}.csv'.format(
+                                                    CSV_OUT_FOLDER,
+                                                    currencyPair)
+                with open(csv_fn, 'r') as f:
+                    f.seek(0, os.SEEK_END)
+                    if(f.tell() > 2):               # Check file size is not 0
+                        f.seek(-2, os.SEEK_END)     # Jump to 2nd last byte
+                        while f.read(1) != b"\n":   # Until EOL is found...
+                            # ...jump back the read byte plus one more.
+                            f.seek(-2, os.SEEK_CUR)
+                        start = pd.to_datetime(f.readline().split(',')[1],
+                                               infer_datetime_format=True)
+
+                if(start is None):
+                    start = time.gmtime()
+                base, market = currencyPair.lower().split('_')
+                symbol = '{market}_{base}'.format(market=market, base=base)
+                symbol_map[currencyPair] = dict(
+                    symbol=symbol,
+                    start_date=start.strftime("%Y-%m-%d")
+                )
+            json.dump(symbol_map, symbols, sort_keys=True, indent=2,
+                      separators=(',', ':'))
 
-        return df[datetime.fromtimestamp(start):datetime.fromtimestamp(end-1)]
 
 if __name__ == '__main__':
     pc = PoloniexCurator()
     pc.get_currency_pairs()
-    pc.append_data()
+    # pc.generate_symbols_json()
+
+    for currencyPair in pc.currency_pairs:
+        pc.retrieve_trade_history(currencyPair)
+        log.debug('{} up to date.'.format(currencyPair))
+        pc.write_ohlcv_file(currencyPair)

catalyst/data/_equities.pyx

@@ -215,13 +215,13 @@ cpdef _read_bcolz_data(ctable_t table,
                 else:
                     continue
 
-        if column_name in ['open', 'high', 'low', 'close']:
+        if column_name in ['open', 'high', 'low', 'close', 'volume']:
             where_nan = (outbuf == 0)
-            outbuf_as_float = outbuf.astype(float64) * .000001
+            outbuf_as_float = outbuf.astype(float64) * .000000001
             outbuf_as_float[where_nan] = NAN
             results.append(outbuf_as_float)
-        elif column_name != 'volume':
-            results.append(outbuf.astype(uint32))
+        elif column_name in ['volume']:
+            results.append(outbuf.astype(float64) * .000000001)
         else:
             results.append(outbuf)
     return results

catalyst/data/_minute_bar_internal.pyx

@@ -35,17 +35,6 @@ def minute_value(ndarray[long_t, ndim=1] market_opens,
 
     return market_opens[q] + r
 
-@cython.cdivision(True)
-def five_minute_value(ndarray[long_t, ndim=1] market_opens,
-                      Py_ssize_t pos,
-                      short five_minutes_per_day):
-
-    cdef short q, r
-    q = cython.cdiv(pos, five_minutes_per_day)
-    r = cython.cmod(pos, five_minutes_per_day)
-
-    return market_opens[q] + r
-
 def find_position_of_minute(ndarray[long_t, ndim=1] market_opens,
                             ndarray[long_t, ndim=1] market_closes,
                             long_t minute_val,
@@ -99,26 +88,6 @@ def find_position_of_minute(ndarray[long_t, ndim=1] market_opens,
 
     return (market_open_loc * minutes_per_day) + delta
 
-def find_position_of_five_minute(ndarray[long_t, ndim=1] market_opens,
-                                 ndarray[long_t, ndim=1] market_closes,
-                                 long_t five_minute_val,
-                                 short five_minutes_per_day,
-                                 bool forward_fill):
-
-    cdef Py_ssize_t market_open_loc, market_open, delta
-
-    market_open_loc = \
-        searchsorted(market_opens, five_minute_val, side='right') - 1
-    market_open = market_opens[market_open_loc]
-    market_close = market_closes[market_open_loc]
-
-    if not forward_fill and ((five_minute_val - market_open) >= five_minutes_per_day):
-        raise ValueError("Given five minutes is not between an open and a close")
-
-    delta = int_min(five_minute_val - market_open, market_close - market_open)
-
-    return (market_open_loc * five_minutes_per_day) + delta
-
 def find_last_traded_position_internal(
         ndarray[long_t, ndim=1] market_opens,
         ndarray[long_t, ndim=1] market_closes,
@@ -189,50 +158,3 @@ def find_last_traded_position_internal(
     # found a trade event
     return -1
 
-def find_last_traded_five_minute_position_internal(
-        ndarray[long_t, ndim=1] market_opens,
-        ndarray[long_t, ndim=1] market_closes,
-        long_t end_five_minute,
-        long_t start_five_minute,
-        volumes,
-        short five_minutes_per_day):
-    cdef Py_ssize_t minute_pos, current_minute, q
-
-    five_minute_pos = int_min(
-        find_position_of_five_minute(
-            market_opens,
-            market_closes,
-            end_five_minute,
-            five_minutes_per_day,
-            True,
-        ),
-        len(volumes) - 1,
-    )
-
-    while five_minute_pos >= 0:
-        current_five_minute = five_minute_value(
-            market_opens, five_minute_pos, five_minutes_per_day
-        )
-
-        q = cython.cdiv(five_minute_pos, five_minutes_per_day)
-        if current_five_minute > market_closes[q]:
-            five_minute_pos = find_position_of_five_minute(
-                market_opens,
-                market_closes,
-                market_closes[q],
-                five_minutes_per_day,
-                False,
-            )
-            continue
-
-        if current_five_minute < start_five_minute:
-            return -1
-
-        if volumes[five_minute_pos] != 0:
-            return five_minute_pos
-
-        five_minute_pos -= 1
-
-    # we've gone to the beginning of this asset's range, and still haven't
-    # found a trade event
-    return -1

catalyst/data/bundles/__init__.py

@@ -1,6 +1,5 @@
 # These imports are necessary to force module-scope register calls to happen.
 from . import quandl  # noqa
-from . import poloniex
 from .core import (
     UnknownBundle,
     bundles,

catalyst/data/bundles/base.py

@@ -13,10 +13,9 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-
 from itertools import count
 import tarfile
-from time import time, sleep
+from time import sleep
 
 from abc import abstractmethod, abstractproperty
 import logbook
@@ -30,11 +29,14 @@ from catalyst.utils.cli import (
 )
 from catalyst.utils.memoize import lazyval
 
+from catalyst.constants import LOG_LEVEL
+
 logbook.StderrHandler().push_application()
-log = logbook.Logger(__name__)
+log = logbook.Logger(__name__, level=LOG_LEVEL)
 
 DEFAULT_RETRIES = 5
 
+
 class BaseBundle(object):
     def __init__(self, asset_filter=[]):
         self._asset_filter = asset_filter
@@ -60,10 +62,6 @@ class BaseBundle(object):
     def minutes_per_day(self):
         raise NotImplementedError()
 
-    @lazyval
-    def five_minutes_per_day(self):
-        raise NotImplementedError()
-
     @lazyval
     def frequencies(self):
         raise NotImplementedError()
@@ -106,16 +104,15 @@ class BaseBundle(object):
 
     def post_process_symbol_metadata(self, metadata, data):
         return metadata
-    
+
     @abstractmethod
     def fetch_raw_symbol_frame(self, api_key, symbol, start_date, end_date):
         raise NotImplementedError()
-    
+
     def ingest(self,
                environ,
                asset_db_writer,
                minute_bar_writer,
-               five_minute_bar_writer,
                daily_bar_writer,
                adjustment_writer,
                calendar,
@@ -131,7 +128,7 @@ class BaseBundle(object):
             retries = environ.get('CATALYST_DOWNLOAD_ATTEMPTS', 5)
 
             if is_compile:
-                # User has instructed local compilation and ingestion of bundle.
+                # User has instructed local compilation & ingestion of bundle.
                 # Fetch raw metadata for all symbols.
                 raw_metadata = self._fetch_metadata_frame(
                     api_key,
@@ -160,9 +157,9 @@ class BaseBundle(object):
                         show_progress=show_progress,
                     )
 
-                # Post-process metadata using cached symbol frames, and write to
-                # disk.  This metadata must be written before any attempt to write
-                # either minute or 5-minute data.
+                # Post-process metadata using cached symbol frames, and write
+                # to disk.  This metadata must be written before any attempt
+                # to write minute data.
                 metadata = self._post_process_metadata(
                     raw_metadata,
                     cache,
@@ -170,24 +167,6 @@ class BaseBundle(object):
                 )
                 asset_db_writer.write(metadata)
 
-                # Compile 5-minute symbol data if bundle supports 5-minute mode and
-                # persist the dataset to disk.
-                if '5-minute' in self.frequencies:
-                    five_minute_bar_writer.write(
-                        self._fetch_symbol_iter(
-                            api_key,
-                            cache,
-                            symbol_map,
-                            calendar,
-                            start_session,
-                            end_session,
-                            '5-minute',
-                            retries,
-                        ),
-                        length=len(symbol_map),
-                        show_progress=show_progress,
-                    )
-
                 # Compile minute symbol data if bundle supports minute mode and
                 # persist the dataset to disk.
                 if 'minute' in self.frequencies:
@@ -205,10 +184,11 @@ class BaseBundle(object):
                         show_progress=show_progress,
                     )
 
-                # For legacy purposes, this call is required to ensure the database
-                # contains an appropriately initialized file structure.  We don't
-                # forsee a usecase for adjustments at this time, but may later
-                # choose to expose this functionality in the future.
+                # For legacy purposes, this call is required to ensure the
+                # database contains an appropriately initialized file
+                # structure.  We don't forsee a usecase for adjustments at
+                # this time, but may later choose to expose this functionality
+                # in the future.
                 adjustment_writer.write(
                     splits=(
                         pd.concat(self.splits, ignore_index=True)
@@ -253,12 +233,12 @@ class BaseBundle(object):
             tar.extractall(output_dir)
 
     def _fetch_metadata_frame(self,
-                             api_key,
-                             cache,
-                             retries=DEFAULT_RETRIES,
-                             environ=None,
-                             show_progress=False):
-        
+                              api_key,
+                              cache,
+                              retries=DEFAULT_RETRIES,
+                              environ=None,
+                              show_progress=False):
+
         # Setup raw metadata iterator to fetch pages if necessary.
         raw_iter = self._fetch_metadata_iter(api_key, cache, retries, environ)
 
@@ -272,7 +252,7 @@ class BaseBundle(object):
             show_percent=False,
         ) as blocks:
             metadata = pd.concat(blocks, ignore_index=True)
-        
+
         return metadata
 
     def _fetch_metadata_iter(self, api_key, cache, retries, environ):
@@ -290,30 +270,28 @@ class BaseBundle(object):
                             page_number,
                         )
                         break
-                    except ValueError as e:
+                    except ValueError:
                         raw = pd.DataFrame([])
                         break
-                    except Exception as e:
+                    except Exception:
                         log.exception(
                             'Failed to load metadata from {}. '
-                            'Retrying.'.format(
-                                name=self.name,
-                            )
-                        ) 
+                            'Retrying.'.format(self.name)
+                        )
                 else:
                     raise ValueError(
-                        'Failed to download metadata page %d after %d '
-                        'attempts.'.format(page_number, retries),
+                        'Failed to download metadata page {} after {} '
+                        'attempts.'.format(page_number, retries)
                     )
 
-
             if raw.empty:
                 # Empty DataFrame signals completion.
                 break
 
             # Apply selective asset filtering, useful for benchmark
             # ingestion.
-            raw = raw[raw.symbol.isin(self._asset_filter)]
+            if self._asset_filter:
+                raw = raw[raw.symbol.isin(self._asset_filter)]
 
             # Update cached value for key.
             cache[key] = raw
@@ -327,7 +305,7 @@ class BaseBundle(object):
             columns=self.md_column_names,
             index=metadata.index,
         )
-        
+
         # Iterate over the available symbols, loading the asset's raw symbol
         # data from the cache.  The final metadata is computed and recorded in
         # the appropriate row depending on the asset's id.
@@ -340,22 +318,22 @@ class BaseBundle(object):
             show_percent=False,
         ) as symbols_map:
             for asset_id, symbol in symbols_map:
-                # Attempt to load data from disk, the cache should have an entry
-                # for each symbol at this point of the execution. If one does
-                # not exist, we should fail.
+                # Attempt to load data from disk, the cache should have an
+                # entry for each symbol at this point of the execution. If one
+                # does not exist, we should fail.
                 key = '{sym}.daily.frame'.format(sym=symbol)
                 try:
                     raw_data = cache[key]
                 except KeyError:
                     raise ValueError(
-                      'Unable to find cached data for symbol: {0}'.format(symbol)
-                    )
+                      'Unable to find cached data for symbol:'
+                      ' {0}'.format(symbol))
 
                 # Perform and require post-processing of metadata.
                 final_symbol_metadata = self.post_process_symbol_metadata(
                     asset_id,
                     metadata.iloc[asset_id],
-                    raw_data,        
+                    raw_data,
                 )
 
                 # Record symbol's final metadata.
@@ -385,8 +363,8 @@ class BaseBundle(object):
             # returns the cached data unaltered.  The `should_sleep` flag
             # indicates that an API call was attempted, and that we should be
             # ensure aren't exceeding our rate limit before proceeding to the
-            # next symbol. If the raw_data is updated, it is cached before being
-            # returned.
+            # next symbol. If the raw_data is updated, it is cached before
+            # being returned.
             raw_data, should_sleep = self._maybe_update_symbol_frame(
                 start_time,
                 api_key,
@@ -436,7 +414,7 @@ class BaseBundle(object):
         last = start_session
         if raw_data is not None and len(raw_data) > 0:
             last = raw_data.index[-1].tz_localize('UTC')
-        
+
         should_sleep = False
 
         # Determine time at which cached data will be considered stale.
@@ -477,7 +455,7 @@ class BaseBundle(object):
                             retries=DEFAULT_RETRIES):
 
         # Data for symbol is old enough to attempt an update or is not
-        # present in the cache.  Fetch raw data for a single symbol 
+        # present in the cache.  Fetch raw data for a single symbol
         # with requested intervals and frequency. Retry as necessary.
         for _ in range(retries):
             try:
@@ -490,7 +468,6 @@ class BaseBundle(object):
                     data_frequency,
                 )
                 raw_data.index = pd.to_datetime(raw_data.index, utc=True)
-                raw_data.index = raw_data.index.tz_localize('UTC')
 
                 # Filter incoming data to fit start and end sessions.
                 raw_data = raw_data[
@@ -504,7 +481,7 @@ class BaseBundle(object):
 
                 return raw_data
 
-            except Exception as e:
+            except Exception:
                 log.exception(
                     'Exception raised fetching {name} data. Retrying.'
                     .format(name=self.name)

catalyst/data/bundles/base_pricing.py

@@ -16,6 +16,7 @@
 from catalyst.data.bundles.base import BaseBundle
 from catalyst.utils.memoize import lazyval
 
+
 class BasePricingBundle(BaseBundle):
     @lazyval
     def md_dtypes(self):
@@ -24,6 +25,7 @@ class BasePricingBundle(BaseBundle):
             ('start_date', 'datetime64[ns]'),
             ('end_date', 'datetime64[ns]'),
             ('ac_date', 'datetime64[ns]'),
+            ('min_trade_size', 'float'),
         ]
 
     @lazyval
@@ -37,6 +39,7 @@ class BasePricingBundle(BaseBundle):
             ('volume', 'float64'),
         ]
 
+
 class BaseCryptoPricingBundle(BasePricingBundle):
     @lazyval
     def calendar_name(self):
@@ -46,10 +49,6 @@ class BaseCryptoPricingBundle(BasePricingBundle):
     def minutes_per_day(self):
         return 1440
 
-    @lazyval
-    def five_minutes_per_day(self):
-        return 288
-
     @property
     def splits(self):
         return []
@@ -58,6 +57,7 @@ class BaseCryptoPricingBundle(BasePricingBundle):
     def dividends(self):
         return []
 
+
 class BaseEquityPricingBundle(BasePricingBundle):
     @lazyval
     def calendar_name(self):
@@ -67,10 +67,6 @@ class BaseEquityPricingBundle(BasePricingBundle):
     def minutes_per_day(self):
         return 390
 
-    @lazyval
-    def five_minutes_per_day(self):
-        return 78
-
     @property
     def splits(self):
         return self._splits

catalyst/data/bundles/core.py

@@ -17,10 +17,6 @@ from ..us_equity_pricing import (
     SQLiteAdjustmentReader,
     SQLiteAdjustmentWriter,
 )
-from ..five_minute_bars import (
-    BcolzFiveMinuteBarReader,
-    BcolzFiveMinuteBarWriter,
-)
 from ..minute_bars import (
     BcolzMinuteBarReader,
     BcolzMinuteBarWriter,
@@ -41,6 +37,7 @@ from catalyst.utils.cli import maybe_show_progress
 
 ONE_MEGABYTE = 1024 * 1024
 
+
 def asset_db_path(bundle_name, timestr, environ=None, db_version=None):
     return pth.data_path(
         asset_db_relative(bundle_name, timestr, environ, db_version),
@@ -54,11 +51,6 @@ def minute_path(bundle_name, timestr, environ=None):
         environ=environ,
     )
 
-def five_minute_path(bundle_name, timestr, environ=None):
-    return pth.data_path(
-        five_minute_relative(bundle_name, timestr, environ),
-        environ=environ,
-    )
 
 def daily_path(bundle_name, timestr, environ=None):
     return pth.data_path(
@@ -90,13 +82,11 @@ def cache_relative(bundle_name, timestr, environ=None):
 
 
 def daily_relative(bundle_name, timestr, environ=None):
-    return bundle_name, timestr, 'daily.bcolz'
+    return bundle_name, timestr, 'daily_equities.bcolz'
 
-def five_minute_relative(bundle_name, timestr, environ=None):
-    return bundle_name, timestr, 'five_minute.bcolz'
 
 def minute_relative(bundle_name, timestr, environ=None):
-    return bundle_name, timestr, 'minute.bcolz'
+    return bundle_name, timestr, 'minute_equities.bcolz'
 
 
 def asset_db_relative(bundle_name, timestr, environ=None, db_version=None):
@@ -146,6 +136,7 @@ def ingestions_for_bundle(bundle, environ=None):
         reverse=True,
     )
 
+
 def download_with_progress(url, chunk_size, **progress_kwargs):
     """
     Download streaming data from a URL, printing progress information to the
@@ -206,14 +197,13 @@ RegisteredBundle = namedtuple(
      'start_session',
      'end_session',
      'minutes_per_day',
-     'five_minutes_per_day',
      'ingest',
      'create_writers']
 )
 
 BundleData = namedtuple(
     'BundleData',
-    'asset_finder minute_bar_reader five_minute_bar_reader daily_bar_reader '
+    'asset_finder minute_bar_reader daily_bar_reader '
     'adjustment_reader',
 )
 
@@ -303,7 +293,6 @@ def _make_bundle_core():
             bundle.ingest,
             calendar_name=bundle.calendar_name,
             minutes_per_day=bundle.minutes_per_day,
-            five_minutes_per_day=bundle.five_minutes_per_day,
             start_session=start_session,
             end_session=end_session,
             create_writers=create_writers,
@@ -316,7 +305,6 @@ def _make_bundle_core():
                  start_session=None,
                  end_session=None,
                  minutes_per_day=1440,
-                 five_minutes_per_day=288,
                  create_writers=True):
         """Register a data bundle ingest function.
 
@@ -397,7 +385,6 @@ def _make_bundle_core():
             start_session=start_session,
             end_session=end_session,
             minutes_per_day=minutes_per_day,
-            five_minutes_per_day=five_minutes_per_day,
             ingest=f,
             create_writers=create_writers,
         )
@@ -496,16 +483,6 @@ def _make_bundle_core():
                 # that it can compute the adjustment ratios for the dividends.
                 daily_bar_writer.write(())
 
-                five_minute_bar_writer = BcolzFiveMinuteBarWriter(
-                    wd.ensure_dir(*five_minute_relative(
-                        name, timestr, environ=environ)
-                    ),
-                    calendar,
-                    start_session,
-                    end_session,
-                    five_minutes_per_day=bundle.five_minutes_per_day,
-                )
-
                 minute_bar_writer = BcolzMinuteBarWriter(
                     wd.ensure_dir(*minute_relative(
                         name, timestr, environ=environ)
@@ -532,7 +509,6 @@ def _make_bundle_core():
                 )
             else:
                 daily_bar_writer = None
-                five_minute_bar_writer = None
                 minute_bar_writer = None
                 asset_db_writer = None
                 adjustment_db_writer = None
@@ -544,7 +520,6 @@ def _make_bundle_core():
                 environ,
                 asset_db_writer,
                 minute_bar_writer,
-                five_minute_bar_writer,
                 daily_bar_writer,
                 adjustment_db_writer,
                 calendar,
@@ -631,9 +606,6 @@ def _make_bundle_core():
             minute_bar_reader=BcolzMinuteBarReader(
                 minute_path(name, timestr, environ=environ),
             ),
-            five_minute_bar_reader=BcolzFiveMinuteBarReader(
-                five_minute_path(name, timestr, environ=environ),
-            ),
             daily_bar_reader=BcolzDailyBarReader(
                 daily_path(name, timestr, environ=environ),
             ),
@@ -735,4 +707,5 @@ def _make_bundle_core():
     )
 
 
-bundles, register_bundle, register, unregister, ingest, load, clean = _make_bundle_core()
+bundles, register_bundle, register, unregister, ingest, load, clean = \
+    _make_bundle_core()

catalyst/data/bundles/poloniex.py

@@ -13,16 +13,18 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from datetime import datetime
+import sys
+from six.moves.urllib.parse import urlencode
 
 import pandas as pd
 
-from six.moves.urllib.parse import urlencode
-
 from catalyst.data.bundles.core import register_bundle
 from catalyst.data.bundles.base_pricing import BaseCryptoPricingBundle
 from catalyst.utils.memoize import lazyval
 
+from catalyst.curate.poloniex import PoloniexCurator
+
+
 class PoloniexBundle(BaseCryptoPricingBundle):
     @lazyval
     def name(self):
@@ -36,14 +38,14 @@ class PoloniexBundle(BaseCryptoPricingBundle):
     def frequencies(self):
         return set((
             'daily',
-            '5-minute',
+            'minute',
         ))
 
     @lazyval
     def tar_url(self):
         return (
-            'https://www.dropbox.com/s/9naqffawnq8o4r2/'
-            'poloniex-bundle.tar?dl=1'
+            'https://s3.amazonaws.com/enigmaco/catalyst-bundles/'
+            'poloniex/poloniex-bundle.tar.gz'
         )
 
     @lazyval
@@ -64,24 +66,25 @@ class PoloniexBundle(BaseCryptoPricingBundle):
 
         raw = raw.sort_index().reset_index()
         raw.rename(
-            columns={'index':'symbol'},
+            columns={'index': 'symbol'},
             inplace=True,
         )
 
         raw = raw[raw['isFrozen'] == 0]
-
         return raw
 
     def post_process_symbol_metadata(self, asset_id, sym_md, sym_data):
         start_date = sym_data.index[0]
         end_date = sym_data.index[-1]
         ac_date = end_date + pd.Timedelta(days=1)
+        min_trade_size = 0.00000001
 
         return (
             sym_md.symbol,
             start_date,
             end_date,
             ac_date,
+            min_trade_size,
         )
 
     def fetch_raw_symbol_frame(self,
@@ -91,19 +94,32 @@ class PoloniexBundle(BaseCryptoPricingBundle):
                                start_date,
                                end_date,
                                frequency):
-        raw = pd.read_json(
-            self._format_data_url(
-                api_key,
-                symbol,
-                start_date,
-                end_date,
-                frequency,
-            ),
-            orient='records',
-        )
-        raw.set_index('date', inplace=True)
 
-        scale = 1000.0
+        # TODO: replace this with direct exchange call
+        # The end date and frequency should be used to
+        # calculate the number of bars
+        if(frequency == 'minute'):
+            pc = PoloniexCurator()
+            raw = pc.onemin_to_dataframe(symbol, start_date, end_date)
+
+        else:
+            raw = pd.read_json(
+                self._format_data_url(
+                    api_key,
+                    symbol,
+                    start_date,
+                    end_date,
+                    frequency,
+                ),
+                orient='records',
+            )
+            raw.set_index('date', inplace=True)
+
+        # BcolzDailyBarReader introduces a 1/1000 factor in the way
+        # pricing is stored on disk, which we compensate here to get
+        # the right pricing amounts
+        # ref: data/us_equity_pricing.py
+        scale = 1
         raw.loc[:, 'open'] /= scale
         raw.loc[:, 'high'] /= scale
         raw.loc[:, 'low'] /= scale
@@ -123,7 +139,6 @@ class PoloniexBundle(BaseCryptoPricingBundle):
 
         return self._format_polo_query(query_params)
 
-
     def _format_data_url(self,
                          api_key,
                          symbol,
@@ -132,7 +147,6 @@ class PoloniexBundle(BaseCryptoPricingBundle):
                          data_frequency):
         period_map = {
             'daily': 86400,
-            '5-minute': 300,
         }
 
         try:
@@ -147,12 +161,26 @@ class PoloniexBundle(BaseCryptoPricingBundle):
             ('end', end_date.value / 10**9),
             ('period', period),
         ]
-            
+
         return self._format_polo_query(query_params)
-    
+
     def _format_polo_query(self, query_params):
+        # TODO: got against the exchange object
         return 'https://poloniex.com/public?{query}'.format(
             query=urlencode(query_params),
         )
 
-register_bundle(PoloniexBundle, ['USDT_BTC'])
+
+'''
+As a second parameter, you can pass an array of currency pairs
+that will be processed as an asset_filter to only process that
+subset of assets in the bundle, such as:
+register_bundle(PoloniexBundle, ['USDT_BTC',])
+
+For a production environment make sure to use (to bundle all pairs):
+register_bundle(PoloniexBundle)
+'''
+if 'ingest' in sys.argv and '-c' in sys.argv:
+    register_bundle(PoloniexBundle)
+else:
+    register_bundle(PoloniexBundle, create_writers=False)

catalyst/data/bundles/quandl.py

@@ -16,7 +16,6 @@
 from datetime import datetime
 
 import pandas as pd
-
 from six.moves.urllib.parse import urlencode
 
 from catalyst.data.bundles.core import register_bundle
@@ -26,23 +25,16 @@ from catalyst.utils.memoize import lazyval
 """
 Module for building a complete daily dataset from Quandl's WIKI dataset.
 """
-from itertools import count
-import tarfile
-from time import time, sleep
-from datetime import datetime
-
 from logbook import Logger
-import pandas as pd
-from six.moves.urllib.parse import urlencode
 
+from catalyst.constants import LOG_LEVEL
 from catalyst.utils.calendars import register_calendar_alias
-from catalyst.utils.cli import maybe_show_progress
 
-from . import core as bundles
 
-log = Logger(__name__)
+log = Logger(__name__, level=LOG_LEVEL)
 seconds_per_call = (pd.Timedelta('10 minutes') / 2000).total_seconds()
 
+
 class QuandlBundle(BaseEquityPricingBundle):
     @lazyval
     def name(self):
@@ -107,8 +99,8 @@ class QuandlBundle(BaseEquityPricingBundle):
         # Filter out invalid symbols
         raw = raw[~raw.symbol.isin(self._excluded_symbols)]
 
-        # cut out all the other stuff in the name column
-        # we need to escape the paren because it is actually splitting on a regex
+        # cut out all the other stuff in the name column. We need to
+        # escape the paren because it is actually splitting on a regex
         raw.asset_name = raw.asset_name.str.split(r' \(', 1).str.get(0)
 
         return raw
@@ -173,7 +165,6 @@ class QuandlBundle(BaseEquityPricingBundle):
         df['sid'] = asset_id
         self.splits.append(df)
 
-
     def _update_dividends(self, asset_id, raw_data):
         divs = raw_data.ex_dividend
         df = pd.DataFrame({'amount': divs[divs != 0]})
@@ -184,7 +175,6 @@ class QuandlBundle(BaseEquityPricingBundle):
         df['record_date'] = df['declared_date'] = df['pay_date'] = pd.NaT
         self.dividends.append(df)
 
-
     def _format_metadata_url(self, api_key, page_number):
         """Build the query RL for the quandl WIKI metadata.
         """
@@ -198,10 +188,10 @@ class QuandlBundle(BaseEquityPricingBundle):
             query_params = [('api_key', api_key)] + query_params
 
         return (
-            'https://www.quandl.com/api/v3/datasets.csv?' + urlencode(query_params)
+            'https://www.quandl.com/api/v3/datasets.csv?'
+            + urlencode(query_params)
         )
 
-
     def _format_wiki_url(self,
                          api_key,
                          symbol,
@@ -227,5 +217,6 @@ class QuandlBundle(BaseEquityPricingBundle):
             )
         )
 
+
 register_calendar_alias('QUANDL', 'NYSE')
 register_bundle(QuandlBundle)

catalyst/data/data_portal.py

@@ -42,7 +42,6 @@ from catalyst.assets.roll_finder import (
 )
 from catalyst.data.dispatch_bar_reader import (
     AssetDispatchMinuteBarReader,
-    AssetDispatchFiveMinuteBarReader,
     AssetDispatchSessionBarReader
 )
 from catalyst.data.resample import (
@@ -69,7 +68,9 @@ from catalyst.errors import (
     HistoryWindowStartsBeforeData,
 )
 
-log = Logger('DataPortal')
+from catalyst.constants import LOG_LEVEL
+
+log = Logger('DataPortal', level=LOG_LEVEL)
 
 BASE_FIELDS = frozenset([
     "open",
@@ -120,10 +121,6 @@ class DataPortal(object):
         daily data backtests or daily history calls in a minute backetest.
         If a daily bar reader is not provided but a minute bar reader is,
         the minutes will be rolled up to serve the daily requests.
-    five_minute_reader : BcolzFiveMinuteBarReader, optional
-        The five minute bar reader for equities. This will be used to service
-        5-minute data backtests or five-minute history calls.  This can be used
-        to serve daily calls if no daily bar reader is provided.
     minute_reader : BcolzMinuteBarReader, optional
         The minute bar reader for equities. This will be used to service
         minute data backtests or minute history calls. This can be used
@@ -150,7 +147,6 @@ class DataPortal(object):
                  trading_calendar,
                  first_trading_day,
                  daily_reader=None,
-                 five_minute_reader=None,
                  minute_reader=None,
                  future_daily_reader=None,
                  future_minute_reader=None,
@@ -202,7 +198,6 @@ class DataPortal(object):
                 reader.last_available_dt
                 for reader in [
                     minute_reader,
-                    five_minute_reader,
                     future_minute_reader,
                 ]
                 if reader is not None
@@ -214,8 +209,6 @@ class DataPortal(object):
 
         aligned_minute_reader = self._ensure_reader_aligned(
             minute_reader)
-        aligned_five_minute_reader = self._ensure_reader_aligned(
-            five_minute_reader)
         aligned_session_reader = self._ensure_reader_aligned(
             daily_reader)
         aligned_future_minute_reader = self._ensure_reader_aligned(
@@ -229,13 +222,10 @@ class DataPortal(object):
         }
 
         aligned_minute_readers = {}
-        aligned_five_minute_readers = {}
         aligned_session_readers = {}
 
         if aligned_minute_reader is not None:
             aligned_minute_readers[Equity] = aligned_minute_reader
-        if aligned_five_minute_reader is not None:
-            aligned_five_minute_readers[Equity] = aligned_five_minute_reader
         if aligned_session_reader is not None:
             aligned_session_readers[Equity] = aligned_session_reader
 
@@ -267,13 +257,6 @@ class DataPortal(object):
             self._last_available_minute,
         )
 
-        _dispatch_five_minute_reader = AssetDispatchFiveMinuteBarReader(
-            self.trading_calendar,
-            self.asset_finder,
-            aligned_five_minute_readers,
-            self._last_available_minute,
-        )
-
         _dispatch_session_reader = AssetDispatchSessionBarReader(
             self.trading_calendar,
             self.asset_finder,
@@ -283,7 +266,6 @@ class DataPortal(object):
 
         self._pricing_readers = {
             'minute': _dispatch_minute_reader,
-            '5-minute': _dispatch_five_minute_reader,
             'daily': _dispatch_session_reader,
         }
 
@@ -674,11 +656,11 @@ class DataPortal(object):
         return spot_value
 
     def _get_minutely_spot_value(self,
-                                asset,
-                                column,
-                                dt,
-                                data_frequency,
-                                ffill=False):
+                                 asset,
+                                 column,
+                                 dt,
+                                 data_frequency,
+                                 ffill=False):
 
         reader = self._get_pricing_reader(data_frequency)
 
@@ -719,23 +701,12 @@ class DataPortal(object):
             spot_value=result
         )
 
-
-    def _get_five_minute_spot_value(self, asset, column, dt, ffill=False):
-        return self._get_minutely_spot_value(
-            asset,
-            column,
-            dt,
-            ffill, 
-            '5-minute',
-        )
-        
-
     def _get_minute_spot_value(self, asset, column, dt, ffill=False):
         return self._get_minutely_spot_value(
             asset,
             column,
             dt,
-            ffill, 
+            ffill,
             'minute',
         )
 

catalyst/data/dispatch_bar_reader.py

@@ -18,6 +18,7 @@ from numpy import (
     full,
     nan,
     int64,
+    float64,
     zeros
 )
 from six import iteritems, with_metaclass
@@ -70,7 +71,9 @@ class AssetDispatchBarReader(with_metaclass(ABCMeta)):
         return self._dt_window_size(start_dt, end_dt), num_sids
 
     def _make_raw_array_out(self, field, shape):
-        if field != 'volume' and field != 'sid':
+        if field == 'volume':
+            out = zeros(shape, dtype=float64)
+        elif field != 'sid':
             out = full(shape, nan)
         else:
             out = zeros(shape, dtype=int64)
@@ -130,17 +133,13 @@ class AssetDispatchBarReader(with_metaclass(ABCMeta)):
 
         return results
 
+
 class AssetDispatchMinuteBarReader(AssetDispatchBarReader):
 
     def _dt_window_size(self, start_dt, end_dt):
         return len(self.trading_calendar.minutes_in_range(start_dt, end_dt))
 
 
-class AssetDispatchFiveMinuteBarReader(AssetDispatchBarReader):
-
-    def _dt_window_size(self, start_dt, end_dt):
-        return len(self.trading_calendar.five_minutes_in_range(start_dt, end_dt))
-
 class AssetDispatchSessionBarReader(AssetDispatchBarReader):
 
     def _dt_window_size(self, start_dt, end_dt):

catalyst/data/history_loader.py

@@ -38,7 +38,7 @@ from catalyst.utils.numpy_utils import float64_dtype
 from catalyst.utils.pandas_utils import find_in_sorted_index
 
 # Default number of decimal places used for rounding asset prices.
-DEFAULT_ASSET_PRICE_DECIMALS = 3
+DEFAULT_ASSET_PRICE_DECIMALS = 9
 
 
 class HistoryCompatibleUSEquityAdjustmentReader(object):

catalyst/data/loader.py

@@ -17,36 +17,32 @@ from collections import OrderedDict
 
 import logbook
 import pandas as pd
-import numpy as np
-from pandas_datareader.data import DataReader
-import datetime
-import time
 import pytz
+from pandas_datareader.data import DataReader
 from six import iteritems
 from six.moves.urllib_error import HTTPError
 
-from .benchmarks import get_benchmark_returns
+from catalyst.utils.calendars import get_calendar
 from . import treasuries, treasuries_can
+from .benchmarks import get_benchmark_returns
+from ..utils.deprecate import deprecated
 from ..utils.paths import (
     cache_root,
     data_root,
 )
-from ..utils.deprecate import deprecated
 
-from catalyst.data.bundles.poloniex import PoloniexBundle
-from catalyst.utils.calendars import get_calendar
+from catalyst.constants import LOG_LEVEL
 
-
-logger = logbook.Logger('Loader')
+logger = logbook.Logger('Loader', level=LOG_LEVEL)
 
 # Mapping from index symbol to appropriate bond data
 INDEX_MAPPING = {
     'SPY':
-    (treasuries, 'treasury_curves.csv', 'www.federalreserve.gov'),
+        (treasuries, 'treasury_curves.csv', 'www.federalreserve.gov'),
     '^GSPTSE':
-    (treasuries_can, 'treasury_curves_can.csv', 'bankofcanada.ca'),
+        (treasuries_can, 'treasury_curves_can.csv', 'bankofcanada.ca'),
     '^FTSE':  # use US treasuries until UK bonds implemented
-    (treasuries, 'treasury_curves.csv', 'www.federalreserve.gov'),
+        (treasuries, 'treasury_curves.csv', 'www.federalreserve.gov'),
 }
 
 ONE_HOUR = pd.Timedelta(hours=1)
@@ -93,20 +89,28 @@ def has_data_for_dates(series_or_df, first_date, last_date):
     dts = series_or_df.index
     if not isinstance(dts, pd.DatetimeIndex):
         raise TypeError("Expected a DatetimeIndex, but got %s." % type(dts))
-    first, last = dts[[0, -1]]
-    return (first <= first_date) and (last >= last_date)
+    first, last = dts[[0, -1]].tz_localize(None)
+    return (first <= first_date.tz_localize(None)) and (
+        last >= last_date.tz_localize(None))
 
-def load_crypto_market_data(trading_day=None,
-                            trading_days=None,
-                            bm_symbol='USDT_BTC',
-                            environ=None):
+
+def load_crypto_market_data(trading_day=None, trading_days=None,
+                            bm_symbol=None, bundle=None, bundle_data=None,
+                            environ=None, exchange=None, start_dt=None,
+                            end_dt=None):
     if trading_day is None:
         trading_day = get_calendar('OPEN').trading_day
-    if trading_days is None:
-        trading_days = get_calendar('OPEN').all_sessions
 
-    first_date = trading_days[0]
-    now = pd.Timestamp.utcnow()
+    # TODO: consider making configurable
+    bm_symbol = 'btc_usdt'
+    # if trading_days is None:
+    #    trading_days = get_calendar('OPEN').schedule
+
+    # if start_dt is None:
+    start_dt = get_calendar('OPEN').first_trading_session
+
+    if end_dt is None:
+        end_dt = pd.Timestamp.utcnow()
 
     # We expect to have benchmark and treasury data that's current up until
     # **two** full trading days prior to the most recently completed trading
@@ -122,29 +126,59 @@ def load_crypto_market_data(trading_day=None,
 
     # We'll attempt to download new data if the latest entry in our cache is
     # before this date.
-    last_date = trading_days[trading_days.get_loc(now, method='ffill') - 2]
+    '''
+    if(bundle_data):
+        # If we are using the bundle to retrieve the cryptobenchmark, find
+        # the last date for which there is trading data in the bundle
+        asset = bundle_data.asset_finder.lookup_symbol(
+                    symbol=bm_symbol,as_of_date=None)
+        ix = bundle_data.daily_bar_reader._last_rows[asset.sid]
+        last_date = pd.to_datetime(
+                    bundle_data.daily_bar_reader._spot_col('day')[ix],unit='s')
+    else:
+        last_date = trading_days[trading_days.get_loc(now, method='ffill') - 2]
+    '''
+    last_date = trading_days[trading_days.get_loc(end_dt, method='ffill') - 1]
 
-    br = ensure_crypto_benchmark_data(
-        bm_symbol,
-        first_date,
-        last_date,
-        now,
-        # We need the trading_day to figure out the close prior to the first
-        # date so that we can compute returns for the first date.
-        trading_day,
-        environ,
-    )
+    if exchange is None:
+        # This is exceptional, since placing the import at the module scope
+        #  breaks things and it's only needed here
+        from catalyst.exchange.factory import get_exchange
+        exchange = get_exchange(
+            exchange_name='poloniex', base_currency='usdt'
+        )
+
+    benchmark_asset = exchange.get_asset(bm_symbol)
+
+    # exchange.get_history_window() already ensures that we have the right data
+    # for the right dates
+    br = exchange.get_history_window_with_bundle(
+        assets=[benchmark_asset],
+        end_dt=last_date,
+        bar_count=pd.Timedelta(last_date - start_dt).days,
+        frequency='1d',
+        field='close',
+        data_frequency='daily',
+        force_auto_ingest=True)
+    br.columns = ['close']
+    br = br.pct_change(1).iloc[1:]
+    br.loc[start_dt] = 0
+    br = br.sort_index()
+
+    # Override first_date for treasury data since we have it for many more
+    # years and is independent of crypto data
+    first_date_treasury = pd.Timestamp('1990-01-02', tz='UTC')
     tc = ensure_treasury_data(
         bm_symbol,
-        first_date,
+        first_date_treasury,
         last_date,
-        now,
+        end_dt,
         environ,
     )
-    benchmark_returns = br[br.index.slice_indexer(first_date, last_date)]
-    treasury_curves = tc[tc.index.slice_indexer(first_date, last_date)]
+    benchmark_returns = br[br.index.slice_indexer(start_dt, last_date)]
+    treasury_curves = tc[
+        tc.index.slice_indexer(first_date_treasury, last_date)]
     return benchmark_returns, treasury_curves
-    
 
 
 def load_market_data(trading_day=None, trading_days=None, bm_symbol='SPY',
@@ -232,20 +266,22 @@ def load_market_data(trading_day=None, trading_days=None, bm_symbol='SPY',
     treasury_curves = tc[tc.index.slice_indexer(first_date, last_date)]
     return benchmark_returns, treasury_curves
 
+
 def ensure_crypto_benchmark_data(symbol,
                                  first_date,
                                  last_date,
                                  now,
                                  trading_day,
+                                 bundle,
+                                 bundle_data,
                                  environ=None):
-
     filename = get_benchmark_filename(symbol)
 
     logger.info(
         ('Loading benchmark data for {symbol!r} '
-            'from {first_date} to {last_date}'),
+         'from {first_date} to {last_date}'),
         symbol=symbol,
-        first_date=first_date - trading_day,
+        first_date=first_date,
         last_date=last_date
     )
 
@@ -258,34 +294,66 @@ def ensure_crypto_benchmark_data(symbol,
         environ,
     )
 
-
     if data is not None:
         return data
 
     # If no cached data was found or it was missing any dates then download the
     # necessary data.
-    logger.info(
-        ('Downloading benchmark data for {symbol!r} '
-            'from {first_date} to {last_date}'),
-        symbol=symbol,
-        first_date=first_date - trading_day,
-        last_date=last_date
-    )
 
-    # Load benchmark symbol from Poloniex API
-    try:
-        bundle = PoloniexBundle()
-        bench_raw = bundle._fetch_symbol_frame(
-            None,
-            symbol,
-            get_calendar(bundle.calendar_name),
-            first_date,
-            last_date,
-            'daily',
-        )
-    except (OSError, IOError, HTTPError):
-        logger.exception('Failed to fetch new crypto benchmark returns')
-        raise
+    if (bundle == 'poloniex'):
+        '''
+        If we're using the Poloniex bundle, we'll get the benchmark from the
+        bundle instead of downloading it from Poloniex every time we need it.
+        Poloniex has a captcha for API queries originating from outside the US
+        that prevents users abroad from getting Catalyst to work
+        '''
+        logger.info(
+            ('Retrieving benchmark data from bundle for {symbol!r}'
+             ' from {first_date} to {last_date}'),
+            symbol=symbol, first_date=first_date, last_date=last_date)
+
+        asset = bundle_data.asset_finder.lookup_symbol(symbol=symbol,
+                                                       as_of_date=None)
+        fields = ['day', 'close']
+        raw = bundle_data.daily_bar_reader.load_raw_arrays(
+            columns=fields,
+            start_date=first_date - trading_day,
+            end_date=last_date,
+            assets=[asset, ])
+        bench_raw = pd.concat([pd.DataFrame(raw[0], columns=['date']),
+                               pd.DataFrame(raw[1], columns=['close'])],
+                              axis=1)
+        bench_raw['date'] = pd.to_datetime(bench_raw['date'], unit='s')
+        bench_raw.set_index('date', inplace=True)
+        bench_raw.sort_index(inplace=True)
+        bench_raw = bench_raw[
+                    pd.to_datetime(first_date - trading_day):pd.to_datetime(
+                        last_date)]
+
+    else:
+        # This is how it used to be: downloading the benchmark everytime.
+        # Leaving this code here to be repurposed in the future for
+        # other bundles.
+        logger.info(
+            ('Downloading benchmark data for {symbol!r}'
+             ' from {first_date} to {last_date}'),
+            symbol=symbol, first_date=first_date, last_date=last_date)
+
+        raise DeprecationWarning('poloniex bundle deprecated')
+        # Load benchmark symbol from Poloniex API
+        # try:
+        #     bundle = PoloniexBundle()
+        #     bench_raw = bundle._fetch_symbol_frame(
+        #         None,
+        #         symbol,
+        #         get_calendar(bundle.calendar_name),
+        #         first_date - trading_day,
+        #         last_date,
+        #         'daily',
+        #     )
+        # except (OSError, IOError, HTTPError):
+        #     logger.exception('Failed to fetch new crypto benchmark returns')
+        #     raise
 
     # select close column and compute percent change between days
     daily_close = bench_raw[['close']]
@@ -344,67 +412,7 @@ def ensure_benchmark_data(symbol, first_date, last_date, now, trading_day,
     # necessary data.
     logger.info(
         ('Downloading benchmark data for {symbol!r} '
-            'from {first_date} to {last_date}'),
-        symbol=symbol,
-        first_date=first_date - trading_day,
-        last_date=last_date
-    )
-
-    try:
-        data = get_benchmark_returns(
-            symbol,
-            first_date - trading_day,
-            last_date,
-        )
-        data.to_csv(get_data_filepath(filename, environ))
-    except (OSError, IOError, HTTPError):
-        logger.exception('Failed to cache the new benchmark returns')
-        raise
-    if not has_data_for_dates(data, first_date, last_date):
-        logger.warn("Still don't have expected data after redownload!")
-    return data
-
-def ensure_benchmark_data(symbol, first_date, last_date, now, trading_day,
-                          environ=None):
-    """
-    Ensure we have benchmark data for `symbol` from `first_date` to `last_date`
-
-    Parameters
-    ----------
-    symbol : str
-        The symbol for the benchmark to load.
-    first_date : pd.Timestamp
-        First required date for the cache.
-    last_date : pd.Timestamp
-        Last required date for the cache.
-    now : pd.Timestamp
-        The current time.  This is used to prevent repeated attempts to
-        re-download data that isn't available due to scheduling quirks or other
-        failures.
-    trading_day : pd.CustomBusinessDay
-        A trading day delta.  Used to find the day before first_date so we can
-        get the close of the day prior to first_date.
-
-    We attempt to download data unless we already have data stored at the data
-    cache for `symbol` whose first entry is before or on `first_date` and whose
-    last entry is on or after `last_date`.
-
-    If we perform a download and the cache criteria are not satisfied, we wait
-    at least one hour before attempting a redownload.  This is determined by
-    comparing the current time to the result of os.path.getmtime on the cache
-    path.
-    """
-    filename = get_benchmark_filename(symbol)
-    data = _load_cached_data(filename, first_date, last_date, now, 'benchmark',
-                             environ)
-    if data is not None:
-        return data
-
-    # If no cached data was found or it was missing any dates then download the
-    # necessary data.
-    logger.info(
-        ('Downloading benchmark data for {symbol!r} '
-            'from {first_date} to {last_date}'),
+         'from {first_date} to {last_date}'),
         symbol=symbol,
         first_date=first_date - trading_day,
         last_date=last_date
@@ -478,11 +486,6 @@ def ensure_treasury_data(symbol, first_date, last_date, now, environ=None):
 
 def _load_cached_data(filename, first_date, last_date, now, resource_name,
                       environ=None):
-    if resource_name == 'benchmark':
-        from_csv = pd.Series.from_csv
-    else:
-        from_csv = pd.DataFrame.from_csv
-
     # Path for the cache.
     path = get_data_filepath(filename, environ)
 
@@ -490,8 +493,11 @@ def _load_cached_data(filename, first_date, last_date, now, resource_name,
     # yet, so don't try to read from 'path'.
     if os.path.exists(path):
         try:
-            data = from_csv(path)
-            data.index = pd.to_datetime(data.index).tz_localize('UTC')
+            data = pd.DataFrame.from_csv(path)
+            if data.empty:
+                raise ValueError("File is empty.")
+            data.index = pd.to_datetime(data.index, infer_datetime_format=True,
+                                        errors='coerce').tz_localize('UTC')
             if has_data_for_dates(data, first_date, last_date):
                 return data
 
@@ -517,7 +523,7 @@ def _load_cached_data(filename, first_date, last_date, now, resource_name,
             )
 
     logger.info(
-        "Cache at {path} does not have data from {start} to {end}.\n",
+        "Cache at {path} does not have data from {start} to {end}.",
         start=first_date,
         end=last_date,
         path=path,

catalyst/data/minute_bars.py

@@ -39,20 +39,21 @@ from catalyst.data._minute_bar_internal import (
 from catalyst.gens.sim_engine import NANOS_IN_MINUTE
 
 from catalyst.data.bar_reader import BarReader, NoDataOnDate
-from catalyst.data.us_equity_pricing import check_uint32_safe
+from catalyst.data.us_equity_pricing import check_uint64_safe
 from catalyst.utils.calendars import get_calendar
 from catalyst.utils.cli import maybe_show_progress
 from catalyst.utils.memoize import lazyval
 
+from catalyst.constants import LOG_LEVEL
 
-logger = logbook.Logger('MinuteBars')
+logger = logbook.Logger('MinuteBars', level=LOG_LEVEL)
 
 US_EQUITIES_MINUTES_PER_DAY = 390
 FUTURES_MINUTES_PER_DAY = 1440
 
 DEFAULT_EXPECTEDLEN = US_EQUITIES_MINUTES_PER_DAY * 252 * 15
 
-OHLC_RATIO = 1000
+OHLC_RATIO = 100000000
 
 
 class BcolzMinuteOverlappingData(Exception):
@@ -114,15 +115,15 @@ def _sid_subdir_path(sid):
 
 
 def convert_cols(cols, scale_factor, sid, invalid_data_behavior):
-    """Adapt OHLCV columns into uint32 columns.
+    """Adapt OHLCV columns into uint64 columns.
 
     Parameters
     ----------
     cols : dict
         A dict mapping each column name (open, high, low, close, volume)
-        to a float column to convert to uint32.
+        to a float column to convert to uint64.
     scale_factor : int
-        Factor to use to scale float values before converting to uint32.
+        Factor to use to scale float values before converting to uint64.
     sid : int
         Sid of the relevant asset, for logging.
     invalid_data_behavior : str
@@ -135,6 +136,7 @@ def convert_cols(cols, scale_factor, sid, invalid_data_behavior):
     scaled_highs = np.nan_to_num(cols['high']) * scale_factor
     scaled_lows = np.nan_to_num(cols['low']) * scale_factor
     scaled_closes = np.nan_to_num(cols['close']) * scale_factor
+    scaled_volumes = np.nan_to_num(cols['volume']) * scale_factor
 
     exclude_mask = np.zeros_like(scaled_opens, dtype=bool)
 
@@ -143,11 +145,12 @@ def convert_cols(cols, scale_factor, sid, invalid_data_behavior):
         ('high', scaled_highs),
         ('low', scaled_lows),
         ('close', scaled_closes),
+        ('volume', scaled_volumes),
     ]:
         max_val = scaled_col.max()
 
         try:
-            check_uint32_safe(max_val, col_name)
+            check_uint64_safe(max_val, col_name)
         except ValueError:
             if invalid_data_behavior == 'raise':
                 raise
@@ -155,20 +158,20 @@ def convert_cols(cols, scale_factor, sid, invalid_data_behavior):
             if invalid_data_behavior == 'warn':
                 logger.warn(
                     'Values for sid={}, col={} contain some too large for '
-                    'uint32 (max={}), filtering them out',
+                    'uint64 (max={}), filtering them out',
                     sid, col_name, max_val,
                 )
 
             # We want to exclude all rows that have an unsafe value in
             # this column.
-            exclude_mask &= (scaled_col >= np.iinfo(np.uint32).max)
+            exclude_mask &= (scaled_col >= np.iinfo(np.uint64).max)
 
     # Convert all cols to uint32.
-    opens = scaled_opens.astype(np.uint32)
-    highs = scaled_highs.astype(np.uint32)
-    lows = scaled_lows.astype(np.uint32)
-    closes = scaled_closes.astype(np.uint32)
-    volumes = cols['volume'].astype(np.uint32)
+    opens = scaled_opens.astype(np.uint64)
+    highs = scaled_highs.astype(np.uint64)
+    lows = scaled_lows.astype(np.uint64)
+    closes = scaled_closes.astype(np.uint64)
+    volumes = scaled_volumes.astype(np.uint64)
 
     # Exclude rows with unsafe values by setting to zero.
     opens[exclude_mask] = 0
@@ -260,14 +263,14 @@ class BcolzMinuteBarMetadata(object):
             )
 
     def __init__(
-        self,
-        default_ohlc_ratio,
-        ohlc_ratios_per_sid,
-        calendar,
-        start_session,
-        end_session,
-        minutes_per_day,
-        version=FORMAT_VERSION,
+            self,
+            default_ohlc_ratio,
+            ohlc_ratios_per_sid,
+            calendar,
+            start_session,
+            end_session,
+            minutes_per_day,
+            version=FORMAT_VERSION,
     ):
         self.calendar = calendar
         self.start_session = start_session
@@ -288,7 +291,7 @@ class BcolzMinuteBarMetadata(object):
         ohlc_ratio : int
             The default ratio by which to multiply the pricing data to
             convert the floats from floats to an integer to fit within
-            the np.uint32. If ohlc_ratios_per_sid is None or does not
+            the np.uint64. If ohlc_ratios_per_sid is None or does not
             contain a mapping for a given sid, this ratio is used.
         ohlc_ratios_per_sid : dict
              A dict mapping each sid in the output to the factor by
@@ -338,12 +341,10 @@ class BcolzMinuteBarMetadata(object):
             'end_session': str(self.end_session.date()),
             # Write these values for backwards compatibility
             'first_trading_day': str(self.start_session.date()),
-            'market_opens': (
-                market_opens.values.astype('datetime64[m]').
-                astype(np.int64).tolist()),
-            'market_closes': (
-                market_closes.values.astype('datetime64[m]').
-                astype(np.int64).tolist()),
+            'market_opens': (market_opens.values.astype('datetime64[m]').
+                             astype(np.int64).tolist()),
+            'market_closes': (market_closes.values.astype('datetime64[m]').
+                              astype(np.int64).tolist()),
         }
         with open(self.metadata_path(rootdir), 'w+') as fp:
             json.dump(metadata, fp)
@@ -372,13 +373,13 @@ class BcolzMinuteBarWriter(object):
         The last trading session in the data set.
     default_ohlc_ratio : int, optional
         The default ratio by which to multiply the pricing data to
-        convert from floats to integers that fit within np.uint32. If
+        convert from floats to integers that fit within np.uint64. If
         ohlc_ratios_per_sid is None or does not contain a mapping for a
-        given sid, this ratio is used. Default is OHLC_RATIO (1000).
+        given sid, this ratio is used. Default is OHLC_RATIO (10^8).
     ohlc_ratios_per_sid : dict, optional
         A dict mapping each sid in the output to the ratio by which to
         multiply the pricing data to convert the floats from floats to
-        an integer to fit within the np.uint32.
+        an integer to fit within the np.uint64.
     expectedlen : int, optional
         The expected length of the dataset, used when creating the initial
         bcolz ctable.
@@ -401,11 +402,9 @@ class BcolzMinuteBarWriter(object):
     Each individual asset's data is stored as a bcolz table with a column for
     each pricing field: (open, high, low, close, volume)
 
-    The open, high, low, and close columns are integers which are 1000 times
+    The open, high, low, close and volume columns are integers which are 10^8 times
     the quoted price, so that the data can represented and stored as an
-    np.uint32, supporting market prices quoted up to the thousands place.
-
-    volume is a np.uint32 with no mutation of the tens place.
+    np.uint64, supporting market prices quoted up to the 1/10^8-th place.
 
     The 'index' for each individual asset are a repeating period of minutes of
     length `minutes_per_day` starting from each market open.
@@ -573,7 +572,7 @@ class BcolzMinuteBarWriter(object):
         if not os.path.exists(sid_containing_dirname):
             # Other sids may have already created the containing directory.
             os.makedirs(sid_containing_dirname)
-        initial_array = np.empty(0, np.uint32)
+        initial_array = np.empty(0, np.uint64)
         table = ctable(
             rootdir=path,
             columns=[
@@ -610,7 +609,7 @@ class BcolzMinuteBarWriter(object):
         minute_offset = len(table) % self._minutes_per_day
         num_to_prepend = numdays * self._minutes_per_day - minute_offset
 
-        prepend_array = np.zeros(num_to_prepend, np.uint32)
+        prepend_array = np.zeros(num_to_prepend, np.uint64)
         # Fill all OHLCV with zeros.
         table.append([prepend_array] * 5)
         table.flush()
@@ -815,11 +814,11 @@ class BcolzMinuteBarWriter(object):
 
         minutes_count = all_minutes_in_window.size
 
-        open_col = np.zeros(minutes_count, dtype=np.uint32)
-        high_col = np.zeros(minutes_count, dtype=np.uint32)
-        low_col = np.zeros(minutes_count, dtype=np.uint32)
-        close_col = np.zeros(minutes_count, dtype=np.uint32)
-        vol_col = np.zeros(minutes_count, dtype=np.uint32)
+        open_col = np.zeros(minutes_count, dtype=np.uint64)
+        high_col = np.zeros(minutes_count, dtype=np.uint64)
+        low_col = np.zeros(minutes_count, dtype=np.uint64)
+        close_col = np.zeros(minutes_count, dtype=np.uint64)
+        vol_col = np.zeros(minutes_count, dtype=np.uint64)
 
         dt_ixs = np.searchsorted(all_minutes_in_window.values,
                                  dts.astype('datetime64[ns]'))
@@ -914,10 +913,10 @@ class BcolzMinuteBarReader(MinuteBarReader):
         )
         self._schedule = self.calendar.schedule[slicer]
         self._market_opens = self._schedule.market_open
-        self._market_open_values = self._market_opens.values.\
+        self._market_open_values = self._market_opens.values. \
             astype('datetime64[m]').astype(np.int64)
         self._market_closes = self._schedule.market_close
-        self._market_close_values = self._market_closes.values.\
+        self._market_close_values = self._market_closes.values. \
             astype('datetime64[m]').astype(np.int64)
 
         self._default_ohlc_inverse = 1.0 / metadata.default_ohlc_ratio
@@ -1125,8 +1124,8 @@ class BcolzMinuteBarReader(MinuteBarReader):
             else:
                 return np.nan
 
-        if field != 'volume':
-            value *= self._ohlc_ratio_inverse_for_sid(sid)
+        # if field != 'volume':
+        value *= self._ohlc_ratio_inverse_for_sid(sid)
         return value
 
     def get_last_traded_dt(self, asset, dt):
@@ -1248,25 +1247,25 @@ class BcolzMinuteBarReader(MinuteBarReader):
             if field != 'volume':
                 out = np.full(shape, np.nan)
             else:
-                out = np.zeros(shape, dtype=np.uint32)
+                out = np.zeros(shape, dtype=np.float64)
 
             for i, sid in enumerate(sids):
                 carray = self._open_minute_file(field, sid)
                 values = carray[start_idx:end_idx + 1]
                 if indices_to_exclude is not None:
                     for excl_start, excl_stop in indices_to_exclude[::-1]:
-                        excl_slice = np.s_[
-                            excl_start - start_idx:excl_stop - start_idx + 1]
+                        excl_slice = np.s_[excl_start - start_idx:excl_stop
+                                           - start_idx + 1]
                         values = np.delete(values, excl_slice)
 
                 where = values != 0
                 # first slice down to len(where) because we might not have
                 # written data for all the minutes requested
-                if field != 'volume':
-                    out[:len(where), i][where] = (
-                        values[where] * self._ohlc_ratio_inverse_for_sid(sid))
-                else:
-                    out[:len(where), i][where] = values[where]
+                # if field != 'volume':
+                out[:len(where), i][where] = (
+                    values[where] * self._ohlc_ratio_inverse_for_sid(sid))
+                # else:
+                #    out[:len(where), i][where] = values[where]
 
             results.append(out)
         return results
@@ -1319,9 +1318,8 @@ class H5MinuteBarUpdateWriter(object):
 
     def __init__(self, path, complevel=None, complib=None):
         self._complevel = complevel if complevel \
-            is not None else self._COMPLEVEL
-        self._complib = complib if complib \
-            is not None else self._COMPLIB
+                            is not None else self._COMPLEVEL
+        self._complib = complib if complib is not None else self._COMPLIB
         self._path = path
 
     def write(self, frames):
@@ -1353,6 +1351,7 @@ class H5MinuteBarUpdateReader(MinuteBarUpdateReader):
     path : str
         The path of the HDF5 file from which to source data.
     """
+
     def __init__(self, path):
         self._panel = pd.read_hdf(path)
 

catalyst/data/resample.py

@@ -156,7 +156,10 @@ class DailyHistoryAggregator(object):
             cache = self._caches[field] = (session, market_open, {})
 
         _, market_open, entries = cache
-        market_open = market_open.tz_localize('UTC')
+        try:
+            market_open = market_open.tz_localize('UTC')
+        except TypeError:
+            market_open = market_open.tz_convert('UTC')
         if dt != market_open:
             prev_dt = dt_value - self._one_min
         else:

catalyst/data/us_equity_pricing.py

@@ -11,6 +11,9 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+
+from __future__ import division  # Python2 req for division of ints yield float
+
 from errno import ENOENT
 from functools import partial
 from os import remove
@@ -80,8 +83,9 @@ from catalyst.utils.cli import (
 from ._equities import _compute_row_slices, _read_bcolz_data
 from ._adjustments import load_adjustments_from_sqlite
 
+from catalyst.constants import LOG_LEVEL
 
-logger = logbook.Logger('UsEquityPricing')
+logger = logbook.Logger('UsEquityPricing', level=LOG_LEVEL)
 
 OHLC = frozenset(['open', 'high', 'low', 'close'])
 OHLCV = frozenset(['open', 'high', 'low', 'close', 'volume'])
@@ -116,6 +120,9 @@ SQLITE_STOCK_DIVIDEND_PAYOUT_COLUMN_DTYPES = {
 UINT32_MAX = iinfo(uint32).max
 UINT64_MAX = iinfo(uint64).max
 
+# Provides 9 decimals resolution. Also affects _equities.pyx L220
+PRICE_ADJUSTMENT_FACTOR = 1000000000
+
 
 def check_uint32_safe(value, colname):
     if value >= UINT32_MAX:
@@ -124,6 +131,7 @@ def check_uint32_safe(value, colname):
             "for uint32" % (value, colname)
         )
 
+
 def check_uint64_safe(value, colname):
     if value >= UINT64_MAX:
         raise ValueError(
@@ -316,8 +324,8 @@ class BcolzDailyBarWriter(object):
         # Maps column name -> output carray.
         columns = {
             k: carray(array([], dtype=uint64))
-               if k in OHLCV
-               else carray(array([], dtype=uint32))
+            if k in OHLCV
+            else carray(array([], dtype=uint32))
             for k in US_EQUITY_PRICING_BCOLZ_COLUMNS
         }
 
@@ -433,11 +441,13 @@ class BcolzDailyBarWriter(object):
             return raw_data
 
         winsorise_uint64(raw_data, invalid_data_behavior, 'volume', *OHLC)
-        processed = (raw_data[list(OHLC)] * 1000000).astype('uint64')
+        processed = (raw_data[list(OHLC)]
+                     * PRICE_ADJUSTMENT_FACTOR).astype('uint64')
         dates = raw_data.index.values.astype('datetime64[s]')
         check_uint32_safe(dates.max().view(np.int64), 'day')
         processed['day'] = dates.astype('uint32')
-        processed['volume'] = raw_data.volume.astype('uint64')
+        processed['volume'] = (raw_data.volume
+                               * PRICE_ADJUSTMENT_FACTOR).astype('uint64')
         return ctable.fromdataframe(processed)
 
 
@@ -490,9 +500,8 @@ class BcolzDailyBarReader(SessionBarReader):
 
     The data in these columns is interpreted as follows:
 
-    - Price columns ('open', 'high', 'low', 'close') are interpreted as 1000 *
-      as-traded dollar value.
-    - Volume is interpreted as as-traded volume.
+    - Price columns ('open', 'high', 'low', 'close') and Volume are interpreted
+      as 10^9 * as-traded dollar value.
     - Day is interpreted as seconds since midnight UTC, Jan 1, 1970.
     - Id is the asset id of the row.
 
@@ -519,7 +528,6 @@ class BcolzDailyBarReader(SessionBarReader):
         # Need to test keeping the entire array in memory for the course of a
         # process first.
         self._spot_cols = {}
-        self.PRICE_ADJUSTMENT_FACTOR = 0.001
         self._read_all_threshold = read_all_threshold
 
     @lazyval
@@ -759,13 +767,10 @@ class BcolzDailyBarReader(SessionBarReader):
         """
         ix = self.sid_day_index(sid, dt)
         price = self._spot_col(field)[ix]
-        if field != 'volume':
-            if price == 0:
-                return nan
-            else:
-                return price * 0.001
+        if field != 'volume' and price == 0:
+            return nan
         else:
-            return price
+            return price / PRICE_ADJUSTMENT_FACTOR
 
 
 class PanelBarReader(SessionBarReader):

catalyst/examples/README.rst

@@ -0,0 +1,3 @@
+An overview of most of the trading strategies in this folder can be found in the 
+`Examples Algorithms <https://enigmampc.github.io/catalyst/example-algos.html>`_
+section of our documentation website.

catalyst/examples/arbitrage_with_interface.py

@@ -0,0 +1,282 @@
+from logbook import Logger
+
+from catalyst.api import (
+    record,
+    order,
+    symbol,
+    get_open_orders
+)
+from catalyst.exchange.stats_utils import get_pretty_stats
+from catalyst.utils.run_algo import run_algorithm
+
+algo_namespace = 'arbitrage_eth_btc'
+log = Logger(algo_namespace)
+
+
+def initialize(context):
+    log.info('initializing arbitrage algorithm')
+
+    # The context contains a new "exchanges" attribute which is a dictionary
+    # of exchange objects by exchange name. This allow easy access to the
+    # exchanges.
+    context.buying_exchange = context.exchanges['poloniex']
+    context.selling_exchange = context.exchanges['bitfinex']
+
+    context.trading_pair_symbol = 'eth_btc'
+    context.trading_pairs = dict()
+
+    # Note the second parameter of the symbol() method
+    # Passing the exchange name here returns a TradingPair object including
+    # the exchange information. This allow all other operations using
+    # the TradingPair to target the correct exchange.
+    context.trading_pairs[context.buying_exchange] = \
+        symbol('eth_btc', context.buying_exchange.name)
+
+    context.trading_pairs[context.selling_exchange] = \
+        symbol(context.trading_pair_symbol, context.selling_exchange.name)
+
+    context.entry_points = [
+        dict(gap=0.03, amount=0.05),
+        dict(gap=0.04, amount=0.1),
+        dict(gap=0.05, amount=0.5),
+    ]
+    context.exit_points = [
+        dict(gap=-0.02, amount=0.5),
+    ]
+
+    context.SLIPPAGE_ALLOWED = 0.02
+    pass
+
+
+def place_orders(context, amount, buying_price, selling_price, action):
+    """
+    This method will always place two orders of the same amount to keep
+    the currency position the same as it moves between the two exchanges.
+
+    :param context: TradingAlgorithm
+    :param amount: float
+        The trading pair amount to trade on both exchanges.
+    :param buying_price: float
+        The current trading pair price on the buying exchange.
+    :param selling_price: float
+        The current trading pair price on the selling exchange.
+    :param action: string
+        "enter": buys on the buying exchange and sells on the selling exchange
+        "exit": buys on the selling exchange and sells on the buying exchange
+
+    :return:
+    """
+    if action == 'enter':
+        enter_exchange = context.buying_exchange
+        entry_price = buying_price
+
+        exit_exchange = context.selling_exchange
+        exit_price = selling_price
+
+    elif action == 'exit':
+        enter_exchange = context.selling_exchange
+        entry_price = selling_price
+
+        exit_exchange = context.buying_exchange
+        exit_price = buying_price
+
+    else:
+        raise ValueError('invalid order action')
+
+    quote_currency = enter_exchange.quote_currency
+    quote_currency_amount = enter_exchange.portfolio.cash
+
+    exit_balances = exit_exchange.get_balances()
+    exit_currency = context.trading_pairs[
+        context.selling_exchange].quote_currency
+
+    if exit_currency in exit_balances:
+        quote_currency_amount = exit_balances[exit_currency]
+    else:
+        log.warn(
+            'the selling exchange {exchange_name} does not hold '
+            'currency {currency}'.format(
+                exchange_name=exit_exchange.name,
+                currency=exit_currency
+            )
+        )
+        return
+
+    if quote_currency_amount < (amount * entry_price):
+        adj_amount = quote_currency_amount / entry_price
+        log.warn(
+            'not enough {quote_currency} ({quote_currency_amount}) to buy '
+            '{amount}, adjusting the amount to {adj_amount}'.format(
+                quote_currency=quote_currency,
+                quote_currency_amount=quote_currency_amount,
+                amount=amount,
+                adj_amount=adj_amount
+            )
+        )
+        amount = adj_amount
+
+    elif quote_currency_amount < amount:
+        log.warn(
+            'not enough {currency} ({currency_amount}) to sell '
+            '{amount}, aborting'.format(
+                currency=exit_currency,
+                currency_amount=quote_currency_amount,
+                amount=amount
+            )
+        )
+        return
+
+    adj_buy_price = entry_price * (1 + context.SLIPPAGE_ALLOWED)
+    log.info(
+        'buying {amount} {trading_pair} on {exchange_name} with price '
+        'limit {limit_price}'.format(
+            amount=amount,
+            trading_pair=context.trading_pair_symbol,
+            exchange_name=enter_exchange.name,
+            limit_price=adj_buy_price
+        )
+    )
+    order(
+        asset=context.trading_pairs[enter_exchange],
+        amount=amount,
+        limit_price=adj_buy_price
+    )
+
+    adj_sell_price = exit_price * (1 - context.SLIPPAGE_ALLOWED)
+    log.info(
+        'selling {amount} {trading_pair} on {exchange_name} with price '
+        'limit {limit_price}'.format(
+            amount=-amount,
+            trading_pair=context.trading_pair_symbol,
+            exchange_name=exit_exchange.name,
+            limit_price=adj_sell_price
+        )
+    )
+    order(
+        asset=context.trading_pairs[exit_exchange],
+        amount=-amount,
+        limit_price=adj_sell_price
+    )
+    pass
+
+
+def handle_data(context, data):
+    log.info('handling bar {}'.format(data.current_dt))
+
+    buying_price = data.current(
+        context.trading_pairs[context.buying_exchange], 'price')
+
+    log.info('price on buying exchange {exchange}: {price}'.format(
+        exchange=context.buying_exchange.name.upper(),
+        price=buying_price,
+    ))
+
+    selling_price = data.current(
+        context.trading_pairs[context.selling_exchange], 'price')
+
+    log.info('price on selling exchange {exchange}: {price}'.format(
+        exchange=context.selling_exchange.name.upper(),
+        price=selling_price,
+    ))
+
+    # If for example,
+    #   selling price = 50
+    #   buying price = 25
+    #   expected gap = 1
+
+    # If follows that,
+    #   selling price - buying price / buying price
+    #   50 - 25 / 25 = 1
+    gap = (selling_price - buying_price) / buying_price
+    log.info(
+        'the price gap: {gap} ({gap_percent}%)'.format(
+            gap=gap,
+            gap_percent=gap * 100
+        )
+    )
+    record(buying_price=buying_price, selling_price=selling_price, gap=gap)
+
+    # Waiting for orders to close before initiating new ones
+    for exchange in context.trading_pairs:
+        asset = context.trading_pairs[exchange]
+
+        orders = get_open_orders(asset)
+        if orders:
+            log.info(
+                'found {order_count} open orders on {exchange_name} '
+                'skipping bar until all open orders execute'.format(
+                    order_count=len(orders),
+                    exchange_name=exchange.name
+                )
+            )
+            return
+
+    # Consider the least ambitious entry point first
+    # Override of wider gap is found
+    entry_points = sorted(
+        context.entry_points,
+        key=lambda point: point['gap'],
+    )
+
+    buy_amount = None
+    for entry_point in entry_points:
+        if gap > entry_point['gap']:
+            buy_amount = entry_point['amount']
+
+    if buy_amount:
+        log.info('found buy trigger for amount: {}'.format(buy_amount))
+        place_orders(
+            context=context,
+            amount=buy_amount,
+            buying_price=buying_price,
+            selling_price=selling_price,
+            action='enter'
+        )
+
+    else:
+        # Consider the narrowest exit gap first
+        # Override of wider gap is found
+        exit_points = sorted(
+            context.exit_points,
+            key=lambda point: point['gap'],
+            reverse=True
+        )
+
+        sell_amount = None
+        for exit_point in exit_points:
+            if gap < exit_point['gap']:
+                sell_amount = exit_point['amount']
+
+        if sell_amount:
+            log.info('found sell trigger for amount: {}'.format(sell_amount))
+            place_orders(
+                context=context,
+                amount=sell_amount,
+                buying_price=buying_price,
+                selling_price=selling_price,
+                action='exit'
+            )
+
+
+def analyze(context, stats):
+    log.info('the daily stats:\n{}'.format(get_pretty_stats(stats)))
+    pass
+
+
+if __name__ == '__main__':
+    # The execution mode: backtest or live
+    MODE = 'live'
+    if MODE == 'live':
+        run_algorithm(
+            capital_base=0.1,
+            initialize=initialize,
+            handle_data=handle_data,
+            analyze=analyze,
+            exchange_name='poloniex,bitfinex',
+            live=True,
+            algo_namespace=algo_namespace,
+            base_currency='btc',
+            live_graph=False,
+            simulate_orders=True,
+            stats_output=None,
+        )

catalyst/examples/buy_and_hodl.py

@@ -14,36 +14,28 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+import pandas as pd
+import matplotlib.pyplot as plt
 
-from catalyst.api import (
-    order_target_value,
-    symbol,
-    record,
-    cancel_order,
-    get_open_orders,
-)
+from catalyst import run_algorithm
+from catalyst.api import (order_target_value, symbol, record,
+                          cancel_order, get_open_orders, )
 
 
 def initialize(context):
-    context.ASSET_NAME = 'USDT_BTC'
+    context.ASSET_NAME = 'btc_usd'
     context.TARGET_HODL_RATIO = 0.8
     context.RESERVE_RATIO = 1.0 - context.TARGET_HODL_RATIO
 
-    # For all trading pairs in the poloniex bundle, the default denomination
-    # currently supported by Catalyst is 1/1000th of a full coin. Use this
-    # constant to scale the price of up to that of a full coin if desired.
-    context.TICK_SIZE = 1000.0
-
     context.is_buying = True
     context.asset = symbol(context.ASSET_NAME)
 
     context.i = 0
 
+
 def handle_data(context, data):
     context.i += 1
 
-    print 'i:', context.i
-
     starting_cash = context.portfolio.starting_cash
     target_hodl_value = context.TARGET_HODL_RATIO * starting_cash
     reserve_value = context.RESERVE_RATIO * starting_cash
@@ -52,60 +44,63 @@ def handle_data(context, data):
     orders = get_open_orders(context.asset) or []
     for order in orders:
         cancel_order(order)
-    
+
     # Stop buying after passing the reserve threshold
     cash = context.portfolio.cash
     if cash <= reserve_value:
         context.is_buying = False
 
     # Retrieve current asset price from pricing data
-    price = data[context.asset].price
+    price = data.current(context.asset, 'price')
 
     # Check if still buying and could (approximately) afford another purchase
     if context.is_buying and cash > price:
+        print('buying')
         # Place order to make position in asset equal to target_hodl_value
         order_target_value(
             context.asset,
             target_hodl_value,
-            limit_price=price*1.1,
-            stop_price=price*0.9,
+            limit_price=price * 1.1,
         )
 
     record(
         price=price,
+        volume=data.current(context.asset, 'volume'),
         cash=cash,
         starting_cash=context.portfolio.starting_cash,
         leverage=context.account.leverage,
     )
 
-def analyze(context=None, results=None):
-    import matplotlib.pyplot as plt
-    # Plot the portfolio and asset data.
-    ax1 = plt.subplot(511)
-    results[['portfolio_value']].plot(ax=ax1)
-    ax1.set_ylabel('Portfolio Value (USD)')
 
-    ax2 = plt.subplot(512, sharex=ax1)
-    ax2.set_ylabel('{asset} (USD)'.format(asset=context.ASSET_NAME))
-    (context.TICK_SIZE * results[['price']]).plot(ax=ax2)
+def analyze(context=None, results=None):
+
+    # Plot the portfolio and asset data.
+    ax1 = plt.subplot(611)
+    results[['portfolio_value']].plot(ax=ax1)
+    ax1.set_ylabel('Portfolio\nValue\n(USD)')
+
+    ax2 = plt.subplot(612, sharex=ax1)
+    ax2.set_ylabel('{asset}\n(USD)'.format(asset=context.ASSET_NAME))
+    results[['price']].plot(ax=ax2)
 
     trans = results.ix[[t != [] for t in results.transactions]]
     buys = trans.ix[
         [t[0]['amount'] > 0 for t in trans.transactions]
     ]
-    ax2.plot(
-        buys.index,
-        context.TICK_SIZE * results.price[buys.index],
-        '^',
-        markersize=10,
-        color='g',
+    ax2.scatter(
+        buys.index.to_pydatetime(),
+        results.price[buys.index],
+        marker='^',
+        s=100,
+        c='g',
+        label=''
     )
 
-    ax3 = plt.subplot(513, sharex=ax1)
+    ax3 = plt.subplot(613, sharex=ax1)
     results[['leverage', 'alpha', 'beta']].plot(ax=ax3)
     ax3.set_ylabel('Leverage ')
 
-    ax4 = plt.subplot(514, sharex=ax1)
+    ax4 = plt.subplot(614, sharex=ax1)
     results[['starting_cash', 'cash']].plot(ax=ax4)
     ax4.set_ylabel('Cash (USD)')
 
@@ -119,16 +114,35 @@ def analyze(context=None, results=None):
         'benchmark_period_return',
     ]]
 
-    ax5 = plt.subplot(515, sharex=ax1)
+    ax5 = plt.subplot(615, sharex=ax1)
     results[[
         'treasury',
         'algorithm',
         'benchmark',
     ]].plot(ax=ax5)
-    ax5.set_ylabel('Percent Change')
+    ax5.set_ylabel('Percent\nChange')
+
+    ax6 = plt.subplot(616, sharex=ax1)
+    results[['volume']].plot(ax=ax6)
+    ax6.set_ylabel('Volume')
 
     plt.legend(loc=3)
 
     # Show the plot.
     plt.gcf().set_size_inches(18, 8)
     plt.show()
+
+
+if __name__ == '__main__':
+    run_algorithm(
+        capital_base=10000,
+        data_frequency='daily',
+        initialize=initialize,
+        handle_data=handle_data,
+        analyze=analyze,
+        exchange_name='bitfinex',
+        algo_namespace='buy_and_hodl',
+        base_currency='usd',
+        start=pd.to_datetime('2015-03-01', utc=True),
+        end=pd.to_datetime('2017-10-31', utc=True),
+    )

catalyst/examples/buy_btc_simple.py

@@ -0,0 +1,49 @@
+'''
+    This is a very simple example referenced in the beginner's tutorial:
+    https://enigmampc.github.io/catalyst/beginner-tutorial.html
+
+    Run this example, by executing the following from your terminal:
+      catalyst ingest-exchange -x bitfinex -f daily -i btc_usdt
+      catalyst run -f buy_btc_simple.py -x bitfinex --start 2016-1-1 \
+        --end 2017-9-30 -o buy_btc_simple_out.pickle
+
+    If you want to run this code using another exchange, make sure that
+    the asset is available on that exchange. For example, if you were to run
+    it for exchange Poloniex, you would need to edit the following line:
+
+        context.asset = symbol('btc_usdt')     # note 'usdt' instead of 'usd'
+
+    and specify exchange poloniex as follows:
+    catalyst ingest-exchange -x poloniex -f daily -i btc_usdt
+    catalyst run -f buy_btc_simple.py -x poloniex --start 2016-1-1 \
+        --end 2017-9-30 -o buy_btc_simple_out.pickle
+
+    To see which assets are available on each exchange, visit:
+    https://www.enigma.co/catalyst/status
+'''
+from catalyst import run_algorithm
+from catalyst.api import order, record, symbol
+import pandas as pd
+
+
+def initialize(context):
+    context.asset = symbol('btc_usd')
+
+
+def handle_data(context, data):
+    order(context.asset, 1)
+    record(btc=data.current(context.asset, 'price'))
+
+
+if __name__ == '__main__':
+    run_algorithm(
+        capital_base=10000,
+        data_frequency='daily',
+        initialize=initialize,
+        handle_data=handle_data,
+        exchange_name='bitfinex',
+        algo_namespace='buy_and_hodl',
+        base_currency='usd',
+        start=pd.to_datetime('2015-03-01', utc=True),
+        end=pd.to_datetime('2017-10-31', utc=True),
+    )

catalyst/examples/buy_low_sell_high.py

@@ -0,0 +1,176 @@
+'''
+This algorithm requires an additional library (ta-lib) beyond those
+required by catalyst. Install it first by running:
+$ pip install TA-Lib
+
+If you get build errors like:
+    "fatal error: ta-lib/ta_libc.h: No such file or directory"
+it typically means that it can't find the underlying TA-Lib library and it
+needs to be installed. See https://mrjbq7.github.io/ta-lib/install.html for
+instructions on how to install the required dependencies.
+'''
+
+import talib
+from logbook import Logger
+
+from catalyst import run_algorithm
+from catalyst.api import (
+    order,
+    order_target_percent,
+    symbol,
+    record,
+    get_open_orders,
+)
+from catalyst.exchange.stats_utils import get_pretty_stats
+import pandas as pd
+
+algo_namespace = 'buy_low_sell_high_xrp'
+log = Logger(algo_namespace)
+
+
+def initialize(context):
+    log.info('initializing algo')
+    context.ASSET_NAME = 'XRP_USDT'
+    context.asset = symbol(context.ASSET_NAME)
+
+    context.TARGET_POSITIONS = 5000
+    context.PROFIT_TARGET = 0.1
+    context.SLIPPAGE_ALLOWED = 0.05
+
+    context.retry_check_open_orders = 10
+    context.retry_update_portfolio = 10
+    context.retry_order = 5
+
+    context.swallow_errors = True
+
+    context.errors = []
+    pass
+
+
+def _handle_data(context, data):
+    prices = data.history(
+        context.asset,
+        fields='price',
+        bar_count=20,
+        frequency='15m'
+    )
+
+    rsi = talib.RSI(prices.values, timeperiod=14)[-1]
+    log.info('got rsi: {}'.format(rsi))
+
+    # Buying more when RSI is low, this should lower our cost basis
+    if rsi <= 30:
+        buy_increment = 50
+    elif rsi <= 40:
+        buy_increment = 20
+    elif rsi <= 70:
+        buy_increment = 5
+    else:
+        buy_increment = None
+
+    cash = context.portfolio.cash
+    log.info('base currency available: {cash}'.format(cash=cash))
+
+    price = data.current(context.asset, 'price')
+    log.info('got price {price}'.format(price=price))
+
+    record(
+        price=price,
+        rsi=rsi,
+    )
+
+    orders = get_open_orders(context.asset)
+    if orders:
+        log.info('skipping bar until all open orders execute')
+        return
+
+    is_buy = False
+    cost_basis = None
+    if context.asset in context.portfolio.positions:
+        position = context.portfolio.positions[context.asset]
+
+        cost_basis = position.cost_basis
+        log.info(
+            'found {amount} positions with cost basis {cost_basis}'.format(
+                amount=position.amount,
+                cost_basis=cost_basis
+            )
+        )
+
+        if position.amount >= context.TARGET_POSITIONS:
+            log.info('reached positions target: {}'.format(position.amount))
+            return
+
+        if price < cost_basis:
+            is_buy = True
+        elif (position.amount > 0
+              and price > cost_basis * (1 + context.PROFIT_TARGET)):
+            profit = (price * position.amount) - (cost_basis * position.amount)
+            log.info('closing position, taking profit: {}'.format(profit))
+            order_target_percent(
+                asset=context.asset,
+                target=0,
+                limit_price=price * (1 - context.SLIPPAGE_ALLOWED),
+            )
+        else:
+            log.info('no buy or sell opportunity found')
+    else:
+        is_buy = True
+
+    if is_buy:
+        if buy_increment is None:
+            log.info('the rsi is too high to consider buying {}'.format(rsi))
+            return
+
+        if price * buy_increment > cash:
+            log.info('not enough base currency to consider buying')
+            return
+
+        log.info(
+            'buying position cheaper than cost basis {} < {}'.format(
+                price,
+                cost_basis
+            )
+        )
+        order(
+            asset=context.asset,
+            amount=buy_increment,
+            limit_price=price * (1 + context.SLIPPAGE_ALLOWED)
+        )
+
+
+def handle_data(context, data):
+    log.info('handling bar {}'.format(data.current_dt))
+    try:
+        _handle_data(context, data)
+    except Exception as e:
+        log.warn('aborting the bar on error {}'.format(e))
+        context.errors.append(e)
+
+    log.info('completed bar {}, total execution errors {}'.format(
+        data.current_dt,
+        len(context.errors)
+    ))
+
+    if len(context.errors) > 0:
+        log.info('the errors:\n{}'.format(context.errors))
+
+
+def analyze(context, stats):
+    log.info('the daily stats:\n{}'.format(get_pretty_stats(stats)))
+    pass
+
+
+if __name__ == '__main__':
+    run_algorithm(
+        capital_base=10000,
+        data_frequency='daily',
+        initialize=initialize,
+        handle_data=handle_data,
+        analyze=analyze,
+        exchange_name='poloniex',
+        algo_namespace='buy_and_hodl',
+        base_currency='usd',
+        start=pd.to_datetime('2015-03-01', utc=True),
+        end=pd.to_datetime('2017-10-31', utc=True),
+    )

catalyst/examples/buy_low_sell_high_live.py

@@ -0,0 +1,160 @@
+import talib
+from logbook import Logger
+
+import pandas as pd
+from catalyst.api import (
+    order,
+    order_target_percent,
+    symbol,
+    record,
+    get_open_orders,
+)
+from catalyst.exchange.stats_utils import get_pretty_stats
+from catalyst.utils.run_algo import run_algorithm
+
+algo_namespace = 'buy_the_dip_live'
+log = Logger('buy low sell high')
+
+
+def initialize(context):
+    log.info('initializing algo')
+    context.ASSET_NAME = 'btc_usdt'
+    context.asset = symbol(context.ASSET_NAME)
+
+    context.TARGET_POSITIONS = 30
+    context.PROFIT_TARGET = 0.1
+    context.SLIPPAGE_ALLOWED = 0.02
+
+    context.retry_check_open_orders = 10
+    context.retry_update_portfolio = 10
+    context.retry_order = 5
+
+    context.errors = []
+    pass
+
+
+def _handle_data(context, data):
+    price = data.current(context.asset, 'price')
+    log.info('got price {price}'.format(price=price))
+
+    prices = data.history(
+        context.asset,
+        fields='price',
+        bar_count=20,
+        frequency='1D'
+    )
+    rsi = talib.RSI(prices.values, timeperiod=14)[-1]
+    log.info('got rsi: {}'.format(rsi))
+
+    # Buying more when RSI is low, this should lower our cost basis
+    if rsi <= 30:
+        buy_increment = 1
+    elif rsi <= 40:
+        buy_increment = 0.5
+    elif rsi <= 70:
+        buy_increment = 0.2
+    else:
+        buy_increment = 0.1
+
+    cash = context.portfolio.cash
+    log.info('base currency available: {cash}'.format(cash=cash))
+
+    record(
+        price=price,
+        rsi=rsi,
+    )
+
+    orders = get_open_orders(context.asset)
+    if orders:
+        log.info('skipping bar until all open orders execute')
+        return
+
+    is_buy = False
+    cost_basis = None
+    if context.asset in context.portfolio.positions:
+        position = context.portfolio.positions[context.asset]
+
+        cost_basis = position.cost_basis
+        log.info(
+            'found {amount} positions with cost basis {cost_basis}'.format(
+                amount=position.amount,
+                cost_basis=cost_basis
+            )
+        )
+
+        if position.amount >= context.TARGET_POSITIONS:
+            log.info('reached positions target: {}'.format(position.amount))
+            return
+
+        if price < cost_basis:
+            is_buy = True
+        elif (position.amount > 0
+              and price > cost_basis * (1 + context.PROFIT_TARGET)):
+            profit = (price * position.amount) - (cost_basis * position.amount)
+            log.info('closing position, taking profit: {}'.format(profit))
+            order_target_percent(
+                asset=context.asset,
+                target=0,
+                limit_price=price * (1 - context.SLIPPAGE_ALLOWED),
+            )
+        else:
+            log.info('no buy or sell opportunity found')
+    else:
+        is_buy = True
+
+    if is_buy:
+        if buy_increment is None:
+            log.info('the rsi is too high to consider buying {}'.format(rsi))
+            return
+
+        if price * buy_increment > cash:
+            log.info('not enough base currency to consider buying')
+            return
+
+        log.info(
+            'buying position cheaper than cost basis {} < {}'.format(
+                price,
+                cost_basis
+            )
+        )
+        order(
+            asset=context.asset,
+            amount=buy_increment,
+            limit_price=price * (1 + context.SLIPPAGE_ALLOWED)
+        )
+
+
+def handle_data(context, data):
+    log.info('handling bar {}'.format(data.current_dt))
+    # try:
+    _handle_data(context, data)
+    # except Exception as e:
+    #     log.warn('aborting the bar on error {}'.format(e))
+    #     context.errors.append(e)
+
+    log.info('completed bar {}, total execution errors {}'.format(
+        data.current_dt,
+        len(context.errors)
+    ))
+
+    if len(context.errors) > 0:
+        log.info('the errors:\n{}'.format(context.errors))
+
+
+def analyze(context, stats):
+    log.info('the daily stats:\n{}'.format(get_pretty_stats(stats)))
+    pass
+
+
+if __name__ == '__main__':
+    run_algorithm(
+        capital_base=0.001,
+        initialize=initialize,
+        handle_data=handle_data,
+        analyze=analyze,
+        exchange_name='binance',
+        live=True,
+        algo_namespace=algo_namespace,
+        base_currency='btc',
+        simulate_orders=True,
+    )

catalyst/examples/dual_moving_average.py

@@ -0,0 +1,162 @@
+import numpy as np
+import pandas as pd
+from logbook import Logger
+import matplotlib.pyplot as plt
+
+from catalyst import run_algorithm
+from catalyst.api import (record, symbol, order_target_percent,
+                          get_open_orders)
+from catalyst.exchange.stats_utils import extract_transactions
+
+NAMESPACE = 'dual_moving_average'
+log = Logger(NAMESPACE)
+
+
+def initialize(context):
+    context.i = 0
+    context.asset = symbol('ltc_usd')
+    context.base_price = None
+
+
+def handle_data(context, data):
+    # define the windows for the moving averages
+    short_window = 50
+    long_window = 200
+
+    # Skip as many bars as long_window to properly compute the average
+    context.i += 1
+    if context.i < long_window:
+        return
+
+    # Compute moving averages calling data.history() for each
+    # moving average with the appropriate parameters. We choose to use
+    # minute bars for this simulation -> freq="1m"
+    # Returns a pandas dataframe.
+    short_mavg = data.history(context.asset,
+                              'price',
+                              bar_count=short_window,
+                              frequency="1m",
+                              ).mean()
+    long_mavg = data.history(context.asset,
+                             'price',
+                             bar_count=long_window,
+                             frequency="1m",
+                             ).mean()
+
+    # Let's keep the price of our asset in a more handy variable
+    price = data.current(context.asset, 'price')
+
+    # If base_price is not set, we use the current value. This is the
+    # price at the first bar which we reference to calculate price_change.
+    if context.base_price is None:
+        context.base_price = price
+    price_change = (price - context.base_price) / context.base_price
+
+    # Save values for later inspection
+    record(price=price,
+           cash=context.portfolio.cash,
+           price_change=price_change,
+           short_mavg=short_mavg,
+           long_mavg=long_mavg)
+
+    # Since we are using limit orders, some orders may not execute immediately
+    # we wait until all orders are executed before considering more trades.
+    orders = get_open_orders(context.asset)
+    if len(orders) > 0:
+        return
+
+    # Exit if we cannot trade
+    if not data.can_trade(context.asset):
+        return
+
+    # We check what's our position on our portfolio and trade accordingly
+    pos_amount = context.portfolio.positions[context.asset].amount
+
+    # Trading logic
+    if short_mavg > long_mavg and pos_amount == 0:
+        # we buy 100% of our portfolio for this asset
+        order_target_percent(context.asset, 1)
+    elif short_mavg < long_mavg and pos_amount > 0:
+        # we sell all our positions for this asset
+        order_target_percent(context.asset, 0)
+
+
+def analyze(context, perf):
+
+    # Get the base_currency that was passed as a parameter to the simulation
+    base_currency = context.exchanges.values()[0].base_currency.upper()
+
+    # First chart: Plot portfolio value using base_currency
+    ax1 = plt.subplot(411)
+    perf.loc[:, ['portfolio_value']].plot(ax=ax1)
+    ax1.legend_.remove()
+    ax1.set_ylabel('Portfolio Value\n({})'.format(base_currency))
+    start, end = ax1.get_ylim()
+    ax1.yaxis.set_ticks(np.arange(start, end, (end-start)/5))
+
+    # Second chart: Plot asset price, moving averages and buys/sells
+    ax2 = plt.subplot(412, sharex=ax1)
+    perf.loc[:, ['price', 'short_mavg', 'long_mavg']].plot(
+        ax=ax2,
+        label='Price')
+    ax2.legend_.remove()
+    ax2.set_ylabel('{asset}\n({base})'.format(
+        asset=context.asset.symbol,
+        base=base_currency
+        ))
+    start, end = ax2.get_ylim()
+    ax2.yaxis.set_ticks(np.arange(start, end, (end-start)/5))
+
+    transaction_df = extract_transactions(perf)
+    if not transaction_df.empty:
+        buy_df = transaction_df[transaction_df['amount'] > 0]
+        sell_df = transaction_df[transaction_df['amount'] < 0]
+        ax2.scatter(
+            buy_df.index.to_pydatetime(),
+            perf.loc[buy_df.index, 'price'],
+            marker='^',
+            s=100,
+            c='green',
+            label=''
+        )
+        ax2.scatter(
+            sell_df.index.to_pydatetime(),
+            perf.loc[sell_df.index, 'price'],
+            marker='v',
+            s=100,
+            c='red',
+            label=''
+        )
+
+    # Third chart: Compare percentage change between our portfolio
+    # and the price of the asset
+    ax3 = plt.subplot(413, sharex=ax1)
+    perf.loc[:, ['algorithm_period_return', 'price_change']].plot(ax=ax3)
+    ax3.legend_.remove()
+    ax3.set_ylabel('Percent Change')
+    start, end = ax3.get_ylim()
+    ax3.yaxis.set_ticks(np.arange(start, end, (end-start)/5))
+
+    # Fourth chart: Plot our cash
+    ax4 = plt.subplot(414, sharex=ax1)
+    perf.cash.plot(ax=ax4)
+    ax4.set_ylabel('Cash\n({})'.format(base_currency))
+    start, end = ax4.get_ylim()
+    ax4.yaxis.set_ticks(np.arange(0, end, end/5))
+
+    plt.show()
+
+
+if __name__ == '__main__':
+    run_algorithm(
+            capital_base=1000,
+            data_frequency='minute',
+            initialize=initialize,
+            handle_data=handle_data,
+            analyze=analyze,
+            exchange_name='bitfinex',
+            algo_namespace=NAMESPACE,
+            base_currency='usd',
+            start=pd.to_datetime('2017-9-22', utc=True),
+            end=pd.to_datetime('2017-9-23', utc=True),
+        )

catalyst/examples/dual_vwap.py

@@ -1,188 +0,0 @@
-#!/usr/bin/env python
-#
-# Copyright 2017 Enigma MPC, Inc.
-# Copyright 2014 Quantopian, Inc.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from catalyst.api import (
-    order_target_percent,
-    record,
-    symbol,
-    get_open_orders,
-    set_max_leverage,
-    schedule_function,
-    date_rules,
-    attach_pipeline,
-    pipeline_output,
-)
-
-from catalyst.pipeline import Pipeline
-from catalyst.pipeline.data import CryptoPricing
-from catalyst.pipeline.factors.crypto import VWAP
-
-
-def initialize(context):
-    context.ASSET_NAME = 'USDT_BTC'
-    context.TARGET_INVESTMENT_RATIO = 0.8
-    context.SHORT_WINDOW = 30
-    context.LONG_WINDOW = 100
-
-    # For all trading pairs in the poloniex bundle, the default denomination
-    # currently supported by Catalyst is 1/1000th of a full coin. Use this
-    # constant to scale the price of up to that of a full coin if desired.
-    context.TICK_SIZE = 1000.0
-
-    context.i = 0
-    context.asset = symbol(context.ASSET_NAME)
-
-    set_max_leverage(1.0)
-
-    attach_pipeline(make_pipeline(context), 'vwap_pipeline')
-
-    schedule_function(
-        rebalance,
-        time_rules=times_rules.every_minute(),
-    )
-
-
-def before_trading_start(context, data):
-    context.pipeline_data = pipeline_output('vwap_pipeline')
-
-def make_pipeline(context):
-    return Pipeline(
-        columns={
-            'price': CryptoPricing.open.latest,
-            'volume': CryptoPricing.volume.latest,
-            'short_mavg': VWAP(window_length=context.SHORT_WINDOW),
-            'long_mavg': VWAP(window_length=context.LONG_WINDOW),
-        }
-    )
-
-def rebalance(context, data):
-    context.i += 1
-
-    # skip first LONG_WINDOW bars to fill windows
-    if context.i < context.LONG_WINDOW:
-        return
-
-    # get pipeline data for asset of interest
-    pipeline_data = context.pipeline_data
-    pipeline_data = pipeline_data[pipeline_data.index == context.asset].iloc[0]
-
-    # retrieve long and short moving averages from pipeline
-    short_mavg = pipeline_data.short_mavg
-    long_mavg = pipeline_data.long_mavg
-    price = pipeline_data.price
-    volume = pipeline_data.volume
-
-    # check that order has not already been placed
-    open_orders = get_open_orders()
-    if context.asset not in open_orders:
-        # check that the asset of interest can currently be traded
-        if data.can_trade(context.asset):
-            # adjust portfolio based on comparison of long and short vwap
-            if short_mavg > long_mavg:
-                order_target_percent(
-                    context.asset,
-                    context.TARGET_INVESTMENT_RATIO,
-                )
-            elif short_mavg < long_mavg:
-                order_target_percent(
-                    context.asset,
-                    0.0,
-                )
-
-    record(
-        price=price,
-        cash=context.portfolio.cash,
-        leverage=context.account.leverage,
-        short_mavg=short_mavg,
-        long_mavg=long_mavg,
-        volume=volume,
-    )
-    
-
-
-def analyze(context=None, results=None):
-    import matplotlib.pyplot as plt
-
-    # Plot the portfolio and asset data.
-    ax1 = plt.subplot(611)
-    results[['portfolio_value']].plot(ax=ax1)
-    ax1.set_ylabel('Portfolio value (USD)')
-
-    ax2 = plt.subplot(612, sharex=ax1)
-    ax2.set_ylabel('{asset} (USD)'.format(asset=context.ASSET_NAME))
-    (context.TICK_SIZE*results[['price', 'short_mavg', 'long_mavg']]).plot(ax=ax2)
-
-    trans = results.ix[[t != [] for t in results.transactions]]
-    amounts = [t[0]['amount'] for t in trans.transactions]
-
-    buys = trans.ix[
-        [t[0]['amount'] > 0 for t in trans.transactions]
-    ]
-    sells = trans.ix[
-        [t[0]['amount'] < 0 for t in trans.transactions]
-    ]
-
-    ax2.plot(
-        buys.index,
-        context.TICK_SIZE * results.price[buys.index],
-        '^',
-        markersize=10,
-        color='g',
-    )
-    ax2.plot(
-        sells.index,
-        context.TICK_SIZE * results.price[sells.index],
-        'v',
-        markersize=10,
-        color='r',
-    )
-
-    ax3 = plt.subplot(613, sharex=ax1)
-    results[['leverage', 'alpha', 'beta']].plot(ax=ax3)
-    ax3.set_ylabel('Leverage (USD)')
-
-    ax4 = plt.subplot(614, sharex=ax1)
-    results[['cash']].plot(ax=ax4)
-    ax4.set_ylabel('Cash (USD)')
-
-    results[[
-        'treasury',
-        'algorithm',
-        'benchmark',
-    ]] = results[[
-        'treasury_period_return',
-        'algorithm_period_return',
-        'benchmark_period_return',
-    ]]
-
-    ax5 = plt.subplot(615, sharex=ax1)
-    results[[
-        'treasury',
-        'algorithm',
-        'benchmark',
-    ]].plot(ax=ax5)
-    ax5.set_ylabel('Percent Change')
-
-    ax6 = plt.subplot(616, sharex=ax1)
-    results[['volume']].plot(ax=ax6)
-    ax6.set_ylabel('Volume (mBTC/day)')
-
-    plt.legend(loc=3)
-
-    # Show the plot.
-    plt.gcf().set_size_inches(18, 8)
-    plt.show()

catalyst/examples/mean_reversion_simple.py

@@ -0,0 +1,289 @@
+# For this example, we're going to write a simple momentum script.  When the
+# stock goes up quickly, we're going to buy; when it goes down quickly, we're
+# going to sell.  Hopefully we'll ride the waves.
+import os
+import tempfile
+import time
+
+import numpy as np
+import pandas as pd
+import talib
+from logbook import Logger
+
+from catalyst import run_algorithm
+from catalyst.api import symbol, record, order_target_percent, get_open_orders
+from catalyst.exchange.stats_utils import extract_transactions
+
+# We give a name to the algorithm which Catalyst will use to persist its state.
+# In this example, Catalyst will create the `.catalyst/data/live_algos`
+# directory. If we stop and start the algorithm, Catalyst will resume its
+# state using the files included in the folder.
+from catalyst.utils.paths import ensure_directory
+
+NAMESPACE = 'mean_reversion_simple'
+log = Logger(NAMESPACE)
+
+
+# To run an algorithm in Catalyst, you need two functions: initialize and
+# handle_data.
+
+def initialize(context):
+    # This initialize function sets any data or variables that you'll use in
+    # your algorithm.  For instance, you'll want to define the trading pair (or
+    # trading pairs) you want to backtest.  You'll also want to define any
+    # parameters or values you're going to use.
+
+    # In our example, we're looking at Neo in Ether.
+    context.market = symbol('neo_eth')
+    context.base_price = None
+    context.current_day = None
+
+    context.RSI_OVERSOLD = 30
+    context.RSI_OVERBOUGHT = 80
+    context.CANDLE_SIZE = '5T'
+
+    context.start_time = time.time()
+
+    # context.set_commission(maker=0.1, taker=0.2)
+    context.set_slippage(spread=0.0001)
+
+
+def handle_data(context, data):
+    # This handle_data function is where the real work is done.  Our data is
+    # minute-level tick data, and each minute is called a frame.  This function
+    # runs on each frame of the data.
+
+    # We flag the first period of each day.
+    # Since cryptocurrencies trade 24/7 the `before_trading_starts` handle
+    # would only execute once. This method works with minute and daily
+    # frequencies.
+    today = data.current_dt.floor('1D')
+    if today != context.current_day:
+        context.traded_today = False
+        context.current_day = today
+
+    # We're computing the volume-weighted-average-price of the security
+    # defined above, in the context.market variable.  For this example, we're
+    # using three bars on the 15 min bars.
+
+    # The frequency attribute determine the bar size. We use this convention
+    # for the frequency alias:
+    # http://pandas.pydata.org/pandas-docs/stable/timeseries.html#offset-aliases
+    prices = data.history(
+        context.market,
+        fields='close',
+        bar_count=50,
+        frequency=context.CANDLE_SIZE
+    )
+
+    # Ta-lib calculates various technical indicator based on price and
+    # volume arrays.
+
+    # In this example, we are comp
+    rsi = talib.RSI(prices.values, timeperiod=14)
+
+    # We need a variable for the current price of the security to compare to
+    # the average. Since we are requesting two fields, data.current()
+    # returns a DataFrame with
+    current = data.current(context.market, fields=['close', 'volume'])
+    price = current['close']
+
+    # If base_price is not set, we use the current value. This is the
+    # price at the first bar which we reference to calculate price_change.
+    if context.base_price is None:
+        context.base_price = price
+
+    price_change = (price - context.base_price) / context.base_price
+    cash = context.portfolio.cash
+
+    # Now that we've collected all current data for this frame, we use
+    # the record() method to save it. This data will be available as
+    # a parameter of the analyze() function for further analysis.
+
+    record(
+        volume=current['volume'],
+        price=price,
+        price_change=price_change,
+        rsi=rsi[-1],
+        cash=cash
+    )
+    # We are trying to avoid over-trading by limiting our trades to
+    # one per day.
+    if context.traded_today:
+        return
+
+    # TODO: retest with open orders
+    # Since we are using limit orders, some orders may not execute immediately
+    # we wait until all orders are executed before considering more trades.
+    orders = get_open_orders(context.market)
+    if len(orders) > 0:
+        log.info('exiting because orders are open: {}'.format(orders))
+        return
+
+    # Exit if we cannot trade
+    if not data.can_trade(context.market):
+        return
+
+    # Another powerful built-in feature of the Catalyst backtester is the
+    # portfolio object.  The portfolio object tracks your positions, cash,
+    # cost basis of specific holdings, and more.  In this line, we calculate
+    # how long or short our position is at this minute.
+    pos_amount = context.portfolio.positions[context.market].amount
+
+    if rsi[-1] <= context.RSI_OVERSOLD and pos_amount == 0:
+        log.info(
+            '{}: buying - price: {}, rsi: {}'.format(
+                data.current_dt, price, rsi[-1]
+            )
+        )
+        # Set a style for limit orders,
+        limit_price = price * 1.005
+        order_target_percent(
+            context.market, 1, limit_price=limit_price
+        )
+        context.traded_today = True
+
+    elif rsi[-1] >= context.RSI_OVERBOUGHT and pos_amount > 0:
+        log.info(
+            '{}: selling - price: {}, rsi: {}'.format(
+                data.current_dt, price, rsi[-1]
+            )
+        )
+        limit_price = price * 0.995
+        order_target_percent(
+            context.market, 0, limit_price=limit_price
+        )
+        context.traded_today = True
+
+
+def analyze(context=None, perf=None):
+    end = time.time()
+    log.info('elapsed time: {}'.format(end - context.start_time))
+
+    import matplotlib.pyplot as plt
+    # The base currency of the algo exchange
+    base_currency = context.exchanges.values()[0].base_currency.upper()
+
+    # Plot the portfolio value over time.
+    ax1 = plt.subplot(611)
+    perf.loc[:, 'portfolio_value'].plot(ax=ax1)
+    ax1.set_ylabel('Portfolio\nValue\n({})'.format(base_currency))
+
+    # Plot the price increase or decrease over time.
+    ax2 = plt.subplot(612, sharex=ax1)
+    perf.loc[:, 'price'].plot(ax=ax2, label='Price')
+
+    ax2.set_ylabel('{asset}\n({base})'.format(
+        asset=context.market.symbol, base=base_currency
+    ))
+
+    transaction_df = extract_transactions(perf)
+    if not transaction_df.empty:
+        buy_df = transaction_df[transaction_df['amount'] > 0]
+        sell_df = transaction_df[transaction_df['amount'] < 0]
+        ax2.scatter(
+            buy_df.index.to_pydatetime(),
+            perf.loc[buy_df.index.floor('1 min'), 'price'],
+            marker='^',
+            s=100,
+            c='green',
+            label=''
+        )
+        ax2.scatter(
+            sell_df.index.to_pydatetime(),
+            perf.loc[sell_df.index.floor('1 min'), 'price'],
+            marker='v',
+            s=100,
+            c='red',
+            label=''
+        )
+
+    ax4 = plt.subplot(613, sharex=ax1)
+    perf.loc[:, 'cash'].plot(
+        ax=ax4, label='Base Currency ({})'.format(base_currency)
+    )
+    ax4.set_ylabel('Cash\n({})'.format(base_currency))
+
+    perf['algorithm'] = perf.loc[:, 'algorithm_period_return']
+
+    ax5 = plt.subplot(614, sharex=ax1)
+    perf.loc[:, ['algorithm', 'price_change']].plot(ax=ax5)
+    ax5.set_ylabel('Percent\nChange')
+
+    ax6 = plt.subplot(615, sharex=ax1)
+    perf.loc[:, 'rsi'].plot(ax=ax6, label='RSI')
+    ax6.set_ylabel('RSI')
+    ax6.axhline(context.RSI_OVERBOUGHT, color='darkgoldenrod')
+    ax6.axhline(context.RSI_OVERSOLD, color='darkgoldenrod')
+
+    if not transaction_df.empty:
+        ax6.scatter(
+            buy_df.index.to_pydatetime(),
+            perf.loc[buy_df.index.floor('1 min'), 'rsi'],
+            marker='^',
+            s=100,
+            c='green',
+            label=''
+        )
+        ax6.scatter(
+            sell_df.index.to_pydatetime(),
+            perf.loc[sell_df.index.floor('1 min'), 'rsi'],
+            marker='v',
+            s=100,
+            c='red',
+            label=''
+        )
+    plt.legend(loc=3)
+    start, end = ax6.get_ylim()
+    ax6.yaxis.set_ticks(np.arange(0, end, end / 5))
+
+    # Show the plot.
+    plt.gcf().set_size_inches(18, 8)
+    plt.show()
+    pass
+
+
+if __name__ == '__main__':
+    # The execution mode: backtest or live
+    MODE = 'backtest'
+
+    if MODE == 'backtest':
+        folder = os.path.join(
+            tempfile.gettempdir(), 'catalyst', NAMESPACE
+        )
+        ensure_directory(folder)
+
+        timestr = time.strftime('%Y%m%d-%H%M%S')
+        out = os.path.join(folder, '{}.p'.format(timestr))
+        # catalyst run -f catalyst/examples/mean_reversion_simple.py \
+        #    -x bitfinex -s 2017-10-1 -e 2017-11-10 -c usdt -n mean-reversion \
+        #   --data-frequency minute --capital-base 10000
+        run_algorithm(
+            capital_base=0.1,
+            data_frequency='minute',
+            initialize=initialize,
+            handle_data=handle_data,
+            analyze=analyze,
+            exchange_name='bitfinex',
+            algo_namespace=NAMESPACE,
+            base_currency='eth',
+            start=pd.to_datetime('2017-10-01', utc=True),
+            end=pd.to_datetime('2017-11-10', utc=True),
+            output=out
+        )
+        log.info('saved perf stats: {}'.format(out))
+
+    elif MODE == 'live':
+        run_algorithm(
+            capital_base=0.05,
+            initialize=initialize,
+            handle_data=handle_data,
+            analyze=analyze,
+            exchange_name='binance',
+            live=True,
+            algo_namespace=NAMESPACE,
+            base_currency='eth',
+            live_graph=False,
+            simulate_orders=True,
+            stats_output=None
+        )

catalyst/examples/portfolio_optimization.py

@@ -0,0 +1,149 @@
+'''Use this code to execute a portfolio optimization model. This code
+   will select the portfolio with the maximum Sharpe Ratio. The parameters
+   are set to use 180 days of historical data and rebalance every 30 days.
+
+   This is the code used in the following article:
+   https://blog.enigma.co/markowitz-portfolio-optimization-for-cryptocurrencies-in-catalyst-b23c38652556
+
+   You can run this code using the Python interpreter:
+
+   $ python portfolio_optimization.py
+'''
+
+from __future__ import division
+import os
+import pytz
+import numpy as np
+import pandas as pd
+import matplotlib.pyplot as plt
+from datetime import datetime
+
+from catalyst.api import record, symbols, order_target_percent
+from catalyst.utils.run_algo import run_algorithm
+
+np.set_printoptions(threshold='nan', suppress=True)
+
+
+def initialize(context):
+    # Portfolio assets list
+    context.assets = symbols('btc_usdt', 'eth_usdt', 'ltc_usdt', 'dash_usdt',
+                             'xmr_usdt')
+    context.nassets = len(context.assets)
+    # Set the time window that will be used to compute expected return
+    # and asset correlations
+    context.window = 180
+    # Set the number of days between each portfolio rebalancing
+    context.rebalance_period = 30
+    context.i = 0
+
+
+def handle_data(context, data):
+    # Only rebalance at the beggining of the algorithm execution and
+    # every multiple of the rebalance period
+    if context.i == 0 or context.i % context.rebalance_period == 0:
+        n = context.window
+        prices = data.history(context.assets, fields='price',
+                              bar_count=n + 1, frequency='1d')
+        pr = np.asmatrix(prices)
+        t_prices = prices.iloc[1:n + 1]
+        t_val = t_prices.values
+        tminus_prices = prices.iloc[0:n]
+        tminus_val = tminus_prices.values
+        # Compute daily returns (r)
+        r = np.asmatrix(t_val / tminus_val - 1)
+        # Compute the expected returns of each asset with the average
+        # daily return for the selected time window
+        m = np.asmatrix(np.mean(r, axis=0))
+        # ###
+        stds = np.std(r, axis=0)
+        # Compute excess returns matrix (xr)
+        xr = r - m
+        # Matrix algebra to get variance-covariance matrix
+        cov_m = np.dot(np.transpose(xr), xr) / n
+        # Compute asset correlation matrix (informative only)
+        corr_m = cov_m / np.dot(np.transpose(stds), stds)
+
+        # Define portfolio optimization parameters
+        n_portfolios = 50000
+        results_array = np.zeros((3 + context.nassets, n_portfolios))
+        for p in xrange(n_portfolios):
+            weights = np.random.random(context.nassets)
+            weights /= np.sum(weights)
+            w = np.asmatrix(weights)
+            p_r = np.sum(np.dot(w, np.transpose(m))) * 365
+            p_std = np.sqrt(np.dot(np.dot(w, cov_m),
+                                   np.transpose(w))) * np.sqrt(365)
+
+            # store results in results array
+            results_array[0, p] = p_r
+            results_array[1, p] = p_std
+            # store Sharpe Ratio (return / volatility) - risk free rate element
+            # excluded for simplicity
+            results_array[2, p] = results_array[0, p] / results_array[1, p]
+            i = 0
+            for iw in weights:
+                results_array[3 + i, p] = weights[i]
+                i += 1
+
+        # convert results array to Pandas DataFrame
+        results_frame = pd.DataFrame(np.transpose(results_array),
+                                     columns=['r', 'stdev', 'sharpe']
+                                             + context.assets)
+        # locate position of portfolio with highest Sharpe Ratio
+        max_sharpe_port = results_frame.iloc[results_frame['sharpe'].idxmax()]
+        # locate positon of portfolio with minimum standard deviation
+        # min_vol_port = results_frame.iloc[results_frame['stdev'].idxmin()]
+
+        # order optimal weights for each asset
+        for asset in context.assets:
+            if data.can_trade(asset):
+                order_target_percent(asset, max_sharpe_port[asset])
+
+        # create scatter plot coloured by Sharpe Ratio
+        plt.scatter(results_frame.stdev,
+                    results_frame.r,
+                    c=results_frame.sharpe,
+                    cmap='RdYlGn')
+        plt.xlabel('Volatility')
+        plt.ylabel('Returns')
+        plt.colorbar()
+        # plot red star to highlight position of portfolio
+        # with highest Sharpe Ratio
+        plt.scatter(max_sharpe_port[1],
+                    max_sharpe_port[0],
+                    marker='o',
+                    color='b',
+                    s=200)
+        # plot green star to highlight position of minimum variance portfolio
+        plt.show()
+        print(max_sharpe_port)
+        record(pr=pr,
+               r=r,
+               m=m,
+               stds=stds,
+               max_sharpe_port=max_sharpe_port,
+               corr_m=corr_m)
+    context.i += 1
+
+
+def analyze(context=None, results=None):
+    # Form DataFrame with selected data
+    data = results[['pr', 'r', 'm', 'stds', 'max_sharpe_port', 'corr_m',
+                    'portfolio_value']]
+
+    # Save results in CSV file
+    filename = os.path.splitext(os.path.basename(__file__))[0]
+    data.to_csv(filename + '.csv')
+
+
+if __name__ == '__main__':
+    # Bitcoin data is available from 2015-3-2. Dates vary for other tokens.
+    start = datetime(2017, 1, 1, 0, 0, 0, 0, pytz.utc)
+    end = datetime(2017, 8, 16, 0, 0, 0, 0, pytz.utc)
+    results = run_algorithm(initialize=initialize,
+                            handle_data=handle_data,
+                            analyze=analyze,
+                            start=start,
+                            end=end,
+                            exchange_name='poloniex',
+                            capital_base=100000, )

catalyst/examples/rsi_profit_target.py

@@ -0,0 +1,265 @@
+from datetime import timedelta
+
+import pandas as pd
+import numpy as np
+import talib
+from logbook import Logger
+
+from catalyst.api import (
+    order,
+    symbol,
+    record,
+    get_open_orders,
+)
+from catalyst.utils.run_algo import run_algorithm
+
+algo_namespace = 'rsi'
+log = Logger(algo_namespace)
+
+
+def initialize(context):
+    log.info('initializing algo')
+    context.asset = symbol('eth_btc')
+    context.base_price = None
+
+    context.MAX_HOLDINGS = 0.2
+    context.RSI_OVERSOLD = 30
+    context.RSI_OVERSOLD_BBANDS = 45
+    context.RSI_OVERBOUGHT_BBANDS = 55
+    context.SLIPPAGE_ALLOWED = 0.03
+
+    context.TARGET = 0.15
+    context.STOP_LOSS = 0.1
+    context.STOP = 0.03
+    context.position = None
+
+    context.last_bar = None
+
+    context.errors = []
+    pass
+
+
+def _handle_buy_sell_decision(context, data, signal, price):
+    orders = get_open_orders(context.asset)
+    if len(orders) > 0:
+        log.info('skipping bar until all open orders execute')
+        return
+
+    positions = context.portfolio.positions
+    if context.position is None and context.asset in positions:
+        position = positions[context.asset]
+        context.position = dict(
+            cost_basis=position['cost_basis'],
+            amount=position['amount'],
+            stop=None
+        )
+
+    # action = None
+    if context.position is not None:
+        cost_basis = context.position['cost_basis']
+        amount = context.position['amount']
+        log.info(
+            'found {amount} positions with cost basis {cost_basis}'.format(
+                amount=amount,
+                cost_basis=cost_basis
+            )
+        )
+        stop = context.position['stop']
+
+        target = cost_basis * (1 + context.TARGET)
+        if price >= target:
+            context.position['cost_basis'] = price
+            context.position['stop'] = context.STOP
+
+        stop_target = context.STOP_LOSS if stop is None else context.STOP
+        if price < cost_basis * (1 - stop_target):
+            log.info('executing stop loss')
+            order(
+                asset=context.asset,
+                amount=-amount,
+                limit_price=price * (1 - context.SLIPPAGE_ALLOWED),
+            )
+            # action = 0
+            context.position = None
+
+    else:
+        if signal == 'long':
+            log.info('opening position')
+            buy_amount = context.MAX_HOLDINGS / price
+            order(
+                asset=context.asset,
+                amount=buy_amount,
+                limit_price=price * (1 + context.SLIPPAGE_ALLOWED),
+            )
+            context.position = dict(
+                cost_basis=price,
+                amount=buy_amount,
+                stop=None
+            )
+            # action = 0
+
+
+def _handle_data_rsi_only(context, data):
+    price = data.current(context.asset, 'close')
+    log.info('got price {price}'.format(price=price))
+
+    if price is np.nan:
+        log.warn('no pricing data')
+        return
+
+    if context.base_price is None:
+        context.base_price = price
+
+    try:
+        prices = data.history(
+            context.asset,
+            fields='price',
+            bar_count=20,
+            frequency='30T'
+        )
+    except Exception as e:
+        log.warn('historical data not available: '.format(e))
+        return
+
+    rsi = talib.RSI(prices.values, timeperiod=16)[-1]
+    log.info('got rsi {}'.format(rsi))
+
+    signal = None
+    if rsi < context.RSI_OVERSOLD:
+        signal = 'long'
+
+    # Making sure that the price is still current
+    price = data.current(context.asset, 'close')
+    cash = context.portfolio.cash
+    log.info(
+        'base currency available: {cash}, cap: {cap}'.format(
+            cash=cash,
+            cap=context.MAX_HOLDINGS
+        )
+    )
+    volume = data.current(context.asset, 'volume')
+    price_change = (price - context.base_price) / context.base_price
+    record(
+        price=price,
+        price_change=price_change,
+        rsi=rsi,
+        volume=volume,
+        cash=cash,
+        starting_cash=context.portfolio.starting_cash,
+        leverage=context.account.leverage,
+    )
+
+    _handle_buy_sell_decision(context, data, signal, price)
+
+
+def handle_data(context, data):
+    dt = data.current_dt
+
+    if context.last_bar is None or (
+            context.last_bar + timedelta(minutes=15)) <= dt:
+        context.last_bar = dt
+    else:
+        return
+
+    log.info('BAR {}'.format(dt))
+    try:
+        _handle_data_rsi_only(context, data)
+    except Exception as e:
+        log.warn('aborting the bar on error {}'.format(e))
+        context.errors.append(e)
+
+    if len(context.errors) > 0:
+        log.info('the errors:\n{}'.format(context.errors))
+
+
+def analyze(context=None, results=None):
+    import matplotlib.pyplot as plt
+
+    base_currency = context.exchanges.values()[0].base_currency.upper()
+    # Plot the portfolio and asset data.
+    ax1 = plt.subplot(611)
+    results.loc[:, 'portfolio_value'].plot(ax=ax1)
+    ax1.set_ylabel('Portfolio Value ({})'.format(base_currency))
+
+    ax2 = plt.subplot(612, sharex=ax1)
+    results.loc[:, 'price'].plot(ax=ax2)
+    ax2.set_ylabel('{asset} ({base})'.format(
+        asset=context.asset.symbol, base=base_currency
+    ))
+
+    trans = results.loc[[t != [] for t in results.transactions], :]
+    buys = trans.loc[[t[0]['amount'] > 0 for t in trans.transactions], :]
+    sells = trans.loc[[t[0]['amount'] < 0 for t in trans.transactions], :]
+    # buys = results.loc[results['action'] == 1, :]
+    # sells = results.loc[results['action'] == 0, :]
+
+    ax2.plot(
+        buys.index,
+        results.loc[buys.index, 'price'],
+        '^',
+        markersize=10,
+        color='g',
+    )
+    ax2.plot(
+        sells.index,
+        results.loc[sells.index, 'price'],
+        'v',
+        markersize=10,
+        color='r',
+    )
+
+    ax3 = plt.subplot(613, sharex=ax1)
+    results.loc[:, ['alpha', 'beta']].plot(ax=ax3)
+    ax3.set_ylabel('Alpha / Beta ')
+
+    ax4 = plt.subplot(614, sharex=ax1)
+    results.loc[:, ['starting_cash', 'cash']].plot(ax=ax4)
+    ax4.set_ylabel('Base Currency ({})'.format(base_currency))
+
+    results['algorithm'] = results.loc[:, 'algorithm_period_return']
+
+    ax5 = plt.subplot(615, sharex=ax1)
+    results.loc[:, ['algorithm', 'price_change']].plot(ax=ax5)
+    ax5.set_ylabel('Percent Change')
+
+    ax6 = plt.subplot(616, sharex=ax1)
+    results.loc[:, 'rsi'].plot(ax=ax6)
+    ax6.set_ylabel('RSI')
+
+    ax6.plot(
+        buys.index,
+        results.loc[buys.index, 'rsi'],
+        '^',
+        markersize=10,
+        color='g',
+    )
+    ax6.plot(
+        sells.index,
+        results.loc[sells.index, 'rsi'],
+        'v',
+        markersize=10,
+        color='r',
+    )
+
+    plt.legend(loc=3)
+
+    # Show the plot.
+    plt.gcf().set_size_inches(18, 8)
+    plt.show()
+    pass
+
+
+if __name__ == '__main__':
+    # Backtest
+    run_algorithm(
+        capital_base=0.5,
+        data_frequency='minute',
+        initialize=initialize,
+        handle_data=handle_data,
+        analyze=analyze,
+        exchange_name='poloniex',
+        algo_namespace=algo_namespace,
+        base_currency='btc',
+        start=pd.to_datetime('2017-9-1', utc=True),
+        end=pd.to_datetime('2017-10-1', utc=True),
+    )

catalyst/examples/simple_loop.py

@@ -0,0 +1,125 @@
+import talib
+import pandas as pd
+
+from catalyst import run_algorithm
+from catalyst.api import symbol, record
+from catalyst.exchange.stats_utils import get_pretty_stats, \
+    extract_transactions
+
+
+def initialize(context):
+    print('initializing')
+    context.asset = symbol('eth_btc')
+    context.base_price = None
+
+
+def handle_data(context, data):
+    print('handling bar: {}'.format(data.current_dt))
+
+    price = data.current(context.asset, 'close')
+    print('got price {price}'.format(price=price))
+
+    prices = data.history(
+        context.asset,
+        fields='price',
+        bar_count=20,
+        frequency='30T'
+    )
+    last_traded = prices.index[-1]
+    print('last candle date: {}'.format(last_traded))
+
+    rsi = talib.RSI(prices.values, timeperiod=14)[-1]
+    print('got rsi: {}'.format(rsi))
+
+    # If base_price is not set, we use the current value. This is the
+    # price at the first bar which we reference to calculate price_change.
+    if context.base_price is None:
+        context.base_price = price
+
+    price_change = (price - context.base_price) / context.base_price
+    cash = context.portfolio.cash
+
+    # Now that we've collected all current data for this frame, we use
+    # the record() method to save it. This data will be available as
+    # a parameter of the analyze() function for further analysis.
+    record(
+        price=price,
+        price_change=price_change,
+        cash=cash
+    )
+
+
+def analyze(context, perf):
+    import matplotlib.pyplot as plt
+    print('the stats: {}'.format(get_pretty_stats(perf)))
+
+    # The base currency of the algo exchange
+    base_currency = context.exchanges.values()[0].base_currency.upper()
+
+    # Plot the portfolio value over time.
+    ax1 = plt.subplot(611)
+    perf.loc[:, 'portfolio_value'].plot(ax=ax1)
+    ax1.set_ylabel('Portfolio Value ({})'.format(base_currency))
+
+    # Plot the price increase or decrease over time.
+    ax2 = plt.subplot(612, sharex=ax1)
+    perf.loc[:, 'price'].plot(ax=ax2, label='Price')
+
+    ax2.set_ylabel('{asset} ({base})'.format(
+        asset=context.asset.symbol, base=base_currency
+    ))
+
+    transaction_df = extract_transactions(perf)
+    if not transaction_df.empty:
+        buy_df = transaction_df[transaction_df['amount'] > 0]
+        sell_df = transaction_df[transaction_df['amount'] < 0]
+        ax2.scatter(
+            buy_df.index.to_pydatetime(),
+            perf.loc[buy_df.index, 'price'],
+            marker='^',
+            s=100,
+            c='green',
+            label=''
+        )
+        ax2.scatter(
+            sell_df.index.to_pydatetime(),
+            perf.loc[sell_df.index, 'price'],
+            marker='v',
+            s=100,
+            c='red',
+            label=''
+        )
+
+    ax4 = plt.subplot(613, sharex=ax1)
+    perf.loc[:, 'cash'].plot(
+        ax=ax4, label='Base Currency ({})'.format(base_currency)
+    )
+    ax4.set_ylabel('Cash ({})'.format(base_currency))
+
+    perf['algorithm'] = perf.loc[:, 'algorithm_period_return']
+
+    ax5 = plt.subplot(614, sharex=ax1)
+    perf.loc[:, ['algorithm', 'price_change']].plot(ax=ax5)
+    ax5.set_ylabel('Percent Change')
+
+    plt.legend(loc=3)
+
+    # Show the plot.
+    plt.gcf().set_size_inches(18, 8)
+    plt.show()
+    pass
+
+
+if __name__ == '__main__':
+    run_algorithm(
+        capital_base=1,
+        initialize=initialize,
+        handle_data=handle_data,
+        analyze=None,
+        exchange_name='poloniex',
+        live=True,
+        algo_namespace='simple_loop',
+        base_currency='eth',
+        live_graph=False,
+        simulate_orders=True
+    )

catalyst/examples/simple_universe.py

@@ -0,0 +1,171 @@
+"""
+Requires Catalyst version 0.3.0 or above
+Tested on Catalyst version 0.3.3
+
+This example aims to provide an easy way for users to learn how to
+collect data from any given exchange and select a subset of the available
+currency pairs for trading. You simply need to specify the exchange and
+the market (base_currency) that you want to focus on. You will then see
+how to create a universe of assets, and filter it based the market you
+desire.
+
+The example prints out the closing price of all the pairs for a given
+market in a given exchange every 30 minutes. The example also contains
+the OHLCV data with minute-resolution for the past seven days which
+could be used to create indicators. Use this code as the backbone to
+create your own trading strategy.
+
+The lookback_date variable is used to ensure data for a coin existed on
+the lookback period specified.
+
+To run, execute the following two commands in a terminal (inside catalyst
+environment). The first one retrieves all the pricing data needed for this
+script to run (only needs to be run once), and the second one executes this
+script with the parameters specified in the run_algorithm() call at the end
+of the file:
+
+catalyst ingest-exchange -x bitfinex -f minute
+
+python simple_universe.py
+
+"""
+from datetime import timedelta
+
+import numpy as np
+import pandas as pd
+
+from catalyst import run_algorithm
+from catalyst.exchange.exchange_utils import get_exchange_symbols
+from catalyst.api import (symbols, )
+
+
+def initialize(context):
+    context.i = -1  # minute counter
+    context.exchange = context.exchanges.values()[0].name.lower()
+    context.base_currency = context.exchanges.values()[0].base_currency.lower()
+
+
+def handle_data(context, data):
+    context.i += 1
+    lookback_days = 7  # 7 days
+
+    # current date & time in each iteration formatted into a string
+    now = data.current_dt
+    date, time = now.strftime('%Y-%m-%d %H:%M:%S').split(' ')
+    lookback_date = now - timedelta(days=lookback_days)
+    # keep only the date as a string, discard the time
+    lookback_date = lookback_date.strftime('%Y-%m-%d %H:%M:%S').split(' ')[0]
+
+    one_day_in_minutes = 1440  # 60 * 24 assumes data_frequency='minute'
+    # update universe everyday at midnight
+    if not context.i % one_day_in_minutes:
+        context.universe = universe(context, lookback_date, date)
+
+    # get data every 30 minutes
+    minutes = 30
+
+    # get lookback_days of history data: that is 'lookback' number of bins
+    lookback = one_day_in_minutes / minutes * lookback_days
+    if not context.i % minutes and context.universe:
+        # we iterate for every pair in the current universe
+        for coin in context.coins:
+            pair = str(coin.symbol)
+
+            # Get 30 minute interval OHLCV data. This is the standard data
+            # required for candlestick or indicators/signals. Return Pandas
+            # DataFrames. 30T means 30-minute re-sampling of one minute data.
+            # Adjust it to your desired time interval as needed.
+            opened = fill(data.history(coin,
+                                       'open',
+                                       bar_count=lookback,
+                                       frequency='30T')).values
+            high = fill(data.history(coin,
+                                     'high',
+                                     bar_count=lookback,
+                                     frequency='30T')).values
+            low = fill(data.history(coin,
+                                    'low',
+                                    bar_count=lookback,
+                                    frequency='30T')).values
+            close = fill(data.history(coin,
+                                      'price',
+                                      bar_count=lookback,
+                                      frequency='30T')).values
+            volume = fill(data.history(coin,
+                                       'volume',
+                                       bar_count=lookback,
+                                       frequency='30T')).values
+
+            # close[-1] is the last value in the set, which is the equivalent
+            # to current price (as in the most recent value)
+            # displays the minute price for each pair every 30 minutes
+            print('{now}: {pair} -\tO:{o},\tH:{h},\tL:{c},\tC{c},'
+                  '\tV:{v}'.format(
+                    now=now,
+                    pair=pair,
+                    o=opened[-1],
+                    h=high[-1],
+                    l=low[-1],
+                    c=close[-1],
+                    v=volume[-1],
+                  ))
+
+            # -------------------------------------------------------------
+            # --------------- Insert Your Strategy Here -------------------
+            # -------------------------------------------------------------
+
+
+def analyze(context=None, results=None):
+    pass
+
+
+# Get the universe for a given exchange and a given base_currency market
+# Example: Poloniex BTC Market
+def universe(context, lookback_date, current_date):
+    # get all the pairs for the given exchange
+    json_symbols = get_exchange_symbols(context.exchange)
+    # convert into a DataFrame for easier processing
+    df = pd.DataFrame.from_dict(json_symbols).transpose().astype(str)
+    df['base_currency'] = df.apply(lambda row: row.symbol.split('_')[1],
+                                   axis=1)
+    df['market_currency'] = df.apply(lambda row: row.symbol.split('_')[0],
+                                     axis=1)
+
+    # Filter all the pairs to get only the ones for a given base_currency
+    df = df[df['base_currency'] == context.base_currency]
+
+    # Filter all pairs to ensure that pair existed in the current date range
+    df = df[df.start_date < lookback_date]
+    df = df[df.end_daily >= current_date]
+    context.coins = symbols(*df.symbol)  # convert all the pairs to symbols
+
+    return df.symbol.tolist()
+
+
+# Replace all NA, NAN or infinite values with its nearest value
+def fill(series):
+    if isinstance(series, pd.Series):
+        return series.replace([np.inf, -np.inf], np.nan).ffill().bfill()
+    elif isinstance(series, np.ndarray):
+        return pd.Series(series).replace(
+                     [np.inf, -np.inf], np.nan
+                    ).ffill().bfill().values
+    else:
+        return series
+
+
+if __name__ == '__main__':
+    start_date = pd.to_datetime('2017-11-10', utc=True)
+    end_date = pd.to_datetime('2017-11-13', utc=True)
+
+    performance = run_algorithm(start=start_date, end=end_date,
+                                capital_base=100.0,  # amount of base_currency
+                                initialize=initialize,
+                                handle_data=handle_data,
+                                analyze=analyze,
+                                exchange_name='poloniex',
+                                data_frequency='minute',
+                                base_currency='btc',
+                                live=False,
+                                live_graph=False,
+                                algo_namespace='simple_universe')

catalyst/examples/talib_simple.py

@@ -0,0 +1,366 @@
+# Run Command
+# catalyst run --start 2017-1-1 --end 2017-11-1 -o talib_simple.pickle \
+#   -f talib_simple.py -x poloniex
+#
+# Description
+# Simple TALib Example showing how to use various indicators
+# in you strategy. Based loosly on
+# https://github.com/mellertson/talib-macd-example/blob/master/talib-macd-matplotlib-example.py
+
+import os
+
+import matplotlib.pyplot as plt
+import numpy as np
+import pandas as pd
+import talib as ta
+from logbook import Logger
+from matplotlib.dates import date2num
+from matplotlib.finance import candlestick_ohlc
+
+from catalyst import run_algorithm
+from catalyst.api import (
+    order,
+    order_target_percent,
+    symbol,
+)
+from catalyst.exchange.stats_utils import get_pretty_stats
+
+algo_namespace = 'talib_sample'
+log = Logger(algo_namespace)
+
+
+def initialize(context):
+    log.info('Starting TALib Simple Example')
+
+    context.ASSET_NAME = 'BTC_USDT'
+    context.asset = symbol(context.ASSET_NAME)
+
+    context.ORDER_SIZE = 10
+    context.SLIPPAGE_ALLOWED = 0.05
+
+    context.swallow_errors = True
+    context.errors = []
+
+    # Bars to look at per iteration should be bigger than SMA_SLOW
+    context.BARS = 365
+    context.COUNT = 0
+
+    # Technical Analysis Settings
+    context.SMA_FAST = 50
+    context.SMA_SLOW = 100
+    context.RSI_PERIOD = 14
+    context.RSI_OVER_BOUGHT = 80
+    context.RSI_OVER_SOLD = 20
+    context.RSI_AVG_PERIOD = 15
+    context.MACD_FAST = 12
+    context.MACD_SLOW = 26
+    context.MACD_SIGNAL = 9
+    context.STOCH_K = 14
+    context.STOCH_D = 3
+    context.STOCH_OVER_BOUGHT = 80
+    context.STOCH_OVER_SOLD = 20
+
+    pass
+
+
+def _handle_data(context, data):
+    # Get price, open, high, low, close
+    prices = data.history(
+        context.asset,
+        bar_count=context.BARS,
+        fields=['price', 'open', 'high', 'low', 'close'],
+        frequency='1d')
+
+    # Create a analysis data frame
+    analysis = pd.DataFrame(index=prices.index)
+
+    # SMA FAST
+    analysis['sma_f'] = ta.SMA(prices.close.as_matrix(), context.SMA_FAST)
+    # SMA SLOW
+    analysis['sma_s'] = ta.SMA(prices.close.as_matrix(), context.SMA_SLOW)
+
+    # Relative Strength Index
+    analysis['rsi'] = ta.RSI(prices.close.as_matrix(), context.RSI_PERIOD)
+    # RSI SMA
+    analysis['sma_r'] = ta.SMA(analysis.rsi.as_matrix(),
+                               context.RSI_AVG_PERIOD)
+
+    # MACD, MACD Signal, MACD Histogram
+    analysis['macd'], analysis['macdSignal'], analysis['macdHist'] = ta.MACD(
+        prices.close.as_matrix(), fastperiod=context.MACD_FAST,
+        slowperiod=context.MACD_SLOW, signalperiod=context.MACD_SIGNAL)
+
+    # Stochastics %K %D
+    # %K = (Current Close - Lowest Low)/(Highest High - Lowest Low) * 100
+    # %D = 3-day SMA of %K
+    analysis['stoch_k'], analysis['stoch_d'] = ta.STOCH(
+        prices.high.as_matrix(), prices.low.as_matrix(),
+        prices.close.as_matrix(), slowk_period=context.STOCH_K,
+        slowd_period=context.STOCH_D)
+
+    # SMA FAST over SLOW Crossover
+    analysis['sma_test'] = np.where(analysis.sma_f > analysis.sma_s, 1, 0)
+
+    # MACD over Signal Crossover
+    analysis['macd_test'] = np.where((analysis.macd > analysis.macdSignal), 1,
+                                     0)
+
+    # Stochastics OVER BOUGHT & Decreasing
+    analysis['stoch_over_bought'] = np.where(
+        (analysis.stoch_k > context.STOCH_OVER_BOUGHT) & (
+            analysis.stoch_k > analysis.stoch_k.shift(1)), 1, 0)
+
+    # Stochastics OVER SOLD & Increasing
+    analysis['stoch_over_sold'] = np.where(
+        (analysis.stoch_k < context.STOCH_OVER_SOLD) & (
+            analysis.stoch_k > analysis.stoch_k.shift(1)), 1, 0)
+
+    # RSI OVER BOUGHT & Decreasing
+    analysis['rsi_over_bought'] = np.where(
+        (analysis.rsi > context.RSI_OVER_BOUGHT) & (
+            analysis.rsi < analysis.rsi.shift(1)), 1, 0)
+
+    # RSI OVER SOLD & Increasing
+    analysis['rsi_over_sold'] = np.where(
+        (analysis.rsi < context.RSI_OVER_SOLD) & (
+            analysis.rsi > analysis.rsi.shift(1)), 1, 0)
+
+    # Save the prices and analysis to send to analyze
+    context.prices = prices
+    context.analysis = analysis
+    context.price = data.current(context.asset, 'price')
+
+    makeOrders(context, analysis)
+
+    # Log the values of this bar
+    logAnalysis(analysis)
+
+
+def handle_data(context, data):
+    log.info('handling bar {}'.format(data.current_dt))
+    try:
+        _handle_data(context, data)
+    except Exception as e:
+        log.warn('aborting the bar on error {}'.format(e))
+        context.errors.append(e)
+
+    log.info('completed bar {}, total execution errors {}'.format(
+        data.current_dt,
+        len(context.errors)
+    ))
+
+    if len(context.errors) > 0:
+        log.info('the errors:\n{}'.format(context.errors))
+
+
+def analyze(context, results):
+    # Save results in CSV file
+    filename = os.path.splitext(os.path.basename('talib_simple'))[0]
+    results.to_csv(filename + '.csv')
+
+    log.info('the daily stats:\n{}'.format(get_pretty_stats(results)))
+    chart(context, context.prices, context.analysis, results)
+    pass
+
+
+def makeOrders(context, analysis):
+    if context.asset in context.portfolio.positions:
+
+        # Current position
+        position = context.portfolio.positions[context.asset]
+
+        if (position == 0):
+            log.info('Position Zero')
+            return
+
+        # Cost Basis
+        cost_basis = position.cost_basis
+
+        log.info(
+            'Holdings: {amount} @ {cost_basis}'.format(
+                amount=position.amount,
+                cost_basis=cost_basis
+            )
+        )
+
+        # Sell when holding and got sell singnal
+        if isSell(context, analysis):
+            profit = (context.price * position.amount) - (
+                cost_basis * position.amount)
+            order_target_percent(
+                asset=context.asset,
+                target=0,
+                limit_price=context.price * (1 - context.SLIPPAGE_ALLOWED),
+            )
+            log.info(
+                'Sold {amount} @ {price} Profit: {profit}'.format(
+                    amount=position.amount,
+                    price=context.price,
+                    profit=profit
+                )
+            )
+        else:
+            log.info('no buy or sell opportunity found')
+    else:
+        # Buy when not holding and got buy signal
+        if isBuy(context, analysis):
+            order(
+                asset=context.asset,
+                amount=context.ORDER_SIZE,
+                limit_price=context.price * (1 + context.SLIPPAGE_ALLOWED)
+            )
+            log.info(
+                'Bought {amount} @ {price}'.format(
+                    amount=context.ORDER_SIZE,
+                    price=context.price
+                )
+            )
+
+
+def isBuy(context, analysis):
+    # Bullish SMA Crossover
+    if (getLast(analysis, 'sma_test') == 1):
+        # Bullish MACD
+        if (getLast(analysis, 'macd_test') == 1):
+            return True
+
+    # # Bullish Stochastics
+    # if(getLast(analysis, 'stoch_over_sold') == 1):
+    #     return True
+
+    # # Bullish RSI
+    # if(getLast(analysis, 'rsi_over_sold') == 1):
+    #     return True
+
+    return False
+
+
+def isSell(context, analysis):
+    # Bearish SMA Crossover
+    if (getLast(analysis, 'sma_test') == 0):
+        # Bearish MACD
+        if (getLast(analysis, 'macd_test') == 0):
+            return True
+
+    # # Bearish Stochastics
+    # if(getLast(analysis, 'stoch_over_bought') == 0):
+    #     return True
+
+    # # Bearish RSI
+    # if(getLast(analysis, 'rsi_over_bought') == 0):
+    #     return True
+
+    return False
+
+
+def chart(context, prices, analysis, results):
+    results.portfolio_value.plot()
+
+    # Data for matplotlib finance plot
+    dates = date2num(prices.index.to_pydatetime())
+
+    # Create the Open High Low Close Tuple
+    prices_ohlc = [tuple([dates[i],
+                          prices.open[i],
+                          prices.high[i],
+                          prices.low[i],
+                          prices.close[i]]) for i in range(len(dates))]
+
+    fig = plt.figure(figsize=(14, 18))
+
+    # Draw the candle sticks
+    ax1 = fig.add_subplot(411)
+    ax1.set_ylabel(context.ASSET_NAME, size=20)
+    candlestick_ohlc(ax1, prices_ohlc, width=0.4, colorup='g', colordown='r')
+
+    # Draw Moving Averages
+    analysis.sma_f.plot(ax=ax1, c='r')
+    analysis.sma_s.plot(ax=ax1, c='g')
+
+    # RSI
+    ax2 = fig.add_subplot(412)
+    ax2.set_ylabel('RSI', size=12)
+    analysis.rsi.plot(ax=ax2, c='g',
+                      label='Period: ' + str(context.RSI_PERIOD))
+    analysis.sma_r.plot(ax=ax2, c='r',
+                        label='MA: ' + str(context.RSI_AVG_PERIOD))
+    ax2.axhline(y=30, c='b')
+    ax2.axhline(y=50, c='black')
+    ax2.axhline(y=70, c='b')
+    ax2.set_ylim([0, 100])
+    handles, labels = ax2.get_legend_handles_labels()
+    ax2.legend(handles, labels)
+
+    # Draw MACD computed with Talib
+    ax3 = fig.add_subplot(413)
+    ax3.set_ylabel('MACD: ' + str(context.MACD_FAST) + ', ' + str(
+        context.MACD_SLOW) + ', ' + str(context.MACD_SIGNAL), size=12)
+    analysis.macd.plot(ax=ax3, color='b', label='Macd')
+    analysis.macdSignal.plot(ax=ax3, color='g', label='Signal')
+    analysis.macdHist.plot(ax=ax3, color='r', label='Hist')
+    ax3.axhline(0, lw=2, color='0')
+    handles, labels = ax3.get_legend_handles_labels()
+    ax3.legend(handles, labels)
+
+    # Stochastic plot
+    ax4 = fig.add_subplot(414)
+    ax4.set_ylabel('Stoch (k,d)', size=12)
+    analysis.stoch_k.plot(ax=ax4, label='stoch_k:' + str(context.STOCH_K),
+                          color='r')
+    analysis.stoch_d.plot(ax=ax4, label='stoch_d:' + str(context.STOCH_D),
+                          color='g')
+    handles, labels = ax4.get_legend_handles_labels()
+    ax4.legend(handles, labels)
+    ax4.axhline(y=20, c='b')
+    ax4.axhline(y=50, c='black')
+    ax4.axhline(y=80, c='b')
+
+    plt.show()
+
+
+def logAnalysis(analysis):
+    # Log only the last value in the array
+    log.info('- sma_f:          {:.2f}'.format(getLast(analysis, 'sma_f')))
+    log.info('- sma_s:          {:.2f}'.format(getLast(analysis, 'sma_s')))
+
+    log.info('- rsi:            {:.2f}'.format(getLast(analysis, 'rsi')))
+    log.info('- sma_r:          {:.2f}'.format(getLast(analysis, 'sma_r')))
+
+    log.info('- macd:           {:.2f}'.format(getLast(analysis, 'macd')))
+    log.info(
+        '- macdSignal:     {:.2f}'.format(getLast(analysis, 'macdSignal')))
+    log.info('- macdHist:       {:.2f}'.format(getLast(analysis, 'macdHist')))
+
+    log.info('- stoch_k:        {:.2f}'.format(getLast(analysis, 'stoch_k')))
+    log.info('- stoch_d:        {:.2f}'.format(getLast(analysis, 'stoch_d')))
+
+    log.info('- sma_test:       {}'.format(getLast(analysis, 'sma_test')))
+    log.info('- macd_test:      {}'.format(getLast(analysis, 'macd_test')))
+
+    log.info('- stoch_over_bought:   {}'.format(
+        getLast(analysis, 'stoch_over_bought')))
+    log.info(
+        '- stoch_over_sold:   {}'.format(getLast(analysis, 'stoch_over_sold')))
+
+    log.info('- rsi_over_bought:       {}'.format(
+        getLast(analysis, 'rsi_over_bought')))
+    log.info(
+        '- rsi_over_sold:       {}'.format(getLast(analysis, 'rsi_over_sold')))
+
+
+def getLast(arr, name):
+    return arr[name][arr[name].index[-1]]
+
+
+if __name__ == '__main__':
+    run_algorithm(
+        capital_base=10000,
+        data_frequency='daily',
+        initialize=initialize,
+        handle_data=handle_data,
+        analyze=analyze,
+        exchange_name='poloniex',
+        base_currency='usdt',
+        start=pd.to_datetime('2016-11-1', utc=True),
+        end=pd.to_datetime('2017-11-10', utc=True),
+    )

catalyst/exchange/asset_finder_exchange.py

@@ -0,0 +1,99 @@
+from logbook import Logger
+
+from catalyst.constants import LOG_LEVEL
+
+log = Logger('AssetFinderExchange', level=LOG_LEVEL)
+
+
+class AssetFinderExchange(object):
+    def __init__(self):
+        self._asset_cache = {}
+
+    @property
+    def sids(self):
+        """
+        This seems to be used to pre-fetch assets.
+        I don't think that we need this for live-trading.
+        Leaving the list empty.
+        """
+        return list()
+
+    def retrieve_all(self, sids, default_none=False):
+        """
+        Retrieve all assets in `sids`.
+
+        Parameters
+        ----------
+        sids : iterable of int
+            Assets to retrieve.
+        default_none : bool
+            If True, return None for failed lookups.
+            If False, raise `SidsNotFound`.
+
+        Returns
+        -------
+        assets : list[Asset or None]
+            A list of the same length as `sids` containing Assets (or Nones)
+            corresponding to the requested sids.
+
+        Raises
+        ------
+        SidsNotFound
+            When a requested sid is not found and default_none=False.
+        """
+        # for sid in sids:
+        #     if sid in self._asset_cache:
+        #         log.debug('got asset from cache: {}'.format(sid))
+        #     else:
+        #         log.debug('fetching asset: {}'.format(sid))
+        return list()
+
+    def lookup_symbol(self, symbol, exchange, data_frequency=None,
+                      as_of_date=None, fuzzy=False):
+        """Lookup an asset by symbol.
+
+        Parameters
+        ----------
+        symbol : str
+            The ticker symbol to resolve.
+        as_of_date : datetime or None
+            Look up the last owner of this symbol as of this datetime.
+            If ``as_of_date`` is None, then this can only resolve the equity
+            if exactly one equity has ever owned the ticker.
+        fuzzy : bool, optional
+            Should fuzzy symbol matching be used? Fuzzy symbol matching
+            attempts to resolve differences in representations for
+            shareclasses. For example, some people may represent the ``A``
+            shareclass of ``BRK`` as ``BRK.A``, where others could write
+            ``BRK_A``.
+
+        Returns
+        -------
+        equity : Asset
+            The equity that held ``symbol`` on the given ``as_of_date``, or the
+            only equity to hold ``symbol`` if ``as_of_date`` is None.
+
+        Raises
+        ------
+        SymbolNotFound
+            Raised when no equity has ever held the given symbol.
+        MultipleSymbolsFound
+            Raised when no ``as_of_date`` is given and more than one equity
+            has held ``symbol``. This is also raised when ``fuzzy=True`` and
+            there are multiple candidates for the given ``symbol`` on the
+            ``as_of_date``.
+        """
+        log.debug('looking up symbol: {} {}'.format(symbol, exchange.name))
+
+        if data_frequency is not None:
+            key = ','.join([exchange.name, symbol, data_frequency])
+
+        else:
+            key = ','.join([exchange.name, symbol])
+
+        if key in self._asset_cache:
+            return self._asset_cache[key]
+        else:
+            asset = exchange.get_asset(symbol, data_frequency)
+            self._asset_cache[key] = asset
+            return asset

catalyst/exchange/bitfinex/bitfinex.py

@@ -0,0 +1,709 @@
+import base64
+import datetime
+import hashlib
+import hmac
+import json
+import re
+import time
+
+import numpy as np
+import pandas as pd
+import pytz
+import requests
+import six
+from catalyst.assets._assets import TradingPair
+from logbook import Logger
+
+from catalyst.constants import LOG_LEVEL
+from catalyst.exchange.exchange import Exchange
+from catalyst.exchange.exchange_bundle import ExchangeBundle
+from catalyst.exchange.exchange_errors import (
+    ExchangeRequestError,
+    InvalidHistoryFrequencyError,
+    InvalidOrderStyle, OrderCancelError)
+from catalyst.exchange.exchange_execution import ExchangeLimitOrder, \
+    ExchangeStopLimitOrder, ExchangeStopOrder
+from catalyst.exchange.exchange_utils import get_exchange_symbols_filename, \
+    download_exchange_symbols, get_symbols_string
+from catalyst.finance.order import Order, ORDER_STATUS
+from catalyst.protocol import Account
+
+# Trying to account for REST api instability
+# https://stackoverflow.com/questions/15431044/can-i-set-max-retries-for-requests-request
+from catalyst.utils.deprecate import deprecated
+
+requests.adapters.DEFAULT_RETRIES = 20
+
+BITFINEX_URL = 'https://api.bitfinex.com'
+
+log = Logger('Bitfinex', level=LOG_LEVEL)
+warning_logger = Logger('AlgoWarning')
+
+
+@deprecated
+class Bitfinex(Exchange):
+    def __init__(self, key, secret, base_currency, portfolio=None):
+        self.url = BITFINEX_URL
+        self.key = key
+        self.secret = secret.encode('UTF-8')
+        self.name = 'bitfinex'
+        self.color = 'green'
+
+        self.assets = dict()
+        self.load_assets()
+
+        self.local_assets = dict()
+        self.load_assets(is_local=True)
+
+        self.base_currency = base_currency
+        self._portfolio = portfolio
+        self.minute_writer = None
+        self.minute_reader = None
+
+        # The candle limit for each request
+        self.num_candles_limit = 1000
+
+        # Max is 90 but playing it safe
+        # https://www.bitfinex.com/posts/188
+        self.max_requests_per_minute = 80
+        self.request_cpt = dict()
+
+        self.bundle = ExchangeBundle(self.name)
+
+    def _request(self, operation, data, version='v1'):
+        payload_object = {
+            'request': '/{}/{}'.format(version, operation),
+            'nonce': '{0:f}'.format(time.time() * 1000000),
+            # convert to string
+            'options': {}
+        }
+
+        if data is None:
+            payload_dict = payload_object
+        else:
+            payload_dict = payload_object.copy()
+            payload_dict.update(data)
+
+        payload_json = json.dumps(payload_dict)
+        if six.PY3:
+            payload = base64.b64encode(bytes(payload_json, 'utf-8'))
+        else:
+            payload = base64.b64encode(payload_json)
+
+        m = hmac.new(self.secret, payload, hashlib.sha384)
+        m = m.hexdigest()
+
+        # headers
+        headers = {
+            'X-BFX-APIKEY': self.key,
+            'X-BFX-PAYLOAD': payload,
+            'X-BFX-SIGNATURE': m
+        }
+
+        if data is None:
+            request = requests.get(
+                '{url}/{version}/{operation}'.format(
+                    url=self.url,
+                    version=version,
+                    operation=operation
+                ), data={},
+                headers=headers)
+        else:
+            request = requests.post(
+                '{url}/{version}/{operation}'.format(
+                    url=self.url,
+                    version=version,
+                    operation=operation
+                ),
+                headers=headers)
+
+        return request
+
+    def _get_v2_symbol(self, asset):
+        pair = asset.symbol.split('_')
+        symbol = 't' + pair[0].upper() + pair[1].upper()
+        return symbol
+
+    def _get_v2_symbols(self, assets):
+        """
+        Workaround to support Bitfinex v2
+        TODO: Might require a separate asset dictionary
+
+        :param assets:
+        :return:
+        """
+
+        v2_symbols = []
+        for asset in assets:
+            v2_symbols.append(self._get_v2_symbol(asset))
+
+        return v2_symbols
+
+    def _create_order(self, order_status):
+        """
+        Create a Catalyst order object from a Bitfinex order dictionary
+        :param order_status:
+        :return: Order
+        """
+        if order_status['is_cancelled']:
+            status = ORDER_STATUS.CANCELLED
+        elif not order_status['is_live']:
+            log.info('found executed order {}'.format(order_status))
+            status = ORDER_STATUS.FILLED
+        else:
+            status = ORDER_STATUS.OPEN
+
+        amount = float(order_status['original_amount'])
+        filled = float(order_status['executed_amount'])
+
+        if order_status['side'] == 'sell':
+            amount = -amount
+            filled = -filled
+
+        price = float(order_status['price'])
+        order_type = order_status['type']
+
+        stop_price = None
+        limit_price = None
+
+        # TODO: is this comprehensive enough?
+        if order_type.endswith('limit'):
+            limit_price = price
+        elif order_type.endswith('stop'):
+            stop_price = price
+
+        executed_price = float(order_status['avg_execution_price'])
+
+        # TODO: bitfinex does not specify comission.
+        # I could calculate it but not sure if it's worth it.
+        commission = None
+
+        date = pd.Timestamp.utcfromtimestamp(float(order_status['timestamp']))
+        date = pytz.utc.localize(date)
+        order = Order(
+            dt=date,
+            asset=self.assets[order_status['symbol']],
+            amount=amount,
+            stop=stop_price,
+            limit=limit_price,
+            filled=filled,
+            id=str(order_status['id']),
+            commission=commission
+        )
+        order.status = status
+
+        return order, executed_price
+
+    def get_balances(self):
+        log.debug('retrieving wallets balances')
+        try:
+            self.ask_request()
+            response = self._request('balances', None)
+            balances = response.json()
+        except Exception as e:
+            raise ExchangeRequestError(error=e)
+
+        if 'message' in balances:
+            raise ExchangeRequestError(
+                error='unable to fetch balance {}'.format(balances['message'])
+            )
+
+        std_balances = dict()
+        for balance in balances:
+            currency = balance['currency'].lower()
+            std_balances[currency] = float(balance['available'])
+
+        return std_balances
+
+    @property
+    def account(self):
+        account = Account()
+
+        account.settled_cash = None
+        account.accrued_interest = None
+        account.buying_power = None
+        account.equity_with_loan = None
+        account.total_positions_value = None
+        account.total_positions_exposure = None
+        account.regt_equity = None
+        account.regt_margin = None
+        account.initial_margin_requirement = None
+        account.maintenance_margin_requirement = None
+        account.available_funds = None
+        account.excess_liquidity = None
+        account.cushion = None
+        account.day_trades_remaining = None
+        account.leverage = None
+        account.net_leverage = None
+        account.net_liquidation = None
+
+        return account
+
+    @property
+    def time_skew(self):
+        # TODO: research the time skew conditions
+        return pd.Timedelta('0s')
+
+    def get_account(self):
+        # TODO: fetch account data and keep in cache
+        return None
+
+    def get_candles(self, freq, assets, bar_count=None,
+                    start_dt=None, end_dt=None):
+        """
+        Retrieve OHLVC candles from Bitfinex
+
+        :param data_frequency:
+        :param assets:
+        :param bar_count:
+        :return:
+
+        Available Frequencies
+        ---------------------
+        '1m', '5m', '15m', '30m', '1h', '3h', '6h', '12h', '1D', '7D', '14D',
+         '1M'
+        """
+        log.debug(
+            'retrieving {bars} {freq} candles on {exchange} from '
+            '{end_dt} for markets {symbols}, '.format(
+                bars=bar_count,
+                freq=freq,
+                exchange=self.name,
+                end_dt=end_dt,
+                symbols=get_symbols_string(assets)
+            )
+        )
+
+        allowed_frequencies = ['1T', '5T', '15T', '30T', '60T', '180T',
+                               '360T', '720T', '1D', '7D', '14D', '30D']
+        if freq not in allowed_frequencies:
+            raise InvalidHistoryFrequencyError(frequency=freq)
+
+        freq_match = re.match(r'([0-9].*)(T|H|D)', freq, re.M | re.I)
+        if freq_match:
+            number = int(freq_match.group(1))
+            unit = freq_match.group(2)
+
+            if unit == 'T':
+                if number in [60, 180, 360, 720]:
+                    number = number / 60
+                    converted_unit = 'h'
+                else:
+                    converted_unit = 'm'
+            else:
+                converted_unit = unit
+
+            frequency = '{}{}'.format(number, converted_unit)
+
+        else:
+            raise InvalidHistoryFrequencyError(frequency=freq)
+
+        # Making sure that assets are iterable
+        asset_list = [assets] if isinstance(assets, TradingPair) else assets
+        ohlc_map = dict()
+        for asset in asset_list:
+            symbol = self._get_v2_symbol(asset)
+            url = '{url}/v2/candles/trade:{frequency}:{symbol}'.format(
+                url=self.url,
+                frequency=frequency,
+                symbol=symbol
+            )
+
+            if bar_count:
+                is_list = True
+                url += '/hist?limit={}'.format(int(bar_count))
+
+                def get_ms(date):
+                    epoch = datetime.datetime.utcfromtimestamp(0)
+                    epoch = epoch.replace(tzinfo=pytz.UTC)
+
+                    return (date - epoch).total_seconds() * 1000.0
+
+                if start_dt is not None:
+                    start_ms = get_ms(start_dt)
+                    url += '&start={0:f}'.format(start_ms)
+
+                if end_dt is not None:
+                    end_ms = get_ms(end_dt)
+                    url += '&end={0:f}'.format(end_ms)
+
+            else:
+                is_list = False
+                url += '/last'
+
+            try:
+                self.ask_request()
+                response = requests.get(url)
+            except Exception as e:
+                raise ExchangeRequestError(error=e)
+
+            if 'error' in response.content:
+                raise ExchangeRequestError(
+                    error='Unable to retrieve candles: {}'.format(
+                        response.content)
+                )
+
+            candles = response.json()
+
+            def ohlc_from_candle(candle):
+                last_traded = pd.Timestamp.utcfromtimestamp(
+                    candle[0] / 1000.0)
+                last_traded = last_traded.replace(tzinfo=pytz.UTC)
+                ohlc = dict(
+                    open=np.float64(candle[1]),
+                    high=np.float64(candle[3]),
+                    low=np.float64(candle[4]),
+                    close=np.float64(candle[2]),
+                    volume=np.float64(candle[5]),
+                    price=np.float64(candle[2]),
+                    last_traded=last_traded
+                )
+                return ohlc
+
+            if is_list:
+                ohlc_bars = []
+                # We can to list candles from old to new
+                for candle in reversed(candles):
+                    ohlc = ohlc_from_candle(candle)
+                    ohlc_bars.append(ohlc)
+
+                ohlc_map[asset] = ohlc_bars
+
+            else:
+                ohlc = ohlc_from_candle(candles)
+                ohlc_map[asset] = ohlc
+
+        return ohlc_map[assets] \
+            if isinstance(assets, TradingPair) else ohlc_map
+
+    def create_order(self, asset, amount, is_buy, style):
+        """
+        Creating order on the exchange.
+
+        :param asset:
+        :param amount:
+        :param is_buy:
+        :param style:
+        :return:
+        """
+        exchange_symbol = self.get_symbol(asset)
+        if isinstance(style, ExchangeLimitOrder) \
+                or isinstance(style, ExchangeStopLimitOrder):
+            price = style.get_limit_price(is_buy)
+            order_type = 'limit'
+
+        elif isinstance(style, ExchangeStopOrder):
+            price = style.get_stop_price(is_buy)
+            order_type = 'stop'
+
+        else:
+            raise InvalidOrderStyle(exchange=self.name,
+                                    style=style.__class__.__name__)
+
+        req = dict(
+            symbol=exchange_symbol,
+            amount=str(float(abs(amount))),
+            price="{:.20f}".format(float(price)),
+            side='buy' if is_buy else 'sell',
+            type='exchange ' + order_type,  # TODO: support margin trades
+            exchange=self.name,
+            is_hidden=False,
+            is_postonly=False,
+            use_all_available=0,
+            ocoorder=False,
+            buy_price_oco=0,
+            sell_price_oco=0
+        )
+
+        date = pd.Timestamp.utcnow()
+        try:
+            self.ask_request()
+            response = self._request('order/new', req)
+            order_status = response.json()
+        except Exception as e:
+            raise ExchangeRequestError(error=e)
+
+        if 'message' in order_status:
+            raise ExchangeRequestError(
+                error='unable to create Bitfinex order {}'.format(
+                    order_status['message'])
+            )
+
+        order_id = str(order_status['id'])
+        order = Order(
+            dt=date,
+            asset=asset,
+            amount=amount,
+            stop=style.get_stop_price(is_buy),
+            limit=style.get_limit_price(is_buy),
+            id=order_id
+        )
+
+        return order
+
+    def get_open_orders(self, asset=None):
+        """Retrieve all of the current open orders.
+
+        Parameters
+        ----------
+        asset : Asset
+            If passed and not None, return only the open orders for the given
+            asset instead of all open orders.
+
+        Returns
+        -------
+        open_orders : dict[list[Order]] or list[Order]
+            If no asset is passed this will return a dict mapping Assets
+            to a list containing all the open orders for the asset.
+            If an asset is passed then this will return a list of the open
+            orders for this asset.
+        """
+        try:
+            self.ask_request()
+            response = self._request('orders', None)
+            order_statuses = response.json()
+        except Exception as e:
+            raise ExchangeRequestError(error=e)
+
+        if 'message' in order_statuses:
+            raise ExchangeRequestError(
+                error='Unable to retrieve open orders: {}'.format(
+                    order_statuses['message'])
+            )
+
+        orders = []
+        for order_status in order_statuses:
+            order, executed_price = self._create_order(order_status)
+            if asset is None or asset == order.sid:
+                orders.append(order)
+
+        return orders
+
+    def get_order(self, order_id):
+        """Lookup an order based on the order id returned from one of the
+        order functions.
+
+        Parameters
+        ----------
+        order_id : str
+            The unique identifier for the order.
+
+        Returns
+        -------
+        order : Order
+            The order object.
+        """
+        try:
+            self.ask_request()
+            response = self._request(
+                'order/status', {'order_id': int(order_id)})
+            order_status = response.json()
+        except Exception as e:
+            raise ExchangeRequestError(error=e)
+
+        if 'message' in order_status:
+            raise ExchangeRequestError(
+                error='Unable to retrieve order status: {}'.format(
+                    order_status['message'])
+            )
+        return self._create_order(order_status)
+
+    def cancel_order(self, order_param):
+        """Cancel an open order.
+
+        Parameters
+        ----------
+        order_param : str or Order
+            The order_id or order object to cancel.
+        """
+        order_id = order_param.id \
+            if isinstance(order_param, Order) else order_param
+
+        try:
+            self.ask_request()
+            response = self._request('order/cancel', {'order_id': order_id})
+            status = response.json()
+        except Exception as e:
+            raise ExchangeRequestError(error=e)
+
+        if 'message' in status:
+            raise OrderCancelError(
+                order_id=order_id,
+                exchange=self.name,
+                error=status['message']
+            )
+
+    def tickers(self, assets):
+        """
+        Fetch ticket data for assets
+        https://docs.bitfinex.com/v2/reference#rest-public-tickers
+
+        :param assets:
+        :return:
+        """
+        symbols = self._get_v2_symbols(assets)
+        log.debug('fetching tickers {}'.format(symbols))
+
+        try:
+            self.ask_request()
+            response = requests.get(
+                '{url}/v2/tickers?symbols={symbols}'.format(
+                    url=self.url,
+                    symbols=','.join(symbols),
+                )
+            )
+        except Exception as e:
+            raise ExchangeRequestError(error=e)
+
+        if 'error' in response.content:
+            raise ExchangeRequestError(
+                error='Unable to retrieve tickers: {}'.format(
+                    response.content)
+            )
+
+        try:
+            tickers = response.json()
+        except Exception as e:
+            raise ExchangeRequestError(error=e)
+
+        ticks = dict()
+        for index, ticker in enumerate(tickers):
+            if not len(ticker) == 11:
+                raise ExchangeRequestError(
+                    error='Invalid ticker in response: {}'.format(ticker)
+                )
+
+            ticks[assets[index]] = dict(
+                timestamp=pd.Timestamp.utcnow(),
+                bid=ticker[1],
+                ask=ticker[3],
+                last_price=ticker[7],
+                low=ticker[10],
+                high=ticker[9],
+                volume=ticker[8],
+            )
+
+        log.debug('got tickers {}'.format(ticks))
+        return ticks
+
+    def generate_symbols_json(self, filename=None, source_dates=False):
+        symbol_map = {}
+
+        if not source_dates:
+            fn, r = download_exchange_symbols(self.name)
+            with open(fn) as data_file:
+                cached_symbols = json.load(data_file)
+
+        response = self._request('symbols', None)
+
+        for symbol in response.json():
+            if (source_dates):
+                start_date = self.get_symbol_start_date(symbol)
+            else:
+                try:
+                    start_date = cached_symbols[symbol]['start_date']
+                except KeyError:
+                    start_date = time.strftime('%Y-%m-%d')
+
+            try:
+                end_daily = cached_symbols[symbol]['end_daily']
+            except KeyError:
+                end_daily = 'N/A'
+
+            try:
+                end_minute = cached_symbols[symbol]['end_minute']
+            except KeyError:
+                end_minute = 'N/A'
+
+            symbol_map[symbol] = dict(
+                symbol=symbol[:-3] + '_' + symbol[-3:],
+                start_date=start_date,
+                end_daily=end_daily,
+                end_minute=end_minute,
+            )
+
+        if (filename is None):
+            filename = get_exchange_symbols_filename(self.name)
+
+        with open(filename, 'w') as f:
+            json.dump(symbol_map, f, sort_keys=True, indent=2,
+                      separators=(',', ':'))
+
+    def get_symbol_start_date(self, symbol):
+
+        print(symbol)
+        symbol_v2 = 't' + symbol.upper()
+
+        """
+            For each symbol we retrieve candles with Monhtly resolution
+            We get the first month, and query again with daily resolution
+            around that date, and we get the first date
+        """
+        url = '{url}/v2/candles/trade:1M:{symbol}/hist'.format(
+            url=self.url,
+            symbol=symbol_v2
+        )
+
+        try:
+            self.ask_request()
+            response = requests.get(url)
+        except Exception as e:
+            raise ExchangeRequestError(error=e)
+
+        """
+            If we don't get any data back for our monthly-resolution query
+            it means that symbol started trading less than a month ago, so
+            arbitrarily set the ref. date to 15 days ago to be safe with
+            +/- 31 days
+        """
+        if (len(response.json())):
+            startmonth = response.json()[-1][0]
+        else:
+            startmonth = int((time.time() - 15 * 24 * 3600) * 1000)
+
+        """
+            Query again with daily resolution setting the start and end around
+            the startmonth we got above. Avoid end dates greater than
+            now: time.time()
+        """
+        url = ('{url}/v2/candles/trade:1D:{symbol}/hist?start={start}'
+               '&end={end}').format(
+            url=self.url,
+            symbol=symbol_v2,
+            start=startmonth - 3600 * 24 * 31 * 1000,
+            end=min(startmonth + 3600 * 24 * 31 * 1000,
+                    int(time.time() * 1000)))
+
+        try:
+            self.ask_request()
+            response = requests.get(url)
+        except Exception as e:
+            raise ExchangeRequestError(error=e)
+
+        return time.strftime('%Y-%m-%d',
+                             time.gmtime(int(response.json()[-1][0] / 1000)))
+
+    def get_orderbook(self, asset, order_type='all', limit=100):
+        exchange_symbol = asset.exchange_symbol
+        try:
+            self.ask_request()
+            # TODO: implement limit
+            response = self._request(
+                'book/{}'.format(exchange_symbol), None)
+            data = response.json()
+
+        except Exception as e:
+            raise ExchangeRequestError(error=e)
+
+        # TODO: filter by type
+        result = dict()
+        for order_type in data:
+            result[order_type] = []
+
+            for entry in data[order_type]:
+                result[order_type].append(dict(
+                    rate=float(entry['price']),
+                    quantity=float(entry['amount'])
+                ))
+
+        return result

catalyst/exchange/bitfinex/symbols.json

@@ -0,0 +1,127 @@
+{
+  "neobtc": {
+    "symbol": "neo_btc",
+    "start_date": "2017-09-07",
+    "precision": 5
+  },
+  "neousd": {
+    "symbol": "neo_usd",
+    "start_date": "2017-09-07"
+  },
+  "neoeth": {
+    "symbol": "neo_eth",
+    "start_date": "2017-09-07"
+  },
+  "btcusd": {
+    "symbol": "btc_usd",
+    "start_date": "2010-01-01"
+  },
+  "bchusd": {
+    "symbol": "bch_usd",
+    "start_date": "2010-01-01"
+  },
+  "ltcusd": {
+    "symbol": "ltc_usd",
+    "start_date": "2010-01-01"
+  },
+  "ltcbtc": {
+    "symbol": "ltc_btc",
+    "start_date": "2010-01-01"
+  },
+  "ethusd": {
+    "symbol": "eth_usd",
+    "start_date": "2017-01-01"
+  },
+  "ethbtc": {
+    "symbol": "eth_btc",
+    "start_date": "2017-01-01"
+  },
+  "etcbtc": {
+    "symbol": "etc_btc",
+    "start_date": "2017-01-01"
+  },
+  "etcusd": {
+    "symbol": "etc_usd",
+    "start_date": "2017-01-01"
+  },
+  "rrtusd": {
+    "symbol": "rrt_usd",
+    "start_date": "2010-01-01"
+  },
+  "rrtbtc": {
+    "symbol": "rrt_btc",
+    "start_date": "2010-01-01"
+  },
+  "zecusd": {
+    "symbol": "zec_usd",
+    "start_date": "2010-01-01"
+  },
+  "zecbtc": {
+    "symbol": "zec_btc",
+    "start_date": "2010-01-01"
+  },
+  "xmrusd": {
+    "symbol": "xmr_usd",
+    "start_date": "2010-01-01"
+  },
+  "xmrbtc": {
+    "symbol": "xmr_btc",
+    "start_date": "2010-01-01"
+  },
+  "dshusd": {
+    "symbol": "dsh_usd",
+    "start_date": "2010-01-01"
+  },
+  "dshbtc": {
+    "symbol": "dsh_btc",
+    "start_date": "2010-01-01"
+  },
+  "bccbtc": {
+    "symbol": "bcc_btc",
+    "start_date": "2010-01-01"
+  },
+  "bcubtc": {
+    "symbol": "bcu_btc",
+    "start_date": "2010-01-01"
+  },
+  "bccusd": {
+    "symbol": "bcc_usd",
+    "start_date": "2010-01-01"
+  },
+  "bcuusd": {
+    "symbol": "bcu_usd",
+    "start_date": "2010-01-01"
+  },
+  "xrpusd": {
+    "symbol": "xrp_usd",
+    "start_date": "2010-01-01"
+  },
+  "xrpbtc": {
+    "symbol": "xrp_btc",
+    "start_date": "2010-01-01"
+  },
+  "iotusd": {
+    "symbol": "iot_usd",
+    "start_date": "2010-01-01"
+  },
+  "iotbtc": {
+    "symbol": "iot_btc",
+    "start_date": "2010-01-01"
+  },
+  "ioteth": {
+    "symbol": "iot_eth",
+    "start_date": "2010-01-01"
+  },
+  "eosusd": {
+    "symbol": "eos_usd",
+    "start_date": "2010-01-01"
+  },
+  "eosbtc": {
+    "symbol": "eos_btc",
+    "start_date": "2010-01-01"
+  },
+  "eoseth": {
+    "symbol": "eos_eth",
+    "start_date": "2010-01-01"
+  }
+}

catalyst/exchange/bittrex/bittrex.py

@@ -0,0 +1,417 @@
+import json
+import time
+
+import pandas as pd
+from catalyst.assets._assets import TradingPair
+from logbook import Logger
+from six.moves import urllib
+
+from catalyst.constants import LOG_LEVEL
+from catalyst.exchange.bittrex.bittrex_api import Bittrex_api
+from catalyst.exchange.exchange import Exchange
+from catalyst.exchange.exchange_bundle import ExchangeBundle
+from catalyst.exchange.exchange_errors import InvalidHistoryFrequencyError, \
+    ExchangeRequestError, InvalidOrderStyle, OrderNotFound, OrderCancelError, \
+    CreateOrderError
+from catalyst.exchange.exchange_utils import get_exchange_symbols_filename, \
+    download_exchange_symbols, get_symbols_string
+from catalyst.finance.execution import LimitOrder, StopLimitOrder
+from catalyst.finance.order import Order, ORDER_STATUS
+
+# TODO: consider using this: https://github.com/mondeja/bittrex_v2
+from catalyst.utils.deprecate import deprecated
+
+log = Logger('Bittrex', level=LOG_LEVEL)
+
+URL2 = 'https://bittrex.com/Api/v2.0'
+
+
+@deprecated
+class Bittrex(Exchange):
+    def __init__(self, key, secret, base_currency, portfolio=None):
+        self.api = Bittrex_api(key=key, secret=secret)
+        self.name = 'bittrex'
+        self.color = 'blue'
+        self.base_currency = base_currency
+        self._portfolio = portfolio
+
+        self.num_candles_limit = 2000
+
+        # Not sure what the rate limit is but trying to play it safe
+        # https://bitcoin.stackexchange.com/questions/53778/bittrex-api-rate-limit
+        self.max_requests_per_minute = 60
+        self.request_cpt = dict()
+
+        self.minute_writer = None
+        self.minute_reader = None
+
+        self.assets = dict()
+        self.load_assets()
+
+        self.local_assets = dict()
+        self.load_assets(is_local=True)
+
+        self.bundle = ExchangeBundle(self.name)
+
+    @property
+    def account(self):
+        pass
+
+    @property
+    def time_skew(self):
+        # TODO: research the time skew conditions
+        return pd.Timedelta('0s')
+
+    def sanitize_curency_symbol(self, exchange_symbol):
+        """
+        Helper method used to build the universal pair.
+        Include any symbol mapping here if appropriate.
+
+        :param exchange_symbol:
+        :return universal_symbol:
+        """
+        return exchange_symbol.lower()
+
+    def get_balances(self):
+        balances = self.api.getbalances()
+        try:
+            log.debug('retrieving wallet balances')
+            self.ask_request()
+
+        except Exception as e:
+            raise ExchangeRequestError(error=e)
+
+        std_balances = dict()
+        try:
+            for balance in balances:
+                currency = balance['Currency'].lower()
+                std_balances[currency] = balance['Available']
+
+        except TypeError:
+            raise ExchangeRequestError(error=balances)
+
+        return std_balances
+
+    def create_order(self, asset, amount, is_buy, style):
+        log.info('creating {} order'.format('buy' if is_buy else 'sell'))
+        exchange_symbol = self.get_symbol(asset)
+
+        if isinstance(style, LimitOrder) or isinstance(style, StopLimitOrder):
+            if isinstance(style, StopLimitOrder):
+                log.warn('{} will ignore the stop price'.format(self.name))
+
+            price = style.get_limit_price(is_buy)
+            try:
+                self.ask_request()
+                if is_buy:
+                    order_status = self.api.buylimit(exchange_symbol, amount,
+                                                     price)
+                else:
+                    order_status = self.api.selllimit(exchange_symbol,
+                                                      abs(amount), price)
+            except Exception as e:
+                raise ExchangeRequestError(error=e)
+
+            if 'uuid' in order_status:
+                order_id = order_status['uuid']
+                order = Order(
+                    dt=pd.Timestamp.utcnow(),
+                    asset=asset,
+                    amount=amount,
+                    stop=style.get_stop_price(is_buy),
+                    limit=style.get_limit_price(is_buy),
+                    id=order_id
+                )
+                return order
+            else:
+                if order_status == 'INSUFFICIENT_FUNDS':
+                    log.warn('not enough funds to create order')
+                    return None
+                elif order_status == 'DUST_TRADE_DISALLOWED_MIN_VALUE_50K_SAT':
+                    log.warn('Your order is too small, order at least 50K'
+                             ' Satoshi')
+                    return None
+                else:
+                    raise CreateOrderError(
+                        exchange=self.name,
+                        error=order_status
+                    )
+        else:
+            raise InvalidOrderStyle(exchange=self.name,
+                                    style=style.__class__.__name__)
+
+    def get_open_orders(self, asset):
+        symbol = self.get_symbol(asset)
+        try:
+            self.ask_request()
+            open_orders = self.api.getopenorders(symbol)
+        except Exception as e:
+            raise ExchangeRequestError(error=e)
+
+        orders = list()
+        for order_status in open_orders:
+            order = self._create_order(order_status)
+            orders.append(order)
+
+        return orders
+
+    def _create_order(self, order_status):
+        log.info(
+            'creating catalyst order from Bittrex {}'.format(order_status))
+        if order_status['CancelInitiated']:
+            status = ORDER_STATUS.CANCELLED
+        elif order_status['Closed'] is not None:
+            status = ORDER_STATUS.FILLED
+        else:
+            status = ORDER_STATUS.OPEN
+
+        date = pd.to_datetime(order_status['Opened'], utc=True)
+        amount = order_status['Quantity']
+        filled = amount - order_status['QuantityRemaining']
+        order = Order(
+            dt=date,
+            asset=self.assets[order_status['Exchange']],
+            amount=amount,
+            stop=None,  # Not yet supported by Bittrex
+            limit=order_status['Limit'],
+            filled=filled,
+            id=order_status['OrderUuid'],
+            commission=order_status['CommissionPaid']
+        )
+        order.status = status
+
+        executed_price = order_status['PricePerUnit']
+
+        return order, executed_price
+
+    def get_order(self, order_id):
+        log.info('retrieving order {}'.format(order_id))
+        try:
+            self.ask_request()
+            order_status = self.api.getorder(order_id)
+        except Exception as e:
+            raise ExchangeRequestError(error=e)
+
+        if order_status is None:
+            raise OrderNotFound(order_id=order_id, exchange=self.name)
+
+        return self._create_order(order_status)
+
+    def cancel_order(self, order_param):
+        order_id = order_param.id \
+            if isinstance(order_param, Order) else order_param
+        log.info('cancelling order {}'.format(order_id))
+
+        try:
+            self.ask_request()
+            status = self.api.cancel(order_id)
+        except Exception as e:
+            raise ExchangeRequestError(error=e)
+
+        if 'message' in status:
+            raise OrderCancelError(
+                order_id=order_id,
+                exchange=self.name,
+                error=status['message']
+            )
+
+    def get_candles(self, freq, assets, bar_count=None,
+                    start_dt=None, end_dt=None):
+        """
+        Supported Intervals
+        -------------------
+        day, oneMin, fiveMin, thirtyMin, hour
+
+        :param freq:
+        :param assets:
+        :param bar_count:
+        :param start_dt
+        :param end_dt
+        :return:
+        """
+
+        # TODO: this has no effect at the moment
+        if end_dt is None:
+            end_dt = pd.Timestamp.utcnow()
+
+        log.debug(
+            'retrieving {bars} {freq} candles on {exchange} from '
+            '{end_dt} for markets {symbols}, '.format(
+                bars=bar_count,
+                freq=freq,
+                exchange=self.name,
+                end_dt=end_dt,
+                symbols=get_symbols_string(assets)
+            )
+        )
+
+        if freq == '1T':
+            frequency = 'oneMin'
+        elif freq == '5T':
+            frequency = 'fiveMin'
+        elif freq == '30T':
+            frequency = 'thirtyMin'
+        elif freq == '60T':
+            frequency = 'hour'
+        elif freq == '1D':
+            frequency = 'day'
+        else:
+            raise InvalidHistoryFrequencyError(frequency=freq)
+
+        # Making sure that assets are iterable
+        asset_list = [assets] if isinstance(assets, TradingPair) else assets
+        for asset in asset_list:
+            end = int(time.mktime(end_dt.timetuple()))
+            url = '{url}/pub/market/GetTicks?marketName={symbol}' \
+                  '&tickInterval={frequency}&_={end}'.format(
+                    url=URL2,
+                    symbol=self.get_symbol(asset),
+                    frequency=frequency,
+                    end=end, )
+
+            try:
+                data = json.loads(urllib.request.urlopen(url).read().decode())
+            except Exception as e:
+                raise ExchangeRequestError(error=e)
+
+            if data['message']:
+                raise ExchangeRequestError(
+                    error='Unable to fetch candles {}'.format(data['message'])
+                )
+
+            candles = data['result']
+
+            def ohlc_from_candle(candle):
+                ohlc = dict(
+                    open=candle['O'],
+                    high=candle['H'],
+                    low=candle['L'],
+                    close=candle['C'],
+                    volume=candle['V'],
+                    price=candle['C'],
+                    last_traded=pd.to_datetime(candle['T'], utc=True)
+                )
+                return ohlc
+
+            ordered_candles = list(reversed(candles))
+            ohlc_map = dict()
+            if bar_count is None:
+                ohlc_map[asset] = ohlc_from_candle(ordered_candles[0])
+            else:
+                # TODO: optimize
+                ohlc_bars = []
+                for candle in ordered_candles[:bar_count]:
+                    ohlc = ohlc_from_candle(candle)
+                    ohlc_bars.append(ohlc)
+
+                ohlc_map[asset] = ohlc_bars
+
+        return ohlc_map[assets] \
+            if isinstance(assets, TradingPair) else ohlc_map
+
+    def tickers(self, assets):
+        """
+        As of v1.1, Bittrex only allows one ticker at the time.
+        So we have to make multiple calls to fetch multiple assets.
+
+        :param assets:
+        :return:
+        """
+        log.info('retrieving tickers')
+
+        ticks = dict()
+        for asset in assets:
+            symbol = self.get_symbol(asset)
+            try:
+                self.ask_request()
+                ticker = self.api.getticker(symbol)
+            except Exception as e:
+                raise ExchangeRequestError(error=e)
+
+            # TODO: catch invalid ticker
+            ticks[asset] = dict(
+                timestamp=pd.Timestamp.utcnow(),
+                bid=ticker['Bid'],
+                ask=ticker['Ask'],
+                last_price=ticker['Last']
+            )
+
+        log.debug('got tickers {}'.format(ticks))
+        return ticks
+
+    def get_account(self):
+        log.info('retrieving account data')
+        pass
+
+    def generate_symbols_json(self, filename=None):
+        symbol_map = {}
+
+        fn, r = download_exchange_symbols(self.name)
+        with open(fn) as data_file:
+            cached_symbols = json.load(data_file)
+
+        markets = self.api.getmarkets()
+        for market in markets:
+            exchange_symbol = market['MarketName']
+            symbol = '{market}_{base}'.format(
+                market=self.sanitize_curency_symbol(market['MarketCurrency']),
+                base=self.sanitize_curency_symbol(market['BaseCurrency'])
+            )
+
+            try:
+                end_daily = cached_symbols[exchange_symbol]['end_daily']
+            except KeyError:
+                end_daily = 'N/A'
+
+            try:
+                end_minute = cached_symbols[exchange_symbol]['end_minute']
+            except KeyError:
+                end_minute = 'N/A'
+
+            symbol_map[exchange_symbol] = dict(
+                symbol=symbol,
+                start_date=pd.to_datetime(market['Created'],
+                                          utc=True).strftime("%Y-%m-%d"),
+                end_daily=end_daily,
+                end_minute=end_minute,
+            )
+
+        if (filename is None):
+            filename = get_exchange_symbols_filename(self.name)
+
+        with open(filename, 'w') as f:
+            json.dump(symbol_map, f, sort_keys=True, indent=2,
+                      separators=(',', ':'))
+
+    def get_orderbook(self, asset, order_type='all', limit=100):
+        if order_type == 'all':
+            order_type = 'both'
+        elif order_type == 'bid':
+            order_type = 'buy'
+        elif order_type == 'ask':
+            order_type = 'sell'
+        else:
+            raise ValueError('invalid type')
+
+        exchange_symbol = asset.exchange_symbol
+        data = self.api.getorderbook(
+            market=exchange_symbol,
+            type=order_type,
+            depth=100
+        )
+
+        result = dict()
+        for exchange_type in data:
+            if exchange_type == 'buy':
+                order_type = 'bids'
+            elif exchange_type == 'sell':
+                order_type = 'asks'
+
+            result[order_type] = []
+            for entry in data[exchange_type]:
+                result[order_type].append(dict(
+                    rate=entry['Rate'],
+                    quantity=entry['Quantity']
+                ))
+
+        return result

catalyst/exchange/bittrex/bittrex_api.py

@@ -0,0 +1,132 @@
+#!/usr/bin/env python
+import json
+import time
+import hmac
+import hashlib
+import ssl
+
+# Workaround for backwards compatibility
+# https://stackoverflow.com/questions/3745771/urllib-request-in-python-2-7
+from six.moves import urllib
+
+urlopen = urllib.request.urlopen
+
+
+class Bittrex_api(object):
+    def __init__(self, key, secret):
+        self.key = key
+        self.secret = secret
+        self.public = ['getmarkets', 'getcurrencies', 'getticker',
+                       'getmarketsummaries', 'getmarketsummary',
+                       'getorderbook', 'getmarkethistory']
+        self.market = ['buylimit', 'buymarket', 'selllimit', 'sellmarket',
+                       'cancel', 'getopenorders']
+        self.account = ['getbalances', 'getbalance', 'getdepositaddress',
+                        'withdraw', 'getorder', 'getorderhistory',
+                        'getwithdrawalhistory', 'getdeposithistory']
+
+    def query(self, method, values={}):
+        if method in self.public:
+            url = 'https://bittrex.com/api/v1.1/public/'
+        elif method in self.market:
+            url = 'https://bittrex.com/api/v1.1/market/'
+        elif method in self.account:
+            url = 'https://bittrex.com/api/v1.1/account/'
+        else:
+            return 'Something went wrong, sorry.'
+
+        url += method + '?' + urllib.parse.urlencode(values)
+
+        if method not in self.public:
+            url += '&apikey=' + self.key
+            url += '&nonce=' + str(int(time.time()))
+
+            signature = hmac.new(self.secret.encode('utf-8'),
+                                 url.encode('utf-8'),
+                                 hashlib.sha512).hexdigest()
+            headers = {'apisign': signature}
+        else:
+            headers = {}
+
+        req = urllib.request.Request(url, headers=headers)
+        response = json.loads(urlopen(
+            req, context=ssl._create_unverified_context()).read())
+
+        if response["result"]:
+            return response["result"]
+        else:
+            return response["message"]
+
+    def getmarkets(self):
+        return self.query('getmarkets')
+
+    def getcurrencies(self):
+        return self.query('getcurrencies')
+
+    def getticker(self, market):
+        return self.query('getticker', {'market': market})
+
+    def getmarketsummaries(self):
+        return self.query('getmarketsummaries')
+
+    def getmarketsummary(self, market):
+        return self.query('getmarketsummary', {'market': market})
+
+    def getorderbook(self, market, type, depth=20):
+        return self.query('getorderbook',
+                          {'market': market, 'type': type, 'depth': depth})
+
+    def getmarkethistory(self, market, count=20):
+        return self.query('getmarkethistory',
+                          {'market': market, 'count': count})
+
+    def buylimit(self, market, quantity, rate):
+        return self.query('buylimit', {'market': market, 'quantity': quantity,
+                                       'rate': rate})
+
+    def buymarket(self, market, quantity):
+        return self.query('buymarket',
+                          {'market': market, 'quantity': quantity})
+
+    def selllimit(self, market, quantity, rate):
+        return self.query('selllimit', {'market': market, 'quantity': quantity,
+                                        'rate': rate})
+
+    def sellmarket(self, market, quantity):
+        return self.query('sellmarket',
+                          {'market': market, 'quantity': quantity})
+
+    def cancel(self, uuid):
+        return self.query('cancel', {'uuid': uuid})
+
+    def getopenorders(self, market):
+        return self.query('getopenorders', {'market': market})
+
+    def getbalances(self):
+        return self.query('getbalances')
+
+    def getbalance(self, currency):
+        return self.query('getbalance', {'currency': currency})
+
+    def getdepositaddress(self, currency):
+        return self.query('getdepositaddress', {'currency': currency})
+
+    def withdraw(self, currency, quantity, address):
+        return self.query('withdraw',
+                          {'currency': currency, 'quantity': quantity,
+                           'address': address})
+
+    def getorder(self, uuid):
+        return self.query('getorder', {'uuid': uuid})
+
+    def getorderhistory(self, market, count):
+        return self.query('getorderhistory',
+                          {'market': market, 'count': count})
+
+    def getwithdrawalhistory(self, currency, count):
+        return self.query('getwithdrawalhistory',
+                          {'currency': currency, 'count': count})
+
+    def getdeposithistory(self, currency, count):
+        return self.query('getdeposithistory',
+                          {'currency': currency, 'count': count})

catalyst/exchange/bittrex/extensions-example.py

@@ -0,0 +1,7 @@
+from catalyst.data.bundles import register
+from catalyst.exchange.exchange_bundle import exchange_bundle
+
+symbols = (
+    'neo_btc',
+)
+register('exchange_bitfinex', exchange_bundle('bitfinex', symbols))

catalyst/exchange/bundle_utils.py

@@ -0,0 +1,358 @@
+import calendar
+import os
+import tarfile
+from datetime import timedelta, datetime, date
+
+import numpy as np
+import pandas as pd
+import pytz
+
+from catalyst.data.bundles.core import download_without_progress
+from catalyst.exchange.exchange_utils import get_exchange_bundles_folder
+
+EXCHANGE_NAMES = ['bitfinex', 'bittrex', 'poloniex']
+API_URL = 'http://data.enigma.co/api/v1'
+
+
+def get_date_from_ms(ms):
+    """
+    The date from the number of miliseconds from the epoch.
+
+    Parameters
+    ----------
+    ms: int
+
+    Returns
+    -------
+    datetime
+
+    """
+    return datetime.fromtimestamp(ms / 1000.0)
+
+
+def get_seconds_from_date(date):
+    """
+    The number of seconds from the epoch.
+
+    Parameters
+    ----------
+    date: datetime
+
+    Returns
+    -------
+    int
+
+    """
+    epoch = datetime.utcfromtimestamp(0)
+    epoch = epoch.replace(tzinfo=pytz.UTC)
+
+    return int((date - epoch).total_seconds())
+
+
+def get_bcolz_chunk(exchange_name, symbol, data_frequency, period):
+    """
+    Download and extract a bcolz bundle.
+
+    Parameters
+    ----------
+    exchange_name: str
+    symbol: str
+    data_frequency: str
+    period: str
+
+    Returns
+    -------
+    str
+        Filename: bitfinex-daily-neo_eth-2017-10.tar.gz
+
+    """
+    root = get_exchange_bundles_folder(exchange_name)
+    name = '{exchange}-{frequency}-{symbol}-{period}'.format(
+        exchange=exchange_name,
+        frequency=data_frequency,
+        symbol=symbol,
+        period=period
+    )
+    path = os.path.join(root, name)
+
+    if not os.path.isdir(path):
+        url = 'https://s3.amazonaws.com/enigmaco/catalyst-bundles/' \
+              'exchange-{exchange}/{name}.tar.gz'.format(
+                exchange=exchange_name,
+                name=name)
+
+        bytes = download_without_progress(url)
+        with tarfile.open('r', fileobj=bytes) as tar:
+            tar.extractall(path)
+
+    return path
+
+
+def get_delta(periods, data_frequency):
+    """
+    Get a time delta based on the specified data frequency.
+
+    Parameters
+    ----------
+    periods: int
+    data_frequency: str
+
+    Returns
+    -------
+    timedelta
+
+    """
+    return timedelta(minutes=periods) \
+        if data_frequency == 'minute' else timedelta(days=periods)
+
+
+def get_periods_range(start_dt, end_dt, freq):
+    """
+    Get a date range for the specified parameters.
+
+    Parameters
+    ----------
+    start_dt: datetime
+    end_dt: datetime
+    freq: str
+
+    Returns
+    -------
+    DateTimeIndex
+
+    """
+    if freq == 'minute':
+        freq = 'T'
+
+    elif freq == 'daily':
+        freq = 'D'
+
+    return pd.date_range(start_dt, end_dt, freq=freq)
+
+
+def get_periods(start_dt, end_dt, freq):
+    """
+    The number of periods in the specified range.
+
+    Parameters
+    ----------
+    start_dt: datetime
+    end_dt: datetime
+    freq: str
+
+    Returns
+    -------
+    int
+
+    """
+    return len(get_periods_range(start_dt, end_dt, freq))
+
+
+def get_start_dt(end_dt, bar_count, data_frequency, include_first=True):
+    """
+    The start date based on specified end date and data frequency.
+
+    Parameters
+    ----------
+    end_dt: datetime
+    bar_count: int
+    data_frequency: str
+
+    Returns
+    -------
+    datetime
+
+    """
+    periods = bar_count
+    if periods > 1:
+        delta = get_delta(periods, data_frequency)
+        start_dt = end_dt - delta
+
+        if not include_first:
+            start_dt += get_delta(1, data_frequency)
+    else:
+        start_dt = end_dt
+
+    return start_dt
+
+
+def get_period_label(dt, data_frequency):
+    """
+    The period label for the specified date and frequency.
+
+    Parameters
+    ----------
+    dt: datetime
+    data_frequency: str
+
+    Returns
+    -------
+    str
+
+    """
+    if data_frequency == 'minute':
+        return '{}-{:02d}'.format(dt.year, dt.month)
+    else:
+        return '{}'.format(dt.year)
+
+
+def get_month_start_end(dt, first_day=None, last_day=None):
+    """
+    The first and last day of the month for the specified date.
+
+    Parameters
+    ----------
+    dt: datetime
+    first_day: datetime
+    last_day: datetime
+
+    Returns
+    -------
+    datetime, datetime
+
+    """
+    month_range = calendar.monthrange(dt.year, dt.month)
+
+    if first_day:
+        month_start = first_day
+    else:
+        month_start = pd.to_datetime(datetime(
+            dt.year, dt.month, 1, 0, 0, 0, 0
+        ), utc=True)
+
+    if last_day:
+        month_end = last_day
+    else:
+        month_end = pd.to_datetime(datetime(
+            dt.year, dt.month, month_range[1], 23, 59, 0, 0
+        ), utc=True)
+
+        if month_end > pd.Timestamp.utcnow():
+            month_end = pd.Timestamp.utcnow().floor('1D')
+
+    return month_start, month_end
+
+
+def get_year_start_end(dt, first_day=None, last_day=None):
+    """
+    The first and last day of the year for the specified date.
+
+    Parameters
+    ----------
+
+    dt: datetime
+    first_day: datetime
+    last_day: datetime
+
+    Returns
+    -------
+    datetime, datetime
+
+    """
+    year_start = first_day if first_day \
+        else pd.to_datetime(date(dt.year, 1, 1), utc=True)
+    year_end = last_day if last_day \
+        else pd.to_datetime(date(dt.year, 12, 31), utc=True)
+
+    if year_end > pd.Timestamp.utcnow():
+        year_end = pd.Timestamp.utcnow().floor('1D')
+
+    return year_start, year_end
+
+
+def get_df_from_arrays(arrays, periods):
+    """
+    A DataFrame from the specified OHCLV arrays.
+
+    Parameters
+    ----------
+    arrays: Object
+    periods: DateTimeIndex
+
+    Returns
+    -------
+    DataFrame
+
+    """
+    ohlcv = dict()
+    for index, field in enumerate(
+            ['open', 'high', 'low', 'close', 'volume']):
+        ohlcv[field] = arrays[index].flatten()
+
+    df = pd.DataFrame(
+        data=ohlcv,
+        index=periods
+    )
+    return df
+
+
+def range_in_bundle(asset, start_dt, end_dt, reader):
+    """
+    Evaluate whether price data of an asset is included has been ingested in
+    the exchange bundle for the given date range.
+
+    Parameters
+    ----------
+    asset: TradingPair
+    start_dt: datetime
+    end_dt: datetime
+    reader: BcolzBarMinuteReader
+
+    Returns
+    -------
+    bool
+
+    """
+    has_data = True
+    dates = [start_dt, end_dt]
+
+    while dates and has_data:
+        try:
+            dt = dates.pop(0)
+            close = reader.get_value(asset.sid, dt, 'close')
+
+            if np.isnan(close):
+                has_data = False
+
+        except Exception:
+            has_data = False
+
+    return has_data
+
+
+def get_assets(exchange, include_symbols, exclude_symbols):
+    """
+    Get assets from an exchange, including or excluding the specified
+    symbols.
+
+    Parameters
+    ----------
+    exchange: Exchange
+    include_symbols: str
+    exclude_symbols: str
+
+    Returns
+    -------
+    list[TradingPair]
+
+    """
+    if include_symbols is not None:
+        include_symbols_list = include_symbols.split(',')
+
+        return exchange.get_assets(include_symbols_list)
+
+    else:
+        all_assets = exchange.get_assets()
+
+        if exclude_symbols is not None:
+            exclude_symbols_list = exclude_symbols.split(',')
+
+            assets = []
+            for asset in all_assets:
+                if asset.symbol not in exclude_symbols_list:
+                    assets.append(asset)
+
+            return assets
+
+        else:
+            return all_assets

catalyst/exchange/ccxt/ccxt_exchange.py

@@ -0,0 +1,638 @@
+import re
+from collections import defaultdict
+
+import ccxt
+import pandas as pd
+import six
+from ccxt import ExchangeNotAvailable, InvalidOrder
+from logbook import Logger
+from six import string_types
+
+from catalyst.algorithm import MarketOrder
+from catalyst.assets._assets import TradingPair
+from catalyst.constants import LOG_LEVEL
+from catalyst.exchange.exchange import Exchange
+from catalyst.exchange.exchange_bundle import ExchangeBundle
+from catalyst.exchange.exchange_errors import InvalidHistoryFrequencyError, \
+    ExchangeSymbolsNotFound, ExchangeRequestError, InvalidOrderStyle, \
+    ExchangeNotFoundError, CreateOrderError
+from catalyst.exchange.exchange_execution import ExchangeLimitOrder
+from catalyst.exchange.exchange_utils import mixin_market_params, \
+    from_ms_timestamp, get_epoch
+from catalyst.finance.order import Order, ORDER_STATUS
+
+log = Logger('CCXT', level=LOG_LEVEL)
+
+SUPPORTED_EXCHANGES = dict(
+    binance=ccxt.binance,
+    bitfinex=ccxt.bitfinex,
+    bittrex=ccxt.bittrex,
+    poloniex=ccxt.poloniex,
+    bitmex=ccxt.bitmex,
+    gdax=ccxt.gdax,
+)
+
+
+class CCXT(Exchange):
+    def __init__(self, exchange_name, key, secret, base_currency):
+        log.debug(
+            'finding {} in CCXT exchanges:\n{}'.format(
+                exchange_name, ccxt.exchanges
+            )
+        )
+        try:
+            # Making instantiation as explicit as possible for code tracking.
+            if exchange_name in SUPPORTED_EXCHANGES:
+                exchange_attr = SUPPORTED_EXCHANGES[exchange_name]
+
+            else:
+                exchange_attr = getattr(ccxt, exchange_name)
+
+            self.api = exchange_attr({
+                'apiKey': key,
+                'secret': secret,
+            })
+
+        except Exception:
+            raise ExchangeNotFoundError(exchange_name=exchange_name)
+
+        self._symbol_maps = [None, None]
+
+        try:
+            markets_symbols = self.api.load_markets()
+            log.debug('the markets:\n{}'.format(markets_symbols))
+
+        except ExchangeNotAvailable as e:
+            raise ExchangeRequestError(error=e)
+
+        self.name = exchange_name
+
+        self.markets = self.api.fetch_markets()
+        self.load_assets()
+
+        self.base_currency = base_currency
+        self.transactions = defaultdict(list)
+
+        self.num_candles_limit = 2000
+        self.max_requests_per_minute = 60
+        self.request_cpt = dict()
+
+        self.bundle = ExchangeBundle(self.name)
+
+    def account(self):
+        return None
+
+    def time_skew(self):
+        return None
+
+    def get_market(self, symbol):
+        """
+        The CCXT market.
+
+        Parameters
+        ----------
+        symbol:
+            The CCXT symbol.
+
+        Returns
+        -------
+        dict[str, Object]
+
+        """
+        s = self.get_symbol(symbol)
+        market = next(
+            (market for market in self.markets if market['symbol'] == s),
+            None,
+        )
+        return market
+
+    def get_symbol(self, asset_or_symbol):
+        """
+        The CCXT symbol.
+
+        Parameters
+        ----------
+        asset_or_symbol
+
+        Returns
+        -------
+
+        """
+        symbol = asset_or_symbol if isinstance(
+            asset_or_symbol, string_types
+        ) else asset_or_symbol.symbol
+
+        parts = symbol.split('_')
+        return '{}/{}'.format(parts[0].upper(), parts[1].upper())
+
+    def get_catalyst_symbol(self, market_or_symbol):
+        """
+        The Catalyst symbol.
+
+        Parameters
+        ----------
+        market_or_symbol
+
+        Returns
+        -------
+
+        """
+        if isinstance(market_or_symbol, string_types):
+            parts = market_or_symbol.split('/')
+            return '{}_{}'.format(parts[0].lower(), parts[1].lower())
+
+        else:
+            return '{}_{}'.format(
+                market_or_symbol['base'].lower(),
+                market_or_symbol['quote'].lower(),
+            )
+
+    def get_timeframe(self, freq):
+        """
+        The CCXT timeframe from the Catalyst frequency.
+
+        Parameters
+        ----------
+        freq: str
+            The Catalyst frequency (Pandas convention)
+
+        Returns
+        -------
+        str
+
+        """
+        freq_match = re.match(r'([0-9].*)?(m|M|d|D|h|H|T)', freq, re.M | re.I)
+        if freq_match:
+            candle_size = int(freq_match.group(1)) \
+                if freq_match.group(1) else 1
+
+            unit = freq_match.group(2)
+
+        else:
+            raise InvalidHistoryFrequencyError(frequency=freq)
+
+        if unit.lower() == 'd':
+            timeframe = '{}d'.format(candle_size)
+
+        elif unit.lower() == 'm' or unit == 'T':
+            timeframe = '{}m'.format(candle_size)
+
+        elif unit.lower() == 'h' or unit == 'T':
+            timeframe = '{}h'.format(candle_size)
+
+        return timeframe
+
+    def get_candles(self, freq, assets, bar_count=None, start_dt=None,
+                    end_dt=None):
+        is_single = (isinstance(assets, TradingPair))
+        if is_single:
+            assets = [assets]
+
+        symbols = self.get_symbols(assets)
+        timeframe = self.get_timeframe(freq)
+
+        ms = None
+        if start_dt is not None:
+            delta = start_dt - get_epoch()
+            ms = int(delta.total_seconds()) * 1000
+
+        candles = dict()
+        for asset in assets:
+            try:
+                ohlcvs = self.api.fetch_ohlcv(
+                    symbol=symbols[0],
+                    timeframe=timeframe,
+                    since=ms,
+                    limit=bar_count,
+                    params={}
+                )
+
+                candles[asset] = []
+                for ohlcv in ohlcvs:
+                    candles[asset].append(dict(
+                        last_traded=pd.to_datetime(
+                            ohlcv[0], unit='ms', utc=True
+                        ),
+                        open=ohlcv[1],
+                        high=ohlcv[2],
+                        low=ohlcv[3],
+                        close=ohlcv[4],
+                        volume=ohlcv[5]
+                    ))
+
+            except Exception as e:
+                raise ExchangeRequestError(error=e)
+
+        if is_single:
+            return six.next(six.itervalues(candles))
+
+        else:
+            return candles
+
+    def _fetch_symbol_map(self, is_local):
+        try:
+            return self.fetch_symbol_map(is_local)
+        except ExchangeSymbolsNotFound:
+            return None
+
+    def get_asset_defs(self, market):
+        """
+        The local and Catalyst definitions of the specified market.
+
+        Parameters
+        ----------
+        market: dict[str, Object]
+            The CCXT market dicts.
+
+        Returns
+        -------
+        dict[str, Object]
+            The asset definition.
+
+        """
+        asset_defs = []
+
+        for is_local in (False, True):
+            asset_def = self.get_asset_def(market, is_local)
+            asset_defs.append((asset_def, is_local))
+
+        return asset_defs
+
+    def get_asset_def(self, market, is_local=False):
+        """
+        The asset definition (in symbols.json files) corresponding
+        to the the specified market.
+
+        Parameters
+        ----------
+        market: dict[str, Object]
+            The CCXT market dict.
+        is_local
+            Whether to search in local or Catalyst asset definitions.
+
+        Returns
+        -------
+        dict[str, Object]
+            The asset definition.
+
+        """
+        exchange_symbol = market['id']
+
+        symbol_map = self._fetch_symbol_map(is_local)
+        if symbol_map is not None:
+            assets_lower = {k.lower(): v for k, v in symbol_map.items()}
+            key = exchange_symbol.lower()
+
+            asset = assets_lower[key] if key in assets_lower else None
+            if asset is not None:
+                return asset
+
+            else:
+                return None
+
+        else:
+            return None
+
+    def create_trading_pair(self, market, asset_def=None, is_local=False):
+        """
+        Creating a TradingPair from market and asset data.
+
+        Parameters
+        ----------
+        market: dict[str, Object]
+        asset_def: dict[str, Object]
+        is_local: bool
+
+        Returns
+        -------
+
+        """
+        data_source = 'local' if is_local else 'catalyst'
+        params = dict(
+            exchange=self.name,
+            data_source=data_source,
+            exchange_symbol=market['id'],
+        )
+        mixin_market_params(self.name, params, market)
+
+        if asset_def is not None:
+            params['symbol'] = asset_def['symbol']
+
+            params['start_date'] = asset_def['start_date'] \
+                if 'start_date' in asset_def else None
+
+            params['end_date'] = asset_def['end_date'] \
+                if 'end_date' in asset_def else None
+
+            params['leverage'] = asset_def['leverage'] \
+                if 'leverage' in asset_def else 1.0
+
+            params['asset_name'] = asset_def['asset_name'] \
+                if 'asset_name' in asset_def else None
+
+            params['end_daily'] = asset_def['end_daily'] \
+                if 'end_daily' in asset_def \
+                   and asset_def['end_daily'] != 'N/A' else None
+
+            params['end_minute'] = asset_def['end_minute'] \
+                if 'end_minute' in asset_def \
+                   and asset_def['end_minute'] != 'N/A' else None
+
+        else:
+            params['symbol'] = self.get_catalyst_symbol(market)
+            # TODO: add as an optional column
+            params['leverage'] = 1.0
+
+        return TradingPair(**params)
+
+    def load_assets(self):
+        self.assets = []
+
+        for market in self.markets:
+            asset_defs = self.get_asset_defs(market)
+
+            asset = None
+            for asset_def in asset_defs:
+                if asset_def[0] is not None or not asset_defs[1]:
+                    try:
+                        asset = self.create_trading_pair(
+                            market=market,
+                            asset_def=asset_def[0],
+                            is_local=asset_def[1]
+                        )
+                        self.assets.append(asset)
+
+                    except TypeError as e:
+                        log.warn('unable to add asset: {}'.format(e))
+
+            if asset is None:
+                asset = self.create_trading_pair(market=market)
+                self.assets.append(asset)
+
+    def get_balances(self):
+        try:
+            log.debug('retrieving wallets balances')
+            balances = self.api.fetch_balance()
+
+            balances_lower = dict()
+            for key in balances:
+                balances_lower[key.lower()] = balances[key]
+
+        except Exception as e:
+            log.debug('error retrieving balances: {}', e)
+            raise ExchangeRequestError(error=e)
+
+        return balances_lower
+
+    def _create_order(self, order_status):
+        """
+        Create a Catalyst order object from a CCXT order dictionary
+
+        Parameters
+        ----------
+        order_status: dict[str, Object]
+            The order dict from the CCXT api.
+
+        Returns
+        -------
+        Order
+            The Catalyst order object
+
+        """
+        if order_status['status'] == 'canceled':
+            status = ORDER_STATUS.CANCELLED
+
+        elif order_status['status'] == 'closed' and order_status['filled'] > 0:
+            log.debug('found executed order {}'.format(order_status))
+            status = ORDER_STATUS.FILLED
+
+        elif order_status['status'] == 'open':
+            status = ORDER_STATUS.OPEN
+
+        else:
+            raise ValueError('invalid state for order')
+
+        amount = order_status['amount']
+        filled = order_status['filled']
+
+        if order_status['side'] == 'sell':
+            amount = -amount
+            filled = -filled
+
+        price = order_status['price']
+        order_type = order_status['type']
+
+        limit_price = price if order_type == 'limit' else None
+        stop_price = None  # TODO: add support
+
+        executed_price = order_status['cost'] / order_status['amount']
+        commission = order_status['fee']
+        date = from_ms_timestamp(order_status['timestamp'])
+
+        # order_id = str(order_status['info']['clientOrderId'])
+        order_id = order_status['id']
+
+        # TODO: this won't work, redo the packages with a different key.
+        symbol = order_status['info']['symbol'] \
+            if 'symbol' in order_status['info'] \
+            else order_status['info']['Exchange']
+
+        order = Order(
+            dt=date,
+            asset=self.get_asset(symbol, is_exchange_symbol=True),
+            amount=amount,
+            stop=stop_price,
+            limit=limit_price,
+            filled=filled,
+            id=order_id,
+            commission=commission
+        )
+        order.status = status
+
+        return order, executed_price
+
+    def create_order(self, asset, amount, is_buy, style):
+        symbol = self.get_symbol(asset)
+
+        if isinstance(style, ExchangeLimitOrder):
+            price = style.get_limit_price(is_buy)
+            order_type = 'limit'
+
+        elif isinstance(style, MarketOrder):
+            price = None
+            order_type = 'market'
+
+        else:
+            raise InvalidOrderStyle(
+                exchange=self.name,
+                style=style.__class__.__name__
+            )
+
+        side = 'buy' if amount > 0 else 'sell'
+
+        if hasattr(self.api, 'amount_to_lots'):
+            adj_amount = self.api.amount_to_lots(
+                symbol=symbol,
+                amount=abs(amount),
+            )
+            if adj_amount != abs(amount):
+                log.info(
+                    'adjusted order amount {} to {} based on lot size'.format(
+                        abs(amount), adj_amount,
+                    )
+                )
+        else:
+            adj_amount = abs(amount)
+
+        try:
+            result = self.api.create_order(
+                symbol=symbol,
+                type=order_type,
+                side=side,
+                amount=adj_amount,
+                price=price
+            )
+        except ExchangeNotAvailable as e:
+            log.debug('unable to create order: {}'.format(e))
+            raise ExchangeRequestError(error=e)
+
+        except InvalidOrder as e:
+            log.warn('the exchange rejected the order: {}'.format(e))
+            raise CreateOrderError(exchange=self.name, error=e)
+
+        if 'info' not in result:
+            raise ValueError('cannot use order without info attribute')
+
+        final_amount = adj_amount if side == 'buy' else -adj_amount
+        order_id = result['id']
+        order = Order(
+            dt=pd.Timestamp.utcnow(),
+            asset=asset,
+            amount=final_amount,
+            stop=style.get_stop_price(is_buy),
+            limit=style.get_limit_price(is_buy),
+            id=order_id
+        )
+        return order
+
+    def get_open_orders(self, asset):
+        try:
+            symbol = self.get_symbol(asset)
+            result = self.api.fetch_open_orders(
+                symbol=symbol,
+                since=None,
+                limit=None,
+                params=dict()
+            )
+        except Exception as e:
+            raise ExchangeRequestError(error=e)
+
+        orders = []
+        for order_status in result:
+            order, executed_price = self._create_order(order_status)
+            if asset is None or asset == order.sid:
+                orders.append(order)
+
+        return orders
+
+    def get_order(self, order_id, asset_or_symbol=None):
+        if asset_or_symbol is None:
+            log.debug(
+                'order not found in memory, the request might fail '
+                'on some exchanges.'
+            )
+        try:
+            symbol = self.get_symbol(asset_or_symbol) \
+                if asset_or_symbol is not None else None
+            order_status = self.api.fetch_order(id=order_id, symbol=symbol)
+            order, executed_price = self._create_order(order_status)
+
+        except Exception as e:
+            raise ExchangeRequestError(error=e)
+
+        return order, executed_price
+
+    def cancel_order(self, order_param, asset_or_symbol=None):
+        order_id = order_param.id \
+            if isinstance(order_param, Order) else order_param
+
+        if asset_or_symbol is None:
+            log.debug(
+                'order not found in memory, cancelling order might fail '
+                'on some exchanges.'
+            )
+        try:
+            symbol = self.get_symbol(asset_or_symbol) \
+                if asset_or_symbol is not None else None
+            self.api.cancel_order(id=order_id, symbol=symbol)
+
+        except Exception as e:
+            raise ExchangeRequestError(error=e)
+
+    def tickers(self, assets):
+        """
+        Retrieve current tick data for the given assets
+
+        Parameters
+        ----------
+        assets: list[TradingPair]
+
+        Returns
+        -------
+        list[dict[str, float]
+
+        """
+        tickers = dict()
+        for asset in assets:
+            try:
+                ccxt_symbol = self.get_symbol(asset)
+                ticker = self.api.fetch_ticker(ccxt_symbol)
+
+                ticker['last_traded'] = from_ms_timestamp(ticker['timestamp'])
+
+                if 'last_price' not in ticker:
+                    # TODO: any more exceptions?
+                    ticker['last_price'] = ticker['last']
+
+                # Using the volume represented in the base currency
+                ticker['volume'] = ticker['baseVolume'] \
+                    if 'baseVolume' in ticker else 0
+
+                tickers[asset] = ticker
+
+            except ExchangeNotAvailable as e:
+                log.warn(
+                    'unable to fetch ticker: {} {}'.format(
+                        self.name, asset.symbol
+                    )
+                )
+                raise ExchangeRequestError(error=e)
+
+        return tickers
+
+    def get_account(self):
+        return None
+
+    def get_orderbook(self, asset, order_type='all', limit=None):
+        ccxt_symbol = self.get_symbol(asset)
+
+        params = dict()
+        if limit is not None:
+            params['depth'] = limit
+
+        order_book = self.api.fetch_order_book(ccxt_symbol, params)
+
+        order_types = ['bids', 'asks'] if order_type == 'all' else [order_type]
+        result = dict(last_traded=from_ms_timestamp(order_book['timestamp']))
+        for index, order_type in enumerate(order_types):
+            if limit is not None and index > limit - 1:
+                break
+
+            result[order_type] = []
+            for entry in order_book[order_type]:
+                result[order_type].append(dict(
+                    rate=float(entry[0]),
+                    quantity=float(entry[1])
+                ))
+
+        return result

catalyst/exchange/exchange.py

@@ -0,0 +1,911 @@
+import abc
+from abc import ABCMeta, abstractmethod, abstractproperty
+from datetime import timedelta
+from time import sleep
+
+import numpy as np
+import pandas as pd
+from logbook import Logger
+
+from catalyst.constants import LOG_LEVEL
+from catalyst.data.data_portal import BASE_FIELDS
+from catalyst.exchange.bundle_utils import get_start_dt, \
+    get_delta, get_periods, get_periods_range
+from catalyst.exchange.exchange_bundle import ExchangeBundle
+from catalyst.exchange.exchange_errors import MismatchingBaseCurrencies, \
+    BaseCurrencyNotFoundError, SymbolNotFoundOnExchange, \
+    PricingDataNotLoadedError, \
+    NoDataAvailableOnExchange, NoValueForField, LastCandleTooEarlyError
+from catalyst.exchange.exchange_utils import get_exchange_symbols, \
+    get_frequency, resample_history_df
+
+log = Logger('Exchange', level=LOG_LEVEL)
+
+
+class Exchange:
+    __metaclass__ = ABCMeta
+
+    def __init__(self):
+        self.name = None
+        self.assets = []
+        self._symbol_maps = [None, None]
+        self.minute_writer = None
+        self.minute_reader = None
+        self.base_currency = None
+
+        self.num_candles_limit = None
+        self.max_requests_per_minute = None
+        self.request_cpt = None
+        self.bundle = ExchangeBundle(self.name)
+
+    @abstractproperty
+    def account(self):
+        pass
+
+    @abstractproperty
+    def time_skew(self):
+        pass
+
+    def is_open(self, dt):
+        """
+        Is the exchange open
+
+        Parameters
+        ----------
+        dt: Timestamp
+
+        Returns
+        -------
+        bool
+
+        """
+        # TODO: implement for each exchange.
+        return True
+
+    def ask_request(self):
+        """
+        Asks permission to issue a request to the exchange.
+        The primary purpose is to avoid hitting rate limits.
+
+        The application will pause if the maximum requests per minute
+        permitted by the exchange is exceeded.
+
+        Returns
+        -------
+        bool
+
+        """
+        now = pd.Timestamp.utcnow()
+        if not self.request_cpt:
+            self.request_cpt = dict()
+            self.request_cpt[now] = 0
+            return True
+
+        cpt_date = list(self.request_cpt.keys())[0]
+        cpt = self.request_cpt[cpt_date]
+
+        if now > cpt_date + timedelta(minutes=1):
+            self.request_cpt = dict()
+            self.request_cpt[now] = 0
+            return True
+
+        if cpt >= self.max_requests_per_minute:
+            delta = now - cpt_date
+
+            sleep_period = 60 - delta.total_seconds()
+            sleep(sleep_period)
+
+            now = pd.Timestamp.utcnow()
+            self.request_cpt = dict()
+            self.request_cpt[now] = 0
+            return True
+        else:
+            self.request_cpt[cpt_date] += 1
+
+    def get_symbol(self, asset):
+        """
+        The exchange specific symbol of the specified market.
+
+        Parameters
+        ----------
+        asset: TradingPair
+
+        Returns
+        -------
+        str
+
+        """
+        symbol = None
+
+        for a in self.assets:
+            if not symbol and a.symbol == asset.symbol:
+                symbol = a.symbol
+
+        if not symbol:
+            raise ValueError('Currency %s not supported by exchange %s' %
+                             (asset['symbol'], self.name.title()))
+
+        return symbol
+
+    def get_symbols(self, assets):
+        """
+        Get a list of symbols corresponding to each given asset.
+
+        Parameters
+        ----------
+        assets: list[TradingPair]
+
+        Returns
+        -------
+        list[str]
+
+        """
+        symbols = []
+        for asset in assets:
+            symbols.append(self.get_symbol(asset))
+
+        return symbols
+
+    def get_assets(self, symbols=None, data_frequency=None,
+                   is_exchange_symbol=False,
+                   is_local=None):
+        """
+        The list of markets for the specified symbols.
+
+        Parameters
+        ----------
+        symbols: list[str]
+        data_frequency: str
+        is_exchange_symbol: bool
+        is_local: bool
+
+        Returns
+        -------
+        list[TradingPair]
+            A list of asset objects.
+
+        Notes
+        -----
+        See get_asset for details of each parameter.
+
+        """
+        if symbols is None:
+            # Make a distinct list of all symbols
+            symbols = list(set([asset.symbol for asset in self.assets]))
+            is_exchange_symbol = False
+
+        assets = []
+        for symbol in symbols:
+            try:
+                asset = self.get_asset(
+                    symbol, data_frequency, is_exchange_symbol, is_local
+                )
+                assets.append(asset)
+
+            except SymbolNotFoundOnExchange:
+                log.debug(
+                    'skipping non-existent market {} {}'.format(
+                        self.name, symbol
+                    )
+                )
+        return assets
+
+    def get_asset(self, symbol, data_frequency=None, is_exchange_symbol=False,
+                  is_local=None):
+        """
+        The market for the specified symbol.
+
+        Parameters
+        ----------
+        symbol: str
+            The Catalyst or exchange symbol.
+
+        data_frequency: str
+            Check for asset corresponding to the specified data_frequency.
+            The same asset might exist in the Catalyst repository or
+            locally (following a CSV ingestion). Filtering by
+            data_frequency picks the right asset.
+
+        is_exchange_symbol: bool
+            Whether the symbol uses the Catalyst or exchange convention.
+
+        is_local: bool
+            For the local or Catalyst asset.
+
+        Returns
+        -------
+        TradingPair
+            The asset object.
+
+        """
+        asset = None
+
+        log.debug(
+            'searching assets for: {} {}'.format(
+                self.name, symbol
+            )
+        )
+        for a in self.assets:
+            if asset is not None:
+                break
+
+            if is_local is not None:
+                data_source = 'local' if is_local else 'catalyst'
+                applies = (a.data_source == data_source)
+
+            elif data_frequency is not None:
+                applies = (
+                        (
+                                data_frequency == 'minute' and a.end_minute is not None)
+                        or (
+                                data_frequency == 'daily' and a.end_daily is not None)
+                )
+
+            else:
+                applies = True
+
+            # The symbol provided may use the Catalyst or the exchange
+            # convention
+            key = a.exchange_symbol if is_exchange_symbol else a.symbol
+            if not asset and key.lower() == symbol.lower() and applies:
+                asset = a
+
+        if asset is None:
+            supported_symbols = sorted([a.symbol for a in self.assets])
+
+            raise SymbolNotFoundOnExchange(
+                symbol=symbol,
+                exchange=self.name.title(),
+                supported_symbols=supported_symbols
+            )
+
+        log.debug('found asset: {}'.format(asset))
+        return asset
+
+    def fetch_symbol_map(self, is_local=False):
+        index = 1 if is_local else 0
+        if self._symbol_maps[index] is not None:
+            return self._symbol_maps[index]
+
+        else:
+            symbol_map = get_exchange_symbols(self.name, is_local)
+            self._symbol_maps[index] = symbol_map
+            return symbol_map
+
+    @abstractmethod
+    def load_assets(self, is_local=False):
+        """
+        Populate the 'assets' attribute with a dictionary of Assets.
+        The key of the resulting dictionary is the exchange specific
+        currency pair symbol. The universal symbol is contained in the
+        'symbol' attribute of each asset.
+
+        Notes
+        -----
+        The sid of each asset is calculated based on a numeric hash of the
+        universal symbol. This simple approach avoids maintaining a mapping
+        of sids.
+
+        This method can be omerridden if an exchange offers equivalent data
+        via its api.
+
+        """
+        pass
+
+    def get_spot_value(self, assets, field, dt=None, data_frequency='minute'):
+        """
+        Public API method that returns a scalar value representing the value
+        of the desired asset's field at either the given dt.
+
+        Parameters
+        ----------
+        assets : Asset, ContinuousFuture, or iterable of same.
+            The asset or assets whose data is desired.
+        field : {'open', 'high', 'low', 'close', 'volume',
+                 'price', 'last_traded'}
+            The desired field of the asset.
+        dt : pd.Timestamp
+            The timestamp for the desired value.
+        data_frequency : str
+            The frequency of the data to query; i.e. whether the data is
+            'daily' or 'minute' bars
+
+        Returns
+        -------
+        value : float, int, or pd.Timestamp
+            The spot value of ``field`` for ``asset`` The return type is based
+            on the ``field`` requested. If the field is one of 'open', 'high',
+            'low', 'close', or 'price', the value will be a float. If the
+            ``field`` is 'volume' the value will be a int. If the ``field`` is
+            'last_traded' the value will be a Timestamp.
+
+        Bitfinex timeframes
+        -------------------
+        Available values: '1m', '5m', '15m', '30m', '1h', '3h', '6h', '12h',
+         '1D', '7D', '14D', '1M'
+        """
+        if field not in BASE_FIELDS:
+            raise KeyError('Invalid column: {}'.format(field))
+
+        tickers = self.tickers(assets)
+        if field == 'close' or field == 'price':
+            return [tickers[asset]['last'] for asset in tickers]
+
+        elif field == 'volume':
+            return [tickers[asset]['volume'] for asset in tickers]
+
+        else:
+            raise NoValueForField(field=field)
+
+    def get_single_spot_value(self, asset, field, data_frequency):
+        """
+        Similar to 'get_spot_value' but for a single asset
+
+        Notes
+        -----
+        We're writing each minute bar to disk using zipline's machinery.
+        This is especially useful when running multiple algorithms
+        concurrently. By using local data when possible, we try to reaching
+        request limits on exchanges.
+
+        Parameters
+        ----------
+        asset: TradingPair
+        field: str
+        data_frequency: str
+
+        Returns
+        -------
+        float
+            The spot value of the given asset / field
+
+        """
+        log.debug(
+            'fetching spot value {field} for symbol {symbol}'.format(
+                symbol=asset.symbol,
+                field=field
+            )
+        )
+
+        freq = '1T' if data_frequency == 'minute' else '1D'
+        ohlc = self.get_candles(freq, asset)
+        if field not in ohlc:
+            raise KeyError('Invalid column: %s' % field)
+
+        value = ohlc[field]
+        log.debug('got spot value: {}'.format(value))
+
+        return value
+
+    def get_series_from_candles(self, candles, start_dt, end_dt,
+                                data_frequency, field, previous_value=None):
+        """
+        Get a series of field data for the specified candles.
+
+        Parameters
+        ----------
+        candles: list[dict[str, float]]
+        start_dt: datetime
+        end_dt: datetime
+        data_frequency: str
+        field: str
+        previous_value: float
+
+        Returns
+        -------
+        Series
+
+        """
+        dates = [candle['last_traded'] for candle in candles]
+        values = [candle[field] for candle in candles]
+        series = pd.Series(values, index=dates)
+
+        periods = get_periods_range(
+            start_dt, end_dt, data_frequency
+        )
+        # TODO: ensure that this working as expected, if not use fillna
+        series = series.reindex(
+            periods,
+            method='ffill',
+            fill_value=previous_value,
+        )
+        series.sort_index(inplace=True)
+        return series
+
+    def get_history_window(self,
+                           assets,
+                           end_dt,
+                           bar_count,
+                           frequency,
+                           field,
+                           data_frequency=None,
+                           is_current=False):
+
+        """
+        Public API method that returns a dataframe containing the requested
+        history window.  Data is fully adjusted.
+
+        Parameters
+        ----------
+        assets : list[TradingPair]
+            The assets whose data is desired.
+
+        end_dt: datetime
+            The date of the last bar
+
+        bar_count: int
+            The number of bars desired.
+
+        frequency: string
+            "1d" or "1m"
+
+        field: string
+            The desired field of the asset.
+
+        data_frequency: string
+            The frequency of the data to query; i.e. whether the data is
+            'daily' or 'minute' bars.
+
+        is_current: bool
+            Skip date filters when current data is requested (last few bars
+            until now).
+
+        Notes
+        -----
+        Catalysts requires an end data with bar count both CCXT wants a
+        start data with bar count. Since we have to make calculations here,
+        we ensure that the last candle match the end_dt parameter.
+
+        Returns
+        -------
+        DataFrame
+            A dataframe containing the requested data.
+
+        """
+        freq, candle_size, unit, data_frequency = get_frequency(
+            frequency, data_frequency
+        )
+        adj_bar_count = candle_size * bar_count
+
+        start_dt = get_start_dt(end_dt, adj_bar_count, data_frequency)
+
+        # The get_history method supports multiple asset
+        candles = self.get_candles(
+            freq=freq,
+            assets=assets,
+            bar_count=bar_count,
+            start_dt=start_dt if not is_current else None,
+            end_dt=end_dt if not is_current else None,
+        )
+
+        series = dict()
+        for asset in candles:
+            asset_series = self.get_series_from_candles(
+                candles=candles[asset],
+                start_dt=start_dt,
+                end_dt=end_dt,
+                data_frequency=frequency,
+                field=field,
+            )
+            if end_dt is not None:
+                delta = get_delta(candle_size, data_frequency)
+                adj_end_dt = end_dt - delta
+                last_traded = asset_series.index[-1]
+
+                if last_traded < adj_end_dt:
+                    raise LastCandleTooEarlyError(
+                        last_traded=last_traded,
+                        end_dt=adj_end_dt,
+                        exchange=self.name,
+                    )
+            series[asset] = asset_series
+
+        df = pd.DataFrame(series)
+        df.dropna(inplace=True)
+
+        return df
+
+    def get_history_window_with_bundle(self,
+                                       assets,
+                                       end_dt,
+                                       bar_count,
+                                       frequency,
+                                       field,
+                                       data_frequency=None,
+                                       ffill=True,
+                                       force_auto_ingest=False):
+
+        """
+        Public API method that returns a dataframe containing the requested
+        history window.  Data is fully adjusted.
+
+        Parameters
+        ----------
+        assets : list[TradingPair]
+            The assets whose data is desired.
+
+        end_dt: datetime
+            The date of the last bar.
+
+        bar_count: int
+            The number of bars desired.
+
+        frequency: string
+            "1d" or "1m"
+
+        field: string
+            The desired field of the asset.
+
+        data_frequency: string
+            The frequency of the data to query; i.e. whether the data is
+            'daily' or 'minute' bars.
+
+        # TODO: fill how?
+        ffill: boolean
+            Forward-fill missing values. Only has effect if field
+            is 'price'.
+
+        Returns
+        -------
+        DataFrame
+            A dataframe containing the requested data.
+
+        """
+        freq, candle_size, unit, data_frequency = get_frequency(
+            frequency, data_frequency
+        )
+        adj_bar_count = candle_size * bar_count
+
+        try:
+            series = self.bundle.get_history_window_series_and_load(
+                assets=assets,
+                end_dt=end_dt,
+                bar_count=adj_bar_count,
+                field=field,
+                data_frequency=data_frequency,
+                force_auto_ingest=force_auto_ingest
+            )
+
+        except (PricingDataNotLoadedError, NoDataAvailableOnExchange):
+            series = dict()
+
+        for asset in assets:
+            if asset not in series or series[asset].index[-1] < end_dt:
+                # Adding bars too recent to be contained in the consolidated
+                # exchanges bundles. We go directly against the exchange
+                # to retrieve the candles.
+                start_dt = get_start_dt(end_dt, adj_bar_count, data_frequency)
+                trailing_dt = \
+                    series[asset].index[-1] + get_delta(1, data_frequency) \
+                        if asset in series else start_dt
+
+                # The get_history method supports multiple asset
+                # Use the original frequency to let each api optimize
+                # the size of result sets
+                trailing_bar_count = get_periods(
+                    trailing_dt, end_dt, freq
+                )
+                candles = self.get_candles(
+                    freq=freq,
+                    assets=asset,
+                    bar_count=trailing_bar_count,
+                    start_dt=start_dt,
+                    end_dt=end_dt
+                )
+
+                last_value = series[asset].iloc(0) if asset in series \
+                    else np.nan
+
+                # Create a series with the common data_frequency, ffill
+                # missing values
+                candle_series = self.get_series_from_candles(
+                    candles=candles,
+                    start_dt=trailing_dt,
+                    end_dt=end_dt,
+                    data_frequency=data_frequency,
+                    field=field,
+                    previous_value=last_value
+                )
+
+                if asset in series:
+                    series[asset].append(candle_series)
+
+                else:
+                    series[asset] = candle_series
+
+        df = resample_history_df(pd.DataFrame(series), freq, field)
+        # TODO: consider this more carefully
+        df.dropna(inplace=True)
+
+        return df
+
+    def calculate_totals(self, check_cash=False, positions=None):
+        """
+        Update the portfolio cash and position balances based on the
+        latest ticker prices.
+
+        """
+        log.debug('synchronizing portfolio with exchange {}'.format(self.name))
+
+        cash = None
+        if check_cash:
+            balances = self.get_balances()
+
+            cash = balances[self.base_currency]['free'] \
+                if self.base_currency in balances else None
+
+            if cash is None:
+                raise BaseCurrencyNotFoundError(
+                    base_currency=self.base_currency,
+                    exchange=self.name
+                )
+            log.debug('found base currency balance: {}'.format(cash))
+
+        positions_value = 0.0
+        if positions:
+            assets = set([position.asset for position in positions])
+            tickers = self.tickers(assets)
+            log.debug('got tickers for positions: {}'.format(tickers))
+
+            for asset in tickers:
+                ticker = tickers[asset]
+                positions = [p for p in positions if p.asset == asset]
+
+                for position in positions:
+                    position.last_sale_price = ticker['last_price']
+                    position.last_sale_date = ticker['last_traded']
+
+                    positions_value += \
+                        position.amount * position.last_sale_price
+
+        return cash, positions_value
+
+    def order(self, asset, amount, style):
+        """Place an order.
+
+        Parameters
+        ----------
+        asset : TradingPair
+            The asset that this order is for.
+
+        amount : int
+            The amount of shares to order. If ``amount`` is positive, this is
+            the number of shares to buy or cover. If ``amount`` is negative,
+            this is the number of shares to sell or short.
+
+        limit_price : float, optional
+            The limit price for the order.
+
+        stop_price : float, optional
+            The stop price for the order.
+
+        style : ExecutionStyle, optional
+            The execution style for the order.
+
+        Returns
+        -------
+        order_id : str or None
+            The unique identifier for this order, or None if no order was
+            placed.
+
+        Notes
+        -----
+        The ``limit_price`` and ``stop_price`` arguments provide shorthands for
+        passing common execution styles. Passing ``limit_price=N`` is
+        equivalent to ``style=LimitOrder(N)``. Similarly, passing
+        ``stop_price=M`` is equivalent to ``style=StopOrder(M)``, and passing
+        ``limit_price=N`` and ``stop_price=M`` is equivalent to
+        ``style=StopLimitOrder(N, M)``. It is an error to pass both a ``style``
+        and ``limit_price`` or ``stop_price``.
+
+        See Also
+        --------
+        :class:`catalyst.finance.execution.ExecutionStyle`
+        :func:`catalyst.api.order_value`
+        :func:`catalyst.api.order_percent`
+
+        """
+        if amount == 0:
+            log.warn('skipping order amount of 0')
+            return None
+
+        if self.base_currency is None:
+            raise ValueError('no base_currency defined for this exchange')
+
+        if asset.quote_currency != self.base_currency.lower():
+            raise MismatchingBaseCurrencies(
+                base_currency=asset.quote_currency,
+                algo_currency=self.base_currency
+            )
+
+        is_buy = (amount > 0)
+        display_price = style.get_limit_price(is_buy)
+
+        log.debug(
+            'issuing {side} order of {amount} {symbol} for {type}:'
+            ' {price}'.format(
+                side='buy' if is_buy else 'sell',
+                amount=amount,
+                symbol=asset.symbol,
+                type=style.__class__.__name__,
+                price='{}{}'.format(display_price, asset.quote_currency)
+            )
+        )
+
+        return self.create_order(asset, amount, is_buy, style)
+
+    # The methods below must be implemented for each exchange.
+    @abstractmethod
+    def get_balances(self):
+        """
+        Retrieve wallet balances for the exchange.
+
+        Returns
+        -------
+        dict[TradingPair, float]
+
+        """
+        pass
+
+    @abstractmethod
+    def create_order(self, asset, amount, is_buy, style):
+        """
+        Place an order on the exchange.
+
+        Parameters
+        ----------
+        asset: TradingPair
+            The target market.
+
+        amount: float
+            The amount of shares to order. If ``amount`` is positive, this is
+            the number of shares to buy or cover. If ``amount`` is negative,
+            this is the number of shares to sell or short.
+
+        is_buy: bool
+            Is it a buy order?
+
+        style: ExecutionStyle
+
+        Returns
+        -------
+        Order
+
+        """
+        pass
+
+    @abstractmethod
+    def get_open_orders(self, asset):
+        """Retrieve all of the current open orders.
+
+        Parameters
+        ----------
+        asset : Asset
+            If passed and not None, return only the open orders for the given
+            asset instead of all open orders.
+
+        Returns
+        -------
+        open_orders : dict[list[Order]] or list[Order]
+            If no asset is passed this will return a dict mapping Assets
+            to a list containing all the open orders for the asset.
+            If an asset is passed then this will return a list of the open
+            orders for this asset.
+        """
+        pass
+
+    @abstractmethod
+    def get_order(self, order_id, symbol_or_asset=None):
+        """Lookup an order based on the order id returned from one of the
+        order functions.
+
+        Parameters
+        ----------
+        order_id : str
+            The unique identifier for the order.
+        symbol_or_asset: str|TradingPair
+            The catalyst symbol, some exchanges need this
+
+        Returns
+        -------
+        order : Order
+            The order object.
+        execution_price: float
+            The execution price per share of the order
+        """
+        pass
+
+    @abstractmethod
+    def cancel_order(self, order_param, symbol_or_asset=None):
+        """Cancel an open order.
+
+        Parameters
+        ----------
+        order_param : str or Order
+            The order_id or order object to cancel.
+        symbol_or_asset: str|TradingPair
+            The catalyst symbol, some exchanges need this
+        """
+        pass
+
+    @abstractmethod
+    def get_candles(self, freq, assets, bar_count=None,
+                    start_dt=None, end_dt=None):
+        """
+        Retrieve OHLCV candles for the given assets
+
+        Parameters
+        ----------
+        freq: str
+            The frequency alias per convention:
+            http://pandas.pydata.org/pandas-docs/stable/timeseries.html#offset-aliases
+
+        assets: list[TradingPair]
+            The targeted assets.
+
+        bar_count: int
+            The number of bar desired. (default 1)
+
+        end_dt: datetime, optional
+            The last bar date.
+
+        start_dt: datetime, optional
+            The first bar date.
+
+        Returns
+        -------
+        dict[TradingPair, dict[str, Object]]
+            A dictionary of OHLCV candles. Each TradingPair instance is
+            mapped to a list of dictionaries with this structure:
+                open: float
+                high: float
+                low: float
+                close: float
+                volume: float
+                last_traded: datetime
+
+            See definition here:
+                http://www.investopedia.com/terms/o/ohlcchart.asp
+        """
+        pass
+
+    @abc.abstractmethod
+    def tickers(self, assets):
+        """
+        Retrieve current tick data for the given assets
+
+        Parameters
+        ----------
+        assets: list[TradingPair]
+
+        Returns
+        -------
+        list[dict[str, float]
+
+        """
+        pass
+
+    @abc.abstractmethod
+    def get_account(self):
+        """
+        Retrieve the account parameters.
+        """
+        pass
+
+    @abc.abstractmethod
+    def get_orderbook(self, asset, order_type, limit):
+        """
+        Retrieve the the orderbook for the given trading pair.
+
+        Parameters
+        ----------
+        asset: TradingPair
+        order_type: str
+            The type of orders: bid, ask or all
+        limit: int
+
+        Returns
+        -------
+        list[dict[str, float]
+        """
+        pass

catalyst/exchange/exchange_algorithm.py

@@ -0,0 +1,839 @@
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import pickle
+import signal
+import sys
+from datetime import timedelta
+from os import listdir
+from os.path import isfile, join
+from time import sleep
+
+import logbook
+import pandas as pd
+
+import catalyst.protocol as zp
+from catalyst.algorithm import TradingAlgorithm
+from catalyst.constants import LOG_LEVEL
+from catalyst.exchange.exchange_blotter import ExchangeBlotter
+from catalyst.exchange.exchange_errors import (
+    ExchangeRequestError,
+    ExchangePortfolioDataError,
+    OrderTypeNotSupported, )
+from catalyst.exchange.exchange_execution import ExchangeLimitOrder
+from catalyst.exchange.exchange_utils import (
+    save_algo_object,
+    get_algo_object,
+    get_algo_folder,
+    get_algo_df,
+    save_algo_df,
+    group_assets_by_exchange, )
+from catalyst.exchange.live_graph_clock import LiveGraphClock
+from catalyst.exchange.simple_clock import SimpleClock
+from catalyst.exchange.stats_utils import get_pretty_stats, stats_to_s3, \
+    stats_to_algo_folder
+from catalyst.finance.execution import MarketOrder
+from catalyst.finance.performance.period import calc_period_stats
+from catalyst.gens.tradesimulation import AlgorithmSimulator
+from catalyst.utils.api_support import api_method
+from catalyst.utils.input_validation import error_keywords, ensure_upper_case
+from catalyst.utils.math_utils import round_nearest
+from catalyst.utils.preprocess import preprocess
+
+log = logbook.Logger('exchange_algorithm', level=LOG_LEVEL)
+
+
+class ExchangeAlgorithmExecutor(AlgorithmSimulator):
+    def __init__(self, *args, **kwargs):
+        super(self.__class__, self).__init__(*args, **kwargs)
+
+
+class ExchangeTradingAlgorithmBase(TradingAlgorithm):
+    def __init__(self, *args, **kwargs):
+        self.exchanges = kwargs.pop('exchanges', None)
+        self.simulate_orders = kwargs.pop('simulate_orders', None)
+
+        super(ExchangeTradingAlgorithmBase, self).__init__(*args, **kwargs)
+
+        self.current_day = None
+
+        if self.simulate_orders is None \
+                and self.sim_params.arena == 'backtest':
+            self.simulate_orders = True
+
+        self.blotter = ExchangeBlotter(
+            data_frequency=self.data_frequency,
+            # Default to NeverCancel in catalyst
+            cancel_policy=self.cancel_policy,
+            simulate_orders=self.simulate_orders,
+            exchanges=self.exchanges
+        )
+
+    @staticmethod
+    def __convert_order_params_for_blotter(limit_price, stop_price, style):
+        """
+        Helper method for converting deprecated limit_price and stop_price
+        arguments into ExecutionStyle instances.
+
+        This function assumes that either style == None or (limit_price,
+        stop_price) == (None, None).
+        """
+        if stop_price:
+            raise OrderTypeNotSupported(order_type='stop')
+
+        if style:
+            if limit_price is not None:
+                raise ValueError(
+                    'An order style and a limit price was included in the '
+                    'order. Please pick one to avoid any possible conflict.'
+                )
+
+            # Currently limiting order types or limit and market to
+            # be in-line with CXXT and many exchanges. We'll consider
+            # adding more order types in the future.
+            if not isinstance(style, ExchangeLimitOrder) or \
+                    not isinstance(style, MarketOrder):
+                raise OrderTypeNotSupported(
+                    order_type=style.__class__.__name__
+                )
+
+            return style
+
+        if limit_price:
+            return ExchangeLimitOrder(limit_price)
+        else:
+            return MarketOrder()
+
+    @api_method
+    def set_commission(self, maker=None, taker=None):
+        key = self.blotter.commission_models.keys()[0]
+        if maker is not None:
+            self.blotter.commission_models[key].maker = maker
+
+        if taker is not None:
+            self.blotter.commission_models[key].taker = taker
+
+    @api_method
+    def set_slippage(self, spread=None):
+        key = self.blotter.slippage_models.keys()[0]
+        if spread is not None:
+            self.blotter.slippage_models[key].spread = spread
+
+    def _calculate_order(self, asset, amount,
+                         limit_price=None, stop_price=None, style=None):
+        # Raises a ZiplineError if invalid parameters are detected.
+        self.validate_order_params(asset,
+                                   amount,
+                                   limit_price,
+                                   stop_price,
+                                   style)
+
+        # Convert deprecated limit_price and stop_price parameters to use
+        # ExecutionStyle objects.
+        style = self.__convert_order_params_for_blotter(limit_price,
+                                                        stop_price,
+                                                        style)
+        return amount, style
+
+    def round_order(self, amount, asset):
+        """
+        We need fractions with cryptocurrencies
+
+        :param amount:
+        :return:
+        """
+        return round_nearest(amount, asset.min_trade_size)
+
+    @api_method
+    @preprocess(symbol_str=ensure_upper_case)
+    def symbol(self, symbol_str, exchange_name=None):
+        """Lookup an Equity by its ticker symbol.
+
+        Parameters
+        ----------
+        symbol_str : str
+            The ticker symbol for the equity to lookup.
+        exchange_name: str
+            The name of the exchange containing the symbol
+
+        Returns
+        -------
+        equity : Equity
+            The equity that held the ticker symbol on the current
+            symbol lookup date.
+
+        Raises
+        ------
+        SymbolNotFound
+            Raised when the symbols was not held on the current lookup date.
+
+        See Also
+        --------
+        :func:`catalyst.api.set_symbol_lookup_date`
+        """
+        # If the user has not set the symbol lookup date,
+        # use the end_session as the date for sybmol->sid resolution.
+
+        _lookup_date = self._symbol_lookup_date \
+            if self._symbol_lookup_date is not None \
+            else self.sim_params.end_session
+
+        if exchange_name is None:
+            exchange = list(self.exchanges.values())[0]
+        else:
+            exchange = self.exchanges[exchange_name]
+
+        data_frequency = self.data_frequency \
+            if self.sim_params.arena == 'backtest' else None
+        return self.asset_finder.lookup_symbol(
+            symbol=symbol_str,
+            exchange=exchange,
+            data_frequency=data_frequency,
+            as_of_date=_lookup_date
+        )
+
+    def prepare_period_stats(self, start_dt, end_dt):
+        """
+        Creates a dictionary representing the state of the tracker.
+
+        Parameters
+        ----------
+        start_dt: datetime
+        end_dt: datetime
+
+        Notes
+        -----
+        I rewrote this in an attempt to better control the stats.
+        I don't want things to happen magically through complex logic
+        pertaining to backtesting.
+
+        """
+        tracker = self.perf_tracker
+        period = tracker.todays_performance
+
+        pos_stats = period.position_tracker.stats()
+        period_stats = calc_period_stats(pos_stats, period.ending_cash)
+
+        stats = dict(
+            period_start=tracker.period_start,
+            period_end=tracker.period_end,
+            capital_base=tracker.capital_base,
+            progress=tracker.progress,
+            ending_value=period.ending_value,
+            ending_exposure=period.ending_exposure,
+            capital_used=period.cash_flow,
+            starting_value=period.starting_value,
+            starting_exposure=period.starting_exposure,
+            starting_cash=period.starting_cash,
+            ending_cash=period.ending_cash,
+            portfolio_value=period.ending_cash + period.ending_value,
+            pnl=period.pnl,
+            returns=period.returns,
+            period_open=period.period_open,
+            period_close=period.period_close,
+            gross_leverage=period_stats.gross_leverage,
+            net_leverage=period_stats.net_leverage,
+            short_exposure=pos_stats.short_exposure,
+            long_exposure=pos_stats.long_exposure,
+            short_value=pos_stats.short_value,
+            long_value=pos_stats.long_value,
+            longs_count=pos_stats.longs_count,
+            shorts_count=pos_stats.shorts_count,
+        )
+
+        # Merging cumulative risk
+        stats.update(tracker.cumulative_risk_metrics.to_dict())
+
+        # Merging latest recorded variables
+        stats.update(self.recorded_vars)
+
+        stats['positions'] = period.position_tracker.get_positions_list()
+
+        # we want the key to be absent, not just empty
+        # Only include transactions for given dt
+        stats['transactions'] = []
+        for date in period.processed_transactions:
+            if start_dt <= date < end_dt:
+                transactions = period.processed_transactions[date]
+                for t in transactions:
+                    stats['transactions'].append(t.to_dict())
+
+        stats['orders'] = []
+        for date in period.orders_by_modified:
+            if start_dt <= date < end_dt:
+                orders = period.orders_by_modified[date]
+                for order in orders:
+                    stats['orders'].append(orders[order].to_dict())
+
+        return stats
+
+
+class ExchangeTradingAlgorithmBacktest(ExchangeTradingAlgorithmBase):
+    def __init__(self, *args, **kwargs):
+        super(ExchangeTradingAlgorithmBacktest, self).__init__(*args, **kwargs)
+
+        self.frame_stats = list()
+        log.info('initialized trading algorithm in backtest mode')
+
+    def is_last_frame_of_day(self, data):
+        # TODO: adjust here to support more intervals
+        next_frame_dt = data.current_dt + timedelta(minutes=1)
+        if next_frame_dt.date() > data.current_dt.date():
+            return True
+        else:
+            return False
+
+    def handle_data(self, data):
+        super(ExchangeTradingAlgorithmBacktest, self).handle_data(data)
+
+        if self.data_frequency == 'minute':
+            frame_stats = self.prepare_period_stats(
+                data.current_dt, data.current_dt + timedelta(minutes=1)
+            )
+            self.frame_stats.append(frame_stats)
+
+        self.current_day = data.current_dt.floor('1D')
+
+    def _create_stats_df(self):
+        stats = pd.DataFrame(self.frame_stats)
+        stats.set_index('period_close', inplace=True, drop=False)
+        return stats
+
+    def analyze(self, perf):
+        stats = self._create_stats_df() if self.data_frequency == 'minute' \
+            else perf
+        super(ExchangeTradingAlgorithmBacktest, self).analyze(stats)
+
+    def run(self, data=None, overwrite_sim_params=True):
+        perf = super(ExchangeTradingAlgorithmBacktest, self).run(
+            data, overwrite_sim_params
+        )
+        # Rebuilding the stats to support minute data
+        stats = self._create_stats_df() if self.data_frequency == 'minute' \
+            else perf
+        return stats
+
+
+class ExchangeTradingAlgorithmLive(ExchangeTradingAlgorithmBase):
+    def __init__(self, *args, **kwargs):
+        self.algo_namespace = kwargs.pop('algo_namespace', None)
+        self.live_graph = kwargs.pop('live_graph', None)
+        self.stats_output = kwargs.pop('stats_output', None)
+
+        self._clock = None
+        self.frame_stats = list()
+
+        self.pnl_stats = get_algo_df(self.algo_namespace, 'pnl_stats')
+
+        self.custom_signals_stats = \
+            get_algo_df(self.algo_namespace, 'custom_signals_stats')
+
+        self.exposure_stats = \
+            get_algo_df(self.algo_namespace, 'exposure_stats')
+
+        self.is_running = True
+
+        self.retry_check_open_orders = 5
+        self.retry_synchronize_portfolio = 5
+        self.retry_get_open_orders = 5
+        self.retry_order = 2
+        self.retry_delay = 5
+
+        self.stats_minutes = 10
+
+        super(ExchangeTradingAlgorithmLive, self).__init__(*args, **kwargs)
+
+        signal.signal(signal.SIGINT, self.signal_handler)
+
+        log.info('initialized trading algorithm in live mode')
+
+    def signal_handler(self, signal, frame):
+        """
+        Handles the keyboard interruption signal.
+
+        Parameters
+        ----------
+        signal
+        frame
+
+        Returns
+        -------
+
+        """
+        self.is_running = False
+
+        if self._analyze is None:
+            log.info('Interruption signal detected {}, exiting the '
+                     'algorithm'.format(signal))
+
+        else:
+            log.info('Interruption signal detected {}, calling `analyze()` '
+                     'before exiting the algorithm'.format(signal))
+
+            algo_folder = get_algo_folder(self.algo_namespace)
+            folder = join(algo_folder, 'daily_perf')
+            files = [f for f in listdir(folder) if isfile(join(folder, f))]
+
+            daily_perf_list = []
+            for item in files:
+                filename = join(folder, item)
+                with open(filename, 'rb') as handle:
+                    daily_perf_list.append(pickle.load(handle))
+
+            stats = pd.DataFrame(daily_perf_list)
+
+            self.analyze(stats)
+
+        sys.exit(0)
+
+    @property
+    def clock(self):
+        if self._clock is None:
+            return self._create_clock()
+        else:
+            return self._clock
+
+    def _create_clock(self):
+
+        # The calendar's execution times are the minutes over which we actually
+        # want to run the clock. Typically the execution times simply adhere to
+        # the market open and close times. In the case of the futures calendar,
+        # for example, we only want to simulate over a subset of the full 24
+        # hour calendar, so the execution times dictate a market open time of
+        # 6:31am US/Eastern and a close of 5:00pm US/Eastern.
+
+        # In our case, we are trading around the clock, so the market close
+        # corresponds to the last minute of the day.
+
+        # This method is taken from TradingAlgorithm.
+        # The clock has been replaced to use RealtimeClock
+        # TODO: should we apply time skew? not sure to understand the utility.
+
+        log.debug('creating clock')
+        if self.live_graph:
+            self._clock = LiveGraphClock(
+                self.sim_params.sessions,
+                context=self
+            )
+        else:
+            self._clock = SimpleClock(
+                self.sim_params.sessions,
+            )
+
+        return self._clock
+
+    def _create_generator(self, sim_params):
+        if self.perf_tracker is None:
+            self.perf_tracker = get_algo_object(
+                algo_name=self.algo_namespace,
+                key='perf_tracker'
+            )
+
+        # Call the simulation trading algorithm for side-effects:
+        # it creates the perf tracker
+        TradingAlgorithm._create_generator(self, sim_params)
+        self.trading_client = ExchangeAlgorithmExecutor(
+            self,
+            sim_params,
+            self.data_portal,
+            self.clock,
+            self._create_benchmark_source(),
+            self.restrictions,
+            universe_func=self._calculate_universe
+        )
+
+        return self.trading_client.transform()
+
+    def updated_portfolio(self):
+        return self.perf_tracker.get_portfolio(False)
+
+    def updated_account(self):
+        return self.perf_tracker.get_account(False)
+
+    def synchronize_portfolio(self, attempt_index=0):
+        """
+        Synchronizes the portfolio tracked by the algorithm to refresh
+        its current value.
+
+        This includes updating the last_sale_price of all tracked
+        positions, returning the available cash, and raising error
+        if the data goes out of sync.
+
+        Parameters
+        ----------
+        attempt_index: int
+
+        Returns
+        -------
+        float
+            The amount of base currency available for trading.
+
+        float
+            The total value of all tracked positions.
+
+        """
+        tracker = self.perf_tracker.position_tracker
+        total_cash = 0.0
+        total_positions_value = 0.0
+
+        try:
+            # Position keys correspond to assets
+            positions = self.portfolio.positions
+            assets = list(positions)
+            exchange_assets = group_assets_by_exchange(assets)
+            for exchange_name in self.exchanges:
+                assets = exchange_assets[exchange_name] \
+                    if exchange_name in exchange_assets else []
+
+                exchange_positions = \
+                    [positions[asset] for asset in assets]
+
+                check_cash = (not self.simulate_orders)
+
+                exchange = self.exchanges[exchange_name]  # Type: Exchange
+                cash, positions_value = exchange.calculate_totals(
+                    positions=exchange_positions,
+                    check_cash=check_cash,
+                )
+                total_positions_value += positions_value
+
+                if cash is not None:
+                    total_cash += cash
+
+                for position in exchange_positions:
+                    tracker.update_position(
+                        asset=position.asset,
+                        last_sale_date=position.last_sale_date,
+                        last_sale_price=position.last_sale_price
+                    )
+
+            if cash is None:
+                total_cash = self.portfolio.cash
+
+            elif total_cash < self.portfolio.cash:
+                raise ValueError('Cash on exchanges is lower than the algo.')
+
+            return total_cash, total_positions_value
+
+        except ExchangeRequestError as e:
+            log.warn(
+                'update portfolio attempt {}: {}'.format(attempt_index, e)
+            )
+            if attempt_index < self.retry_synchronize_portfolio:
+                sleep(self.retry_delay)
+                return self.synchronize_portfolio(attempt_index + 1)
+            else:
+                raise ExchangePortfolioDataError(
+                    data_type='update-portfolio',
+                    attempts=attempt_index,
+                    error=e
+                )
+
+    def add_pnl_stats(self, period_stats):
+        """
+        Save p&l stats.
+
+        Parameters
+        ----------
+        period_stats
+
+        Returns
+        -------
+
+        """
+        starting = period_stats['starting_cash']
+        current = period_stats['portfolio_value']
+        appreciation = (current / starting) - 1
+        perc = (appreciation * 100) if current != 0 else 0
+
+        log.debug('adding pnl stats: {:6f}%'.format(perc))
+
+        df = pd.DataFrame(
+            data=[dict(performance=perc)],
+            index=[period_stats['period_close']]
+        )
+        self.pnl_stats = pd.concat([self.pnl_stats, df])
+
+        save_algo_df(self.algo_namespace, 'pnl_stats', self.pnl_stats)
+
+    def add_custom_signals_stats(self, period_stats):
+        """
+        Save custom signals stats.
+
+        Parameters
+        ----------
+        period_stats
+
+        Returns
+        -------
+
+        """
+        log.debug('adding custom signals stats: {}'.format(self.recorded_vars))
+        df = pd.DataFrame(
+            data=[self.recorded_vars],
+            index=[period_stats['period_close']],
+        )
+        self.custom_signals_stats = pd.concat([self.custom_signals_stats, df])
+
+        save_algo_df(self.algo_namespace, 'custom_signals_stats',
+                     self.custom_signals_stats)
+
+    def add_exposure_stats(self, period_stats):
+        """
+        Save exposure stats.
+
+        Parameters
+        ----------
+        period_stats
+
+        Returns
+        -------
+
+        """
+        data = dict(
+            long_exposure=period_stats['long_exposure'],
+            base_currency=period_stats['ending_cash']
+        )
+        log.debug('adding exposure stats: {}'.format(data))
+
+        df = pd.DataFrame(
+            data=[data],
+            index=[period_stats['period_close']],
+        )
+        self.exposure_stats = pd.concat([self.exposure_stats, df])
+
+        save_algo_df(
+            self.algo_namespace, 'exposure_stats', self.exposure_stats
+        )
+
+    def handle_data(self, data):
+        """
+        Wrapper around the handle_data method of each algo.
+
+        Parameters
+        ----------
+        data
+
+        """
+        if not self.is_running:
+            return
+
+        # Resetting the frame stats every day to minimize memory footprint
+        today = data.current_dt.floor('1D')
+        if self.current_day is not None and today > self.current_day:
+            self.frame_stats = list()
+
+        new_transactions, new_commissions, closed_orders = \
+            self.blotter.get_transactions(data)
+
+        if len(new_transactions) > 0:
+            self.perf_tracker.update_performance()
+
+        cash, positions_value = self.synchronize_portfolio()
+        log.info(
+            'got totals from exchanges, cash: {} positions: {}'.format(
+                cash, positions_value
+            )
+        )
+        if self._handle_data:
+            self._handle_data(self, data)
+
+        # Unlike trading controls which remain constant unless placing an
+        # order, account controls can change each bar. Thus, must check
+        # every bar no matter if the algorithm places an order or not.
+        self.validate_account_controls()
+
+        try:
+            self._save_stats_csv(self._process_stats(data))
+        except Exception as e:
+            log.warn('unable to calculate performance: {}'.format(e))
+
+        # TODO: pickle does not seem to work in python 3
+        try:
+            save_algo_object(
+                algo_name=self.algo_namespace,
+                key='perf_tracker',
+                obj=self.perf_tracker
+            )
+        except Exception as e:
+            log.warn('unable to save minute perfs to disk: {}'.format(e))
+
+        self.current_day = data.current_dt.floor('1D')
+
+    def _process_stats(self, data):
+        today = data.current_dt.floor('1D')
+
+        # Since the clock runs 24/7, I trying to disable the daily
+        # Performance tracker and keep only minute and cumulative
+        self.perf_tracker.update_performance()
+
+        frame_stats = self.prepare_period_stats(
+            data.current_dt, data.current_dt + timedelta(minutes=1))
+
+        # Saving the last hour in memory
+        self.frame_stats.append(frame_stats)
+
+        self.add_pnl_stats(frame_stats)
+        if self.recorded_vars:
+            self.add_custom_signals_stats(frame_stats)
+            recorded_cols = list(self.recorded_vars.keys())
+
+        else:
+            recorded_cols = None
+
+        self.add_exposure_stats(frame_stats)
+
+        log.info(
+            'statistics for the last {stats_minutes} minutes:\n'
+            '{stats}'.format(
+                stats_minutes=self.stats_minutes,
+                stats=get_pretty_stats(
+                    stats=self.frame_stats,
+                    recorded_cols=recorded_cols,
+                    num_rows=self.stats_minutes
+                )
+            ))
+
+        # Saving the daily stats in a format usable for performance
+        # analysis.
+        daily_stats = self.prepare_period_stats(
+            start_dt=today,
+            end_dt=data.current_dt
+        )
+        save_algo_object(
+            algo_name=self.algo_namespace,
+            key=today.strftime('%Y-%m-%d'),
+            obj=daily_stats,
+            rel_path='daily_perf'
+        )
+
+        return recorded_cols
+
+    def _save_stats_csv(self, recorded_cols):
+        # Writing the stats output
+        csv_bytes = None
+        try:
+            csv_bytes = stats_to_algo_folder(
+                stats=self.frame_stats,
+                algo_namespace=self.algo_namespace,
+                recorded_cols=recorded_cols,
+            )
+        except Exception as e:
+            log.warn('unable save stats locally: {}'.format(e))
+
+        try:
+            if self.stats_output is not None:
+                if 's3://' in self.stats_output:
+                    stats_to_s3(
+                        uri=self.stats_output,
+                        stats=self.frame_stats,
+                        algo_namespace=self.algo_namespace,
+                        recorded_cols=recorded_cols,
+                        bytes_to_write=csv_bytes
+                    )
+                else:
+                    raise ValueError(
+                        'Only S3 stats output is supported for now.'
+                    )
+        except Exception as e:
+            log.warn('unable save stats externally: {}'.format(e))
+
+    @api_method
+    def batch_market_order(self, share_counts):
+        raise NotImplementedError()
+
+    def _get_open_orders(self, asset=None, attempt_index=0):
+        try:
+            if asset:
+                exchange = self.exchanges[asset.exchange]
+                return exchange.get_open_orders(asset)
+
+            else:
+                open_orders = []
+                for exchange_name in self.exchanges:
+                    exchange = self.exchanges[exchange_name]
+                    exchange_orders = exchange.get_open_orders()
+                    open_orders.append(exchange_orders)
+
+                return open_orders
+        except ExchangeRequestError as e:
+            log.warn(
+                'open orders attempt {}: {}'.format(attempt_index, e)
+            )
+            if attempt_index < self.retry_get_open_orders:
+                sleep(self.retry_delay)
+                return self._get_open_orders(asset, attempt_index + 1)
+            else:
+                raise ExchangePortfolioDataError(
+                    data_type='open-orders',
+                    attempts=attempt_index,
+                    error=e
+                )
+
+    @error_keywords(sid='Keyword argument `sid` is no longer supported for '
+                        'get_open_orders. Use `asset` instead.')
+    @api_method
+    def get_open_orders(self, asset=None):
+        """Retrieve all of the current open orders.
+
+        Parameters
+        ----------
+        asset : Asset
+            If passed and not None, return only the open orders for the given
+            asset instead of all open orders.
+
+        Returns
+        -------
+        open_orders : dict[list[Order]] or list[Order]
+            If no asset is passed this will return a dict mapping Assets
+            to a list containing all the open orders for the asset.
+            If an asset is passed then this will return a list of the open
+            orders for this asset.
+        """
+        return self._get_open_orders(asset)
+
+    @api_method
+    def get_order(self, order_id, exchange_name):
+        """Lookup an order based on the order id returned from one of the
+        order functions.
+
+        Parameters
+        ----------
+        order_id : str
+            The unique identifier for the order.
+
+        Returns
+        -------
+        order : Order
+            The order object.
+        execution_price: float
+            The execution price per share of the order
+        """
+        exchange = self.exchanges[exchange_name]
+        return exchange.get_order(order_id)
+
+    @api_method
+    def cancel_order(self, order_param, exchange_name):
+        """Cancel an open order.
+
+        Parameters
+        ----------
+        order_param : str or Order
+            The order_id or order object to cancel.
+        """
+        exchange = self.exchanges[exchange_name]
+
+        order_id = order_param
+        if isinstance(order_param, zp.Order):
+            order_id = order_param.id
+
+        exchange.cancel_order(order_id)

catalyst/exchange/exchange_bcolz.py

@@ -0,0 +1,98 @@
+import numpy as np
+
+from catalyst import get_calendar
+from catalyst.data.minute_bars import BcolzMinuteBarReader, \
+    BcolzMinuteBarWriter
+
+
+class BcolzExchangeBarWriter(BcolzMinuteBarWriter):
+    def __init__(self, *args, **kwargs):
+        self._data_frequency = kwargs.pop('data_frequency', None)
+        kwargs.pop('minutes_per_day', None)
+        kwargs.pop('calendar', None)
+
+        end_session = kwargs.pop('end_session', None)
+        if end_session is not None:
+            end_session = end_session.floor('1d')
+
+        minutes_per_day = 1440 if self._data_frequency == 'minute' else 1
+        default_ohlc_ratio = kwargs.pop('default_ohlc_ratio', 100000000)
+        calendar = get_calendar('OPEN')
+
+        super(BcolzExchangeBarWriter, self) \
+            .__init__(*args, **dict(kwargs,
+                                    minutes_per_day=minutes_per_day,
+                                    default_ohlc_ratio=default_ohlc_ratio,
+                                    calendar=calendar,
+                                    end_session=end_session
+                                    ))
+
+
+class BcolzExchangeBarReader(BcolzMinuteBarReader):
+    def __init__(self, *args, **kwargs):
+        self._data_frequency = kwargs.pop('data_frequency', None)
+
+        super(BcolzExchangeBarReader, self).__init__(*args, **kwargs)
+
+    @property
+    def data_frequency(self):
+        return self._data_frequency
+
+    def load_raw_arrays(self, fields, start_dt, end_dt, sids):
+        """
+        Parameters
+        ----------
+        fields : list of str
+           'open', 'high', 'low', 'close', or 'volume'
+        start_dt: Timestamp
+           Beginning of the window range.
+        end_dt: Timestamp
+           End of the window range.
+        sids : list of int
+           The asset identifiers in the window.
+
+        Returns
+        -------
+        list of np.ndarray
+            A list with an entry per field of ndarrays with shape
+            (minutes in range, sids) with a dtype of float64, containing the
+            values for the respective field over start and end dt range.
+        """
+        start_idx = self._find_position_of_minute(start_dt)
+        end_idx = self._find_position_of_minute(end_dt)
+
+        periods = self.calendar.minutes_in_range(start_dt, end_dt) \
+            if self.data_frequency == 'minute' \
+            else self.calendar.sessions_in_range(start_dt, end_dt)
+
+        num_days = len(periods)
+        shape = num_days, len(sids)
+
+        all_fields = fields[:]
+        if len(all_fields) == 1 and all_fields[0] == 'volume':
+            all_fields.insert(0, 'close')
+
+        mask = None
+        data = []
+        for field in all_fields:
+            if field != 'volume':
+                out = np.full(shape, np.nan)
+            else:
+                out = np.zeros(shape, dtype=np.float64)
+
+            for i, sid in enumerate(sids):
+                carray = self._open_minute_file(field, sid)
+                a = carray[start_idx:end_idx + 1]
+
+                if mask is None:
+                    mask = a != 0
+
+                inverse_ratio = self._ohlc_ratio_inverse_for_sid(sid)
+                out[:len(mask), i][mask] = (
+                    a[mask] * inverse_ratio
+                )
+
+            if field in fields:
+                data.append(out)
+
+        return data

catalyst/exchange/exchange_blotter.py

@@ -0,0 +1,297 @@
+from time import sleep
+
+import pandas as pd
+from catalyst.assets._assets import TradingPair
+from logbook import Logger
+
+from catalyst.constants import LOG_LEVEL
+from catalyst.exchange.exchange_errors import ExchangeRequestError, \
+    ExchangePortfolioDataError, ExchangeTransactionError
+from catalyst.finance.blotter import Blotter
+from catalyst.finance.commission import CommissionModel
+from catalyst.finance.order import ORDER_STATUS, Order
+from catalyst.finance.slippage import SlippageModel
+from catalyst.finance.transaction import create_transaction, Transaction
+from catalyst.utils.input_validation import expect_types
+
+log = Logger('exchange_blotter', level=LOG_LEVEL)
+
+
+class TradingPairFeeSchedule(CommissionModel):
+    """
+    Calculates a commission for a transaction based on a per percentage fee.
+
+    Parameters
+    ----------
+    maker : float, optional
+        The percentage maker fee.
+
+    taker: float, optional
+        The percentage taker fee.
+    """
+
+    def __init__(self, maker=None, taker=None):
+        self.maker = maker
+        self.taker = taker
+
+    def __repr__(self):
+        return (
+            '{class_name}(maker={maker}, '
+            'taker={taker})'.format(
+                class_name=self.__class__.__name__,
+                maker=self.maker,
+                taker=self.taker,
+            )
+        )
+
+    def calculate(self, order, transaction):
+        """
+        Calculate the final fee based on the order parameters.
+
+        :param order: Order
+        :param transaction: Transaction
+
+        :return float:
+            The total commission.
+        """
+        cost = abs(transaction.amount) * transaction.price
+
+        asset = order.asset
+        maker = self.maker if self.maker is not None else asset.maker
+        taker = self.taker if self.taker is not None else asset.taker
+
+        multiplier = maker \
+            if ((order.amount > 0 and order.limit < transaction.price)
+                or (order.amount < 0 and order.limit > transaction.price)) \
+               and order.limit_reached else taker
+
+        # Assuming just the taker fee for now
+        fee = cost * multiplier
+        return fee
+
+
+class TradingPairFixedSlippage(SlippageModel):
+    """
+    Model slippage as a fixed spread.
+
+    Parameters
+    ----------
+    spread : float, optional
+        spread / 2 will be added to buys and subtracted from sells.
+    """
+
+    def __init__(self, spread=0.0001):
+        super(TradingPairFixedSlippage, self).__init__()
+        self.spread = spread
+
+    def __repr__(self):
+        return '{class_name}(spread={spread})'.format(
+            class_name=self.__class__.__name__, spread=self.spread,
+        )
+
+    def simulate(self, data, asset, orders_for_asset):
+        self._volume_for_bar = 0
+
+        price = data.current(asset, 'close')
+
+        dt = data.current_dt
+        for order in orders_for_asset:
+            if order.open_amount == 0:
+                continue
+
+            order.check_triggers(price, dt)
+            if not order.triggered:
+                log.debug('order has not reached the trigger at current '
+                          'price {}'.format(price))
+                continue
+
+            execution_price, execution_volume = self.process_order(data, order)
+
+            transaction = create_transaction(
+                order, dt, execution_price, execution_volume
+            )
+
+            self._volume_for_bar += abs(transaction.amount)
+            yield order, transaction
+
+    def process_order(self, data, order):
+        price = data.current(order.asset, 'close')
+
+        if order.amount > 0:
+            # Buy order
+            adj_price = price * (1 + self.spread)
+        else:
+            # Sell order
+            adj_price = price * (1 - self.spread)
+
+        log.debug('added slippage to price: {} => {}'.format(price, adj_price))
+
+        return adj_price, order.amount
+
+
+class ExchangeBlotter(Blotter):
+    def __init__(self, *args, **kwargs):
+        self.simulate_orders = kwargs.pop('simulate_orders', False)
+
+        self.exchanges = kwargs.pop('exchanges', None)
+        if not self.exchanges:
+            raise ValueError(
+                'ExchangeBlotter must have an `exchanges` attribute.'
+            )
+
+        super(ExchangeBlotter, self).__init__(*args, **kwargs)
+
+        # Using the equity models for now
+        # We may be able to define more sophisticated models based on the fee
+        # structure of each exchange.
+        self.slippage_models = {
+            TradingPair: TradingPairFixedSlippage()
+        }
+        self.commission_models = {
+            TradingPair: TradingPairFeeSchedule()
+        }
+
+        self.retry_delay = 5
+        self.retry_check_open_orders = 5
+
+    def exchange_order(self, asset, amount, style=None, attempt_index=0):
+        try:
+            exchange = self.exchanges[asset.exchange]
+            return exchange.order(
+                asset, amount, style
+            )
+        except ExchangeRequestError as e:
+            log.warn(
+                'order attempt {}: {}'.format(attempt_index, e)
+            )
+            if attempt_index < self.retry_order:
+                sleep(self.retry_delay)
+
+                return self.exchange_order(
+                    asset, amount, style, attempt_index + 1
+                )
+            else:
+                raise ExchangeTransactionError(
+                    transaction_type='order',
+                    attempts=attempt_index,
+                    error=e
+                )
+
+    @expect_types(asset=TradingPair)
+    def order(self, asset, amount, style, order_id=None):
+        log.debug('ordering {} {}'.format(amount, asset.symbol))
+        if amount == 0:
+            log.warn('skipping 0 amount orders')
+            return None
+
+        if self.simulate_orders:
+            return super(ExchangeBlotter, self).order(
+                asset, amount, style, order_id
+            )
+
+        else:
+            order = self.exchange_order(
+                asset, amount, style
+            )
+
+            self.open_orders[order.asset].append(order)
+            self.orders[order.id] = order
+            self.new_orders.append(order)
+
+            return order.id
+
+    def check_open_orders(self):
+        """
+        Loop through the list of open orders in the Portfolio object.
+        For each executed order found, create a transaction and apply to the
+        Portfolio.
+
+        Returns
+        -------
+        list[Transaction]
+
+        """
+        for asset in self.open_orders:
+            exchange = self.exchanges[asset.exchange]
+
+            for order in self.open_orders[asset]:
+                log.debug('found open order: {}'.format(order.id))
+
+                new_order, executed_price = exchange.get_order(order.id, asset)
+                log.debug(
+                    'got updated order {} {}'.format(
+                        new_order, executed_price
+                    )
+                )
+                order.status = new_order.status
+
+                if order.status == ORDER_STATUS.FILLED:
+                    order.commission = new_order.commission
+                    if order.amount != new_order.amount:
+                        log.warn(
+                            'executed order amount {} differs '
+                            'from original'.format(
+                                new_order.amount, order.amount
+                            )
+                        )
+                        order.amount = new_order.amount
+
+                    transaction = Transaction(
+                        asset=order.asset,
+                        amount=order.amount,
+                        dt=pd.Timestamp.utcnow(),
+                        price=executed_price,
+                        order_id=order.id,
+                        commission=order.commission
+                    )
+                    yield order, transaction
+
+                elif order.status == ORDER_STATUS.CANCELLED:
+                    yield order, None
+
+                else:
+                    delta = pd.Timestamp.utcnow() - order.dt
+                    log.info(
+                        'order {order_id} still open after {delta}'.format(
+                            order_id=order.id,
+                            delta=delta
+                        )
+                    )
+
+    def get_exchange_transactions(self, attempt_index=0):
+        closed_orders = []
+        transactions = []
+        commissions = []
+
+        try:
+            for order, txn in self.check_open_orders():
+                order.dt = txn.dt
+
+                transactions.append(txn)
+
+                if not order.open:
+                    closed_orders.append(order)
+
+            return transactions, commissions, closed_orders
+
+        except ExchangeRequestError as e:
+            log.warn(
+                'check open orders attempt {}: {}'.format(attempt_index, e)
+            )
+            if attempt_index < self.retry_check_open_orders:
+                sleep(self.retry_delay)
+                return self.get_exchange_transactions(attempt_index + 1)
+
+            else:
+                raise ExchangePortfolioDataError(
+                    data_type='order-status',
+                    attempts=attempt_index,
+                    error=e
+                )
+
+    def get_transactions(self, bar_data):
+        if self.simulate_orders:
+            return super(ExchangeBlotter, self).get_transactions(bar_data)
+
+        else:
+            return self.get_exchange_transactions()

catalyst/exchange/exchange_data_portal.py

@@ -0,0 +1,402 @@
+import abc
+from time import sleep
+
+import numpy as np
+import pandas as pd
+from catalyst.assets._assets import TradingPair
+from logbook import Logger
+
+from catalyst.constants import LOG_LEVEL, AUTO_INGEST
+from catalyst.data.data_portal import DataPortal
+from catalyst.exchange.exchange_bundle import ExchangeBundle
+from catalyst.exchange.exchange_errors import (
+    ExchangeRequestError,
+    ExchangeBarDataError,
+    PricingDataNotLoadedError)
+from catalyst.exchange.exchange_utils import get_frequency, \
+    resample_history_df, group_assets_by_exchange
+
+log = Logger('DataPortalExchange', level=LOG_LEVEL)
+
+
+class DataPortalExchangeBase(DataPortal):
+    def __init__(self, *args, **kwargs):
+
+        # TODO: put somewhere accessible by each algo
+        self.retry_get_history_window = 5
+        self.retry_get_spot_value = 5
+        self.retry_delay = 5
+
+        super(DataPortalExchangeBase, self).__init__(*args, **kwargs)
+
+    def _get_history_window(self,
+                            assets,
+                            end_dt,
+                            bar_count,
+                            frequency,
+                            field,
+                            data_frequency,
+                            ffill=True,
+                            attempt_index=0):
+        try:
+            exchange_assets = group_assets_by_exchange(assets)
+            if len(exchange_assets) > 1:
+                df_list = []
+                for exchange_name in exchange_assets:
+                    assets = exchange_assets[exchange_name]
+
+                    df_exchange = self.get_exchange_history_window(
+                        exchange_name,
+                        assets,
+                        end_dt,
+                        bar_count,
+                        frequency,
+                        field,
+                        data_frequency,
+                        ffill)
+
+                    df_list.append(df_exchange)
+
+                # Merging the values values of each exchange
+                return pd.concat(df_list)
+
+            else:
+                exchange_name = list(exchange_assets.keys())[0]
+                return self.get_exchange_history_window(
+                    exchange_name,
+                    assets,
+                    end_dt,
+                    bar_count,
+                    frequency,
+                    field,
+                    data_frequency,
+                    ffill)
+
+        except ExchangeRequestError as e:
+            log.warn(
+                'get history attempt {}: {}'.format(attempt_index, e)
+            )
+            if attempt_index < self.retry_get_history_window:
+                sleep(self.retry_delay)
+                return self._get_history_window(assets,
+                                                end_dt,
+                                                bar_count,
+                                                frequency,
+                                                field,
+                                                data_frequency,
+                                                ffill,
+                                                attempt_index + 1)
+            else:
+                raise ExchangeBarDataError(
+                    data_type='history',
+                    attempts=attempt_index,
+                    error=e
+                )
+
+    def get_history_window(self,
+                           assets,
+                           end_dt,
+                           bar_count,
+                           frequency,
+                           field,
+                           data_frequency=None,
+                           ffill=True):
+
+        if field == 'price':
+            field = 'close'
+
+        return self._get_history_window(assets,
+                                        end_dt,
+                                        bar_count,
+                                        frequency,
+                                        field,
+                                        data_frequency,
+                                        ffill)
+
+    @abc.abstractmethod
+    def get_exchange_history_window(self,
+                                    exchange_name,
+                                    assets,
+                                    end_dt,
+                                    bar_count,
+                                    frequency,
+                                    field,
+                                    data_frequency,
+                                    ffill=True):
+        pass
+
+    def _get_spot_value(self, assets, field, dt, data_frequency,
+                        attempt_index=0):
+        try:
+            if isinstance(assets, TradingPair):
+                spot_values = self.get_exchange_spot_value(
+                    assets.exchange, [assets], field, dt, data_frequency)
+
+                if not spot_values:
+                    return np.nan
+
+                return spot_values[0]
+
+            else:
+                exchange_assets = dict()
+                for asset in assets:
+                    if asset.exchange not in exchange_assets:
+                        exchange_assets[asset.exchange] = list()
+
+                    exchange_assets[asset.exchange].append(asset)
+
+                if len(list(exchange_assets.keys())) == 1:
+                    exchange_name = list(exchange_assets.keys())[0]
+                    return self.get_exchange_spot_value(
+                        exchange_name, assets, field, dt, data_frequency)
+
+                else:
+                    spot_values = []
+                    for exchange_name in exchange_assets:
+                        assets = exchange_assets[exchange_name]
+                        exchange_spot_values = self.get_exchange_spot_value(
+                            exchange_name,
+                            assets,
+                            field,
+                            dt,
+                            data_frequency
+                        )
+                        if len(assets) == 1:
+                            spot_values.append(exchange_spot_values)
+                        else:
+                            spot_values += exchange_spot_values
+
+                    return spot_values
+
+        except ExchangeRequestError as e:
+            log.warn(
+                'get spot value attempt {}: {}'.format(attempt_index, e)
+            )
+            if attempt_index < self.retry_get_spot_value:
+                sleep(self.retry_delay)
+                return self._get_spot_value(assets, field, dt, data_frequency,
+                                            attempt_index + 1)
+            else:
+                raise ExchangeBarDataError(
+                    data_type='spot',
+                    attempts=attempt_index,
+                    error=e
+                )
+
+    def get_spot_value(self, assets, field, dt, data_frequency):
+        if field == 'price':
+            field = 'close'
+
+        return self._get_spot_value(assets, field, dt, data_frequency)
+
+    @abc.abstractmethod
+    def get_exchange_spot_value(self, exchange_name, assets, field, dt,
+                                data_frequency):
+        return
+
+    def get_adjusted_value(self, asset, field, dt,
+                           perspective_dt,
+                           data_frequency,
+                           spot_value=None):
+        # TODO: does this pertain to cryptocurrencies?
+        log.warn('get_adjusted_value is not implemented yet!')
+        return spot_value
+
+
+class DataPortalExchangeLive(DataPortalExchangeBase):
+    def __init__(self, *args, **kwargs):
+        self.exchanges = kwargs.pop('exchanges', None)
+        super(DataPortalExchangeLive, self).__init__(*args, **kwargs)
+
+    def get_exchange_history_window(self,
+                                    exchange_name,
+                                    assets,
+                                    end_dt,
+                                    bar_count,
+                                    frequency,
+                                    field,
+                                    data_frequency,
+                                    ffill=True):
+        """
+        Fetching price history window from the exchange.
+
+        Parameters
+        ----------
+        exchange_name: Exchange
+        assets: list[TradingPair]
+        end_dt: datetime
+        bar_count: int
+        frequency: str
+        field: str
+        data_frequency: str
+        ffill: bool
+
+        Returns
+        -------
+        DataFrame
+
+        """
+        exchange = self.exchanges[exchange_name]
+
+        df = exchange.get_history_window(
+            assets,
+            end_dt,
+            bar_count,
+            frequency,
+            field,
+            data_frequency,
+            False)
+        return df
+
+    def get_exchange_spot_value(self, exchange_name, assets, field, dt,
+                                data_frequency):
+        """
+        A spot value for the exchange.
+
+        Parameters
+        ----------
+        exchange_name: str
+        assets: list[TradingPair]
+        field: str
+        dt: datetime
+        data_frequency: str
+
+        Returns
+        -------
+        float
+
+        """
+        exchange = self.exchanges[exchange_name]
+        exchange_spot_values = exchange.get_spot_value(
+            assets, field, dt, data_frequency)
+
+        return exchange_spot_values
+
+
+class DataPortalExchangeBacktest(DataPortalExchangeBase):
+    def __init__(self, *args, **kwargs):
+        self.exchange_names = kwargs.pop('exchange_names', None)
+
+        super(DataPortalExchangeBacktest, self).__init__(*args, **kwargs)
+
+        self.exchange_bundles = dict()
+        self.history_loaders = dict()
+        self.minute_history_loaders = dict()
+
+        for name in self.exchange_names:
+            self.exchange_bundles[name] = ExchangeBundle(name)
+
+    def _get_first_trading_day(self, assets):
+        first_date = None
+        for asset in assets:
+            if first_date is None or asset.start_date > first_date:
+                first_date = asset.start_date
+        return first_date
+
+    def get_exchange_history_window(self,
+                                    exchange_name,
+                                    assets,
+                                    end_dt,
+                                    bar_count,
+                                    frequency,
+                                    field,
+                                    data_frequency,
+                                    ffill=True):
+        """
+        Fetching price history window from the exchange bundle.
+
+        Parameters
+        ----------
+        exchange: Exchange
+        assets: list[TradingPair]
+        end_dt: datetime
+        bar_count: int
+        frequency: str
+        field: str
+        data_frequency: str
+        ffill: bool
+
+        Returns
+        -------
+        DataFrame
+
+        """
+        bundle = self.exchange_bundles[exchange_name]  # type: ExchangeBundle
+
+        freq, candle_size, unit, adj_data_frequency = get_frequency(
+            frequency, data_frequency
+        )
+        adj_bar_count = candle_size * bar_count
+        trailing_bar_count = candle_size - 1
+
+        if data_frequency == 'minute' and adj_data_frequency == 'daily':
+            end_dt = end_dt.floor('1D')
+
+        series = bundle.get_history_window_series_and_load(
+            assets=assets,
+            end_dt=end_dt,
+            bar_count=adj_bar_count,
+            field=field,
+            data_frequency=adj_data_frequency,
+            algo_end_dt=self._last_available_session,
+            trailing_bar_count=trailing_bar_count
+        )
+
+        df = resample_history_df(pd.DataFrame(series), freq, field)
+        return df
+
+    def get_exchange_spot_value(self,
+                                exchange_name,
+                                assets,
+                                field,
+                                dt,
+                                data_frequency
+                                ):
+        """
+        A spot value for the exchange bundle. Try to ingest data if not in
+        the bundle.
+
+        Parameters
+        ----------
+        exchange_name: str
+        assets: list[TradingPair]
+        field: str
+        dt: datetime
+        data_frequency: str
+
+        Returns
+        -------
+        float
+
+        """
+        bundle = self.exchange_bundles[exchange_name]
+        if data_frequency == 'daily':
+            dt = dt.floor('1D')
+        else:
+            dt = dt.floor('1 min')
+
+        if AUTO_INGEST:
+            try:
+                return bundle.get_spot_values(
+                    assets, field, dt, data_frequency
+                )
+            except PricingDataNotLoadedError:
+                log.info(
+                    'pricing data for {symbol} not found on {dt}'
+                    ', updating the bundles.'.format(
+                        symbol=[asset.symbol for asset in assets],
+                        dt=dt
+                    )
+                )
+                bundle.ingest_assets(
+                    assets=assets,
+                    start_dt=self._first_trading_day,
+                    end_dt=self._last_available_session,
+                    data_frequency=data_frequency,
+                    show_progress=True
+                )
+                return bundle.get_spot_values(
+                    assets, field, dt, data_frequency, True
+                )
+        else:
+            return bundle.get_spot_values(assets, field, dt, data_frequency)

catalyst/exchange/exchange_errors.py

@@ -0,0 +1,271 @@
+import sys
+import traceback
+
+from catalyst.errors import ZiplineError
+
+
+def silent_except_hook(exctype, excvalue, exctraceback):
+    if exctype in [PricingDataBeforeTradingError, PricingDataNotLoadedError,
+                   SymbolNotFoundOnExchange, NoDataAvailableOnExchange,
+                   ExchangeAuthEmpty]:
+        fn = traceback.extract_tb(exctraceback)[-1][0]
+        ln = traceback.extract_tb(exctraceback)[-1][1]
+        print("Error traceback: {1} (line {2})\n"
+              "{0.__name__}:  {3}".format(exctype, fn, ln, excvalue))
+    else:
+        sys.__excepthook__(exctype, excvalue, exctraceback)
+
+
+sys.excepthook = silent_except_hook
+
+
+class ExchangeRequestError(ZiplineError):
+    msg = (
+        'Request failed: {error}'
+    ).strip()
+
+
+class ExchangeRequestErrorTooManyAttempts(ZiplineError):
+    msg = (
+        'Request failed: {error}, giving up after {attempts} attempts'
+    ).strip()
+
+
+class ExchangeBarDataError(ZiplineError):
+    msg = (
+        'Unable to retrieve bar data: {data_type}, ' +
+        'giving up after {attempts} attempts: {error}'
+    ).strip()
+
+
+class ExchangePortfolioDataError(ZiplineError):
+    msg = (
+        'Unable to retrieve portfolio data: {data_type}, ' +
+        'giving up after {attempts} attempts: {error}'
+    ).strip()
+
+
+class ExchangeTransactionError(ZiplineError):
+    msg = (
+        'Unable to execute transaction: {transaction_type}, ' +
+        'giving up after {attempts} attempts: {error}'
+    ).strip()
+
+
+class ExchangeNotFoundError(ZiplineError):
+    msg = (
+        'Exchange {exchange_name} not found. Please specify exchanges '
+        'supported by Catalyst and verify spelling for accuracy.'
+    ).strip()
+
+
+class ExchangeAuthNotFound(ZiplineError):
+    msg = (
+        'Please create an auth.json file containing the api token and key for '
+        'exchange {exchange}. Place the file here: {filename}'
+    ).strip()
+
+
+class ExchangeAuthEmpty(ZiplineError):
+    msg = (
+        'Please enter your API token key and secret for exchange {exchange} '
+        'in the following file: {filename}'
+    ).strip()
+
+
+class ExchangeSymbolsNotFound(ZiplineError):
+    msg = (
+        'Unable to download or find a local copy of symbols.json for exchange '
+        '{exchange}. The file should be here: {filename}'
+    ).strip()
+
+
+class AlgoPickleNotFound(ZiplineError):
+    msg = (
+        'Pickle not found for algo {algo} in path {filename}'
+    ).strip()
+
+
+class InvalidHistoryFrequencyAlias(ZiplineError):
+    msg = (
+        'Invalid frequency alias {freq}. Valid suffixes are M (minute) '
+        'and D (day). For example, these aliases would be valid '
+        '1M, 5M, 1D.'
+    ).strip()
+
+
+class InvalidHistoryFrequencyError(ZiplineError):
+    msg = (
+        'Frequency {frequency} not supported by the exchange.'
+    ).strip()
+
+
+class MismatchingFrequencyError(ZiplineError):
+    msg = (
+        'Bar aggregate frequency {frequency} not compatible with '
+        'data frequency {data_frequency}.'
+    ).strip()
+
+
+class InvalidSymbolError(ZiplineError):
+    msg = (
+        'Invalid trading pair symbol: {symbol}. '
+        'Catalyst symbols must follow this convention: '
+        '[Market Currency]_[Base Currency]. For example: eth_usd, btc_usd, '
+        'neo_eth, ubq_btc. Error details: {error}'
+    ).strip()
+
+
+class InvalidOrderStyle(ZiplineError):
+    msg = (
+        'Order style {style} not supported by exchange {exchange}.'
+    ).strip()
+
+
+class CreateOrderError(ZiplineError):
+    msg = (
+        'Unable to create order on exchange {exchange} {error}.'
+    ).strip()
+
+
+class OrderNotFound(ZiplineError):
+    msg = (
+        'Order {order_id} not found on exchange {exchange}.'
+    ).strip()
+
+
+class OrphanOrderError(ZiplineError):
+    msg = (
+        'Order {order_id} found in exchange {exchange} but not tracked by '
+        'the algorithm.'
+    ).strip()
+
+
+class OrphanOrderReverseError(ZiplineError):
+    msg = (
+        'Order {order_id} tracked by algorithm, but not found in exchange '
+        '{exchange}.'
+    ).strip()
+
+
+class OrderCancelError(ZiplineError):
+    msg = (
+        'Unable to cancel order {order_id} on exchange {exchange} {error}.'
+    ).strip()
+
+
+class SidHashError(ZiplineError):
+    msg = (
+        'Unable to hash sid from symbol {symbol}.'
+    ).strip()
+
+
+class BaseCurrencyNotFoundError(ZiplineError):
+    msg = (
+        'Algorithm base currency {base_currency} not found in exchange '
+        '{exchange}.'
+    ).strip()
+
+
+class MismatchingBaseCurrencies(ZiplineError):
+    msg = (
+        'Unable to trade with base currency {base_currency} when the '
+        'algorithm uses {algo_currency}.'
+    ).strip()
+
+
+class MismatchingBaseCurrenciesExchanges(ZiplineError):
+    msg = (
+        'Unable to trade with base currency {base_currency} when the '
+        'exchange {exchange_name} users {exchange_currency}.'
+    ).strip()
+
+
+class SymbolNotFoundOnExchange(ZiplineError):
+    """
+    Raised when a symbol() call contains a non-existent symbol.
+    """
+    msg = ('Symbol {symbol} not found on exchange {exchange}. '
+           'Choose from: {supported_symbols}').strip()
+
+
+class BundleNotFoundError(ZiplineError):
+    msg = ('Unable to find bundle data for exchange {exchange} and '
+           'data frequency {data_frequency}.'
+           'Please ingest some price data.'
+           'See `catalyst ingest-exchange --help` for details.').strip()
+
+
+class TempBundleNotFoundError(ZiplineError):
+    msg = ('Temporary bundle not found in: {path}.').strip()
+
+
+class EmptyValuesInBundleError(ZiplineError):
+    msg = ('{name} with end minute {end_minute} has empty rows '
+           'in ranges: {dates}').strip()
+
+
+class PricingDataBeforeTradingError(ZiplineError):
+    msg = ('Pricing data for trading pairs {symbols} on exchange {exchange} '
+           'starts on {first_trading_day}, but you are either trying to trade '
+           'or retrieve pricing data on {dt}. Adjust your dates accordingly.'
+           ).strip()
+
+
+class PricingDataNotLoadedError(ZiplineError):
+    msg = ('Missing data for {exchange} {symbols} in date range '
+           '[{start_dt} - {end_dt}]'
+           '\nPlease run: `catalyst ingest-exchange -x {exchange} -f '
+           '{data_frequency} -i {symbol_list}`. See catalyst documentation '
+           'for details.').strip()
+
+
+class PricingDataValueError(ZiplineError):
+    msg = ('Unable to retrieve pricing data for {exchange} {symbol} '
+           '[{start_dt} - {end_dt}]: {error}').strip()
+
+
+class DataCorruptionError(ZiplineError):
+    msg = ('Unable to validate data for {exchange} {symbols} in date range '
+           '[{start_dt} - {end_dt}]. The data is either corrupted or '
+           'unavailable. Please try deleting this bundle:'
+           '\n`catalyst clean-exchange -x {exchange}\n'
+           'Then, ingest the data again. Please contact the Catalyst team if '
+           'the issue persists.').strip()
+
+
+class ApiCandlesError(ZiplineError):
+    msg = ('Unable to fetch candles from the remote API: {error}.').strip()
+
+
+class NoDataAvailableOnExchange(ZiplineError):
+    msg = (
+        'Requested data for trading pair {symbol} is not available on '
+        'exchange {exchange} '
+        'in `{data_frequency}` frequency at this time. '
+        'Check `http://enigma.co/catalyst/status` for market coverage.'
+    ).strip()
+
+
+class NoValueForField(ZiplineError):
+    msg = ('Value not found for field: {field}.').strip()
+
+
+class OrderTypeNotSupported(ZiplineError):
+    msg = (
+        'Order type `{order_type}` not currencly supported by Catalyst. '
+        'Please use `limit` or `market` orders only.').strip()
+
+
+class NotEnoughCapitalError(ZiplineError):
+    msg = (
+        'Not enough capital on exchange {exchange} for trading. Each '
+        'exchange should contain at least as much {base_currency} '
+        'as the specified `capital_base`. The current balance {balance} is '
+        'lower than the `capital_base`: {capital_base}').strip()
+
+class LastCandleTooEarlyError(ZiplineError):
+    msg = (
+        'The trade date of the last candle {last_traded} is before the '
+        'specified end date minus one candle {end_dt}. Please verify how '
+        '{exchange} calculates the start date of OHLCV candles.').strip()

catalyst/exchange/exchange_execution.py

@@ -0,0 +1,67 @@
+from catalyst.finance.execution import LimitOrder, StopOrder, StopLimitOrder
+
+
+class ExchangeLimitOrder(LimitOrder):
+    def get_limit_price(self, is_buy):
+        """
+        We may be trading Satoshis with 8 decimals, we cannot round numbers.
+
+        Parameters
+        ----------
+        is_buy: bool
+
+        Returns
+        -------
+        float
+
+        """
+        return self.limit_price
+
+
+class ExchangeStopOrder(StopOrder):
+    def get_stop_price(self, is_buy):
+        """
+        We may be trading Satoshis with 8 decimals, we cannot round numbers.
+
+        Parameters
+        ----------
+        is_buy: bool
+
+        Returns
+        -------
+        float
+
+        """
+        return self.stop_price
+
+
+class ExchangeStopLimitOrder(StopLimitOrder):
+    def get_limit_price(self, is_buy):
+        """
+        We may be trading Satoshis with 8 decimals, we cannot round numbers.
+
+        Parameters
+        ----------
+        is_buy: bool
+
+        Returns
+        -------
+        float
+
+        """
+        return self.limit_price
+
+    def get_stop_price(self, is_buy):
+        """
+        We may be trading Satoshis with 8 decimals, we cannot round numbers.
+
+        Parameters
+        ----------
+        is_buy: bool
+
+        Returns
+        -------
+        float
+
+        """
+        return self.stop_price

catalyst/exchange/exchange_portfolio.py

@@ -0,0 +1,132 @@
+import numpy as np
+from logbook import Logger
+
+from catalyst.constants import LOG_LEVEL
+from catalyst.protocol import Portfolio, Positions, Position
+
+log = Logger('ExchangePortfolio', level=LOG_LEVEL)
+
+
+class ExchangePortfolio(Portfolio):
+    """
+    Since the goal is to support multiple exchanges, it makes sense to
+    include additional stats in the portfolio object. This fills the role
+    of Blotter and Portfolio in live mode.
+
+    Instead of relying on the performance tracker, each exchange portfolio
+    tracks its own holding. This offers a separation between tracking an
+    exchange and the statistics of the algorithm.
+    """
+
+    def __init__(self, start_date, starting_cash=None):
+        self.capital_used = 0.0
+        self.starting_cash = starting_cash
+        self.portfolio_value = starting_cash
+        self.pnl = 0.0
+        self.returns = 0.0
+        self.cash = starting_cash
+        self.positions = Positions()
+        self.start_date = start_date
+        self.positions_value = 0.0
+        self.open_orders = dict()
+
+    def create_order(self, order):
+        """
+        Create an open order and store in memory.
+
+        Parameters
+        ----------
+        order: Order
+
+        """
+        log.debug('creating order {}'.format(order.id))
+
+        open_orders = self.open_orders[order.asset] \
+            if order.asset is self.open_orders else []
+
+        open_orders.append(order)
+
+        self.open_orders[order.asset] = open_orders
+
+        order_position = self.positions[order.asset] \
+            if order.asset in self.positions else None
+
+        if order_position is None:
+            order_position = Position(order.asset)
+            self.positions[order.asset] = order_position
+
+        order_position.amount += order.amount
+        log.debug('open order added to portfolio')
+
+    def _remove_open_order(self, order):
+        try:
+            open_orders = self.open_orders[order.asset]
+            if order in open_orders:
+                open_orders.remove(order)
+
+        except Exception:
+            raise ValueError(
+                'unable to clear order not found in open order list.'
+            )
+
+    def execute_order(self, order, transaction):
+        """
+        Update the open orders and positions to apply an executed order.
+
+        Unlike with backtesting, we do not need to add slippage and fees.
+        The executed price includes transaction fees.
+
+        Parameters
+        ----------
+        order: Order
+        transaction: Transaction
+
+        """
+        log.debug('executing order {}'.format(order.id))
+        self._remove_open_order(order)
+
+        order_position = self.positions[order.asset] \
+            if order.asset in self.positions else None
+
+        if order_position is None:
+            raise ValueError(
+                'Trying to execute order for a position not held:'
+                ' {}'.format(order.id)
+            )
+
+        self.capital_used += order.amount * transaction.price
+
+        if order.amount > 0:
+            if order_position.cost_basis > 0:
+                order_position.cost_basis = np.average(
+                    [order_position.cost_basis, transaction.price],
+                    weights=[order_position.amount, order.amount]
+                )
+            else:
+                order_position.cost_basis = transaction.price
+
+        log.debug('updated portfolio with executed order')
+
+    def remove_order(self, order):
+        """
+        Removing an open order.
+
+        Parameters
+        ----------
+        order: Order
+
+        """
+        log.info('removing cancelled order {}'.format(order.id))
+        self._remove_open_order(order)
+
+        order_position = self.positions[order.asset] \
+            if order.asset in self.positions else None
+
+        if order_position is None:
+            raise ValueError(
+                'Trying to remove order for a position not held: %s' % order.id
+            )
+
+        order_position.amount -= order.amount
+
+        log.debug('removed order from portfolio')

catalyst/exchange/exchange_utils.py

@@ -0,0 +1,648 @@
+import hashlib
+import json
+import os
+import pickle
+import re
+import shutil
+from datetime import date, datetime
+
+import pandas as pd
+from catalyst.assets._assets import TradingPair
+from six import string_types
+from six.moves.urllib import request
+
+from catalyst.constants import DATE_FORMAT, SYMBOLS_URL
+from catalyst.exchange.exchange_errors import ExchangeSymbolsNotFound, \
+    InvalidHistoryFrequencyError, InvalidHistoryFrequencyAlias
+from catalyst.utils.paths import data_root, ensure_directory, \
+    last_modified_time
+
+
+def get_sid(symbol):
+    """
+    Create a sid by hashing the symbol of a currency pair.
+
+    Parameters
+    ----------
+    symbol: str
+
+    Returns
+    -------
+    int
+        The resulting sid.
+
+    """
+    sid = int(
+        hashlib.sha256(symbol.encode('utf-8')).hexdigest(), 16
+    ) % 10 ** 6
+    return sid
+
+
+def get_exchange_folder(exchange_name, environ=None):
+    """
+    The root path of an exchange folder.
+
+    Parameters
+    ----------
+    exchange_name: str
+    environ:
+
+    Returns
+    -------
+    str
+
+    """
+    if not environ:
+        environ = os.environ
+
+    root = data_root(environ)
+    exchange_folder = os.path.join(root, 'exchanges', exchange_name)
+    ensure_directory(exchange_folder)
+
+    return exchange_folder
+
+
+def get_exchange_symbols_filename(exchange_name, is_local=False, environ=None):
+    """
+    The absolute path of the exchange's symbol.json file.
+
+    Parameters
+    ----------
+    exchange_name:
+    environ:
+
+    Returns
+    -------
+    str
+
+    """
+    name = 'symbols.json' if not is_local else 'symbols_local.json'
+    exchange_folder = get_exchange_folder(exchange_name, environ)
+    return os.path.join(exchange_folder, name)
+
+
+def download_exchange_symbols(exchange_name, environ=None):
+    """
+    Downloads the exchange's symbols.json from the repository.
+
+    Parameters
+    ----------
+    exchange_name: str
+    environ:
+
+    Returns
+    -------
+    str
+
+    """
+    filename = get_exchange_symbols_filename(exchange_name)
+    url = SYMBOLS_URL.format(exchange=exchange_name)
+    response = request.urlretrieve(url=url, filename=filename)
+    return response
+
+
+def symbols_parser(asset_def):
+    for key, value in asset_def.items():
+        match = isinstance(value, string_types) \
+                and re.search(r'(\d{4}-\d{2}-\d{2})', value)
+
+        if match:
+            try:
+                asset_def[key] = pd.to_datetime(value, utc=True)
+            except ValueError:
+                pass
+
+    return asset_def
+
+
+def get_exchange_symbols(exchange_name, is_local=False, environ=None):
+    """
+    The de-serialized content of the exchange's symbols.json.
+
+    Parameters
+    ----------
+    exchange_name: str
+    is_local: bool
+    environ:
+
+    Returns
+    -------
+    Object
+
+    """
+    filename = get_exchange_symbols_filename(exchange_name, is_local)
+
+    if not is_local and (not os.path.isfile(filename) or pd.Timedelta(
+                pd.Timestamp('now', tz='UTC') - last_modified_time(
+                    filename)).days > 1):
+        download_exchange_symbols(exchange_name, environ)
+
+    if os.path.isfile(filename):
+        with open(filename) as data_file:
+            try:
+                data = json.load(data_file, object_hook=symbols_parser)
+                return data
+
+            except ValueError:
+                return dict()
+    else:
+        raise ExchangeSymbolsNotFound(
+            exchange=exchange_name,
+            filename=filename
+        )
+
+
+def save_exchange_symbols(exchange_name, assets, is_local=False, environ=None):
+    """
+    Save assets into an exchange_symbols file.
+
+    Parameters
+    ----------
+    exchange_name: str
+    assets: list[dict[str, object]]
+    is_local: bool
+    environ
+
+    Returns
+    -------
+
+    """
+    asset_dicts = dict()
+    for symbol in assets:
+        asset_dicts[symbol] = assets[symbol].to_dict()
+
+    filename = get_exchange_symbols_filename(
+        exchange_name, is_local, environ
+    )
+    with open(filename, 'wt') as handle:
+        json.dump(asset_dicts, handle, indent=4, default=symbols_serial)
+
+
+def get_symbols_string(assets):
+    """
+    A concatenated string of symbols from a list of assets.
+
+    Parameters
+    ----------
+    assets: list[TradingPair]
+
+    Returns
+    -------
+    str
+
+    """
+    array = [assets] if isinstance(assets, TradingPair) else assets
+    return ', '.join([asset.symbol for asset in array])
+
+
+def get_exchange_auth(exchange_name, environ=None):
+    """
+    The de-serialized contend of the exchange's auth.json file.
+
+    Parameters
+    ----------
+    exchange_name: str
+    environ:
+
+    Returns
+    -------
+    Object
+
+    """
+    exchange_folder = get_exchange_folder(exchange_name, environ)
+    filename = os.path.join(exchange_folder, 'auth.json')
+
+    if os.path.isfile(filename):
+        with open(filename) as data_file:
+            data = json.load(data_file)
+            return data
+    else:
+        data = dict(name=exchange_name, key='', secret='')
+        with open(filename, 'w') as f:
+            json.dump(data, f, sort_keys=False, indent=2,
+                      separators=(',', ':'))
+            return data
+
+
+def delete_algo_folder(algo_name, environ=None):
+    """
+    Delete the folder containing the algo state.
+
+    Parameters
+    ----------
+    algo_name: str
+    environ:
+
+    Returns
+    -------
+    str
+
+    """
+    folder = get_algo_folder(algo_name, environ)
+    shutil.rmtree(folder)
+
+
+def get_algo_folder(algo_name, environ=None):
+    """
+    The algorithm root folder of the algorithm.
+
+    Parameters
+    ----------
+    algo_name: str
+    environ:
+
+    Returns
+    -------
+    str
+
+    """
+    if not environ:
+        environ = os.environ
+
+    root = data_root(environ)
+    algo_folder = os.path.join(root, 'live_algos', algo_name)
+    ensure_directory(algo_folder)
+
+    return algo_folder
+
+
+def get_algo_object(algo_name, key, environ=None, rel_path=None):
+    """
+    The de-serialized object of the algo name and key.
+
+    Parameters
+    ----------
+    algo_name: str
+    key: str
+    environ:
+    rel_path: str
+
+    Returns
+    -------
+    Object
+
+    """
+    if algo_name is None:
+        return None
+
+    folder = get_algo_folder(algo_name, environ)
+
+    if rel_path is not None:
+        folder = os.path.join(folder, rel_path)
+
+    filename = os.path.join(folder, key + '.p')
+
+    if os.path.isfile(filename):
+        try:
+            with open(filename, 'rb') as handle:
+                return pickle.load(handle)
+        except Exception:
+            return None
+    else:
+        return None
+
+
+def save_algo_object(algo_name, key, obj, environ=None, rel_path=None):
+    """
+    Serialize and save an object by algo name and key.
+
+    Parameters
+    ----------
+    algo_name: str
+    key: str
+    obj: Object
+    environ:
+    rel_path: str
+
+    """
+    folder = get_algo_folder(algo_name, environ)
+
+    if rel_path is not None:
+        folder = os.path.join(folder, rel_path)
+        ensure_directory(folder)
+
+    filename = os.path.join(folder, key + '.p')
+
+    with open(filename, 'wb') as handle:
+        pickle.dump(obj, handle, protocol=pickle.HIGHEST_PROTOCOL)
+
+
+def get_algo_df(algo_name, key, environ=None, rel_path=None):
+    """
+    The de-serialized DataFrame of an algo name and key.
+
+    Parameters
+    ----------
+    algo_name: str
+    key: str
+    environ:
+    rel_path: str
+
+    Returns
+    -------
+    DataFrame
+
+    """
+    folder = get_algo_folder(algo_name, environ)
+
+    if rel_path is not None:
+        folder = os.path.join(folder, rel_path)
+
+    filename = os.path.join(folder, key + '.csv')
+
+    if os.path.isfile(filename):
+        try:
+            with open(filename, 'rb') as handle:
+                return pd.read_csv(handle, index_col=0, parse_dates=True)
+        except IOError:
+            return pd.DataFrame()
+    else:
+        return pd.DataFrame()
+
+
+def save_algo_df(algo_name, key, df, environ=None, rel_path=None):
+    """
+    Serialize to csv and save a DataFrame by algo name and key.
+
+    Parameters
+    ----------
+    algo_name: str
+    key: str
+    df: pd.DataFrame
+    environ:
+    rel_path: str
+
+    """
+    folder = get_algo_folder(algo_name, environ)
+    if rel_path is not None:
+        folder = os.path.join(folder, rel_path)
+        ensure_directory(folder)
+
+    filename = os.path.join(folder, key + '.csv')
+
+    with open(filename, 'wt') as handle:
+        df.to_csv(handle, encoding='UTF_8')
+
+
+def get_exchange_minute_writer_root(exchange_name, environ=None):
+    """
+    The minute writer folder for the exchange.
+
+    Parameters
+    ----------
+    exchange_name: str
+    environ:
+
+    Returns
+    -------
+    BcolzExchangeBarWriter
+
+    """
+    exchange_folder = get_exchange_folder(exchange_name, environ)
+
+    minute_data_folder = os.path.join(exchange_folder, 'minute_data')
+    ensure_directory(minute_data_folder)
+
+    return minute_data_folder
+
+
+def get_exchange_bundles_folder(exchange_name, environ=None):
+    """
+    The temp folder for bundle downloads by algo name.
+
+    Parameters
+    ----------
+    exchange_name: str
+    environ:
+
+    Returns
+    -------
+    str
+
+    """
+    exchange_folder = get_exchange_folder(exchange_name, environ)
+
+    temp_bundles = os.path.join(exchange_folder, 'temp_bundles')
+    ensure_directory(temp_bundles)
+
+    return temp_bundles
+
+
+def symbols_serial(obj):
+    """
+    JSON serializer for objects not serializable by default json code
+
+    Parameters
+    ----------
+    obj: Object
+
+    Returns
+    -------
+    str
+
+    """
+    if isinstance(obj, (datetime, date)):
+        return obj.floor('1D').strftime(DATE_FORMAT)
+
+    raise TypeError("Type %s not serializable" % type(obj))
+
+
+def perf_serial(obj):
+    """
+    JSON serializer for objects not serializable by default json code
+
+    Parameters
+    ----------
+    obj: Object
+
+    Returns
+    -------
+    str
+
+    """
+    if isinstance(obj, (datetime, date)):
+        return obj.isoformat()
+
+    raise TypeError("Type %s not serializable" % type(obj))
+
+
+def get_common_assets(exchanges):
+    """
+    The assets available in all specified exchanges.
+
+    Parameters
+    ----------
+    exchanges: list[Exchange]
+
+    Returns
+    -------
+    list[TradingPair]
+
+    """
+    symbols = []
+    for exchange_name in exchanges:
+        s = [asset.symbol for asset in exchanges[exchange_name].get_assets()]
+        symbols.append(s)
+
+    inter_symbols = set.intersection(*map(set, symbols))
+
+    assets = []
+    for symbol in inter_symbols:
+        for exchange_name in exchanges:
+            asset = exchanges[exchange_name].get_asset(symbol)
+            assets.append(asset)
+
+    return assets
+
+
+def get_frequency(freq, data_frequency):
+    """
+    Get the frequency parameters.
+
+    Notes
+    -----
+    We're trying to use Pandas convention for frequency aliases.
+
+    Parameters
+    ----------
+    freq: str
+    data_frequency: str
+
+    Returns
+    -------
+    str, int, str, str
+
+    """
+    if freq == 'minute':
+        unit = 'T'
+        candle_size = 1
+
+    elif freq == 'daily':
+        unit = 'D'
+        candle_size = 1
+
+    else:
+        freq_match = re.match(r'([0-9].*)?(m|M|d|D|h|H|T)', freq, re.M | re.I)
+        if freq_match:
+            candle_size = int(freq_match.group(1)) if freq_match.group(1) \
+                else 1
+            unit = freq_match.group(2)
+
+        else:
+            raise InvalidHistoryFrequencyError(frequency=freq)
+
+    if unit.lower() == 'd':
+        alias = '{}D'.format(candle_size)
+
+        if data_frequency == 'minute':
+            data_frequency = 'daily'
+
+    elif unit.lower() == 'm' or unit == 'T':
+        alias = '{}T'.format(candle_size)
+
+        if data_frequency == 'daily':
+            data_frequency = 'minute'
+
+    # elif unit.lower() == 'h':
+    #     candle_size = candle_size * 60
+    #
+    #     alias = '{}T'.format(candle_size)
+    #     if data_frequency == 'daily':
+    #         data_frequency = 'minute'
+
+    else:
+        raise InvalidHistoryFrequencyAlias(freq=freq)
+
+    return alias, candle_size, unit, data_frequency
+
+
+def resample_history_df(df, freq, field):
+    """
+    Resample the OHCLV DataFrame using the specified frequency.
+
+    Parameters
+    ----------
+    df: DataFrame
+    freq: str
+    field: str
+
+    Returns
+    -------
+    DataFrame
+
+    """
+    if field == 'open':
+        agg = 'first'
+    elif field == 'high':
+        agg = 'max'
+    elif field == 'low':
+        agg = 'min'
+    elif field == 'close':
+        agg = 'last'
+    elif field == 'volume':
+        agg = 'sum'
+    else:
+        raise ValueError('Invalid field.')
+
+    resampled_df = df.resample(freq).agg(agg)
+    return resampled_df
+
+
+def mixin_market_params(exchange_name, params, market):
+    """
+    Applies a CCXT market dict to parameters of TradingPair init.
+
+    Parameters
+    ----------
+    params: dict[Object]
+    market: dict[Object]
+
+    Returns
+    -------
+
+    """
+    # TODO: make this more externalized / configurable
+    if 'lot' in market:
+        params['min_trade_size'] = market['lot']
+        params['lot'] = market['lot']
+
+    if exchange_name == 'bitfinex':
+        params['maker'] = 0.001
+        params['taker'] = 0.002
+
+    elif 'maker' in market and 'taker' in market \
+            and market['maker'] is not None and market['taker'] is not None:
+        params['maker'] = market['maker']
+        params['taker'] = market['taker']
+
+    else:
+        # TODO: default commission, make configurable
+        params['maker'] = 0.0015
+        params['taker'] = 0.0025
+
+    info = market['info'] if 'info' in market else None
+    if info:
+        if 'minimum_order_size' in info:
+            params['min_trade_size'] = float(info['minimum_order_size'])
+
+            if 'lot' not in params:
+                params['lot'] = params['min_trade_size']
+
+
+def from_ms_timestamp(ms):
+    return pd.to_datetime(ms, unit='ms', utc=True)
+
+
+def get_epoch():
+    return pd.to_datetime('1970-1-1', utc=True)
+
+
+def group_assets_by_exchange(assets):
+    exchange_assets = dict()
+    for asset in assets:
+        if asset.exchange not in exchange_assets:
+            exchange_assets[asset.exchange] = list()
+
+        exchange_assets[asset.exchange].append(asset)
+
+    return exchange_assets

catalyst/exchange/factory.py

@@ -0,0 +1,34 @@
+import os
+
+from catalyst.exchange.ccxt.ccxt_exchange import CCXT
+from catalyst.exchange.exchange_errors import ExchangeAuthEmpty
+from catalyst.exchange.exchange_utils import get_exchange_auth, \
+    get_exchange_folder
+
+
+def get_exchange(exchange_name, base_currency=None, must_authenticate=False):
+    exchange_auth = get_exchange_auth(exchange_name)
+
+    has_auth = (exchange_auth['key'] != '' and exchange_auth['secret'] != '')
+    if must_authenticate and not has_auth:
+        raise ExchangeAuthEmpty(
+            exchange=exchange_name.title(),
+            filename=os.path.join(
+                get_exchange_folder(exchange_name), 'auth.json'
+            )
+        )
+
+    return CCXT(
+        exchange_name=exchange_name,
+        key=exchange_auth['key'],
+        secret=exchange_auth['secret'],
+        base_currency=base_currency,
+    )
+
+
+def get_exchanges(exchange_names):
+    exchanges = dict()
+    for exchange_name in exchange_names:
+        exchanges[exchange_name] = get_exchange(exchange_name)
+
+    return exchanges

catalyst/exchange/live_graph_clock.py

@@ -0,0 +1,233 @@
+import pandas as pd
+from catalyst.gens.sim_engine import (
+    BAR,
+    SESSION_START
+)
+from logbook import Logger
+
+from catalyst.constants import LOG_LEVEL
+from catalyst.exchange.exchange_errors import \
+    MismatchingBaseCurrenciesExchanges
+
+log = Logger('LiveGraphClock', level=LOG_LEVEL)
+
+
+class LiveGraphClock(object):
+    """Realtime clock for live trading.
+
+    This class is a drop-in replacement for
+    :class:`zipline.gens.sim_engine.MinuteSimulationClock`.
+
+    This mixes the clock with a live graph.
+
+    Notes
+    -----
+    This seemingly awkward approach allows us to run the program using a single
+    thread. This is important because Matplotlib does not play nice with
+    multi-threaded environments. Zipline probably does not either.
+
+
+    Matplotlib has a pause() method which is a wrapper around time.sleep()
+    used in the SimpleClock. The key difference is that users
+    can still interact with the chart during the pause cycles. This is
+    what enables us to keep a single thread. This is also why we are not using
+    the 'animate' callback of Matplotlib. We need to direct access to the
+    __iter__ method in order to yield events to Zipline.
+
+    The :param:`time_skew` parameter represents the time difference between
+    the exchange and the live trading machine's clock. It's not used currently.
+    """
+
+    def __init__(self, sessions, context, time_skew=pd.Timedelta('0s')):
+
+        global mdates, plt  # TODO: Could be cleaner
+        import matplotlib.dates as mdates
+        from matplotlib import pyplot as plt
+        from matplotlib import style
+
+        self.sessions = sessions
+        self.time_skew = time_skew
+        self._last_emit = None
+        self._before_trading_start_bar_yielded = True
+        self.context = context
+        self.fmt = mdates.DateFormatter('%Y-%m-%d %H:%M')
+
+        style.use('dark_background')
+
+        fig = plt.figure()
+        fig.canvas.set_window_title('Enigma Catalyst: {}'.format(
+            self.context.algo_namespace))
+
+        self.ax_pnl = fig.add_subplot(311)
+
+        self.ax_custom_signals = fig.add_subplot(312, sharex=self.ax_pnl)
+
+        self.ax_exposure = fig.add_subplot(313, sharex=self.ax_pnl)
+
+        if len(context.minute_stats) > 0:
+            self.draw_pnl()
+            self.draw_custom_signals()
+            self.draw_exposure()
+
+        # rotates and right aligns the x labels, and moves the bottom of the
+        # axes up to make room for them
+        fig.autofmt_xdate()
+        fig.subplots_adjust(hspace=0.5)
+
+        plt.tight_layout()
+        plt.ion()
+        plt.show()
+
+    def format_ax(self, ax):
+        """
+        Trying to assign reasonable parameters to the time axis.
+
+        Parameters
+        ----------
+        ax:
+
+        """
+        # TODO: room for improvement
+        ax.xaxis.set_major_locator(mdates.DayLocator(interval=1))
+        ax.xaxis.set_major_formatter(self.fmt)
+
+        locator = mdates.HourLocator(interval=4)
+        locator.MAXTICKS = 5000
+        ax.xaxis.set_minor_locator(locator)
+
+        datemin = pd.Timestamp.utcnow()
+        ax.set_xlim(datemin)
+
+        ax.grid(True)
+
+    def set_legend(self, ax):
+        """
+        Set legend on the chart.
+
+        Parameters
+        ----------
+        ax
+
+        """
+        ax.legend(loc='upper left', ncol=1, fontsize=10, numpoints=1)
+
+    def draw_pnl(self):
+        """
+        Draw p&l line on the chart.
+
+        """
+        ax = self.ax_pnl
+        df = self.context.pnl_stats
+
+        ax.clear()
+        ax.set_title('Performance')
+        ax.plot(df.index, df['performance'], '-',
+                color='green',
+                linewidth=1.0,
+                label='Performance'
+                )
+
+        def perc(val):
+            return '{:2f}'.format(val)
+
+        ax.format_ydata = perc
+
+        self.set_legend(ax)
+        self.format_ax(ax)
+
+    def draw_custom_signals(self):
+        """
+        Draw custom signals on the chart.
+
+        """
+        ax = self.ax_custom_signals
+        df = self.context.custom_signals_stats
+
+        colors = ['blue', 'green', 'red', 'black', 'orange', 'yellow', 'pink']
+
+        ax.clear()
+        ax.set_title('Custom Signals')
+        for index, column in enumerate(df.columns.values.tolist()):
+            ax.plot(df.index, df[column], '-',
+                    color=colors[index],
+                    linewidth=1.0,
+                    label=column
+                    )
+
+        self.set_legend(ax)
+        self.format_ax(ax)
+
+    def draw_exposure(self):
+        """
+        Draw exposure line on the chart.
+
+        """
+        ax = self.ax_exposure
+        context = self.context
+        df = context.exposure_stats
+
+        # TODO: list exchanges in graph
+        base_currency = None
+        positions = []
+        for exchange_name in context.exchanges:
+            exchange = context.exchanges[exchange_name]
+
+            if not base_currency:
+                base_currency = exchange.base_currency
+            elif base_currency != exchange.base_currency:
+                raise MismatchingBaseCurrenciesExchanges(
+                    base_currency=base_currency,
+                    exchange_name=exchange.name,
+                    exchange_currency=exchange.base_currency
+                )
+
+            positions += exchange.portfolio.positions
+
+        ax.clear()
+        ax.set_title('Exposure')
+        ax.plot(df.index, df['base_currency'], '-',
+                color='green',
+                linewidth=1.0,
+                label='Base Currency: {}'.format(base_currency.upper())
+                )
+
+        symbols = []
+        for position in positions:
+            symbols.append(position.symbol)
+
+        ax.plot(df.index, df['long_exposure'], '-',
+                color='blue',
+                linewidth=1.0,
+                label='Long Exposure: {}'.format(', '.join(symbols).upper()))
+
+        self.set_legend(ax)
+        self.format_ax(ax)
+
+    def __iter__(self):
+        yield pd.Timestamp.utcnow(), SESSION_START
+
+        while True:
+            current_time = pd.Timestamp.utcnow()
+            current_minute = current_time.floor('1 min')
+
+            if self._last_emit is None or current_minute > self._last_emit:
+                log.debug('emitting minutely bar: {}'.format(current_minute))
+
+                self._last_emit = current_minute
+                yield current_minute, BAR
+
+                try:
+                    self.draw_pnl()
+                    self.draw_custom_signals()
+                    self.draw_exposure()
+
+                    plt.draw()
+                except Exception as e:
+                    log.warn('Unable to update the graph: {}'.format(e))
+
+            else:
+                # I can't use the "animate" reactive approach here because
+                # I need to yield from the main loop.
+
+                # Workaround: https://stackoverflow.com/a/33050617/814633
+                plt.pause(1)

catalyst/exchange/poloniex/poloniex.py

@@ -0,0 +1,661 @@
+import json
+import time
+from collections import defaultdict
+
+import numpy as np
+import pandas as pd
+import pytz
+from catalyst.assets._assets import TradingPair
+from logbook import Logger
+# import six
+from six import iteritems
+
+from catalyst.constants import LOG_LEVEL
+# from websocket import create_connection
+from catalyst.exchange.exchange import Exchange
+from catalyst.exchange.exchange_bundle import ExchangeBundle
+from catalyst.exchange.exchange_errors import (
+    ExchangeRequestError,
+    InvalidHistoryFrequencyError,
+    InvalidOrderStyle,
+    OrphanOrderError,
+    OrphanOrderReverseError)
+from catalyst.exchange.exchange_execution import ExchangeLimitOrder, \
+    ExchangeStopLimitOrder
+from catalyst.exchange.exchange_utils import get_exchange_symbols_filename, \
+    download_exchange_symbols, get_symbols_string
+from catalyst.exchange.poloniex.poloniex_api import Poloniex_api
+from catalyst.finance.order import Order, ORDER_STATUS
+from catalyst.finance.transaction import Transaction
+from catalyst.protocol import Account
+from catalyst.utils.deprecate import deprecated
+
+log = Logger('Poloniex', level=LOG_LEVEL)
+
+
+@deprecated
+class Poloniex(Exchange):
+    def __init__(self, key, secret, base_currency, portfolio=None):
+        self.api = Poloniex_api(key=key, secret=secret)
+        self.name = 'poloniex'
+
+        self.assets = dict()
+        self.load_assets()
+
+        self.local_assets = dict()
+        self.load_assets(is_local=True)
+
+        self.base_currency = base_currency
+        self._portfolio = portfolio
+        self.minute_writer = None
+        self.minute_reader = None
+        self.transactions = defaultdict(list)
+
+        self.num_candles_limit = 2000
+        self.max_requests_per_minute = 60
+        self.request_cpt = dict()
+
+        self.bundle = ExchangeBundle(self.name)
+
+    def sanitize_curency_symbol(self, exchange_symbol):
+        """
+        Helper method used to build the universal pair.
+        Include any symbol mapping here if appropriate.
+
+        :param exchange_symbol:
+        :return universal_symbol:
+        """
+        return exchange_symbol.lower()
+
+    def _create_order(self, order_status):
+        """
+        Create a Catalyst order object from the Exchange order dictionary
+        :param order_status:
+        :return: Order
+        """
+        # if order_status['is_cancelled']:
+        #    status = ORDER_STATUS.CANCELLED
+        # elif not order_status['is_live']:
+        #    log.info('found executed order {}'.format(order_status))
+        #    status = ORDER_STATUS.FILLED
+        # else:
+        status = ORDER_STATUS.OPEN
+
+        amount = float(order_status['amount'])
+        # filled = float(order_status['executed_amount'])
+        filled = None
+
+        if order_status['type'] == 'sell':
+            amount = -amount
+            # filled = -filled
+
+        price = float(order_status['rate'])
+
+        stop_price = None
+        limit_price = None
+
+        # TODO: is this comprehensive enough?
+        # if order_type.endswith('limit'):
+        #    limit_price = price
+        # elif order_type.endswith('stop'):
+        #    stop_price = price
+
+        # executed_price = float(order_status['avg_execution_price'])
+        executed_price = price
+
+        # TODO: Set Poloniex comission
+        commission = None
+
+        # date=pd.Timestamp.utcfromtimestamp(float(order_status['timestamp']))
+        # date=pytz.utc.localize(date)
+        date = None
+
+        order = Order(
+            dt=date,
+            asset=self.assets[order_status['symbol']],
+            # No such field in Poloniex
+            amount=amount,
+            stop=stop_price,
+            limit=limit_price,
+            filled=filled,
+            id=str(order_status['orderNumber']),
+            commission=commission
+        )
+        order.status = status
+
+        return order, executed_price
+
+    def get_balances(self):
+        balances = self.api.returnbalances()
+        try:
+            log.debug('retrieving wallets balances')
+        except Exception as e:
+            log.debug(e)
+            raise ExchangeRequestError(error=e)
+
+        if 'error' in balances:
+            raise ExchangeRequestError(
+                error='unable to fetch balance {}'.format(balances['error'])
+            )
+
+        std_balances = dict()
+        for (key, value) in iteritems(balances):
+            currency = key.lower()
+            std_balances[currency] = float(value)
+
+        return std_balances
+
+    @property
+    def account(self):
+        account = Account()
+
+        account.settled_cash = None
+        account.accrued_interest = None
+        account.buying_power = None
+        account.equity_with_loan = None
+        account.total_positions_value = None
+        account.total_positions_exposure = None
+        account.regt_equity = None
+        account.regt_margin = None
+        account.initial_margin_requirement = None
+        account.maintenance_margin_requirement = None
+        account.available_funds = None
+        account.excess_liquidity = None
+        account.cushion = None
+        account.day_trades_remaining = None
+        account.leverage = None
+        account.net_leverage = None
+        account.net_liquidation = None
+
+        return account
+
+    @property
+    def time_skew(self):
+        # TODO: research the time skew conditions
+        return pd.Timedelta('0s')
+
+    def get_account(self):
+        # TODO: fetch account data and keep in cache
+        return None
+
+    def get_candles(self, freq, assets, bar_count=None,
+                    start_dt=None, end_dt=None):
+        """
+        Retrieve OHLVC candles from Poloniex
+
+        :param freq:
+        :param assets:
+        :param bar_count:
+        :return:
+
+        Available Frequencies
+        ---------------------
+        '5m', '15m', '30m', '2h', '4h', '1D'
+        """
+
+        if end_dt is None:
+            end_dt = pd.Timestamp.utcnow()
+
+        log.debug(
+            'retrieving {bars} {freq} candles on {exchange} from '
+            '{end_dt} for markets {symbols}, '.format(
+                bars=bar_count,
+                freq=freq,
+                exchange=self.name,
+                end_dt=end_dt,
+                symbols=get_symbols_string(assets)
+            )
+        )
+
+        if freq == '1T' and (bar_count == 1 or bar_count is None):
+            # TODO: use the order book instead
+            # We use the 5m to fetch the last bar
+            frequency = 300
+        elif freq == '5T':
+            frequency = 300
+        elif freq == '15T':
+            frequency = 900
+        elif freq == '30T':
+            frequency = 1800
+        elif freq == '120T':
+            frequency = 7200
+        elif freq == '240T':
+            frequency = 14400
+        elif freq == '1D':
+            frequency = 86400
+        else:
+            # Poloniex does not offer 1m data candles
+            # It is likely to error out there frequently
+            raise InvalidHistoryFrequencyError(frequency=freq)
+
+        # Making sure that assets are iterable
+        asset_list = [assets] if isinstance(assets, TradingPair) else assets
+        ohlc_map = dict()
+
+        for asset in asset_list:
+            delta = end_dt - pd.to_datetime('1970-1-1', utc=True)
+            end = int(delta.total_seconds())
+
+            if bar_count is None:
+                start = end - 2 * frequency
+            else:
+                start = end - bar_count * frequency
+
+            try:
+                response = self.api.returnchartdata(
+                    self.get_symbol(asset), frequency, start, end
+                )
+            except Exception as e:
+                raise ExchangeRequestError(error=e)
+
+            if 'error' in response:
+                raise ExchangeRequestError(
+                    error='Unable to retrieve candles: {}'.format(
+                        response.content)
+                )
+
+            def ohlc_from_candle(candle):
+                last_traded = pd.Timestamp.utcfromtimestamp(candle['date'])
+                last_traded = last_traded.replace(tzinfo=pytz.UTC)
+
+                ohlc = dict(
+                    open=np.float64(candle['open']),
+                    high=np.float64(candle['high']),
+                    low=np.float64(candle['low']),
+                    close=np.float64(candle['close']),
+                    volume=np.float64(candle['volume']),
+                    price=np.float64(candle['close']),
+                    last_traded=last_traded
+                )
+
+                return ohlc
+
+            if bar_count is None:
+                ohlc_map[asset] = ohlc_from_candle(response[0])
+            else:
+                ohlc_bars = []
+                for candle in response:
+                    ohlc = ohlc_from_candle(candle)
+                    ohlc_bars.append(ohlc)
+                ohlc_map[asset] = ohlc_bars
+
+        return ohlc_map[assets] \
+            if isinstance(assets, TradingPair) else ohlc_map
+
+    def create_order(self, asset, amount, is_buy, style):
+        """
+        Creating order on the exchange.
+
+        :param asset:
+        :param amount:
+        :param is_buy:
+        :param style:
+        :return:
+        """
+        exchange_symbol = self.get_symbol(asset)
+
+        if (isinstance(style, ExchangeLimitOrder)
+                or isinstance(style, ExchangeStopLimitOrder)):
+            if isinstance(style, ExchangeStopLimitOrder):
+                log.warn('{} will ignore the stop price'.format(self.name))
+
+            price = style.get_limit_price(is_buy)
+
+            try:
+                if (is_buy):
+                    response = self.api.buy(exchange_symbol, amount, price)
+                else:
+                    response = self.api.sell(exchange_symbol, -amount, price)
+            except Exception as e:
+                raise ExchangeRequestError(error=e)
+
+            date = pd.Timestamp.utcnow()
+
+            if ('orderNumber' in response):
+                order_id = str(response['orderNumber'])
+                order = Order(
+                    dt=date,
+                    asset=asset,
+                    amount=amount,
+                    stop=style.get_stop_price(is_buy),
+                    limit=style.get_limit_price(is_buy),
+                    id=order_id
+                )
+                return order
+            else:
+                log.warn(
+                    '{} order failed: {}'.format('buy' if is_buy else 'sell',
+                                                 response['error']))
+                return None
+        else:
+            raise InvalidOrderStyle(exchange=self.name,
+                                    style=style.__class__.__name__)
+
+    def get_open_orders(self, asset='all'):
+        """Retrieve all of the current open orders.
+
+        Parameters
+        ----------
+        asset : Asset
+            If passed and not 'all', return only the open orders for the given
+            asset instead of all open orders.
+
+        Returns
+        -------
+        open_orders : dict[list[Order]] or list[Order]
+            If 'all' is passed this will return a dict mapping Assets
+            to a list containing all the open orders for the asset.
+            If an asset is passed then this will return a list of the open
+            orders for this asset.
+        """
+
+        return self.portfolio.open_orders
+
+        """
+        TODO: Why going to the exchange if we already have this info locally?
+              And why creating all these Orders if we later discard them?
+        """
+
+        try:
+            if (asset == 'all'):
+                response = self.api.returnopenorders('all')
+            else:
+                response = self.api.returnopenorders(self.get_symbol(asset))
+        except Exception as e:
+            raise ExchangeRequestError(error=e)
+
+        if 'error' in response:
+            raise ExchangeRequestError(
+                error='Unable to retrieve open orders: {}'.format(
+                    response['message'])
+            )
+
+        print(self.portfolio.open_orders)
+
+        # TODO: Need to handle openOrders for 'all'
+        orders = list()
+        for order_status in response:
+            # will Throw error b/c Polo doesn't track order['symbol']
+            order, executed_price = self._create_order(order_status)
+            if asset is None or asset == order.sid:
+                orders.append(order)
+
+        return orders
+
+    def get_order(self, order_id):
+        """Lookup an order based on the order id returned from one of the
+        order functions.
+
+        Parameters
+        ----------
+        order_id : str
+            The unique identifier for the order.
+
+        Returns
+        -------
+        order : Order
+            The order object.
+        """
+
+        try:
+            order = self._portfolio.open_orders[order_id]
+        except Exception as e:
+            raise OrphanOrderError(order_id=order_id, exchange=self.name)
+
+        return order
+
+        # TODO: Need to decide whether we fetch orders locally or from exchnage
+        # The code below is ignored
+
+        try:
+            response = self.api.returnopenorders(self.get_symbol(order.sid))
+        except Exception as e:
+            raise ExchangeRequestError(error=e)
+
+        for o in response:
+            if (int(o['orderNumber']) == int(order_id)):
+                return order
+
+        return None
+
+    def cancel_order(self, order_param):
+        """Cancel an open order.
+
+        Parameters
+        ----------
+        order_param : str or Order
+            The order_id or order object to cancel.
+        """
+
+        if (isinstance(order_param, Order)):
+            order = order_param
+        else:
+            order = self._portfolio.open_orders[order_param]
+
+        try:
+            response = self.api.cancelorder(order.id)
+        except Exception as e:
+            raise ExchangeRequestError(error=e)
+
+        if 'error' in response:
+            log.info(
+                'Unable to cancel order {order_id} on exchange {exchange} '
+                '{error}.'.format(
+                    order_id=order.id,
+                    exchange=self.name,
+                    error=response['error']
+                ))
+
+            # raise OrderCancelError(
+            #    order_id=order.id,
+            #    exchange=self.name,
+            #    error=response['error']
+            # )
+
+        self.portfolio.remove_order(order)
+
+    def tickers(self, assets):
+        """
+        Fetch ticket data for assets
+        https://docs.bitfinex.com/v2/reference#rest-public-tickers
+
+        :param assets:
+        :return:
+        """
+        symbols = self.get_symbols(assets)
+
+        log.debug('fetching tickers {}'.format(symbols))
+
+        try:
+            response = self.api.returnticker()
+        except Exception as e:
+            raise ExchangeRequestError(error=e)
+
+        if 'error' in response:
+            raise ExchangeRequestError(
+                error='Unable to retrieve tickers: {}'.format(
+                    response['error'])
+            )
+
+        ticks = dict()
+
+        for index, symbol in enumerate(symbols):
+            ticks[assets[index]] = dict(
+                timestamp=pd.Timestamp.utcnow(),
+                bid=float(response[symbol]['highestBid']),
+                ask=float(response[symbol]['lowestAsk']),
+                last_price=float(response[symbol]['last']),
+                low=float(response[symbol]['lowestAsk']),
+                # TODO: Polo does not provide low
+                high=float(response[symbol]['highestBid']),
+                # TODO: Polo does not provide high
+                volume=float(response[symbol]['baseVolume']),
+            )
+
+        log.debug('got tickers {}'.format(ticks))
+        return ticks
+
+    def generate_symbols_json(self, filename=None, source_dates=False):
+        symbol_map = {}
+
+        if not source_dates:
+            fn, r = download_exchange_symbols(self.name)
+            with open(fn) as data_file:
+                cached_symbols = json.load(data_file)
+
+        response = self.api.returnticker()
+
+        for exchange_symbol in response:
+            base, market = self.sanitize_curency_symbol(exchange_symbol).split(
+                '_')
+            symbol = '{market}_{base}'.format(market=market, base=base)
+
+            if (source_dates):
+                start_date = self.get_symbol_start_date(exchange_symbol)
+            else:
+                try:
+                    start_date = cached_symbols[exchange_symbol]['start_date']
+                except KeyError:
+                    start_date = time.strftime('%Y-%m-%d')
+
+            try:
+                end_daily = cached_symbols[exchange_symbol]['end_daily']
+            except KeyError:
+                end_daily = 'N/A'
+
+            try:
+                end_minute = cached_symbols[exchange_symbol]['end_minute']
+            except KeyError:
+                end_minute = 'N/A'
+
+            symbol_map[exchange_symbol] = dict(
+                symbol=symbol,
+                start_date=start_date,
+                end_daily=end_daily,
+                end_minute=end_minute,
+            )
+
+        if (filename is None):
+            filename = get_exchange_symbols_filename(self.name)
+
+        with open(filename, 'w') as f:
+            json.dump(symbol_map, f, sort_keys=True, indent=2,
+                      separators=(',', ':'))
+
+    def get_symbol_start_date(self, symbol):
+        try:
+            r = self.api.returnchartdata(symbol, 86400, pd.to_datetime(
+                '2010-1-1').value // 10 ** 9)
+        except Exception as e:
+            raise ExchangeRequestError(error=e)
+
+        return time.strftime('%Y-%m-%d', time.gmtime(int(r[0]['date'])))
+
+    def check_open_orders(self):
+        """
+        Need to override this function for Poloniex:
+
+        Loop through the list of open orders in the Portfolio object.
+        Check if any transactions have been executed:
+            If so, create a transaction and apply to the Portfolio.
+        Check if the order is still open:
+            If not, remove it from open orders
+
+        :return:
+        transactions: Transaction[]
+        """
+        transactions = list()
+        if self.portfolio.open_orders:
+            for order_id in list(self.portfolio.open_orders):
+
+                order = self._portfolio.open_orders[order_id]
+                log.debug('found open order: {}'.format(order_id))
+
+                try:
+                    order_open = self.get_order(order_id)
+                except Exception as e:
+                    raise ExchangeRequestError(error=e)
+
+                if (order_open):
+                    delta = pd.Timestamp.utcnow() - order.dt
+                    log.info(
+                        'order {order_id} still open after {delta}'.format(
+                            order_id=order_id,
+                            delta=delta)
+                    )
+
+                try:
+                    response = self.api.returnordertrades(order_id)
+                except Exception as e:
+                    raise ExchangeRequestError(error=e)
+
+                if ('error' in response):
+                    if (not order_open):
+                        raise OrphanOrderReverseError(order_id=order_id,
+                                                      exchange=self.name)
+                else:
+                    for tx in response:
+                        """
+                        We maintain a list of dictionaries of transactions that
+                        correspond to partially filled orders, indexed by
+                        order_id. Every time we query executed transactions
+                        from the exchange, we check if we had that transaction
+                        for that order already. If not, we process it.
+
+                        When an order if fully filled, we flush the dict of
+                        transactions associated with that order.
+                        """
+                        if (not filter(
+                                lambda item: item['order_id'] == tx['tradeID'],
+                                self.transactions[order_id])):
+                            log.debug(
+                                'Got new transaction for order {}: amount {}, '
+                                'price {}'.format(
+                                    order_id, tx['amount'], tx['rate']))
+                            tx['amount'] = float(tx['amount'])
+                            if (tx['type'] == 'sell'):
+                                tx['amount'] = -tx['amount']
+                            transaction = Transaction(
+                                asset=order.asset,
+                                amount=tx['amount'],
+                                dt=pd.to_datetime(tx['date'], utc=True),
+                                price=float(tx['rate']),
+                                order_id=tx['tradeID'],
+                                # it's a misnomer, but keep for compatibility
+                                commission=float(tx['fee'])
+                            )
+                            self.transactions[order_id].append(transaction)
+                            self.portfolio.execute_transaction(transaction)
+                            transactions.append(transaction)
+
+                    if (not order_open):
+                        """
+                            Since transactions have been executed individually
+                            the only thing left to do is remove them from list
+                            of open_orders
+                        """
+                        del self.portfolio.open_orders[order_id]
+                        del self.transactions[order_id]
+
+        return transactions
+
+    def get_orderbook(self, asset, order_type='all'):
+        exchange_symbol = asset.exchange_symbol
+        data = self.api.returnOrderBook(market=exchange_symbol)
+
+        result = dict()
+        for order_type in data:
+            # TODO: filter by type
+            if order_type != 'asks' and order_type != 'bids':
+                continue
+
+            result[order_type] = []
+            for entry in data[order_type]:
+                if len(entry) == 2:
+                    result[order_type].append(
+                        dict(
+                            rate=float(entry[0]),
+                            quantity=float(entry[1])
+                        )
+                    )
+        return result

catalyst/exchange/poloniex/poloniex_api.py

@@ -0,0 +1,212 @@
+#!/usr/bin/env python
+import json
+import time
+import hmac
+import hashlib
+import ssl
+
+from six.moves import urllib
+
+# Workaround for backwards compatibility
+# https://stackoverflow.com/questions/3745771/urllib-request-in-python-2-7
+urlopen = urllib.request.urlopen
+
+
+class Poloniex_api(object):
+    def __init__(self, key, secret):
+        self.key = key
+        self.secret = secret
+
+        self.max_requests_per_second = 6
+        self.request_cpt = dict()
+
+        self.public = ['returnTicker', 'return24Volume', 'returnOrderBook',
+                       'returnTradeHistory', 'returnChartData',
+                       'returnCurrencies', 'returnLoanOrders']
+        self.trading = ['returnBalances', 'returnCompleteBalances',
+                        'returnDepositAddresses',
+                        'generateNewAddress', 'returnDepositsWithdrawals',
+                        'returnOpenOrders',
+                        'returnTradeHistory', 'returnOrderTrades',
+                        'buy', 'sell', 'cancelOrder', 'moveOrder',
+                        'withdraw', 'returnFeeInfo',
+                        'returnAvailableAccountBalances',
+                        'returnTradableBalances', 'transferBalance',
+                        'returnMarginAccountSummary', 'marginBuy',
+                        'marginSell',
+                        'getMarginPosition', 'closeMarginPosition',
+                        'createLoanOffer',
+                        'cancelLoanOffer', 'returnOpenLoanOffers',
+                        'returnActiveLoans',
+                        'returnLendingHistory', 'toggleAutoRenew']
+
+    def ask_request(self):
+        """
+        Asks permission to issue a request to the exchange.
+        The primary purpose is to avoid hitting rate limits.
+
+        The application will pause if the maximum requests per minute
+        permitted by the exchange is exceeded.
+
+        :return boolean:
+
+        """
+        now = time.time()
+        if not self.request_cpt:
+            self.request_cpt = dict()
+            self.request_cpt[now] = 0
+            return True
+
+        cpt_date = list(self.request_cpt.keys())[0]
+        cpt = self.request_cpt[cpt_date]
+
+        if now > cpt_date + 1:
+            self.request_cpt = dict()
+            self.request_cpt[now] = 0
+            return True
+
+        if cpt >= self.max_requests_per_second:
+
+            time.sleep(1)
+
+            now = time.time()
+            self.request_cpt = dict()
+            self.request_cpt[now] = 0
+            return True
+        else:
+            self.request_cpt[cpt_date] += 1
+
+    def query(self, method, req={}):
+
+        if method in self.public:
+            url = 'https://poloniex.com/public?command=' + method + '&' + \
+                  urllib.parse.urlencode(req)
+            headers = {}
+            post_data = None
+        elif method in self.trading:
+            url = 'https://poloniex.com/tradingApi'
+            req['command'] = method
+            req['nonce'] = int(time.time() * 1000)
+            post_data = urllib.parse.urlencode(req)
+
+            signature = hmac.new(self.secret.encode('utf-8'),
+                                 post_data.encode('utf-8'),
+                                 hashlib.sha512).hexdigest()
+            headers = {'Sign': signature, 'Key': self.key}
+
+            post_data = post_data.encode('utf-8')
+        else:
+            raise ValueError(
+                'Method "' + method + '" not found in neither the Public API '
+                                      'or Trading API endpoints'
+            )
+
+        self.ask_request()
+        req = urllib.request.Request(
+            url,
+            data=post_data,
+            headers=headers,
+        )
+        resource = urlopen(req, context=ssl._create_unverified_context())
+        content = resource.read().decode('utf-8')
+        return json.loads(content)
+
+    def returnticker(self):
+        return self.query('returnTicker', {})
+
+    def return24volume(self):
+        return self.query('return24Volume', {})
+
+    def returnOrderBook(self, market='all'):
+        return self.query('returnOrderBook', {'currencyPair': market})
+
+    def returntradehistory(self, market, start=None, end=None):
+        if (start is not None and end is not None):
+            return self.query('returntradehistory',
+                              {'currencyPair': market, 'start': start,
+                               'end': end})
+        else:
+            return self.query('returntradehistory', {'currencyPair': market})
+
+    def returnchartdata(self, market, period, start, end=9999999999):
+        return self.query('returnChartData',
+                          {'currencyPair': market, 'period': period,
+                           'start': start, 'end': end})
+
+    def returncurrencies(self):
+        return self.query('returnCurrencies', {})
+
+    def returnloadorders(self, market):
+        return self.query('returnLoanOrders', {'currency': market})
+
+    def returnbalances(self):
+        return self.query('returnBalances')
+
+    def returncompletebalances(self, account):
+        if (account):
+            return self.query('returnCompleteBalances', {'account': account})
+        else:
+            return self.query('returnCompleteBalances')
+
+    def returndepositaddresses(self):
+        return self.query('returnDepositAddresses')
+
+    def generatenewaddress(self, currency):
+        return self.query('generateNewAddress', {'currency': currency})
+
+    def returnDepositsWithdrawals(self, start, end):
+        return self.query('returnDepositsWithdrawals',
+                          {'start': start, 'end': end})
+
+    def returnopenorders(self, market):
+        return self.query('returnOpenOrders', {'currencyPair': market})
+
+    def returnordertrades(self, ordernumber):
+        return self.query('returnOrderTrades', {'orderNumber': ordernumber})
+
+    def buy(self, market, amount, rate, fillorkill=0, immediateorcancel=0,
+            postonly=0):
+        if (fillorkill):
+            return self.query('buy', {'currencyPair': market, 'rate': rate,
+                                      'amount': amount,
+                                      'fillOrKill': fillorkill, })
+        elif (immediateorcancel):
+            return self.query('buy', {'currencyPair': market, 'rate': rate,
+                                      'amount': amount,
+                                      'immediateOrCancel': immediateorcancel})
+        elif (postonly):
+            return self.query('buy', {'currencyPair': market, 'rate': rate,
+                                      'amount': amount,
+                                      'postOnly': postonly, })
+        else:
+            return self.query('buy', {'currencyPair': market, 'rate': rate,
+                                      'amount': amount, })
+
+    def sell(self, market, amount, rate, fillorkill=0, immediateorcancel=0,
+             postonly=0):
+        if (fillorkill):
+            return self.query('sell', {'currencyPair': market, 'rate': rate,
+                                       'amount': amount,
+                                       'fillOrKill': fillorkill, })
+        elif (immediateorcancel):
+            return self.query('sell', {'currencyPair': market, 'rate': rate,
+                                       'amount': amount,
+                                       'immediateOrCancel': immediateorcancel})
+        elif (postonly):
+            return self.query('sell', {'currencyPair': market, 'rate': rate,
+                                       'amount': amount,
+                                       'postOnly': postonly, })
+        else:
+            return self.query('sell', {'currencyPair': market, 'rate': rate,
+                                       'amount': amount, })
+
+    def cancelorder(self, ordernumber):
+        return self.query('cancelOrder', {'orderNumber': ordernumber})
+
+    def withdraw(self, currency, quantity, address):
+        return self.query('withdraw',
+                          {'currency': currency, 'amount': quantity,
+                           'address': address})
+
+    def returnfeeinfo(self):
+        return self.query('returnFeeInfo')

catalyst/exchange/simple_clock.py

@@ -0,0 +1,61 @@
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from time import sleep
+
+import pandas as pd
+from catalyst.gens.sim_engine import (
+    BAR,
+    SESSION_START
+)
+from logbook import Logger
+
+from catalyst.constants import LOG_LEVEL
+
+log = Logger('ExchangeClock', level=LOG_LEVEL)
+
+
+class SimpleClock(object):
+    """Realtime clock for live trading.
+
+    This class is a drop-in replacement for
+    :class:`zipline.gens.sim_engine.MinuteSimulationClock`.
+
+    This is a stripped down version because crypto exchanges run
+    around the clock.
+
+    The :param:`time_skew` parameter represents the time difference between
+    the Broker and the live trading machine's clock.
+    """
+
+    def __init__(self, sessions, time_skew=pd.Timedelta("0s")):
+
+        self.sessions = sessions
+        self.time_skew = time_skew
+        self._last_emit = None
+        self._before_trading_start_bar_yielded = True
+
+    def __iter__(self):
+        yield pd.Timestamp.utcnow(), SESSION_START
+
+        while True:
+            current_time = pd.Timestamp.utcnow()
+            current_minute = current_time.floor('1 min')
+
+            if self._last_emit is None or current_minute > self._last_emit:
+                log.debug('emitting minutely bar: {}'.format(current_minute))
+
+                self._last_emit = current_minute
+                yield current_minute, BAR
+            else:
+                sleep(1)

catalyst/exchange/stats_utils.py

@@ -0,0 +1,426 @@
+import csv
+import numbers
+
+import copy
+import numpy as np
+import os
+import pandas as pd
+import boto3
+import time
+
+from catalyst.assets._assets import TradingPair
+
+from catalyst.exchange.exchange_utils import get_algo_folder
+
+s3 = boto3.resource('s3')
+
+
+def trend_direction(series):
+    if series[-1] is np.nan or series[-1] is np.nan:
+        return None
+
+    if series[-1] > series[-2]:
+        return 'up'
+    else:
+        return 'down'
+
+
+def crossover(source, target):
+    """
+    The `x`-series is defined as having crossed over `y`-series if the value
+    of `x` is greater than the value of `y` and the value of `x` was less than
+    the value of `y` on the bar immediately preceding the current bar.
+
+    Parameters
+    ----------
+    source: Series
+    target: Series
+
+    Returns
+    -------
+    bool
+
+    """
+    if isinstance(target, numbers.Number):
+        if source[-1] is np.nan or source[-2] is np.nan \
+                or target is np.nan:
+            return False
+
+        if source[-1] >= target > source[-2]:
+            return True
+        else:
+            return False
+
+    else:
+        if source[-1] is np.nan or source[-2] is np.nan \
+                or target[-1] is np.nan or target[-2] is np.nan:
+            return False
+
+        if source[-1] > target[-1] and source[-2] < target[-2]:
+            return True
+        else:
+            return False
+
+
+def crossunder(source, target):
+    """
+    The `x`-series is defined as having crossed under `y`-series if the value
+    of `x` is less than the value of `y` and the value of `x` was greater than
+    the value of `y` on the bar immediately preceding the current bar.
+
+    Parameters
+    ----------
+    source: Series
+    target: Series
+
+    Returns
+    -------
+    bool
+
+    """
+    if isinstance(target, numbers.Number):
+        if source[-1] is np.nan or source[-2] is np.nan \
+                or target is np.nan:
+            return False
+
+        if source[-1] < target <= source[-2]:
+            return True
+        else:
+            return False
+    else:
+        if source[-1] is np.nan or source[-2] is np.nan \
+                or target[-1] is np.nan or target[-2] is np.nan:
+            return False
+
+        if source[-1] < target[-1] and source[-2] >= target[-2]:
+            return True
+        else:
+            return False
+
+
+def vwap(df):
+    """
+    Volume-weighted average price (VWAP) is a ratio generally used by
+    institutional investors and mutual funds to make buys and sells so as not
+    to disturb the market prices with large orders. It is the average share
+    price of a stock weighted against its trading volume within a particular
+    time frame, generally one day.
+
+    Read more: Volume Weighted Average Price - VWAP
+    https://www.investopedia.com/terms/v/vwap.asp#ixzz4xt922daE
+
+    Parameters
+    ----------
+    df: pd.DataFrame
+
+    Returns
+    -------
+
+    """
+    if 'close' not in df.columns or 'volume' not in df.columns:
+        raise ValueError('price data must include `volume` and `close`')
+
+    vol_sum = np.nansum(df['volume'].values)
+
+    try:
+        ret = np.nansum(df['close'].values * df['volume'].values) / vol_sum
+    except ZeroDivisionError:
+        ret = np.nan
+
+    return ret
+
+
+def set_position_row(row, asset, asset_values=list()):
+    """
+    Apply the position data as individual columns.
+
+    Parameters
+    ----------
+    row: dict[str, Object]
+    asset: TradingPair
+    asset_values: list[str]
+        If a recorded_col contains a tuple which first value is an asset
+        matching a position, its value will be displayed with the
+        position and not in the index.
+
+    Returns
+    -------
+
+    """
+    asset_cols = ['symbol']
+    row['symbol'] = asset.symbol
+
+    position = next((p for p in row['positions'] if p['sid'] == asset), None)
+
+    columns = ['amount', 'cost_basis', 'last_sale_price']
+    for column in columns:
+        if position is not None:
+            row[column] = position[column]
+
+        else:
+            row[column] = 0
+
+        asset_cols.append(column)
+
+    values = asset_values[asset] if asset in asset_values else list()
+    for column in values:
+        row[column] = values[column]
+
+        asset_cols.append(column)
+
+    return asset_cols
+
+
+def prepare_stats(stats, recorded_cols=list()):
+    """
+    Prepare the stats DataFrame for user-friendly output.
+
+    Parameters
+    ----------
+    stats: list[Object]
+    recorded_cols: list[str]
+
+    Returns
+    -------
+
+    """
+    asset_cols = list()
+
+    stats = copy.deepcopy(stats)
+    # Using a copy since we are adding rows inside the loop.
+    for row_index, row_data in enumerate(list(stats)):
+        assets = [p['sid'] for p in row_data['positions']]
+
+        asset_values = dict()
+        if recorded_cols is not None:
+            for column in recorded_cols[:]:
+                value = row_data[column]
+                if type(value) is dict:
+                    for asset in value:
+                        if not isinstance(asset, TradingPair):
+                            break
+
+                        if asset not in assets:
+                            assets.append(asset)
+
+                        if asset not in asset_values:
+                            asset_values[asset] = dict()
+
+                        asset_values[asset][column] = value[asset]
+
+        if len(assets) == 1:
+            row = stats[row_index]
+            asset_cols = set_position_row(row, assets[0], asset_values)
+
+        elif len(assets) > 1:
+            for asset_index, asset in enumerate(assets):
+                if asset_index > 0:
+                    row = copy.deepcopy(row_data)
+                    stats.append(row)
+
+                else:
+                    row = stats[row_index]
+
+                asset_cols = set_position_row(row, assets[asset_index],
+                                              asset_values)
+
+    df = pd.DataFrame(stats)
+
+    index_cols = [
+        'period_close', 'starting_cash', 'ending_cash', 'portfolio_value',
+        'pnl', 'long_exposure', 'short_exposure', 'orders', 'transactions',
+    ]
+
+    # Removing the asset specific entries
+    if recorded_cols is not None:
+        recorded_cols = [x for x in recorded_cols if x not in asset_cols]
+        for column in recorded_cols:
+            index_cols.append(column)
+
+    df['orders'] = df['orders'].apply(lambda orders: len(orders))
+    df['transactions'] = df['transactions'].apply(
+        lambda transactions: len(transactions)
+    )
+
+    if asset_cols:
+        columns = asset_cols
+        df.set_index(index_cols, drop=True, inplace=True)
+
+    else:
+        columns = index_cols
+        columns.remove('period_close')
+        df.set_index('period_close', drop=False, inplace=True)
+
+    df.dropna(axis=1, how='all', inplace=True)
+    df.sort_index(axis=0, level=0, inplace=True)
+
+    return df, columns
+
+
+def get_pretty_stats(stats, recorded_cols=None, num_rows=10):
+    """
+    Format and print the last few rows of a statistics DataFrame.
+    See the pyfolio project for the data structure.
+
+    Parameters
+    ----------
+    stats: list[Object]
+        An array of statistics for the period.
+
+    num_rows: int
+        The number of rows to display on the screen.
+
+    Returns
+    -------
+    str
+
+    """
+    if isinstance(stats, pd.DataFrame):
+        stats = stats.T.to_dict().values()
+
+    df, columns = prepare_stats(stats, recorded_cols=recorded_cols)
+
+    pd.set_option('display.expand_frame_repr', False)
+    pd.set_option('precision', 8)
+    pd.set_option('display.width', 1000)
+    pd.set_option('display.max_colwidth', 1000)
+
+    formatters = {
+        'returns': lambda returns: "{0:.4f}".format(returns),
+    }
+
+    return df.tail(num_rows).to_string(
+        columns=columns,
+        formatters=formatters
+    )
+
+
+def get_csv_stats(stats, recorded_cols=None):
+    """
+    Create a CSV buffer from the stats DataFrame.
+
+    Parameters
+    ----------
+    path: str
+    stats: list[Object]
+    recorded_cols: list[str]
+
+    Returns
+    -------
+
+    """
+    df, columns = prepare_stats(stats, recorded_cols=recorded_cols)
+
+    return df.to_csv(
+        None,
+        columns=columns,
+        # encoding='utf-8',
+        quoting=csv.QUOTE_NONNUMERIC
+    ).encode()
+
+
+def stats_to_s3(uri, stats, algo_namespace, recorded_cols=None,
+                folder='catalyst/stats', bytes_to_write=None):
+    """
+    Uploads the performance stats to a S3 bucket.
+
+    Parameters
+    ----------
+    uri: str
+    stats: list[Object]
+    algo_namespace: str
+    recorded_cols: list[str]
+    folder: str
+    bytes_to_write: str
+        Option to reuse bytes instead of re-computing the csv
+
+    Returns
+    -------
+
+    """
+    if bytes_to_write is None:
+        bytes_to_write = get_csv_stats(stats, recorded_cols=recorded_cols)
+
+    now = pd.Timestamp.utcnow()
+    timestr = now.strftime('%Y%m%d')
+    pid = os.getpid()
+
+    parts = uri.split('//')
+    obj = s3.Object(parts[1], '{}/{}-{}-{}.csv'.format(
+        folder, timestr, algo_namespace, pid
+    ))
+    obj.put(Body=bytes_to_write)
+
+
+def stats_to_algo_folder(stats, algo_namespace, recorded_cols=None):
+    """
+    Saves the performance stats to the algo local folder.
+
+    Parameters
+    ----------
+    stats: list[Object]
+    algo_namespace: str
+    recorded_cols: list[str]
+
+    Returns
+    -------
+    str
+
+    """
+    bytes_to_write = get_csv_stats(stats, recorded_cols=recorded_cols)
+
+    timestr = time.strftime('%Y%m%d')
+    folder = get_algo_folder(algo_namespace)
+
+    filename = os.path.join(folder, '{}-{}.csv'.format(timestr, 'frames'))
+
+    with open(filename, 'wb') as handle:
+        handle.write(bytes_to_write)
+
+    return bytes_to_write
+
+
+def df_to_string(df):
+    """
+    Create a formatted str representation of the DataFrame.
+
+    Parameters
+    ----------
+    df: DataFrame
+
+    Returns
+    -------
+    str
+
+    """
+    pd.set_option('display.expand_frame_repr', False)
+    pd.set_option('precision', 8)
+    pd.set_option('display.width', 1000)
+    pd.set_option('display.max_colwidth', 1000)
+
+    return df.to_string()
+
+
+def extract_transactions(perf):
+    """
+    Compute indexes for buy and sell transactions
+
+    Parameters
+    ----------
+    perf: DataFrame
+        The algo performance DataFrame.
+
+    Returns
+    -------
+    DataFrame
+        A DataFrame of transactions.
+
+    """
+    trans_list = perf.transactions.values
+    all_trans = [t for sublist in trans_list for t in sublist]
+    all_trans.sort(key=lambda t: t['dt'])
+
+    transactions = pd.DataFrame(all_trans)
+    if not transactions.empty:
+        transactions.set_index('dt', inplace=True, drop=True)
+    return transactions

catalyst/exchange/validator.py

@@ -0,0 +1,142 @@
+import os
+import tempfile
+
+import pandas as pd
+import six
+from catalyst.assets._assets import TradingPair, get_calendar
+from logbook import Logger
+from pandas.util.testing import assert_frame_equal
+
+from catalyst.constants import LOG_LEVEL
+from catalyst.exchange.asset_finder_exchange import AssetFinderExchange
+from catalyst.exchange.exchange_data_portal import DataPortalExchangeBacktest
+from catalyst.exchange.factory import get_exchanges
+from catalyst.utils.paths import ensure_directory
+
+log = Logger('Validator', level=LOG_LEVEL)
+
+
+def output_df(df, assets, name=None):
+    """
+    Outputs a price DataFrame to a temp folder.
+
+    Parameters
+    ----------
+    df: pd.DataFrame
+    assets
+    name
+
+    Returns
+    -------
+
+    """
+    if isinstance(assets, TradingPair):
+        exchange_folder = assets.exchange
+        asset_folder = assets.symbol
+    else:
+        exchange_folder = ','.join([asset.exchange for asset in assets])
+        asset_folder = ','.join([asset.symbol for asset in assets])
+
+    folder = os.path.join(
+        tempfile.gettempdir(), 'catalyst', exchange_folder, asset_folder
+    )
+    ensure_directory(folder)
+
+    if name is None:
+        name = 'output'
+
+    path = os.path.join(folder, '{}.csv'.format(name))
+    df.to_csv(path)
+
+    return path
+
+
+class Validator(object):
+    def __init__(self, data_portal):
+        self.data_portal = data_portal
+
+    def compare_bundle_with_exchange(self, exchange, assets, end_dt, bar_count,
+                                     sample_minutes):
+        """
+        Creates DataFrames from the bundle and exchange for the specified
+        data set.
+
+        Parameters
+        ----------
+        exchange: Exchange
+        assets
+        end_dt
+        bar_count
+        sample_minutes
+
+        Returns
+        -------
+
+        """
+        freq = '{}T'.format(sample_minutes)
+
+        log.info('creating data sample from bundle')
+        df1 = self.data_portal.get_history_window(
+            assets=assets,
+            end_dt=end_dt,
+            bar_count=bar_count,
+            frequency=freq,
+            field='close',
+            data_frequency='minute'
+        )
+        path = output_df(df1, assets, '{}_resampled'.format(freq))
+        log.info('saved resampled bundle candles: {}\n{}'.format(
+            path, df1.tail(10))
+        )
+
+        log.info('creating data sample from exchange api')
+        candles = exchange.get_candles(
+            end_dt=end_dt,
+            freq='{}T'.format(sample_minutes),
+            assets=assets,
+            bar_count=bar_count
+        )
+
+        series = dict()
+        for asset in assets:
+            series[asset] = pd.Series(
+                data=[candle['close'] for candle in candles[asset]],
+                index=[candle['last_traded'] for candle in candles[asset]]
+            )
+
+        df2 = pd.DataFrame(series)
+        path = output_df(df2, assets, '{}_api'.format(freq))
+        log.info('saved exchange api candles: {}\n{}'.format(
+            path, df2.tail(10))
+        )
+
+        try:
+            assert_frame_equal(df1, df2)
+            return True
+        except:
+            log.warn('differences found in dataframes')
+            return False
+
+
+if __name__ == '__main__':
+    exchanges = get_exchanges(['poloniex'])
+    exchange = six.next(six.itervalues(exchanges))
+    assets = exchange.get_assets(symbols=['eth_btc'])
+
+    open_calendar = get_calendar('OPEN')
+    asset_finder = AssetFinderExchange()
+    data_portal = DataPortalExchangeBacktest(
+        exchanges=exchanges,
+        asset_finder=asset_finder,
+        trading_calendar=open_calendar,
+        first_trading_day=None  # will set dynamically based on assets
+    )
+    validator = Validator(data_portal=data_portal)
+
+    validator.compare_bundle_with_exchange(
+        exchange=exchange,
+        assets=assets,
+        end_dt=pd.to_datetime('2017-11-10 1:00', utc=True),
+        bar_count=200,
+        sample_minutes=30
+    )

catalyst/finance/blotter.py

@@ -34,7 +34,9 @@ from catalyst.finance.commission import (
 from catalyst.finance.cancel_policy import NeverCancel
 from catalyst.utils.input_validation import expect_types
 
-log = Logger('Blotter')
+from catalyst.constants import LOG_LEVEL
+
+log = Logger('Blotter', level=LOG_LEVEL)
 warning_logger = Logger('AlgoWarning')
 
 

catalyst/finance/controls.py

@@ -24,7 +24,9 @@ from catalyst.errors import (
     TradingControlViolation,
 )
 
-log = logbook.Logger('TradingControl')
+from catalyst.constants import LOG_LEVEL
+
+log = logbook.Logger('TradingControl', level=LOG_LEVEL)
 
 
 class TradingControl(with_metaclass(abc.ABCMeta)):

catalyst/finance/execution.py

@@ -15,13 +15,8 @@
 
 import abc
 
-from sys import float_info
-
-from six import with_metaclass
-
-import catalyst.utils.math_utils as zp_math
-
 from numpy import isfinite
+from six import with_metaclass
 
 from catalyst.errors import BadOrderParameters
 
@@ -77,6 +72,7 @@ class LimitOrder(ExecutionStyle):
     Execution style representing an order to be executed at a price equal to or
     better than a specified limit price.
     """
+
     def __init__(self, limit_price, exchange=None):
         """
         Store the given price.
@@ -99,6 +95,7 @@ class StopOrder(ExecutionStyle):
     Execution style representing an order to be placed once the market price
     reaches a specified stop price.
     """
+
     def __init__(self, stop_price, exchange=None):
         """
         Store the given price.
@@ -121,6 +118,7 @@ class StopLimitOrder(ExecutionStyle):
     Execution style representing a limit order to be placed with a specified
     limit price once the market reaches a specified stop price.
     """
+
     def __init__(self, limit_price, stop_price, exchange=None):
         """
         Store the given prices
@@ -144,31 +142,20 @@ class StopLimitOrder(ExecutionStyle):
 def asymmetric_round_price_to_penny(price, prefer_round_down,
                                     diff=(0.0095 - .005)):
     """
-    Asymmetric rounding function for adjusting prices to two places in a way
-    that "improves" the price.  For limit prices, this means preferring to
-    round down on buys and preferring to round up on sells.  For stop prices,
-    it means the reverse.
+    Modified the original function because we do not want to round
+    prices on crypto exchange.
 
-    If prefer_round_down == True:
-        When .05 below to .95 above a penny, use that penny.
-    If prefer_round_down == False:
-        When .95 below to .05 above a penny, use that penny.
+    Parameters
+    ----------
+    price: float
+
+    Returns
+    -------
+    float
 
-    In math-speak:
-    If prefer_round_down: [<X-1>.0095, X.0195) -> round to X.01.
-    If not prefer_round_down: (<X-1>.0005, X.0105] -> round to X.01.
     """
-    # Subtracting an epsilon from diff to enforce the open-ness of the upper
-    # bound on buys and the lower bound on sells.  Using the actual system
-    # epsilon doesn't quite get there, so use a slightly less epsilon-ey value.
-    epsilon = float_info.epsilon * 10
-    diff = diff - epsilon
-
-    # relies on rounding half away from zero, unlike numpy's bankers' rounding
-    rounded = round(price - (diff if prefer_round_down else -diff), 2)
-    if zp_math.tolerant_equals(rounded, 0.0):
-        return 0.0
-    return rounded
+    # TODO: consider overriding outside of the original function
+    return price
 
 
 def check_stoplimit_prices(price, label):

catalyst/finance/performance/period.py

@@ -88,7 +88,10 @@ from six import itervalues, iteritems
 
 import catalyst.protocol as zp
 
-log = logbook.Logger('Performance')
+from catalyst.constants import LOG_LEVEL
+
+log = logbook.Logger('Performance', level=LOG_LEVEL)
+
 TRADE_TYPE = zp.DATASOURCE_TYPE.TRADE
 
 

catalyst/finance/performance/position.py

@@ -40,7 +40,9 @@ import logbook
 from catalyst.assets import Future, Asset
 from catalyst.utils.input_validation import expect_types
 
-log = logbook.Logger('Performance')
+from catalyst.constants import LOG_LEVEL
+
+log = logbook.Logger('Performance', level=LOG_LEVEL)
 
 
 class Position(object):

catalyst/finance/performance/position_tracker.py

@@ -32,7 +32,9 @@ from catalyst.assets import (
 )
 from . position import positiondict
 
-log = logbook.Logger('Performance')
+from catalyst.constants import LOG_LEVEL
+
+log = logbook.Logger('Performance', level=LOG_LEVEL)
 
 
 PositionStats = namedtuple('PositionStats',

catalyst/finance/performance/tracker.py

@@ -70,7 +70,9 @@ import catalyst.finance.risk as risk
 
 from . position_tracker import PositionTracker
 
-log = logbook.Logger('Performance')
+from catalyst.constants import LOG_LEVEL
+
+log = logbook.Logger('Performance', level=LOG_LEVEL)
 
 
 class PerformanceTracker(object):
@@ -111,27 +113,11 @@ class PerformanceTracker(object):
                     self.treasury_curves,
                     self.trading_calendar
                 )
-        elif self.emission_rate == '5-minute':
-            self.all_benchmark_returns = pd.Series(
-                index=pd.date_range(
-                    self.sim_params.first_open,
-                    self.sim_params.last_close,
-                    freq='5min'
-                ),
-            )
-            self.cumulative_risk_metrics = \
-                risk.RiskMetricsCumulative(
-                    self.sim_params,
-                    self.treasury_curves,
-                    self.trading_calendar,
-                    create_first_day_stats=True,
-                )
         elif self.emission_rate == 'minute':
             self.all_benchmark_returns = pd.Series(index=pd.date_range(
                 self.sim_params.first_open, self.sim_params.last_close,
                 freq='Min')
             )
-
             self.cumulative_risk_metrics = \
                 risk.RiskMetricsCumulative(
                     self.sim_params,

catalyst/finance/risk/cumulative.py

@@ -22,7 +22,7 @@ from pandas.tseries.tools import normalize_date
 
 from six import iteritems
 
-from . risk import (
+from .risk import (
     check_entry,
     choose_treasury
 )
@@ -37,9 +37,10 @@ from empyrical import (
     sharpe_ratio,
     sortino_ratio,
 )
+import warnings
+from catalyst.constants import LOG_LEVEL
 
-log = logbook.Logger('Risk Cumulative')
-
+log = logbook.Logger('Risk Cumulative', level=LOG_LEVEL)
 
 choose_treasury = functools.partial(choose_treasury, lambda *args: '10year',
                                     compound=False)
@@ -143,6 +144,8 @@ class RiskMetricsCumulative(object):
         self.num_trading_days = 0
 
     def update(self, dt, algorithm_returns, benchmark_returns, leverage):
+        warnings.filterwarnings('error')
+
         # Keep track of latest dt for use in to_dict and other methods
         # that report current state.
         self.latest_dt = dt
@@ -189,9 +192,12 @@ class RiskMetricsCumulative(object):
             if len(self.benchmark_returns) == 1:
                 self.benchmark_returns = np.append(0.0, self.benchmark_returns)
 
-        self.benchmark_cumulative_returns[dt_loc] = cum_returns(
-            self.benchmark_returns
-        )[-1]
+        try:
+            self.benchmark_cumulative_returns[dt_loc] = cum_returns(
+                self.benchmark_returns
+            )[-1]
+        except Exception:
+            self.benchmark_cumulative_returns[dt_loc] = 0
 
         benchmark_cumulative_returns_to_date = \
             self.benchmark_cumulative_returns[:dt_loc + 1]
@@ -266,10 +272,17 @@ algorithm_returns ({algo_count}) in range {start} : {end} on {dt}"
         self.downside_risk[dt_loc] = downside_risk(
             self.algorithm_returns
         )
-        self.sortino[dt_loc] = sortino_ratio(
-            self.algorithm_returns,
-            _downside_risk=self.downside_risk[dt_loc]
-        )
+
+        try:
+            risk = self.downside_risk[dt_loc]
+            self.sortino[dt_loc] = sortino_ratio(
+                self.algorithm_returns,
+                _downside_risk=risk
+            )
+        except Exception:
+            # TODO: what causes it to error out?
+            self.sortino[dt_loc] = 0
+
         self.information[dt_loc] = information_ratio(
             self.algorithm_returns,
             self.benchmark_returns,
@@ -281,6 +294,8 @@ algorithm_returns ({algo_count}) in range {start} : {end} on {dt}"
         self.max_leverage = self.calculate_max_leverage()
         self.max_leverages[dt_loc] = self.max_leverage
 
+        warnings.resetwarnings()
+
     def to_dict(self):
         """
         Creates a dictionary representing the state of the risk report.
@@ -292,18 +307,18 @@ algorithm_returns ({algo_count}) in range {start} : {end} on {dt}"
         rval = {
             'trading_days': self.num_trading_days,
             'benchmark_volatility':
-            self.benchmark_volatility[dt_loc],
+                self.benchmark_volatility[dt_loc],
             'algo_volatility':
-            self.algorithm_volatility[dt_loc],
+                self.algorithm_volatility[dt_loc],
             'treasury_period_return': self.treasury_period_return,
             # Though the two following keys say period return,
             # they would be more accurately called the cumulative return.
             # However, the keys need to stay the same, for now, for backwards
             # compatibility with existing consumers.
             'algorithm_period_return':
-            self.algorithm_cumulative_returns[dt_loc],
+                self.algorithm_cumulative_returns[dt_loc],
             'benchmark_period_return':
-            self.benchmark_cumulative_returns[dt_loc],
+                self.benchmark_cumulative_returns[dt_loc],
             'beta': self.beta[dt_loc],
             'alpha': self.alpha[dt_loc],
             'sharpe': self.sharpe[dt_loc],

catalyst/finance/risk/period.py

@@ -14,6 +14,7 @@
 # limitations under the License.
 
 import functools
+import warnings
 
 import logbook
 
@@ -23,7 +24,7 @@ import numpy as np
 import pandas as pd
 
 from . import risk
-from . risk import check_entry
+from .risk import check_entry
 
 from empyrical import (
     alpha_beta_aligned,
@@ -36,7 +37,9 @@ from empyrical import (
     sortino_ratio
 )
 
-log = logbook.Logger('Risk Period')
+from catalyst.constants import LOG_LEVEL
+
+log = logbook.Logger('Risk Period', level=LOG_LEVEL)
 
 choose_treasury = functools.partial(risk.choose_treasury,
                                     risk.select_treasury_duration)
@@ -76,14 +79,20 @@ class RiskMetricsPeriod(object):
         self.calculate_metrics()
 
     def calculate_metrics(self):
-        self.benchmark_period_returns = \
-            cum_returns(self.benchmark_returns).iloc[-1]
+        warnings.filterwarnings('error')
+
+        try:
+            self.benchmark_period_returns = \
+                cum_returns(self.benchmark_returns).iloc[-1]
+        except Exception:
+            # TODO: why is there an error
+            self.benchmark_period_returns = 0
 
         self.algorithm_period_returns = \
             cum_returns(self.algorithm_returns).iloc[-1]
 
         if not self.algorithm_returns.index.equals(
-            self.benchmark_returns.index
+                self.benchmark_returns.index
         ):
             message = "Mismatch between benchmark_returns ({bm_count}) and \
             algorithm_returns ({algo_count}) in range {start} : {end}"
@@ -126,10 +135,17 @@ class RiskMetricsPeriod(object):
         self.downside_risk = downside_risk(
             self.algorithm_returns.values
         )
-        self.sortino = sortino_ratio(
-            self.algorithm_returns.values,
-            _downside_risk=self.downside_risk,
-        )
+
+        try:
+            risk = self.downside_risk
+            self.sortino = sortino_ratio(
+                self.algorithm_returns.values,
+                _downside_risk=risk,
+            )
+        except Exception:
+            # TODO: what causes it to error out?
+            self.sortino = 0
+
         self.information = information_ratio(
             self.algorithm_returns.values,
             self.benchmark_returns.values,
@@ -138,11 +154,13 @@ class RiskMetricsPeriod(object):
             self.algorithm_returns.values,
             self.benchmark_returns.values,
         )
-        self.excess_return = self.algorithm_period_returns - \
-            self.treasury_period_return
+        self.excess_return = self.algorithm_period_returns \
+            - self.treasury_period_return
         self.max_drawdown = max_drawdown(self.algorithm_returns.values)
         self.max_leverage = self.calculate_max_leverage()
 
+        warnings.resetwarnings()
+
     def to_dict(self):
         """
         Creates a dictionary representing the state of the risk report.

catalyst/finance/risk/report.py

@@ -63,7 +63,9 @@ from dateutil.relativedelta import relativedelta
 
 from . period import RiskMetricsPeriod
 
-log = logbook.Logger('Risk Report')
+from catalyst.constants import LOG_LEVEL
+
+log = logbook.Logger('Risk Report', level=LOG_LEVEL)
 
 
 class RiskReport(object):

catalyst/finance/risk/risk.py

@@ -61,7 +61,9 @@ Risk Report
 import logbook
 import numpy as np
 
-log = logbook.Logger('Risk')
+from catalyst.constants import LOG_LEVEL
+
+log = logbook.Logger('Risk', level=LOG_LEVEL)
 
 
 TREASURY_DURATIONS = [
@@ -158,7 +160,8 @@ def choose_treasury(select_treasury, treasury_curves, start_session,
                 )
                 break
 
-        if search_day:
+        # Supress warning for 'OPEN' calendar
+        if search_day and trading_calendar.name != 'OPEN':
             if (search_dist is None or search_dist > 1) and \
                     search_days[0] <= end_session <= search_days[-1]:
                 message = "No rate within 1 trading day of end date = \

catalyst/finance/slippage.py

@@ -205,20 +205,22 @@ class VolumeShareSlippage(SlippageModel):
     def process_order(self, data, order):
         volume = data.current(order.asset, "volume")
 
+        min_trade_size = order.asset.min_trade_size
+
         max_volume = self.volume_limit * volume
 
         # price impact accounts for the total volume of transactions
         # created against the current minute bar
         remaining_volume = max_volume - self.volume_for_bar
-        if remaining_volume < 1:
+        if remaining_volume < min_trade_size:
             # we can't fill any more transactions
             raise LiquidityExceeded()
 
         # the current order amount will be the min of the
         # volume available in the bar or the open amount.
-        cur_volume = int(min(remaining_volume, abs(order.open_amount)))
+        cur_volume = min(remaining_volume, abs(order.open_amount))
 
-        if cur_volume < 1:
+        if cur_volume < min_trade_size:
             return None, None
 
         # tally the current amount into our total amount ordered.

catalyst/finance/trading.py

@@ -26,7 +26,9 @@ from catalyst.data.loader import load_market_data
 from catalyst.utils.calendars import get_calendar
 from catalyst.utils.memoize import remember_last
 
-log = logbook.Logger('Trading')
+from catalyst.constants import LOG_LEVEL
+
+log = logbook.Logger('Trading', level=LOG_LEVEL)
 
 
 DEFAULT_CAPITAL_BASE = 1e5

catalyst/finance/transaction.py

@@ -65,14 +65,10 @@ def create_transaction(order, dt, price, amount):
     # floor the amount to protect against non-whole number orders
     # TODO: Investigate whether we can add a robust check in blotter
     # and/or tradesimulation, as well.
-    amount_magnitude = int(abs(amount))
-
-    if amount_magnitude < 1:
-        raise Exception("Transaction magnitude must be at least 1.")
 
     transaction = Transaction(
         asset=order.asset,
-        amount=int(amount),
+        amount=amount,
         dt=dt,
         price=price,
         order_id=order.id

catalyst/gens/sim_engine.pyx

@@ -20,9 +20,7 @@ cimport cython
 from cpython cimport bool
 
 cdef np.int64_t _nanos_in_minute = 60000000000
-cdef np.int64_t _nanos_in_five_minutes = 5 * _nanos_in_minute
 NANOS_IN_MINUTE = _nanos_in_minute
-NANOS_IN_FIVE_MINUTES = _nanos_in_five_minutes
 
 cpdef enum:
     BAR = 0
@@ -117,24 +115,3 @@ cdef class MinuteSimulationClock:
             yield minute, BAR
             if minute_emission:
                 yield minute, MINUTE_END
-
-cdef class FiveMinuteSimulationClock(MinuteSimulationClock):
-    @cython.boundscheck(False)
-    @cython.wraparound(False)
-    cdef dict calc_minutes_by_session(self):
-        cdef dict five_minutes_by_session
-        cdef int session_idx
-        cdef np.int64_t session_nano
-        cdef np.ndarray[np.int64_t, ndim=1] five_minutes_nanos
-
-        five_minutes_by_session = {}
-        for session_idx, session_nano in enumerate(self.sessions_nanos):
-            five_minutes_nanos = np.arange(
-                self.market_opens_nanos[session_idx],
-                self.market_closes_nanos[session_idx],
-                _nanos_in_five_minutes
-            )
-            five_minutes_by_session[session_nano] = pd.to_datetime(
-                five_minutes_nanos, utc=True, box=True
-            )
-        return five_minutes_by_session

catalyst/gens/tradesimulation.py

@@ -27,14 +27,15 @@ from catalyst.gens.sim_engine import (
     BEFORE_TRADING_START_BAR
 )
 
-log = Logger('Trade Simulation')
+from catalyst.constants import LOG_LEVEL
+
+log = Logger('Trade Simulation', level=LOG_LEVEL)
 
 
 class AlgorithmSimulator(object):
 
     EMISSION_TO_PERF_KEY_MAP = {
         'minute': 'minute_perf',
-        '5-minute': '5_minute_perf',
         'daily': 'daily_perf'
     }
 
@@ -202,7 +203,7 @@ class AlgorithmSimulator(object):
             stack.enter_context(self.processor)
             stack.enter_context(ZiplineAPI(self.algo))
 
-            if algo.data_frequency in set(('minute', '5-minute')):
+            if algo.data_frequency == 'minute':
                 def execute_order_cancellation_policy():
                     algo.blotter.execute_cancel_policy(SESSION_END)
 

catalyst/pipeline/factors/equity/__init__.py

@@ -1,9 +1,6 @@
 from .statistical import (
-    RollingPearson,
-    RollingLinearRegression,
     RollingLinearRegressionOfReturns,
     RollingPearsonOfReturns,
-    RollingSpearman,
     RollingSpearmanOfReturns,
 )
 from .technical import (

catalyst/pipeline/loaders/crypto_pricing_loader.py

@@ -41,11 +41,7 @@ class CryptoPricingLoader(PipelineLoader):
             reader = bundle.daily_bar_reader
             all_sessions = cal.all_sessions
 
-        elif data_frequency == '5-minute':
-            reader = bundle.five_minute_bar_reader
-            all_sessions = cal.all_five_minutes
-
-        elif daily_bar_reader == 'minute':
+        elif data_frequency == 'minute':
             reader = bundle.minute_bar_reader
             all_sessions = cal.all_minutes
 
@@ -106,12 +102,6 @@ class CryptoPricingLoader(PipelineLoader):
 
 
 def _shift_dates(dates, start_date, end_date, shift):
-    print 'dates.head:\n', dates[:10]
-    print 'dates.tail:\n', dates[:-10]
-
-    print 'start_date:', start_date
-    print 'end_date:', end_date
-    print 'shift:', shift
 
     try:
         start = dates.get_loc(start_date)

catalyst/pipeline/loaders/equity_pricing_loader.py

@@ -38,11 +38,11 @@ class USEquityPricingLoader(PipelineLoader):
 
     def __init__(self, bundle, data_frequency, dataset):
 
-        if data_frequency == 'daily':
-            reader = bundle.daily_bar_reader
-        elif data_frequency == '5-minute':
-            reader = bundle.five_minute_bar_reader
-        elif daily_bar_reader == 'minute':
+        # TODO: This is currently broken, No Pipeline support for Catalyst
+        # if data_frequency == 'daily':
+        #    reader = bundle.daily_bar_reader
+        # elif daily_bar_reader == 'minute':
+        if data_frequency == 'minute':
             reader = bundle.minute_bar_reader
         else:
             raise ValueError(
@@ -53,10 +53,9 @@ class USEquityPricingLoader(PipelineLoader):
 
         if data_frequency == 'daily':
             all_sessions = cal.all_sessions
-        elif data_frequency == '5-minute':
-            reader = bundle.five_minute_bar_reader
-            all_sessions = cal.all_five_minutes
-        elif daily_bar_reader == 'minute':
+        # TODO: this cannot be right, but no pipeline support at the moment
+        # elif daily_bar_reader == 'minute':
+        elif data_frequency == 'minute':
             reader = bundle.minute_bar_reader
             all_sessions = cal.all_minutes
 

catalyst/pipeline/loaders/events.py

@@ -231,7 +231,7 @@ class EventsLoader(PipelineLoader):
             self.load_next_events(n, dates, sids, mask),
             self.load_previous_events(p, dates, sids, mask),
         )
-    
+
     @property
     def columns(self):
         return self._columns

catalyst/pipeline/loaders/frame.py

@@ -180,4 +180,3 @@ class DataFrameLoader(PipelineLoader):
     @property
     def columns(self):
         return self._columns
-

catalyst/pipeline/loaders/synthetic.py

@@ -163,7 +163,7 @@ class SeededRandomLoader(PrecomputedLoader):
             bool_dtype: self._bool_values,
             object_dtype: self._object_values,
         }[dtype](shape)
-    
+
     @property
     def columns(self):
         return self._columns

catalyst/sources/benchmark_source.py

@@ -51,10 +51,7 @@ class BenchmarkSource(object):
         elif benchmark_returns is not None:
             daily_series = benchmark_returns[sessions[0]:sessions[-1]]
 
-            print 'BENCHMARK_RETURNS'
-
             if self.emission_rate == "minute":
-                print 'BENCHMARK_RETURNS minute'
                 # we need to take the env's benchmark returns, which are daily,
                 # and resample them to minute
                 minutes = trading_calendar.minutes_for_sessions_in_range(
@@ -68,29 +65,20 @@ class BenchmarkSource(object):
                 )
 
                 self._precalculated_series = minute_series
-            elif self.emission_rate == '5-minute':
-                print 'BENCHMARK_RETURNS 5-minute'
-                five_minutes = \
-                    trading_calendar.five_minutes_for_sessions_in_range(
-                        sessions[0],
-                        sessions[-1],
-                    )
-
-                five_minute_series = daily_series.reindex(
-                    index=five_minutes,
-                    method='ffill',
-                )
-                
-                self._precalculated_series = five_minute_series
             else:
-                print 'BENCHMARK_RETURNS daily'
                 self._precalculated_series = daily_series
         else:
             raise Exception("Must provide either benchmark_asset or "
                             "benchmark_returns.")
 
     def get_value(self, dt):
-        return self._precalculated_series.loc[dt]
+        try:
+            series = self._precalculated_series
+            value = series.loc[dt]
+            return value
+        except Exception:
+            # TODO: workaround, find permanent fix
+            return 0
 
     def get_range(self, start_dt, end_dt):
         return self._precalculated_series.loc[start_dt:end_dt]
@@ -173,24 +161,8 @@ class BenchmarkSource(object):
                 ffill=True
             )[asset]
 
-            return benchmark_series.pct_change()[1:]
-        elif self.emission_rate == '5-minute':
-            five_minutes = trading_calendar.five_minutes_for_sessions_in_range(
-                self.sessions[0], self.sessions[-1]
-            )
-            benchmark_series = data_portal.get_history_window(
-                [asset],
-                five_minutes[-1],
-                bar_count=len(five_minutes) + 1,
-                frequency='5m',
-                field='price',
-                data_frequency=self.emission_rate,
-                ffill=True,
-            )[asset]
-
             return benchmark_series.pct_change()[1:]
         else:
-            print '----------------------------------------'
             start_date = asset.start_date
             if start_date < trading_days[0]:
                 # get the window of close prices for benchmark_asset from the

catalyst/sources/requests_csv.py

@@ -23,7 +23,9 @@ from catalyst.protocol import (
 )
 from catalyst.assets import Equity
 
-logger = Logger('Requests Source Logger')
+from catalyst.constants import LOG_LEVEL
+
+logger = Logger('Requests Source Logger', level=LOG_LEVEL)
 
 
 def roll_dts_to_midnight(dts, trading_day):

catalyst/support/issue_55.py

@@ -0,0 +1,41 @@
+import pandas as pd
+
+from catalyst import run_algorithm
+from catalyst.api import symbol
+
+
+def initialize(context):
+    print('initializing')
+    context.asset = symbol('xcp_btc')
+
+
+def handle_data(context, data):
+    print('handling bar: {}'.format(data.current_dt))
+
+    price = data.current(context.asset, 'close')
+    print('got price {price}'.format(price=price))
+
+    try:
+        prices = data.history(
+            context.asset,
+            fields='close',
+            bar_count=1,
+            frequency='1D'
+        )
+        print('got {} price entries\n'.format(len(prices), prices))
+    except Exception as e:
+        print(e)
+
+
+run_algorithm(
+    capital_base=1,
+    start=pd.to_datetime('2015-3-2', utc=True),
+    end=pd.to_datetime('2017-8-31', utc=True),
+    data_frequency='daily',
+    initialize=initialize,
+    handle_data=handle_data,
+    analyze=None,
+    exchange_name='poloniex',
+    algo_namespace='issue_55',
+    base_currency='btc'
+)

catalyst/utils/calendars/exchange_calendar_open.py

@@ -1,6 +1,7 @@
 from datetime import time
 from pytz import timezone
 
+from pandas import Timestamp
 from pandas.tseries.offsets import DateOffset
 
 from catalyst.utils.memoize import lazyval
@@ -28,3 +29,7 @@ class OpenExchangeCalendar(TradingCalendar):
     @lazyval
     def day(self):
         return DateOffset(days=1)
+
+    def __init__(self, *args, **kwargs):
+        super(OpenExchangeCalendar, self).__init__(
+            start=Timestamp('2015-3-1', tz='UTC'), **kwargs)

catalyst/utils/calendars/trading_calendar.py

@@ -117,9 +117,6 @@ class TradingCalendar(with_metaclass(ABCMeta)):
 
         self._trading_minutes_nanos = self.all_minutes.values.\
             astype(np.int64)
-        
-        self._trading_five_minutes_nanos = self.all_five_minutes.values.\
-            astype(np.int64)
 
         self.first_trading_session = _all_days[0]
         self.last_trading_session = _all_days[-1]
@@ -182,18 +179,6 @@ class TradingCalendar(with_metaclass(ABCMeta)):
         """
         return int(self._minutes_per_session[start_session:end_session].sum())
 
-    @lazyval
-    def _five_minutes_per_session(self):
-        diff = self.schedule.market_close - self.schedule.market_open
-        diff = diff.astype('timedelta64[m]')
-        return (diff + 1) // 5
-
-    def five_minutes_count_for_sessions_in_range(self,
-                                                 start_session,
-                                                 end_session):
-        five_mins = self._five_minutes_per_session[start_session:end_session]
-        return int(five_mins.sum())
-
     @property
     def regular_holidays(self):
         """
@@ -386,10 +371,6 @@ class TradingCalendar(with_metaclass(ABCMeta)):
         idx = next_divider_idx(self._trading_minutes_nanos, dt.value)
         return self.all_minutes[idx]
 
-    def next_five_minute(self, dt):
-        idx = next_divider_idx(self._trading_five_minutes_nanos, dt.values)
-        return self.all_five_mintutes[idx]
-
     def previous_minute(self, dt):
         """
         Given a dt, return the previous exchange minute.
@@ -484,12 +465,6 @@ class TradingCalendar(with_metaclass(ABCMeta)):
             end_minute=self.schedule.at[session_label, 'market_close'],
         )
 
-    def five_minutes_for_session(self, session_label):
-        return self.five_minutes_in_range(
-            start_five_minute=self.schedule.at[session_label, 'market_open'],
-            end_five_minute=self.schedule.at[session_label, 'market_close'],
-        )
-
     def minutes_window(self, start_dt, count):
         start_dt_nanos = start_dt.value
         all_minutes_nanos = self._trading_minutes_nanos
@@ -591,20 +566,6 @@ class TradingCalendar(with_metaclass(ABCMeta)):
 
         return abs(end_idx - start_idx)
 
-    def five_minutes_in_range(self, start_five_minute, end_five_minute):
-        start_idx = searchsorted(self._trading_five_minutes_nanos,
-                                 start_five_minute.value)
-
-        end_idx = searchsorted(self._trading_five_minutes_nanos,
-                               end_five_minute.value)
-
-        if end_five_minute.value == self._trading_five_minutes_nanos[end_idx]:
-            # if the end minute is a market minute, increase by 1
-            end_idx += 1
-
-        return self.all_five_minutes[start_idx:end_idx]
-        
-
     def minutes_in_range(self, start_minute, end_minute):
         """
         Given start and end minutes, return all the calendar minutes
@@ -662,15 +623,6 @@ class TradingCalendar(with_metaclass(ABCMeta)):
 
         return self.minutes_in_range(first_minute, last_minute)
 
-    def five_minutes_for_sessions_in_range(self,
-                                           start_session_label,
-                                           end_session_label):
-
-        first_minute, _ = self.open_and_close_for_session(start_session_label)
-        _, last_minute = self.open_and_close_for_session(end_session_label)
-
-        return self.five_minutes_in_range(first_minute, last_minute)
-
     def open_and_close_for_session(self, session_label):
         """
         Returns a tuple of timestamps of the open and close of the session
@@ -777,13 +729,6 @@ class TradingCalendar(with_metaclass(ABCMeta)):
 
         return DatetimeIndex(all_minutes).tz_localize("UTC")
 
-    @lazyval
-    def all_five_minutes(self):
-        """
-        Returns a DatetimeIndex representing all the five minutes in this calendar.
-        """
-        return self._all_minutes_with_interval(5) 
-
     @lazyval
     def all_minutes(self):
         """