Merge branch 'develop' - Release 0.3.1

FIX: bundle start_dt cannot be earlier than asset_start
FIX: prior raise of AuthNotFound, now generates empty auth.json, and raises AuthEmpty when live
FIX: os.path.join to make BUNDLE_NAME_TEMPLATE compatible across OSes

.github/ISSUE_TEMPLATE.md

@@ -1,4 +1,4 @@
-Dear Zipline Maintainers,
+Dear Catalyst Maintainers,
 
 Before I tell you about my issue, let me describe my environment:
 
@@ -7,7 +7,7 @@ Before I tell you about my issue, let me describe my environment:
 * Operating System: (Windows Version or `$ uname --all`)
 * Python Version: `$ python --version`
 * Python Bitness: `$ python -c 'import math, sys;print(int(math.log(sys.maxsize + 1, 2) + 1))'`
-* How did you install Zipline: (`pip`, `conda`, or `other (please explain)`)
+* How did you install Catalyst: (`pip`, `conda`, or `other (please explain)`)
 * Python packages: `$ pip freeze` or `$ conda list`
 
 Now that you know a little about me, let me tell you about the issue I am

.gitignore

@@ -78,3 +78,7 @@ zipline.iml
 ./data
 
 TAGS
+
+python2
+python3
+scratch

README.rst

@@ -1,196 +1 @@
-========
-Catalyst
-========
-
-|version status|
-
-Catalyst is an algorithmic trading library for crypto-assets written in Python.
-It allows trading strategies to be easily expressed and backtested against historical data, providing analytics and insights regarding a particular strategy's performance.
-Catalyst will be expanded to support live-trading of crypto-assets in the coming months.
-Please visit `<enigma.co>`_ to learn about Catalyst, or refer to the 
-`whitepaper <https://www.enigma.co/enigma_catalyst.pdf>`_ for further technical details.
-
-Catalyst builds on top of the well-established `Zipline <https://github.com/quantopian/zipline>`_ project.
-We did our best to minimize structural changes to the general API to maximize compatibility with existing trading algorithms, developer knowledge, and tutorials.
-For now, please refer to the `Zipline API Docs <http://zipline.io>`_ as a general reference and bring any other questions you have to our #dev channel on `Slack <https://join.slack.com/enigmacatalyst/shared_invite/MTkzMjQ0MTg1NTczLTE0OTY3MjE3MDEtZGZmMTI5YzI3ZA>`_.
-
-Our primary contributions include the:
-
-- Intruction of an open trading calendar, that permits simulation to allow trades on weekends, holidays, and outside of normal business hours.
-- Curation of OHLCV data bundle from `Poloniex's API <https://poloniex.com/support/api/>`_, which contains data in five-minute intervals as early as 2/19/2015.
-- Support for backtesting for daily trading strategies, support for five-minute backtesting is in development.
-- Addition Bitcoin price (USDT_BTC) as a benchmark asset for comparing performance.
-
-Interested in getting involved?
-`Join us on Slack! <https://join.slack.com/enigmacatalyst/shared_invite/MTkzMjQ0MTg1NTczLTE0OTY3MjE3MDEtZGZmMTI5YzI3ZA>`_
-
-
-Installation
-============
-
-At the moment, Catalyst has some fairly specific and strict depedency requirements.
-We recommend the use of Python virtual environments if you wish to simplify the installation process, or otherwise isolate Catalyst's dependencies from your other projects.
-If you don't have ``virtualenv`` installed, see our later section on Virtual Environments.
-
-.. code-block:: bash
-
-    $ virtualenv catalyst-venv
-    $ source ./catalyst-venv/bin/activate
-    $ pip install enigma-catalyst
-
-**Note:** A successful installation will require several minutes in order to compile dependencies that expose C APIs.
-
-Dependencies
-------------
-
-Catalyst's depedencies can be found in the ``etc/requirements.txt`` file.
-If you need to install them outside of a typical ``pip install``, this is done using:
-
-.. code-block:: bash
-
-    $ pip install -r etc/requirements.txt
-
-Though not required by Catalyst directly, our example algorithms use matplotlib to visually display backtest results.
-If you wish to run any examples or use matplotlib during development, it can be installed using:
-
-.. code-block:: bash
-
-    $ pip install matplotlib
-
-**Note:** If you plan to use matplotlib and virtualenv on Mac OS X, see our later section for additional setup instructions.
-
-Getting Started
-===============
-
-The following code implements a simple buy and hodl algorithm.  The full source can be found in ``catalyst/examples/buy_and_hodl.py``.
-
-.. code:: python
-
-    import numpy as np
-    
-    from catalyst.api import (
-        order_target_value,
-        symbol,
-        record,
-        cancel_order,
-        get_open_orders,
-    )
-    
-    ASSET = 'USDT_BTC'
-    
-    TARGET_HODL_RATIO = 0.8
-    RESERVE_RATIO = 1.0 - TARGET_HODL_RATIO
-
-    def initialize(context):
-        context.is_buying = True
-        context.asset = symbol(ASSET)
-
-    def handle_data(context, data):
-        cash = context.portfolio.cash
-        target_hodl_value = TARGET_HODL_RATIO * context.portfolio.starting_cash
-        reserve_value = RESERVE_RATIO * context.portfolio.starting_cash
-        
-        # Cancel any outstanding orders from the previous day
-        orders = get_open_orders(context.asset) or []
-        for order in orders:
-            cancel_order(order)
-        
-        # Stop buying after passing reserve threshold
-        if cash <= reserve_value:
-            context.is_buying = False
-        
-        # Retrieve current price from pricing data
-        price = data[context.asset].price
-        
-        # Check if still buying and could (approximately) afford another purchase                    
-        if context.is_buying and cash > price:
-            # Place order to make position in asset equal to target_hodl_value
-            order_target_value(
-                context.asset,
-                target_hodl_value,
-                limit_price=1.1 * price,
-                stop_price=0.9 * price,
-            )
-        
-        # Record any state for later analysis
-        record(
-            price=price,
-            cash=context.portfolio.cash,
-            leverage=context.account.leverage,
-        )
-
-
-You can then run this algorithm using the Catalyst CLI. From the command
-line, run:
-
-.. code:: bash
-
-    $ catalyst ingest
-    $ catalyst run -f buy_and_hodl.py --start 2015-3-1 --end 2017-6-28 --capital-base 100000 -o bah.pickle
-
-This will download the crypto-asset price data from a poloniex bundle
-curated by Enigma in the specified time range and stream it through
-the algorithm and plot the resulting performance using matplotlib.
-
-You can find other examples in the ``catalyst/examples`` directory.
-
-Limitations
------------
-
-This project is currently in a pre-alpha state and has some limitations we'd like to address:
-
-- *Minimum Denomination:* The smallest tradable unit in Catalyst is equal to 1/1000th of a full coin. We plan to enable more granular increments, but have capped it at 1/1000th for the time being.
-- *Supported Assets:* Currently the poloniex bundle comes prepopulated with data for all 90 registered trading pairs. However, due to limitations in how portfolios are currently modeled, we recommend sticking to ``USDT_*`` trading pairs. USDT is an independent currency listed on Poloniex whose price is pegged to the US dollar. Currently, this list includes: ``USDT_BTC``, ``USDT_DASH``, ``USDT_ETC``, ``USDT_ETH``, ``USDT_LTC``, ``USDT_NXT``, ``USDT_REP``, ``USDT_STR``, ``USDT_XMR``, ``USDT_XRP``, and ``USDT_ZEC``. We plan to add support for basing your portfolio in arbitrary currencies and provide native support for modeling ForEx trades in the near future!
-
-Virtual Environments
-====================
-
-Here we will provide a brief tutorial for installing ``virtualenv`` and its basic usage.
-For more information regarding ``virtualenv``, please refer to this `virtualenv guide <http://python-guide-pt-br.readthedocs.io/en/latest/dev/virtualenvs/>`_.
-
-The ``virtualenv`` command can be installed using:
-
-.. code-block:: bash
-
-    $ pip install virtualenv
-
-To create a new virtual environment, choose a directory, e.g. ``/path/to/venv-dir``, where project-specific packages and files will be stored.  The environment is created by running:
-
-.. code-block:: bash
-
-    $ virtualenv /path/to/venv-dir
-
-To enter an environment, run the ``bin/activate`` script located in ``/path/to/venv-dir`` using:
-
-.. code-block:: bash
-
-    $ source /path/to/venv-dir/bin/activate
-
-Exiting an environment is accomplished using ``deactivate``, and removing it entirely is done by deleting ``/path/to/venv-dir``.
-
-OS X + virtualenv + matplotlib
--------------------------------------
-
-A note about using matplotlib in virtual enviroments on OS X: it may be necessary to run
-
-.. code-block:: python
-
-    echo "backend: TkAgg" > ~/.matplotlib/matplotlibrc
-
-in order to override the default ``macosx`` backend for your system, which may not be accessible from inside the virtual environment.
-This will allow Catalyst to open matplotlib charts from within a virtual environment, which is useful for displaying the performance of your backtests.  To learn more about matplotlib backends, please refer to the
-`matplotlib backend documentation <https://matplotlib.org/faq/usage_faq.html#what-is-a-backend>`_.
-
-Disclaimer
-==========
-
-Keep in mind that this project is still under active development, and is not recommended for production use in its current state.
-We are deeply committed to improving the overall user experience, reliability, and feature-set offered by Catalyst.
-If you have any suggestions, feedback, or general improvements regarding any of these topics, please let us know!
-
-Hello World,
-
-The Enigma Team
-
-.. |version status| image:: https://img.shields.io/pypi/pyversions/enigma-catalyst.svg
-   :target: https://pypi.python.org/pypi/enigma-catalyst
+All the documentation for `Catalyst <https://github.com/enigmampc/catalyst>`_ can be found in the `catalyst-docs wiki <https://github.com/enigmampc/catalyst-docs/wiki>`_.

catalyst/__main__.py

@@ -8,6 +8,8 @@ import pandas as pd
 from six import text_type
 
 from catalyst.data import bundles as bundles_module
+from catalyst.exchange.exchange_bundle import ExchangeBundle
+from catalyst.exchange.init_utils import get_exchange
 from catalyst.utils.cli import Date, Timestamp
 from catalyst.utils.run_algo import _run, load_extensions
 
@@ -28,9 +30,9 @@ except NameError:
     '--strict-extensions/--non-strict-extensions',
     is_flag=True,
     help='If --strict-extensions is passed then catalyst will not run if it'
-    ' cannot load all of the specified extensions. If this is not passed or'
-    ' --non-strict-extensions is passed then the failure will be logged but'
-    ' execution will continue.',
+         ' cannot load all of the specified extensions. If this is not passed or'
+         ' --non-strict-extensions is passed then the failure will be logged but'
+         ' execution will continue.',
 )
 @click.option(
     '--default-extension/--no-default-extension',
@@ -38,6 +40,7 @@ except NameError:
     default=True,
     help="Don't load the default catalyst extension.py file in $ZIPLINE_HOME.",
 )
+@click.version_option()
 def main(extension, strict_extensions, default_extension):
     """Top level catalyst entry point.
     """
@@ -64,6 +67,7 @@ def extract_option_object(option):
     option_object : click.Option
         The option object that this decorator will create.
     """
+
     @option
     def opt():
         pass
@@ -95,7 +99,9 @@ def ipython_only(option):
         def _(*args, **kwargs):
             kwargs[argname] = None
             return f(*args, **kwargs)
+
         return _
+
     return d
 
 
@@ -117,9 +123,9 @@ def ipython_only(option):
     '--define',
     multiple=True,
     help="Define a name to be bound in the namespace before executing"
-    " the algotext. For example '-Dname=value'. The value may be any python"
-    " expression. These are evaluated in order so they may refer to previously"
-    " defined names.",
+         " the algotext. For example '-Dname=value'. The value may be any python"
+         " expression. These are evaluated in order so they may refer to previously"
+         " defined names.",
 )
 @click.option(
     '--data-frequency',
@@ -149,7 +155,7 @@ def ipython_only(option):
     default=pd.Timestamp.utcnow(),
     show_default=False,
     help='The date to lookup data on or before.\n'
-    '[default: <current-time>]'
+         '[default: <current-time>]'
 )
 @click.option(
     '-s',
@@ -170,7 +176,7 @@ def ipython_only(option):
     metavar='FILENAME',
     show_default=True,
     help="The location to write the perf data. If this is '-' the perf will"
-    " be written to stdout.",
+         " be written to stdout.",
 )
 @click.option(
     '--print-algo/--no-print-algo',
@@ -184,6 +190,23 @@ def ipython_only(option):
     default=None,
     help='Should the algorithm methods be resolved in the local namespace.'
 ))
+@click.option(
+    '-x',
+    '--exchange-name',
+    type=click.Choice({'bitfinex', 'bittrex', 'poloniex'}),
+    help='The name of the targeted exchange (supported: bitfinex, bittrex, poloniex).',
+)
+@click.option(
+    '-n',
+    '--algo-namespace',
+    help='A label assigned to the algorithm for data storage purposes.'
+)
+@click.option(
+    '-c',
+    '--base-currency',
+    help='The base currency used to calculate statistics '
+         '(e.g. usd, btc, eth).',
+)
 @click.pass_context
 def run(ctx,
         algofile,
@@ -197,9 +220,19 @@ def run(ctx,
         end,
         output,
         print_algo,
-        local_namespace):
+        local_namespace,
+        exchange_name,
+        algo_namespace,
+        base_currency):
     """Run a backtest for the given algorithm.
     """
+
+    if (algotext is not None) == (algofile is not None):
+        ctx.fail(
+            "must specify exactly one of '-f' / '--algofile' or"
+            " '-t' / '--algotext'",
+        )
+
     # check that the start and end dates are passed correctly
     if start is None and end is None:
         # check both at the same time to avoid the case where a user
@@ -213,11 +246,8 @@ def run(ctx,
     if end is None:
         ctx.fail("must specify an end date with '-e' / '--end'")
 
-    if (algotext is not None) == (algofile is not None):
-        ctx.fail(
-            "must specify exactly one of '-f' / '--algofile' or"
-            " '-t' / '--algotext'",
-        )
+    if exchange_name is None:
+        ctx.fail("must specify an exchange name '-x'")
 
     perf = _run(
         initialize=None,
@@ -238,6 +268,11 @@ def run(ctx,
         print_algo=print_algo,
         local_namespace=local_namespace,
         environ=os.environ,
+        live=False,
+        exchange=exchange_name,
+        algo_namespace=algo_namespace,
+        base_currency=base_currency,
+        live_graph=False
     )
 
     if output == '-':
@@ -265,11 +300,11 @@ def catalyst_magic(line, cell=None):
                 '--algotext', cell,
                 '--output', os.devnull,  # don't write the results by default
             ] + ([
-                # these options are set when running in line magic mode
-                # set a non None algo text to use the ipython user_ns
-                '--algotext', '',
-                '--local-namespace',
-            ] if cell is None else []) + line.split(),
+                     # these options are set when running in line magic mode
+                     # set a non None algo text to use the ipython user_ns
+                     '--algotext', '',
+                     '--local-namespace',
+                 ] if cell is None else []) + line.split(),
             '%s%%catalyst' % ((cell or '') and '%'),
             # don't use system exit and propogate errors to the caller
             standalone_mode=False,
@@ -281,15 +316,222 @@ def catalyst_magic(line, cell=None):
             raise ValueError('main returned non-zero status code: %d' % e.code)
 
 
+@main.command()
+@click.option(
+    '-f',
+    '--algofile',
+    default=None,
+    type=click.File('r'),
+    help='The file that contains the algorithm to run.',
+)
+@click.option(
+    '-t',
+    '--algotext',
+    help='The algorithm script to run.',
+)
+@click.option(
+    '-D',
+    '--define',
+    multiple=True,
+    help="Define a name to be bound in the namespace before executing"
+         " the algotext. For example '-Dname=value'. The value may be any python"
+         " expression. These are evaluated in order so they may refer to previously"
+         " defined names.",
+)
+@click.option(
+    '-o',
+    '--output',
+    default='-',
+    metavar='FILENAME',
+    show_default=True,
+    help="The location to write the perf data. If this is '-' the perf will"
+         " be written to stdout.",
+)
+@click.option(
+    '--print-algo/--no-print-algo',
+    is_flag=True,
+    default=False,
+    help='Print the algorithm to stdout.',
+)
+@ipython_only(click.option(
+    '--local-namespace/--no-local-namespace',
+    is_flag=True,
+    default=None,
+    help='Should the algorithm methods be resolved in the local namespace.'
+))
+@click.option(
+    '-x',
+    '--exchange-name',
+    type=click.Choice({'bitfinex', 'bittrex', 'poloniex'}),
+    help='The name of the targeted exchange (supported: bitfinex, bittrex, poloniex).',
+)
+@click.option(
+    '-n',
+    '--algo-namespace',
+    help='A label assigned to the algorithm for data storage purposes.'
+)
+@click.option(
+    '-c',
+    '--base-currency',
+    help='The base currency used to calculate statistics '
+         '(e.g. usd, btc, eth).',
+)
+@click.option(
+    '--live-graph/--no-live-graph',
+    is_flag=True,
+    default=False,
+    help='Display live graph.',
+)
+@click.pass_context
+def live(ctx,
+         algofile,
+         algotext,
+         define,
+         output,
+         print_algo,
+         local_namespace,
+         exchange_name,
+         algo_namespace,
+         base_currency,
+         live_graph):
+    """Trade live with the given algorithm.
+    """
+    if (algotext is not None) == (algofile is not None):
+        ctx.fail(
+            "must specify exactly one of '-f' / '--algofile' or"
+            " '-t' / '--algotext'",
+        )
+
+    if exchange_name is None:
+        ctx.fail("must specify an exchange name '-x'")
+    if algo_namespace is None:
+        ctx.fail("must specify an algorithm name '-n' in live execution mode")
+    if base_currency is None:
+        ctx.fail("must specify a base currency '-c' in live execution mode")
+
+    perf = _run(
+        initialize=None,
+        handle_data=None,
+        before_trading_start=None,
+        analyze=None,
+        algofile=algofile,
+        algotext=algotext,
+        defines=define,
+        data_frequency=None,
+        capital_base=None,
+        data=None,
+        bundle=None,
+        bundle_timestamp=None,
+        start=None,
+        end=None,
+        output=output,
+        print_algo=print_algo,
+        local_namespace=local_namespace,
+        environ=os.environ,
+        live=True,
+        exchange=exchange_name,
+        algo_namespace=algo_namespace,
+        base_currency=base_currency,
+        live_graph=live_graph
+    )
+
+    if output == '-':
+        click.echo(str(perf))
+    elif output != os.devnull:  # make the catalyst magic not write any data
+        perf.to_pickle(output)
+
+    return perf
+
+
+@main.command(name='ingest-exchange')
+@click.option(
+    '-x',
+    '--exchange-name',
+    type=click.Choice({'bitfinex', 'bittrex', 'poloniex'}),
+    help='The name of the exchange bundle to ingest (supported: bitfinex,'
+         ' bittrex, poloniex).',
+)
+@click.option(
+    '-f',
+    '--data-frequency',
+    type=click.Choice({'daily', 'minute', 'daily,minute', 'minute,daily'}),
+    default='daily',
+    show_default=True,
+    help='The data frequency of the desired OHLCV bars.',
+)
+@click.option(
+    '-s',
+    '--start',
+    default=None,
+    type=Date(tz='utc', as_timestamp=True),
+    help='The start date of the data range. (default: one year from end date)',
+)
+@click.option(
+    '-e',
+    '--end',
+    default=None,
+    type=Date(tz='utc', as_timestamp=True),
+    help='The end date of the data range. (default: today)',
+)
+@click.option(
+    '-i',
+    '--include-symbols',
+    default=None,
+    help='A list of symbols to ingest (optional comma separated list)',
+)
+@click.option(
+    '--exclude-symbols',
+    default=None,
+    help='A list of symbols to exclude from the ingestion '
+         '(optional comma separated list)',
+)
+@click.option(
+    '--show-progress/--no-show-progress',
+    default=True,
+    help='Print progress information to the terminal.'
+)
+def ingest_exchange(exchange_name, data_frequency, start, end,
+                    include_symbols, exclude_symbols, show_progress):
+    """
+    Ingest data for the given exchange.
+    """
+    exchange = get_exchange(exchange_name)
+    exchange_bundle = ExchangeBundle(exchange)
+
+    click.echo('Ingesting exchange bundle {}...'.format(exchange_name))
+    exchange_bundle.ingest(
+        data_frequency=data_frequency,
+        include_symbols=include_symbols,
+        exclude_symbols=exclude_symbols,
+        start=start,
+        end=end,
+        show_progress=show_progress
+    )
+
+
 @main.command()
 @click.option(
     '-b',
     '--bundle',
-    default='poloniex',
     metavar='BUNDLE-NAME',
-    show_default=True,
+    default=None,
+    show_default=False,
     help='The data bundle to ingest.',
 )
+@click.option(
+    '-x',
+    '--exchange-name',
+    type=click.Choice({'bitfinex', 'bittrex', 'poloniex'}),
+    help='The name of the exchange bundle to ingest (supported: bitfinex,'
+         ' bittrex, poloniex).',
+)
+@click.option(
+    '-c',
+    '--compile-locally',
+    is_flag=True,
+    default=False,
+    help='Download dataset from source and compile bundle locally.',
+)
 @click.option(
     '--assets-version',
     type=int,
@@ -301,15 +543,19 @@ def catalyst_magic(line, cell=None):
     default=True,
     help='Print progress information to the terminal.'
 )
-def ingest(bundle, assets_version, show_progress):
+@click.pass_context
+def ingest(ctx, bundle, exchange_name, compile_locally, assets_version,
+           show_progress):
     """Ingest the data for the given bundle.
     """
+
     bundles_module.ingest(
         bundle,
         os.environ,
         pd.Timestamp.utcnow(),
         assets_version,
         show_progress,
+        compile_locally,
     )
 
 
@@ -322,19 +568,26 @@ def ingest(bundle, assets_version, show_progress):
     show_default=True,
     help='The data bundle to clean.',
 )
+@click.option(
+    '-x',
+    '--exchange_name',
+    metavar='EXCHANGE-NAME',
+    show_default=True,
+    help='The exchange bundle name to clean.',
+)
 @click.option(
     '-e',
     '--before',
     type=Timestamp(),
     help='Clear all data before TIMESTAMP.'
-    ' This may not be passed with -k / --keep-last',
+         ' This may not be passed with -k / --keep-last',
 )
 @click.option(
     '-a',
     '--after',
     type=Timestamp(),
     help='Clear all data after TIMESTAMP'
-    ' This may not be passed with -k / --keep-last',
+         ' This may not be passed with -k / --keep-last',
 )
 @click.option(
     '-k',
@@ -342,7 +595,7 @@ def ingest(bundle, assets_version, show_progress):
     type=int,
     metavar='N',
     help='Clear all but the last N downloads.'
-    ' This may not be passed with -e / --before or -a / --after',
+         ' This may not be passed with -e / --before or -a / --after',
 )
 def clean(bundle, before, after, keep_last):
     """Clean up data downloaded with the ingest command.

catalyst/algorithm.py

@@ -125,6 +125,7 @@ from catalyst.utils.factory import create_simulation_parameters
 from catalyst.utils.math_utils import (
     tolerant_equals,
     round_if_near_integer,
+    round_nearest
 )
 from catalyst.utils.pandas_utils import clear_dataframe_indexer_caches
 from catalyst.utils.preprocess import preprocess
@@ -137,8 +138,9 @@ from catalyst.gens.sim_engine import MinuteSimulationClock
 from catalyst.sources.benchmark_source import BenchmarkSource
 from catalyst.catalyst_warnings import ZiplineDeprecationWarning
 
+from catalyst.constants import LOG_LEVEL
 
-log = logbook.Logger("ZiplineLog")
+log = logbook.Logger("CatalystLog", level=LOG_LEVEL)
 
 
 class TradingAlgorithm(object):
@@ -305,7 +307,10 @@ class TradingAlgorithm(object):
         self.asset_finder = self.trading_environment.asset_finder
 
         # Initialize Pipeline API data.
-        self.init_engine(kwargs.pop('get_pipeline_loader', None))
+        self.init_engine(
+            kwargs.pop('get_pipeline_loader', None),
+            self.sim_params.data_frequency,
+        )
         self._pipelines = {}
         # Create an always-expired cache so that we compute the first time data
         # is requested.
@@ -419,16 +424,26 @@ class TradingAlgorithm(object):
 
         self.restrictions = NoRestrictions()
 
-    def init_engine(self, get_loader):
+    def init_engine(self, get_loader, data_frequency):
         """
         Construct and store a PipelineEngine from loader.
 
         If get_loader is None, constructs an ExplodingPipelineEngine
         """
         if get_loader is not None:
+            if data_frequency == 'daily':
+                all_dates = self.trading_calendar.all_sessions
+            elif data_frequency == 'minute':
+                all_dates = self.trading_calendar.all_minutes
+            else:
+                raise ValueError(
+                    'Cannot initialize engine with '
+                    'data frequency: {}'.format(data_frequency)
+                )
+
             self.engine = SimplePipelineEngine(
                 get_loader,
-                self.trading_calendar.all_sessions,
+                all_dates,
                 self.asset_finder,
             )
         else:
@@ -449,7 +464,7 @@ class TradingAlgorithm(object):
         self._in_before_trading_start = True
 
         with handle_non_market_minutes(data) if \
-                self.data_frequency == "minute" else ExitStack():
+                self.data_frequency == 'minute' else ExitStack():
             self._before_trading_start(self, data)
 
         self._in_before_trading_start = False
@@ -508,7 +523,7 @@ class TradingAlgorithm(object):
         if self.sim_params.data_frequency == 'minute':
             market_opens = trading_o_and_c['market_open']
 
-            minutely_emission = self.sim_params.emission_rate == "minute"
+            minutely_emission = self.sim_params.emission_rate == 'minute'
         else:
             # in daily mode, we want to have one bar per session, timestamped
             # as the last minute of the session.
@@ -528,8 +543,8 @@ class TradingAlgorithm(object):
         # FIXME generalize these values
         before_trading_start_minutes = days_at_time(
             self.sim_params.sessions,
-            time(8, 45),
-            "US/Eastern"
+            time(0, 0),
+            'UTC',
         )
 
         return MinuteSimulationClock(
@@ -660,7 +675,8 @@ class TradingAlgorithm(object):
                     # Assume data is daily if timestamp times are
                     # standardized, otherwise assume minute bars.
                     times = data.major_axis.time
-                    if np.all(times == times[0]):
+                    time_count = times.nunique()
+                    if time_count == 1:
                         self.sim_params.data_frequency = 'daily'
                     else:
                         self.sim_params.data_frequency = 'minute'
@@ -928,7 +944,7 @@ class TradingAlgorithm(object):
                   live trading from backtesting.
               data_frequency : {'daily', 'minute'}
                   data_frequency tells the algorithm if it is running with
-                  daily data or minute data.
+                  daily or minute mode.
               start : datetime
                   The start date for the simulation.
               end : datetime
@@ -1448,7 +1464,7 @@ class TradingAlgorithm(object):
 
     def _calculate_order(self, asset, amount,
                          limit_price=None, stop_price=None, style=None):
-        amount = self.round_order(amount)
+        amount = self.round_order(amount, asset)
 
         # Raises a ZiplineError if invalid parameters are detected.
         self.validate_order_params(asset,
@@ -1465,16 +1481,13 @@ class TradingAlgorithm(object):
         return amount, style
 
     @staticmethod
-    def round_order(amount):
+    def round_order(amount, asset):
         """
-        Convert number of shares to an integer.
-
-        By default, truncates to the integer share count that's either within
-        .0001 of amount or closer to zero.
-
-        E.g. 3.9999 -> 4.0; 5.5 -> 5.0; -5.5 -> -5.0
+        Converts the number of shares to the smallest tradable lot size for
+        the asset being ordered.
+        
         """
-        return int(round_if_near_integer(amount))
+        return round_nearest(amount, asset.min_trade_size)
 
     def validate_order_params(self,
                               asset,
@@ -1510,7 +1523,6 @@ class TradingAlgorithm(object):
                              self.updated_portfolio(),
                              self.get_datetime(),
                              self.trading_client.current_data)
-
     @staticmethod
     def __convert_order_params_for_blotter(limit_price, stop_price, style):
         """

catalyst/assets/_assets.pyx

@@ -20,31 +20,31 @@ Cythonized Asset object.
 cimport cython
 from cpython.number cimport PyNumber_Index
 from cpython.object cimport (
-    Py_EQ,
-    Py_NE,
-    Py_GE,
-    Py_LE,
-    Py_GT,
-    Py_LT,
+Py_EQ,
+Py_NE,
+Py_GE,
+Py_LE,
+Py_GT,
+Py_LT,
 )
 from cpython cimport bool
 
+import pandas as pd
+from datetime import timedelta
 import numpy as np
 from numpy cimport int64_t
 import warnings
 cimport numpy as np
 
 from catalyst.utils.calendars import get_calendar
-
+from catalyst.exchange.exchange_errors import InvalidSymbolError, SidHashError
 
 # IMPORTANT NOTE: You must change this template if you change
 # Asset.__reduce__, or else we'll attempt to unpickle an old version of this
 # class
 CACHE_FILE_TEMPLATE = '/tmp/.%s-%s.v7.cache'
 
-
 cdef class Asset:
-
     cdef readonly int sid
     # Cached hash of self.sid
     cdef int sid_hash
@@ -59,6 +59,7 @@ cdef class Asset:
 
     cdef readonly object exchange
     cdef readonly object exchange_full
+    cdef readonly object min_trade_size
 
     _kwargnames = frozenset({
         'sid',
@@ -70,18 +71,20 @@ cdef class Asset:
         'auto_close_date',
         'exchange',
         'exchange_full',
+        'min_trade_size',
     })
 
     def __init__(self,
-                 int sid, # sid is required
-                 object exchange, # exchange is required
+                 int sid,  # sid is required
+                 object exchange,  # exchange is required
                  object symbol="",
                  object asset_name="",
                  object start_date=None,
                  object end_date=None,
                  object first_traded=None,
                  object auto_close_date=None,
-                 object exchange_full=None):
+                 object exchange_full=None,
+                 object min_trade_size=None):
 
         self.sid = sid
         self.sid_hash = hash(sid)
@@ -94,6 +97,7 @@ cdef class Asset:
         self.end_date = end_date
         self.first_traded = first_traded
         self.auto_close_date = auto_close_date
+        self.min_trade_size = min_trade_size
 
     def __int__(self):
         return self.sid
@@ -148,7 +152,8 @@ cdef class Asset:
 
     def __repr__(self):
         attrs = ('symbol', 'asset_name', 'exchange',
-                 'start_date', 'end_date', 'first_traded', 'auto_close_date')
+                 'start_date', 'end_date', 'first_traded', 'auto_close_date',
+                 'min_trade_size')
         tuples = ((attr, repr(getattr(self, attr, None)))
                   for attr in attrs)
         strings = ('%s=%s' % (t[0], t[1]) for t in tuples)
@@ -170,7 +175,8 @@ cdef class Asset:
                                  self.end_date,
                                  self.first_traded,
                                  self.auto_close_date,
-                                 self.exchange_full))
+                                 self.exchange_full,
+                                 self.min_trade_size))
 
     cpdef to_dict(self):
         """
@@ -186,6 +192,7 @@ cdef class Asset:
             'auto_close_date': self.auto_close_date,
             'exchange': self.exchange,
             'exchange_full': self.exchange_full,
+            'min_trade_size': self.min_trade_size
         }
 
     @classmethod
@@ -230,13 +237,11 @@ cdef class Asset:
         calendar = get_calendar(self.exchange)
         return calendar.is_open_on_minute(dt_minute)
 
-
 cdef class Equity(Asset):
-
     def __repr__(self):
         attrs = ('symbol', 'asset_name', 'exchange',
                  'start_date', 'end_date', 'first_traded', 'auto_close_date',
-                 'exchange_full')
+                 'exchange_full', 'min_trade_size')
         tuples = ((attr, repr(getattr(self, attr, None)))
                   for attr in attrs)
         strings = ('%s=%s' % (t[0], t[1]) for t in tuples)
@@ -250,8 +255,8 @@ cdef class Equity(Asset):
         """
         def __get__(self):
             warnings.warn("The security_start_date property will soon be "
-            "retired. Please use the start_date property instead.",
-            DeprecationWarning)
+                          "retired. Please use the start_date property instead.",
+                          DeprecationWarning)
             return self.start_date
 
     property security_end_date:
@@ -261,8 +266,8 @@ cdef class Equity(Asset):
         """
         def __get__(self):
             warnings.warn("The security_end_date property will soon be "
-            "retired. Please use the end_date property instead.",
-            DeprecationWarning)
+                          "retired. Please use the end_date property instead.",
+                          DeprecationWarning)
             return self.end_date
 
     property security_name:
@@ -272,13 +277,11 @@ cdef class Equity(Asset):
         """
         def __get__(self):
             warnings.warn("The security_name property will soon be "
-            "retired. Please use the asset_name property instead.",
-            DeprecationWarning)
+                          "retired. Please use the asset_name property instead.",
+                          DeprecationWarning)
             return self.asset_name
 
-
 cdef class Future(Asset):
-
     cdef readonly object root_symbol
     cdef readonly object notice_date
     cdef readonly object expiration_date
@@ -303,8 +306,8 @@ cdef class Future(Asset):
     })
 
     def __init__(self,
-                 int sid, # sid is required
-                 object exchange, # exchange is required
+                 int sid,  # sid is required
+                 object exchange,  # exchange is required
                  object symbol="",
                  object root_symbol="",
                  object asset_name="",
@@ -388,6 +391,186 @@ cdef class Future(Asset):
         super_dict['multiplier'] = self.multiplier
         return super_dict
 
+cdef class TradingPair(Asset):
+    cdef readonly float leverage
+    cdef readonly object market_currency
+    cdef readonly object base_currency
+    cdef readonly object end_daily
+    cdef readonly object end_minute
+    cdef readonly object exchange_symbol
+
+    _kwargnames = frozenset({
+        'sid',
+        'symbol',
+        'asset_name',
+        'start_date',
+        'end_date',
+        'first_traded',
+        'auto_close_date',
+        'exchange',
+        'exchange_full',
+        'leverage',
+        'market_currency',
+        'base_currency',
+        'end_daily',
+        'end_minute',
+        'exchange_symbol',
+        'min_trade_size'
+    })
+    def __init__(self,
+                 object symbol,
+                 object exchange,
+                 object start_date=None,
+                 object asset_name=None,
+                 int sid=0,
+                 float leverage=1.0,
+                 object end_daily=None,
+                 object end_minute=None,
+                 object end_date=None,
+                 object exchange_symbol=None,
+                 object first_traded=None,
+                 object auto_close_date=None,
+                 object exchange_full=None,
+                 object min_trade_size=None):
+        """
+        Replicates the Asset constructor with some built-in conventions
+        and a new 'leverage' attribute.
+
+        Symbol
+        ------
+        Catalyst defines its own set of "universal" symbols to reference
+        trading pairs across exchanges. This is required because exchanges
+        are not adhering to a universal symbolism. For example, Bitfinex
+        uses the BTC symbol for Bitcon while Kraken uses XBT. In addition,
+        pairs are sometimes presented differently. For example, Bitfinex
+        puts the market currency before the base currency without a
+        separator, Bittrex puts the base currency first and uses a dash
+        seperator.
+
+        Here is the Catalyst convention: [Market Currency]_[Base Currency]
+        For example: btc_usd, eth_btc, neo_eth, ltc_eur.
+
+        The symbol for each currency (e.g. btc, eth, ltc) is generally
+        aligned with the Bittrex exchange.
+
+        Sid
+        ---
+        The sid of each asset is calculated based on a numeric hash of the
+        universal symbol. This simple approach avoids maintaining a mapping
+        of sids.
+
+        Leverage
+        --------
+        In contrast with equities, crypto exchanges generally assign
+        leverage values to specific trading pairs. Pairs with the
+        highest volume and market cap generally benefit from high leverage.
+        New currencies from ICO generally cannot be leveraged.
+
+        The leverage value is either None or and integer.
+
+        Leverage allows you to open a larger position with a smaller amount
+        of funds. For example, if you open a $5,000 position in BTC/USD
+        with 5:1 leverage, only one-fifth of this amount, or $1000, will be
+        tied to the position from your balance. Your remaining balance will
+        be available for opening more positions. If you open this same
+        position with 2:1 leverage, $2,500 of your balance will be tied to
+        the position. If you open with 1:1 leverage, $5,000 of your balance
+        will be tied to the position.
+
+        :param symbol:
+        :param exchange:
+        :param start_date:
+        :param asset_name:
+        :param sid:
+        :param leverage:
+        :param end_daily
+        :param end_minute
+        :param end_date:
+        :param exchange_symbol:
+        :param first_traded:
+        :param auto_close_date:
+        :param exchange_full:
+        :param min_trade_size:
+        """
+
+        symbol = symbol.lower()
+        try:
+            self.market_currency, self.base_currency = symbol.split('_')
+        except Exception as e:
+            raise InvalidSymbolError(symbol=symbol, error=e)
+
+        if sid == 0 or sid is None:
+            try:
+                sid = abs(hash(symbol)) % (10 ** 4)
+            except Exception as e:
+                raise SidHashError(symbol=symbol)
+
+        if asset_name is None:
+            asset_name = ' / '.join(symbol.split('_')).upper()
+
+        if start_date is None:
+            start_date = pd.Timestamp.utcnow()
+
+        if end_date is None:
+            end_date = pd.Timestamp.utcnow() + timedelta(days=365)
+
+        super().__init__(
+            sid,
+            exchange,
+            symbol=symbol,
+            asset_name=asset_name,
+            start_date=start_date,
+            end_date=end_date,
+            first_traded=first_traded,
+            auto_close_date=auto_close_date,
+            exchange_full=exchange_full,
+            min_trade_size=min_trade_size
+        )
+
+        self.leverage = leverage
+        self.end_daily = end_daily
+        self.end_minute = end_minute
+        self.exchange_symbol = exchange_symbol
+
+    def __repr__(self):
+        return 'Trading Pair {symbol}({sid}) Exchange: {exchange}, ' \
+               'Introduced On: {start_date}, ' \
+               'Market Currency: {market_currency}, ' \
+               'Base Currency: {base_currency}, ' \
+               'Exchange Leverage: {leverage}, ' \
+               'Minimum Trade Size: {min_trade_size} ' \
+               'Last daily ingestion: {end_daily} ' \
+               'Last minutely ingestion: {end_minute}'.format(
+            symbol=self.symbol,
+            sid=self.sid,
+            exchange=self.exchange,
+            start_date=self.start_date,
+            market_currency=self.market_currency,
+            base_currency=self.base_currency,
+            leverage=self.leverage,
+            min_trade_size=self.min_trade_size,
+            end_daily=self.end_daily,
+            end_minute=self.end_minute
+        )
+
+    cpdef __reduce__(self):
+        """
+        Function used by pickle to determine how to serialize/deserialize this
+        class.  Should return a tuple whose first element is self.__class__,
+        and whose second element is a tuple of all the attributes that should
+        be serialized/deserialized during pickling.
+        """
+        return (self.__class__, (self.symbol,
+                                 self.exchange,
+                                 self.start_date,
+                                 self.asset_name,
+                                 self.sid,
+                                 self.leverage,
+                                 self.end_date,
+                                 self.first_traded,
+                                 self.auto_close_date,
+                                 self.exchange_full,
+                                 self.min_trade_size))
 
 def make_asset_array(int size, Asset asset):
     cdef np.ndarray out = np.empty([size], dtype=object)

catalyst/assets/asset_db_schema.py

@@ -39,7 +39,8 @@ equities = sa.Table(
     sa.Column('first_traded', sa.Integer),
     sa.Column('auto_close_date', sa.Integer),
     sa.Column('exchange', sa.Text),
-    sa.Column('exchange_full', sa.Text)
+    sa.Column('exchange_full', sa.Text),
+    sa.Column('min_trade_size', sa.Float)
 )
 
 equity_symbol_mappings = sa.Table(

catalyst/assets/asset_writer.py

@@ -73,6 +73,7 @@ _equities_defaults = {
     'exchange': None,
     # optional, something like "New York Stock Exchange"
     'exchange_full': None,
+    'min_trade_size': 1
 }
 
 # Default values for the futures DataFrame
@@ -390,6 +391,8 @@ class AssetDBWriter(object):
                   The date on which to close any positions in this asset.
               exchange : str
                   The exchange where this asset is traded.
+              min_trade_size: float, optional
+                  The minimum denomination this asset can be traded.
 
             The index of this dataframe should contain the sids.
         futures : pd.DataFrame, optional

catalyst/assets/assets.py

@@ -76,7 +76,9 @@ from catalyst.utils.numpy_utils import as_column
 from catalyst.utils.preprocess import preprocess
 from catalyst.utils.sqlite_utils import group_into_chunks, coerce_string_to_eng
 
-log = Logger('assets.py')
+from catalyst.constants import LOG_LEVEL
+
+log = Logger('assets.py', level=LOG_LEVEL)
 
 # A set of fields that need to be converted to strings before building an
 # Asset to avoid unicode fields

catalyst/constants.py

@@ -0,0 +1,5 @@
+# -*- coding: utf-8 -*-
+
+import logbook
+
+LOG_LEVEL = logbook.INFO

catalyst/curate/poloniex.py

@@ -1,22 +1,23 @@
 import json, time, csv
 from datetime import datetime
 import pandas as pd
-import os
-import time
-import requests
-import logbook
+import os, time, shutil, requests, logbook
+from catalyst.exchange.exchange_utils import get_exchange_symbols_filename
 
-DT_START        = time.mktime(datetime(2010, 01, 01, 0, 0).timetuple())
+
+DT_START        = int(time.mktime(datetime(2010, 1, 1, 0, 0).timetuple()))
+DT_END          = int(time.time())
 CSV_OUT_FOLDER  = '/var/tmp/catalyst/data/poloniex/'
+CSV_OUT_FOLDER  = '/Volumes/enigma/data/poloniex/'
 CONN_RETRIES    = 2
 
 logbook.StderrHandler().push_application()
 log = logbook.Logger(__name__)
 
 class PoloniexCurator(object):
-    """
+    '''
     OHLCV data feed generator for crypto data. Based on Poloniex market data
-    """
+    '''
 
     _api_path       = 'https://poloniex.com/public?'
     currency_pairs  = []
@@ -29,6 +30,9 @@ class PoloniexCurator(object):
                 log.error('Failed to create data folder: %s' % CSV_OUT_FOLDER)
                 log.exception(e)
 
+    '''
+        Retrieves and returns all currency pairs from the exchange 
+    '''
     def get_currency_pairs(self):
         url = self._api_path + 'command=returnTicker'
 
@@ -47,98 +51,244 @@ class PoloniexCurator(object):
 
         log.debug('Currency pairs retrieved successfully: %d' % (len(self.currency_pairs)))
 
-    def _get_start_date(self, csv_fn):
-        ''' Function returns latest appended date, if the file has been previously written
-            the last line is an empty one, so we have to read the second to last line 
+
+    '''
+        Helper function that reads tradeID and date fields from CSV readline
+    '''
+    def _retrieve_tradeID_date(self, row):
+        tId = int(row.split(',')[0])
+        d   = pd.to_datetime( row.split(',')[1], infer_datetime_format=True).value // 10 ** 9
+        return tId, d
+
+    '''
+        Retrieves TradeHistory from exchange for a given currencyPair between start and end dates.
+        If no start date is provided, uses a system-wide one (beginning of time for cryptotrading)
+        If no end date is provided, 'now' is used
+        Stores results in CSV file on disk.
+        This function is called recursively to work around the limitations imposed by the provider API.
+    '''
+    def retrieve_trade_history(self, currencyPair, start=DT_START, end=DT_END, temp=None):
+        csv_fn = CSV_OUT_FOLDER + 'crypto_trades-' + currencyPair + '.csv'
+
+        '''
+            Check what data we already have on disk, reading first and last lines from file.
+            Data is stored on file from NEWEST to OLDEST.
         '''
         try:
             with open(csv_fn, 'ab+') as f: 
-                f.seek(0, os.SEEK_END)              # First check file is not zero size
-                if(f.tell() > 2):
-                    f.seek(-2, os.SEEK_END)         # Jump to the second last byte.
-                    while f.read(1) != b"\n":       # Until EOL is found...
-                        f.seek(-2, os.SEEK_CUR)     # ...jump back the read byte plus one more.
-                    lastrow = f.readline() 
-                    return int(lastrow.split(',')[0]) + 300
+                f.seek(0, os.SEEK_END)
+                if(f.tell() > 2):                           # First check file is not zero size
+                    f.seek(0)                               # Go to the beginning to read first line
+                    last_tradeID, end_file    = self._retrieve_tradeID_date(f.readline())
+                    f.seek(-2, os.SEEK_END)                 # Jump to the second last byte.
+                    while f.read(1) != b"\n":               # Until EOL is found...
+                        f.seek(-2, os.SEEK_CUR)             # ...jump back the read byte plus one more.
+                    first_tradeID, start_file = self._retrieve_tradeID_date(f.readline())
+
+                    if( first_tradeID == 1 and end_file + 3600 > DT_END ):
+                        return
 
         except Exception as e:
             log.error('Error opening file: %s' % csv_fn)
             log.exception(e)
 
-        return DT_START
+        '''
+        Poloniex API limits querying TradeHistory to intervals smaller than 1 month,
+        so we make sure that start date is never more than 1 month apart from end date
+        '''
+        if( end - start > 2419200 ):       # 60 s/min * 60 min/hr * 24 hr/day * 28 days     
+            newstart = end - 2419200
+        else:
+            newstart = start
+
+        log.debug(currencyPair+': Retrieving from '+str(newstart)+' to '+str(end) +'\t '
+                    + time.ctime(newstart) + ' - '+ time.ctime(end))
+
+        url = self._api_path + 'command=returnTradeHistory&currencyPair=' + currencyPair + '&start=' + str(newstart) + '&end=' + str(end)
 
-    def get_data(self, currencyPair, start, end=9999999999, period=300):
-    	url = self._api_path + 'command=returnChartData&currencyPair=' + currencyPair + '&start=' + str(start) + '&end=' + str(end) + '&period=' + str(period)
-        
         try:
             response = requests.get(url)
         except Exception as e:
-            log.error('Failed to retrieve candlestick chart data for %s' % currencyPair)
+            log.error('Failed to retrieve trade history data for %s' % currencyPair)
             log.exception(e)
             return None
+        else:
+            if isinstance(response.json(), dict) and response.json()['error']:
+                log.error('Failed to to retrieve trade history data for %s: %s' % (currencyPair,response.json()['error']))
+                exit(1)
+
+        '''
+            If we get to transactionId == 1, and we already have that on disk, 
+            we got to the end of TradeHistory for this coin.
+        '''
+        if('first_tradeID' in locals() and response.json()[-1]['tradeID'] == first_tradeID): 
+            return
+
+        '''
+            There are primarily two scenarios:
+                a) There is newer data available that we need to add at the beginning
+                   of the file. We'll retrieve all what we need until we get to what 
+                   we already have, writing it to a temporary file; and we will write
+                   that at the beginning of our existing file.
+                b) We are going back in time, appending at the end of our existing
+                   TradeHistory until the first transaction for this currencyPair
+        '''
+        try: 
+            if( 'end_file' in locals() and end_file + 3600 < end):
+                if (temp is None):
+                    temp = os.tmpfile()
+                tempcsv = csv.writer(temp)
+                for item in response.json():
+                    if( item['tradeID'] <= last_tradeID ):
+                        continue
+                    tempcsv.writerow([
+                        item['tradeID'],
+                        item['date'],
+                        item['type'],
+                        item['rate'],
+                        item['amount'],
+                        item['total'],
+                        item['globalTradeID']        
+                    ])
+                if( response.json()[-1]['tradeID'] > last_tradeID ):
+                    end = pd.to_datetime( response.json()[-1]['date'], infer_datetime_format=True).value // 10 ** 9
+                    self.retrieve_trade_history(currencyPair, start, end, temp=temp) 
+                else:
+                    with open(csv_fn,'rb+') as f:
+                        shutil.copyfileobj(f,temp)
+                        f.seek(0)
+                        temp.seek(0)
+                        shutil.copyfileobj(temp,f)
+                    temp.close()
+                    end = start_file
+            else:
+                with open(csv_fn, 'ab') as csvfile:
+                    csvwriter = csv.writer(csvfile)
+                    for item in response.json():
+                        if( 'first_tradeID' in locals() and item['tradeID'] >= first_tradeID ):
+                            continue
+                        csvwriter.writerow([
+                            item['tradeID'],
+                            item['date'],
+                            item['type'],
+                            item['rate'],
+                            item['amount'],
+                            item['total'],
+                            item['globalTradeID']
+                        ])
+                end = pd.to_datetime( response.json()[-1]['date'], infer_datetime_format=True).value // 10 ** 9
+
+        except Exception as e:
+            log.error('Error opening %s' % csv_fn)
+            log.exception(e)
+
+        '''
+            If we got here, we aren't done yet. Call recursively with 'end' times
+            that go sequentially back in time.
+        '''
+        self.retrieve_trade_history(currencyPair, start, end)
 
-    	return response.json()
 
     '''
-    Pulls latest data for a single pair
+        Generates OHLCV dataframe from a dataframe containing all TradeHistory
+        by resampling with 1-minute period
     '''
-    def append_data_single_pair(self, currencyPair, repeat=0):
-    	log.debug('Getting data for %s' % currencyPair)
-        csv_fn = CSV_OUT_FOLDER + 'crypto_prices-' + currencyPair + '.csv'
-        start  = self._get_start_date(csv_fn)
-        # Only fetch data if more than 5min have passed since last fetch
-        if (time.time() > start):
-            data   = self.get_data(currencyPair, start)
-            if data is not None:
-                try: 
-                    with open(csv_fn, 'ab') as csvfile:
-                        csvwriter = csv.writer(csvfile)
-                        for item in data:
-                            if item['date'] == 0:
-                                continue
-                            csvwriter.writerow([
-                                item['date'],
-                                item['open'],
-                                item['high'],
-                                item['low'],
-                                item['close'],
-                                item['volume'],
-                            ])
-                except Exception as e:
-                    log.error('Error opening %s' % csv_fn)
-                    log.exception(e)
-            elif (repeat < CONN_RETRIES):
-                    log.debug('Retrying: attemt %d' % (repeat+1) )
-                    self.append_data_single_pair(currencyPair, repeat + 1)    
+    def generate_ohlcv(self, df):
+        df.set_index('date', inplace=True)                      # Index by date
+        vol = df['total'].to_frame('volume')                    # Will deal with vol separately, as ohlc() messes it up
+        df.drop('total', axis=1, inplace=True)                  # Drop volume data from dataframe
+        ohlc = df.resample('T').ohlc()                          # Resample OHLC in 1min bins
+        ohlc.columns = ohlc.columns.map(lambda t: t[1])         # Raname columns by dropping 'rate'
+        closes = ohlc['close'].fillna(method='pad')             # Pad forward missing 'close'
+        ohlc = ohlc.apply(lambda x: x.fillna(closes))           # Fill N/A with last close
+        vol = vol.resample('T').sum().fillna(0)                 # Add volumes by bin
+        ohlcv = pd.concat([ohlc,vol], axis=1)                   # Concatenate OHLC + Volume
+        return ohlcv
+
 
     '''
-    Pulls latest data for all currency pairs
+        Generates OHLCV data file with 1minute bars from TradeHistory on disk
     '''
-    def append_data(self):
-    	for currencyPair in self.currency_pairs:
-    		self.append_data_single_pair(currencyPair)
-        # Rate limit is 6 calls per second, sleep 1sec/6 to be safe
-    		time.sleep(0.17)
+    def write_ohlcv_file(self, currencyPair):        
+        csv_trades = CSV_OUT_FOLDER + 'crypto_trades-' + currencyPair + '.csv'
+        csv_1min   = CSV_OUT_FOLDER + 'crypto_1min-' + currencyPair + '.csv'
+        if( os.path.isfile(csv_1min) ):
+            log.debug(currencyPair+': 1min data already present. Delete the file if you want to rebuild it.')
+        else:
+            df = pd.read_csv(csv_trades, names=['tradeID','date','type','rate','amount','total','globalTradeID'], 
+                    dtype = {'tradeID': int, 'date': str, 'type': str, 'rate': float, 'amount': float, 'total': float, 'globalTradeID': int } )
+            df.drop(['tradeID','type','amount','globalTradeID'], axis=1, inplace=True)
+            df['date'] = pd.to_datetime(df['date'], infer_datetime_format=True) 
+            ohlcv = self.generate_ohlcv(df)
+            try: 
+                with open(csv_1min, 'ab') as csvfile:
+                    csvwriter = csv.writer(csvfile)
+                    for item in ohlcv.itertuples():
+                        if item.Index == 0:
+                            continue
+                        csvwriter.writerow([
+                            item.Index.value // 10 ** 9,
+                            item.open,
+                            item.high,
+                            item.low,
+                            item.close,
+                            item.volume,
+                        ])
+            except Exception as e:
+                log.error('Error opening %s' % csv_fn)
+                log.exception(e)
+            log.debug(currencyPair+': Generated 1min OHLCV data.')
+
 
     '''
-    Returns a data frame for all pairs, or for the requests currency pair.
-    Makes sure data is up to date
+    Returns a data frame for a given currencyPair from data on disk
     '''
-    def to_dataframe(self, start, end, currencyPair=None):
-        csv_fn = CSV_OUT_FOLDER + 'crypto_prices-' + currencyPair + '.csv'
-        last_date = self._get_start_date(csv_fn)
-        if last_date + 300 < end or not os.path.exists(csv_fn):
-            # get latest data
-            self.append_data_single_pair(currencyPair)
-
-        # CSV holds the latest snapshot
-        df = pd.read_csv(csv_fn, names=['date', 'open', 'high', 'low', 'close', 'volume'])
-        df['date']=pd.to_datetime(df['date'],unit='s')
+    def onemin_to_dataframe(self, currencyPair, start, end):
+        csv_fn     = CSV_OUT_FOLDER + 'crypto_1min-' + currencyPair + '.csv'
+        df         = pd.read_csv(csv_fn, names=['date', 'open', 'high', 'low', 'close', 'volume'])
+        df['date'] = pd.to_datetime(df['date'],unit='s')
         df.set_index('date', inplace=True)
+        return df[start : end]
+
+    '''
+    Generates a symbols.json file with corresponding start_date for each currencyPair
+    '''
+    def generate_symbols_json(self, filename=None):
+        symbol_map = {}
+
+        if(filename is None):
+            filename = get_exchange_symbols_filename('poloniex')
+
+        with open(filename, 'w') as symbols:
+            for currencyPair in self.currency_pairs:
+                start = None
+                csv_fn     = CSV_OUT_FOLDER + 'crypto_trades-' + currencyPair + '.csv'
+                with open(csv_fn, 'r') as f: 
+                    f.seek(0, os.SEEK_END)
+                    if(f.tell() > 2):                           # First check file is not zero size
+                        f.seek(-2, os.SEEK_END)                 # Jump to the second last byte.
+                        while f.read(1) != b"\n":               # Until EOL is found...
+                            f.seek(-2, os.SEEK_CUR)             # ...jump back the read byte plus one more.
+                        start = pd.to_datetime( f.readline().split(',')[1], infer_datetime_format=True)
+
+                if(start is None):
+                    start = time.gmtime()
+                base, market = currencyPair.lower().split('_')
+                symbol = '{market}_{base}'.format( market=market, base=base )
+                symbol_map[currencyPair] = dict(
+                    symbol = symbol,
+                    start_date = start.strftime("%Y-%m-%d")
+                )
+            json.dump(symbol_map, symbols, sort_keys=True, indent=2, separators=(',',':'))    
 
-        return df[datetime.fromtimestamp(start):datetime.fromtimestamp(end-1)]
 
 if __name__ == '__main__':
     pc = PoloniexCurator()
     pc.get_currency_pairs()
-    pc.append_data()
+    #pc.generate_symbols_json()
+    
+    for currencyPair in pc.currency_pairs:
+        pc.retrieve_trade_history(currencyPair)
+        pc.write_ohlcv_file(currencyPair)
+
+    

catalyst/data/_equities.pyx

@@ -215,13 +215,13 @@ cpdef _read_bcolz_data(ctable_t table,
                 else:
                     continue
 
-        if column_name in ['open', 'high', 'low', 'close']:
+        if column_name in ['open', 'high', 'low', 'close', 'volume']:
             where_nan = (outbuf == 0)
-            outbuf_as_float = outbuf.astype(float64) * .000001
+            outbuf_as_float = outbuf.astype(float64) * .000000001
             outbuf_as_float[where_nan] = NAN
             results.append(outbuf_as_float)
-        elif column_name != 'volume':
-            results.append(outbuf.astype(uint32))
+        elif column_name in ['volume']:
+            results.append(outbuf.astype(float64) * .000000001)
         else:
             results.append(outbuf)
     return results

catalyst/data/_minute_bar_internal.pyx

@@ -157,3 +157,4 @@ def find_last_traded_position_internal(
     # we've gone to the beginning of this asset's range, and still haven't
     # found a trade event
     return -1
+

catalyst/data/bundles/base.py

@@ -0,0 +1,501 @@
+#
+# Copyright 2017 Enigma MPC, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+from itertools import count
+import tarfile
+from time import time, sleep
+
+from abc import abstractmethod, abstractproperty
+import logbook
+import pandas as pd
+
+from . import core as bundles
+
+from catalyst.utils.cli import (
+    item_show_count,
+    maybe_show_progress
+)
+from catalyst.utils.memoize import lazyval
+
+from catalyst.constants import LOG_LEVEL
+
+logbook.StderrHandler().push_application()
+log = logbook.Logger(__name__, level=LOG_LEVEL)
+
+DEFAULT_RETRIES = 5
+
+class BaseBundle(object):
+    def __init__(self, asset_filter=[]):
+        self._asset_filter = asset_filter
+        self._reset()
+
+    def _reset(self):
+        self._splits = []
+        self._dividends = []
+
+    @lazyval
+    def name(self):
+        raise NotImplementedError()
+
+    @lazyval
+    def exchange(self):
+        raise NotImplementedError()
+
+    @lazyval
+    def calendar_name(self):
+        raise NotImplementedError()
+
+    @lazyval
+    def minutes_per_day(self):
+        raise NotImplementedError()
+
+    @lazyval
+    def frequencies(self):
+        raise NotImplementedError()
+
+    @lazyval
+    def md_column_names(self):
+        return _dtypes_to_cols(self.md_dtypes)
+
+    @lazyval
+    def md_dtypes(self):
+        raise NotImplementedError()
+
+    @lazyval
+    def column_names(self):
+        return _dtypes_to_cols(self.dtypes)
+
+    @lazyval
+    def dtypes(self):
+        raise NotImplementedError()
+
+    @lazyval
+    def tar_url(self):
+        raise NotImplementedError()
+
+    @lazyval
+    def wait_time(self):
+        raise NotImplementedError()
+
+    @abstractproperty
+    def splits(self):
+        raise NotImplementedError()
+
+    @abstractproperty
+    def dividends(self):
+        raise NotImplementedError()
+
+    @abstractmethod
+    def fetch_raw_metadata_frame(self, api_key, page_number):
+        raise NotImplementedError()
+
+    def post_process_symbol_metadata(self, metadata, data):
+        return metadata
+    
+    @abstractmethod
+    def fetch_raw_symbol_frame(self, api_key, symbol, start_date, end_date):
+        raise NotImplementedError()
+    
+    def ingest(self,
+               environ,
+               asset_db_writer,
+               minute_bar_writer,
+               daily_bar_writer,
+               adjustment_writer,
+               calendar,
+               start_session,
+               end_session,
+               cache,
+               show_progress,
+               is_compile,
+               output_dir):
+
+        try:
+            api_key = environ.get('CATALYST_API_KEY')
+            retries = environ.get('CATALYST_DOWNLOAD_ATTEMPTS', 5)
+
+            if is_compile:
+                # User has instructed local compilation and ingestion of bundle.
+                # Fetch raw metadata for all symbols.
+                raw_metadata = self._fetch_metadata_frame(
+                    api_key,
+                    cache=cache,
+                    retries=retries,
+                    environ=environ,
+                    show_progress=show_progress,
+                )
+
+                # Compile daily symbol data if bundle supports daily mode and
+                # persist the dataset to disk.
+                symbol_map = raw_metadata.symbol
+                if 'daily' in self.frequencies:
+                    daily_bar_writer.write(
+                        self._fetch_symbol_iter(
+                            api_key,
+                            cache,
+                            symbol_map,
+                            calendar,
+                            start_session,
+                            end_session,
+                            'daily',
+                            retries,
+                        ),
+                        assets=raw_metadata.index,
+                        show_progress=show_progress,
+                    )
+
+                # Post-process metadata using cached symbol frames, and write to
+                # disk.  This metadata must be written before any attempt to write
+                # minute data.
+                metadata = self._post_process_metadata(
+                    raw_metadata,
+                    cache,
+                    show_progress=show_progress,
+                )
+                asset_db_writer.write(metadata)
+
+                # Compile minute symbol data if bundle supports minute mode and
+                # persist the dataset to disk.
+                if 'minute' in self.frequencies:
+                    minute_bar_writer.write(
+                        self._fetch_symbol_iter(
+                            api_key,
+                            cache,
+                            symbol_map,
+                            calendar,
+                            start_session,
+                            end_session,
+                            'minute',
+                            retries,
+                        ),
+                        show_progress=show_progress,
+                    )
+
+                # For legacy purposes, this call is required to ensure the database
+                # contains an appropriately initialized file structure.  We don't
+                # forsee a usecase for adjustments at this time, but may later
+                # choose to expose this functionality in the future.
+                adjustment_writer.write(
+                    splits=(
+                        pd.concat(self.splits, ignore_index=True)
+                        if len(self.splits) > 0 else
+                        None
+                    ),
+                    dividends=(
+                        pd.concat(self.dividends, ignore_index=True)
+                        if len(self.dividends) > 0 else
+                        None
+                    ),
+                )
+            else:
+                # Otherwise, user has instructed to download and untar bundle
+                # directly from the bundles `tar_url`.
+                self._download_and_untar(show_progress, output_dir)
+        except Exception as e:
+            log.exception(
+                ' Failed to ingest {name}:\n{msg}'.format(
+                    name=self.name,
+                    msg=str(e),
+                )
+            )
+        else:
+            self._reset()
+
+    def _download_and_untar(self, show_progress, output_dir):
+        # Download bundle conditioned on whether the user would like progress
+        # information to be displayed in the CLI.
+        if show_progress:
+            data = bundles.download_with_progress(
+                self.tar_url,
+                chunk_size=bundles.ONE_MEGABYTE,
+                label='Downloading {name} bundle'.format(name=self.name),
+            )
+        else:
+            data = bundles.download_without_progress(self.tar_url)
+
+        # File transfer has completed, untar the bundle to the appropriate
+        # data directory.
+        with tarfile.open('r', fileobj=data) as tar:
+            tar.extractall(output_dir)
+
+    def _fetch_metadata_frame(self,
+                             api_key,
+                             cache,
+                             retries=DEFAULT_RETRIES,
+                             environ=None,
+                             show_progress=False):
+        
+        # Setup raw metadata iterator to fetch pages if necessary.
+        raw_iter = self._fetch_metadata_iter(api_key, cache, retries, environ)
+
+        # Concatenate all frame in iterator to compute a single metadata frame.
+        with maybe_show_progress(
+            raw_iter,
+            show_progress,
+            label='Fetching symbol metadata',
+            item_show_func=item_show_count(),
+            length=3,
+            show_percent=False,
+        ) as blocks:
+            metadata = pd.concat(blocks, ignore_index=True)
+        
+        return metadata
+
+    def _fetch_metadata_iter(self, api_key, cache, retries, environ):
+        for page_number in count(1):
+            # Attempt to load metadata page from cache.  If it does not exist,
+            # poll the API upto `retries` times in order to get raw DataFrame.
+            key = 'metadata-page-{pn}.frame'.format(pn=page_number)
+            try:
+                raw = cache[key]
+            except KeyError:
+                for _ in range(retries):
+                    try:
+                        raw = self.fetch_raw_metadata_frame(
+                            api_key,
+                            page_number,
+                        )
+                        break
+                    except ValueError as e:
+                        raw = pd.DataFrame([])
+                        break
+                    except Exception as e:
+                        log.exception(
+                            'Failed to load metadata from {}. '
+                            'Retrying.'.format(self.name)
+                        ) 
+                else:
+                    raise ValueError(
+                        'Failed to download metadata page {} after {} '
+                        'attempts.'.format(page_number, retries)
+                    )
+
+
+            if raw.empty:
+                # Empty DataFrame signals completion.
+                break
+
+            # Apply selective asset filtering, useful for benchmark
+            # ingestion.
+            if self._asset_filter:
+                raw = raw[raw.symbol.isin(self._asset_filter)]
+
+            # Update cached value for key.
+            cache[key] = raw
+
+            # Return metadata frame to application.
+            yield raw
+
+    def _post_process_metadata(self, metadata, cache, show_progress=False):
+        # Create empty data frame using target metadata column names and dtypes
+        final_metadata = pd.DataFrame(
+            columns=self.md_column_names,
+            index=metadata.index,
+        )
+        
+        # Iterate over the available symbols, loading the asset's raw symbol
+        # data from the cache.  The final metadata is computed and recorded in
+        # the appropriate row depending on the asset's id.
+        with maybe_show_progress(
+            metadata.symbol.iteritems(),
+            show_progress,
+            label='Post-processing symbol metadata',
+            item_show_func=item_show_count(len(metadata)),
+            length=len(metadata),
+            show_percent=False,
+        ) as symbols_map:
+            for asset_id, symbol in symbols_map:
+                # Attempt to load data from disk, the cache should have an entry
+                # for each symbol at this point of the execution. If one does
+                # not exist, we should fail.
+                key = '{sym}.daily.frame'.format(sym=symbol)
+                try:
+                    raw_data = cache[key]
+                except KeyError:
+                    raise ValueError(
+                      'Unable to find cached data for symbol: {0}'.format(symbol)
+                    )
+
+                # Perform and require post-processing of metadata.
+                final_symbol_metadata = self.post_process_symbol_metadata(
+                    asset_id,
+                    metadata.iloc[asset_id],
+                    raw_data,        
+                )
+
+                # Record symbol's final metadata.
+                final_metadata.iloc[asset_id] = final_symbol_metadata
+
+            # Register all assets with the bundle's default exchange.
+            final_metadata['exchange'] = self.exchange
+
+        return final_metadata
+
+    def _fetch_symbol_iter(self,
+                           api_key,
+                           cache,
+                           symbol_map,
+                           calendar,
+                           start_session,
+                           end_session,
+                           data_frequency,
+                           retries):
+
+        for asset_id, symbol in symbol_map.iteritems():
+            # Record start time of iteration, compare at end of iteration to
+            # adhere to the datas source's rate limit policy.
+            start_time = pd.Timestamp.utcnow()
+
+            # Fetch new data if cached data is absent or stale, otherwise
+            # returns the cached data unaltered.  The `should_sleep` flag
+            # indicates that an API call was attempted, and that we should be
+            # ensure aren't exceeding our rate limit before proceeding to the
+            # next symbol. If the raw_data is updated, it is cached before being
+            # returned.
+            raw_data, should_sleep = self._maybe_update_symbol_frame(
+                start_time,
+                api_key,
+                cache,
+                symbol,
+                calendar,
+                start_session,
+                end_session,
+                data_frequency,
+                retries,
+            )
+
+            # TODO(cfromknecht) further data validation?
+
+            # Pass asset_id and symbol data to writer.
+            yield asset_id, raw_data
+
+            # If an API call was made during this iteration and the time to
+            # reach this point was less than the inter-request `wait_time`,
+            # sleep until after enough time has elapsed to prevent getting rate
+            # limited.
+            if should_sleep:
+                remaining = pd.Timestamp.utcnow() - start_time + self.wait_time
+                if remaining.value > 0:
+                    sleep(remaining.value / 10**9)
+
+    def _maybe_update_symbol_frame(self,
+                                   start_time,
+                                   api_key,
+                                   cache,
+                                   symbol,
+                                   calendar,
+                                   start_session,
+                                   end_session,
+                                   data_frequency,
+                                   retries):
+
+        # Attempt to load pre-existing symbol data from cache.
+        key = '{sym}.{freq}.frame'.format(sym=symbol, freq=data_frequency)
+        try:
+            raw_data = cache[key]
+        except KeyError:
+            raw_data = None
+
+        # Select the most recent date in cached dataset if it exists,
+        # otherwise use the provided `start_session`.
+        last = start_session
+        if raw_data is not None and len(raw_data) > 0:
+            last = raw_data.index[-1].tz_localize('UTC')
+        
+        should_sleep = False
+
+        # Determine time at which cached data will be considered stale.
+        cache_expiration = last + pd.Timedelta(days=2)
+        if start_time <= cache_expiration and raw_data is not None:
+            # Data is fresh enough to reuse, no need to update. Iterator can
+            # proceed to next symbol directly since no API call was required.
+            return raw_data, should_sleep
+
+        # If we arrive here, we must have attempted an API call.
+        # Setting this flag tells the iterator to pause before starting
+        # the next asset, that we don't exceed the data source's rate
+        # limit.
+        should_sleep = True
+
+        raw_data = self._fetch_symbol_frame(
+            api_key,
+            symbol,
+            calendar,
+            start_session,
+            end_session,
+            data_frequency,
+            retries=retries,
+        )
+
+        # Cache latest symbol data.
+        cache[key] = raw_data
+
+        return raw_data, should_sleep
+
+    def _fetch_symbol_frame(self,
+                            api_key,
+                            symbol,
+                            calendar,
+                            start_session,
+                            end_session,
+                            data_frequency,
+                            retries=DEFAULT_RETRIES):
+
+        # Data for symbol is old enough to attempt an update or is not
+        # present in the cache.  Fetch raw data for a single symbol 
+        # with requested intervals and frequency. Retry as necessary.
+        for _ in range(retries):
+            try:
+                raw_data = self.fetch_raw_symbol_frame(
+                    api_key,
+                    symbol,
+                    calendar,
+                    start_session,
+                    end_session,
+                    data_frequency,
+                )
+                raw_data.index = pd.to_datetime(raw_data.index, utc=True)
+                #raw_data.index = raw_data.index.tz_localize('UTC')
+
+                # Filter incoming data to fit start and end sessions.
+                raw_data = raw_data[
+                    (raw_data.index >= start_session) &
+                    (raw_data.index <= end_session)
+                ]
+
+                # Filter out any duplicates entries, keep last one, since
+                # previous frame is probably an incomplete.
+                raw_data = raw_data[~raw_data.index.duplicated(keep='last')]
+
+                return raw_data
+
+            except Exception as e:
+                log.exception(
+                    'Exception raised fetching {name} data. Retrying.'
+                    .format(name=self.name)
+                )
+        else:
+            raise ValueError(
+                'Failed to download data for symbol {sym} '
+                'after {n} attempts.'.format(
+                    sym=symbol,
+                    n=retries,
+                )
+            )
+
+
+def _dtypes_to_cols(dtypes):
+    return [name for name, _ in dtypes]

catalyst/data/bundles/base_pricing.py

@@ -0,0 +1,73 @@
+#
+# Copyright 2017 Enigma MPC, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from catalyst.data.bundles.base import BaseBundle
+from catalyst.utils.memoize import lazyval
+
+class BasePricingBundle(BaseBundle):
+    @lazyval
+    def md_dtypes(self):
+        return [
+            ('symbol', 'object'),
+            ('start_date', 'datetime64[ns]'),
+            ('end_date', 'datetime64[ns]'),
+            ('ac_date', 'datetime64[ns]'),
+            ('min_trade_size', 'float'),
+        ]
+
+    @lazyval
+    def dtypes(self):
+        return [
+            ('date', 'datetime64[ns]'),
+            ('open', 'float64'),
+            ('high', 'float64'),
+            ('low', 'float64'),
+            ('close', 'float64'),
+            ('volume', 'float64'),
+        ]
+
+class BaseCryptoPricingBundle(BasePricingBundle):
+    @lazyval
+    def calendar_name(self):
+        return 'OPEN'
+
+    @lazyval
+    def minutes_per_day(self):
+        return 1440
+
+    @property
+    def splits(self):
+        return []
+
+    @property
+    def dividends(self):
+        return []
+
+class BaseEquityPricingBundle(BasePricingBundle):
+    @lazyval
+    def calendar_name(self):
+        return 'NYSE'
+
+    @lazyval
+    def minutes_per_day(self):
+        return 390
+
+    @property
+    def splits(self):
+        return self._splits
+
+    @property
+    def dividends(self):
+        return self._dividends

catalyst/data/bundles/core.py

@@ -33,6 +33,7 @@ from catalyst.utils.input_validation import ensure_timestamp, optionally
 import catalyst.utils.paths as pth
 from catalyst.utils.preprocess import preprocess
 from catalyst.utils.calendars import get_calendar
+from catalyst.utils.cli import maybe_show_progress
 
 ONE_MEGABYTE = 1024 * 1024
 
@@ -43,16 +44,16 @@ def asset_db_path(bundle_name, timestr, environ=None, db_version=None):
     )
 
 
-def minute_equity_path(bundle_name, timestr, environ=None):
+def minute_path(bundle_name, timestr, environ=None):
     return pth.data_path(
-        minute_equity_relative(bundle_name, timestr, environ),
+        minute_relative(bundle_name, timestr, environ),
         environ=environ,
     )
 
 
-def daily_equity_path(bundle_name, timestr, environ=None):
+def daily_path(bundle_name, timestr, environ=None):
     return pth.data_path(
-        daily_equity_relative(bundle_name, timestr, environ),
+        daily_relative(bundle_name, timestr, environ),
         environ=environ,
     )
 
@@ -79,11 +80,11 @@ def cache_relative(bundle_name, timestr, environ=None):
     return bundle_name, '.cache'
 
 
-def daily_equity_relative(bundle_name, timestr, environ=None):
+def daily_relative(bundle_name, timestr, environ=None):
     return bundle_name, timestr, 'daily_equities.bcolz'
 
 
-def minute_equity_relative(bundle_name, timestr, environ=None):
+def minute_relative(bundle_name, timestr, environ=None):
     return bundle_name, timestr, 'minute_equities.bcolz'
 
 
@@ -158,7 +159,9 @@ def download_with_progress(url, chunk_size, **progress_kwargs):
 
     total_size = int(resp.headers['content-length'])
     data = BytesIO()
-    with click.progressbar(length=total_size, **progress_kwargs) as pbar:
+
+    progress_kwargs['length'] = total_size
+    with maybe_show_progress(None, True, **progress_kwargs) as pbar:
         for chunk in resp.iter_content(chunk_size=chunk_size):
             data.write(chunk)
             pbar.update(len(chunk))
@@ -198,13 +201,13 @@ RegisteredBundle = namedtuple(
 
 BundleData = namedtuple(
     'BundleData',
-    'asset_finder equity_minute_bar_reader equity_daily_bar_reader '
+    'asset_finder minute_bar_reader daily_bar_reader '
     'adjustment_reader',
 )
 
 BundleCore = namedtuple(
     'BundleCore',
-    'bundles register unregister ingest load clean',
+    'bundles register_bundle register unregister ingest load clean',
 )
 
 
@@ -258,6 +261,8 @@ def _make_bundle_core():
     -------
     bundles : mappingproxy
         The mapping of bundles to bundle payloads.
+    register_bundle : Bundle
+        A bundle instance to add to the ``bundles`` mapping.
     register : callable
         The function which registers new bundles in the ``bundles`` mapping.
     unregister : callable
@@ -275,13 +280,29 @@ def _make_bundle_core():
     # warn when trampling another bundle.
     bundles = mappingproxy(_bundles)
 
+    def register_bundle(bundle_cls,
+                        asset_filter=None,
+                        start_session=None,
+                        end_session=None,
+                        create_writers=True):
+        bundle = bundle_cls(asset_filter=asset_filter)
+        return register(
+            bundle.name,
+            bundle.ingest,
+            calendar_name=bundle.calendar_name,
+            minutes_per_day=bundle.minutes_per_day,
+            start_session=start_session,
+            end_session=end_session,
+            create_writers=create_writers,
+        )
+
     @curry
     def register(name,
                  f,
-                 calendar_name='NYSE',
+                 calendar_name='OPEN',
                  start_session=None,
                  end_session=None,
-                 minutes_per_day=390,
+                 minutes_per_day=1440,
                  create_writers=True):
         """Register a data bundle ingest function.
 
@@ -393,7 +414,8 @@ def _make_bundle_core():
                environ=os.environ,
                timestamp=None,
                assets_versions=(),
-               show_progress=False):
+               show_progress=False,
+               is_compile=False):
         """Ingest data for a given bundle.
 
         Parameters
@@ -443,7 +465,7 @@ def _make_bundle_core():
                     pth.data_path([], environ=environ))
                 )
                 daily_bars_path = wd.ensure_dir(
-                    *daily_equity_relative(
+                    *daily_relative(
                         name, timestr, environ=environ,
                     )
                 )
@@ -457,10 +479,10 @@ def _make_bundle_core():
                 # when we create the SQLiteAdjustmentWriter below. The
                 # SQLiteAdjustmentWriter needs to open the daily ctables so
                 # that it can compute the adjustment ratios for the dividends.
-
                 daily_bar_writer.write(())
+
                 minute_bar_writer = BcolzMinuteBarWriter(
-                    wd.ensure_dir(*minute_equity_relative(
+                    wd.ensure_dir(*minute_relative(
                         name, timestr, environ=environ)
                     ),
                     calendar,
@@ -468,6 +490,7 @@ def _make_bundle_core():
                     end_session,
                     minutes_per_day=bundle.minutes_per_day,
                 )
+
                 assets_db_path = wd.getpath(*asset_db_relative(
                     name, timestr, environ=environ,
                 ))
@@ -502,6 +525,7 @@ def _make_bundle_core():
                 end_session,
                 cache,
                 show_progress,
+                is_compile,
                 pth.data_path([name, timestr], environ=environ),
             )
 
@@ -577,11 +601,11 @@ def _make_bundle_core():
             asset_finder=AssetFinder(
                 asset_db_path(name, timestr, environ=environ),
             ),
-            equity_minute_bar_reader=BcolzMinuteBarReader(
-                minute_equity_path(name, timestr, environ=environ),
+            minute_bar_reader=BcolzMinuteBarReader(
+                minute_path(name, timestr, environ=environ),
             ),
-            equity_daily_bar_reader=BcolzDailyBarReader(
-                daily_equity_path(name, timestr, environ=environ),
+            daily_bar_reader=BcolzDailyBarReader(
+                daily_path(name, timestr, environ=environ),
             ),
             adjustment_reader=SQLiteAdjustmentReader(
                 adjustment_db_path(name, timestr, environ=environ),
@@ -670,7 +694,15 @@ def _make_bundle_core():
 
         return cleaned
 
-    return BundleCore(bundles, register, unregister, ingest, load, clean)
+    return BundleCore(
+        bundles,
+        register_bundle,
+        register,
+        unregister,
+        ingest,
+        load,
+        clean,
+    )
 
 
-bundles, register, unregister, ingest, load, clean = _make_bundle_core()
+bundles, register_bundle, register, unregister, ingest, load, clean = _make_bundle_core()

catalyst/data/bundles/poloniex.py

@@ -1,38 +1,188 @@
-from io import BytesIO
-import tarfile
+#
+# Copyright 2017 Enigma MPC, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
 
-from . import core as bundles
+import sys
 
-POLONIEX_BUNDLE_URL = (
-    'https://www.dropbox.com/s/9naqffawnq8o4r2/poloniex-bundle.tar?dl=1'
-)
+from datetime import datetime
 
-@bundles.register(
-    'poloniex',
-    create_writers=False,
-    calendar_name='OPEN',
-    minutes_per_day=1440)
-def quantopian_quandl_bundle(environ,
-                             asset_db_writer,
-                             minute_bar_writer,
-                             daily_bar_writer,
-                             adjustment_writer,
-                             calendar,
-                             start_session,
-                             end_session,
-                             cache,
-                             show_progress,
-                             output_dir):
-    if show_progress:
-        data = bundles.download_with_progress(
-            POLONIEX_BUNDLE_URL,
-            chunk_size=bundles.ONE_MEGABYTE,
-            label="Downloading Bundle: poloniex",
+import pandas as pd
+
+from six.moves.urllib.parse import urlencode
+
+from catalyst.data.bundles.core import register_bundle
+from catalyst.data.bundles.base_pricing import BaseCryptoPricingBundle
+from catalyst.utils.memoize import lazyval
+
+from catalyst.curate.poloniex import PoloniexCurator
+
+class PoloniexBundle(BaseCryptoPricingBundle):
+    @lazyval
+    def name(self):
+        return 'poloniex'
+
+    @lazyval
+    def exchange(self):
+        return 'POLO'
+
+    @lazyval
+    def frequencies(self):
+        return set((
+            'daily',
+            'minute',
+        ))
+
+    @lazyval
+    def tar_url(self):
+        return (
+            'https://s3.amazonaws.com/enigmaco/catalyst-bundles/poloniex/poloniex-bundle.tar.gz'
         )
-    else:
-        data = bundles.download_without_progress(POLONIEX_BUNDLE_URL)
 
-    with tarfile.open('r', fileobj=data) as tar:
-        if show_progress:
-            print("Writing data to %s." % output_dir)
-        tar.extractall(output_dir)
+    @lazyval
+    def wait_time(self):
+        return pd.Timedelta(milliseconds=170)
+
+    def fetch_raw_metadata_frame(self, api_key, page_number):
+        if page_number > 1:
+            return pd.DataFrame([])
+
+        raw = pd.read_json(
+            self._format_metadata_url(
+              api_key,
+              page_number,
+            ),
+            orient='index',
+        )
+
+        raw = raw.sort_index().reset_index()
+        raw.rename(
+            columns={'index':'symbol'},
+            inplace=True,
+        )
+
+        raw = raw[raw['isFrozen'] == 0]
+
+        return raw
+
+    def post_process_symbol_metadata(self, asset_id, sym_md, sym_data):
+        start_date = sym_data.index[0]
+        end_date = sym_data.index[-1]
+        ac_date = end_date + pd.Timedelta(days=1)
+        min_trade_size = 0.00000001
+
+        return (
+            sym_md.symbol,
+            start_date,
+            end_date,
+            ac_date,
+            min_trade_size,
+        )
+
+    def fetch_raw_symbol_frame(self,
+                               api_key,
+                               symbol,
+                               calendar,
+                               start_date,
+                               end_date,
+                               frequency):
+
+        # TODO: replace this with direct exchange call
+        # The end date and frequency should be used to calculate the number of bars
+        if(frequency == 'minute'):
+            pc = PoloniexCurator()
+            raw = pc.onemin_to_dataframe(symbol, start_date, end_date)
+
+        else:
+            raw = pd.read_json(
+                self._format_data_url(
+                    api_key,
+                    symbol,
+                    start_date,
+                    end_date,
+                    frequency,
+                ),
+                orient='records',
+            )
+            raw.set_index('date', inplace=True)
+
+        # BcolzDailyBarReader introduces a 1/1000 factor in the way pricing is stored
+        # on disk, which we compensate here to get the right pricing amounts
+        # ref: data/us_equity_pricing.py
+        scale = 1
+        raw.loc[:, 'open'] /= scale
+        raw.loc[:, 'high'] /= scale
+        raw.loc[:, 'low'] /= scale
+        raw.loc[:, 'close'] /= scale
+        raw.loc[:, 'volume'] *= scale
+
+        return raw
+
+    '''
+    HELPER METHODS
+    '''
+
+    def _format_metadata_url(self, api_key, page_number):
+        query_params = [
+            ('command', 'returnTicker'),
+        ]
+
+        return self._format_polo_query(query_params)
+
+
+    def _format_data_url(self,
+                         api_key,
+                         symbol,
+                         start_date,
+                         end_date,
+                         data_frequency):
+        period_map = {
+            'daily': 86400,
+        }
+
+        try:
+            period = period_map[data_frequency]
+        except KeyError:
+            return None
+
+        query_params = [
+            ('command', 'returnChartData'),
+            ('currencyPair', symbol),
+            ('start', start_date.value / 10**9),
+            ('end', end_date.value / 10**9),
+            ('period', period),
+        ]
+            
+        return self._format_polo_query(query_params)
+    
+    def _format_polo_query(self, query_params):
+        # TODO: got against the exchange object
+        return 'https://poloniex.com/public?{query}'.format(
+            query=urlencode(query_params),
+        )
+
+''' 
+As a second parameter, you can pass an array of currency pairs 
+that will be processed as an asset_filter to only process that 
+subset of assets in the bundle, such as:
+register_bundle(PoloniexBundle, ['USDT_BTC',])
+
+For a production environment make sure to use (to bundle all pairs):
+register_bundle(PoloniexBundle)
+'''
+
+if 'ingest' in sys.argv and '-c' in sys.argv:
+    register_bundle(PoloniexBundle)
+else:
+    register_bundle(PoloniexBundle, create_writers=False)
+

catalyst/data/bundles/quandl.py

@@ -1,3 +1,28 @@
+#
+# Copyright 2017 Enigma MPC, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from datetime import datetime
+
+import pandas as pd
+
+from six.moves.urllib.parse import urlencode
+
+from catalyst.data.bundles.core import register_bundle
+from catalyst.data.bundles.base_pricing import BaseEquityPricingBundle
+from catalyst.utils.memoize import lazyval
+
 """
 Module for building a complete daily dataset from Quandl's WIKI dataset.
 """
@@ -15,352 +40,194 @@ from catalyst.utils.cli import maybe_show_progress
 
 from . import core as bundles
 
-log = Logger(__name__)
+from catalyst.constants import LOG_LEVEL
+
+log = Logger(__name__, level=LOG_LEVEL)
 seconds_per_call = (pd.Timedelta('10 minutes') / 2000).total_seconds()
-# Invalid symbols that quandl has had in its metadata:
-excluded_symbols = frozenset({'TEST123456789'})
 
+class QuandlBundle(BaseEquityPricingBundle):
+    @lazyval
+    def name(self):
+        return 'quandl'
 
-def _fetch_raw_metadata(api_key, cache, retries, environ):
-    """Generator that yields each page of data from the metadata endpoint
-    as a dataframe.
-    """
-    for page_number in count(1):
-        key = 'metadata-page-%d' % page_number
-        try:
-            raw = cache[key]
-        except KeyError:
-            for _ in range(retries):
-                try:
-                    raw = pd.read_csv(
-                        format_metadata_url(api_key, page_number),
-                        date_parser=pd.tseries.tools.to_datetime,
-                        parse_dates=[
-                            'oldest_available_date',
-                            'newest_available_date',
-                        ],
-                        dtypes={
-                            'dataset_code': 'int',
-                            'name': 'str',
-                            'oldest_available_date': 'str',
-                            'newest_available_date': 'str',
-                        },
-                        usecols=[
-                            'dataset_code',
-                            'name',
-                            'oldest_available_date',
-                            'newest_available_date',
-                        ],
-                    )
-                    break
-                except ValueError:
-                    # when we are past the last page we will get a value
-                    # error because there will be no columns
-                    raw = pd.DataFrame([])
-                    break
-                except Exception:
-                    pass
-            else:
-                raise ValueError(
-                    'Failed to download metadata page %d after %d'
-                    ' attempts.' % (page_number, retries),
-                )
+    @lazyval
+    def exchange(self):
+        return 'QUANDL'
 
-            cache[key] = raw
+    @lazyval
+    def frequencies(self):
+        return set(('daily',))
 
-        if raw.empty:
-            # use the empty dataframe to signal completion
-            break
-        yield raw
+    @lazyval
+    def tar_url(self):
+        return 'https://s3.amazonaws.com/quantopian-public-zipline-data/quandl'
 
+    @lazyval
+    def wait_time(self):
+        return pd.Timedelta(milliseconds=300)
 
-def fetch_symbol_metadata_frame(api_key,
-                                cache,
-                                retries=5,
-                                environ=None,
-                                show_progress=False):
-    """
-    Download Quandl symbol metadata.
+    @lazyval
+    def _excluded_symbols(self):
+        """
+        Invalid symbols that quandl has had in its metadata:
+        """
+        return frozenset({'TEST123456789'})
 
-    Parameters
-    ----------
-    api_key : str
-        The quandl api key to use. If this is None then no api key will be
-        sent.
-    cache : DataFrameCache
-        The cache to use for persisting the intermediate data.
-    retries : int, optional
-        The number of times to retry each request before failing.
-    environ : mapping[str -> str], optional
-        The environment to use to find the catalyst home. By default this
-        is ``os.environ``.
-    show_progress : bool, optional
-        Show a progress bar for the download of this data.
-
-    Returns
-    -------
-    metadata_frame : pd.DataFrame
-        A dataframe with the following columns:
-          symbol: the asset's symbol
-          name: the full name of the asset
-          start_date: the first date of data for this asset
-          end_date: the last date of data for this asset
-          auto_close_date: end_date + one day
-          exchange: the exchange for the asset; this is always 'quandl'
-        The index of the dataframe will be used for symbol->sid mappings but
-        otherwise does not have specific meaning.
-    """
-    raw_iter = _fetch_raw_metadata(api_key, cache, retries, environ)
-
-    def item_show_func(_, _it=iter(count())):
-        'Downloading page: %d' % next(_it)
-
-    with maybe_show_progress(raw_iter,
-                             show_progress,
-                             item_show_func=item_show_func,
-                             label='Downloading WIKI metadata: ') as blocks:
-        data = pd.concat(blocks, ignore_index=True).rename(columns={
-            'dataset_code': 'symbol',
-            'name': 'asset_name',
-            'oldest_available_date': 'start_date',
-            'newest_available_date': 'end_date',
-        }).sort_values('symbol')
-
-    data = data[~data.symbol.isin(excluded_symbols)]
-    # cut out all the other stuff in the name column
-    # we need to escape the paren because it is actually splitting on a regex
-    data.asset_name = data.asset_name.str.split(r' \(', 1).str.get(0)
-    data['exchange'] = 'QUANDL'
-
-    data['start_date'] = data['start_date'].astype(datetime)
-    data['end_date'] = data['end_date'].astype(datetime)
-
-    data['auto_close_date'] = data['end_date'] + pd.Timedelta(days=1)
-    return data
-
-
-def format_metadata_url(api_key, page_number):
-    """Build the query RL for the quandl WIKI metadata.
-    """
-    query_params = [
-        ('per_page', '100'),
-        ('sort_by', 'id'),
-        ('page', str(page_number)),
-        ('database_code', 'WIKI'),
-    ]
-    if api_key is not None:
-        query_params = [('api_key', api_key)] + query_params
-    return (
-        'https://www.quandl.com/api/v3/datasets.csv?' + urlencode(query_params)
-    )
-
-
-def format_wiki_url(api_key, symbol, start_date, end_date):
-    """
-    Build a query URL for a quandl WIKI dataset.
-    """
-    query_params = [
-        ('start_date', start_date.strftime('%Y-%m-%d')),
-        ('end_date', end_date.strftime('%Y-%m-%d')),
-        ('order', 'asc'),
-    ]
-    if api_key is not None:
-        query_params = [('api_key', api_key)] + query_params
-
-    return (
-        "https://www.quandl.com/api/v3/datasets/WIKI/"
-        "{symbol}.csv?{query}".format(
-            symbol=symbol,
-            query=urlencode(query_params),
-        )
-    )
-
-
-def fetch_single_equity(api_key,
-                        symbol,
-                        start_date,
-                        end_date,
-                        retries=5):
-    """
-    Download data for a single equity.
-    """
-    for _ in range(retries):
-        try:
-            return pd.read_csv(
-                format_wiki_url(api_key, symbol, start_date, end_date),
-                parse_dates=['Date'],
-                index_col='Date',
-                usecols=[
-                    'Open',
-                    'High',
-                    'Low',
-                    'Close',
-                    'Volume',
-                    'Date',
-                    'Ex-Dividend',
-                    'Split Ratio',
-                ],
-                na_values=['NA'],
-            ).rename(columns={
-                'Open': 'open',
-                'High': 'high',
-                'Low': 'low',
-                'Close': 'close',
-                'Volume': 'volume',
-                'Date': 'date',
-                'Ex-Dividend': 'ex_dividend',
-                'Split Ratio': 'split_ratio',
-            })
-        except Exception:
-            log.exception("Exception raised reading Quandl data. Retrying.")
-    else:
-        raise ValueError(
-            "Failed to download data for %r after %d attempts." % (
-                symbol, retries
-            )
+    def fetch_raw_metadata_frame(self, api_key, page_number):
+        raw = pd.read_csv(
+            self._format_metadata_url(api_key, page_number),
+            date_parser=pd.tseries.tools.to_datetime,
+            parse_dates=[
+                'oldest_available_date',
+                'newest_available_date',
+            ],
+            dtype={
+                'dataset_code': 'str',
+                'name': 'str',
+                'oldest_available_date': 'str',
+                'newest_available_date': 'str',
+            },
+            usecols=[
+                'dataset_code',
+                'name',
+                'oldest_available_date',
+                'newest_available_date',
+            ],
+        ).rename(
+            columns={
+                'dataset_code': 'symbol',
+                'name': 'asset_name',
+                'oldest_available_date': 'start_date',
+                'newest_available_date': 'end_date',
+            },
         )
 
+        raw['start_date'] = raw['start_date'].astype(datetime)
+        raw['end_date'] = raw['end_date'].astype(datetime)
+        raw['ac_date'] = raw['end_date'] + pd.Timedelta(days=1)
 
-def _update_splits(splits, asset_id, raw_data):
-    split_ratios = raw_data.split_ratio
-    df = pd.DataFrame({'ratio': 1 / split_ratios[split_ratios != 1]})
-    df.index.name = 'effective_date'
-    df.reset_index(inplace=True)
-    df['sid'] = asset_id
-    splits.append(df)
+        # Filter out invalid symbols
+        raw = raw[~raw.symbol.isin(self._excluded_symbols)]
 
+        # cut out all the other stuff in the name column
+        # we need to escape the paren because it is actually splitting on a regex
+        raw.asset_name = raw.asset_name.str.split(r' \(', 1).str.get(0)
 
-def _update_dividends(dividends, asset_id, raw_data):
-    divs = raw_data.ex_dividend
-    df = pd.DataFrame({'amount': divs[divs != 0]})
-    df.index.name = 'ex_date'
-    df.reset_index(inplace=True)
-    df['sid'] = asset_id
-    # we do not have this data in the WIKI dataset
-    df['record_date'] = df['declared_date'] = df['pay_date'] = pd.NaT
-    dividends.append(df)
+        return raw
 
-
-def gen_symbol_data(api_key,
-                    cache,
-                    symbol_map,
-                    calendar,
-                    start_session,
-                    end_session,
-                    splits,
-                    dividends,
-                    retries):
-    for asset_id, symbol in symbol_map.iteritems():
-        start_time = time()
-        try:
-            # see if we have this data cached.
-            raw_data = cache[symbol]
-            should_sleep = False
-        except KeyError:
-            # we need to fetch the data and then write it to our cache
-            raw_data = cache[symbol] = fetch_single_equity(
+    def fetch_raw_symbol_frame(self,
+                               api_key,
+                               symbol,
+                               calendar,
+                               start_session,
+                               end_session,
+                               data_frequency):
+        raw_data = pd.read_csv(
+            self._format_wiki_url(
                 api_key,
                 symbol,
-                start_date=start_session,
-                end_date=end_session,
-            )
-            should_sleep = True
-
-        _update_splits(splits, asset_id, raw_data)
-        _update_dividends(dividends, asset_id, raw_data)
+                start_session,
+                end_session,
+                data_frequency,
+            ),
+            parse_dates=['Date'],
+            index_col='Date',
+            usecols=[
+                'Open',
+                'High',
+                'Low',
+                'Close',
+                'Volume',
+                'Date',
+                'Ex-Dividend',
+                'Split Ratio',
+            ],
+            na_values=['NA'],
+        ).rename(columns={
+            'Open': 'open',
+            'High': 'high',
+            'Low': 'low',
+            'Close': 'close',
+            'Volume': 'volume',
+            'Date': 'date',
+            'Ex-Dividend': 'ex_dividend',
+            'Split Ratio': 'split_ratio',
+        })
 
         sessions = calendar.sessions_in_range(start_session, end_session)
 
-        raw_data = raw_data.reindex(
+        return raw_data.reindex(
             sessions.tz_localize(None),
             copy=False,
         ).fillna(0.0)
-        yield asset_id, raw_data
 
-        if should_sleep:
-            remaining = seconds_per_call - time() - start_time
-            if remaining > 0:
-                sleep(remaining)
+    def post_process_symbol_metadata(self, asset_id, sym_md, sym_data):
+        self._update_splits(asset_id, sym_data)
+        self._update_dividends(asset_id, sym_data)
+
+        return sym_md
+
+    def _update_splits(self, asset_id, raw_data):
+        split_ratios = raw_data.split_ratio
+        df = pd.DataFrame({'ratio': 1 / split_ratios[split_ratios != 1]})
+        df.index.name = 'effective_date'
+        df.reset_index(inplace=True)
+        df['sid'] = asset_id
+        self.splits.append(df)
 
 
-@bundles.register('quandl')
-def quandl_bundle(environ,
-                  asset_db_writer,
-                  minute_bar_writer,
-                  daily_bar_writer,
-                  adjustment_writer,
-                  calendar,
-                  start_session,
-                  end_session,
-                  cache,
-                  show_progress,
-                  output_dir):
-    """Build a catalyst data bundle from the Quandl WIKI dataset.
-    """
-    api_key = environ.get('QUANDL_API_KEY')
-    metadata = fetch_symbol_metadata_frame(
-        api_key,
-        cache=cache,
-        show_progress=show_progress,
-    )
-    symbol_map = metadata.symbol
-
-    # data we will collect in `gen_symbol_data`
-    splits = []
-    dividends = []
-
-    asset_db_writer.write(metadata)
-    daily_bar_writer.write(
-        gen_symbol_data(
-            api_key,
-            cache,
-            symbol_map,
-            calendar,
-            start_session,
-            end_session,
-            splits,
-            dividends,
-            environ.get('QUANDL_DOWNLOAD_ATTEMPTS', 5),
-        ),
-        assets=metadata.index,
-        show_progress=show_progress,
-    )
-    adjustment_writer.write(
-        splits=pd.concat(splits, ignore_index=True),
-        dividends=pd.concat(dividends, ignore_index=True),
-    )
+    def _update_dividends(self, asset_id, raw_data):
+        divs = raw_data.ex_dividend
+        df = pd.DataFrame({'amount': divs[divs != 0]})
+        df.index.name = 'ex_date'
+        df.reset_index(inplace=True)
+        df['sid'] = asset_id
+        # we do not have this data in the WIKI dataset
+        df['record_date'] = df['declared_date'] = df['pay_date'] = pd.NaT
+        self.dividends.append(df)
 
 
-QUANTOPIAN_QUANDL_URL = (
-    'https://s3.amazonaws.com/quantopian-public-zipline-data/quandl'
-)
+    def _format_metadata_url(self, api_key, page_number):
+        """Build the query RL for the quandl WIKI metadata.
+        """
+        query_params = [
+            ('per_page', '100'),
+            ('sort_by', 'id'),
+            ('page', str(page_number)),
+            ('database_code', 'WIKI'),
+        ]
+        if api_key is not None:
+            query_params = [('api_key', api_key)] + query_params
 
-
-@bundles.register('quantopian-quandl', create_writers=False)
-def quantopian_quandl_bundle(environ,
-                             asset_db_writer,
-                             minute_bar_writer,
-                             daily_bar_writer,
-                             adjustment_writer,
-                             calendar,
-                             start_session,
-                             end_session,
-                             cache,
-                             show_progress,
-                             output_dir):
-    if show_progress:
-        data = bundles.download_with_progress(
-            QUANTOPIAN_QUANDL_URL,
-            chunk_size=bundles.ONE_MEGABYTE,
-            label="Downloading Bundle: quantopian-quandl",
+        return (
+            'https://www.quandl.com/api/v3/datasets.csv?' + urlencode(query_params)
         )
-    else:
-        data = bundles.download_without_progress(QUANTOPIAN_QUANDL_URL)
-
-    with tarfile.open('r', fileobj=data) as tar:
-        if show_progress:
-            print("Writing data to %s." % output_dir)
-        tar.extractall(output_dir)
 
 
-register_calendar_alias("QUANDL", "NYSE")
+    def _format_wiki_url(self,
+                         api_key,
+                         symbol,
+                         start_date,
+                         end_date,
+                         data_frequency):
+        """
+        Build a query URL for a quandl WIKI dataset.
+        """
+        query_params = [
+            ('start_date', start_date.strftime('%Y-%m-%d')),
+            ('end_date', end_date.strftime('%Y-%m-%d')),
+            ('order', 'asc'),
+        ]
+        if api_key is not None:
+            query_params = [('api_key', api_key)] + query_params
+
+        return (
+            "https://www.quandl.com/api/v3/datasets/WIKI/"
+            "{symbol}.csv?{query}".format(
+                symbol=symbol,
+                query=urlencode(query_params),
+            )
+        )
+
+register_calendar_alias('QUANDL', 'NYSE')
+register_bundle(QuandlBundle)

catalyst/data/data_portal.py

@@ -68,7 +68,9 @@ from catalyst.errors import (
     HistoryWindowStartsBeforeData,
 )
 
-log = Logger('DataPortal')
+from catalyst.constants import LOG_LEVEL
+
+log = Logger('DataPortal', level=LOG_LEVEL)
 
 BASE_FIELDS = frozenset([
     "open",
@@ -114,12 +116,12 @@ class DataPortal(object):
         The calendar instance used to provide minute->session information.
     first_trading_day : pd.Timestamp
         The first trading day for the simulation.
-    equity_daily_reader : BcolzDailyBarReader, optional
+    daily_reader : BcolzDailyBarReader, optional
         The daily bar reader for equities. This will be used to service
         daily data backtests or daily history calls in a minute backetest.
         If a daily bar reader is not provided but a minute bar reader is,
         the minutes will be rolled up to serve the daily requests.
-    equity_minute_reader : BcolzMinuteBarReader, optional
+    minute_reader : BcolzMinuteBarReader, optional
         The minute bar reader for equities. This will be used to service
         minute data backtests or minute history calls. This can be used
         to serve daily calls if no daily bar reader is provided.
@@ -144,8 +146,8 @@ class DataPortal(object):
                  asset_finder,
                  trading_calendar,
                  first_trading_day,
-                 equity_daily_reader=None,
-                 equity_minute_reader=None,
+                 daily_reader=None,
+                 minute_reader=None,
                  future_daily_reader=None,
                  future_minute_reader=None,
                  adjustment_reader=None,
@@ -180,7 +182,7 @@ class DataPortal(object):
             # Infer the last session from the provided readers.
             last_sessions = [
                 reader.last_available_dt
-                for reader in [equity_daily_reader, future_daily_reader]
+                for reader in [daily_reader, future_daily_reader]
                 if reader is not None
             ]
             if last_sessions:
@@ -194,7 +196,10 @@ class DataPortal(object):
             # Infer the last minute from the provided readers.
             last_minutes = [
                 reader.last_available_dt
-                for reader in [equity_minute_reader, future_minute_reader]
+                for reader in [
+                    minute_reader,
+                    future_minute_reader,
+                ]
                 if reader is not None
             ]
             if last_minutes:
@@ -202,10 +207,10 @@ class DataPortal(object):
             else:
                 self._last_available_minute = None
 
-        aligned_equity_minute_reader = self._ensure_reader_aligned(
-            equity_minute_reader)
-        aligned_equity_session_reader = self._ensure_reader_aligned(
-            equity_daily_reader)
+        aligned_minute_reader = self._ensure_reader_aligned(
+            minute_reader)
+        aligned_session_reader = self._ensure_reader_aligned(
+            daily_reader)
         aligned_future_minute_reader = self._ensure_reader_aligned(
             future_minute_reader)
         aligned_future_session_reader = self._ensure_reader_aligned(
@@ -219,10 +224,10 @@ class DataPortal(object):
         aligned_minute_readers = {}
         aligned_session_readers = {}
 
-        if aligned_equity_minute_reader is not None:
-            aligned_minute_readers[Equity] = aligned_equity_minute_reader
-        if aligned_equity_session_reader is not None:
-            aligned_session_readers[Equity] = aligned_equity_session_reader
+        if aligned_minute_reader is not None:
+            aligned_minute_readers[Equity] = aligned_minute_reader
+        if aligned_session_reader is not None:
+            aligned_session_readers[Equity] = aligned_session_reader
 
         if aligned_future_minute_reader is not None:
             aligned_minute_readers[Future] = aligned_future_minute_reader
@@ -514,15 +519,17 @@ class DataPortal(object):
                     )
             else:
                 if field == "last_traded":
-                    return self.get_last_traded_dt(asset, dt, 'minute')
+                    return self.get_last_traded_dt(asset, dt, data_frequency)
                 elif field == "price":
-                    return self._get_minute_spot_value(
-                        asset, "close", dt, ffill=True,
+                    return self._get_minutely_spot_value(
+                        asset, "close", dt, data_frequency, ffill=True,
                     )
                 elif field == "contract":
                     return self._get_current_contract(asset, dt)
                 else:
-                    return self._get_minute_spot_value(asset, field, dt)
+                    return self._get_minutely_spot_value(
+                        asset, field, dt, data_frequency,
+                     )
 
         if assets_is_scalar:
             return get_single_asset_value(assets)
@@ -648,8 +655,14 @@ class DataPortal(object):
 
         return spot_value
 
-    def _get_minute_spot_value(self, asset, column, dt, ffill=False):
-        reader = self._get_pricing_reader('minute')
+    def _get_minutely_spot_value(self,
+                                asset,
+                                column,
+                                dt,
+                                data_frequency,
+                                ffill=False):
+
+        reader = self._get_pricing_reader(data_frequency)
 
         if ffill:
             # If forward filling, we want the last minute with values (up to
@@ -680,8 +693,21 @@ class DataPortal(object):
         # the value we found came from a different day, so we have to adjust
         # the data if there are any adjustments on that day barrier
         return self.get_adjusted_value(
-            asset, column, query_dt,
-            dt, "minute", spot_value=result
+            asset,
+            column,
+            query_dt,
+            dt,
+            data_frequency,
+            spot_value=result
+        )
+
+    def _get_minute_spot_value(self, asset, column, dt, ffill=False):
+        return self._get_minutely_spot_value(
+            asset,
+            column,
+            dt,
+            ffill, 
+            'minute',
         )
 
     def _get_daily_spot_value(self, asset, column, dt):

catalyst/data/dispatch_bar_reader.py

@@ -18,6 +18,7 @@ from numpy import (
     full,
     nan,
     int64,
+    float64,
     zeros
 )
 from six import iteritems, with_metaclass
@@ -70,7 +71,9 @@ class AssetDispatchBarReader(with_metaclass(ABCMeta)):
         return self._dt_window_size(start_dt, end_dt), num_sids
 
     def _make_raw_array_out(self, field, shape):
-        if field != 'volume' and field != 'sid':
+        if field == 'volume':
+            out = zeros(shape, dtype=float64)
+        elif field != 'sid':
             out = full(shape, nan)
         else:
             out = zeros(shape, dtype=int64)
@@ -130,13 +133,11 @@ class AssetDispatchBarReader(with_metaclass(ABCMeta)):
 
         return results
 
-
 class AssetDispatchMinuteBarReader(AssetDispatchBarReader):
 
     def _dt_window_size(self, start_dt, end_dt):
         return len(self.trading_calendar.minutes_in_range(start_dt, end_dt))
 
-
 class AssetDispatchSessionBarReader(AssetDispatchBarReader):
 
     def _dt_window_size(self, start_dt, end_dt):

catalyst/data/history_loader.py

@@ -38,7 +38,7 @@ from catalyst.utils.numpy_utils import float64_dtype
 from catalyst.utils.pandas_utils import find_in_sorted_index
 
 # Default number of decimal places used for rounding asset prices.
-DEFAULT_ASSET_PRICE_DECIMALS = 3
+DEFAULT_ASSET_PRICE_DECIMALS = 9
 
 
 class HistoryCompatibleUSEquityAdjustmentReader(object):

catalyst/data/loader.py

@@ -12,40 +12,38 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+import datetime
 import os
 from collections import OrderedDict
 
 import logbook
 import pandas as pd
-import numpy as np
-from pandas_datareader.data import DataReader
-import datetime
 import pytz
+from pandas_datareader.data import DataReader
 from six import iteritems
 from six.moves.urllib_error import HTTPError
 
-from .benchmarks import get_benchmark_returns
+from catalyst.utils.calendars import get_calendar
 from . import treasuries, treasuries_can
+from .benchmarks import get_benchmark_returns
+from ..utils.deprecate import deprecated
 from ..utils.paths import (
     cache_root,
     data_root,
 )
-from ..utils.deprecate import deprecated
 
-from catalyst.curate.poloniex import PoloniexCurator
-from catalyst.utils.calendars import get_calendar
+from catalyst.constants import LOG_LEVEL
 
-
-logger = logbook.Logger('Loader')
+logger = logbook.Logger('Loader', level=LOG_LEVEL)
 
 # Mapping from index symbol to appropriate bond data
 INDEX_MAPPING = {
     'SPY':
-    (treasuries, 'treasury_curves.csv', 'www.federalreserve.gov'),
+        (treasuries, 'treasury_curves.csv', 'www.federalreserve.gov'),
     '^GSPTSE':
-    (treasuries_can, 'treasury_curves_can.csv', 'bankofcanada.ca'),
+        (treasuries_can, 'treasury_curves_can.csv', 'bankofcanada.ca'),
     '^FTSE':  # use US treasuries until UK bonds implemented
-    (treasuries, 'treasury_curves.csv', 'www.federalreserve.gov'),
+        (treasuries, 'treasury_curves.csv', 'www.federalreserve.gov'),
 }
 
 ONE_HOUR = pd.Timedelta(hours=1)
@@ -92,20 +90,28 @@ def has_data_for_dates(series_or_df, first_date, last_date):
     dts = series_or_df.index
     if not isinstance(dts, pd.DatetimeIndex):
         raise TypeError("Expected a DatetimeIndex, but got %s." % type(dts))
-    first, last = dts[[0, -1]]
-    return (first <= first_date) and (last >= last_date)
+    first, last = dts[[0, -1]].tz_localize(None)
+    return (first <= first_date.tz_localize(None)) and (
+        last >= last_date.tz_localize(None))
 
-def load_crypto_market_data(trading_day=None,
-                            trading_days=None,
-                            bm_symbol='USDT_BTC',
-                            environ=None):
+
+def load_crypto_market_data(trading_day=None, trading_days=None,
+                            bm_symbol=None, bundle=None, bundle_data=None,
+                            environ=None, exchange=None, start_dt=None,
+                            end_dt=None):
     if trading_day is None:
         trading_day = get_calendar('OPEN').trading_day
-    if trading_days is None:
-        trading_days = get_calendar('OPEN').all_sessions
 
-    first_date = trading_days[0]
-    now = pd.Timestamp.utcnow()
+    # TODO: consider making configurable
+    bm_symbol = 'btc_usdt'
+    # if trading_days is None:
+    #    trading_days = get_calendar('OPEN').schedule
+
+    # if start_dt is None:
+    start_dt = get_calendar('OPEN').first_trading_session
+
+    if end_dt is None:
+        end_dt = pd.Timestamp.utcnow()
 
     # We expect to have benchmark and treasury data that's current up until
     # **two** full trading days prior to the most recently completed trading
@@ -121,29 +127,54 @@ def load_crypto_market_data(trading_day=None,
 
     # We'll attempt to download new data if the latest entry in our cache is
     # before this date.
-    last_date = trading_days[trading_days.get_loc(now, method='ffill') - 2]
+    '''
+    if(bundle_data):
+        # If we are using the bundle to retrieve the cryptobenchmark, find the last
+        # date for which there is trading data in the bundle
+        asset = bundle_data.asset_finder.lookup_symbol(symbol=bm_symbol,as_of_date=None)
+        ix = bundle_data.daily_bar_reader._last_rows[asset.sid]
+        last_date = pd.to_datetime(bundle_data.daily_bar_reader._spot_col('day')[ix],unit='s')
+    else:
+        last_date = trading_days[trading_days.get_loc(now, method='ffill') - 2]
+    '''
+    last_date = trading_days[trading_days.get_loc(end_dt, method='ffill') - 1]
 
-    br = ensure_crypto_benchmark_data(
-        bm_symbol,
-        first_date,
-        last_date,
-        now,
-        # We need the trading_day to figure out the close prior to the first
-        # date so that we can compute returns for the first date.
-        trading_day,
-        environ,
-    )
+    if exchange is None:
+        # This is exceptional, since placing the import at the module scope
+        #  breaks things and it's only needed here
+        from catalyst.exchange.poloniex.poloniex import Poloniex
+        exchange = Poloniex('', '', '')
+
+    benchmark_asset = exchange.get_asset(bm_symbol)
+
+    # exchange.get_history_window() already ensures that we have the right data
+    # for the right dates
+    br = exchange.get_history_window(
+        assets=[benchmark_asset],
+        end_dt=last_date,
+        bar_count=pd.Timedelta(last_date - start_dt).days,
+        frequency='1d',
+        field='close',
+        data_frequency='daily')
+    br.columns = ['close']
+    br = br.pct_change(1).iloc[1:]
+    br.loc[start_dt] = 0
+    br = br.sort_index()
+
+    # Override first_date for treasury data since we have it for many more years
+    # and is independent of crypto data
+    first_date_treasury = pd.Timestamp('1990-01-02', tz='UTC')
     tc = ensure_treasury_data(
         bm_symbol,
-        first_date,
+        first_date_treasury,
         last_date,
-        now,
+        end_dt,
         environ,
     )
-    benchmark_returns = br[br.index.slice_indexer(first_date, last_date)]
-    treasury_curves = tc[tc.index.slice_indexer(first_date, last_date)]
+    benchmark_returns = br[br.index.slice_indexer(start_dt, last_date)]
+    treasury_curves = tc[
+        tc.index.slice_indexer(first_date_treasury, last_date)]
     return benchmark_returns, treasury_curves
-    
 
 
 def load_market_data(trading_day=None, trading_days=None, bm_symbol='SPY',
@@ -231,17 +262,22 @@ def load_market_data(trading_day=None, trading_days=None, bm_symbol='SPY',
     treasury_curves = tc[tc.index.slice_indexer(first_date, last_date)]
     return benchmark_returns, treasury_curves
 
-def ensure_crypto_benchmark_data(symbol, first_date, last_date, now,
-                                trading_day, environ=None):
+
+def ensure_crypto_benchmark_data(symbol,
+                                 first_date,
+                                 last_date,
+                                 now,
+                                 trading_day,
+                                 bundle,
+                                 bundle_data,
+                                 environ=None):
     filename = get_benchmark_filename(symbol)
-    source_filename = '/var/tmp/catalyst/data/poloniex/crypto_prices-{0}.csv'.\
-        format(symbol)
 
     logger.info(
         ('Loading benchmark data for {symbol!r} '
-            'from {first_date} to {last_date}'),
+         'from {first_date} to {last_date}'),
         symbol=symbol,
-        first_date=first_date - trading_day,
+        first_date=first_date,
         last_date=last_date
     )
 
@@ -254,106 +290,68 @@ def ensure_crypto_benchmark_data(symbol, first_date, last_date, now,
         environ,
     )
 
-
     if data is not None:
         return data
 
     # If no cached data was found or it was missing any dates then download the
     # necessary data.
-    logger.info(
-        ('Downloading benchmark data for {symbol!r} '
-            'from {first_date} to {last_date}'),
-        symbol=symbol,
-        first_date=first_date - trading_day,
-        last_date=last_date
-    )
 
-    def dateparse(time_in_secs):
-        return datetime.datetime.fromtimestamp(float(time_in_secs), pytz.utc)
+    if (bundle == 'poloniex'):
+        '''
+        If we're using the Poloniex bundle, we'll get the benchmark from the bundle
+        instead of downloading it from Poloniex every time we need it.
+        Poloniex has a captcha for API queries originating from outside the US that 
+        prevents users abroad from getting Catalyst to work
+        '''
+        logger.info(
+            (
+                'Retrieving benchmark data from bundle for {symbol!r} from {first_date} to {last_date}'),
+            symbol=symbol, first_date=first_date, last_date=last_date)
 
-    def compute_daily_bars(five_min_bars, schedule):
-        # filter and copy the entry at the beginning of each session
-        daily_bars = five_min_bars[
-            five_min_bars.index.isin(schedule)
-        ].copy()
+        asset = bundle_data.asset_finder.lookup_symbol(symbol=symbol,
+                                                       as_of_date=None)
+        fields = ['day', 'close']
+        raw = bundle_data.daily_bar_reader.load_raw_arrays(
+            columns=fields,
+            start_date=first_date - trading_day,
+            end_date=last_date,
+            assets=[asset, ])
+        bench_raw = pd.concat([pd.DataFrame(raw[0], columns=['date']),
+                               pd.DataFrame(raw[1], columns=['close'])],
+                              axis=1)
+        bench_raw['date'] = pd.to_datetime(bench_raw['date'], unit='s')
+        bench_raw.set_index('date', inplace=True)
+        bench_raw.sort_index(inplace=True)
+        bench_raw = bench_raw[
+                    pd.to_datetime(first_date - trading_day):pd.to_datetime(
+                        last_date)]
 
-        day_offset = pd.Timedelta(days=1)
+    else:
+        # This is how it used to be: downloading the benchmark everytime. 
+        # Leaving this code here to be repurposed in the future for other bundles.
+        logger.info(
+            (
+                'Downloading benchmark data for {symbol!r} from {first_date} to {last_date}'),
+            symbol=symbol, first_date=first_date, last_date=last_date)
 
-        # iterate through session starts doing:
-        # 1. filter five_min_bars to get all entries in one day
-        # 2. compute daily bar entry
-        # 3. record in rid-th row of daily_bars
-        for rid, start_date in enumerate(daily_bars.index):
-            # compute beginning of next session
-            end_date = start_date + day_offset
-
-            # filter for entries session entries
-            day_data = five_min_bars[
-                (five_min_bars.index >= start_date) &
-                (five_min_bars.index < end_date)
-            ]
-
-            # compute and record daily bar
-            daily_bars.iloc[rid] = (
-                day_data.open.iloc[0],   # first open price
-                day_data.high.max(),     # max of high prices
-                day_data.low.min(),      # min of low prices
-                day_data.close.iloc[-1], # last close prices
-                day_data.volume.sum(),   # sum of all volumes
-            )
-
-        # scale to allow trading 10-ths of a coin
-        scale = 10.0
-        daily_bars.loc[:, 'open'] /= scale
-        daily_bars.loc[:, 'high'] /= scale
-        daily_bars.loc[:, 'low'] /= scale
-        daily_bars.loc[:, 'close'] /= scale
-        daily_bars.loc[:, 'volume'] *= scale
-        
-        return daily_bars
-
-    
-    five_min_bars = None
-    try:
-        # load five minute bars from csv cache
-        five_min_bars = pd.read_csv(
-            source_filename,
-            names=['date', 'open', 'high', 'low', 'close', 'volume'],
-            index_col=[0],
-            parse_dates=True,
-            date_parser=dateparse,
-        )
-        five_min_bars.index = pd.to_datetime(five_min_bars.index, utc=True, unit='s')
-    except (OSError, IOError):
-        # Otherwise load from Poloniex API
-        try:
-            pc = PoloniexCurator()
-            pc.append_data_single_pair(symbol)
-
-            five_min_bars = pc.to_dataframe(
-                time.mktime(first_date.timetuple()),
-                time.mktime(last_date.timetuple()),
-                currencyPair=symbol,
-            )
-        except (OSError, IOError, HTTPError):
-            logger.exception('Failed to new crypto benchmark returns')
-            raise
-
-    # compute daily bars for open calendar
-    open_calendar = get_calendar('OPEN')
-    daily_bars = compute_daily_bars(
-        five_min_bars,
-        open_calendar.all_sessions,
-    )
-
-    # filter daily bars to include first_date and last_date
-    daily_bars = daily_bars[
-        (daily_bars.index >= (first_date - trading_day)) &
-        (daily_bars.index <= last_date)
-    ]
+        raise DeprecationWarning('poloniex bundle deprecated')
+        # Load benchmark symbol from Poloniex API
+        # try:
+        #     bundle = PoloniexBundle()
+        #     bench_raw = bundle._fetch_symbol_frame(
+        #         None,
+        #         symbol,
+        #         get_calendar(bundle.calendar_name),
+        #         first_date - trading_day,
+        #         last_date,
+        #         'daily',
+        #     )
+        # except (OSError, IOError, HTTPError):
+        #     logger.exception('Failed to fetch new crypto benchmark returns')
+        #     raise
 
     # select close column and compute percent change between days
-    daily_close = daily_bars[['close']]
+    daily_close = bench_raw[['close']]
     daily_close = daily_close.pct_change(1).iloc[1:]
 
     try:
@@ -409,7 +407,7 @@ def ensure_benchmark_data(symbol, first_date, last_date, now, trading_day,
     # necessary data.
     logger.info(
         ('Downloading benchmark data for {symbol!r} '
-            'from {first_date} to {last_date}'),
+         'from {first_date} to {last_date}'),
         symbol=symbol,
         first_date=first_date - trading_day,
         last_date=last_date
@@ -429,6 +427,7 @@ def ensure_benchmark_data(symbol, first_date, last_date, now, trading_day,
         logger.warn("Still don't have expected data after redownload!")
     return data
 
+
 def ensure_benchmark_data(symbol, first_date, last_date, now, trading_day,
                           environ=None):
     """
@@ -469,7 +468,7 @@ def ensure_benchmark_data(symbol, first_date, last_date, now, trading_day,
     # necessary data.
     logger.info(
         ('Downloading benchmark data for {symbol!r} '
-            'from {first_date} to {last_date}'),
+         'from {first_date} to {last_date}'),
         symbol=symbol,
         first_date=first_date - trading_day,
         last_date=last_date
@@ -543,11 +542,6 @@ def ensure_treasury_data(symbol, first_date, last_date, now, environ=None):
 
 def _load_cached_data(filename, first_date, last_date, now, resource_name,
                       environ=None):
-    if resource_name == 'benchmark':
-        from_csv = pd.Series.from_csv
-    else:
-        from_csv = pd.DataFrame.from_csv
-
     # Path for the cache.
     path = get_data_filepath(filename, environ)
 
@@ -555,8 +549,11 @@ def _load_cached_data(filename, first_date, last_date, now, resource_name,
     # yet, so don't try to read from 'path'.
     if os.path.exists(path):
         try:
-            data = from_csv(path)
-            data.index = pd.to_datetime(data.index).tz_localize('UTC')
+            data = pd.DataFrame.from_csv(path)
+            if data.empty:
+                raise ValueError("File is empty.")
+            data.index = pd.to_datetime(data.index, infer_datetime_format=True,
+                                        errors='coerce').tz_localize('UTC')
             if has_data_for_dates(data, first_date, last_date):
                 return data
 
@@ -582,7 +579,7 @@ def _load_cached_data(filename, first_date, last_date, now, resource_name,
             )
 
     logger.info(
-        "Cache at {path} does not have data from {start} to {end}.\n",
+        "Cache at {path} does not have data from {start} to {end}.",
         start=first_date,
         end=last_date,
         path=path,

catalyst/data/minute_bars.py

@@ -39,20 +39,21 @@ from catalyst.data._minute_bar_internal import (
 from catalyst.gens.sim_engine import NANOS_IN_MINUTE
 
 from catalyst.data.bar_reader import BarReader, NoDataOnDate
-from catalyst.data.us_equity_pricing import check_uint32_safe
+from catalyst.data.us_equity_pricing import check_uint64_safe
 from catalyst.utils.calendars import get_calendar
 from catalyst.utils.cli import maybe_show_progress
 from catalyst.utils.memoize import lazyval
 
+from catalyst.constants import LOG_LEVEL
 
-logger = logbook.Logger('MinuteBars')
+logger = logbook.Logger('MinuteBars', level=LOG_LEVEL)
 
 US_EQUITIES_MINUTES_PER_DAY = 390
 FUTURES_MINUTES_PER_DAY = 1440
 
 DEFAULT_EXPECTEDLEN = US_EQUITIES_MINUTES_PER_DAY * 252 * 15
 
-OHLC_RATIO = 1000
+OHLC_RATIO = 100000000
 
 
 class BcolzMinuteOverlappingData(Exception):
@@ -114,15 +115,15 @@ def _sid_subdir_path(sid):
 
 
 def convert_cols(cols, scale_factor, sid, invalid_data_behavior):
-    """Adapt OHLCV columns into uint32 columns.
+    """Adapt OHLCV columns into uint64 columns.
 
     Parameters
     ----------
     cols : dict
         A dict mapping each column name (open, high, low, close, volume)
-        to a float column to convert to uint32.
+        to a float column to convert to uint64.
     scale_factor : int
-        Factor to use to scale float values before converting to uint32.
+        Factor to use to scale float values before converting to uint64.
     sid : int
         Sid of the relevant asset, for logging.
     invalid_data_behavior : str
@@ -135,6 +136,7 @@ def convert_cols(cols, scale_factor, sid, invalid_data_behavior):
     scaled_highs = np.nan_to_num(cols['high']) * scale_factor
     scaled_lows = np.nan_to_num(cols['low']) * scale_factor
     scaled_closes = np.nan_to_num(cols['close']) * scale_factor
+    scaled_volumes = np.nan_to_num(cols['volume']) * scale_factor
 
     exclude_mask = np.zeros_like(scaled_opens, dtype=bool)
 
@@ -143,11 +145,12 @@ def convert_cols(cols, scale_factor, sid, invalid_data_behavior):
         ('high', scaled_highs),
         ('low', scaled_lows),
         ('close', scaled_closes),
+        ('volume', scaled_volumes),
     ]:
         max_val = scaled_col.max()
 
         try:
-            check_uint32_safe(max_val, col_name)
+            check_uint64_safe(max_val, col_name)
         except ValueError:
             if invalid_data_behavior == 'raise':
                 raise
@@ -155,20 +158,20 @@ def convert_cols(cols, scale_factor, sid, invalid_data_behavior):
             if invalid_data_behavior == 'warn':
                 logger.warn(
                     'Values for sid={}, col={} contain some too large for '
-                    'uint32 (max={}), filtering them out',
+                    'uint64 (max={}), filtering them out',
                     sid, col_name, max_val,
                 )
 
             # We want to exclude all rows that have an unsafe value in
             # this column.
-            exclude_mask &= (scaled_col >= np.iinfo(np.uint32).max)
+            exclude_mask &= (scaled_col >= np.iinfo(np.uint64).max)
 
     # Convert all cols to uint32.
-    opens = scaled_opens.astype(np.uint32)
-    highs = scaled_highs.astype(np.uint32)
-    lows = scaled_lows.astype(np.uint32)
-    closes = scaled_closes.astype(np.uint32)
-    volumes = cols['volume'].astype(np.uint32)
+    opens = scaled_opens.astype(np.uint64)
+    highs = scaled_highs.astype(np.uint64)
+    lows = scaled_lows.astype(np.uint64)
+    closes = scaled_closes.astype(np.uint64)
+    volumes = scaled_volumes.astype(np.uint64)
 
     # Exclude rows with unsafe values by setting to zero.
     opens[exclude_mask] = 0
@@ -260,14 +263,14 @@ class BcolzMinuteBarMetadata(object):
             )
 
     def __init__(
-        self,
-        default_ohlc_ratio,
-        ohlc_ratios_per_sid,
-        calendar,
-        start_session,
-        end_session,
-        minutes_per_day,
-        version=FORMAT_VERSION,
+            self,
+            default_ohlc_ratio,
+            ohlc_ratios_per_sid,
+            calendar,
+            start_session,
+            end_session,
+            minutes_per_day,
+            version=FORMAT_VERSION,
     ):
         self.calendar = calendar
         self.start_session = start_session
@@ -288,7 +291,7 @@ class BcolzMinuteBarMetadata(object):
         ohlc_ratio : int
             The default ratio by which to multiply the pricing data to
             convert the floats from floats to an integer to fit within
-            the np.uint32. If ohlc_ratios_per_sid is None or does not
+            the np.uint64. If ohlc_ratios_per_sid is None or does not
             contain a mapping for a given sid, this ratio is used.
         ohlc_ratios_per_sid : dict
              A dict mapping each sid in the output to the factor by
@@ -340,10 +343,10 @@ class BcolzMinuteBarMetadata(object):
             'first_trading_day': str(self.start_session.date()),
             'market_opens': (
                 market_opens.values.astype('datetime64[m]').
-                astype(np.int64).tolist()),
+                    astype(np.int64).tolist()),
             'market_closes': (
                 market_closes.values.astype('datetime64[m]').
-                astype(np.int64).tolist()),
+                    astype(np.int64).tolist()),
         }
         with open(self.metadata_path(rootdir), 'w+') as fp:
             json.dump(metadata, fp)
@@ -372,13 +375,13 @@ class BcolzMinuteBarWriter(object):
         The last trading session in the data set.
     default_ohlc_ratio : int, optional
         The default ratio by which to multiply the pricing data to
-        convert from floats to integers that fit within np.uint32. If
+        convert from floats to integers that fit within np.uint64. If
         ohlc_ratios_per_sid is None or does not contain a mapping for a
-        given sid, this ratio is used. Default is OHLC_RATIO (1000).
+        given sid, this ratio is used. Default is OHLC_RATIO (10^8).
     ohlc_ratios_per_sid : dict, optional
         A dict mapping each sid in the output to the ratio by which to
         multiply the pricing data to convert the floats from floats to
-        an integer to fit within the np.uint32.
+        an integer to fit within the np.uint64.
     expectedlen : int, optional
         The expected length of the dataset, used when creating the initial
         bcolz ctable.
@@ -401,11 +404,9 @@ class BcolzMinuteBarWriter(object):
     Each individual asset's data is stored as a bcolz table with a column for
     each pricing field: (open, high, low, close, volume)
 
-    The open, high, low, and close columns are integers which are 1000 times
+    The open, high, low, close and volume columns are integers which are 10^8 times
     the quoted price, so that the data can represented and stored as an
-    np.uint32, supporting market prices quoted up to the thousands place.
-
-    volume is a np.uint32 with no mutation of the tens place.
+    np.uint64, supporting market prices quoted up to the 1/10^8-th place.
 
     The 'index' for each individual asset are a repeating period of minutes of
     length `minutes_per_day` starting from each market open.
@@ -573,7 +574,7 @@ class BcolzMinuteBarWriter(object):
         if not os.path.exists(sid_containing_dirname):
             # Other sids may have already created the containing directory.
             os.makedirs(sid_containing_dirname)
-        initial_array = np.empty(0, np.uint32)
+        initial_array = np.empty(0, np.uint64)
         table = ctable(
             rootdir=path,
             columns=[
@@ -610,7 +611,7 @@ class BcolzMinuteBarWriter(object):
         minute_offset = len(table) % self._minutes_per_day
         num_to_prepend = numdays * self._minutes_per_day - minute_offset
 
-        prepend_array = np.zeros(num_to_prepend, np.uint32)
+        prepend_array = np.zeros(num_to_prepend, np.uint64)
         # Fill all OHLCV with zeros.
         table.append([prepend_array] * 5)
         table.flush()
@@ -815,11 +816,11 @@ class BcolzMinuteBarWriter(object):
 
         minutes_count = all_minutes_in_window.size
 
-        open_col = np.zeros(minutes_count, dtype=np.uint32)
-        high_col = np.zeros(minutes_count, dtype=np.uint32)
-        low_col = np.zeros(minutes_count, dtype=np.uint32)
-        close_col = np.zeros(minutes_count, dtype=np.uint32)
-        vol_col = np.zeros(minutes_count, dtype=np.uint32)
+        open_col = np.zeros(minutes_count, dtype=np.uint64)
+        high_col = np.zeros(minutes_count, dtype=np.uint64)
+        low_col = np.zeros(minutes_count, dtype=np.uint64)
+        close_col = np.zeros(minutes_count, dtype=np.uint64)
+        vol_col = np.zeros(minutes_count, dtype=np.uint64)
 
         dt_ixs = np.searchsorted(all_minutes_in_window.values,
                                  dts.astype('datetime64[ns]'))
@@ -914,10 +915,10 @@ class BcolzMinuteBarReader(MinuteBarReader):
         )
         self._schedule = self.calendar.schedule[slicer]
         self._market_opens = self._schedule.market_open
-        self._market_open_values = self._market_opens.values.\
+        self._market_open_values = self._market_opens.values. \
             astype('datetime64[m]').astype(np.int64)
         self._market_closes = self._schedule.market_close
-        self._market_close_values = self._market_closes.values.\
+        self._market_close_values = self._market_closes.values. \
             astype('datetime64[m]').astype(np.int64)
 
         self._default_ohlc_inverse = 1.0 / metadata.default_ohlc_ratio
@@ -1125,8 +1126,8 @@ class BcolzMinuteBarReader(MinuteBarReader):
             else:
                 return np.nan
 
-        if field != 'volume':
-            value *= self._ohlc_ratio_inverse_for_sid(sid)
+        # if field != 'volume':
+        value *= self._ohlc_ratio_inverse_for_sid(sid)
         return value
 
     def get_last_traded_dt(self, asset, dt):
@@ -1248,7 +1249,7 @@ class BcolzMinuteBarReader(MinuteBarReader):
             if field != 'volume':
                 out = np.full(shape, np.nan)
             else:
-                out = np.zeros(shape, dtype=np.uint32)
+                out = np.zeros(shape, dtype=np.float64)
 
             for i, sid in enumerate(sids):
                 carray = self._open_minute_file(field, sid)
@@ -1256,17 +1257,17 @@ class BcolzMinuteBarReader(MinuteBarReader):
                 if indices_to_exclude is not None:
                     for excl_start, excl_stop in indices_to_exclude[::-1]:
                         excl_slice = np.s_[
-                            excl_start - start_idx:excl_stop - start_idx + 1]
+                                     excl_start - start_idx:excl_stop - start_idx + 1]
                         values = np.delete(values, excl_slice)
 
                 where = values != 0
                 # first slice down to len(where) because we might not have
                 # written data for all the minutes requested
-                if field != 'volume':
-                    out[:len(where), i][where] = (
-                        values[where] * self._ohlc_ratio_inverse_for_sid(sid))
-                else:
-                    out[:len(where), i][where] = values[where]
+                # if field != 'volume':
+                out[:len(where), i][where] = (
+                    values[where] * self._ohlc_ratio_inverse_for_sid(sid))
+                # else:
+                #    out[:len(where), i][where] = values[where]
 
             results.append(out)
         return results
@@ -1319,9 +1320,9 @@ class H5MinuteBarUpdateWriter(object):
 
     def __init__(self, path, complevel=None, complib=None):
         self._complevel = complevel if complevel \
-            is not None else self._COMPLEVEL
+                                       is not None else self._COMPLEVEL
         self._complib = complib if complib \
-            is not None else self._COMPLIB
+                                   is not None else self._COMPLIB
         self._path = path
 
     def write(self, frames):
@@ -1353,6 +1354,7 @@ class H5MinuteBarUpdateReader(MinuteBarUpdateReader):
     path : str
         The path of the HDF5 file from which to source data.
     """
+
     def __init__(self, path):
         self._panel = pd.read_hdf(path)
 

catalyst/data/resample.py

@@ -156,7 +156,10 @@ class DailyHistoryAggregator(object):
             cache = self._caches[field] = (session, market_open, {})
 
         _, market_open, entries = cache
-        market_open = market_open.tz_localize('UTC')
+        try:
+            market_open = market_open.tz_localize('UTC')
+        except TypeError:
+            market_open = market_open.tz_convert('UTC')
         if dt != market_open:
             prev_dt = dt_value - self._one_min
         else:

catalyst/data/us_equity_pricing.py

@@ -11,6 +11,9 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+
+from __future__ import division     # Python2 req to have division of ints yield float
+
 from errno import ENOENT
 from functools import partial
 from os import remove
@@ -73,12 +76,16 @@ from catalyst.utils.sqlite_utils import (
     coerce_string_to_conn,
 )
 from catalyst.utils.memoize import lazyval
-from catalyst.utils.cli import maybe_show_progress
+from catalyst.utils.cli import (
+    item_show_count,
+    maybe_show_progress,
+)
 from ._equities import _compute_row_slices, _read_bcolz_data
 from ._adjustments import load_adjustments_from_sqlite
 
+from catalyst.constants import LOG_LEVEL
 
-logger = logbook.Logger('UsEquityPricing')
+logger = logbook.Logger('UsEquityPricing', level=LOG_LEVEL)
 
 OHLC = frozenset(['open', 'high', 'low', 'close'])
 OHLCV = frozenset(['open', 'high', 'low', 'close', 'volume'])
@@ -113,11 +120,21 @@ SQLITE_STOCK_DIVIDEND_PAYOUT_COLUMN_DTYPES = {
 UINT32_MAX = iinfo(uint32).max
 UINT64_MAX = iinfo(uint64).max
 
+PRICE_ADJUSTMENT_FACTOR = 1000000000    # Provides 9 decimals resolution. Also affects _equities.pyx L220
+
 
 def check_uint32_safe(value, colname):
     if value >= UINT32_MAX:
         raise ValueError(
-            "Value %s from column '%s' is too large" % (value, colname)
+            "Value %s from column '%s' is too large "
+            "for uint32" % (value, colname)
+        )
+
+def check_uint64_safe(value, colname):
+    if value >= UINT64_MAX:
+        raise ValueError(
+            "Value %s from column '%s' is too large "
+            "for uint64" % (value, colname)
         )
 
 
@@ -218,10 +235,7 @@ class BcolzDailyBarWriter(object):
 
     @property
     def progress_bar_message(self):
-        return "Merging daily equity files:"
-
-    def progress_bar_item_show_func(self, value):
-        return value if value is None else str(value[0])
+        return 'Compiling daily data'
 
     def write(self,
               data,
@@ -249,15 +263,17 @@ class BcolzDailyBarWriter(object):
         table : bcolz.ctable
             The newly-written table.
         """
+        total = None if assets is None else len(assets)
         ctx = maybe_show_progress(
             (
                 (sid, self.to_ctable(df, invalid_data_behavior))
                 for sid, df in data
             ),
             show_progress=show_progress,
-            item_show_func=self.progress_bar_item_show_func,
             label=self.progress_bar_message,
-            length=len(assets) if assets is not None else None,
+            item_show_func=item_show_count(total),
+            length=total,
+            show_percent=False,
         )
         with ctx as it:
             return self._write_internal(it, assets)
@@ -423,11 +439,11 @@ class BcolzDailyBarWriter(object):
             return raw_data
 
         winsorise_uint64(raw_data, invalid_data_behavior, 'volume', *OHLC)
-        processed = (raw_data[list(OHLC)] * 1000000).astype('uint64')
+        processed = (raw_data[list(OHLC)] * PRICE_ADJUSTMENT_FACTOR).astype('uint64')
         dates = raw_data.index.values.astype('datetime64[s]')
         check_uint32_safe(dates.max().view(np.int64), 'day')
         processed['day'] = dates.astype('uint32')
-        processed['volume'] = raw_data.volume.astype('uint64')
+        processed['volume'] = (raw_data.volume * PRICE_ADJUSTMENT_FACTOR).astype('uint64')
         return ctable.fromdataframe(processed)
 
 
@@ -480,9 +496,8 @@ class BcolzDailyBarReader(SessionBarReader):
 
     The data in these columns is interpreted as follows:
 
-    - Price columns ('open', 'high', 'low', 'close') are interpreted as 1000 *
-      as-traded dollar value.
-    - Volume is interpreted as as-traded volume.
+    - Price columns ('open', 'high', 'low', 'close') and Volume are interpreted 
+      as 10^9 * as-traded dollar value.
     - Day is interpreted as seconds since midnight UTC, Jan 1, 1970.
     - Id is the asset id of the row.
 
@@ -509,7 +524,6 @@ class BcolzDailyBarReader(SessionBarReader):
         # Need to test keeping the entire array in memory for the course of a
         # process first.
         self._spot_cols = {}
-        self.PRICE_ADJUSTMENT_FACTOR = 0.001
         self._read_all_threshold = read_all_threshold
 
     @lazyval
@@ -749,13 +763,10 @@ class BcolzDailyBarReader(SessionBarReader):
         """
         ix = self.sid_day_index(sid, dt)
         price = self._spot_col(field)[ix]
-        if field != 'volume':
-            if price == 0:
-                return nan
-            else:
-                return price * 0.001
+        if field != 'volume' and price == 0:
+            return nan
         else:
-            return price
+            return price / PRICE_ADJUSTMENT_FACTOR
 
 
 class PanelBarReader(SessionBarReader):

catalyst/examples/arbitrage_with_interface.py

@@ -0,0 +1,275 @@
+from logbook import Logger
+
+from catalyst.api import (
+    record,
+    order,
+    symbol,
+    get_open_orders
+)
+from catalyst.exchange.stats_utils import get_pretty_stats
+from catalyst.utils.run_algo import run_algorithm
+
+algo_namespace = 'arbitrage_eth_btc'
+log = Logger(algo_namespace)
+
+
+def initialize(context):
+    log.info('initializing arbitrage algorithm')
+
+    # The context contains a new "exchanges" attribute which is a dictionary
+    # of exchange objects by exchange name. This allow easy access to the
+    # exchanges.
+    context.buying_exchange = context.exchanges['poloniex']
+    context.selling_exchange = context.exchanges['bitfinex']
+
+    context.trading_pair_symbol = 'eth_btc'
+    context.trading_pairs = dict()
+
+    # Note the second parameter of the symbol() method
+    # Passing the exchange name here returns a TradingPair object including
+    # the exchange information. This allow all other operations using
+    # the TradingPair to target the correct exchange.
+    context.trading_pairs[context.buying_exchange] = \
+        symbol('eth_btc', context.buying_exchange.name)
+
+    context.trading_pairs[context.selling_exchange] = \
+        symbol(context.trading_pair_symbol, context.selling_exchange.name)
+
+    context.entry_points = [
+        dict(gap=0.03, amount=0.05),
+        dict(gap=0.04, amount=0.1),
+        dict(gap=0.05, amount=0.5),
+    ]
+    context.exit_points = [
+        dict(gap=-0.02, amount=0.5),
+    ]
+
+    context.SLIPPAGE_ALLOWED = 0.02
+    pass
+
+
+def place_orders(context, amount, buying_price, selling_price, action):
+    """
+    This method will always place two orders of the same amount to keep
+    the currency position the same as it moves between the two exchanges.
+
+    :param context: TradingAlgorithm
+    :param amount: float
+        The trading pair amount to trade on both exchanges.
+    :param buying_price: float
+        The current trading pair price on the buying exchange.
+    :param selling_price: float
+        The current trading pair price on the selling exchange.
+    :param action: string
+        "enter": buys on the buying exchange and sells on the selling exchange
+        "exit": buys on the selling exchange and sells on the buying exchange
+
+    :return:
+    """
+    if action == 'enter':
+        enter_exchange = context.buying_exchange
+        entry_price = buying_price
+
+        exit_exchange = context.selling_exchange
+        exit_price = selling_price
+
+    elif action == 'exit':
+        enter_exchange = context.selling_exchange
+        entry_price = selling_price
+
+        exit_exchange = context.buying_exchange
+        exit_price = buying_price
+
+    else:
+        raise ValueError('invalid order action')
+
+    base_currency = enter_exchange.base_currency
+    base_currency_amount = enter_exchange.portfolio.cash
+
+    exit_balances = exit_exchange.get_balances()
+    exit_currency = context.trading_pairs[
+        context.selling_exchange].market_currency
+
+    if exit_currency in exit_balances:
+        market_currency_amount = exit_balances[exit_currency]
+    else:
+        log.warn(
+            'the selling exchange {exchange_name} does not hold '
+            'currency {currency}'.format(
+                exchange_name=exit_exchange.name,
+                currency=exit_currency
+            )
+        )
+        return
+
+    if base_currency_amount < (amount * entry_price):
+        adj_amount = base_currency_amount / entry_price
+        log.warn(
+            'not enough {base_currency} ({base_currency_amount}) to buy '
+            '{amount}, adjusting the amount to {adj_amount}'.format(
+                base_currency=base_currency,
+                base_currency_amount=base_currency_amount,
+                amount=amount,
+                adj_amount=adj_amount
+            )
+        )
+        amount = adj_amount
+
+    elif market_currency_amount < amount:
+        log.warn(
+            'not enough {currency} ({currency_amount}) to sell '
+            '{amount}, aborting'.format(
+                currency=exit_currency,
+                currency_amount=market_currency_amount,
+                amount=amount
+            )
+        )
+        return
+
+    adj_buy_price = entry_price * (1 + context.SLIPPAGE_ALLOWED)
+    log.info(
+        'buying {amount} {trading_pair} on {exchange_name} with price '
+        'limit {limit_price}'.format(
+            amount=amount,
+            trading_pair=context.trading_pair_symbol,
+            exchange_name=enter_exchange.name,
+            limit_price=adj_buy_price
+        )
+    )
+    order(
+        asset=context.trading_pairs[enter_exchange],
+        amount=amount,
+        limit_price=adj_buy_price
+    )
+
+    adj_sell_price = exit_price * (1 - context.SLIPPAGE_ALLOWED)
+    log.info(
+        'selling {amount} {trading_pair} on {exchange_name} with price '
+        'limit {limit_price}'.format(
+            amount=-amount,
+            trading_pair=context.trading_pair_symbol,
+            exchange_name=exit_exchange.name,
+            limit_price=adj_sell_price
+        )
+    )
+    order(
+        asset=context.trading_pairs[exit_exchange],
+        amount=-amount,
+        limit_price=adj_sell_price
+    )
+    pass
+
+
+def handle_data(context, data):
+    log.info('handling bar {}'.format(data.current_dt))
+
+    buying_price = data.current(
+        context.trading_pairs[context.buying_exchange], 'price')
+
+    log.info('price on buying exchange {exchange}: {price}'.format(
+        exchange=context.buying_exchange.name.upper(),
+        price=buying_price,
+    ))
+
+    selling_price = data.current(
+        context.trading_pairs[context.selling_exchange], 'price')
+
+    log.info('price on selling exchange {exchange}: {price}'.format(
+        exchange=context.selling_exchange.name.upper(),
+        price=selling_price,
+    ))
+
+    # If for example,
+    #   selling price = 50
+    #   buying price = 25
+    #   expected gap = 1
+
+    # If follows that,
+    #   selling price - buying price / buying price
+    #   50 - 25 / 25 = 1
+    gap = (selling_price - buying_price) / buying_price
+    log.info(
+        'the price gap: {gap} ({gap_percent}%)'.format(
+            gap=gap,
+            gap_percent=gap * 100
+        )
+    )
+    record(buying_price=buying_price, selling_price=selling_price, gap=gap)
+
+    # Waiting for orders to close before initiating new ones
+    for exchange in context.trading_pairs:
+        asset = context.trading_pairs[exchange]
+
+        orders = get_open_orders(asset)
+        if orders:
+            log.info(
+                'found {order_count} open orders on {exchange_name} '
+                'skipping bar until all open orders execute'.format(
+                    order_count=len(orders),
+                    exchange_name=exchange.name
+                )
+            )
+            return
+
+    # Consider the least ambitious entry point first
+    # Override of wider gap is found
+    entry_points = sorted(
+        context.entry_points,
+        key=lambda point: point['gap'],
+    )
+
+    buy_amount = None
+    for entry_point in entry_points:
+        if gap > entry_point['gap']:
+            buy_amount = entry_point['amount']
+
+    if buy_amount:
+        log.info('found buy trigger for amount: {}'.format(buy_amount))
+        place_orders(
+            context=context,
+            amount=buy_amount,
+            buying_price=buying_price,
+            selling_price=selling_price,
+            action='enter'
+        )
+
+    else:
+        # Consider the narrowest exit gap first
+        # Override of wider gap is found
+        exit_points = sorted(
+            context.exit_points,
+            key=lambda point: point['gap'],
+            reverse=True
+        )
+
+        sell_amount = None
+        for exit_point in exit_points:
+            if gap < exit_point['gap']:
+                sell_amount = exit_point['amount']
+
+        if sell_amount:
+            log.info('found sell trigger for amount: {}'.format(sell_amount))
+            place_orders(
+                context=context,
+                amount=sell_amount,
+                buying_price=buying_price,
+                selling_price=selling_price,
+                action='exit'
+            )
+
+
+def analyze(context, stats):
+    log.info('the daily stats:\n{}'.format(get_pretty_stats(stats)))
+    pass
+
+
+run_algorithm(
+    initialize=initialize,
+    handle_data=handle_data,
+    analyze=analyze,
+    exchange_name='poloniex,bitfinex',
+    live=True,
+    algo_namespace=algo_namespace,
+    base_currency='btc',
+    live_graph=False
+)

catalyst/examples/buy_and_hodl.py

@@ -23,9 +23,8 @@ from catalyst.api import (
     get_open_orders,
 )
 
-
 def initialize(context):
-    context.ASSET_NAME = 'USDT_ETH'
+    context.ASSET_NAME = 'USDT_BTC'
     context.TARGET_HODL_RATIO = 0.8
     context.RESERVE_RATIO = 1.0 - context.TARGET_HODL_RATIO
 
@@ -37,7 +36,11 @@ def initialize(context):
     context.is_buying = True
     context.asset = symbol(context.ASSET_NAME)
 
+    context.i = 0
+
 def handle_data(context, data):
+    context.i += 1
+
     starting_cash = context.portfolio.starting_cash
     target_hodl_value = context.TARGET_HODL_RATIO * starting_cash
     reserve_value = context.RESERVE_RATIO * starting_cash
@@ -67,6 +70,7 @@ def handle_data(context, data):
 
     record(
         price=price,
+        volume=data[context.asset].volume,
         cash=cash,
         starting_cash=context.portfolio.starting_cash,
         leverage=context.account.leverage,
@@ -74,12 +78,13 @@ def handle_data(context, data):
 
 def analyze(context=None, results=None):
     import matplotlib.pyplot as plt
+
     # Plot the portfolio and asset data.
-    ax1 = plt.subplot(511)
+    ax1 = plt.subplot(611)
     results[['portfolio_value']].plot(ax=ax1)
     ax1.set_ylabel('Portfolio Value (USD)')
 
-    ax2 = plt.subplot(512, sharex=ax1)
+    ax2 = plt.subplot(612, sharex=ax1)
     ax2.set_ylabel('{asset} (USD)'.format(asset=context.ASSET_NAME))
     (context.TICK_SIZE * results[['price']]).plot(ax=ax2)
 
@@ -95,11 +100,11 @@ def analyze(context=None, results=None):
         color='g',
     )
 
-    ax3 = plt.subplot(513, sharex=ax1)
+    ax3 = plt.subplot(613, sharex=ax1)
     results[['leverage', 'alpha', 'beta']].plot(ax=ax3)
     ax3.set_ylabel('Leverage ')
 
-    ax4 = plt.subplot(514, sharex=ax1)
+    ax4 = plt.subplot(614, sharex=ax1)
     results[['starting_cash', 'cash']].plot(ax=ax4)
     ax4.set_ylabel('Cash (USD)')
 
@@ -113,7 +118,7 @@ def analyze(context=None, results=None):
         'benchmark_period_return',
     ]]
 
-    ax5 = plt.subplot(515, sharex=ax1)
+    ax5 = plt.subplot(615, sharex=ax1)
     results[[
         'treasury',
         'algorithm',
@@ -121,8 +126,12 @@ def analyze(context=None, results=None):
     ]].plot(ax=ax5)
     ax5.set_ylabel('Percent Change')
 
+    ax6 = plt.subplot(616, sharex=ax1)
+    results[['volume']].plot(ax=ax6)
+    ax6.set_ylabel('Volume (mCoins/5min)')
+
     plt.legend(loc=3)
 
     # Show the plot.
     plt.gcf().set_size_inches(18, 8)
-    plt.show()
+    plt.show()

catalyst/examples/buy_btc.py

@@ -0,0 +1,10 @@
+from catalyst.api import order, record, symbol
+
+
+def initialize(context):
+    context.asset = symbol('btc_usd')
+
+
+def handle_data(context, data):
+    order(context.asset, 1)
+    record(btc=data.current(context.asset, 'price'))

catalyst/examples/buy_btc_simple.py

@@ -0,0 +1,8 @@
+from catalyst.api import order, record, symbol
+
+def initialize(context):
+    context.asset = symbol('btc_usd')
+
+def handle_data(context, data):
+    order(context.asset, 1)
+    record(btc = data.current(context.asset, 'price'))

catalyst/examples/buy_low_sell_high.py

@@ -0,0 +1,158 @@
+'''
+This algorithm requires an additional library (ta-lib) beyond those required by catalyst.
+Install it first by running: 
+$ pip install TA-Lib
+
+If you get build errors like "fatal error: ta-lib/ta_libc.h: No such file or directory"
+it typically means that it can't find the underlying TA-Lib library and needs to be installed.
+See https://mrjbq7.github.io/ta-lib/install.html for instructions on how to install 
+the required dependencies.
+'''
+
+import talib
+from logbook import Logger
+
+from catalyst.api import (
+    order,
+    order_target_percent,
+    symbol,
+    record,
+    get_open_orders,
+)
+from catalyst.exchange.stats_utils import get_pretty_stats
+
+algo_namespace = 'buy_low_sell_high_xrp'
+log = Logger(algo_namespace)
+
+
+def initialize(context):
+    log.info('initializing algo')
+    context.ASSET_NAME = 'XRP_USD'
+    context.asset = symbol(context.ASSET_NAME)
+
+    context.TARGET_POSITIONS = 5000
+    context.PROFIT_TARGET = 0.1
+    context.SLIPPAGE_ALLOWED = 0.05
+
+    context.retry_check_open_orders = 10
+    context.retry_update_portfolio = 10
+    context.retry_order = 5
+
+    context.swallow_errors = True
+
+    context.errors = []
+    pass
+
+
+def _handle_data(context, data):
+    prices = data.history(
+        context.asset,
+        fields='price',
+        bar_count=20,
+        frequency='15m'
+    )
+
+    rsi = talib.RSI(prices.values, timeperiod=14)[-1]
+    log.info('got rsi: {}'.format(rsi))
+
+    # Buying more when RSI is low, this should lower our cost basis
+    if rsi <= 30:
+        buy_increment = 50
+    elif rsi <= 40:
+        buy_increment = 20
+    elif rsi <= 70:
+        buy_increment = 5
+    else:
+        buy_increment = None
+
+    cash = context.portfolio.cash
+    log.info('base currency available: {cash}'.format(cash=cash))
+
+    price = data.current(context.asset, 'price')
+    log.info('got price {price}'.format(price=price))
+
+    record(
+        price=price,
+        rsi=rsi,
+    )
+
+    orders = get_open_orders(context.asset)
+    if orders:
+        log.info('skipping bar until all open orders execute')
+        return
+
+    is_buy = False
+    cost_basis = None
+    if context.asset in context.portfolio.positions:
+        position = context.portfolio.positions[context.asset]
+
+        cost_basis = position.cost_basis
+        log.info(
+            'found {amount} positions with cost basis {cost_basis}'.format(
+                amount=position.amount,
+                cost_basis=cost_basis
+            )
+        )
+
+        if position.amount >= context.TARGET_POSITIONS:
+            log.info('reached positions target: {}'.format(position.amount))
+            return
+
+        if price < cost_basis:
+            is_buy = True
+        elif position.amount > 0 and \
+                        price > cost_basis * (1 + context.PROFIT_TARGET):
+            profit = (price * position.amount) - (cost_basis * position.amount)
+            log.info('closing position, taking profit: {}'.format(profit))
+            order_target_percent(
+                asset=context.asset,
+                target=0,
+                limit_price=price * (1 - context.SLIPPAGE_ALLOWED),
+            )
+        else:
+            log.info('no buy or sell opportunity found')
+    else:
+        is_buy = True
+
+    if is_buy:
+        if buy_increment is None:
+            log.info('the rsi is too high to consider buying {}'.format(rsi))
+            return
+
+        if price * buy_increment > cash:
+            log.info('not enough base currency to consider buying')
+            return
+
+        log.info(
+            'buying position cheaper than cost basis {} < {}'.format(
+                price,
+                cost_basis
+            )
+        )
+        order(
+            asset=context.asset,
+            amount=buy_increment,
+            limit_price=price * (1 + context.SLIPPAGE_ALLOWED)
+        )
+
+
+def handle_data(context, data):
+    log.info('handling bar {}'.format(data.current_dt))
+    try:
+        _handle_data(context, data)
+    except Exception as e:
+        log.warn('aborting the bar on error {}'.format(e))
+        context.errors.append(e)
+
+    log.info('completed bar {}, total execution errors {}'.format(
+        data.current_dt,
+        len(context.errors)
+    ))
+
+    if len(context.errors) > 0:
+        log.info('the errors:\n{}'.format(context.errors))
+
+
+def analyze(context, stats):
+    log.info('the daily stats:\n{}'.format(get_pretty_stats(stats)))
+    pass

catalyst/examples/buy_low_sell_high_live.py

@@ -0,0 +1,168 @@
+import talib
+from logbook import Logger
+
+import pandas as pd
+from catalyst.api import (
+    order,
+    order_target_percent,
+    symbol,
+    record,
+    get_open_orders,
+)
+from catalyst.exchange.stats_utils import get_pretty_stats
+from catalyst.utils.run_algo import run_algorithm
+
+algo_namespace = 'buy_the_dip_live'
+log = Logger('buy low sell high')
+
+
+def initialize(context):
+    log.info('initializing algo')
+    context.ASSET_NAME = 'btc_usdt'
+    context.asset = symbol(context.ASSET_NAME)
+
+    context.TARGET_POSITIONS = 30
+    context.PROFIT_TARGET = 0.1
+    context.SLIPPAGE_ALLOWED = 0.02
+
+    context.retry_check_open_orders = 10
+    context.retry_update_portfolio = 10
+    context.retry_order = 5
+
+    context.errors = []
+    pass
+
+
+def _handle_data(context, data):
+    price = data.current(context.asset, 'price')
+    log.info('got price {price}'.format(price=price))
+
+    prices = data.history(
+        context.asset,
+        fields='price',
+        bar_count=20,
+        frequency='1d'
+    )
+    rsi = talib.RSI(prices.values, timeperiod=14)[-1]
+    log.info('got rsi: {}'.format(rsi))
+
+    # Buying more when RSI is low, this should lower our cost basis
+    if rsi <= 30:
+        buy_increment = 1
+    elif rsi <= 40:
+        buy_increment = 0.5
+    elif rsi <= 70:
+        buy_increment = 0.2
+    else:
+        buy_increment = 0.1
+
+    cash = context.portfolio.cash
+    log.info('base currency available: {cash}'.format(cash=cash))
+
+    record(
+        price=price,
+        rsi=rsi,
+    )
+
+    orders = get_open_orders(context.asset)
+    if orders:
+        log.info('skipping bar until all open orders execute')
+        return
+
+    is_buy = False
+    cost_basis = None
+    if context.asset in context.portfolio.positions:
+        position = context.portfolio.positions[context.asset]
+
+        cost_basis = position.cost_basis
+        log.info(
+            'found {amount} positions with cost basis {cost_basis}'.format(
+                amount=position.amount,
+                cost_basis=cost_basis
+            )
+        )
+
+        if position.amount >= context.TARGET_POSITIONS:
+            log.info('reached positions target: {}'.format(position.amount))
+            return
+
+        if price < cost_basis:
+            is_buy = True
+        elif position.amount > 0 and \
+                        price > cost_basis * (1 + context.PROFIT_TARGET):
+            profit = (price * position.amount) - (cost_basis * position.amount)
+            log.info('closing position, taking profit: {}'.format(profit))
+            order_target_percent(
+                asset=context.asset,
+                target=0,
+                limit_price=price * (1 - context.SLIPPAGE_ALLOWED),
+            )
+        else:
+            log.info('no buy or sell opportunity found')
+    else:
+        is_buy = True
+
+    if is_buy:
+        if buy_increment is None:
+            log.info('the rsi is too high to consider buying {}'.format(rsi))
+            return
+
+        if price * buy_increment > cash:
+            log.info('not enough base currency to consider buying')
+            return
+
+        log.info(
+            'buying position cheaper than cost basis {} < {}'.format(
+                price,
+                cost_basis
+            )
+        )
+        order(
+            asset=context.asset,
+            amount=buy_increment,
+            limit_price=price * (1 + context.SLIPPAGE_ALLOWED)
+        )
+
+
+def handle_data(context, data):
+    log.info('handling bar {}'.format(data.current_dt))
+    # try:
+    _handle_data(context, data)
+    # except Exception as e:
+    #     log.warn('aborting the bar on error {}'.format(e))
+    #     context.errors.append(e)
+
+    log.info('completed bar {}, total execution errors {}'.format(
+        data.current_dt,
+        len(context.errors)
+    ))
+
+    if len(context.errors) > 0:
+        log.info('the errors:\n{}'.format(context.errors))
+
+
+def analyze(context, stats):
+    log.info('the daily stats:\n{}'.format(get_pretty_stats(stats)))
+    pass
+
+
+run_algorithm(
+    capital_base=100000,
+    initialize=initialize,
+    handle_data=handle_data,
+    analyze=analyze,
+    exchange_name='poloniex',
+    start=pd.to_datetime('2017-5-01', utc=True),
+    end=pd.to_datetime('2017-10-16', utc=True),
+    base_currency='usdt',
+    data_frequency='daily'
+)
+# run_algorithm(
+#     initialize=initialize,
+#     handle_data=handle_data,
+#     analyze=analyze,
+#     exchange_name='poloniex',
+#     live=True,
+#     algo_namespace=algo_namespace,
+#     base_currency='btc'
+# )

catalyst/examples/buy_low_sell_high_neo_with_interface.py

@@ -0,0 +1,173 @@
+import talib
+from logbook import Logger
+import pandas as pd
+
+from catalyst.api import (
+    order,
+    order_target_percent,
+    symbol,
+    record,
+    get_open_orders,
+)
+from catalyst.exchange.stats_utils import get_pretty_stats
+from catalyst.utils.run_algo import run_algorithm
+
+algo_namespace = 'buy_low_sell_high_neo'
+log = Logger(algo_namespace)
+
+
+def initialize(context):
+    log.info('initializing algo')
+    context.asset = symbol('neo_btc', 'bitfinex')
+
+    context.TARGET_POSITIONS = 50000
+    context.PROFIT_TARGET = 0.1
+    context.SLIPPAGE_ALLOWED = 0.02
+
+    context.retry_check_open_orders = 10
+    context.retry_update_portfolio = 10
+    context.retry_order = 5
+
+    context.errors = []
+    pass
+
+
+def _handle_data(context, data):
+    price = data.current(context.asset, 'close')
+    log.info('got price {price}'.format(price=price))
+
+    if price is None:
+        log.warn('no pricing data')
+        return
+
+    prices = data.history(
+        context.asset,
+        fields='price',
+        bar_count=1,
+        frequency='1m'
+    )
+    rsi = talib.RSI(prices.values, timeperiod=14)[-1]
+    log.info('got rsi: {}'.format(rsi))
+
+    # Buying more when RSI is low, this should lower our cost basis
+    if rsi <= 30:
+        buy_increment = 1
+    elif rsi <= 40:
+        buy_increment = 0.5
+    elif rsi <= 70:
+        buy_increment = 0.1
+    else:
+        buy_increment = None
+
+    cash = context.portfolio.cash
+    log.info('base currency available: {cash}'.format(cash=cash))
+
+    record(price=price)
+
+    orders = get_open_orders(context.asset)
+    if len(orders) > 0:
+        log.info('skipping bar until all open orders execute')
+        return
+
+    is_buy = False
+    cost_basis = None
+    if context.asset in context.portfolio.positions:
+        position = context.portfolio.positions[context.asset]
+
+        cost_basis = position.cost_basis
+        log.info(
+            'found {amount} positions with cost basis {cost_basis}'.format(
+                amount=position.amount,
+                cost_basis=cost_basis
+            )
+        )
+
+        if position.amount >= context.TARGET_POSITIONS:
+            log.info('reached positions target: {}'.format(position.amount))
+            return
+
+        if price < cost_basis:
+            is_buy = True
+        elif position.amount > 0 and \
+                        price > cost_basis * (1 + context.PROFIT_TARGET):
+            profit = (price * position.amount) - (cost_basis * position.amount)
+
+            log.info('closing position, taking profit: {}'.format(profit))
+            order_target_percent(
+                asset=context.asset,
+                target=0,
+                limit_price=price * (1 - context.SLIPPAGE_ALLOWED),
+            )
+        else:
+            log.info('no buy or sell opportunity found')
+    else:
+        is_buy = True
+
+    if is_buy:
+        if buy_increment is None:
+            return
+
+        if price * buy_increment > cash:
+            log.info('not enough base currency to consider buying')
+            return
+
+        log.info(
+            'buying position cheaper than cost basis {} < {}'.format(
+                price,
+                cost_basis
+            )
+        )
+        limit_price = price * (1 + context.SLIPPAGE_ALLOWED)
+        order(
+            asset=context.asset,
+            amount=buy_increment,
+            limit_price=limit_price
+        )
+        pass
+
+
+def handle_data(context, data):
+    log.info('handling bar {}'.format(data.current_dt))
+    # try:
+    _handle_data(context, data)
+    # except Exception as e:
+    #     log.warn('aborting the bar on error {}'.format(e))
+    #     context.errors.append(e)
+
+    log.info('completed bar {}, total execution errors {}'.format(
+        data.current_dt,
+        len(context.errors)
+    ))
+
+    if len(context.errors) > 0:
+        log.info('the errors:\n{}'.format(context.errors))
+
+
+def analyze(context, stats):
+    log.info('the daily stats:\n{}'.format(get_pretty_stats(stats)))
+
+    pass
+
+
+# run_algorithm(
+#     initialize=initialize,
+#     handle_data=handle_data,
+#     analyze=analyze,
+#     exchange_name='bitfinex',
+#     live=True,
+#     algo_namespace=algo_namespace,
+#     base_currency='btc',
+#     live_graph=False
+# )
+
+# Backtest
+run_algorithm(
+    capital_base=250,
+    data_frequency='minute',
+    initialize=initialize,
+    handle_data=handle_data,
+    analyze=analyze,
+    exchange_name='bitfinex',
+    algo_namespace=algo_namespace,
+    base_currency='btc'
+)

catalyst/examples/dual_vwap.py

@@ -52,7 +52,7 @@ def initialize(context):
 
     schedule_function(
         rebalance,
-        date_rules.every_day(),
+        time_rules=times_rules.every_minute(),
     )
 
 

catalyst/examples/simple_loop.py

@@ -0,0 +1,51 @@
+import pandas as pd
+import talib
+
+from catalyst import run_algorithm
+from catalyst.api import symbol
+
+
+def initialize(context):
+    print('initializing')
+    context.asset = symbol('xrp_btc')
+
+
+def handle_data(context, data):
+    print('handling bar: {}'.format(data.current_dt))
+
+    price = data.current(context.asset, 'close')
+    print('got price {price}'.format(price=price))
+
+    prices = data.history(
+        context.asset,
+        fields='price',
+        bar_count=15,
+        frequency='1d'
+    )
+    rsi = talib.RSI(prices.values, timeperiod=14)[-1]
+    print('got rsi: {}'.format(rsi))
+    pass
+
+
+# run_algorithm(
+#     capital_base=250,
+#     start=pd.to_datetime('2015-08-01', utc=True),
+#     end=pd.to_datetime('2017-9-30', utc=True),
+#     data_frequency='daily',
+#     initialize=initialize,
+#     handle_data=handle_data,
+#     analyze=None,
+#     exchange_name='poloniex',
+#     algo_namespace='simple_loop',
+#     base_currency='eth'
+# )
+run_algorithm(
+    initialize=initialize,
+    handle_data=handle_data,
+    analyze=None,
+    exchange_name='bitfinex',
+    live=True,
+    algo_namespace='simple_loop',
+    base_currency='eth',
+    live_graph=False
+)

catalyst/exchange/asset_finder_exchange.py

@@ -0,0 +1,93 @@
+from logbook import Logger
+
+from catalyst.constants import LOG_LEVEL
+
+log = Logger('AssetFinderExchange', level=LOG_LEVEL)
+
+
+class AssetFinderExchange(object):
+    def __init__(self):
+        self._asset_cache = {}
+
+    @property
+    def sids(self):
+        """
+        This seems to be used to pre-fetch assets.
+        I don't think that we need this for live-trading.
+        Leaving the list empty.
+        """
+        return list()
+
+    def retrieve_all(self, sids, default_none=False):
+        """
+        Retrieve all assets in `sids`.
+
+        Parameters
+        ----------
+        sids : iterable of int
+            Assets to retrieve.
+        default_none : bool
+            If True, return None for failed lookups.
+            If False, raise `SidsNotFound`.
+
+        Returns
+        -------
+        assets : list[Asset or None]
+            A list of the same length as `sids` containing Assets (or Nones)
+            corresponding to the requested sids.
+
+        Raises
+        ------
+        SidsNotFound
+            When a requested sid is not found and default_none=False.
+        """
+        for sid in sids:
+            if sid in self._asset_cache:
+                log.debug('got asset from cache: {}'.format(sid))
+            else:
+                log.debug('fetching asset: {}'.format(sid))
+        return list()
+
+    def lookup_symbol(self, symbol, exchange, as_of_date=None, fuzzy=False):
+        """Lookup an asset by symbol.
+
+        Parameters
+        ----------
+        symbol : str
+            The ticker symbol to resolve.
+        as_of_date : datetime or None
+            Look up the last owner of this symbol as of this datetime.
+            If ``as_of_date`` is None, then this can only resolve the equity
+            if exactly one equity has ever owned the ticker.
+        fuzzy : bool, optional
+            Should fuzzy symbol matching be used? Fuzzy symbol matching
+            attempts to resolve differences in representations for
+            shareclasses. For example, some people may represent the ``A``
+            shareclass of ``BRK`` as ``BRK.A``, where others could write
+            ``BRK_A``.
+
+        Returns
+        -------
+        equity : Asset
+            The equity that held ``symbol`` on the given ``as_of_date``, or the
+            only equity to hold ``symbol`` if ``as_of_date`` is None.
+
+        Raises
+        ------
+        SymbolNotFound
+            Raised when no equity has ever held the given symbol.
+        MultipleSymbolsFound
+            Raised when no ``as_of_date`` is given and more than one equity
+            has held ``symbol``. This is also raised when ``fuzzy=True`` and
+            there are multiple candidates for the given ``symbol`` on the
+            ``as_of_date``.
+        """
+        log.debug('looking up symbol: {} {}'.format(symbol, exchange.name))
+
+        key = ','.join([exchange.name, symbol])
+        if key in self._asset_cache:
+            return self._asset_cache[key]
+        else:
+            asset = exchange.get_asset(symbol)
+            self._asset_cache[key] = asset
+            return asset

catalyst/exchange/bitfinex/bitfinex.py

@@ -0,0 +1,693 @@
+import base64
+import datetime
+import hashlib
+import hmac
+import json
+import re
+import time
+
+import numpy as np
+import pandas as pd
+import pytz
+import requests
+import six
+from catalyst.assets._assets import TradingPair
+from logbook import Logger
+
+from catalyst.exchange.exchange import Exchange
+from catalyst.exchange.exchange_bundle import ExchangeBundle
+from catalyst.exchange.exchange_errors import (
+    ExchangeRequestError,
+    InvalidHistoryFrequencyError,
+    InvalidOrderStyle, OrderCancelError)
+from catalyst.exchange.exchange_execution import ExchangeLimitOrder, \
+    ExchangeStopLimitOrder, ExchangeStopOrder
+from catalyst.exchange.exchange_utils import get_exchange_symbols_filename, \
+    download_exchange_symbols
+from catalyst.finance.order import Order, ORDER_STATUS
+from catalyst.protocol import Account
+
+# Trying to account for REST api instability
+# https://stackoverflow.com/questions/15431044/can-i-set-max-retries-for-requests-request
+requests.adapters.DEFAULT_RETRIES = 20
+
+BITFINEX_URL = 'https://api.bitfinex.com'
+
+from catalyst.constants import LOG_LEVEL
+
+log = Logger('Bitfinex', level=LOG_LEVEL)
+warning_logger = Logger('AlgoWarning')
+
+
+class Bitfinex(Exchange):
+    def __init__(self, key, secret, base_currency, portfolio=None):
+        self.url = BITFINEX_URL
+        self.key = key
+        self.secret = secret.encode('UTF-8')
+        self.name = 'bitfinex'
+        self.color = 'green'
+        self.assets = {}
+        self.load_assets()
+        self.base_currency = base_currency
+        self._portfolio = portfolio
+        self.minute_writer = None
+        self.minute_reader = None
+
+        # The candle limit for each request
+        self.num_candles_limit = 1000
+
+        # Max is 90 but playing it safe
+        # https://www.bitfinex.com/posts/188
+        self.max_requests_per_minute = 80
+        self.request_cpt = dict()
+
+        self.bundle = ExchangeBundle(self)
+
+    def _request(self, operation, data, version='v1'):
+        payload_object = {
+            'request': '/{}/{}'.format(version, operation),
+            'nonce': '{0:f}'.format(time.time() * 1000000),
+            # convert to string
+            'options': {}
+        }
+
+        if data is None:
+            payload_dict = payload_object
+        else:
+            payload_dict = payload_object.copy()
+            payload_dict.update(data)
+
+        payload_json = json.dumps(payload_dict)
+        if six.PY3:
+            payload = base64.b64encode(bytes(payload_json, 'utf-8'))
+        else:
+            payload = base64.b64encode(payload_json)
+
+        m = hmac.new(self.secret, payload, hashlib.sha384)
+        m = m.hexdigest()
+
+        # headers
+        headers = {
+            'X-BFX-APIKEY': self.key,
+            'X-BFX-PAYLOAD': payload,
+            'X-BFX-SIGNATURE': m
+        }
+
+        if data is None:
+            request = requests.get(
+                '{url}/{version}/{operation}'.format(
+                    url=self.url,
+                    version=version,
+                    operation=operation
+                ), data={},
+                headers=headers)
+        else:
+            request = requests.post(
+                '{url}/{version}/{operation}'.format(
+                    url=self.url,
+                    version=version,
+                    operation=operation
+                ),
+                headers=headers)
+
+        return request
+
+    def _get_v2_symbol(self, asset):
+        pair = asset.symbol.split('_')
+        symbol = 't' + pair[0].upper() + pair[1].upper()
+        return symbol
+
+    def _get_v2_symbols(self, assets):
+        """
+        Workaround to support Bitfinex v2
+        TODO: Might require a separate asset dictionary
+
+        :param assets:
+        :return:
+        """
+
+        v2_symbols = []
+        for asset in assets:
+            v2_symbols.append(self._get_v2_symbol(asset))
+
+        return v2_symbols
+
+    def _create_order(self, order_status):
+        """
+        Create a Catalyst order object from a Bitfinex order dictionary
+        :param order_status:
+        :return: Order
+        """
+        if order_status['is_cancelled']:
+            status = ORDER_STATUS.CANCELLED
+        elif not order_status['is_live']:
+            log.info('found executed order {}'.format(order_status))
+            status = ORDER_STATUS.FILLED
+        else:
+            status = ORDER_STATUS.OPEN
+
+        amount = float(order_status['original_amount'])
+        filled = float(order_status['executed_amount'])
+
+        if order_status['side'] == 'sell':
+            amount = -amount
+            filled = -filled
+
+        price = float(order_status['price'])
+        order_type = order_status['type']
+
+        stop_price = None
+        limit_price = None
+
+        # TODO: is this comprehensive enough?
+        if order_type.endswith('limit'):
+            limit_price = price
+        elif order_type.endswith('stop'):
+            stop_price = price
+
+        executed_price = float(order_status['avg_execution_price'])
+
+        # TODO: bitfinex does not specify comission. I could calculate it but not sure if it's worth it.
+        commission = None
+
+        date = pd.Timestamp.utcfromtimestamp(float(order_status['timestamp']))
+        date = pytz.utc.localize(date)
+        order = Order(
+            dt=date,
+            asset=self.assets[order_status['symbol']],
+            amount=amount,
+            stop=stop_price,
+            limit=limit_price,
+            filled=filled,
+            id=str(order_status['id']),
+            commission=commission
+        )
+        order.status = status
+
+        return order, executed_price
+
+    def get_balances(self):
+        log.debug('retrieving wallets balances')
+        try:
+            self.ask_request()
+            response = self._request('balances', None)
+            balances = response.json()
+        except Exception as e:
+            raise ExchangeRequestError(error=e)
+
+        if 'message' in balances:
+            raise ExchangeRequestError(
+                error='unable to fetch balance {}'.format(balances['message'])
+            )
+
+        std_balances = dict()
+        for balance in balances:
+            currency = balance['currency'].lower()
+            std_balances[currency] = float(balance['available'])
+
+        return std_balances
+
+    @property
+    def account(self):
+        account = Account()
+
+        account.settled_cash = None
+        account.accrued_interest = None
+        account.buying_power = None
+        account.equity_with_loan = None
+        account.total_positions_value = None
+        account.total_positions_exposure = None
+        account.regt_equity = None
+        account.regt_margin = None
+        account.initial_margin_requirement = None
+        account.maintenance_margin_requirement = None
+        account.available_funds = None
+        account.excess_liquidity = None
+        account.cushion = None
+        account.day_trades_remaining = None
+        account.leverage = None
+        account.net_leverage = None
+        account.net_liquidation = None
+
+        return account
+
+    @property
+    def time_skew(self):
+        # TODO: research the time skew conditions
+        return pd.Timedelta('0s')
+
+    def get_account(self):
+        # TODO: fetch account data and keep in cache
+        return None
+
+    def get_candles(self, data_frequency, assets, bar_count=None,
+                    start_dt=None, end_dt=None):
+        """
+        Retrieve OHLVC candles from Bitfinex
+
+        :param data_frequency:
+        :param assets:
+        :param bar_count:
+        :return:
+
+        Available Frequencies
+        ---------------------
+        '1m', '5m', '15m', '30m', '1h', '3h', '6h', '12h', '1D', '7D', '14D',
+         '1M'
+        """
+
+        freq_match = re.match(r'([0-9].*)(m|h|d)', data_frequency, re.M | re.I)
+        if freq_match:
+            number = int(freq_match.group(1))
+            unit = freq_match.group(2)
+
+            if unit == 'd':
+                converted_unit = 'D'
+            else:
+                converted_unit = unit
+
+            frequency = '{}{}'.format(number, converted_unit)
+            allowed_frequencies = ['1m', '5m', '15m', '30m', '1h', '3h', '6h',
+                                   '12h', '1D', '7D', '14D', '1M']
+
+            if frequency not in allowed_frequencies:
+                raise InvalidHistoryFrequencyError(
+                    frequency=data_frequency
+                )
+        elif data_frequency == 'minute':
+            frequency = '1m'
+        elif data_frequency == 'daily':
+            frequency = '1D'
+        else:
+            raise InvalidHistoryFrequencyError(
+                frequency=data_frequency
+            )
+
+        # Making sure that assets are iterable
+        asset_list = [assets] if isinstance(assets, TradingPair) else assets
+        ohlc_map = dict()
+        for asset in asset_list:
+            symbol = self._get_v2_symbol(asset)
+            url = '{url}/v2/candles/trade:{frequency}:{symbol}'.format(
+                url=self.url,
+                frequency=frequency,
+                symbol=symbol
+            )
+
+            if bar_count:
+                is_list = True
+                url += '/hist?limit={}'.format(int(bar_count))
+
+                def get_ms(date):
+                    epoch = datetime.datetime.utcfromtimestamp(0)
+                    epoch = epoch.replace(tzinfo=pytz.UTC)
+
+                    return (date - epoch).total_seconds() * 1000.0
+
+                if start_dt is not None:
+                    start_ms = get_ms(start_dt)
+                    url += '&start={0:f}'.format(start_ms)
+
+                if end_dt is not None:
+                    end_ms = get_ms(end_dt)
+                    url += '&end={0:f}'.format(end_ms)
+
+            else:
+                is_list = False
+                url += '/last'
+
+            try:
+                self.ask_request()
+                response = requests.get(url)
+            except Exception as e:
+                raise ExchangeRequestError(error=e)
+
+            if 'error' in response.content:
+                raise ExchangeRequestError(
+                    error='Unable to retrieve candles: {}'.format(
+                        response.content)
+                )
+
+            candles = response.json()
+
+            def ohlc_from_candle(candle):
+                last_traded = pd.Timestamp.utcfromtimestamp(
+                    candle[0] / 1000.0)
+                last_traded = last_traded.replace(tzinfo=pytz.UTC)
+                ohlc = dict(
+                    open=np.float64(candle[1]),
+                    high=np.float64(candle[3]),
+                    low=np.float64(candle[4]),
+                    close=np.float64(candle[2]),
+                    volume=np.float64(candle[5]),
+                    price=np.float64(candle[2]),
+                    last_traded=last_traded
+                )
+                return ohlc
+
+            if is_list:
+                ohlc_bars = []
+                # We can to list candles from old to new
+                for candle in reversed(candles):
+                    ohlc = ohlc_from_candle(candle)
+                    ohlc_bars.append(ohlc)
+
+                ohlc_map[asset] = ohlc_bars
+
+            else:
+                ohlc = ohlc_from_candle(candles)
+                ohlc_map[asset] = ohlc
+
+        return ohlc_map[assets] \
+            if isinstance(assets, TradingPair) else ohlc_map
+
+    def create_order(self, asset, amount, is_buy, style):
+        """
+        Creating order on the exchange.
+
+        :param asset:
+        :param amount:
+        :param is_buy:
+        :param style:
+        :return:
+        """
+        exchange_symbol = self.get_symbol(asset)
+        if isinstance(style, ExchangeLimitOrder) \
+                or isinstance(style, ExchangeStopLimitOrder):
+            price = style.get_limit_price(is_buy)
+            order_type = 'limit'
+
+        elif isinstance(style, ExchangeStopOrder):
+            price = style.get_stop_price(is_buy)
+            order_type = 'stop'
+
+        else:
+            raise InvalidOrderStyle(exchange=self.name,
+                                    style=style.__class__.__name__)
+
+        req = dict(
+            symbol=exchange_symbol,
+            amount=str(float(abs(amount))),
+            price="{:.20f}".format(float(price)),
+            side='buy' if is_buy else 'sell',
+            type='exchange ' + order_type,  # TODO: support margin trades
+            exchange=self.name,
+            is_hidden=False,
+            is_postonly=False,
+            use_all_available=0,
+            ocoorder=False,
+            buy_price_oco=0,
+            sell_price_oco=0
+        )
+
+        date = pd.Timestamp.utcnow()
+        try:
+            self.ask_request()
+            response = self._request('order/new', req)
+            order_status = response.json()
+        except Exception as e:
+            raise ExchangeRequestError(error=e)
+
+        if 'message' in order_status:
+            raise ExchangeRequestError(
+                error='unable to create Bitfinex order {}'.format(
+                    order_status['message'])
+            )
+
+        order_id = str(order_status['id'])
+        order = Order(
+            dt=date,
+            asset=asset,
+            amount=amount,
+            stop=style.get_stop_price(is_buy),
+            limit=style.get_limit_price(is_buy),
+            id=order_id
+        )
+
+        return order
+
+    def get_open_orders(self, asset=None):
+        """Retrieve all of the current open orders.
+
+        Parameters
+        ----------
+        asset : Asset
+            If passed and not None, return only the open orders for the given
+            asset instead of all open orders.
+
+        Returns
+        -------
+        open_orders : dict[list[Order]] or list[Order]
+            If no asset is passed this will return a dict mapping Assets
+            to a list containing all the open orders for the asset.
+            If an asset is passed then this will return a list of the open
+            orders for this asset.
+        """
+        try:
+            self.ask_request()
+            response = self._request('orders', None)
+            order_statuses = response.json()
+        except Exception as e:
+            raise ExchangeRequestError(error=e)
+
+        if 'message' in order_statuses:
+            raise ExchangeRequestError(
+                error='Unable to retrieve open orders: {}'.format(
+                    order_statuses['message'])
+            )
+
+        orders = []
+        for order_status in order_statuses:
+            order, executed_price = self._create_order(order_status)
+            if asset is None or asset == order.sid:
+                orders.append(order)
+
+        return orders
+
+    def get_order(self, order_id):
+        """Lookup an order based on the order id returned from one of the
+        order functions.
+
+        Parameters
+        ----------
+        order_id : str
+            The unique identifier for the order.
+
+        Returns
+        -------
+        order : Order
+            The order object.
+        """
+        try:
+            self.ask_request()
+            response = self._request(
+                'order/status', {'order_id': int(order_id)})
+            order_status = response.json()
+        except Exception as e:
+            raise ExchangeRequestError(error=e)
+
+        if 'message' in order_status:
+            raise ExchangeRequestError(
+                error='Unable to retrieve order status: {}'.format(
+                    order_status['message'])
+            )
+        return self._create_order(order_status)
+
+    def cancel_order(self, order_param):
+        """Cancel an open order.
+
+        Parameters
+        ----------
+        order_param : str or Order
+            The order_id or order object to cancel.
+        """
+        order_id = order_param.id \
+            if isinstance(order_param, Order) else order_param
+
+        try:
+            self.ask_request()
+            response = self._request('order/cancel', {'order_id': order_id})
+            status = response.json()
+        except Exception as e:
+            raise ExchangeRequestError(error=e)
+
+        if 'message' in status:
+            raise OrderCancelError(
+                order_id=order_id,
+                exchange=self.name,
+                error=status['message']
+            )
+
+    def tickers(self, assets):
+        """
+        Fetch ticket data for assets
+        https://docs.bitfinex.com/v2/reference#rest-public-tickers
+
+        :param assets:
+        :return:
+        """
+        symbols = self._get_v2_symbols(assets)
+        log.debug('fetching tickers {}'.format(symbols))
+
+        try:
+            self.ask_request()
+            response = requests.get(
+                '{url}/v2/tickers?symbols={symbols}'.format(
+                    url=self.url,
+                    symbols=','.join(symbols),
+                )
+            )
+        except Exception as e:
+            raise ExchangeRequestError(error=e)
+
+        if 'error' in response.content:
+            raise ExchangeRequestError(
+                error='Unable to retrieve tickers: {}'.format(
+                    response.content)
+            )
+
+        try:
+            tickers = response.json()
+        except Exception as e:
+            raise ExchangeRequestError(error=e)
+
+        ticks = dict()
+        for index, ticker in enumerate(tickers):
+            if not len(ticker) == 11:
+                raise ExchangeRequestError(
+                    error='Invalid ticker in response: {}'.format(ticker)
+                )
+
+            ticks[assets[index]] = dict(
+                timestamp=pd.Timestamp.utcnow(),
+                bid=ticker[1],
+                ask=ticker[3],
+                last_price=ticker[7],
+                low=ticker[10],
+                high=ticker[9],
+                volume=ticker[8],
+            )
+
+        log.debug('got tickers {}'.format(ticks))
+        return ticks
+
+    def generate_symbols_json(self, filename=None, source_dates=False):
+        symbol_map = {}
+
+        if not source_dates:
+            fn, r = download_exchange_symbols(self.name)
+            with open(fn) as data_file:
+                cached_symbols = json.load(data_file)
+
+        response = self._request('symbols', None)
+
+        for symbol in response.json():
+            if (source_dates):
+                start_date = self.get_symbol_start_date(symbol)
+            else:
+                try:
+                    start_date = cached_symbols[symbol]['start_date']
+                except KeyError as e:
+                    start_date = time.strftime('%Y-%m-%d')
+
+            try:
+                end_daily = cached_symbols[symbol]['end_daily']
+            except KeyError as e:
+                end_daily = 'N/A'
+
+            try:
+                end_minute = cached_symbols[symbol]['end_minute']
+            except KeyError as e:
+                end_minute = 'N/A'
+
+            symbol_map[symbol] = dict(
+                symbol=symbol[:-3] + '_' + symbol[-3:],
+                start_date=start_date,
+                end_daily=end_daily,
+                end_minute=end_minute,
+            )
+
+        if (filename is None):
+            filename = get_exchange_symbols_filename(self.name)
+
+        with open(filename, 'w') as f:
+            json.dump(symbol_map, f, sort_keys=True, indent=2,
+                      separators=(',', ':'))
+
+    def get_symbol_start_date(self, symbol):
+
+        print(symbol)
+        symbol_v2 = 't' + symbol.upper()
+
+        """
+            For each symbol we retrieve candles with Monhtly resolution
+            We get the first month, and query again with daily resolution
+            around that date, and we get the first date
+        """
+        url = '{url}/v2/candles/trade:1M:{symbol}/hist'.format(
+            url=self.url,
+            symbol=symbol_v2
+        )
+
+        try:
+            self.ask_request()
+            response = requests.get(url)
+        except Exception as e:
+            raise ExchangeRequestError(error=e)
+
+        """
+            If we don't get any data back for our monthly-resolution query
+            it means that symbol started trading less than a month ago, so
+            arbitrarily set the ref. date to 15 days ago to be safe with
+            +/- 31 days
+        """
+        if (len(response.json())):
+            startmonth = response.json()[-1][0]
+        else:
+            startmonth = int((time.time() - 15 * 24 * 3600) * 1000)
+
+        """
+            Query again with daily resolution setting the start and end around
+            the startmonth we got above. Avoid end dates greater than now: time.time()
+        """
+        url = '{url}/v2/candles/trade:1D:{symbol}/hist?start={start}&end={end}'.format(
+            url=self.url,
+            symbol=symbol_v2,
+            start=startmonth - 3600 * 24 * 31 * 1000,
+            end=min(startmonth + 3600 * 24 * 31 * 1000,
+                    int(time.time() * 1000))
+        )
+
+        try:
+            self.ask_request()
+            response = requests.get(url)
+        except Exception as e:
+            raise ExchangeRequestError(error=e)
+
+        return time.strftime('%Y-%m-%d',
+                             time.gmtime(int(response.json()[-1][0] / 1000)))
+
+    def get_orderbook(self, asset, order_type='all', limit=100):
+        exchange_symbol = asset.exchange_symbol
+        try:
+            self.ask_request()
+            # TODO: implement limit
+            response = self._request(
+                'book/{}'.format(exchange_symbol), None)
+            data = response.json()
+
+        except Exception as e:
+            raise ExchangeRequestError(error=e)
+
+        # TODO: filter by type
+        result = dict()
+        for order_type in data:
+            result[order_type] = []
+
+            for entry in data[order_type]:
+                result[order_type].append(dict(
+                    rate=float(entry['price']),
+                    quantity=float(entry['amount'])
+                ))
+
+        return result

catalyst/exchange/bitfinex/symbols.json

@@ -0,0 +1,127 @@
+{
+  "neobtc": {
+    "symbol": "neo_btc",
+    "start_date": "2017-09-07",
+    "precision": 5
+  },
+  "neousd": {
+    "symbol": "neo_usd",
+    "start_date": "2017-09-07"
+  },
+  "neoeth": {
+    "symbol": "neo_eth",
+    "start_date": "2017-09-07"
+  },
+  "btcusd": {
+    "symbol": "btc_usd",
+    "start_date": "2010-01-01"
+  },
+  "bchusd": {
+    "symbol": "bch_usd",
+    "start_date": "2010-01-01"
+  },
+  "ltcusd": {
+    "symbol": "ltc_usd",
+    "start_date": "2010-01-01"
+  },
+  "ltcbtc": {
+    "symbol": "ltc_btc",
+    "start_date": "2010-01-01"
+  },
+  "ethusd": {
+    "symbol": "eth_usd",
+    "start_date": "2017-01-01"
+  },
+  "ethbtc": {
+    "symbol": "eth_btc",
+    "start_date": "2017-01-01"
+  },
+  "etcbtc": {
+    "symbol": "etc_btc",
+    "start_date": "2017-01-01"
+  },
+  "etcusd": {
+    "symbol": "etc_usd",
+    "start_date": "2017-01-01"
+  },
+  "rrtusd": {
+    "symbol": "rrt_usd",
+    "start_date": "2010-01-01"
+  },
+  "rrtbtc": {
+    "symbol": "rrt_btc",
+    "start_date": "2010-01-01"
+  },
+  "zecusd": {
+    "symbol": "zec_usd",
+    "start_date": "2010-01-01"
+  },
+  "zecbtc": {
+    "symbol": "zec_btc",
+    "start_date": "2010-01-01"
+  },
+  "xmrusd": {
+    "symbol": "xmr_usd",
+    "start_date": "2010-01-01"
+  },
+  "xmrbtc": {
+    "symbol": "xmr_btc",
+    "start_date": "2010-01-01"
+  },
+  "dshusd": {
+    "symbol": "dsh_usd",
+    "start_date": "2010-01-01"
+  },
+  "dshbtc": {
+    "symbol": "dsh_btc",
+    "start_date": "2010-01-01"
+  },
+  "bccbtc": {
+    "symbol": "bcc_btc",
+    "start_date": "2010-01-01"
+  },
+  "bcubtc": {
+    "symbol": "bcu_btc",
+    "start_date": "2010-01-01"
+  },
+  "bccusd": {
+    "symbol": "bcc_usd",
+    "start_date": "2010-01-01"
+  },
+  "bcuusd": {
+    "symbol": "bcu_usd",
+    "start_date": "2010-01-01"
+  },
+  "xrpusd": {
+    "symbol": "xrp_usd",
+    "start_date": "2010-01-01"
+  },
+  "xrpbtc": {
+    "symbol": "xrp_btc",
+    "start_date": "2010-01-01"
+  },
+  "iotusd": {
+    "symbol": "iot_usd",
+    "start_date": "2010-01-01"
+  },
+  "iotbtc": {
+    "symbol": "iot_btc",
+    "start_date": "2010-01-01"
+  },
+  "ioteth": {
+    "symbol": "iot_eth",
+    "start_date": "2010-01-01"
+  },
+  "eosusd": {
+    "symbol": "eos_usd",
+    "start_date": "2010-01-01"
+  },
+  "eosbtc": {
+    "symbol": "eos_btc",
+    "start_date": "2010-01-01"
+  },
+  "eoseth": {
+    "symbol": "eos_eth",
+    "start_date": "2010-01-01"
+  }
+}

catalyst/exchange/bittrex/bittrex.py

@@ -0,0 +1,393 @@
+import json
+
+import pandas as pd
+from catalyst.assets._assets import TradingPair
+from logbook import Logger
+from six.moves import urllib
+
+from catalyst.constants import LOG_LEVEL
+from catalyst.exchange.bittrex.bittrex_api import Bittrex_api
+from catalyst.exchange.exchange import Exchange
+from catalyst.exchange.exchange_bundle import ExchangeBundle
+from catalyst.exchange.exchange_errors import InvalidHistoryFrequencyError, \
+    ExchangeRequestError, InvalidOrderStyle, OrderNotFound, OrderCancelError, \
+    CreateOrderError
+from catalyst.exchange.exchange_utils import get_exchange_symbols_filename, \
+    download_exchange_symbols
+from catalyst.finance.execution import LimitOrder, StopLimitOrder
+from catalyst.finance.order import Order, ORDER_STATUS
+
+log = Logger('Bittrex', level=LOG_LEVEL)
+
+URL2 = 'https://bittrex.com/Api/v2.0'
+
+
+class Bittrex(Exchange):
+    def __init__(self, key, secret, base_currency, portfolio=None):
+        self.api = Bittrex_api(key=key, secret=secret.encode('UTF-8'))
+        self.name = 'bittrex'
+        self.color = 'blue'
+        self.base_currency = base_currency
+        self._portfolio = portfolio
+
+        self.num_candles_limit = 2000
+
+        # Not sure what the rate limit is but trying to play it safe
+        # https://bitcoin.stackexchange.com/questions/53778/bittrex-api-rate-limit
+        self.max_requests_per_minute = 60
+        self.request_cpt = dict()
+
+        self.minute_writer = None
+        self.minute_reader = None
+
+        self.assets = dict()
+        self.load_assets()
+
+        self.bundle = ExchangeBundle(self)
+
+    @property
+    def account(self):
+        pass
+
+    @property
+    def time_skew(self):
+        # TODO: research the time skew conditions
+        return pd.Timedelta('0s')
+
+    def sanitize_curency_symbol(self, exchange_symbol):
+        """
+        Helper method used to build the universal pair.
+        Include any symbol mapping here if appropriate.
+
+        :param exchange_symbol:
+        :return universal_symbol:
+        """
+        return exchange_symbol.lower()
+
+    def get_balances(self):
+        try:
+            log.debug('retrieving wallet balances')
+            self.ask_request()
+            balances = self.api.getbalances()
+
+        except Exception as e:
+            raise ExchangeRequestError(error=e)
+
+        std_balances = dict()
+        try:
+            for balance in balances:
+                currency = balance['Currency'].lower()
+                std_balances[currency] = balance['Available']
+
+        except TypeError:
+            raise ExchangeRequestError(error=balances)
+
+        return std_balances
+
+    def create_order(self, asset, amount, is_buy, style):
+        log.info('creating {} order'.format('buy' if is_buy else 'sell'))
+        exchange_symbol = self.get_symbol(asset)
+
+        if isinstance(style, LimitOrder) or isinstance(style, StopLimitOrder):
+            if isinstance(style, StopLimitOrder):
+                log.warn('{} will ignore the stop price'.format(self.name))
+
+            price = style.get_limit_price(is_buy)
+            try:
+                self.ask_request()
+                if is_buy:
+                    order_status = self.api.buylimit(exchange_symbol, amount,
+                                                     price)
+                else:
+                    order_status = self.api.selllimit(exchange_symbol,
+                                                      abs(amount), price)
+            except Exception as e:
+                raise ExchangeRequestError(error=e)
+
+            if 'uuid' in order_status:
+                order_id = order_status['uuid']
+                order = Order(
+                    dt=pd.Timestamp.utcnow(),
+                    asset=asset,
+                    amount=amount,
+                    stop=style.get_stop_price(is_buy),
+                    limit=style.get_limit_price(is_buy),
+                    id=order_id
+                )
+                return order
+            else:
+                if order_status == 'INSUFFICIENT_FUNDS':
+                    log.warn('not enough funds to create order')
+                    return None
+                elif order_status == 'DUST_TRADE_DISALLOWED_MIN_VALUE_50K_SAT':
+                    log.warn('Your order is too small, order at least 50K'
+                             ' Satoshi')
+                    return None
+                else:
+                    raise CreateOrderError(
+                        exchange=self.name,
+                        error=order_status
+                    )
+        else:
+            raise InvalidOrderStyle(exchange=self.name,
+                                    style=style.__class__.__name__)
+
+    def get_open_orders(self, asset):
+        symbol = self.get_symbol(asset)
+        try:
+            self.ask_request()
+            open_orders = self.api.getopenorders(symbol)
+        except Exception as e:
+            raise ExchangeRequestError(error=e)
+
+        orders = list()
+        for order_status in open_orders:
+            order = self._create_order(order_status)
+            orders.append(order)
+
+        return orders
+
+    def _create_order(self, order_status):
+        log.info(
+            'creating catalyst order from Bittrex {}'.format(order_status))
+        if order_status['CancelInitiated']:
+            status = ORDER_STATUS.CANCELLED
+        elif order_status['Closed'] is not None:
+            status = ORDER_STATUS.FILLED
+        else:
+            status = ORDER_STATUS.OPEN
+
+        date = pd.to_datetime(order_status['Opened'], utc=True)
+        amount = order_status['Quantity']
+        filled = amount - order_status['QuantityRemaining']
+        order = Order(
+            dt=date,
+            asset=self.assets[order_status['Exchange']],
+            amount=amount,
+            stop=None,  # Not yet supported by Bittrex
+            limit=order_status['Limit'],
+            filled=filled,
+            id=order_status['OrderUuid'],
+            commission=order_status['CommissionPaid']
+        )
+        order.status = status
+
+        executed_price = order_status['PricePerUnit']
+
+        return order, executed_price
+
+    def get_order(self, order_id):
+        log.info('retrieving order {}'.format(order_id))
+        try:
+            self.ask_request()
+            order_status = self.api.getorder(order_id)
+        except Exception as e:
+            raise ExchangeRequestError(error=e)
+
+        if order_status is None:
+            raise OrderNotFound(order_id=order_id, exchange=self.name)
+
+        return self._create_order(order_status)
+
+    def cancel_order(self, order_param):
+        order_id = order_param.id \
+            if isinstance(order_param, Order) else order_param
+        log.info('cancelling order {}'.format(order_id))
+
+        try:
+            self.ask_request()
+            status = self.api.cancel(order_id)
+        except Exception as e:
+            raise ExchangeRequestError(error=e)
+
+        if 'message' in status:
+            raise OrderCancelError(
+                order_id=order_id,
+                exchange=self.name,
+                error=status['message']
+            )
+
+    def get_candles(self, data_frequency, assets, bar_count=None,
+                    start_date=None):
+        """
+        Supported Intervals
+        -------------------
+        day, oneMin, fiveMin, thirtyMin, hour
+
+        :param data_frequency:
+        :param assets:
+        :param bar_count:
+        :return:
+        """
+        log.info('retrieving candles')
+
+        if data_frequency == 'minute' or data_frequency == '1m':
+            frequency = 'oneMin'
+        elif data_frequency == '5m':
+            frequency = 'fiveMin'
+        elif data_frequency == '30m':
+            frequency = 'thirtyMin'
+        elif data_frequency == '1h':
+            frequency = 'hour'
+        elif data_frequency == 'daily' or data_frequency == '1D':
+            frequency = 'day'
+        else:
+            raise InvalidHistoryFrequencyError(
+                frequency=data_frequency
+            )
+
+        # Making sure that assets are iterable
+        asset_list = [assets] if isinstance(assets, TradingPair) else assets
+        ohlc_map = dict()
+        for asset in asset_list:
+            url = '{url}/pub/market/GetTicks?marketName={symbol}' \
+                  '&tickInterval={frequency}&_=1499127220008'.format(
+                url=URL2,
+                symbol=self.get_symbol(asset),
+                frequency=frequency
+            )
+
+            try:
+                data = json.loads(urllib.request.urlopen(url).read().decode())
+            except Exception as e:
+                raise ExchangeRequestError(error=e)
+
+            if data['message']:
+                raise ExchangeRequestError(
+                    error='Unable to fetch candles {}'.format(data['message'])
+                )
+
+            candles = data['result']
+
+            def ohlc_from_candle(candle):
+                ohlc = dict(
+                    open=candle['O'],
+                    high=candle['H'],
+                    low=candle['L'],
+                    close=candle['C'],
+                    volume=candle['V'],
+                    price=candle['C'],
+                    last_traded=pd.to_datetime(candle['T'], utc=True)
+                )
+                return ohlc
+
+            ordered_candles = list(reversed(candles))
+            if bar_count is None:
+                ohlc_map[asset] = ohlc_from_candle(ordered_candles[0])
+            else:
+                ohlc_bars = []
+                for candle in ordered_candles[:bar_count]:
+                    ohlc = ohlc_from_candle(candle)
+                    ohlc_bars.append(ohlc)
+
+                ohlc_map[asset] = ohlc_bars
+
+        return ohlc_map[assets] \
+            if isinstance(assets, TradingPair) else ohlc_map
+
+    def tickers(self, assets):
+        """
+        As of v1.1, Bittrex only allows one ticker at the time.
+        So we have to make multiple calls to fetch multiple assets.
+
+        :param assets:
+        :return:
+        """
+        log.info('retrieving tickers')
+
+        ticks = dict()
+        for asset in assets:
+            symbol = self.get_symbol(asset)
+            try:
+                self.ask_request()
+                ticker = self.api.getticker(symbol)
+            except Exception as e:
+                raise ExchangeRequestError(error=e)
+
+            # TODO: catch invalid ticker
+            ticks[asset] = dict(
+                timestamp=pd.Timestamp.utcnow(),
+                bid=ticker['Bid'],
+                ask=ticker['Ask'],
+                last_price=ticker['Last']
+            )
+
+        log.debug('got tickers {}'.format(ticks))
+        return ticks
+
+    def get_account(self):
+        log.info('retrieving account data')
+        pass
+
+    def generate_symbols_json(self, filename=None):
+        symbol_map = {}
+
+        fn, r = download_exchange_symbols(self.name)
+        with open(fn) as data_file:
+            cached_symbols = json.load(data_file)
+
+        markets = self.api.getmarkets()
+        for market in markets:
+            exchange_symbol = market['MarketName']
+            symbol = '{market}_{base}'.format(
+                market=self.sanitize_curency_symbol(market['MarketCurrency']),
+                base=self.sanitize_curency_symbol(market['BaseCurrency'])
+            )
+
+            try:
+                end_daily = cached_symbols[exchange_symbol]['end_daily']
+            except KeyError as e:
+                end_daily = 'N/A'
+
+            try:
+                end_minute = cached_symbols[exchange_symbol]['end_minute']
+            except KeyError as e:
+                end_minute = 'N/A'
+
+            symbol_map[exchange_symbol] = dict(
+                symbol=symbol,
+                start_date=pd.to_datetime(market['Created'],
+                                          utc=True).strftime("%Y-%m-%d"),
+                end_daily=end_daily,
+                end_minute=end_minute,
+            )
+
+        if (filename is None):
+            filename = get_exchange_symbols_filename(self.name)
+
+        with open(filename, 'w') as f:
+            json.dump(symbol_map, f, sort_keys=True, indent=2,
+                      separators=(',', ':'))
+
+    def get_orderbook(self, asset, order_type='all', limit=100):
+        if order_type == 'all':
+            order_type = 'both'
+        elif order_type == 'bid':
+            order_type = 'buy'
+        elif order_type == 'ask':
+            order_type = 'sell'
+        else:
+            raise ValueError('invalid type')
+
+        exchange_symbol = asset.exchange_symbol
+        data = self.api.getorderbook(
+            market=exchange_symbol,
+            type=order_type,
+            depth=100
+        )
+
+        result = dict()
+        for exchange_type in data:
+            if exchange_type == 'buy':
+                order_type = 'bids'
+            elif exchange_type == 'sell':
+                order_type = 'asks'
+
+            result[order_type] = []
+            for entry in data[exchange_type]:
+                result[order_type].append(dict(
+                    rate=entry['Rate'],
+                    quantity=entry['Quantity']
+                ))
+
+        return result

catalyst/exchange/bittrex/bittrex_api.py

@@ -0,0 +1,127 @@
+#!/usr/bin/env python
+import json
+import time
+import hmac
+import hashlib
+
+from six.moves import urllib
+
+# Workaround for backwards compatibility
+# https://stackoverflow.com/questions/3745771/urllib-request-in-python-2-7
+urlopen = urllib.request.urlopen
+
+
+class Bittrex_api(object):
+    def __init__(self, key, secret):
+        self.key = key
+        self.secret = secret
+        self.public = ['getmarkets', 'getcurrencies', 'getticker',
+                       'getmarketsummaries', 'getmarketsummary',
+                       'getorderbook', 'getmarkethistory']
+        self.market = ['buylimit', 'buymarket', 'selllimit', 'sellmarket',
+                       'cancel', 'getopenorders']
+        self.account = ['getbalances', 'getbalance', 'getdepositaddress',
+                        'withdraw', 'getorder', 'getorderhistory',
+                        'getwithdrawalhistory', 'getdeposithistory']
+
+    def query(self, method, values={}):
+        if method in self.public:
+            url = 'https://bittrex.com/api/v1.1/public/'
+        elif method in self.market:
+            url = 'https://bittrex.com/api/v1.1/market/'
+        elif method in self.account:
+            url = 'https://bittrex.com/api/v1.1/account/'
+        else:
+            return 'Something went wrong, sorry.'
+
+        url += method + '?' + urllib.parse.urlencode(values)
+
+        if method not in self.public:
+            url += '&apikey=' + self.key
+            url += '&nonce=' + str(int(time.time()))
+            signature = hmac.new(self.secret, url, hashlib.sha512).hexdigest()
+            headers = {'apisign': signature}
+        else:
+            headers = {}
+
+        req = urllib.request.Request(url, headers=headers)
+        response = json.loads(urlopen(req).read())
+
+        if response["result"]:
+            return response["result"]
+        else:
+            return response["message"]
+
+    def getmarkets(self):
+        return self.query('getmarkets')
+
+    def getcurrencies(self):
+        return self.query('getcurrencies')
+
+    def getticker(self, market):
+        return self.query('getticker', {'market': market})
+
+    def getmarketsummaries(self):
+        return self.query('getmarketsummaries')
+
+    def getmarketsummary(self, market):
+        return self.query('getmarketsummary', {'market': market})
+
+    def getorderbook(self, market, type, depth=20):
+        return self.query('getorderbook',
+                          {'market': market, 'type': type, 'depth': depth})
+
+    def getmarkethistory(self, market, count=20):
+        return self.query('getmarkethistory',
+                          {'market': market, 'count': count})
+
+    def buylimit(self, market, quantity, rate):
+        return self.query('buylimit', {'market': market, 'quantity': quantity,
+                                       'rate': rate})
+
+    def buymarket(self, market, quantity):
+        return self.query('buymarket',
+                          {'market': market, 'quantity': quantity})
+
+    def selllimit(self, market, quantity, rate):
+        return self.query('selllimit', {'market': market, 'quantity': quantity,
+                                        'rate': rate})
+
+    def sellmarket(self, market, quantity):
+        return self.query('sellmarket',
+                          {'market': market, 'quantity': quantity})
+
+    def cancel(self, uuid):
+        return self.query('cancel', {'uuid': uuid})
+
+    def getopenorders(self, market):
+        return self.query('getopenorders', {'market': market})
+
+    def getbalances(self):
+        return self.query('getbalances')
+
+    def getbalance(self, currency):
+        return self.query('getbalance', {'currency': currency})
+
+    def getdepositaddress(self, currency):
+        return self.query('getdepositaddress', {'currency': currency})
+
+    def withdraw(self, currency, quantity, address):
+        return self.query('withdraw',
+                          {'currency': currency, 'quantity': quantity,
+                           'address': address})
+
+    def getorder(self, uuid):
+        return self.query('getorder', {'uuid': uuid})
+
+    def getorderhistory(self, market, count):
+        return self.query('getorderhistory',
+                          {'market': market, 'count': count})
+
+    def getwithdrawalhistory(self, currency, count):
+        return self.query('getwithdrawalhistory',
+                          {'currency': currency, 'count': count})
+
+    def getdeposithistory(self, currency, count):
+        return self.query('getdeposithistory',
+                          {'currency': currency, 'count': count})

catalyst/exchange/bittrex/extensions-example.py

@@ -0,0 +1,7 @@
+from catalyst.data.bundles import register
+from catalyst.exchange.exchange_bundle import exchange_bundle
+
+symbols = (
+    'neo_btc',
+)
+register('exchange_bitfinex', exchange_bundle('bitfinex', symbols))

catalyst/exchange/bundle_utils.py

@@ -0,0 +1,254 @@
+import calendar
+import os
+import tarfile
+from datetime import timedelta, datetime, date
+
+import numpy as np
+import pandas as pd
+import pytz
+
+from catalyst.data.bundles import from_bundle_ingest_dirname
+from catalyst.data.bundles.core import download_without_progress
+from catalyst.exchange.exchange_errors import NoDataAvailableOnExchange
+from catalyst.exchange.exchange_utils import get_exchange_bundles_folder
+from catalyst.utils.deprecate import deprecated
+from catalyst.utils.paths import data_path
+
+EXCHANGE_NAMES = ['bitfinex', 'bittrex', 'poloniex']
+API_URL = 'http://data.enigma.co/api/v1'
+
+
+def get_date_from_ms(ms):
+    return datetime.fromtimestamp(ms / 1000.0)
+
+
+def get_seconds_from_date(date):
+    epoch = datetime.utcfromtimestamp(0)
+    epoch = epoch.replace(tzinfo=pytz.UTC)
+
+    return int((date - epoch).total_seconds())
+
+
+def get_bcolz_chunk(exchange_name, symbol, data_frequency, period):
+    """
+    Download and extract a bcolz bundle.
+
+    :param exchange_name:
+    :param symbol:
+    :param data_frequency:
+    :param period:
+    :return:
+
+    Note:
+        Filename: bitfinex-daily-neo_eth-2017-10.tar.gz
+    """
+
+    root = get_exchange_bundles_folder(exchange_name)
+    name = '{exchange}-{frequency}-{symbol}-{period}'.format(
+        exchange=exchange_name,
+        frequency=data_frequency,
+        symbol=symbol,
+        period=period
+    )
+    path = os.path.join(root, name)
+
+    if not os.path.isdir(path):
+        url = 'https://s3.amazonaws.com/enigmaco/catalyst-bundles/' \
+              'exchange-{exchange}/{name}.tar.gz'.format(
+            exchange=exchange_name,
+            name=name
+        )
+
+        bytes = download_without_progress(url)
+        with tarfile.open('r', fileobj=bytes) as tar:
+            tar.extractall(path)
+
+    return path
+
+
+def get_delta(periods, data_frequency):
+    return timedelta(minutes=periods) \
+        if data_frequency == 'minute' else timedelta(days=periods)
+
+
+def get_periods_range(start_dt, end_dt, data_frequency):
+    freq = 'T' if data_frequency == 'minute' else 'D'
+
+    return pd.date_range(start_dt, end_dt, freq=freq)
+
+
+def get_periods(start_dt, end_dt, data_frequency):
+    delta = end_dt - start_dt
+
+    if data_frequency == 'minute':
+        delta_periods = delta.total_seconds() / 60
+
+    elif data_frequency == 'daily':
+        delta_periods = delta.total_seconds() / 60 / 60 / 24
+
+    else:
+        raise ValueError('frequency not supported')
+
+    return int(delta_periods)
+
+
+def get_start_dt(end_dt, bar_count, data_frequency):
+    periods = bar_count
+    if periods > 1:
+        delta = get_delta(periods, data_frequency)
+        start_dt = end_dt - delta
+    else:
+        start_dt = end_dt
+
+    return start_dt
+
+
+def get_adj_dates(start, end, assets, data_frequency):
+    """
+    Contains a date range to the trading availability of the specified pairs.
+
+    :param start:
+    :param end:
+    :param assets:
+    :param data_frequency:
+    :return:
+    """
+    earliest_trade = None
+    last_entry = None
+    for asset in assets:
+        if earliest_trade is None or earliest_trade > asset.start_date:
+            earliest_trade = asset.start_date
+
+        end_asset = asset.end_minute if data_frequency == 'minute' else \
+            asset.end_daily
+        if end_asset is not None and \
+                (last_entry is None or end_asset > last_entry):
+            last_entry = end_asset
+
+    if start is None or earliest_trade > start:
+        start = earliest_trade
+
+    if end is None or (last_entry is not None and end > last_entry):
+        end = last_entry
+
+    if end is None or start >= end:
+        raise NoDataAvailableOnExchange(
+            exchange=asset.exchange.title(),
+            symbol=[asset.symbol.encode('utf-8')],
+            data_frequency=data_frequency,
+        )
+
+    return start, end
+
+
+def get_month_start_end(dt):
+    """
+    Returns the first and last day of the month for the specified date.
+
+    :param dt:
+    :return:
+    """
+    month_range = calendar.monthrange(dt.year, dt.month)
+    month_start = pd.to_datetime(datetime(
+        dt.year, dt.month, 1, 0, 0, 0, 0
+    ), utc=True)
+
+    month_end = pd.to_datetime(datetime(
+        dt.year, dt.month, month_range[1], 23, 59, 0, 0
+    ), utc=True)
+
+    return month_start, month_end
+
+
+def get_year_start_end(dt):
+    """
+    Returns the first and last day of the year for the specified date.
+
+    :param dt:
+    :return:
+    """
+    year_start = pd.to_datetime(date(dt.year, 1, 1), utc=True)
+    year_end = pd.to_datetime(date(dt.year, 12, 31), utc=True)
+
+    return year_start, year_end
+
+
+def get_df_from_arrays(arrays, periods):
+    ohlcv = dict()
+    for index, field in enumerate(
+            ['open', 'high', 'low', 'close', 'volume']):
+        ohlcv[field] = arrays[index].flatten()
+
+    df = pd.DataFrame(
+        data=ohlcv,
+        index=periods
+    )
+    return df
+
+
+def range_in_bundle(asset, start_dt, end_dt, reader):
+    """
+    Evaluate whether price data of an asset is included has been ingested in
+    the exchange bundle for the given date range.
+
+    :param asset:
+    :param start_dt:
+    :param end_dt:
+    :param reader:
+    :return:
+    """
+    has_data = True
+    if has_data and reader is not None:
+        try:
+            start_close = \
+                reader.get_value(asset.sid, start_dt, 'close')
+
+            if np.isnan(start_close):
+                has_data = False
+
+            else:
+                end_close = reader.get_value(asset.sid, end_dt, 'close')
+
+                if np.isnan(end_close):
+                    has_data = False
+
+        except Exception as e:
+            has_data = False
+
+    else:
+        has_data = False
+
+    return has_data
+
+
+@deprecated
+def find_most_recent_time(bundle_name):
+    """
+    Find most recent "time folder" for a given bundle.
+
+    :param bundle_name:
+        The name of the targeted bundle.
+
+    :return folder:
+        The name of the time folder.
+    """
+    try:
+        bundle_folders = os.listdir(
+            data_path([bundle_name]),
+        )
+    except OSError:
+        return None
+
+    most_recent_bundle = dict()
+    for folder in bundle_folders:
+        date = from_bundle_ingest_dirname(folder)
+        if not most_recent_bundle or date > \
+                most_recent_bundle[most_recent_bundle.keys()[0]]:
+            most_recent_bundle = dict()
+            most_recent_bundle[folder] = date
+
+    if most_recent_bundle:
+        return most_recent_bundle.keys()[0]
+    else:
+        return None
+

catalyst/exchange/data_portal_exchange.py

@@ -0,0 +1,343 @@
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import abc
+from time import sleep
+
+import numpy as np
+import pandas as pd
+from catalyst.assets._assets import TradingPair
+from logbook import Logger
+
+from catalyst.constants import LOG_LEVEL
+from catalyst.data.data_portal import DataPortal
+from catalyst.exchange.exchange_bundle import ExchangeBundle
+from catalyst.exchange.exchange_errors import (
+    ExchangeRequestError,
+    ExchangeBarDataError,
+    PricingDataNotLoadedError)
+
+log = Logger('DataPortalExchange', level=LOG_LEVEL)
+
+
+class DataPortalExchangeBase(DataPortal):
+    def __init__(self, *args, **kwargs):
+
+        self.exchanges = kwargs.pop('exchanges', None)
+        # TODO: put somewhere accessible by each algo
+        self.retry_get_history_window = 5
+        self.retry_get_spot_value = 5
+        self.retry_delay = 5
+
+        super(DataPortalExchangeBase, self).__init__(*args, **kwargs)
+
+    def _get_history_window(self,
+                            assets,
+                            end_dt,
+                            bar_count,
+                            frequency,
+                            field,
+                            data_frequency,
+                            ffill=True,
+                            attempt_index=0):
+        try:
+            exchange_assets = dict()
+            for asset in assets:
+                if asset.exchange not in exchange_assets:
+                    exchange_assets[asset.exchange] = list()
+
+                exchange_assets[asset.exchange].append(asset)
+
+            if len(exchange_assets) > 1:
+                df_list = []
+                for exchange_name in exchange_assets:
+                    exchange = self.exchanges[exchange_name]
+                    assets = exchange_assets[exchange_name]
+
+                    df_exchange = self.get_exchange_history_window(
+                        exchange,
+                        assets,
+                        end_dt,
+                        bar_count,
+                        frequency,
+                        field,
+                        data_frequency,
+                        ffill)
+
+                    df_list.append(df_exchange)
+
+                # Merging the values values of each exchange
+                return pd.concat(df_list)
+
+            else:
+                exchange = self.exchanges[exchange_assets.keys()[0]]
+                return self.get_exchange_history_window(
+                    exchange,
+                    assets,
+                    end_dt,
+                    bar_count,
+                    frequency,
+                    field,
+                    data_frequency,
+                    ffill)
+
+        except ExchangeRequestError as e:
+            log.warn(
+                'get history attempt {}: {}'.format(attempt_index, e)
+            )
+            if attempt_index < self.retry_get_history_window:
+                sleep(self.retry_delay)
+                return self._get_history_window(assets,
+                                                end_dt,
+                                                bar_count,
+                                                frequency,
+                                                field,
+                                                data_frequency,
+                                                ffill,
+                                                attempt_index + 1)
+            else:
+                raise ExchangeBarDataError(
+                    data_type='history',
+                    attempts=attempt_index,
+                    error=e
+                )
+
+    def get_history_window(self,
+                           assets,
+                           end_dt,
+                           bar_count,
+                           frequency,
+                           field,
+                           data_frequency=None,
+                           ffill=True):
+
+        if field == 'price':
+            field = 'close'
+
+        return self._get_history_window(assets,
+                                        end_dt,
+                                        bar_count,
+                                        frequency,
+                                        field,
+                                        data_frequency,
+                                        ffill)
+
+    @abc.abstractmethod
+    def get_exchange_history_window(self,
+                                    exchange,
+                                    assets,
+                                    end_dt,
+                                    bar_count,
+                                    frequency,
+                                    field,
+                                    data_frequency,
+                                    ffill=True):
+        pass
+
+    def _get_spot_value(self, assets, field, dt, data_frequency,
+                        attempt_index=0):
+        try:
+            if isinstance(assets, TradingPair):
+                exchange = self.exchanges[assets.exchange]
+                spot_values = self.get_exchange_spot_value(
+                    exchange, [assets], field, dt, data_frequency)
+
+                if not spot_values:
+                    return np.nan
+
+                return spot_values[0]
+
+            else:
+                exchange_assets = dict()
+                for asset in assets:
+                    if asset.exchange not in exchange_assets:
+                        exchange_assets[asset.exchange] = list()
+
+                    exchange_assets[asset.exchange].append(asset)
+
+                if len(exchange_assets.keys()) == 1:
+                    exchange = self.exchanges[exchange_assets.keys()[0]]
+                    return self.get_exchange_spot_value(
+                        exchange, assets, field, dt, data_frequency)
+
+                else:
+                    spot_values = []
+                    for exchange_name in exchange_assets:
+                        exchange = self.exchanges[exchange_name]
+                        assets = exchange_assets[exchange_name]
+                        exchange_spot_values = self.get_exchange_spot_value(
+                            exchange,
+                            assets,
+                            field,
+                            dt,
+                            data_frequency
+                        )
+                        if len(assets) == 1:
+                            spot_values.append(exchange_spot_values)
+                        else:
+                            spot_values += exchange_spot_values
+
+                    return spot_values
+
+        except ExchangeRequestError as e:
+            log.warn(
+                'get spot value attempt {}: {}'.format(attempt_index, e)
+            )
+            if attempt_index < self.retry_get_spot_value:
+                sleep(self.retry_delay)
+                return self._get_spot_value(assets, field, dt, data_frequency,
+                                            attempt_index + 1)
+            else:
+                raise ExchangeBarDataError(
+                    data_type='spot',
+                    attempts=attempt_index,
+                    error=e
+                )
+
+    def get_spot_value(self, assets, field, dt, data_frequency):
+        if field == 'price':
+            field = 'close'
+
+        return self._get_spot_value(assets, field, dt, data_frequency)
+
+    @abc.abstractmethod
+    def get_exchange_spot_value(self, exchange, assets, field, dt,
+                                data_frequency):
+        return
+
+    def get_adjusted_value(self, asset, field, dt,
+                           perspective_dt,
+                           data_frequency,
+                           spot_value=None):
+        # TODO: does this pertain to cryptocurrencies?
+        log.warn('get_adjusted_value is not implemented yet!')
+        return spot_value
+
+
+class DataPortalExchangeLive(DataPortalExchangeBase):
+    def __init__(self, *args, **kwargs):
+        super(DataPortalExchangeLive, self).__init__(*args, **kwargs)
+
+    def get_exchange_history_window(self,
+                                    exchange,
+                                    assets,
+                                    end_dt,
+                                    bar_count,
+                                    frequency,
+                                    field,
+                                    data_frequency,
+                                    ffill=True):
+        df = exchange.get_history_window(
+            assets,
+            end_dt,
+            bar_count,
+            frequency,
+            field,
+            data_frequency,
+            ffill)
+        return df
+
+    def get_exchange_spot_value(self, exchange, assets, field, dt,
+                                data_frequency):
+        exchange_spot_values = exchange.get_spot_value(
+            assets, field, dt, data_frequency)
+
+        return exchange_spot_values
+
+
+class DataPortalExchangeBacktest(DataPortalExchangeBase):
+    def __init__(self, *args, **kwargs):
+        super(DataPortalExchangeBacktest, self).__init__(*args, **kwargs)
+
+        self.exchange_bundles = dict()
+
+        self.history_loaders = dict()
+        self.minute_history_loaders = dict()
+
+        for exchange_name in self.exchanges:
+            exchange = self.exchanges[exchange_name]
+            self.exchange_bundles[exchange_name] = ExchangeBundle(exchange)
+
+    def _get_first_trading_day(self, assets):
+        first_date = None
+        for asset in assets:
+            if first_date is None or asset.start_date > first_date:
+                first_date = asset.start_date
+        return first_date
+
+    def get_exchange_history_window(self,
+                                    exchange,
+                                    assets,
+                                    end_dt,
+                                    bar_count,
+                                    frequency,
+                                    field,
+                                    data_frequency,
+                                    ffill=True):
+        """
+        Fetching price history window from the exchange bundle.
+
+        Using a try... except approach to minimize reads most of the time,
+        when the data exists.
+
+        :param exchange:
+        :param assets:
+        :param end_dt:
+        :param bar_count:
+        :param frequency:
+        :param field:
+        :param data_frequency:
+        :param ffill:
+        :return:
+        """
+
+        bundle = self.exchange_bundles[exchange.name]
+        series = bundle.get_history_window_series_and_load(
+            assets=assets,
+            end_dt=end_dt,
+            bar_count=bar_count,
+            field=field,
+            data_frequency=data_frequency
+        )
+        return pd.DataFrame(series)
+
+    def get_exchange_spot_value(self, exchange, assets, field, dt,
+                                data_frequency):
+        bundle = self.exchange_bundles[exchange.name]
+
+        if data_frequency == 'daily':
+            dt = dt.floor('1D')
+        else:
+            dt = dt.floor('1 min')
+
+        try:
+            return bundle.get_spot_values(assets, field, dt, data_frequency)
+
+        except PricingDataNotLoadedError:
+            log.info(
+                'pricing data for {symbol} not found on {dt}'
+                ', updating the bundles.'.format(
+                    symbol=[asset.symbol for asset in assets],
+                    dt=dt
+                )
+            )
+            bundle.ingest_assets(
+                assets=assets,
+                start_dt=self._first_trading_day,
+                end_dt=self._last_available_session,
+                data_frequency=data_frequency,
+                show_progress=True
+            )
+            return bundle.get_spot_values(
+                assets, field, dt, data_frequency, True
+            )

catalyst/exchange/exchange.py

@@ -0,0 +1,795 @@
+import abc
+import re
+from abc import ABCMeta, abstractmethod, abstractproperty
+from datetime import timedelta
+from time import sleep
+
+import numpy as np
+import pandas as pd
+from catalyst.assets._assets import TradingPair
+from logbook import Logger
+
+from catalyst.constants import LOG_LEVEL
+from catalyst.data.data_portal import BASE_FIELDS
+from catalyst.exchange.bundle_utils import get_start_dt, \
+    get_delta, get_periods
+from catalyst.exchange.exchange_bundle import ExchangeBundle
+from catalyst.exchange.exchange_errors import MismatchingBaseCurrencies, \
+    InvalidOrderStyle, BaseCurrencyNotFoundError, SymbolNotFoundOnExchange, \
+    InvalidHistoryFrequencyError, PricingDataNotLoadedError
+from catalyst.exchange.exchange_execution import ExchangeStopLimitOrder, \
+    ExchangeLimitOrder, ExchangeStopOrder
+from catalyst.exchange.exchange_portfolio import ExchangePortfolio
+from catalyst.exchange.exchange_utils import get_exchange_symbols
+from catalyst.finance.order import ORDER_STATUS
+from catalyst.finance.transaction import Transaction
+
+log = Logger('Exchange', level=LOG_LEVEL)
+
+
+class Exchange:
+    __metaclass__ = ABCMeta
+
+    def __init__(self):
+        self.name = None
+        self.assets = {}
+        self._portfolio = None
+        self.minute_writer = None
+        self.minute_reader = None
+        self.base_currency = None
+
+        self.num_candles_limit = None
+        self.max_requests_per_minute = None
+        self.request_cpt = None
+        self.bundle = ExchangeBundle(self)
+
+    @property
+    def positions(self):
+        return self.portfolio.positions
+
+    @property
+    def portfolio(self):
+        """
+        Return the Portfolio
+
+        :return:
+        """
+        if self._portfolio is None:
+            self._portfolio = ExchangePortfolio(
+                start_date=pd.Timestamp.utcnow()
+            )
+            self.synchronize_portfolio()
+
+        return self._portfolio
+
+    @abstractproperty
+    def account(self):
+        pass
+
+    @abstractproperty
+    def time_skew(self):
+        pass
+
+    def ask_request(self):
+        """
+        Asks permission to issue a request to the exchange.
+        The primary purpose is to avoid hitting rate limits.
+
+        The application will pause if the maximum requests per minute
+        permitted by the exchange is exceeded.
+
+        :return boolean:
+
+        """
+        now = pd.Timestamp.utcnow()
+        if not self.request_cpt:
+            self.request_cpt = dict()
+            self.request_cpt[now] = 0
+            return True
+
+        cpt_date = self.request_cpt.keys()[0]
+        cpt = self.request_cpt[cpt_date]
+
+        if now > cpt_date + timedelta(minutes=1):
+            self.request_cpt = dict()
+            self.request_cpt[now] = 0
+            return True
+
+        if cpt >= self.max_requests_per_minute:
+            delta = now - cpt_date
+
+            sleep_period = 60 - delta.total_seconds()
+            sleep(sleep_period)
+
+            now = pd.Timestamp.utcnow()
+            self.request_cpt = dict()
+            self.request_cpt[now] = 0
+            return True
+        else:
+            self.request_cpt[cpt_date] += 1
+
+    def get_symbol(self, asset):
+        """
+        Get the exchange specific symbol of the given asset.
+
+        :param asset: Asset
+        :return: symbol: str
+        """
+        symbol = None
+
+        for key in self.assets:
+            if not symbol and self.assets[key].symbol == asset.symbol:
+                symbol = key
+
+        if not symbol:
+            raise ValueError('Currency %s not supported by exchange %s' %
+                             (asset['symbol'], self.name.title()))
+
+        return symbol
+
+    def get_symbols(self, assets):
+        """
+        Get a list of symbols corresponding to each given asset.
+
+        :param assets: Asset[]
+        :return:
+        """
+        symbols = []
+
+        for asset in assets:
+            symbols.append(self.get_symbol(asset))
+
+        return symbols
+
+    def get_assets(self, symbols=None):
+        assets = []
+
+        if symbols is not None:
+            for symbol in symbols:
+                asset = self.get_asset(symbol)
+                assets.append(asset)
+        else:
+            for key in self.assets:
+                assets.append(self.assets[key])
+
+        return assets
+
+    def get_asset(self, symbol):
+        """
+        Find an Asset on the current exchange based on its Catalyst symbol
+        :param symbol: the [target]_[base] currency pair symbol
+        :return: Asset
+        """
+        asset = None
+
+        for key in self.assets:
+            if not asset and self.assets[key].symbol.lower() == symbol.lower():
+                asset = self.assets[key]
+
+        if not asset:
+            supported_symbols = [pair.symbol.encode('utf-8') for pair in
+                                 self.assets.values()]
+            raise SymbolNotFoundOnExchange(
+                symbol=symbol,
+                exchange=self.name.title(),
+                supported_symbols=supported_symbols
+            )
+
+        return asset
+
+    def fetch_symbol_map(self):
+        return get_exchange_symbols(self.name)
+
+    def load_assets(self):
+        """
+        Populate the 'assets' attribute with a dictionary of Assets.
+        The key of the resulting dictionary is the exchange specific
+        currency pair symbol. The universal symbol is contained in the
+        'symbol' attribute of each asset.
+
+
+        Notes
+        -----
+        The sid of each asset is calculated based on a numeric hash of the
+        universal symbol. This simple approach avoids maintaining a mapping
+        of sids.
+
+        This method can be overridden if an exchange offers equivalent data
+        via its api.
+        """
+
+        symbol_map = self.fetch_symbol_map()
+        for exchange_symbol in symbol_map:
+            asset = symbol_map[exchange_symbol]
+
+            if 'start_date' in asset:
+                start_date = pd.to_datetime(asset['start_date'], utc=True)
+            else:
+                start_date = None
+
+            if 'end_date' in asset:
+                end_date = pd.to_datetime(asset['end_date'], utc=True)
+            else:
+                end_date = None
+
+            if 'leverage' in asset:
+                leverage = asset['leverage']
+            else:
+                leverage = 1.0
+
+            if 'asset_name' in asset:
+                asset_name = asset['asset_name']
+            else:
+                asset_name = None
+
+            if 'min_trade_size' in asset:
+                min_trade_size = asset['min_trade_size']
+            else:
+                min_trade_size = 0.0000001
+
+            if 'end_daily' in asset and asset['end_daily'] != 'N/A':
+                end_daily = pd.to_datetime(asset['end_daily'], utc=True)
+            else:
+                end_daily = None
+
+            if 'end_minute' in asset and asset['end_minute'] != 'N/A':
+                end_minute = pd.to_datetime(asset['end_minute'], utc=True)
+            else:
+                end_minute = None
+
+            trading_pair = TradingPair(
+                symbol=asset['symbol'],
+                exchange=self.name,
+                start_date=start_date,
+                end_date=end_date,
+                leverage=leverage,
+                asset_name=asset_name,
+                min_trade_size=min_trade_size,
+                end_daily=end_daily,
+                end_minute=end_minute,
+                exchange_symbol=exchange_symbol
+            )
+
+            self.assets[exchange_symbol] = trading_pair
+
+    def check_open_orders(self):
+        """
+        Loop through the list of open orders in the Portfolio object.
+        For each executed order found, create a transaction and apply to the
+        Portfolio.
+
+        :return:
+        transactions: Transaction[]
+        """
+        transactions = list()
+        if self.portfolio.open_orders:
+            for order_id in list(self.portfolio.open_orders):
+                log.debug('found open order: {}'.format(order_id))
+
+                order, executed_price = self.get_order(order_id)
+                log.debug('got updated order {} {}'.format(
+                    order, executed_price))
+
+                if order.status == ORDER_STATUS.FILLED:
+                    transaction = Transaction(
+                        asset=order.asset,
+                        amount=order.amount,
+                        dt=pd.Timestamp.utcnow(),
+                        price=executed_price,
+                        order_id=order.id,
+                        commission=order.commission
+                    )
+                    transactions.append(transaction)
+
+                    self.portfolio.execute_order(order, transaction)
+
+                elif order.status == ORDER_STATUS.CANCELLED:
+                    self.portfolio.remove_order(order)
+
+                else:
+                    delta = pd.Timestamp.utcnow() - order.dt
+                    log.info(
+                        'order {order_id} still open after {delta}'.format(
+                            order_id=order_id,
+                            delta=delta
+                        )
+                    )
+        return transactions
+
+    def get_spot_value(self, assets, field, dt=None, data_frequency='minute'):
+        """
+        Public API method that returns a scalar value representing the value
+        of the desired asset's field at either the given dt.
+
+        Parameters
+        ----------
+        assets : Asset, ContinuousFuture, or iterable of same.
+            The asset or assets whose data is desired.
+        field : {'open', 'high', 'low', 'close', 'volume',
+                 'price', 'last_traded'}
+            The desired field of the asset.
+        dt : pd.Timestamp
+            The timestamp for the desired value.
+        data_frequency : str
+            The frequency of the data to query; i.e. whether the data is
+            'daily' or 'minute' bars
+
+        Returns
+        -------
+        value : float, int, or pd.Timestamp
+            The spot value of ``field`` for ``asset`` The return type is based
+            on the ``field`` requested. If the field is one of 'open', 'high',
+            'low', 'close', or 'price', the value will be a float. If the
+            ``field`` is 'volume' the value will be a int. If the ``field`` is
+            'last_traded' the value will be a Timestamp.
+
+        Bitfinex timeframes
+        -------------------
+        Available values: '1m', '5m', '15m', '30m', '1h', '3h', '6h', '12h',
+         '1D', '7D', '14D', '1M'
+        """
+        if field not in BASE_FIELDS:
+            raise KeyError('Invalid column: {}'.format(field))
+
+        values = []
+        for asset in assets:
+            value = self.get_single_spot_value(asset, field, data_frequency)
+            values.append(value)
+
+        return values
+
+    def get_single_spot_value(self, asset, field, data_frequency):
+        """
+        Similar to 'get_spot_value' but for a single asset
+
+        Note
+        ----
+        We're writing each minute bar to disk using zipline's machinery.
+        This is especially useful when running multiple algorithms
+        concurrently. By using local data when possible, we try to reaching
+        request limits on exchanges.
+
+        :param asset:
+        :param field:
+        :param data_frequency:
+        :return value: The spot value of the given asset / field
+        """
+        log.debug(
+            'fetching spot value {field} for symbol {symbol}'.format(
+                symbol=asset.symbol,
+                field=field
+            )
+        )
+
+        ohlc = self.get_candles(data_frequency, asset)
+        if field not in ohlc:
+            raise KeyError('Invalid column: %s' % field)
+
+        value = ohlc[field]
+        log.debug('got spot value: {}'.format(value))
+
+        return value
+
+    def get_series_from_candles(self, candles, start_dt, end_dt,
+                                field, previous_value=None):
+        """
+        Get a series of field data for the specified candles.
+
+        :param candles:
+        :param start_dt:
+        :param end_dt:
+        :param field:
+        :param previous_value:
+        :return:
+        """
+
+        dates = [candle['last_traded'] for candle in candles]
+        values = [candle[field] for candle in candles]
+
+        periods = pd.date_range(start_dt, end_dt)
+        series = pd.Series(values, index=dates)
+
+        series.reindex(periods, method='ffill', fill_value=previous_value)
+
+        return series
+
+    def get_history_window(self,
+                           assets,
+                           end_dt,
+                           bar_count,
+                           frequency,
+                           field,
+                           data_frequency=None,
+                           ffill=True):
+
+        """
+        Public API method that returns a dataframe containing the requested
+        history window.  Data is fully adjusted.
+
+        Parameters
+        ----------
+        assets : list of catalyst.data.Asset objects
+            The assets whose data is desired.
+
+        end_dt: not applicable to cryptocurrencies
+
+        bar_count: int
+            The number of bars desired.
+
+        frequency: string
+            "1d" or "1m"
+
+        field: string
+            The desired field of the asset.
+
+        data_frequency: string
+            The frequency of the data to query; i.e. whether the data is
+            'daily' or 'minute' bars.
+
+        # TODO: fill how?
+        ffill: boolean
+            Forward-fill missing values. Only has effect if field
+            is 'price'.
+
+        Returns
+        -------
+        A dataframe containing the requested data.
+        """
+
+        freq_match = re.match(r'([0-9].*)(m|M|d|D)', frequency, re.M | re.I)
+        if freq_match:
+            candle_size = int(freq_match.group(1))
+            unit = freq_match.group(2)
+
+        else:
+            raise InvalidHistoryFrequencyError(frequency)
+
+        if unit.lower() == 'd':
+            if data_frequency == 'minute':
+                data_frequency = 'daily'
+
+        elif unit.lower() == 'm':
+            if data_frequency == 'daily':
+                data_frequency = 'minute'
+
+        else:
+            raise InvalidHistoryFrequencyError(frequency)
+
+        adj_bar_count = candle_size * bar_count
+        try:
+            series = self.bundle.get_history_window_series_and_load(
+                assets=assets,
+                end_dt=end_dt,
+                bar_count=adj_bar_count,
+                field=field,
+                data_frequency=data_frequency
+            )
+        except PricingDataNotLoadedError:
+            series = dict()
+
+        for asset in assets:
+            if asset not in series or series[asset].index[-1] < end_dt:
+                # Adding bars too recent to be contained in the consolidated
+                # exchanges bundles. We go directly against the exchange
+                # to retrieve the candles.
+                start_dt = get_start_dt(end_dt, adj_bar_count, data_frequency)
+                trailing_dt = \
+                    series[asset].index[-1] + get_delta(1, data_frequency) \
+                        if asset in series else start_dt
+
+                trailing_bar_count = \
+                    get_periods(trailing_dt, end_dt, data_frequency)
+
+                # The get_history method supports multiple asset
+                candles = self.get_candles(
+                    data_frequency=data_frequency,
+                    assets=asset,
+                    bar_count=trailing_bar_count,
+                    end_dt=end_dt
+                )
+
+                last_value = series[asset].iloc(0) if asset in series \
+                    else np.nan
+
+                candle_series = self.get_series_from_candles(
+                    candles=candles,
+                    start_dt=trailing_dt,
+                    end_dt=end_dt,
+                    field=field,
+                    previous_value=last_value
+                )
+
+                if asset in series:
+                    series[asset].append(candle_series)
+
+                else:
+                    series[asset] = candle_series
+
+        df = pd.DataFrame(series)
+
+        if candle_size > 1:
+            if field == 'open':
+                agg = 'first'
+            elif field == 'high':
+                agg = 'max'
+            elif field == 'low':
+                agg = 'min'
+            elif field == 'close':
+                agg = 'last'
+            elif field == 'volume':
+                agg = 'sum'
+            else:
+                raise ValueError('Invalid field.')
+
+            df = df.resample('{}T'.format(candle_size)).agg(agg)
+
+        return df
+
+    def synchronize_portfolio(self):
+        """
+        Update the portfolio cash and position balances based on the
+        latest ticker prices.
+
+        :return:
+        """
+        log.debug('synchronizing portfolio with exchange {}'.format(self.name))
+        balances = self.get_balances()
+
+        base_position_available = balances[self.base_currency] \
+            if self.base_currency in balances else None
+
+        if base_position_available is None:
+            raise BaseCurrencyNotFoundError(
+                base_currency=self.base_currency,
+                exchange=self.name.title()
+            )
+
+        portfolio = self._portfolio
+        portfolio.cash = base_position_available
+        log.debug('found base currency balance: {}'.format(portfolio.cash))
+
+        if portfolio.starting_cash is None:
+            portfolio.starting_cash = portfolio.cash
+
+        if portfolio.positions:
+            assets = portfolio.positions.keys()
+            tickers = self.tickers(assets)
+
+            portfolio.positions_value = 0.0
+            for asset in tickers:
+                # TODO: convert if the position is not in the base currency
+                ticker = tickers[asset]
+                position = portfolio.positions[asset]
+                position.last_sale_price = ticker['last_price']
+                position.last_sale_date = ticker['timestamp']
+
+                portfolio.positions_value += \
+                    position.amount * position.last_sale_price
+                portfolio.portfolio_value = \
+                    portfolio.positions_value + portfolio.cash
+
+    def order(self, asset, amount, limit_price=None, stop_price=None,
+              style=None):
+        """Place an order.
+
+        Parameters
+        ----------
+        asset : Asset
+            The asset that this order is for.
+        amount : int
+            The amount of shares to order. If ``amount`` is positive, this is
+            the number of shares to buy or cover. If ``amount`` is negative,
+            this is the number of shares to sell or short.
+        limit_price : float, optional
+            The limit price for the order.
+        stop_price : float, optional
+            The stop price for the order.
+        style : ExecutionStyle, optional
+            The execution style for the order.
+
+        Returns
+        -------
+        order_id : str or None
+            The unique identifier for this order, or None if no order was
+            placed.
+
+        Notes
+        -----
+        The ``limit_price`` and ``stop_price`` arguments provide shorthands for
+        passing common execution styles. Passing ``limit_price=N`` is
+        equivalent to ``style=LimitOrder(N)``. Similarly, passing
+        ``stop_price=M`` is equivalent to ``style=StopOrder(M)``, and passing
+        ``limit_price=N`` and ``stop_price=M`` is equivalent to
+        ``style=StopLimitOrder(N, M)``. It is an error to pass both a ``style``
+        and ``limit_price`` or ``stop_price``.
+
+        See Also
+        --------
+        :class:`catalyst.finance.execution.ExecutionStyle`
+        :func:`catalyst.api.order_value`
+        :func:`catalyst.api.order_percent`
+        """
+        if amount == 0:
+            log.warn('skipping order amount of 0')
+            return None
+
+        if asset.base_currency != self.base_currency.lower():
+            raise MismatchingBaseCurrencies(
+                base_currency=asset.base_currency,
+                algo_currency=self.base_currency
+            )
+
+        is_buy = (amount > 0)
+
+        if limit_price is not None and stop_price is not None:
+            style = ExchangeStopLimitOrder(limit_price, stop_price,
+                                           exchange=self.name)
+        elif limit_price is not None:
+            style = ExchangeLimitOrder(limit_price, exchange=self.name)
+
+        elif stop_price is not None:
+            style = ExchangeStopOrder(stop_price, exchange=self.name)
+
+        elif style is not None:
+            raise InvalidOrderStyle(exchange=self.name.title(),
+                                    style=style.__class__.__name__)
+        else:
+            raise ValueError('Incomplete order data.')
+
+        display_price = limit_price if limit_price is not None else stop_price
+        log.debug(
+            'issuing {side} order of {amount} {symbol} for {type}: {price}'.format(
+                side='buy' if is_buy else 'sell',
+                amount=amount,
+                symbol=asset.symbol,
+                type=style.__class__.__name__,
+                price='{}{}'.format(display_price, asset.base_currency)
+            )
+        )
+        order = self.create_order(asset, amount, is_buy, style)
+        if order:
+            self._portfolio.create_order(order)
+            return order.id
+        else:
+            return None
+
+    # The methods below must be implemented for each exchange.
+    @abstractmethod
+    def get_balances(self):
+        """
+        Retrieve wallet balances for the exchange
+        :return balances: A dict of currency => available balance
+        """
+        pass
+
+    @abstractmethod
+    def create_order(self, asset, amount, is_buy, style):
+        """
+        Place an order on the exchange.
+
+        :param asset : Asset
+            The asset that this order is for.
+        :param amount : int
+            The amount of shares to order. If ``amount`` is positive, this is
+            the number of shares to buy or cover. If ``amount`` is negative,
+            this is the number of shares to sell or short.
+        :param style : ExecutionStyle
+            The execution style for the order.
+        :param is_buy: boolean
+            Is it a buy order?
+        :return:
+        """
+        pass
+
+    @abstractmethod
+    def get_open_orders(self, asset):
+        """Retrieve all of the current open orders.
+
+        Parameters
+        ----------
+        asset : Asset
+            If passed and not None, return only the open orders for the given
+            asset instead of all open orders.
+
+        Returns
+        -------
+        open_orders : dict[list[Order]] or list[Order]
+            If no asset is passed this will return a dict mapping Assets
+            to a list containing all the open orders for the asset.
+            If an asset is passed then this will return a list of the open
+            orders for this asset.
+        """
+        pass
+
+    @abstractmethod
+    def get_order(self, order_id):
+        """Lookup an order based on the order id returned from one of the
+        order functions.
+
+        Parameters
+        ----------
+        order_id : str
+            The unique identifier for the order.
+
+        Returns
+        -------
+        order : Order
+            The order object.
+        execution_price: float
+            The execution price per share of the order
+        """
+        pass
+
+    @abstractmethod
+    def cancel_order(self, order_param):
+        """Cancel an open order.
+
+        Parameters
+        ----------
+        order_param : str or Order
+            The order_id or order object to cancel.
+        """
+        pass
+
+    @abstractmethod
+    def get_candles(self, data_frequency, assets, bar_count=None,
+                    start_dt=None, end_dt=None):
+        """
+        Retrieve OHLCV candles for the given assets
+
+        :param data_frequency:
+            The candle frequency: minute or daily
+        :param assets: list[TradingPair]
+            The targeted assets.
+        :param bar_count:
+            The number of bar desired. (default 1)
+        :param end_dt: datetime, optional
+            The last bar date.
+        :param start_dt: datetime, optional
+            The first bar date.
+
+        :return dict[TradingPair, dict[str, Object]]: OHLCV data
+            A dictionary of OHLCV candles. Each TradingPair instance is
+            mapped to a list of dictionaries with this structure:
+                open: float
+                high: float
+                low: float
+                close: float
+                volume: float
+                last_traded: datetime
+
+            See definition here:
+                http://www.investopedia.com/terms/o/ohlcchart.asp
+        """
+        pass
+
+    @abc.abstractmethod
+    def tickers(self, assets):
+        """
+        Retrieve current tick data for the given assets
+
+        :param assets:
+        :return:
+        """
+        pass
+
+    @abc.abstractmethod
+    def get_account(self):
+        """
+        Retrieve the account parameters.
+        :return:
+        """
+        pass
+
+    @abc.abstractmethod
+    def get_orderbook(self, asset, order_type):
+        """
+        Retrieve the the orderbook for the given trading pair.
+
+        :param asset: TradingPair
+        :param order_type: str
+            The type of orders: bid, ask or all
+
+        :return:
+        """
+        pass

catalyst/exchange/exchange_algorithm.py

@@ -0,0 +1,706 @@
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import os
+import pickle
+import signal
+import sys
+from collections import deque
+from datetime import timedelta
+from os import listdir
+from os.path import isfile, join
+from time import sleep
+
+import logbook
+import pandas as pd
+from catalyst.assets._assets import TradingPair
+
+import catalyst.protocol as zp
+from catalyst.algorithm import TradingAlgorithm
+from catalyst.constants import LOG_LEVEL
+from catalyst.data.minute_bars import BcolzMinuteBarWriter, \
+    BcolzMinuteBarReader
+from catalyst.errors import OrderInBeforeTradingStart
+from catalyst.exchange.exchange_blotter import ExchangeBlotter
+from catalyst.exchange.exchange_errors import (
+    ExchangeRequestError,
+    ExchangePortfolioDataError,
+    ExchangeTransactionError,
+    OrphanOrderError)
+from catalyst.exchange.exchange_execution import ExchangeStopLimitOrder, \
+    ExchangeLimitOrder, ExchangeStopOrder
+from catalyst.exchange.exchange_utils import get_exchange_minute_writer_root, \
+    save_algo_object, get_algo_object, get_algo_folder, get_algo_df, \
+    save_algo_df
+from catalyst.exchange.live_graph_clock import LiveGraphClock
+from catalyst.exchange.simple_clock import SimpleClock
+from catalyst.exchange.stats_utils import get_pretty_stats
+from catalyst.finance.execution import MarketOrder
+from catalyst.finance.performance.period import calc_period_stats
+from catalyst.gens.tradesimulation import AlgorithmSimulator
+from catalyst.utils.api_support import (
+    api_method,
+    disallowed_in_before_trading_start)
+from catalyst.utils.input_validation import error_keywords, ensure_upper_case, \
+    expect_types
+from catalyst.utils.math_utils import round_nearest
+from catalyst.utils.preprocess import preprocess
+
+log = logbook.Logger('exchange_algorithm', level=LOG_LEVEL)
+
+
+class ExchangeAlgorithmExecutor(AlgorithmSimulator):
+    def __init__(self, *args, **kwargs):
+        super(self.__class__, self).__init__(*args, **kwargs)
+
+
+class ExchangeTradingAlgorithmBase(TradingAlgorithm):
+    def __init__(self, *args, **kwargs):
+        self.exchanges = kwargs.pop('exchanges', None)
+
+        super(ExchangeTradingAlgorithmBase, self).__init__(*args, **kwargs)
+
+    def round_order(self, amount, asset):
+        """
+        We need fractions with cryptocurrencies
+
+        :param amount:
+        :return:
+        """
+        return round_nearest(amount, asset.min_trade_size)
+
+    @api_method
+    @preprocess(symbol_str=ensure_upper_case)
+    def symbol(self, symbol_str, exchange_name=None):
+        """Lookup an Equity by its ticker symbol.
+
+        Parameters
+        ----------
+        symbol_str : str
+            The ticker symbol for the equity to lookup.
+        exchange_name: str
+            The name of the exchange containing the symbol
+
+        Returns
+        -------
+        equity : Equity
+            The equity that held the ticker symbol on the current
+            symbol lookup date.
+
+        Raises
+        ------
+        SymbolNotFound
+            Raised when the symbols was not held on the current lookup date.
+
+        See Also
+        --------
+        :func:`catalyst.api.set_symbol_lookup_date`
+        """
+        # If the user has not set the symbol lookup date,
+        # use the end_session as the date for sybmol->sid resolution.
+
+        _lookup_date = self._symbol_lookup_date \
+            if self._symbol_lookup_date is not None \
+            else self.sim_params.end_session
+
+        if exchange_name is None:
+            exchange = self.exchanges.values()[0]
+        else:
+            exchange = self.exchanges[exchange_name]
+
+        return self.asset_finder.lookup_symbol(
+            symbol=symbol_str,
+            exchange=exchange,
+            as_of_date=_lookup_date
+        )
+
+    def prepare_period_stats(self, start_dt, end_dt):
+        """
+        Creates a dictionary representing the state of the tracker.
+
+
+        I rewrote this in an attempt to better control the stats.
+        I don't want things to happen magically through complex logic
+        pertaining to backtesting.
+
+        """
+        tracker = self.perf_tracker
+        period = tracker.todays_performance
+
+        pos_stats = period.position_tracker.stats()
+        period_stats = calc_period_stats(pos_stats, period.ending_cash)
+
+        stats = dict(
+            period_start=tracker.period_start,
+            period_end=tracker.period_end,
+            capital_base=tracker.capital_base,
+            progress=tracker.progress,
+            ending_value=period.ending_value,
+            ending_exposure=period.ending_exposure,
+            capital_used=period.cash_flow,
+            starting_value=period.starting_value,
+            starting_exposure=period.starting_exposure,
+            starting_cash=period.starting_cash,
+            ending_cash=period.ending_cash,
+            portfolio_value=period.ending_cash + period.ending_value,
+            pnl=period.pnl,
+            returns=period.returns,
+            period_open=period.period_open,
+            period_close=period.period_close,
+            gross_leverage=period_stats.gross_leverage,
+            net_leverage=period_stats.net_leverage,
+            short_exposure=pos_stats.short_exposure,
+            long_exposure=pos_stats.long_exposure,
+            short_value=pos_stats.short_value,
+            long_value=pos_stats.long_value,
+            longs_count=pos_stats.longs_count,
+            shorts_count=pos_stats.shorts_count,
+        )
+
+        # Merging cumulative risk
+        stats.update(tracker.cumulative_risk_metrics.to_dict())
+
+        # Merging latest recorded variables
+        stats.update(self.recorded_vars)
+
+        stats['positions'] = period.position_tracker.get_positions_list()
+
+        # we want the key to be absent, not just empty
+        # Only include transactions for given dt
+        stats['transactions'] = dict()
+        for date in period.processed_transactions:
+            if start_dt <= date < end_dt:
+                stats['transactions'][date] = \
+                    period.processed_transactions[date]
+
+        stats['orders'] = dict()
+        for date in period.orders_by_modified:
+            if start_dt <= date < end_dt:
+                stats['orders'][date] = \
+                    period.orders_by_modified[date]
+
+        return stats
+
+
+class ExchangeTradingAlgorithmBacktest(ExchangeTradingAlgorithmBase):
+    def __init__(self, *args, **kwargs):
+        super(ExchangeTradingAlgorithmBacktest, self).__init__(*args, **kwargs)
+
+        self.blotter = ExchangeBlotter(
+            data_frequency=self.data_frequency,
+            # Default to NeverCancel in catalyst
+            cancel_policy=self.cancel_policy,
+        )
+        log.info('initialized trading algorithm in backtest mode')
+
+    def _calculate_order(self, asset, amount,
+                         limit_price=None, stop_price=None, style=None):
+        # Raises a ZiplineError if invalid parameters are detected.
+        self.validate_order_params(asset,
+                                   amount,
+                                   limit_price,
+                                   stop_price,
+                                   style)
+
+        # Convert deprecated limit_price and stop_price parameters to use
+        # ExecutionStyle objects.
+        style = self.__convert_order_params_for_blotter(limit_price,
+                                                        stop_price,
+                                                        style)
+        return amount, style
+
+    @staticmethod
+    def __convert_order_params_for_blotter(limit_price, stop_price, style):
+        """
+        Helper method for converting deprecated limit_price and stop_price
+        arguments into ExecutionStyle instances.
+
+        This function assumes that either style == None or (limit_price,
+        stop_price) == (None, None).
+        """
+        if style:
+            assert (limit_price, stop_price) == (None, None)
+            return style
+        if limit_price and stop_price:
+            return ExchangeStopLimitOrder(limit_price, stop_price)
+        if limit_price:
+            return ExchangeLimitOrder(limit_price)
+        if stop_price:
+            return ExchangeStopOrder(stop_price)
+        else:
+            return MarketOrder()
+
+
+class ExchangeTradingAlgorithmLive(ExchangeTradingAlgorithmBase):
+    def __init__(self, *args, **kwargs):
+        self.algo_namespace = kwargs.pop('algo_namespace', None)
+        self.live_graph = kwargs.pop('live_graph', None)
+
+        self._clock = None
+        self.minute_stats = deque(maxlen=60)
+
+        self.pnl_stats = get_algo_df(self.algo_namespace, 'pnl_stats')
+
+        self.custom_signals_stats = \
+            get_algo_df(self.algo_namespace, 'custom_signals_stats')
+
+        self.exposure_stats = \
+            get_algo_df(self.algo_namespace, 'exposure_stats')
+
+        self.is_running = True
+
+        self.retry_check_open_orders = 5
+        self.retry_synchronize_portfolio = 5
+        self.retry_get_open_orders = 5
+        self.retry_order = 2
+        self.retry_delay = 5
+
+        self.stats_minutes = 5
+
+        super(ExchangeTradingAlgorithmLive, self).__init__(*args, **kwargs)
+        # TODO: fix precision before re-enabling
+        # self._create_minute_writer()
+
+        signal.signal(signal.SIGINT, self.signal_handler)
+
+        log.info('initialized trading algorithm in live mode')
+
+    def _create_minute_writer(self):
+        root = get_exchange_minute_writer_root(self.exchange.name)
+        filename = os.path.join(root, 'metadata.json')
+
+        if os.path.isfile(filename):
+            writer = BcolzMinuteBarWriter.open(
+                root, self.sim_params.end_session)
+        else:
+            # TODO: need to be able to write more precise numbers
+            writer = BcolzMinuteBarWriter(
+                rootdir=root,
+                calendar=self.trading_calendar,
+                minutes_per_day=1440,
+                start_session=self.sim_params.start_session,
+                end_session=self.sim_params.end_session,
+                write_metadata=True
+            )
+
+        self.exchange.minute_writer = writer
+        self.exchange.minute_reader = BcolzMinuteBarReader(root)
+
+    def signal_handler(self, signal, frame):
+        self.is_running = False
+
+        if self._analyze is None:
+            log.info('Interruption signal detected {}, exiting the '
+                     'algorithm'.format(signal))
+
+        else:
+            log.info('Interruption signal detected {}, calling `analyze()` '
+                     'before exiting the algorithm'.format(signal))
+
+            algo_folder = get_algo_folder(self.algo_namespace)
+            folder = join(algo_folder, 'daily_perf')
+            files = [f for f in listdir(folder) if isfile(join(folder, f))]
+
+            daily_perf_list = []
+            for item in files:
+                filename = join(folder, item)
+                with open(filename, 'rb') as handle:
+                    daily_perf_list.append(pickle.load(handle))
+
+            stats = pd.DataFrame(daily_perf_list)
+
+            self.analyze(stats)
+
+        sys.exit(0)
+
+    @property
+    def clock(self):
+        if self._clock is None:
+            return self._create_clock()
+        else:
+            return self._clock
+
+    def _create_clock(self):
+
+        # The calendar's execution times are the minutes over which we actually
+        # want to run the clock. Typically the execution times simply adhere to
+        # the market open and close times. In the case of the futures calendar,
+        # for example, we only want to simulate over a subset of the full 24
+        # hour calendar, so the execution times dictate a market open time of
+        # 6:31am US/Eastern and a close of 5:00pm US/Eastern.
+
+        # In our case, we are trading around the clock, so the market close
+        # corresponds to the last minute of the day.
+
+        # This method is taken from TradingAlgorithm.
+        # The clock has been replaced to use RealtimeClock
+        # TODO: should we apply a time skew? not sure to understand the utility.
+
+        log.debug('creating clock')
+        if self.live_graph:
+            self._clock = LiveGraphClock(
+                self.sim_params.sessions,
+                context=self
+            )
+        else:
+            self._clock = SimpleClock(
+                self.sim_params.sessions,
+            )
+
+        return self._clock
+
+    def _create_generator(self, sim_params):
+        if self.perf_tracker is None:
+            self.perf_tracker = get_algo_object(
+                algo_name=self.algo_namespace,
+                key='perf_tracker'
+            )
+
+        # Call the simulation trading algorithm for side-effects:
+        # it creates the perf tracker
+        TradingAlgorithm._create_generator(self, sim_params)
+        self.trading_client = ExchangeAlgorithmExecutor(
+            self,
+            sim_params,
+            self.data_portal,
+            self.clock,
+            self._create_benchmark_source(),
+            self.restrictions,
+            universe_func=self._calculate_universe
+        )
+
+        return self.trading_client.transform()
+
+    def updated_portfolio(self):
+        """
+        We skip the entire performance tracker business and update the
+        portfolio directly.
+        :return:
+        """
+        # TODO: build cumulative portfolio
+        return self.perf_tracker.get_portfolio(False)
+
+    def updated_account(self):
+        return self.perf_tracker.get_account(False)
+
+    def _synchronize_portfolio(self, attempt_index=0):
+        try:
+            for exchange_name in self.exchanges:
+                exchange = self.exchanges[exchange_name]
+
+                exchange.synchronize_portfolio()
+
+                # Applying the updated last_sales_price to the positions
+                # in the performance tracker. This seems a bit redundant
+                # but it will make sense when we have multiple exchange portfolios
+                # feeding into the same performance tracker.
+                tracker = self.perf_tracker.todays_performance.position_tracker
+                for asset in exchange.portfolio.positions:
+                    position = exchange.portfolio.positions[asset]
+                    tracker.update_position(
+                        asset=asset,
+                        last_sale_date=position.last_sale_date,
+                        last_sale_price=position.last_sale_price
+                    )
+        except ExchangeRequestError as e:
+            log.warn(
+                'update portfolio attempt {}: {}'.format(attempt_index, e)
+            )
+            if attempt_index < self.retry_synchronize_portfolio:
+                sleep(self.retry_delay)
+                self._synchronize_portfolio(attempt_index + 1)
+            else:
+                raise ExchangePortfolioDataError(
+                    data_type='update-portfolio',
+                    attempts=attempt_index,
+                    error=e
+                )
+
+    def _check_open_orders(self, attempt_index=0):
+        try:
+            orders = list()
+            for exchange_name in self.exchanges:
+                exchange = self.exchanges[exchange_name]
+                exchange_orders = exchange.check_open_orders()
+
+                orders += exchange_orders
+
+            return orders
+        except ExchangeRequestError as e:
+            log.warn(
+                'check open orders attempt {}: {}'.format(attempt_index, e)
+            )
+            if attempt_index < self.retry_check_open_orders:
+                sleep(self.retry_delay)
+                return self._check_open_orders(attempt_index + 1)
+            else:
+                raise ExchangePortfolioDataError(
+                    data_type='order-status',
+                    attempts=attempt_index,
+                    error=e
+                )
+
+    def add_pnl_stats(self, period_stats):
+        starting = period_stats['starting_cash']
+        current = period_stats['portfolio_value']
+        appreciation = (current / starting) - 1
+        perc = (appreciation * 100) if current != 0 else 0
+
+        log.debug('adding pnl stats: {:6f}%'.format(perc))
+
+        df = pd.DataFrame(
+            data=[dict(performance=perc)],
+            index=[period_stats['period_close']]
+        )
+        self.pnl_stats = pd.concat([self.pnl_stats, df])
+
+        save_algo_df(self.algo_namespace, 'pnl_stats', self.pnl_stats)
+
+    def add_custom_signals_stats(self, period_stats):
+        log.debug('adding custom signals stats: {}'.format(self.recorded_vars))
+        df = pd.DataFrame(
+            data=[self.recorded_vars],
+            index=[period_stats['period_close']],
+        )
+        self.custom_signals_stats = pd.concat([self.custom_signals_stats, df])
+
+        save_algo_df(self.algo_namespace, 'custom_signals_stats',
+                     self.custom_signals_stats)
+
+    def add_exposure_stats(self, period_stats):
+        data = dict(
+            long_exposure=period_stats['long_exposure'],
+            base_currency=period_stats['ending_cash']
+        )
+        log.debug('adding exposure stats: {}'.format(data))
+
+        df = pd.DataFrame(
+            data=[data],
+            index=[period_stats['period_close']],
+        )
+        self.exposure_stats = pd.concat([self.exposure_stats, df])
+
+        save_algo_df(self.algo_namespace, 'exposure_stats',
+                     self.exposure_stats)
+
+    def handle_data(self, data):
+        if not self.is_running:
+            return
+
+        self._synchronize_portfolio()
+
+        transactions = self._check_open_orders()
+        for transaction in transactions:
+            self.perf_tracker.process_transaction(transaction)
+
+        if self._handle_data:
+            self._handle_data(self, data)
+
+        # Unlike trading controls which remain constant unless placing an
+        # order, account controls can change each bar. Thus, must check
+        # every bar no matter if the algorithm places an order or not.
+        self.validate_account_controls()
+
+        try:
+            # Since the clock runs 24/7, I trying to disable the daily
+            # Performance tracker and keep only minute and cumulative
+            self.perf_tracker.update_performance()
+
+            minute_stats = self.prepare_period_stats(
+                data.current_dt, data.current_dt + timedelta(minutes=1))
+
+            # Saving the last hour in memory
+            self.minute_stats.append(minute_stats)
+
+            self.add_pnl_stats(minute_stats)
+            if self.recorded_vars:
+                self.add_custom_signals_stats(minute_stats)
+                recorded_cols = self.recorded_vars.keys()
+            else:
+                recorded_cols = None
+
+            self.add_exposure_stats(minute_stats)
+
+            print_df = pd.DataFrame(list(self.minute_stats))
+            log.info(
+                'statistics for the last {stats_minutes} minutes:\n{stats}'.format(
+                    stats_minutes=self.stats_minutes,
+                    stats=get_pretty_stats(
+                        stats_df=print_df,
+                        recorded_cols=recorded_cols,
+                        num_rows=self.stats_minutes
+                    )
+                ))
+
+            today = pd.to_datetime('today', utc=True)
+            daily_stats = self.prepare_period_stats(
+                start_dt=today,
+                end_dt=pd.Timestamp.utcnow()
+            )
+            save_algo_object(
+                algo_name=self.algo_namespace,
+                key=today.strftime('%Y-%m-%d'),
+                obj=daily_stats,
+                rel_path='daily_perf'
+            )
+
+        except Exception as e:
+            log.warn('unable to calculate performance: {}'.format(e))
+
+        try:
+            save_algo_object(
+                algo_name=self.algo_namespace,
+                key='perf_tracker',
+                obj=self.perf_tracker
+            )
+        except Exception as e:
+            log.warn('unable to save minute perfs to disk: {}'.format(e))
+
+        try:
+            for exchange_name in self.exchanges:
+                exchange = self.exchanges[exchange_name]
+                save_algo_object(
+                    algo_name=self.algo_namespace,
+                    key='portfolio_{}'.format(exchange_name),
+                    obj=exchange.portfolio
+                )
+        except Exception as e:
+            log.warn('unable to save portfolio to disk: {}'.format(e))
+
+    def _order(self,
+               asset,
+               amount,
+               limit_price=None,
+               stop_price=None,
+               style=None,
+               attempt_index=0):
+        try:
+            exchange = self.exchanges[asset.exchange]
+            return exchange.order(asset, amount, limit_price,
+                                  stop_price,
+                                  style)
+        except ExchangeRequestError as e:
+            log.warn(
+                'order attempt {}: {}'.format(attempt_index, e)
+            )
+            if attempt_index < self.retry_order:
+                sleep(self.retry_delay)
+                return self._order(
+                    asset, amount, limit_price, stop_price, style,
+                    attempt_index + 1)
+            else:
+                raise ExchangeTransactionError(
+                    transaction_type='order',
+                    attempts=attempt_index,
+                    error=e
+                )
+
+    @api_method
+    @disallowed_in_before_trading_start(OrderInBeforeTradingStart())
+    @expect_types(asset=TradingPair)
+    def order(self,
+              asset,
+              amount,
+              limit_price=None,
+              stop_price=None,
+              style=None):
+        """
+        We use the exchange specific portfolio to place orders.
+        The cumulative portfolio does not contain open orders but exchange
+        portfolios do.
+
+        :param asset: TradingPair
+        :param amount: float
+        :param limit_price: float
+        :param stop_price: float
+        :param style: Style
+        :return order: Order
+            The catalyst order object or None
+        """
+
+        amount, style = self._calculate_order(asset, amount,
+                                              limit_price, stop_price,
+                                              style)
+
+        order_id = self._order(asset, amount, limit_price, stop_price, style)
+
+        exchange = self.exchanges[asset.exchange]
+        exchange_portfolio = exchange.portfolio
+        if order_id is not None:
+
+            if order_id in exchange_portfolio.open_orders:
+                order = exchange_portfolio.open_orders[order_id]
+                self.perf_tracker.process_order(order)
+                return order
+
+            else:
+                raise OrphanOrderError(
+                    order_id=order_id,
+                    exchange=exchange.name
+                )
+        else:
+            log.warn('unable to order {} {} on exchange {}'.format(
+                amount, asset.symbol, asset.exchange))
+            return None
+
+    @api_method
+    def batch_market_order(self, share_counts):
+        raise NotImplementedError()
+
+    def _get_open_orders(self, asset=None, attempt_index=0):
+        try:
+            if asset:
+                exchange = self.exchanges[asset.exchange]
+                return exchange.get_open_orders(asset)
+
+            else:
+                open_orders = []
+                for exchange_name in self.exchanges:
+                    exchange = self.exchanges[exchange_name]
+                    exchange_orders = exchange.get_open_orders()
+                    open_orders.append(exchange_orders)
+
+                return open_orders
+        except ExchangeRequestError as e:
+            log.warn(
+                'open orders attempt {}: {}'.format(attempt_index, e)
+            )
+            if attempt_index < self.retry_get_open_orders:
+                sleep(self.retry_delay)
+                return self._get_open_orders(asset, attempt_index + 1)
+            else:
+                raise ExchangePortfolioDataError(
+                    data_type='open-orders',
+                    attempts=attempt_index,
+                    error=e
+                )
+
+    @error_keywords(sid='Keyword argument `sid` is no longer supported for '
+                        'get_open_orders. Use `asset` instead.')
+    @api_method
+    def get_open_orders(self, asset=None):
+        return self._get_open_orders(asset)
+
+    @api_method
+    def get_order(self, order_id, exchange_name):
+        exchange = self.exchanges[exchange_name]
+        return exchange.get_order(order_id)
+
+    @api_method
+    def cancel_order(self, order_param, exchange_name):
+        exchange = self.exchanges[exchange_name]
+
+        order_id = order_param
+        if isinstance(order_param, zp.Order):
+            order_id = order_param.id
+
+        exchange.cancel_order(order_id)

catalyst/exchange/exchange_bcolz.py

@@ -0,0 +1,89 @@
+import numpy as np
+
+from catalyst import get_calendar
+from catalyst.data.minute_bars import BcolzMinuteBarReader, \
+    BcolzMinuteBarWriter
+
+
+class BcolzExchangeBarWriter(BcolzMinuteBarWriter):
+    def __init__(self, *args, **kwargs):
+        self._data_frequency = kwargs.pop('data_frequency', None)
+        kwargs.pop('minutes_per_day', None)
+        kwargs.pop('calendar', None)
+
+        end_session = kwargs.pop('end_session', None)
+        if end_session is not None:
+            end_session = end_session.floor('1d')
+
+        minutes_per_day = 1440 if self._data_frequency == 'minute' else 1
+        default_ohlc_ratio = kwargs.pop('default_ohlc_ratio', 1000000)
+        calendar = get_calendar('OPEN')
+
+        super(BcolzExchangeBarWriter, self) \
+            .__init__(*args, **dict(kwargs,
+                                    minutes_per_day=minutes_per_day,
+                                    default_ohlc_ratio=default_ohlc_ratio,
+                                    calendar=calendar,
+                                    end_session=end_session
+                                    ))
+
+
+class BcolzExchangeBarReader(BcolzMinuteBarReader):
+    def __init__(self, *args, **kwargs):
+        self._data_frequency = kwargs.pop('data_frequency', None)
+
+        super(BcolzExchangeBarReader, self).__init__(*args, **kwargs)
+
+    @property
+    def data_frequency(self):
+        return self._data_frequency
+
+    def load_raw_arrays(self, fields, start_dt, end_dt, sids):
+
+        # if self._data_frequency == 'minute':
+        #     return super(BcolzExchangeBarReader, self) \
+        #         .load_raw_arrays(fields, start_dt, end_dt, sids)
+        #
+        # else:
+        #     return self._load_daily_raw_arrays(fields, start_dt, end_dt, sids)
+
+        return self._load_raw_arrays(fields, start_dt, end_dt, sids)
+
+    def _load_raw_arrays(self, fields, start_dt, end_dt, sids):
+        start_idx = self._find_position_of_minute(start_dt)
+        end_idx = self._find_position_of_minute(end_dt)
+
+        periods = self.calendar.minutes_in_range(start_dt, end_dt) \
+            if self.data_frequency == 'minute' \
+            else self.calendar.sessions_in_range(start_dt, end_dt)
+
+        num_days = len(periods)
+        shape = num_days, len(sids)
+
+        all_fields = fields[:]
+        if len(all_fields) == 1 and all_fields[0] == 'volume':
+            all_fields.insert(0, 'close')
+
+        mask = None
+        data = []
+        for field in all_fields:
+            if field != 'volume':
+                out = np.full(shape, np.nan)
+            else:
+                out = np.zeros(shape, dtype=np.float64)
+
+            for i, sid in enumerate(sids):
+                carray = self._open_minute_file(field, sid)
+                a = carray[start_idx:end_idx + 1]
+
+                if mask is None:
+                    mask = a != 0
+
+                out[:len(mask), i][mask] = (
+                    a[mask] * self._ohlc_ratio_inverse_for_sid(sid)
+                )
+
+            if field in fields:
+                data.append(out)
+
+        return data

catalyst/exchange/exchange_blotter.py

@@ -0,0 +1,138 @@
+from catalyst.assets._assets import TradingPair
+from logbook import Logger
+
+from catalyst.constants import LOG_LEVEL
+from catalyst.finance.blotter import Blotter
+from catalyst.finance.commission import CommissionModel
+from catalyst.finance.slippage import SlippageModel
+from catalyst.finance.transaction import Transaction
+
+log = Logger('exchange_blotter', level=LOG_LEVEL)
+
+# It seems like we need to accept greater slippage risk in cryptos
+# Orders won't often close at Equity levels.
+# TODO: consider adjusting dynamically based on trading pair
+DEFAULT_SLIPPAGE_SPREAD = 0.02
+DEFAULT_MAKER_FEE = 0.001
+DEFAULT_TAKER_FEE = 0.002
+
+
+class TradingPairFeeSchedule(CommissionModel):
+    """
+    Calculates a commission for a transaction based on a per percentage fee.
+
+    Parameters
+    ----------
+    fee : float, optional
+        The percentage fee.
+    """
+
+    def __init__(self,
+                 maker_fee=DEFAULT_MAKER_FEE,
+                 taker_fee=DEFAULT_TAKER_FEE):
+        self.maker_fee = maker_fee
+        self.taker_fee = taker_fee
+
+    def __repr__(self):
+        return (
+            '{class_name}(maker_fee={maker_fee}, '
+            'taker_fee={taker_fee})'.format(
+                class_name=self.__class__.__name__,
+                maker_fee=self.maker_fee,
+                taker_fee=self.taker_fee,
+            )
+        )
+
+    def calculate(self, order, transaction):
+        """
+        Calculate the final fee based on the order parameters.
+
+        :param order:
+        :param transaction:
+
+        :return float:
+            The total commission.
+        """
+        cost = abs(transaction.amount) * transaction.price
+
+        # Assuming just the taker fee for now
+        fee = cost * self.taker_fee
+        return fee
+
+
+class TradingPairFixedSlippage(SlippageModel):
+    """
+    Model slippage as a fixed spread.
+
+    Parameters
+    ----------
+    spread : float, optional
+        spread / 2 will be added to buys and subtracted from sells.
+    """
+
+    def __init__(self, spread=DEFAULT_SLIPPAGE_SPREAD):
+        super(TradingPairFixedSlippage, self).__init__()
+        self.spread = spread
+
+    def __repr__(self):
+        return '{class_name}(spread={spread})'.format(
+            class_name=self.__class__.__name__, spread=self.spread,
+        )
+
+    def simulate(self, data, asset, orders_for_asset):
+        self._volume_for_bar = 0
+
+        price = data.current(asset, 'close')
+
+        dt = data.current_dt
+        for order in orders_for_asset:
+            if order.open_amount == 0:
+                continue
+
+            order.check_triggers(price, dt)
+            if not order.triggered:
+                log.debug('order has not reached the trigger at current '
+                          'price {}'.format(price))
+                continue
+
+            execution_price, execution_volume = self.process_order(data, order)
+
+            transaction = Transaction(
+                asset=order.asset,
+                amount=abs(execution_volume),
+                dt=dt,
+                price=execution_price,
+                order_id=order.id
+            )
+
+            self._volume_for_bar += abs(transaction.amount)
+            yield order, transaction
+
+    def process_order(self, data, order):
+        price = data.current(order.asset, 'close')
+
+        if order.amount > 0:
+            # Buy order
+            adj_price = price * (1 + self.spread)
+        else:
+            # Sell order
+            adj_price = price * (1 - self.spread)
+
+        log.debug('added slippage to price: {} => {}'.format(price, adj_price))
+
+        return adj_price, order.amount
+
+
+class ExchangeBlotter(Blotter):
+    def __init__(self, *args, **kwargs):
+        super(ExchangeBlotter, self).__init__(*args, **kwargs)
+
+        # Using the equity models for now
+        # We may be able to define more sophisticated models based on the fee
+        # structure of each exchange.
+        self.slippage_models = {
+            TradingPair: TradingPairFixedSlippage()
+        }
+        self.commission_models = {
+            TradingPair: TradingPairFeeSchedule()
+        }

catalyst/exchange/exchange_bundle.py

@@ -0,0 +1,604 @@
+import os
+import shutil
+from datetime import timedelta
+
+import pandas as pd
+from logbook import Logger
+
+from catalyst import get_calendar
+from catalyst.constants import LOG_LEVEL
+from catalyst.data.minute_bars import BcolzMinuteOverlappingData, \
+    BcolzMinuteBarMetadata
+from catalyst.exchange.bundle_utils import range_in_bundle, \
+    get_bcolz_chunk, get_delta, get_adj_dates, get_month_start_end, \
+    get_year_start_end, get_periods_range, get_df_from_arrays, get_start_dt
+from catalyst.exchange.exchange_bcolz import BcolzExchangeBarReader, \
+    BcolzExchangeBarWriter
+from catalyst.exchange.exchange_errors import EmptyValuesInBundleError, \
+    InvalidHistoryFrequencyError, TempBundleNotFoundError, \
+    NoDataAvailableOnExchange, \
+    PricingDataNotLoadedError
+from catalyst.exchange.exchange_utils import get_exchange_folder
+from catalyst.utils.cli import maybe_show_progress
+from catalyst.utils.paths import ensure_directory
+
+log = Logger('exchange_bundle', level=LOG_LEVEL)
+
+BUNDLE_NAME_TEMPLATE = os.path.join('{root}','{frequency}_bundle')
+
+def _cachpath(symbol, type_):
+    return '-'.join([symbol, type_])
+
+class ExchangeBundle:
+    def __init__(self, exchange):
+        self.exchange = exchange
+        self.minutes_per_day = 1440
+        self.default_ohlc_ratio = 1000000
+        self._writers = dict()
+        self._readers = dict()
+        self.calendar = get_calendar('OPEN')
+
+    def get_assets(self, include_symbols, exclude_symbols):
+        # TODO: filter exclude symbols assets
+        if include_symbols is not None:
+            include_symbols_list = include_symbols.split(',')
+
+            return self.exchange.get_assets(include_symbols_list)
+
+        else:
+            return self.exchange.get_assets()
+
+    def get_reader(self, data_frequency, path=None):
+        """
+        Get a data writer object, either a new object or from cache
+
+        :return: BcolzMinuteBarReader or BcolzDailyBarReader
+        """
+        if path is None:
+            root = get_exchange_folder(self.exchange.name)
+            path = BUNDLE_NAME_TEMPLATE.format(
+                root=root,
+                frequency=data_frequency
+            )
+
+        if path in self._readers and self._readers[path] is not None:
+            return self._readers[path]
+
+        try:
+            self._readers[path] = BcolzExchangeBarReader(
+                rootdir=path,
+                data_frequency=data_frequency
+            )
+        except IOError:
+            self._readers[path] = None
+
+        return self._readers[path]
+
+    def update_metadata(self, writer, start_dt, end_dt):
+        pass
+
+    def get_writer(self, start_dt, end_dt, data_frequency):
+        """
+        Get a data writer object, either a new object or from cache
+
+        :return: BcolzMinuteBarWriter or BcolzDailyBarWriter
+        """
+        root = get_exchange_folder(self.exchange.name)
+        path = BUNDLE_NAME_TEMPLATE.format(
+            root=root,
+            frequency=data_frequency
+        )
+
+        if path in self._writers:
+            return self._writers[path]
+
+        ensure_directory(path)
+
+        if len(os.listdir(path)) > 0:
+
+            metadata = BcolzMinuteBarMetadata.read(path)
+
+            write_metadata = False
+            if start_dt < metadata.start_session:
+                write_metadata = True
+                start_session = start_dt
+            else:
+                start_session = metadata.start_session
+
+            if end_dt > metadata.end_session:
+                write_metadata = True
+
+                end_session = end_dt
+            else:
+                end_session = metadata.end_session
+
+            self._writers[path] = \
+                BcolzExchangeBarWriter(
+                    rootdir=path,
+                    start_session=start_session,
+                    end_session=end_session,
+                    write_metadata=write_metadata,
+                    data_frequency=data_frequency
+                )
+        else:
+            self._writers[path] = BcolzExchangeBarWriter(
+                rootdir=path,
+                start_session=start_dt,
+                end_session=end_dt,
+                write_metadata=True,
+                data_frequency=data_frequency
+            )
+
+        return self._writers[path]
+
+    def filter_existing_assets(self, assets, start_dt, end_dt, data_frequency):
+        """
+        For each asset, get the close on the start and end dates of the chunk.
+            If the data exists, the chunk ingestion is complete.
+            If any data is missing we ingest the data.
+
+        :param assets: list[TradingPair]
+            The assets is scope.
+        :param start_dt:
+            The chunk start date.
+        :param end_dt:
+            The chunk end date.
+        :return: list[TradingPair]
+            The assets missing from the bundle
+        """
+        reader = self.get_reader(data_frequency)
+        missing_assets = []
+        for asset in assets:
+            has_data = range_in_bundle(asset, start_dt, end_dt, reader)
+
+            if not has_data:
+                missing_assets.append(asset)
+
+        return missing_assets
+
+    def _write(self, data, writer, data_frequency):
+        """
+        Write data to the writer
+
+        :param df:
+        :param writer:
+        :return:
+        """
+        try:
+            writer.write(
+                data=data,
+                show_progress=False,
+                invalid_data_behavior='raise'
+            )
+        except BcolzMinuteOverlappingData as e:
+            log.debug('chunk already exists: {}'.format(e))
+        except Exception as e:
+            log.warn('error when writing data: {}, trying again'.format(e))
+
+            # This is workaround, there is an issue with empty
+            # session_label when using a newly created writer
+            key = writer._rootdir if data_frequency == 'minute' \
+                else writer._filename
+
+            del self._writers[key]
+
+            writer = self.get_writer(writer._start_session,
+                                     writer._end_session, data_frequency)
+            writer.write(
+                data=data,
+                show_progress=False,
+                invalid_data_behavior='raise'
+            )
+
+    def get_calendar_periods_range(self, start_dt, end_dt, data_frequency):
+        return self.calendar.minutes_in_range(start_dt, end_dt) \
+            if data_frequency == 'minute' \
+            else self.calendar.sessions_in_range(start_dt, end_dt)
+
+    def ingest_ctable(self, asset, data_frequency, period, start_dt, end_dt,
+                      writer, empty_rows_behavior='strip', cleanup=False):
+        """
+        Merge a ctable bundle chunk into the main bundle for the exchange.
+
+        :param asset: TradingPair
+        :param data_frequency: str
+        :param period: str
+        :param writer:
+        :param empty_rows_behavior: str
+            Ensure that the bundle does not have any missing data.
+
+        :param cleanup: bool
+            Remove the temp bundle directory after ingestion.
+
+        :return:
+        """
+
+        path = get_bcolz_chunk(
+            exchange_name=self.exchange.name,
+            symbol=asset.symbol,
+            data_frequency=data_frequency,
+            period=period
+        )
+
+        reader = self.get_reader(data_frequency, path=path)
+        if reader is None:
+            raise TempBundleNotFoundError(path=path)
+
+        arrays = reader.load_raw_arrays(
+            sids=[asset.sid],
+            fields=['open', 'high', 'low', 'close', 'volume'],
+            start_dt=start_dt,
+            end_dt=end_dt
+        )
+
+        if not arrays:
+            return path
+
+        periods = self.get_calendar_periods_range(
+            start_dt, end_dt, data_frequency
+        )
+
+        df = get_df_from_arrays(arrays, periods)
+
+        if empty_rows_behavior is not 'ignore':
+            nan_rows = df[df.isnull().T.any().T].index
+
+            if len(nan_rows) > 0:
+                dates = []
+                previous_date = None
+                for row_date in nan_rows.values:
+                    row_date = pd.to_datetime(row_date)
+
+                    if previous_date is None:
+                        dates.append(row_date)
+
+                    else:
+                        seq_date = previous_date + get_delta(1, data_frequency)
+
+                        if row_date > seq_date:
+                            dates.append(previous_date)
+                            dates.append(row_date)
+
+                    previous_date = row_date
+
+                dates.append(pd.to_datetime(nan_rows.values[-1]))
+
+                name = path.split('/')[-1]
+                if empty_rows_behavior == 'warn':
+                    log.warn(
+                        '\n{name} with end minute {end_minute} has empty rows '
+                        'in ranges: {dates}'.format(
+                            name=name,
+                            end_minute=asset.end_minute,
+                            dates=dates
+                        )
+                    )
+
+                elif empty_rows_behavior == 'raise':
+                    raise EmptyValuesInBundleError(
+                        name=name,
+                        end_minute=asset.end_minute,
+                        dates=dates
+                    )
+                else:
+                    df.dropna(inplace=True)
+
+        data = []
+        if not df.empty:
+            df.sort_index(inplace=True)
+            data.append((asset.sid, df))
+        self._write(data, writer, data_frequency)
+
+        if cleanup:
+            log.debug('removing bundle folder following '
+                      'ingestion: {}'.format(path))
+            shutil.rmtree(path)
+
+        return path
+
+    def prepare_chunks(self, assets, data_frequency, start_dt, end_dt):
+        """
+        Split a price data request into chunks corresponding to individual
+        bundles.
+
+        :param assets:
+        :param data_frequency:
+        :param start_dt:
+        :param end_dt:
+        :return:
+        """
+        reader = self.get_reader(data_frequency)
+
+        chunks = []
+        for asset in assets:
+            try:
+                asset_start, asset_end = \
+                    get_adj_dates(start_dt, end_dt, [asset], data_frequency)
+
+            except NoDataAvailableOnExchange:
+                continue
+
+            start_dt = max(start_dt, self.calendar.first_trading_session)
+            start_dt = max(start_dt, asset_start)
+
+            # Aligning start / end dates with the daily calendar
+            sessions = get_periods_range(start_dt, end_dt, data_frequency) \
+                if data_frequency == 'minute' \
+                else self.calendar.sessions_in_range(start_dt, end_dt)
+
+            if asset_start < sessions[0]:
+                asset_start = sessions[0]
+
+            if asset_end > sessions[-1]:
+                asset_end = sessions[-1]
+
+            chunk_labels = []
+            dt = sessions[0]
+            while dt <= sessions[-1]:
+                label = '{}-{:02d}'.format(dt.year, dt.month) \
+                    if data_frequency == 'minute' else '{}'.format(dt.year)
+
+                if label not in chunk_labels:
+                    chunk_labels.append(label)
+
+                    # Adjusting the period dates to match the availability
+                    # of the trading pair
+                    if data_frequency == 'minute':
+                        period_start, period_end = get_month_start_end(dt)
+                        asset_start_month, _ = get_month_start_end(asset_start)
+
+                        if asset_start_month == period_start \
+                                and period_start < asset_start:
+                            period_start = asset_start
+
+                        _, asset_end_month = get_month_start_end(asset_end)
+                        if asset_end_month == period_end \
+                                and period_end > asset_end:
+                            period_end = asset_end
+
+                    elif data_frequency == 'daily':
+                        period_start, period_end = get_year_start_end(dt)
+                        asset_start_year, _ = get_year_start_end(asset_start)
+
+                        if asset_start_year == period_start \
+                                and period_start < asset_start:
+                            period_start = asset_start
+
+                        _, asset_end_year = get_year_start_end(asset_end)
+                        if asset_end_year == period_end \
+                                and period_end > asset_end:
+                            period_end = asset_end
+                    else:
+                        raise InvalidHistoryFrequencyError(
+                            frequency=data_frequency
+                        )
+
+                    # Currencies don't always start trading at midnight.
+                    # Checking the last minute of the day instead.
+                    range_start = period_start.replace(hour=23, minute=59) \
+                        if data_frequency == 'minute' else period_start
+                    has_data = range_in_bundle(
+                        asset, range_start, period_end, reader
+                    )
+
+                    if not has_data:
+                        log.debug('adding period: {}'.format(label))
+                        chunks.append(
+                            dict(
+                                asset=asset,
+                                period_start=period_start,
+                                period_end=period_end,
+                                period=label
+                            )
+                        )
+
+                dt += timedelta(days=1)
+
+        chunks.sort(key=lambda chunk: chunk['period_end'])
+
+        return chunks
+
+    def ingest_assets(self, assets, start_dt, end_dt, data_frequency,
+                      show_progress=False):
+        """
+        Determine if data is missing from the bundle and attempt to ingest it.
+
+        :param assets:
+        :param start_dt:
+        :param end_dt:
+        :return:
+        """
+        writer = self.get_writer(start_dt, end_dt, data_frequency)
+        chunks = self.prepare_chunks(
+            assets=assets,
+            data_frequency=data_frequency,
+            start_dt=start_dt,
+            end_dt=end_dt
+        )
+        with maybe_show_progress(
+                chunks,
+                show_progress,
+                label='Fetching {exchange} {frequency} candles: '.format(
+                    exchange=self.exchange.name,
+                    frequency=data_frequency
+                )) as it:
+            for chunk in it:
+                self.ingest_ctable(
+                    asset=chunk['asset'],
+                    data_frequency=data_frequency,
+                    period=chunk['period'],
+                    start_dt=chunk['period_start'],
+                    end_dt=chunk['period_end'],
+                    writer=writer,
+                    empty_rows_behavior='strip'
+                )
+
+    def ingest(self, data_frequency, include_symbols=None,
+               exclude_symbols=None, start=None, end=None,
+               show_progress=True, environ=os.environ):
+        """
+
+        :param data_frequency:
+        :param include_symbols:
+        :param exclude_symbols:
+        :param start:
+        :param end:
+        :param show_progress:
+        :param environ:
+        :return:
+        """
+        assets = self.get_assets(include_symbols, exclude_symbols)
+        start_dt, end_dt = get_adj_dates(start, end, assets, data_frequency)
+
+        for frequency in data_frequency.split(','):
+            self.ingest_assets(assets, start_dt, end_dt, frequency,
+                               show_progress)
+
+    def get_history_window_series_and_load(self,
+                                           assets,
+                                           end_dt,
+                                           bar_count,
+                                           field,
+                                           data_frequency):
+        try:
+            series = self.get_history_window_series(
+                assets=assets,
+                end_dt=end_dt,
+                bar_count=bar_count,
+                field=field,
+                data_frequency=data_frequency
+            )
+            return pd.DataFrame(series)
+
+        except PricingDataNotLoadedError:
+            start_dt = get_start_dt(end_dt, bar_count, data_frequency)
+            log.info(
+                'pricing data for {symbol} not found in range '
+                '{start} to {end}, updating the bundles.'.format(
+                    symbol=[asset.symbol for asset in assets],
+                    start=start_dt,
+                    end=end_dt
+                )
+            )
+            self.ingest_assets(
+                assets=assets,
+                start_dt=start_dt,
+                end_dt=end_dt,
+                data_frequency=data_frequency,
+                show_progress=True
+            )
+            series = self.get_history_window_series(
+                assets=assets,
+                end_dt=end_dt,
+                bar_count=bar_count,
+                field=field,
+                data_frequency=data_frequency,
+                reset_reader=True
+            )
+            return series
+
+    def get_spot_values(self, assets, field, dt, data_frequency,
+                        reset_reader=False):
+        values = []
+        try:
+            reader = self.get_reader(data_frequency)
+            if reset_reader:
+                del self._readers[reader._rootdir]
+                reader = self.get_reader(data_frequency)
+
+            for asset in assets:
+                value = reader.get_value(
+                    sid=asset.sid,
+                    dt=dt,
+                    field=field
+                )
+                values.append(value)
+
+            return values
+
+        except Exception:
+            symbols = [asset.symbol.encode('utf-8') for asset in assets]
+            raise PricingDataNotLoadedError(
+                field=field,
+                first_trading_day=min([asset.start_date for asset in assets]),
+                exchange=self.exchange.name,
+                symbols=symbols,
+                symbol_list=','.join(symbols),
+                data_frequency=data_frequency
+            )
+
+    def get_history_window_series(self,
+                                  assets,
+                                  end_dt,
+                                  bar_count,
+                                  field,
+                                  data_frequency,
+                                  reset_reader=False):
+        start_dt = get_start_dt(end_dt, bar_count, data_frequency)
+        start_dt, end_dt = \
+            get_adj_dates(start_dt, end_dt, assets, data_frequency)
+
+        reader = self.get_reader(data_frequency)
+        if reset_reader:
+            del self._readers[reader._rootdir]
+            reader = self.get_reader(data_frequency)
+
+        if reader is None:
+            symbols = [asset.symbol.encode('utf-8') for asset in assets]
+            raise PricingDataNotLoadedError(
+                field=field,
+                first_trading_day=min([asset.start_date for asset in assets]),
+                exchange=self.exchange.name,
+                symbols=symbols,
+                symbol_list=','.join(symbols),
+                data_frequency=data_frequency
+            )
+
+        for asset in assets:
+            asset_start_dt, asset_end_dt = \
+                get_adj_dates(start_dt, end_dt, assets, data_frequency)
+
+            in_bundle = range_in_bundle(
+                asset, asset_start_dt, asset_end_dt, reader
+            )
+            if not in_bundle:
+                raise PricingDataNotLoadedError(
+                    field=field,
+                    first_trading_day=asset.start_date,
+                    exchange=self.exchange.name,
+                    symbols=asset.symbol,
+                    symbol_list=asset.symbol,
+                    data_frequency=data_frequency
+                )
+
+        series = dict()
+        try:
+            arrays = reader.load_raw_arrays(
+                sids=[asset.sid for asset in assets],
+                fields=[field],
+                start_dt=start_dt,
+                end_dt=end_dt
+            )
+
+        except Exception:
+            symbols = [asset.symbol.encode('utf-8') for asset in assets]
+            raise PricingDataNotLoadedError(
+                field=field,
+                first_trading_day=min([asset.start_date for asset in assets]),
+                exchange=self.exchange.name,
+                symbols=symbols,
+                symbol_list=','.join(symbols),
+                data_frequency=data_frequency
+            )
+
+        periods = self.get_calendar_periods_range(
+            start_dt, end_dt, data_frequency
+        )
+
+        for asset_index, asset in enumerate(assets):
+            asset_values = arrays[asset_index]
+
+            value_series = pd.Series(asset_values.flatten(), index=periods)
+            series[asset] = value_series
+
+        return series

catalyst/exchange/exchange_errors.py

@@ -0,0 +1,220 @@
+import sys
+import traceback
+
+from catalyst.errors import ZiplineError
+
+
+def silent_except_hook(exctype, excvalue, exctraceback):
+    if exctype in [PricingDataBeforeTradingError, PricingDataNotLoadedError,
+                   SymbolNotFoundOnExchange, NoDataAvailableOnExchange, 
+                   ExchangeAuthEmpty ]:
+        fn = traceback.extract_tb(exctraceback)[-1][0]
+        ln = traceback.extract_tb(exctraceback)[-1][1]
+        print "Error traceback: {1} (line {2})\n" \
+              "{0.__name__}:  {3}".format(exctype, fn, ln, excvalue)
+    else:
+        sys.__excepthook__(exctype, excvalue, exctraceback)
+
+
+sys.excepthook = silent_except_hook
+
+
+class ExchangeRequestError(ZiplineError):
+    msg = (
+        'Request failed: {error}'
+    ).strip()
+
+
+class ExchangeRequestErrorTooManyAttempts(ZiplineError):
+    msg = (
+        'Request failed: {error}, giving up after {attempts} attempts'
+    ).strip()
+
+
+class ExchangeBarDataError(ZiplineError):
+    msg = (
+        'Unable to retrieve bar data: {data_type}, ' +
+        'giving up after {attempts} attempts: {error}'
+    ).strip()
+
+
+class ExchangePortfolioDataError(ZiplineError):
+    msg = (
+        'Unable to retrieve portfolio data: {data_type}, ' +
+        'giving up after {attempts} attempts: {error}'
+    ).strip()
+
+
+class ExchangeTransactionError(ZiplineError):
+    msg = (
+        'Unable to execute transaction: {transaction_type}, ' +
+        'giving up after {attempts} attempts: {error}'
+    ).strip()
+
+
+class ExchangeNotFoundError(ZiplineError):
+    msg = (
+        'Exchange {exchange_name} not found. Please specify exchanges '
+        'supported by Catalyst and verify spelling for accuracy.'
+    ).strip()
+
+
+class ExchangeAuthNotFound(ZiplineError):
+    msg = (
+        'Please create an auth.json file containing the api token and key for '
+        'exchange {exchange}. Place the file here: {filename}'
+    ).strip()
+
+
+class ExchangeAuthEmpty(ZiplineError):
+    msg = (
+        'Please enter your API token key and secret for exchange {exchange} '
+        'in the following file: {filename}'
+    ).strip()
+
+
+class ExchangeSymbolsNotFound(ZiplineError):
+    msg = (
+        'Unable to download or find a local copy of symbols.json for exchange '
+        '{exchange}. The file should be here: {filename}'
+    ).strip()
+
+
+class AlgoPickleNotFound(ZiplineError):
+    msg = (
+        'Pickle not found for algo {algo} in path {filename}'
+    ).strip()
+
+
+class InvalidHistoryFrequencyError(ZiplineError):
+    msg = (
+        'Frequency {frequency} not supported by the exchange.'
+    ).strip()
+
+
+class MismatchingFrequencyError(ZiplineError):
+    msg = (
+        'Bar aggregate frequency {frequency} not compatible with '
+        'data frequency {data_frequency}.'
+    ).strip()
+
+
+class InvalidSymbolError(ZiplineError):
+    msg = (
+        'Invalid trading pair symbol: {symbol}. '
+        'Catalyst symbols must follow this convention: '
+        '[Market Currency]_[Base Currency]. For example: eth_usd, btc_usd, '
+        'neo_eth, ubq_btc. Error details: {error}'
+    ).strip()
+
+
+class InvalidOrderStyle(ZiplineError):
+    msg = (
+        'Order style {style} not supported by exchange {exchange}.'
+    ).strip()
+
+
+class CreateOrderError(ZiplineError):
+    msg = (
+        'Unable to create order on exchange {exchange} {error}.'
+    ).strip()
+
+
+class OrderNotFound(ZiplineError):
+    msg = (
+        'Order {order_id} not found on exchange {exchange}.'
+    ).strip()
+
+
+class OrphanOrderError(ZiplineError):
+    msg = (
+        'Order {order_id} found in exchange {exchange} but not tracked by '
+        'the algorithm.'
+    ).strip()
+
+
+class OrphanOrderReverseError(ZiplineError):
+    msg = (
+        'Order {order_id} tracked by algorithm, but not found in exchange {exchange}.'
+    ).strip()
+
+
+class OrderCancelError(ZiplineError):
+    msg = (
+        'Unable to cancel order {order_id} on exchange {exchange} {error}.'
+    ).strip()
+
+
+class SidHashError(ZiplineError):
+    msg = (
+        'Unable to hash sid from symbol {symbol}.'
+    ).strip()
+
+
+class BaseCurrencyNotFoundError(ZiplineError):
+    msg = (
+        'Algorithm base currency {base_currency} not found in exchange '
+        '{exchange}.'
+    ).strip()
+
+
+class MismatchingBaseCurrencies(ZiplineError):
+    msg = (
+        'Unable to trade with base currency {base_currency} when the '
+        'algorithm uses {algo_currency}.'
+    ).strip()
+
+
+class MismatchingBaseCurrenciesExchanges(ZiplineError):
+    msg = (
+        'Unable to trade with base currency {base_currency} when the '
+        'exchange {exchange_name} users {exchange_currency}.'
+    ).strip()
+
+
+class SymbolNotFoundOnExchange(ZiplineError):
+    """
+    Raised when a symbol() call contains a non-existent symbol.
+    """
+    msg = ('Symbol {symbol} not found on exchange {exchange}. '
+           'Choose from: {supported_symbols}').strip()
+
+
+class BundleNotFoundError(ZiplineError):
+    msg = ('Unable to find bundle data for exchange {exchange} and '
+           'data frequency {data_frequency}.'
+           'Please ingest some price data.'
+           'See `catalyst ingest-exchange --help` for details.').strip()
+
+
+class TempBundleNotFoundError(ZiplineError):
+    msg = ('Temporary bundle not found in: {path}.').strip()
+
+
+class EmptyValuesInBundleError(ZiplineError):
+    msg = ('{name} with end minute {end_minute} has empty rows '
+           'in ranges: {dates}').strip()
+
+
+class PricingDataBeforeTradingError(ZiplineError):
+    msg = ('Pricing data for trading pairs {symbols} on exchange {exchange} '
+           'starts on {first_trading_day}, but you are either trying to trade or '
+           'retrieve pricing data on {dt}. Adjust your dates accordingly.').strip()
+
+
+class PricingDataNotLoadedError(ZiplineError):
+    msg = ('Pricing data {field} for trading pairs {symbols} trading on '
+           'exchange {exchange} since {first_trading_day} is unavailable. '
+           'The bundle data is either out-of-date or has not been loaded yet. '
+           'Please ingest data using the command '
+           '`catalyst ingest-exchange -x {exchange} -f {data_frequency} -i {symbol_list}`. '
+           'See catalyst documentation for details.').strip()
+
+
+class ApiCandlesError(ZiplineError):
+    msg = ('Unable to fetch candles from the remote API: {error}.').strip()
+
+class NoDataAvailableOnExchange(ZiplineError):
+    msg = ('Requested data for trading pair {symbol} is not available on exchange {exchange} '
+           'in `{data_frequency}` frequency at this time. '
+           'Check `http://enigma.co/catalyst/status` for market coverage.').strip()

catalyst/exchange/exchange_execution.py

@@ -0,0 +1,39 @@
+from catalyst.finance.execution import LimitOrder, StopOrder, StopLimitOrder
+
+
+class ExchangeLimitOrder(LimitOrder):
+    def get_limit_price(self, is_buy):
+        """
+        We may be trading Satoshis with 8 decimals, we cannot round numbers
+        :param is_buy:
+        :return:
+        """
+        return self.limit_price
+
+
+class ExchangeStopOrder(StopOrder):
+    def get_stop_price(self, is_buy):
+        """
+        We may be trading Satoshis with 8 decimals, we cannot round numbers
+        :param is_buy:
+        :return:
+        """
+        return self.stop_price
+
+
+class ExchangeStopLimitOrder(StopLimitOrder):
+    def get_limit_price(self, is_buy):
+        """
+        We may be trading Satoshis with 8 decimals, we cannot round numbers
+        :param is_buy:
+        :return:
+        """
+        return self.limit_price
+
+    def get_stop_price(self, is_buy):
+        """
+        We may be trading Satoshis with 8 decimals, we cannot round numbers
+        :param is_buy:
+        :return:
+        """
+        return self.stop_price

catalyst/exchange/exchange_portfolio.py

@@ -0,0 +1,112 @@
+import numpy as np
+from logbook import Logger
+
+from catalyst.constants import LOG_LEVEL
+from catalyst.protocol import Portfolio, Positions, Position
+
+log = Logger('ExchangePortfolio', level=LOG_LEVEL)
+
+
+class ExchangePortfolio(Portfolio):
+    """
+    Since the goal is to support multiple exchanges, it makes sense to
+    include additional stats in the portfolio object.
+
+    Instead of relying on the performance tracker, each exchange portfolio
+    tracks its own holding. This offers a separation between tracking an
+    exchange and the statistics of the algorithm.
+    """
+
+    def __init__(self, start_date, starting_cash=None):
+        self.capital_used = 0.0
+        self.starting_cash = starting_cash
+        self.portfolio_value = starting_cash
+        self.pnl = 0.0
+        self.returns = 0.0
+        self.cash = starting_cash
+        self.positions = Positions()
+        self.start_date = start_date
+        self.positions_value = 0.0
+        self.open_orders = dict()
+
+    def calculate_pnl(self):
+        log.debug('calculating pnl')
+
+    def create_order(self, order):
+        log.debug('creating order {}'.format(order.id))
+        self.open_orders[order.id] = order
+
+        order_position = self.positions[order.asset] \
+            if order.asset in self.positions else None
+
+        if order_position is None:
+            order_position = Position(order.asset)
+            self.positions[order.asset] = order_position
+
+        order_position.amount += order.amount
+        log.debug('open order added to portfolio')
+
+    def execute_order(self, order, transaction):
+        log.debug('executing order {}'.format(order.id))
+        del self.open_orders[order.id]
+
+        order_position = self.positions[order.asset] \
+            if order.asset in self.positions else None
+
+        if order_position is None:
+            raise ValueError(
+                'Trying to execute order for a position not held: %s' % order.id
+            )
+
+        self.capital_used += order.amount * transaction.price
+
+        if order.amount > 0:
+            if order_position.cost_basis > 0:
+                order_position.cost_basis = np.average(
+                    [order_position.cost_basis, transaction.price],
+                    weights=[order_position.amount, order.amount]
+                )
+            else:
+                order_position.cost_basis = transaction.price
+
+        log.debug('updated portfolio with executed order')
+
+    def execute_transaction(self, transaction):
+        log.debug('executing transaction {}'.format(transaction.order_id))
+
+        order_position = self.positions[transaction.asset] \
+            if transaction.asset in self.positions else None
+
+        if order_position is None:
+            raise ValueError(
+                'Trying to execute transaction for a position not held: %s' % transaction.order_id
+            )
+
+        self.capital_used += transaction.amount * transaction.price
+
+        if transaction.amount > 0:
+            if order_position.cost_basis > 0:
+                order_position.cost_basis = np.average(
+                    [order_position.cost_basis, transaction.price],
+                    weights=[order_position.amount, transaction.amount]
+                )
+            else:
+                order_position.cost_basis = transaction.price
+
+        log.debug('updated portfolio with executed order')
+
+    def remove_order(self, order):
+        log.info('removing cancelled order {}'.format(order.id))
+        del self.open_orders[order.id]
+
+        order_position = self.positions[order.asset] \
+            if order.asset in self.positions else None
+
+        if order_position is None:
+            raise ValueError(
+                'Trying to remove order for a position not held: %s' % order.id
+            )
+
+        order_position.amount -= order.amount
+
+        log.debug('removed order from portfolio')

catalyst/exchange/exchange_utils.py

@@ -0,0 +1,180 @@
+import json
+import os
+import pickle
+import urllib
+from datetime import date, datetime
+
+import pandas as pd
+
+from catalyst.exchange.exchange_errors import ExchangeAuthNotFound, \
+    ExchangeSymbolsNotFound
+from catalyst.utils.paths import data_root, ensure_directory, \
+    last_modified_time
+
+SYMBOLS_URL = 'https://s3.amazonaws.com/enigmaco/catalyst-exchanges/' \
+              '{exchange}/symbols.json'
+
+
+def get_exchange_folder(exchange_name, environ=None):
+    if not environ:
+        environ = os.environ
+
+    root = data_root(environ)
+    exchange_folder = os.path.join(root, 'exchanges', exchange_name)
+    ensure_directory(exchange_folder)
+
+    return exchange_folder
+
+
+def get_exchange_symbols_filename(exchange_name, environ=None):
+    exchange_folder = get_exchange_folder(exchange_name, environ)
+    return os.path.join(exchange_folder, 'symbols.json')
+
+
+def download_exchange_symbols(exchange_name, environ=None):
+    filename = get_exchange_symbols_filename(exchange_name)
+    url = SYMBOLS_URL.format(exchange=exchange_name)
+    response = urllib.urlretrieve(url=url, filename=filename)
+    return response
+
+
+def get_exchange_symbols(exchange_name, environ=None):
+    filename = get_exchange_symbols_filename(exchange_name)
+
+    if not os.path.isfile(filename) or \
+            pd.Timedelta(pd.Timestamp('now', tz='UTC') - last_modified_time(filename)).days > 1:
+        download_exchange_symbols(exchange_name, environ)
+
+    if os.path.isfile(filename):
+        with open(filename) as data_file:
+            data = json.load(data_file)
+            return data
+    else:
+        raise ExchangeSymbolsNotFound(
+            exchange=exchange_name,
+            filename=filename
+        )
+
+
+def get_exchange_auth(exchange_name, environ=None):
+    exchange_folder = get_exchange_folder(exchange_name, environ)
+    filename = os.path.join(exchange_folder, 'auth.json')
+
+    if os.path.isfile(filename):
+        with open(filename) as data_file:
+            data = json.load(data_file)
+            return data
+    else:
+        data = dict(name=exchange_name, key='', secret='')
+        with open(filename, 'w') as f:
+            json.dump(data, f, sort_keys=False, indent=2, separators=(',', ':'))
+            return data
+
+def get_algo_folder(algo_name, environ=None):
+    if not environ:
+        environ = os.environ
+
+    root = data_root(environ)
+    algo_folder = os.path.join(root, 'live_algos', algo_name)
+    ensure_directory(algo_folder)
+
+    return algo_folder
+
+
+def get_algo_object(algo_name, key, environ=None, rel_path=None):
+    if algo_name is None:
+        return None
+
+    folder = get_algo_folder(algo_name, environ)
+
+    if rel_path is not None:
+        folder = os.path.join(folder, rel_path)
+
+    filename = os.path.join(folder, key + '.p')
+
+    if os.path.isfile(filename):
+        try:
+            with open(filename, 'rb') as handle:
+                return pickle.load(handle)
+        except Exception as e:
+            return None
+    else:
+        return None
+
+
+def save_algo_object(algo_name, key, obj, environ=None, rel_path=None):
+    folder = get_algo_folder(algo_name, environ)
+
+    if rel_path is not None:
+        folder = os.path.join(folder, rel_path)
+        ensure_directory(folder)
+
+    filename = os.path.join(folder, key + '.p')
+
+    with open(filename, 'wb') as handle:
+        pickle.dump(obj, handle, protocol=pickle.HIGHEST_PROTOCOL)
+
+
+def append_algo_object(algo_name, key, obj, environ=None):
+    algo_folder = get_algo_folder(algo_name, environ)
+    filename = os.path.join(algo_folder, key + '.p')
+
+    mode = 'a+b' if os.path.isfile(filename) else 'wb'
+    with open(filename, mode) as handle:
+        pickle.dump(obj, handle, protocol=pickle.HIGHEST_PROTOCOL)
+
+
+def get_algo_df(algo_name, key, environ=None, rel_path=None):
+    folder = get_algo_folder(algo_name, environ)
+
+    if rel_path is not None:
+        folder = os.path.join(folder, rel_path)
+
+    filename = os.path.join(folder, key + '.csv')
+
+    if os.path.isfile(filename):
+        try:
+            with open(filename, 'rb') as handle:
+                return pd.read_csv(handle, index_col=0, parse_dates=True)
+        except IOError:
+            return pd.DataFrame()
+    else:
+        return pd.DataFrame()
+
+
+def save_algo_df(algo_name, key, df, environ=None, rel_path=None):
+    folder = get_algo_folder(algo_name, environ)
+
+    if rel_path is not None:
+        folder = os.path.join(folder, rel_path)
+        ensure_directory(folder)
+
+    filename = os.path.join(folder, key + '.csv')
+
+    with open(filename, 'wb') as handle:
+        df.to_csv(handle)
+
+
+def get_exchange_minute_writer_root(exchange_name, environ=None):
+    exchange_folder = get_exchange_folder(exchange_name, environ)
+
+    minute_data_folder = os.path.join(exchange_folder, 'minute_data')
+    ensure_directory(minute_data_folder)
+
+    return minute_data_folder
+
+def get_exchange_bundles_folder(exchange_name, environ=None):
+    exchange_folder = get_exchange_folder(exchange_name, environ)
+
+    temp_bundles = os.path.join(exchange_folder, 'temp_bundles')
+    ensure_directory(temp_bundles)
+
+    return temp_bundles
+
+
+def perf_serial(obj):
+    """JSON serializer for objects not serializable by default json code"""
+
+    if isinstance(obj, (datetime, date)):
+        return obj.isoformat()
+    raise TypeError("Type %s not serializable" % type(obj))

catalyst/exchange/init_utils.py

@@ -0,0 +1,32 @@
+from catalyst.exchange.bitfinex.bitfinex import Bitfinex
+from catalyst.exchange.bittrex.bittrex import Bittrex
+from catalyst.exchange.exchange_errors import ExchangeNotFoundError
+from catalyst.exchange.exchange_utils import get_exchange_auth
+from catalyst.exchange.poloniex.poloniex import Poloniex
+
+
+def get_exchange(exchange_name):
+    exchange_auth = get_exchange_auth(exchange_name)
+    if exchange_name == 'bitfinex':
+        return Bitfinex(
+            key=exchange_auth['key'],
+            secret=exchange_auth['secret'],
+            base_currency=None,  # TODO: make optional at the exchange
+            portfolio=None
+        )
+    elif exchange_name == 'bittrex':
+        return Bittrex(
+            key=exchange_auth['key'],
+            secret=exchange_auth['secret'],
+            base_currency=None,
+            portfolio=None
+        )
+    elif exchange_name == 'poloniex':
+        return Poloniex(
+            key=exchange_auth['key'],
+            secret=exchange_auth['secret'],
+            base_currency=None,
+            portfolio=None
+        )
+    else:
+        raise ExchangeNotFoundError(exchange_name=exchange_name)

catalyst/exchange/live_graph_clock.py

@@ -0,0 +1,225 @@
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import pandas as pd
+from catalyst.gens.sim_engine import (
+    BAR,
+    SESSION_START
+)
+from logbook import Logger
+
+from catalyst.constants import LOG_LEVEL
+from catalyst.exchange.exchange_errors import \
+    MismatchingBaseCurrenciesExchanges
+
+log = Logger('LiveGraphClock', level=LOG_LEVEL)
+
+
+class LiveGraphClock(object):
+    """Realtime clock for live trading.
+
+    This class is a drop-in replacement for
+    :class:`zipline.gens.sim_engine.MinuteSimulationClock`.
+
+    This mixes the clock with a live graph.
+
+    Note
+    ----
+    This seemingly awkward approach allows us to run the program using a single
+    thread. This is important because Matplotlib does not play nice with
+    multi-threaded environments. Zipline probably does not either.
+
+
+    Matplotlib has a pause() method which is a wrapper around time.sleep()
+    used in the SimpleClock. The key difference is that users
+    can still interact with the chart during the pause cycles. This is
+    what enables us to keep a single thread. This is also why we are not using
+    the 'animate' callback of Matplotlib. We need to direct access to the
+    __iter__ method in order to yield events to Zipline.
+
+    The :param:`time_skew` parameter represents the time difference between
+    the exchange and the live trading machine's clock. It's not used currently.
+    """
+
+    def __init__(self, sessions, context, time_skew=pd.Timedelta('0s')):
+
+        global mdates, plt                  #TODO: Could be cleaner
+        import matplotlib.dates as mdates
+        from matplotlib import pyplot as plt
+        from matplotlib import style
+
+        self.sessions = sessions
+        self.time_skew = time_skew
+        self._last_emit = None
+        self._before_trading_start_bar_yielded = True
+        self.context = context
+        self.fmt = mdates.DateFormatter('%Y-%m-%d %H:%M')
+
+        style.use('dark_background')
+
+        fig = plt.figure()
+        fig.canvas.set_window_title('Enigma Catalyst: {}'.format(
+            self.context.algo_namespace))
+
+        self.ax_pnl = fig.add_subplot(311)
+
+        self.ax_custom_signals = fig.add_subplot(312, sharex=self.ax_pnl)
+
+        self.ax_exposure = fig.add_subplot(313, sharex=self.ax_pnl)
+
+        if len(context.minute_stats) > 0:
+            self.draw_pnl()
+            self.draw_custom_signals()
+            self.draw_exposure()
+
+        # rotates and right aligns the x labels, and moves the bottom of the
+        # axes up to make room for them
+        fig.autofmt_xdate()
+        fig.subplots_adjust(hspace=0.5)
+
+        plt.tight_layout()
+        plt.ion()
+        plt.show()
+
+    def format_ax(self, ax):
+        """
+        Trying to assign reasonable parameters to the time axis.
+
+        TODO: room for improvement
+
+        :param ax:
+        :return:
+        """
+        ax.xaxis.set_major_locator(mdates.DayLocator(interval=1))
+        ax.xaxis.set_major_formatter(self.fmt)
+
+        locator = mdates.HourLocator(interval=4)
+        locator.MAXTICKS = 5000
+        ax.xaxis.set_minor_locator(locator)
+
+        datemin = pd.Timestamp.utcnow()
+        ax.set_xlim(datemin)
+
+        ax.grid(True)
+
+    def set_legend(self, ax):
+        ax.legend(loc='upper left', ncol=1, fontsize=10, numpoints=1)
+
+    def draw_pnl(self):
+        ax = self.ax_pnl
+        df = self.context.pnl_stats
+
+        ax.clear()
+        ax.set_title('Performance')
+        ax.plot(df.index, df['performance'], '-',
+                color='green',
+                linewidth=1.0,
+                label='Performance'
+                )
+
+        def perc(val):
+            return '{:2f}'.format(val)
+
+        ax.format_ydata = perc
+
+        self.set_legend(ax)
+        self.format_ax(ax)
+
+    def draw_custom_signals(self):
+        ax = self.ax_custom_signals
+        df = self.context.custom_signals_stats
+
+        colors = ['blue', 'green', 'red', 'black', 'orange', 'yellow', 'pink']
+
+        ax.clear()
+        ax.set_title('Custom Signals')
+        for index, column in enumerate(df.columns.values.tolist()):
+            ax.plot(df.index, df[column], '-',
+                    color=colors[index],
+                    linewidth=1.0,
+                    label=column
+                    )
+
+        self.set_legend(ax)
+        self.format_ax(ax)
+
+    def draw_exposure(self):
+        ax = self.ax_exposure
+        context = self.context
+        df = context.exposure_stats
+
+        # TODO: list exchanges in graph
+        base_currency = None
+        positions = []
+        for exchange_name in context.exchanges:
+            exchange = context.exchanges[exchange_name]
+
+            if not base_currency:
+                base_currency = exchange.base_currency
+            elif base_currency != exchange.base_currency:
+                raise MismatchingBaseCurrenciesExchanges(
+                    base_currency=base_currency,
+                    exchange_name=exchange.name,
+                    exchange_currency=exchange.base_currency
+                )
+
+            positions += exchange.portfolio.positions
+
+        ax.clear()
+        ax.set_title('Exposure')
+        ax.plot(df.index, df['base_currency'], '-',
+                color='green',
+                linewidth=1.0,
+                label='Base Currency: {}'.format(base_currency.upper())
+                )
+
+        symbols = []
+        for position in positions:
+            symbols.append(position.symbol)
+
+        ax.plot(df.index, df['long_exposure'], '-',
+                color='blue',
+                linewidth=1.0,
+                label='Long Exposure: {}'.format(', '.join(symbols).upper()))
+
+        self.set_legend(ax)
+        self.format_ax(ax)
+
+    def __iter__(self):
+        yield pd.Timestamp.utcnow(), SESSION_START
+
+        while True:
+            current_time = pd.Timestamp.utcnow()
+            current_minute = current_time.floor('1 min')
+
+            if self._last_emit is None or current_minute > self._last_emit:
+                log.debug('emitting minutely bar: {}'.format(current_minute))
+
+                self._last_emit = current_minute
+                yield current_minute, BAR
+
+                try:
+                    self.draw_pnl()
+                    self.draw_custom_signals()
+                    self.draw_exposure()
+
+                    plt.draw()
+                except Exception as e:
+                    log.warn('Unable to update the graph: {}'.format(e))
+
+            else:
+                # I can't use the "animate" reactive approach here because
+                # I need to yield from the main loop.
+
+                # Workaround: https://stackoverflow.com/a/33050617/814633
+                plt.pause(1)

catalyst/exchange/poloniex/poloniex.py

@@ -0,0 +1,633 @@
+import json
+import json
+import time
+from collections import defaultdict
+
+import numpy as np
+import pandas as pd
+import pytz
+from catalyst.assets._assets import TradingPair
+from logbook import Logger
+# import six
+from six import iteritems
+
+from catalyst.constants import LOG_LEVEL
+# from websocket import create_connection
+from catalyst.exchange.exchange import Exchange
+from catalyst.exchange.exchange_bundle import ExchangeBundle
+from catalyst.exchange.exchange_errors import (
+    ExchangeRequestError,
+    InvalidHistoryFrequencyError,
+    InvalidOrderStyle, OrphanOrderReverseError)
+from catalyst.exchange.exchange_execution import ExchangeLimitOrder, \
+    ExchangeStopLimitOrder
+from catalyst.exchange.exchange_utils import get_exchange_symbols_filename, \
+    download_exchange_symbols
+from catalyst.exchange.poloniex.poloniex_api import Poloniex_api
+from catalyst.finance.order import Order, ORDER_STATUS
+from catalyst.finance.transaction import Transaction
+from catalyst.protocol import Account
+
+log = Logger('Poloniex', level=LOG_LEVEL)
+
+
+class Poloniex(Exchange):
+    def __init__(self, key, secret, base_currency, portfolio=None):
+        self.api = Poloniex_api(key=key, secret=secret.encode('UTF-8'))
+        self.name = 'poloniex'
+        self.assets = {}
+        self.load_assets()
+        self.base_currency = base_currency
+        self._portfolio = portfolio
+        self.minute_writer = None
+        self.minute_reader = None
+        self.transactions = defaultdict(list)
+
+        self.num_candles_limit = 2000
+        self.max_requests_per_minute = 60
+        self.request_cpt = dict()
+
+        self.bundle = ExchangeBundle(self)
+
+    def sanitize_curency_symbol(self, exchange_symbol):
+        """
+        Helper method used to build the universal pair.
+        Include any symbol mapping here if appropriate.
+
+        :param exchange_symbol:
+        :return universal_symbol:
+        """
+        return exchange_symbol.lower()
+
+    def _create_order(self, order_status):
+        """
+        Create a Catalyst order object from the Exchange order dictionary
+        :param order_status:
+        :return: Order
+        """
+        # if order_status['is_cancelled']:
+        #    status = ORDER_STATUS.CANCELLED
+        # elif not order_status['is_live']:
+        #    log.info('found executed order {}'.format(order_status))
+        #    status = ORDER_STATUS.FILLED
+        # else:
+        status = ORDER_STATUS.OPEN
+
+        amount = float(order_status['amount'])
+        # filled = float(order_status['executed_amount'])
+        filled = None
+
+        if order_status['type'] == 'sell':
+            amount = -amount
+            # filled = -filled
+
+        price = float(order_status['rate'])
+        order_type = order_status['type']
+
+        stop_price = None
+        limit_price = None
+
+        # TODO: is this comprehensive enough?
+        # if order_type.endswith('limit'):
+        #    limit_price = price
+        # elif order_type.endswith('stop'):
+        #    stop_price = price
+
+        # executed_price = float(order_status['avg_execution_price'])
+        executed_price = price
+
+        # TODO: bitfinex does not specify comission. I could calculate it but not sure if it's worth it.
+        commission = None
+
+        # date = pd.Timestamp.utcfromtimestamp(float(order_status['timestamp']))
+        # date = pytz.utc.localize(date)
+        date = None
+
+        order = Order(
+            dt=date,
+            asset=self.assets[order_status['symbol']],
+            # No such field in Poloniex
+            amount=amount,
+            stop=stop_price,
+            limit=limit_price,
+            filled=filled,
+            id=str(order_status['orderNumber']),
+            commission=commission
+        )
+        order.status = status
+
+        return order, executed_price
+
+    def get_balances(self):
+        log.debug('retrieving wallets balances')
+        try:
+            balances = self.api.returnbalances()
+        except Exception as e:
+            log.debug(e)
+            raise ExchangeRequestError(error=e)
+
+        if 'error' in balances:
+            raise ExchangeRequestError(
+                error='unable to fetch balance {}'.format(balances['error'])
+            )
+
+        std_balances = dict()
+        for (key, value) in iteritems(balances):
+            currency = key.lower()
+            std_balances[currency] = float(value)
+
+        return std_balances
+
+    @property
+    def account(self):
+        account = Account()
+
+        account.settled_cash = None
+        account.accrued_interest = None
+        account.buying_power = None
+        account.equity_with_loan = None
+        account.total_positions_value = None
+        account.total_positions_exposure = None
+        account.regt_equity = None
+        account.regt_margin = None
+        account.initial_margin_requirement = None
+        account.maintenance_margin_requirement = None
+        account.available_funds = None
+        account.excess_liquidity = None
+        account.cushion = None
+        account.day_trades_remaining = None
+        account.leverage = None
+        account.net_leverage = None
+        account.net_liquidation = None
+
+        return account
+
+    @property
+    def time_skew(self):
+        # TODO: research the time skew conditions
+        return pd.Timedelta('0s')
+
+    def get_account(self):
+        # TODO: fetch account data and keep in cache
+        return None
+
+    def get_candles(self, data_frequency, assets, bar_count=None,
+                    start_dt=None, end_dt=None):
+        """
+        Retrieve OHLVC candles from Poloniex
+
+        :param data_frequency:
+        :param assets:
+        :param bar_count:
+        :return:
+
+        Available Frequencies
+        ---------------------
+        '5m', '15m', '30m', '2h', '4h', '1D'
+        """
+
+        # TODO: implement end_dt and start_dt filters
+
+        if (
+                        data_frequency == '5m' or data_frequency == 'minute'):  # TODO: Polo does not have '1m'
+            frequency = 300
+        elif (data_frequency == '15m'):
+            frequency = 900
+        elif (data_frequency == '30m'):
+            frequency = 1800
+        elif (data_frequency == '2h'):
+            frequency = 7200
+        elif (data_frequency == '4h'):
+            frequency = 14400
+        elif (data_frequency == '1D' or data_frequency == 'daily'):
+            frequency = 86400
+        else:
+            raise InvalidHistoryFrequencyError(
+                frequency=data_frequency
+            )
+
+        # Making sure that assets are iterable
+        asset_list = [assets] if isinstance(assets, TradingPair) else assets
+        ohlc_map = dict()
+
+        for asset in asset_list:
+
+            end = int(time.time())
+            if (bar_count is None):
+                start = end - 2 * frequency
+            else:
+                start = end - bar_count * frequency
+
+            try:
+                response = self.api.returnchartdata(self.get_symbol(asset),
+                                                    frequency, start, end)
+            except Exception as e:
+                raise ExchangeRequestError(error=e)
+
+            if 'error' in response:
+                raise ExchangeRequestError(
+                    error='Unable to retrieve candles: {}'.format(
+                        response.content)
+                )
+
+            def ohlc_from_candle(candle):
+                last_traded = pd.Timestamp.utcfromtimestamp(candle['date'])
+                last_traded = last_traded.replace(tzinfo=pytz.UTC)
+
+                ohlc = dict(
+                    open=np.float64(candle['open']),
+                    high=np.float64(candle['high']),
+                    low=np.float64(candle['low']),
+                    close=np.float64(candle['close']),
+                    volume=np.float64(candle['volume']),
+                    price=np.float64(candle['close']),
+                    last_traded=last_traded
+                )
+
+                return ohlc
+
+            if bar_count is None:
+                ohlc_map[asset] = ohlc_from_candle(response[0])
+            else:
+                ohlc_bars = []
+                for candle in response:
+                    ohlc = ohlc_from_candle(candle)
+                    ohlc_bars.append(ohlc)
+                ohlc_map[asset] = ohlc_bars
+
+        return ohlc_map[assets] \
+            if isinstance(assets, TradingPair) else ohlc_map
+
+    def create_order(self, asset, amount, is_buy, style):
+        """
+        Creating order on the exchange.
+
+        :param asset:
+        :param amount:
+        :param is_buy:
+        :param style:
+        :return:
+        """
+        exchange_symbol = self.get_symbol(asset)
+
+        if isinstance(style, ExchangeLimitOrder) or isinstance(style,
+                                                               ExchangeStopLimitOrder):
+            if isinstance(style, ExchangeStopLimitOrder):
+                log.warn('{} will ignore the stop price'.format(self.name))
+
+            price = style.get_limit_price(is_buy)
+
+            try:
+                if (is_buy):
+                    response = self.api.buy(exchange_symbol, amount, price)
+                else:
+                    response = self.api.sell(exchange_symbol, -amount, price)
+            except Exception as e:
+                raise ExchangeRequestError(error=e)
+
+            date = pd.Timestamp.utcnow()
+
+            if ('orderNumber' in response):
+                order_id = str(response['orderNumber'])
+                order = Order(
+                    dt=date,
+                    asset=asset,
+                    amount=amount,
+                    stop=style.get_stop_price(is_buy),
+                    limit=style.get_limit_price(is_buy),
+                    id=order_id
+                )
+                return order
+            else:
+                log.warn(
+                    '{} order failed: {}'.format('buy' if is_buy else 'sell',
+                                                 response['error']))
+                return None
+        else:
+            raise InvalidOrderStyle(exchange=self.name,
+                                    style=style.__class__.__name__)
+
+    def get_open_orders(self, asset='all'):
+        """Retrieve all of the current open orders.
+
+        Parameters
+        ----------
+        asset : Asset
+            If passed and not 'all', return only the open orders for the given
+            asset instead of all open orders.
+
+        Returns
+        -------
+        open_orders : dict[list[Order]] or list[Order]
+            If 'all' is passed this will return a dict mapping Assets
+            to a list containing all the open orders for the asset.
+            If an asset is passed then this will return a list of the open
+            orders for this asset.
+        """
+
+        return self.portfolio.open_orders
+
+        """
+            TODO: Why going to the exchange if we already have this info locally?
+                  And why creating all these Orders if we later discard them?
+        """
+
+        try:
+            if (asset == 'all'):
+                response = self.api.returnopenorders('all')
+            else:
+                response = self.api.returnopenorders(self.get_symbol(asset))
+        except Exception as e:
+            raise ExchangeRequestError(error=e)
+
+        if 'error' in response:
+            raise ExchangeRequestError(
+                error='Unable to retrieve open orders: {}'.format(
+                    order_statuses['message'])
+            )
+
+        print(self.portfolio.open_orders)
+
+        # TODO: Need to handle openOrders for 'all'
+        orders = list()
+        for order_status in response:
+            order, executed_price = self._create_order(
+                order_status)  # will Throw error b/c Polo doesn't track order['symbol']
+            if asset is None or asset == order.sid:
+                orders.append(order)
+
+        return orders
+
+    def get_order(self, order_id):
+        """Lookup an order based on the order id returned from one of the
+        order functions.
+
+        Parameters
+        ----------
+        order_id : str
+            The unique identifier for the order.
+
+        Returns
+        -------
+        order : Order
+            The order object.
+        """
+
+        try:
+            order = self._portfolio.open_orders[order_id]
+        except Exception as e:
+            raise OrphanOrderError(order_id=order_id, exchange=self.name)
+
+        return order
+
+        # TODO: Need to decide whether we fetch orders locally or from exchnage
+        # The code below is ignored
+
+        try:
+            response = self.api.returnopenorders(self.get_symbol(order.sid))
+        except Exception as e:
+            raise ExchangeRequestError(error=e)
+
+        for o in response:
+            if (int(o['orderNumber']) == int(order_id)):
+                return order
+
+        return None
+
+    def cancel_order(self, order_param):
+        """Cancel an open order.
+
+        Parameters
+        ----------
+        order_param : str or Order
+            The order_id or order object to cancel.
+        """
+
+        if (isinstance(order_param, Order)):
+            order = order_param
+        else:
+            order = self._portfolio.open_orders[order_param]
+
+        try:
+            response = self.api.cancelorder(order.id)
+        except Exception as e:
+            raise ExchangeRequestError(error=e)
+
+        if 'error' in response:
+            log.info(
+                'Unable to cancel order {order_id} on exchange {exchange} {error}.'.format(
+                    order_id=order.id,
+                    exchange=self.name,
+                    error=response['error']
+                ))
+
+            # raise OrderCancelError(
+            #    order_id=order.id,
+            #    exchange=self.name,
+            #    error=response['error']
+            # )
+
+        self.portfolio.remove_order(order)
+
+    def tickers(self, assets):
+        """
+        Fetch ticket data for assets
+        https://docs.bitfinex.com/v2/reference#rest-public-tickers
+
+        :param assets:
+        :return:
+        """
+        symbols = self.get_symbols(assets)
+
+        log.debug('fetching tickers {}'.format(symbols))
+
+        try:
+            response = self.api.returnticker()
+        except Exception as e:
+            raise ExchangeRequestError(error=e)
+
+        if 'error' in response:
+            raise ExchangeRequestError(
+                error='Unable to retrieve tickers: {}'.format(
+                    response['error'])
+            )
+
+        ticks = dict()
+
+        for index, symbol in enumerate(symbols):
+            ticks[assets[index]] = dict(
+                timestamp=pd.Timestamp.utcnow(),
+                bid=float(response[symbol]['highestBid']),
+                ask=float(response[symbol]['lowestAsk']),
+                last_price=float(response[symbol]['last']),
+                low=float(response[symbol]['lowestAsk']),
+                # TODO: Polo does not provide low
+                high=float(response[symbol]['highestBid']),
+                # TODO: Polo does not provide high
+                volume=float(response[symbol]['baseVolume']),
+            )
+
+        log.debug('got tickers {}'.format(ticks))
+        return ticks
+
+    def generate_symbols_json(self, filename=None, source_dates=False):
+        symbol_map = {}
+
+        if not source_dates:
+            fn, r = download_exchange_symbols(self.name)
+            with open(fn) as data_file:
+                cached_symbols = json.load(data_file)
+
+        response = self.api.returnticker()
+
+        for exchange_symbol in response:
+            base, market = self.sanitize_curency_symbol(exchange_symbol).split(
+                '_')
+            symbol = '{market}_{base}'.format(market=market, base=base)
+
+            if (source_dates):
+                start_date = self.get_symbol_start_date(exchange_symbol)
+            else:
+                try:
+                    start_date = cached_symbols[exchange_symbol]['start_date']
+                except KeyError as e:
+                    start_date = time.strftime('%Y-%m-%d')
+
+            try:
+                end_daily = cached_symbols[exchange_symbol]['end_daily']
+            except KeyError as e:
+                end_daily = 'N/A'
+
+            try:
+                end_minute = cached_symbols[exchange_symbol]['end_minute']
+            except KeyError as e:
+                end_minute = 'N/A'
+
+            symbol_map[exchange_symbol] = dict(
+                symbol=symbol,
+                start_date=start_date,
+                end_daily=end_daily,
+                end_minute=end_minute,
+            )
+
+        if (filename is None):
+            filename = get_exchange_symbols_filename(self.name)
+
+        with open(filename, 'w') as f:
+            json.dump(symbol_map, f, sort_keys=True, indent=2,
+                      separators=(',', ':'))
+
+    def get_symbol_start_date(self, symbol):
+        try:
+            r = self.api.returnchartdata(symbol, 86400, pd.to_datetime(
+                '2010-1-1').value // 10 ** 9)
+        except Exception as e:
+            raise ExchangeRequestError(error=e)
+
+        return time.strftime('%Y-%m-%d', time.gmtime(int(r[0]['date'])))
+
+    def check_open_orders(self):
+        """
+        Need to override this function for Poloniex:
+
+        Loop through the list of open orders in the Portfolio object.
+        Check if any transactions have been executed:
+            If so, create a transaction and apply to the Portfolio.
+        Check if the order is still open:
+            If not, remove it from open orders
+
+        :return:
+        transactions: Transaction[]
+        """
+        transactions = list()
+        if self.portfolio.open_orders:
+            for order_id in list(self.portfolio.open_orders):
+
+                order = self._portfolio.open_orders[order_id]
+                log.debug('found open order: {}'.format(order_id))
+
+                try:
+                    order_open = self.get_order(order_id)
+                except Exception as e:
+                    raise ExchangeRequestError(error=e)
+
+                if (order_open):
+                    delta = pd.Timestamp.utcnow() - order.dt
+                    log.info(
+                        'order {order_id} still open after {delta}'.format(
+                            order_id=order_id,
+                            delta=delta)
+                    )
+
+                try:
+                    response = self.api.returnordertrades(order_id)
+                except Exception as e:
+                    raise ExchangeRequestError(error=e)
+
+                if ('error' in response):
+                    if (not order_open):
+                        raise OrphanOrderReverseError(order_id=order_id,
+                                                      exchange=self.name)
+                else:
+                    for tx in response:
+                        """
+                            We maintain a list of dictionaries of transactions that correspond to
+                            partially filled orders, indexed by order_id. Every time we query 
+                            executed transactions from the exchange, we check if we had that 
+                            transaction for that order already. If not, we process it.
+
+                            When an order if fully filled, we flush the dict of transactions  
+                            associated with that order.
+                        """
+                        if (not filter(
+                                lambda item: item['order_id'] == tx['tradeID'],
+                                self.transactions[order_id])):
+                            log.debug(
+                                'Got new transaction for order {}: amount {}, price {}'.format(
+                                    order_id, tx['amount'], tx['rate']))
+                            tx['amount'] = float(tx['amount'])
+                            if (tx['type'] == 'sell'):
+                                tx['amount'] = -tx['amount']
+                            transaction = Transaction(
+                                asset=order.asset,
+                                amount=tx['amount'],
+                                dt=pd.to_datetime(tx['date'], utc=True),
+                                price=float(tx['rate']),
+                                order_id=tx['tradeID'],
+                                # it's a misnomer, but keeping it for compatibility
+                                commission=float(tx['fee'])
+                            )
+                            self.transactions[order_id].append(transaction)
+                            self.portfolio.execute_transaction(transaction)
+                            transactions.append(transaction)
+
+                    if (not order_open):
+                        """
+                            Since transactions have been executed individually
+                            the only thing left to do is remove them from list of open_orders
+                        """
+                        del self.portfolio.open_orders[order_id]
+                        del self.transactions[order_id]
+
+        return transactions
+
+    def get_orderbook(self, asset, order_type='all'):
+        exchange_symbol = asset.exchange_symbol
+        data = self.api.returnOrderBook(market=exchange_symbol)
+
+        result = dict()
+        for order_type in data:
+            # TODO: filter by type
+            if order_type != 'asks' and order_type != 'bids':
+                continue
+
+            result[order_type] = []
+            for entry in data[order_type]:
+                if len(entry) == 2:
+                    result[order_type].append(
+                        dict(
+                            rate=float(entry[0]),
+                            quantity=float(entry[1])
+                        )
+                    )
+        return result

catalyst/exchange/poloniex/poloniex_api.py

@@ -0,0 +1,183 @@
+#!/usr/bin/env python
+import json
+import time
+import hmac
+import hashlib
+
+from six.moves import urllib
+
+# Workaround for backwards compatibility
+# https://stackoverflow.com/questions/3745771/urllib-request-in-python-2-7
+urlopen = urllib.request.urlopen
+
+
+class Poloniex_api(object):
+    def __init__(self, key, secret):
+        self.key = key
+        self.secret = secret
+
+        self.max_requests_per_second = 6
+        self.request_cpt = dict()
+
+        self.public  = ['returnTicker', 'return24Volume', 'returnOrderBook',
+                        'returnTradeHistory', 'returnChartData',
+                        'returnCurrencies', 'returnLoanOrders']
+        self.trading = ['returnBalances','returnCompleteBalances','returnDepositAddresses',
+                        'generateNewAddress','returnDepositsWithdrawals','returnOpenOrders',
+                        'returnTradeHistory','returnOrderTrades',
+                        'buy', 'sell', 'cancelOrder', 'moveOrder',
+                        'withdraw', 'returnFeeInfo','returnAvailableAccountBalances',
+                        'returnTradableBalances', 'transferBalance',
+                        'returnMarginAccountSummary','marginBuy','marginSell',
+                        'getMarginPosition', 'closeMarginPosition','createLoanOffer',
+                        'cancelLoanOffer','returnOpenLoanOffers','returnActiveLoans',
+                        'returnLendingHistory','toggleAutoRenew']
+
+    def ask_request(self):
+        """
+        Asks permission to issue a request to the exchange.
+        The primary purpose is to avoid hitting rate limits.
+
+        The application will pause if the maximum requests per minute
+        permitted by the exchange is exceeded.
+
+        :return boolean:
+
+        """
+        now = time.time()
+        if not self.request_cpt:
+            self.request_cpt = dict()
+            self.request_cpt[now] = 0
+            return True
+
+        cpt_date = self.request_cpt.keys()[0]
+        cpt = self.request_cpt[cpt_date]
+
+        if now > cpt_date + 1:
+            self.request_cpt = dict()
+            self.request_cpt[now] = 0
+            return True
+
+        if cpt >= self.max_requests_per_second:
+            
+            log.debug('max requests 6 reached, sleeping for 1 seconds')
+            sleep(1)
+
+            now = time.time()
+            self.request_cpt = dict()
+            self.request_cpt[now] = 0
+            return True
+        else:
+            self.request_cpt[cpt_date] += 1
+
+    def query(self, method, req={}):
+
+        if method in self.public:
+            url = 'https://poloniex.com/public?command=' + method + '&' + urllib.parse.urlencode(req)
+            headers = {}
+            post_data = None
+        elif method in self.trading:
+            url = 'https://poloniex.com/tradingApi'
+            req['command'] = method
+            req['nonce']   = int(time.time()*1000)
+            post_data      = urllib.parse.urlencode(req)
+            signature      = hmac.new(self.secret, post_data, hashlib.sha512).hexdigest()
+            headers        = { 'Sign': signature, 'Key': self.key}
+        else:
+            raise ValueError('Method "' + method + '" not found in neither the Public API or Trading API endpoints')
+
+        self.ask_request()
+        req = urllib.request.Request(url, data=post_data, headers=headers)
+        return json.loads(urlopen(req).read())
+
+    def returnticker(self):
+        return self.query('returnTicker', {})
+
+    def return24volume(self):
+        return self.query('return24Volume', {})
+
+    def returnOrderBook(self, market='all'):
+        return self.query('returnOrderBook', {'currencyPair': market})
+
+    def returntradehistory(self, market, start=None, end=None):
+        if(start is not None and end is not None):
+            return self.query('returntradehistory', 
+                              {'currencyPair': market, 'start': start, 'end': end })
+        else:
+            return self.query('returntradehistory', {'currencyPair': market })
+
+    def returnchartdata(self, market, period, start, end=9999999999):
+        return self.query('returnChartData', {'currencyPair': market, 'period': period,
+                          'start': start, 'end': end})
+
+    def returncurrencies(self):
+        return self.query('returnCurrencies', {})
+
+    def returnloadorders(self, market):
+        return self.query('returnLoanOrders', {'currency': market})
+
+    def returnbalances(self):
+        return self.query('returnBalances')
+
+    def returncompletebalances(self, account):
+        if(account):
+            return self.query('returnCompleteBalances', {'account': account})
+        else:
+            return self.query('returnCompleteBalances')
+
+    def returndepositaddresses(self):
+        return self.query('returnDepositAddresses')
+
+    def generatenewaddress(self, currency):
+        return self.query('generateNewAddress', {'currency': currency})
+
+    def returnDepositsWithdrawals(self, start, end):
+        return self.query('returnDepositsWithdrawals', {'start': start, 'end': end})
+
+    def returnopenorders(self, market):
+        return self.query('returnOpenOrders', {'currencyPair': market})
+
+    def returntradehistory(self, market):
+        #TODO: optional start and/or end and limit
+        return self.query('returnTradeHistory', {'currencyPair': market})
+
+    def returnordertrades(self, ordernumber):
+        return self.query('returnOrderTrades', {'orderNumber': ordernumber})
+
+    def buy(self, market, amount, rate, fillorkill=0, immediateorcancel=0, postonly=0):
+        if(fillorkill):
+            return self.query('buy', {'currencyPair': market, 'rate':rate, 'amount': amount,
+                                      'fillOrKill': fillorkill, })
+        elif(immediateorcancel):
+            return self.query('buy', {'currencyPair': market, 'rate':rate, 'amount': amount,
+                                      'immediateOrCancel': immediateorcancel, })
+        elif(postonly):
+            return self.query('buy', {'currencyPair': market, 'rate':rate, 'amount': amount,
+                                      'postOnly': postonly, })
+        else:
+            return self.query('buy', {'currencyPair': market, 'rate':rate, 'amount': amount, })
+
+    def sell(self, market, amount, rate, fillorkill=0, immediateorcancel=0, postonly=0):
+        if(fillorkill):
+            return self.query('sell', {'currencyPair': market, 'rate':rate, 'amount': amount,
+                                      'fillOrKill': fillorkill, })
+        elif(immediateorcancel):
+            return self.query('sell', {'currencyPair': market, 'rate':rate, 'amount': amount,
+                                      'immediateOrCancel': immediateorcancel, })
+        elif(postonly):
+            return self.query('sell', {'currencyPair': market, 'rate':rate, 'amount': amount,
+                                      'postOnly': postonly, })
+        else:
+            return self.query('sell', {'currencyPair': market, 'rate':rate, 'amount': amount, })
+
+    def cancelorder(self, ordernumber):
+        return self.query('cancelOrder', {'orderNumber': ordernumber})
+
+    def withdraw(self, currency, quantity, address):
+        return self.query('withdraw',
+                          {'currency': currency, 'amount': quantity,
+                           'address': address})
+
+    def returnfeeinfo(self):
+        return self.query('returnFeeInfo')
+

catalyst/exchange/simple_clock.py

@@ -0,0 +1,60 @@
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from time import sleep
+
+import pandas as pd
+from catalyst.gens.sim_engine import (
+    BAR,
+    SESSION_START
+)
+from logbook import Logger
+
+from catalyst.constants import LOG_LEVEL
+
+log = Logger('ExchangeClock', level=LOG_LEVEL)
+
+
+class SimpleClock(object):
+    """Realtime clock for live trading.
+
+    This class is a drop-in replacement for
+    :class:`zipline.gens.sim_engine.MinuteSimulationClock`.
+
+    This is a stripped down version because crypto exchanges run around the clock.
+
+    The :param:`time_skew` parameter represents the time difference between
+    the Broker and the live trading machine's clock.
+    """
+
+    def __init__(self, sessions, time_skew=pd.Timedelta("0s")):
+
+        self.sessions = sessions
+        self.time_skew = time_skew
+        self._last_emit = None
+        self._before_trading_start_bar_yielded = True
+
+    def __iter__(self):
+        yield pd.Timestamp.utcnow(), SESSION_START
+
+        while True:
+            current_time = pd.Timestamp.utcnow()
+            current_minute = current_time.floor('1 min')
+
+            if self._last_emit is None or current_minute > self._last_emit:
+                log.debug('emitting minutely bar: {}'.format(current_minute))
+
+                self._last_emit = current_minute
+                yield current_minute, BAR
+            else:
+                sleep(1)

catalyst/exchange/stats_utils.py

@@ -0,0 +1,51 @@
+import pandas as pd
+
+
+def get_pretty_stats(stats_df, recorded_cols=None, num_rows=10):
+    """
+    Format and print the last few rows of a statistics DataFrame.
+    See the pyfolio project for the data structure.
+
+    :param stats_df:
+    :param num_rows:
+    :return:
+    """
+    stats_df.set_index('period_close', drop=True, inplace=True)
+    stats_df.dropna(axis=1, how='all', inplace=True)
+
+    pd.set_option('display.expand_frame_repr', False)
+    pd.set_option('precision', 3)
+    pd.set_option('display.width', 1000)
+    pd.set_option('display.max_colwidth', 1000)
+
+    columns = ['starting_cash', 'ending_cash', 'portfolio_value',
+               'pnl', 'long_exposure', 'short_exposure', 'orders',
+               'transactions', 'positions']
+
+    if recorded_cols is not None:
+        for column in recorded_cols:
+            columns.append(column)
+
+    def format_positions(positions):
+        parts = []
+        for position in positions:
+            msg = '{amount:.2f}{market} cost basis {cost_basis:.4f}{base}'.format(
+                amount=position['amount'],
+                market=position['sid'].market_currency,
+                cost_basis=position['cost_basis'],
+                base=position['sid'].base_currency
+            )
+            parts.append(msg)
+        return ', '.join(parts)
+
+    formatters = {
+        'orders': lambda orders: len(orders),
+        'transactions': lambda transactions: len(transactions),
+        'returns': lambda returns: "{0:.4f}".format(returns),
+        'positions': format_positions
+    }
+
+    return stats_df.tail(num_rows).to_string(
+        columns=columns,
+        formatters=formatters
+    )

catalyst/finance/blotter.py

@@ -34,7 +34,9 @@ from catalyst.finance.commission import (
 from catalyst.finance.cancel_policy import NeverCancel
 from catalyst.utils.input_validation import expect_types
 
-log = Logger('Blotter')
+from catalyst.constants import LOG_LEVEL
+
+log = Logger('Blotter', level=LOG_LEVEL)
 warning_logger = Logger('AlgoWarning')
 
 

catalyst/finance/controls.py

@@ -24,7 +24,9 @@ from catalyst.errors import (
     TradingControlViolation,
 )
 
-log = logbook.Logger('TradingControl')
+from catalyst.constants import LOG_LEVEL
+
+log = logbook.Logger('TradingControl', level=LOG_LEVEL)
 
 
 class TradingControl(with_metaclass(abc.ABCMeta)):

catalyst/finance/performance/period.py

@@ -88,7 +88,10 @@ from six import itervalues, iteritems
 
 import catalyst.protocol as zp
 
-log = logbook.Logger('Performance')
+from catalyst.constants import LOG_LEVEL
+
+log = logbook.Logger('Performance', level=LOG_LEVEL)
+
 TRADE_TYPE = zp.DATASOURCE_TYPE.TRADE
 
 

catalyst/finance/performance/position.py

@@ -40,7 +40,9 @@ import logbook
 from catalyst.assets import Future, Asset
 from catalyst.utils.input_validation import expect_types
 
-log = logbook.Logger('Performance')
+from catalyst.constants import LOG_LEVEL
+
+log = logbook.Logger('Performance', level=LOG_LEVEL)
 
 
 class Position(object):

catalyst/finance/performance/position_tracker.py

@@ -32,7 +32,9 @@ from catalyst.assets import (
 )
 from . position import positiondict
 
-log = logbook.Logger('Performance')
+from catalyst.constants import LOG_LEVEL
+
+log = logbook.Logger('Performance', level=LOG_LEVEL)
 
 
 PositionStats = namedtuple('PositionStats',

catalyst/finance/performance/tracker.py

@@ -70,7 +70,9 @@ import catalyst.finance.risk as risk
 
 from . position_tracker import PositionTracker
 
-log = logbook.Logger('Performance')
+from catalyst.constants import LOG_LEVEL
+
+log = logbook.Logger('Performance', level=LOG_LEVEL)
 
 
 class PerformanceTracker(object):
@@ -116,7 +118,6 @@ class PerformanceTracker(object):
                 self.sim_params.first_open, self.sim_params.last_close,
                 freq='Min')
             )
-
             self.cumulative_risk_metrics = \
                 risk.RiskMetricsCumulative(
                     self.sim_params,

catalyst/finance/risk/cumulative.py

@@ -38,7 +38,9 @@ from empyrical import (
     sortino_ratio,
 )
 
-log = logbook.Logger('Risk Cumulative')
+from catalyst.constants import LOG_LEVEL
+
+log = logbook.Logger('Risk Cumulative', level=LOG_LEVEL)
 
 
 choose_treasury = functools.partial(choose_treasury, lambda *args: '10year',

catalyst/finance/risk/period.py

@@ -36,7 +36,9 @@ from empyrical import (
     sortino_ratio
 )
 
-log = logbook.Logger('Risk Period')
+from catalyst.constants import LOG_LEVEL
+
+log = logbook.Logger('Risk Period', level=LOG_LEVEL)
 
 choose_treasury = functools.partial(risk.choose_treasury,
                                     risk.select_treasury_duration)

catalyst/finance/risk/report.py

@@ -63,7 +63,9 @@ from dateutil.relativedelta import relativedelta
 
 from . period import RiskMetricsPeriod
 
-log = logbook.Logger('Risk Report')
+from catalyst.constants import LOG_LEVEL
+
+log = logbook.Logger('Risk Report', level=LOG_LEVEL)
 
 
 class RiskReport(object):

catalyst/finance/risk/risk.py

@@ -61,7 +61,9 @@ Risk Report
 import logbook
 import numpy as np
 
-log = logbook.Logger('Risk')
+from catalyst.constants import LOG_LEVEL
+
+log = logbook.Logger('Risk', level=LOG_LEVEL)
 
 
 TREASURY_DURATIONS = [
@@ -158,7 +160,7 @@ def choose_treasury(select_treasury, treasury_curves, start_session,
                 )
                 break
 
-        if search_day:
+        if search_day and trading_calendar.name != 'OPEN':          # Supress warning for 'OPEN' calendar
             if (search_dist is None or search_dist > 1) and \
                     search_days[0] <= end_session <= search_days[-1]:
                 message = "No rate within 1 trading day of end date = \

catalyst/finance/slippage.py

@@ -41,6 +41,7 @@ DEFAULT_EQUITY_VOLUME_SLIPPAGE_BAR_LIMIT = 0.025
 DEFAULT_FUTURE_VOLUME_SLIPPAGE_BAR_LIMIT = 0.05
 
 
+
 class LiquidityExceeded(Exception):
     pass
 
@@ -205,20 +206,22 @@ class VolumeShareSlippage(SlippageModel):
     def process_order(self, data, order):
         volume = data.current(order.asset, "volume")
 
+        min_trade_size = order.asset.min_trade_size
+
         max_volume = self.volume_limit * volume
 
         # price impact accounts for the total volume of transactions
         # created against the current minute bar
         remaining_volume = max_volume - self.volume_for_bar
-        if remaining_volume < 1:
+        if remaining_volume < min_trade_size:
             # we can't fill any more transactions
             raise LiquidityExceeded()
 
         # the current order amount will be the min of the
         # volume available in the bar or the open amount.
-        cur_volume = int(min(remaining_volume, abs(order.open_amount)))
+        cur_volume = min(remaining_volume, abs(order.open_amount))
 
-        if cur_volume < 1:
+        if cur_volume < min_trade_size:
             return None, None
 
         # tally the current amount into our total amount ordered.

catalyst/finance/trading.py

@@ -26,7 +26,9 @@ from catalyst.data.loader import load_market_data
 from catalyst.utils.calendars import get_calendar
 from catalyst.utils.memoize import remember_last
 
-log = logbook.Logger('Trading')
+from catalyst.constants import LOG_LEVEL
+
+log = logbook.Logger('Trading', level=LOG_LEVEL)
 
 
 DEFAULT_CAPITAL_BASE = 1e5

catalyst/finance/transaction.py

@@ -65,14 +65,10 @@ def create_transaction(order, dt, price, amount):
     # floor the amount to protect against non-whole number orders
     # TODO: Investigate whether we can add a robust check in blotter
     # and/or tradesimulation, as well.
-    amount_magnitude = int(abs(amount))
-
-    if amount_magnitude < 1:
-        raise Exception("Transaction magnitude must be at least 1.")
 
     transaction = Transaction(
         asset=order.asset,
-        amount=int(amount),
+        amount=amount,
         dt=dt,
         price=price,
         order_id=order.id

catalyst/gens/tradesimulation.py

@@ -27,7 +27,9 @@ from catalyst.gens.sim_engine import (
     BEFORE_TRADING_START_BAR
 )
 
-log = Logger('Trade Simulation')
+from catalyst.constants import LOG_LEVEL
+
+log = Logger('Trade Simulation', level=LOG_LEVEL)
 
 
 class AlgorithmSimulator(object):

catalyst/pipeline/loaders/crypto_pricing_loader.py

@@ -33,13 +33,26 @@ class CryptoPricingLoader(PipelineLoader):
     Delegates loading of baselines and adjustments.
     """
 
-    def __init__(self, raw_price_loader, dataset):
-        self.raw_price_loader = raw_price_loader
-        self._columns = dataset.columns
+    def __init__(self, bundle, data_frequency, dataset):
 
         cal = get_calendar('OPEN')
 
-        self._all_sessions = cal.all_sessions
+        if data_frequency == 'daily':
+            reader = bundle.daily_bar_reader
+            all_sessions = cal.all_sessions
+
+        elif data_frequency == 'minute':
+            reader = bundle.minute_bar_reader
+            all_sessions = cal.all_minutes
+
+        else:
+            raise ValueError(
+                'Invalid data frequency: {}'.format(data_frequency)
+            )
+
+        self.raw_price_loader = reader
+        self._columns = dataset.columns
+        self._all_sessions = all_sessions
 
     @classmethod
     def from_files(cls, pricing_path):
@@ -89,6 +102,7 @@ class CryptoPricingLoader(PipelineLoader):
 
 
 def _shift_dates(dates, start_date, end_date, shift):
+
     try:
         start = dates.get_loc(start_date)
     except KeyError:

catalyst/pipeline/loaders/equity_pricing_loader.py

@@ -36,15 +36,29 @@ class USEquityPricingLoader(PipelineLoader):
     Delegates loading of baselines and adjustments.
     """
 
-    def __init__(self, raw_price_loader, adjustments_loader, dataset):
-        self.raw_price_loader = raw_price_loader
-        self.adjustments_loader = adjustments_loader
+    def __init__(self, bundle, data_frequency, dataset):
+
+        if data_frequency == 'daily':
+            reader = bundle.daily_bar_reader
+        elif daily_bar_reader == 'minute':
+            reader = bundle.minute_bar_reader
+        else:
+            raise ValueError(
+                'Invalid data frequency: {}'.format(data_frequency)
+            )
+
+        cal = reader.trading_calendar or get_calendar('NYSE')
+
+        if data_frequency == 'daily':
+            all_sessions = cal.all_sessions
+        elif daily_bar_reader == 'minute':
+            reader = bundle.minute_bar_reader
+            all_sessions = cal.all_minutes
+
+        self.raw_price_loader = reader
+        self.adjustments_loader = bundle.adjustments_loader
         self._columns = dataset.columns
-
-        cal = self.raw_price_loader.trading_calendar or \
-            get_calendar("NYSE")
-
-        self._all_sessions = cal.all_sessions
+        self._all_sessions = all_sessions
 
     @classmethod
     def from_files(cls, pricing_path, adjustments_path):

catalyst/sources/benchmark_source.py

@@ -72,7 +72,13 @@ class BenchmarkSource(object):
                             "benchmark_returns.")
 
     def get_value(self, dt):
-        return self._precalculated_series.loc[dt]
+        try:
+            series = self._precalculated_series
+            value = series.loc[dt]
+            return value
+        except Exception:
+            # TODO: workaround, find permanent fix
+            return 0
 
     def get_range(self, start_dt, end_dt):
         return self._precalculated_series.loc[start_dt:end_dt]

catalyst/sources/requests_csv.py

@@ -23,7 +23,9 @@ from catalyst.protocol import (
 )
 from catalyst.assets import Equity
 
-logger = Logger('Requests Source Logger')
+from catalyst.constants import LOG_LEVEL
+
+logger = Logger('Requests Source Logger', level=LOG_LEVEL)
 
 
 def roll_dts_to_midnight(dts, trading_day):

catalyst/utils/calendars/calendar_utils.py

@@ -25,7 +25,7 @@ _default_calendar_factories = {
     'us_futures': QuantopianUSFuturesCalendar,
 }
 _default_calendar_aliases = {
-    'CATX': 'OPEN',
+    'POLO': 'OPEN',
     'NASDAQ': 'NYSE',
     'BATS': 'NYSE',
     'CBOT': 'CME',

catalyst/utils/calendars/exchange_calendar_open.py

@@ -1,6 +1,7 @@
 from datetime import time
 from pytz import timezone
 
+from pandas import Timestamp
 from pandas.tseries.offsets import DateOffset
 
 from catalyst.utils.memoize import lazyval
@@ -28,3 +29,6 @@ class OpenExchangeCalendar(TradingCalendar):
     @lazyval
     def day(self):
         return DateOffset(days=1)
+
+    def __init__(self, *args, **kwargs):
+        super(OpenExchangeCalendar, self).__init__(start=Timestamp('2015-3-1', tz='UTC'), **kwargs)

catalyst/utils/calendars/trading_calendar.py

@@ -690,8 +690,7 @@ class TradingCalendar(with_metaclass(ABCMeta)):
     def execution_time_from_close(self, close_dates):
         return close_dates
 
-    @lazyval
-    def all_minutes(self):
+    def _all_minutes_with_interval(self, interval):
         """
         Returns a DatetimeIndex representing all the minutes in this calendar.
         """
@@ -703,8 +702,10 @@ class TradingCalendar(with_metaclass(ABCMeta)):
 
         deltas = closes_in_ns - opens_in_ns
 
+        nanos_in_interval = interval * NANOS_IN_MINUTE
+
         # + 1 because we want 390 days per standard day, not 389
-        daily_sizes = (deltas / NANOS_IN_MINUTE) + 1
+        daily_sizes = (deltas / nanos_in_interval) + 1
         num_minutes = np.sum(daily_sizes).astype(np.int64)
 
         # One allocation for the entire thing. This assumes that each day
@@ -721,13 +722,20 @@ class TradingCalendar(with_metaclass(ABCMeta)):
                 np.arange(
                     opens_in_ns[day_idx],
                     closes_in_ns[day_idx] + NANOS_IN_MINUTE,
-                    NANOS_IN_MINUTE
+                    nanos_in_interval
                 )
 
             idx += size_int
 
         return DatetimeIndex(all_minutes).tz_localize("UTC")
 
+    @lazyval
+    def all_minutes(self):
+        """
+        Returns a DatetimeIndex representing all the minutes in this calendar.
+        """
+        return self._all_minutes_with_interval(1)
+
     @preprocess(dt=coerce(pd.Timestamp, attrgetter('value')))
     def minute_to_session_label(self, dt, direction="next"):
         """

catalyst/utils/cli.py

@@ -1,10 +1,34 @@
+from itertools import count
+
 import click
 import pandas as pd
 
 from .context_tricks import CallbackManager
 
+DEFAULT_BAR_TEMPLATE = '    [%(bar)s]  %(label)s:  %(info)s'
+DEFAULT_EMPTY_CHAR = ' '
+DEFAULT_FILL_CHAR = '='
 
-def maybe_show_progress(it, show_progress, **kwargs):
+def item_show_count(total=None):
+    def maybe_show_total(index):
+        if total is not None:
+            return '{0}/{1}'.format(index, total)
+        return str(index)
+
+    def item_show_func(item, _it=iter(count())):
+        if item is not None:
+            starting = False
+            return maybe_show_total(next(_it))
+        return 'DONE'
+
+    return item_show_func
+
+def maybe_show_progress(it,
+                        show_progress,
+                        empty_char=DEFAULT_EMPTY_CHAR,
+                        fill_char=DEFAULT_FILL_CHAR,
+                        bar_template=DEFAULT_BAR_TEMPLATE,
+                        **kwargs):
     """Optionally show a progress bar for the given iterator.
 
     Parameters
@@ -30,6 +54,9 @@ def maybe_show_progress(it, show_progress, **kwargs):
                 ...
     """
     if show_progress:
+        kwargs['bar_template'] = bar_template
+        kwargs['empty_char'] = empty_char
+        kwargs['fill_char'] = fill_char
         return click.progressbar(it, **kwargs)
 
     # context manager that just return `it` when we enter it

catalyst/utils/math_utils.py

@@ -17,6 +17,8 @@ import math
 
 from numpy import isnan
 
+def round_nearest(x, a):
+    return round(round(x / a) * a, -int(math.floor(math.log10(a))))
 
 def tolerant_equals(a, b, atol=10e-7, rtol=10e-7, equal_nan=False):
     """Check if a and b are equal with some tolerance.

catalyst/utils/run_algo.py

@@ -1,34 +1,52 @@
 import os
-import re
-from runpy import run_path
 import sys
 import warnings
+from datetime import timedelta
+from runpy import run_path
+from time import sleep
 
 import click
+import pandas as pd
+
+from catalyst.exchange.bittrex.bittrex import Bittrex
+from catalyst.exchange.bitfinex.bitfinex import Bitfinex
+from catalyst.exchange.poloniex.poloniex import Poloniex
+
 try:
     from pygments import highlight
     from pygments.lexers import PythonLexer
     from pygments.formatters import TerminalFormatter
+
     PYGMENTS = True
 except:
     PYGMENTS = False
 from toolz import valfilter, concatv
 from functools import partial
 
-from catalyst.algorithm import TradingAlgorithm
-from catalyst.data.bundles.core import load
-from catalyst.data.data_portal import DataPortal
-from catalyst.data.loader import load_crypto_market_data
 from catalyst.finance.trading import TradingEnvironment
-from catalyst.pipeline.data import USEquityPricing, CryptoPricing
-from catalyst.pipeline.loaders import (
-    USEquityPricingLoader,
-    CryptoPricingLoader,
-)
 from catalyst.utils.calendars import get_calendar
 from catalyst.utils.factory import create_simulation_parameters
+from catalyst.data.loader import load_crypto_market_data
 import catalyst.utils.paths as pth
 
+from catalyst.exchange.exchange_algorithm import ExchangeTradingAlgorithmLive, \
+    ExchangeTradingAlgorithmBacktest
+from catalyst.exchange.data_portal_exchange import DataPortalExchangeLive, \
+    DataPortalExchangeBacktest
+from catalyst.exchange.asset_finder_exchange import AssetFinderExchange
+from catalyst.exchange.exchange_portfolio import ExchangePortfolio
+from catalyst.exchange.exchange_errors import (
+    ExchangeRequestError, ExchangeAuthEmpty,
+    ExchangeRequestErrorTooManyAttempts,
+    BaseCurrencyNotFoundError, ExchangeNotFoundError)
+from catalyst.exchange.exchange_utils import get_exchange_auth, \
+    get_algo_object, get_exchange_folder
+from logbook import Logger
+
+from catalyst.constants import LOG_LEVEL
+
+log = Logger('run_algo', level=LOG_LEVEL)
+
 
 class _RunAlgoError(click.ClickException, ValueError):
     """Signal an error that should have a different message if invoked from
@@ -68,7 +86,12 @@ def _run(handle_data,
          output,
          print_algo,
          local_namespace,
-         environ):
+         environ,
+         live,
+         exchange,
+         algo_namespace,
+         base_currency,
+         live_graph):
     """Run a backtest for the given algorithm.
 
     This is shared between the cli and :func:`catalyst.run_algo`.
@@ -117,106 +140,188 @@ def _run(handle_data,
         else:
             click.echo(algotext)
 
-    if bundle is not None:
-        bundles = bundle.split(',')
+    mode = 'live' if live else 'backtest'
+    log.info('running algo in {mode} mode'.format(mode=mode))
 
-        def get_trading_env_and_data(bundles):
-            env = data = None
+    exchange_name = exchange
+    if exchange_name is None:
+        raise ValueError('Please specify at least one exchange.')
 
-            b = 'poloniex'
-            if len(bundles) == 0:
-                return env, data
-            elif len(bundles) == 1:
-                b = bundles[0]
+    exchange_list = [x.strip().lower() for x in exchange.split(',')]
 
-            bundle_data = load(
-                b,
-                environ,
-                bundle_timestamp,
+    exchanges = dict()
+    for exchange_name in exchange_list:
+
+        # Looking for the portfolio from the cache first
+        portfolio = get_algo_object(
+            algo_name=algo_namespace,
+            key='portfolio_{}'.format(exchange_name),
+            environ=environ
+        )
+
+        if portfolio is None:
+            portfolio = ExchangePortfolio(
+                start_date=pd.Timestamp.utcnow()
             )
 
-            prefix, connstr = re.split(
-                r'sqlite:///',
-                str(bundle_data.asset_finder.engine.url),
-                maxsplit=1,
+        # This corresponds to the json file containing api token info
+        exchange_auth = get_exchange_auth(exchange_name)
+
+        if live and (exchange_auth['key'] == '' or exchange_auth['secret'] == ''):
+            raise ExchangeAuthEmpty(
+                    exchange=exchange_name.title(), 
+                    filename=os.path.join(get_exchange_folder(exchange_name, environ), 'auth.json') )
+
+        if exchange_name == 'bitfinex':
+            exchanges[exchange_name] = Bitfinex(
+                key=exchange_auth['key'],
+                secret=exchange_auth['secret'],
+                base_currency=base_currency,
+                portfolio=portfolio
             )
-            if prefix:
-                raise ValueError(
-                    "invalid url %r, must begin with 'sqlite:///'" %
-                    str(bundle_data.asset_finder.engine.url),
+        elif exchange_name == 'bittrex':
+            exchanges[exchange_name] = Bittrex(
+                key=exchange_auth['key'],
+                secret=exchange_auth['secret'],
+                base_currency=base_currency,
+                portfolio=portfolio
+            )
+        elif exchange_name == 'poloniex':
+            exchanges[exchange_name] = Poloniex(
+                key=exchange_auth['key'],
+                secret=exchange_auth['secret'],
+                base_currency=base_currency,
+                portfolio=portfolio
+            )
+        else:
+            raise ExchangeNotFoundError(exchange_name=exchange_name)
+
+    open_calendar = get_calendar('OPEN')
+
+    env = TradingEnvironment(
+        load=partial(
+            load_crypto_market_data,
+            environ=environ,
+            start_dt=start,
+            end_dt=end
+        ),
+        environ=environ,
+        exchange_tz='UTC',
+        asset_db_path=None  # We don't need an asset db, we have exchanges
+    )
+    env.asset_finder = AssetFinderExchange()
+    choose_loader = None  # TODO: use the DataPortal for in the algorithm class for this
+
+    if live:
+        start = pd.Timestamp.utcnow()
+
+        # TODO: fix the end data.
+        end = start + timedelta(hours=8760)
+
+        data = DataPortalExchangeLive(
+            exchanges=exchanges,
+            asset_finder=env.asset_finder,
+            trading_calendar=open_calendar,
+            first_trading_day=pd.to_datetime('today', utc=True)
+        )
+
+        def fetch_capital_base(exchange, attempt_index=0):
+            """
+            Fetch the base currency amount required to bootstrap
+            the algorithm against the exchange.
+
+            The algorithm cannot continue without this value.
+
+            :param exchange: the targeted exchange
+            :param attempt_index:
+            :return capital_base: the amount of base currency available for
+            trading
+            """
+            try:
+                log.debug('retrieving capital base in {} to bootstrap '
+                          'exchange {}'.format(base_currency, exchange_name))
+                balances = exchange.get_balances()
+            except ExchangeRequestError as e:
+                if attempt_index < 20:
+                    log.warn(
+                        'could not retrieve balances on {}: {}'.format(
+                            exchange.name, e
+                        )
+                    )
+                    sleep(5)
+                    return fetch_capital_base(exchange, attempt_index + 1)
+
+                else:
+                    raise ExchangeRequestErrorTooManyAttempts(
+                        attempts=attempt_index,
+                        error=e
+                    )
+
+            if base_currency in balances:
+                return balances[base_currency]
+            else:
+                raise BaseCurrencyNotFoundError(
+                    base_currency=base_currency,
+                    exchange=exchange_name
                 )
 
-            open_calendar = get_calendar('OPEN')
+        capital_base = 0
+        for exchange_name in exchanges:
+            exchange = exchanges[exchange_name]
+            capital_base += fetch_capital_base(exchange)
 
-            env = TradingEnvironment(
-                load=partial(load_crypto_market_data, environ=environ),
-                bm_symbol='USDT_BTC',
-                trading_calendar=open_calendar,
-                asset_db_path=connstr,
-                environ=environ,
-            )
+        sim_params = create_simulation_parameters(
+            start=start,
+            end=end,
+            capital_base=capital_base,
+            emission_rate='minute',
+            data_frequency='minute'
+        )
 
-            first_trading_day =\
-                bundle_data.equity_minute_bar_reader.first_trading_day
-
-            data = DataPortal(
-                env.asset_finder,
-                open_calendar,
-                first_trading_day=first_trading_day,
-                equity_minute_reader=bundle_data.equity_minute_bar_reader,
-                equity_daily_reader=bundle_data.equity_daily_bar_reader,
-                adjustment_reader=bundle_data.adjustment_reader,
-            )
-
-            return env, data
-
-        def get_loader_for_bundle(b):
-            bundle_data = load(
-                b,
-                environ,
-                bundle_timestamp,
-            )
-
-            if b == 'poloniex':
-                return CryptoPricingLoader(
-                           bundle_data.equity_daily_bar_reader,
-                           CryptoPricing,
-                       )
-            elif b == 'quantopian-quandl':
-                return USEquityPricingLoader(
-                           bundle_data.equity_daily_bar_reader,
-                           bundle_data.adjustment_reader,
-                           USEquityPricing,
-                       )
-            raise ValueError(
-                "No PipelineLoader registered for bundle %s." % b
-            )
-
-        loaders = [get_loader_for_bundle(b) for b in bundles]
-        env, data = get_trading_env_and_data(bundles)
-
-        def choose_loader(column):
-            for loader in loaders:
-                if column in loader.columns:
-                    return loader
-            raise ValueError(
-                "No PipelineLoader registered for column %s." % column
-            )
+        # TODO: use the constructor instead
+        sim_params._arena = 'live'
 
+        algorithm_class = partial(
+            ExchangeTradingAlgorithmLive,
+            exchanges=exchanges,
+            algo_namespace=algo_namespace,
+            live_graph=live_graph
+        )
     else:
-        env = TradingEnvironment(environ=environ)
-        choose_loader = None
+        # Removed the existing Poloniex fork to keep things simple
+        # We can add back the complexity if required.
 
-    perf = TradingAlgorithm(
-        namespace=namespace,
-        env=env,
-        get_pipeline_loader=choose_loader,
-        sim_params=create_simulation_parameters(
+        # I don't think that we should have arbitrary price data bundles
+        # Instead, we should center this data around exchanges.
+        # We still need to support bundles for other misc data, but we
+        # can handle this later.
+
+        data = DataPortalExchangeBacktest(
+            exchanges=exchanges,
+            asset_finder=None,
+            trading_calendar=open_calendar,
+            first_trading_day=start,
+            last_available_session=end
+        )
+
+        sim_params = create_simulation_parameters(
             start=start,
             end=end,
             capital_base=capital_base,
             data_frequency=data_frequency,
-        ),
+            emission_rate=data_frequency,
+        )
+
+        algorithm_class = partial(
+            ExchangeTradingAlgorithmBacktest,
+            exchanges=exchanges
+        )
+
+    perf = algorithm_class(
+        namespace=namespace,
+        env=env,
+        get_pipeline_loader=choose_loader,
+        sim_params=sim_params,
         **{
             'initialize': initialize,
             'handle_data': handle_data,
@@ -292,10 +397,10 @@ def load_extensions(default, extensions, strict, environ, reload=False):
             _loaded_extensions.add(ext)
 
 
-def run_algorithm(start,
-                  end,
-                  initialize,
-                  capital_base,
+def run_algorithm(initialize,
+                  capital_base=None,
+                  start=None,
+                  end=None,
                   handle_data=None,
                   before_trading_start=None,
                   analyze=None,
@@ -306,7 +411,12 @@ def run_algorithm(start,
                   default_extension=True,
                   extensions=(),
                   strict_extensions=True,
-                  environ=os.environ):
+                  environ=os.environ,
+                  live=False,
+                  exchange_name=None,
+                  base_currency=None,
+                  algo_namespace=None,
+                  live_graph=False):
     """Run a trading algorithm.
 
     Parameters
@@ -360,6 +470,12 @@ def run_algorithm(start,
     environ : mapping[str -> str], optional
         The os environment to use. Many extensions use this to get parameters.
         This defaults to ``os.environ``.
+    live: execute live trading
+    exchange_conn: The exchange connection parameters
+
+    Supported Exchanges
+    -------------------
+    bitfinex
 
     Returns
     -------
@@ -372,25 +488,30 @@ def run_algorithm(start,
     """
     load_extensions(default_extension, extensions, strict_extensions, environ)
 
-    non_none_data = valfilter(bool, {
-        'data': data is not None,
-        'bundle': bundle is not None,
-    })
-    if not non_none_data:
-        # if neither data nor bundle are passed use 'quantopian-quandl'
-        bundle = 'quantopian-quandl'
+    # I'm not sure that we need this since the modified DataPortal
+    # does not require extensions to be explicitly loaded.
 
-    elif len(non_none_data) != 1:
-        raise ValueError(
-            'must specify one of `data`, `data_portal`, or `bundle`,'
-            ' got: %r' % non_none_data,
-        )
+    # This will be useful for arbitrary non-pricing bundles but we may
+    # need to modify the logic.
+    if not live:
+        non_none_data = valfilter(bool, {
+            'data': data is not None,
+            'bundle': bundle is not None,
+        })
+        if not non_none_data:
+            # if neither data nor bundle are passed use 'quantopian-quandl'
+            bundle = 'quantopian-quandl'
 
-    elif 'bundle' not in non_none_data and bundle_timestamp is not None:
-        raise ValueError(
-            'cannot specify `bundle_timestamp` without passing `bundle`',
-        )
+        elif len(non_none_data) != 1:
+            raise ValueError(
+                'must specify one of `data`, `data_portal`, or `bundle`,'
+                ' got: %r' % non_none_data,
+            )
 
+        elif 'bundle' not in non_none_data and bundle_timestamp is not None:
+            raise ValueError(
+                'cannot specify `bundle_timestamp` without passing `bundle`',
+            )
     return _run(
         handle_data=handle_data,
         initialize=initialize,
@@ -410,4 +531,9 @@ def run_algorithm(start,
         print_algo=False,
         local_namespace=False,
         environ=environ,
+        live=live,
+        exchange=exchange_name,
+        algo_namespace=algo_namespace,
+        base_currency=base_currency,
+        live_graph=live_graph
     )

docs/CNAME

@@ -1 +1 @@
-www.zipline.io
+enigma-catalyst.readthedocs.io

docs/live-trading-blueprint.md

@@ -0,0 +1,207 @@
+<h1>Live Trading Blueprint</h1>
+The purpose of this document is to allow project contributors navigate
+through the ongoing live trading implementation.
+
+<h2>Components</h2>
+At a high level, the following components have been implemented to coerce
+zipline into live trading.
+
+<h3>Exchange</h3>
+
+*catalyst/exchange*
+
+Exchange is a new package introducing cryptocurrency
+exchanges to zipline. The package contains mostly new implementations
+of existing components, adapted to characteristics of exchanges.
+
+Here are some key characteristics which make cryptocurrency exchanges
+exchanges different compared to equity brokers.
+* They trade around the clock.
+* Currency symbols are inconsistent across exchanges.
+* They trade currency pairs (i.e. the base currency is not always be USD).
+This is a paradigm shift in context of zipline. Additional
+business logic will be required to manage the portfolio data and orders.
+* The price of a single asset might vary across exchanges. This means
+arbitrage opportunities. Consequently, to extract maximum alpha, the
+platform should not only support multiple exchanges, but also multiple
+exchanges per algorithm.
+* The fee model is usually more complex than that of an equity broker.
+It can vary drastically between exchanges.
+* There are no splits, mergers, etc. to worry about.
+* A complete order book is usually available, the platform should
+offer access to it order to help traders reduce slippage.
+
+<h3>New Components</h3>
+These components of the exchange package were added to the zipline
+sources.
+
+<h4>Exchange</h4>
+
+*catalyst/exchange/exchange.py*
+
+Abstract class which acts as an interface for the implementation of
+various exchanges. It also contains logic common to all exchanges.
+
+<h4>Bitfinex</h4>
+
+*catalyst/exchange/bitfinex.py*
+
+The Bitfinex exchange implementation. It extends the Exchange class.
+
+<h4>DataPortalExchange</h4>
+
+*catalyst/exchange/data_portal_exchange.py*
+
+Extends the zipline DataPortal to route spot data to the exchange.
+This is critical because it allows the algoritm to request data in
+real-time.
+
+For example, `data.current(asset, 'price')` retrieves the current price
+of the asset, not the price at the time of yielding the bar this
+is critical to minimize slippage.
+
+At the time of writing, it only supports spot data but I believe that
+it should be extended to historical data as well. Some exchanges
+have better historical data APIs than others. This will need to
+be considered during each individual implementation.
+
+<h4>ExchangeClock</h4>
+
+*catalyst/exchange/exchange_clock.py*
+
+An implementation to the zipline Clock which runs 24/7. It yields a
+bar every minute.
+
+<h4>AssetFinderExchange</h4>
+
+*catalyst/exchange/asset_finder_exchange.py*
+
+An alternate implementation of AssetFinder which locates each asset
+against the exchanges instead of bundle databases.
+
+For example, `symbol('eth_usd')` should return an Ethereum/USD asset
+regardless of currency notation of the target exchange.
+
+To acheive this, I have created a dictionary of currencies for the
+Bitfinex exchange. Here is what it looks like.
+* Each key represents the exchange specific symbol.
+* The symbol attribute represents the abstract symbol common across
+all exchanges for the given currency.
+* The start_date attribute should correspond to its first trading day
+on the exchange.
+
+```json
+{
+  "btcusd": {
+    "symbol": "btc_usd",
+    "start_date": "2010-01-01"
+  },
+  "ltcusd": {
+    "symbol": "ltc_usd",
+    "start_date": "2010-01-01"
+  },
+  "ltcbtc": {
+    "symbol": "ltc_btc",
+    "start_date": "2010-01-01"
+  },
+  "ethusd": {
+    "symbol": "eth_usd",
+    "start_date": "2010-01-01"
+  },
+  "ethbtc": {
+    "symbol": "eth_btc",
+    "start_date": "2010-01-01"
+  }
+}
+```
+
+<h4>ExchangeTradingAlgorithm</h4>
+
+*catalyst/exchange/algorithm_exchange.py*
+
+Extends the TradingAlgorithm class which orchestrates the api
+operations. This class brings together most of the components
+described above.
+
+<h3>Modified Components</h3>
+
+The following components have been modified to include conditional
+business logic to enable live trading.
+
+<h4>run_algorithm</h4>
+
+*catalyst/utils/run_algo.py*
+
+The run_algorithm interface is an entry point to execute an
+algorithm in zipline. This component was already modified for
+the catalyst concurrency bundles. I added conditional logic
+which should not interfere with backtesting.
+
+In a nutshell, the run_algorithm method now contains three additional
+parameters:
+* live: If True, zipline will attempt to trade live. If False or not
+specified, it will run a backtest as normal.
+* algo_namespace: An arbitrary namespace for the current algorithm.
+It will be used to persist data between runs.
+* exchange_conn: A dictionary containing the attributes required
+to instantiate an exchange. Here is an example for Bitfinex:
+
+```python
+exchange_conn = dict(
+    name='bitfinex',
+    key='',
+    secret=b'',
+    base_currency='usd'
+)
+```
+
+The following sample algorithm uses the run_algorithm interface:
+
+*catalyst/examples/buy_and_hold_live.py*
+
+<h2>Portfolio Management</h2>
+
+Zipline has a Portfolio class containing key metrics used by zipline
+for, but not only, these reasons:
+
+* Placing orders: When placing orders (e.g. order_target_percent),
+zipline queries the portfolio to assess the size of current positions,
+cash available, etc.
+* Measuring performance: The portfolio contains attributes like
+cost basis of each asset, p&l, etc. which zipline uses to compute all
+of its performance criteria.
+
+When backtesting, zipline automatically updates the Portfolio object
+of its corresponding algorithm. When live trading, these updates should
+be the responsibility of the exchange as it holds the truth for:
+
+* Executed price of each order (including fees and slippage)
+* Partial / failed orders
+* Cash (i.e. base currency) available
+* Cost basis of each position
+
+If each exchange account had a one-to-one relationship with an
+algorithm, portfolio metrics could be retrieved directly from the
+exchange without persisting any data to the algorithm. However,
+doing this would have at least the following drawbacks:
+
+* It may not be reasonable to ask users to dedicate an
+exchange account to a single algorithm. Exchanges are not easy
+to partition.
+* If an exchange account contains existing positions, the calculated
+cost basis would correspond to all positions, not just those
+initiated by the algorithm.
+* It would not be possible impose trading limits on algorithms.
+
+It follows that Portfolio metrics should be calculated using a strategic
+combination of the exchange data and algorithm activity. While tracking
+the activity of an algorithm works well in backtesting, it is more
+challenging during live trading. A live algorithm might run over
+several months. It might have to stop and start for many reasons.
+This means that the platform should have the ability to persist
+algorithm activity in order to be reliable.
+
+In the interest of time, I will start by persisting algorithm
+activity in memory. Data will be lost when the algorithm execution stops.
+The intent it to offer a simple basis from which to implement data
+persistence strategies in the future.

docs/live-trading-wiki.md

@@ -0,0 +1,105 @@
+<h1>Live Trading</h1>
+This document explains how to get started with live trading.
+
+<h2>Supported Exchanges</h2>
+Catalyst can trade against these exchanges:
+
+* Bitfinex, id=`bitfinex`
+* Bittrex, id=`bittrex`
+
+<h3>Authentication</h3>
+Most exchanges require key/token combination for authentication. By
+convention, Catalyst uses an "auth.json" file to hold this data.
+
+This example illustrates the convention using the Bitfinex exchange.
+Here is how to generate key and secret values for bitfinex:
+https://docs.bitfinex.com/v1/docs/api-access. Most exchanges follow
+a similar process.
+
+The auth.json file:
+```json
+{
+  "name": "bitfinex",
+  "key": "my-key",
+  "secret": "my-secret"
+}
+```
+
+The file goes here:
+```
+~/.catalyst/data/exchanges/bitfinex/auth.json
+```
+
+Note that the 'bitfinex' directory corresponds to the id of the Bitfinex
+exchange as defined in the "Supported Exchanges" section above.
+Attempting to run an algorithm where the targeted exchange is missing
+its "auth.json" file will create the directory structure but result
+in an error.
+
+<h3>Currency Symbols</h3>
+Catalyst introduces a universal convention to reference
+trading pairs and individual currencies. This
+is required to ensure that the `symbol()` api predictably
+returns the correct asset regardless of the targeted exchange.
+
+Exchanges tend to use their own convention to represent currencies
+(e.g. XBT and BTC both represent Bitcoin on different exchanges).
+Trading pairs are also inconsistent. For example, Bitfinex
+puts the market currency before the base currency without a
+separator, Bittrex puts the base currency first and uses a dash
+seperator.
+
+Here is the Catalyst convention:
+
+*[Market Currency]_[Base Currency]* all lowercase.
+
+Currency symbols (e.g. btc, eth, ltc) follow the Bittrex convention.
+
+Here are some examples:
+```python
+# With Bitfinex
+bitcoin_usd_asset = symbol('btc_usd')
+ethereum_bitcoin_asset = symbol('eth_btc')
+
+# With Bittrex
+ethereum_bitcoin_asset = symbol('eth_btc')
+neo_ethereum_asset = symbol('neo_eth)
+```
+
+Note that the trading pairs are always referenced in the same manner.
+However, not all trading pairs are available on all exchanges. An
+error will occur if the specified trading pair is not trading
+on the exchange.
+
+<h2>Trading an Algorithm</h2>
+There is no special convention to follow when writing an
+algorithm for live trading. The same algorithm should work in
+backtest and live execution mode without modification.
+
+What differs are the arguments provided to the catalyst client or
+`run_algorithm()` interface. Here is example:
+
+```python
+run_algorithm(
+    initialize=initialize,
+    handle_data=handle_data,
+    analyze=analyze,
+    exchange_name='bitfinex',
+    live=True,
+    algo_namespace='my_algo_trading_xrp',
+    base_currency='btc'
+)
+```
+
+Here is the breakdown of the new arguments:
+* live: Boolean flag which enables live trading.
+* exchange_name: The name of the targeted exchange
+    (supported values: *bitfinex*, *bittrex*).
+* algo_namespace: A arbitrary label assigned to your algorithm for
+    data storage purposes.
+* base_currency: The base currency used to calculate the
+    statistics of your algorithm. Currently, the base currency of all
+    trading pairs of your algorithm must match this value.
+
+Here is a complete algorithm for reference:
+[Buy Low and Sell High](../catalyst/examples/buy_low_sell_high_live.py)

docs/source/conf.py

@@ -1,7 +1,7 @@
 import sys
 import os
 
-from zipline import __version__ as version
+#from catalyst import __version__ as version
 
 # If extensions (or modules to document with autodoc) are in another directory,
 # add these directories to sys.path here. If the directory is relative to the
@@ -21,14 +21,14 @@ extensions = [
 
 
 extlinks = {
-    'issue': ('https://github.com/quantopian/zipline/issues/%s', '#'),
-    'commit': ('https://github.com/quantopian/zipline/commit/%s', ''),
+    'issue': ('https://github.com/enigmampc/catalyst/issues/%s', '#'),
+    'commit': ('https://github.com/enigmampc/catalyst/commit/%s', ''),
 }
 
 # -- Docstrings ---------------------------------------------------------------
 
-extensions += ['numpydoc']
-numpydoc_show_class_members = False
+#extensions += ['numpydoc']
+#numpydoc_show_class_members = False
 
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['.templates']
@@ -40,11 +40,12 @@ source_suffix = '.rst'
 master_doc = 'index'
 
 # General information about the project.
-project = u'Zipline'
-copyright = u'2016, Quantopian Inc.'
+project = u'Catalyst'
+copyright = u'2017, Enigma MPC, Inc.'
 
 # The full version, including alpha/beta/rc tags, but excluding the commit hash
-release = version.split('+', 1)[0]
+#release = version.split('+', 1)[0]
+release = '0.3'
 
 # List of patterns, relative to source directory, that match files and
 # directories to ignore when looking for source files.
@@ -84,7 +85,7 @@ html_show_sphinx = True
 html_show_copyright = True
 
 # Output file base name for HTML help builder.
-htmlhelp_basename = 'ziplinedoc'
+htmlhelp_basename = 'catalystdoc'
 
 intersphinx_mapping = {
     'http://docs.python.org/dev': None,
@@ -93,6 +94,6 @@ intersphinx_mapping = {
     'pandas': ('http://pandas.pydata.org/pandas-docs/stable/', None),
 }
 
-doctest_global_setup = "import zipline"
+doctest_global_setup = "import catalyst"
 
 todo_include_todos = True

docs/source/index.rst

@@ -1,12 +1,17 @@
-.. include:: ../../README.rst
+.. include:: welcome.rst
+|
+|
+Table of Contents
+-----------------
 
 .. toctree::
    :maxdepth: 1
 
    install
    beginner-tutorial
-   bundles
-   development-guidelines
-   appendix
-   release-process
-   releases
+   naming-convention
+..   bundles
+..   development-guidelines
+..   appendix
+..   release-process
+..   releases

docs/source/install.rst

@@ -4,16 +4,16 @@ Install
 Installing with ``pip``
 -----------------------
 
-Installing Zipline via ``pip`` is slightly more involved than the average
+Installing Catalyst via ``pip`` is slightly more involved than the average
 Python package.
 
 There are two reasons for the additional complexity:
 
-1. Zipline ships several C extensions that require access to the CPython C API.
+1. Catalyst ships several C extensions that require access to the CPython C API.
    In order to build the C extensions, ``pip`` needs access to the CPython
    header files for your Python installation.
 
-2. Zipline depends on `numpy <http://www.numpy.org/>`_, the core library for
+2. Catalyst depends on `numpy <http://www.numpy.org/>`_, the core library for
    numerical array computing in Python.  Numpy depends on having the `LAPACK
    <http://www.netlib.org/lapack>`_ linear algebra routines available.
 
@@ -28,13 +28,28 @@ your particular platform), you should be able to simply run
 
 .. code-block:: bash
 
-   $ pip install zipline
+   $ pip install enigma-catalyst
 
-If you use Python for anything other than Zipline, we **strongly** recommend
+If you use Python for anything other than Catalyst, we **strongly** recommend
 that you install in a `virtualenv
 <https://virtualenv.readthedocs.org/en/latest>`_.  The `Hitchhiker's Guide to
 Python`_ provides an `excellent tutorial on virtualenv
-<http://docs.python-guide.org/en/latest/dev/virtualenvs/>`_.
+<http://docs.python-guide.org/en/latest/dev/virtualenvs/>`_. Here's a summarized
+version:
+
+.. code-block:: bash
+
+   $ virtualenv catalyst-venv
+   $ source ./catalyst-venv/bin/activate
+   $ pip install enigma-
+
+Though not required by Catalyst directly, our example algorithms use matplotlib 
+to visually display the results of the trading algorithms. If you wish to run 
+any examples or use matplotlib during development, it can be installed using:
+
+.. code-block:: bash
+
+    $ pip install matplotlib
 
 GNU/Linux
 ~~~~~~~~~
@@ -60,15 +75,17 @@ On `Arch Linux`_, you can acquire the additional dependencies via ``pacman``:
 
    $ pacman -S lapack gcc gcc-fortran pkg-config
 
-There are also AUR packages available for installing `Python 3.4
-<https://aur.archlinux.org/packages/python34/>`_ (Arch's default python is now
-3.5, but Zipline only currently supports 3.4), and `ta-lib
-<https://aur.archlinux.org/packages/ta-lib/>`_, an optional Zipline dependency.
-Python 2 is also installable via:
+.. Commenting it out until Catalyst fully supports Python 3.X
+..
+.. There are also AUR packages available for installing `Python 3.4
+.. <https://aur.archlinux.org/packages/python34/>`_ (Arch's default python is now
+.. 3.5, but Catalyst only currently supports 3.4), and `ta-lib
+.. <https://aur.archlinux.org/packages/ta-lib/>`_, an optional Catalyst dependency.
+.. Python 2 is also installable via:
 
-.. code-block:: bash
+.. 
 
-   $ pacman -S python2
+..   $ pacman -S python2
 
 OSX
 ~~~
@@ -87,36 +104,238 @@ following brew packages:
 
    $ brew install freetype pkg-config gcc openssl
 
+OSX + virtualenv + matplotlib
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+A note about using matplotlib in virtual enviroments on OSX: it may be necessary to run
+
+.. code-block:: bash
+
+   echo "backend: TkAgg" > ~/.matplotlib/matplotlibrc
+
+in order to override the default ``macosx`` backend for your system, which may not 
+be accessible from inside the virtual environment. This will allow Catalyst to open 
+matplotlib charts from within a virtual environment, which is useful for displaying 
+the performance of your backtests.  To learn more about matplotlib backends, please refer to the
+`matplotlib backend documentation <https://matplotlib.org/faq/usage_faq.html#what-is-a-backend>`_.
+
+
 Windows
 ~~~~~~~
 
-For windows, the easiest and best supported way to install zipline is to use
+In Windows, you will need the `Microsoft Visual C++ Compiler for Python 2.7 
+<https://www.microsoft.com/en-us/download/details.aspx?id=44266>`_. This package 
+contains the compiler and the set of system headers necessary for producing 
+binary wheels for Python 2.7 packages. If it's not already in your system, download
+it and install it before proceeding to the next step.
+
+For windows, the easiest and best supported way to install Catalyst is to use
 :ref:`Conda <conda>`.
 
+Amazon Linux AMI
+~~~~~~~~~~~~~~~~
+
+The packages ``pip`` and ``setuptools`` that come shipped by default are very outdated. 
+Thus, you first need to run:
+
+.. code-block:: bash
+
+   pip install --upgrade pip setuptools
+
+The default installation is also missing the C and C++ compilers, which you install by:
+
+.. code-block:: bash
+
+   sudo yum install gcc gcc-c++
+
+Then you should follow the regular installation instructions outlined at the beginning 
+of this page.
+
+
+Troubleshooting ``pip`` Install
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+**Issue**: 
+   Package enigma-catalyst cannot be found 
+ 
+**Solution**: 
+   Make sure you have the most up-to-date version of pip installed, by running: 
+
+   .. code-block:: bash
+
+      pip install --upgrade pip
+
+   On Windows, the recommended command is:
+
+   .. code-block:: bash
+
+      python -m pip install --upgrade pip
+
+----
+
+**Issue**: 
+   Package enigma-catalyst cannot still be found, even after upgrading pip (see above), with an error similar to:
+
+   .. code-block:: bash
+
+      Downloading/unpacking enigma-catalyst
+      Could not find a version that satisfies the requirement enigma-catalyst (from versions: 0.1.dev9, 0.2.dev2, 0.1.dev4, 0.1.dev5, 0.1.dev3, 0.2.dev1, 0.1.dev8, 0.1.dev6)
+      Cleaning up...
+      No distributions matching the version for enigma-catalyst
+
+**Solution**:
+   In some systems (this error has been reported in Ubuntu), pip is configured to only find stable versions by default. Since Catalyst is in alpha version, pip cannot find a matching version that satisfies the installation requirements. The solution is to include the `--pre` flag to include pre-release and development versions:
+
+   .. code-block:: bash
+
+      pip install --pre enigma-catalyst
+
+----
+
+**Issue**: 
+   Package enigma-catalyst fails to install because of outdated setuptools
+
+**Solution**: 
+   Upgrade to the most up-to-date setuptools package by running: 
+
+   .. code-block:: bash
+
+      pip install --upgrade pip setuptools
+
+----
+
+**Issue**:
+   Missing required packages  
+
+**Solution**:
+   Download `requirements.txt 
+   <https://github.com/enigmampc/catalyst/blob/master/etc/requirements.txt>`_ 
+   (click on the *Raw* button and Right click -> Save As...) and use it to
+   install all the required dependencies by running:
+
+   .. code-block:: bash
+
+      pip install -r requirements.txt
+
+----
+
+**Issue**: 
+   Installation fails with error: ``fatal error: Python.h: No such file or directory``
+
+**Solution**: 
+   Some systems (this issue has been reported in Ubuntu) require `python-dev` for the proper build and installation of package dependencies. The solution is to install python-dev, which is independent of the virtual environment. In Ubuntu, you would need to run:
+
+   .. code-block:: bash
+
+      sudo apt-get install python-dev
+
+
 .. _conda:
 
 Installing with ``conda``
 -------------------------
 
-Another way to install Zipline is via the ``conda`` package manager, which
+Another way to install Catalyst is via the ``conda`` package manager, which
 comes as part of Continuum Analytics' `Anaconda
 <http://continuum.io/downloads>`_ distribution.
 
 The primary advantage of using Conda over ``pip`` is that conda natively
 understands the complex binary dependencies of packages like ``numpy`` and
-``scipy``.  This means that ``conda`` can install Zipline and its dependencies
-without requiring the use of a second tool to acquire Zipline's non-Python
+``scipy``.  This means that ``conda`` can install Catalyst and its dependencies
+without requiring the use of a second tool to acquire Catalyst's non-Python
 dependencies.
 
 For instructions on how to install ``conda``, see the `Conda Installation
-Documentation <http://conda.pydata.org/docs/download.html>`_
+Documentation <http://conda.pydata.org/docs/download.html>`_. Alternatively, you 
+can install MiniConda, which is a smaller footprint (fewer packages and smaller 
+size) than its big brother Anaconda, but it still contains all the main packages 
+needed. To install MiniConda, you can follow these steps:
 
-Once conda has been set up you can install Zipline from our ``Quantopian``
-channel:
+1. Download `MiniConda <https://conda.io/miniconda.html>`_. Select Python 2.7 for 
+   your Operating System.
+2. Install MiniConda. See the `Installation Instructions <https://conda.io/docs/user-guide/install/index.html>`_
+   if you need help.
+3. Ensure the correct installation by running ``conda list`` in a Terminal window,
+   which should print the list of packages installed with Conda.
 
-.. code-block:: bash
+Once either Conda or MiniConda has been set up you can install Catalyst:
+
+1. Download the file `python2.7-environment.yml <https://github.com/enigmampc/catalyst/blob/master/etc/python2.7-environment.yml>`_.
+2. Open a Terminal window and enter [``cd/dir``] into the directory where you saved
+   the above ``python2.7-environment.yml`` file.
+3. Install using this file. This step can take about 5-10 minutes to install.
+
+   .. code-block:: bash
+
+      conda env create -f python2.7-environment.yml
+
+4. Activate the environment (which you need to do every time you start a new session
+   to run Catalyst):
+
+   **Linux or OSX:**
+
+   .. code-block:: bash
+
+      source activate catalyst
+
+   **Windows:**
+
+   .. code-block:: bash
+
+      activate catalyst
+
+Congratulations! You now have Catalyst installed.
+
+Troubleshooting ``conda`` Install
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If the command  ``conda env create -f python2.7-environment.yml`` in step 3 above failed 
+for any reason, you can try setting up the environment manually with the following steps:
+
+1. Create the environment:
+
+   .. code-block:: bash
+
+      conda create --name catalyst python=2.7 scipy
+
+2. Activate the environment:
+
+   **Linux or OSX:**
+
+   .. code-block:: bash
+
+      source activate catalyst
+
+   **Windows:**
+
+   .. code-block:: bash
+
+      activate catalyst
+
+3. Install the Catalyst inside the environment:
+
+   .. code-block:: bash
+
+      pip install enigma-catalyst matplotlib
+
+Getting Help
+------------
+
+If after following the instructions above, and going through the *Troubleshooting* sections, 
+you still experience problems installing Catalyst, you can seek additional help through the 
+following channels:
+
+- Join our `Discord community <https://discord.gg/SJK32GY>`_, and head over the #catalyst_dev 
+  channel where many other users (as well as the project developers) hang out, and can assist 
+  you with your particular issue. The more descriptive and the more information you can provide, 
+  the easiest will be for others to help you out.
+
+- Report the problem you are experiencing on our 
+  `GitHub repository <https://github.com/enigmampc/catalyst/issues>`_ following the guidelines 
+  provided therein. Before you do so, take a moment to browse through all `previous reported issues
+  <https://github.com/enigmampc/catalyst/issues?utf8=%E2%9C%93&q=is%3Aissue>`_ in the likely case
+  that someone else experienced that same issue before, and you get a hint on how to solve it.
 
-    conda install -c Quantopian zipline
 
 .. _`Debian-derived`: https://www.debian.org/misc/children-distros
 .. _`RHEL-derived`: https://en.wikipedia.org/wiki/Red_Hat_Enterprise_Linux_derivatives

docs/source/naming-convention.rst

@@ -0,0 +1,66 @@
+Naming Convention
+=================
+
+Catalyst introduces a standardized naming convention for all asset pairs 
+trading on any exchange in the following form:
+
+
+    **{market_currency}_{base_currency}**
+
+Where {market_currency} is the asset to be traded using {base_currency} as 
+the reference, both written in lowercase and separated with an underscore.
+
+This standardization is needed to overcome the lack of consistency in the 
+naming of assets across different exchanges, and making it easier to the user
+to refer to the asset pairs that you want to trade.
+
+Catalyst maintains a `Market Coverage Overview <https://www.enigma.co/catalyst/status>`_ 
+where you can check the mapping between Catalyst naming pairs and that of each 
+exchange. Catalyst will always expect in all its functions that you will refer to 
+the asset pairs by using the Catalyst naming convention.
+
+If at any point, you input the wrong name for an asset pair, you will get an error 
+of that pair not found in the given exchange, and a list of pairs available on that exchange:
+
+.. code-block:: bash
+
+   $ catalyst ingest-exchange -x poloniex -i btc_usd
+
+.. parsed-literal::
+
+	Ingesting exchange bundle poloniex...
+	Error traceback: /Volumes/Data/Users/victoris/Desktop/Enigma/user-install/catalyst-dev/catalyst/exchange/exchange.py (line 175)
+	SymbolNotFoundOnExchange:  Symbol btc_usd not found on exchange Poloniex. 
+	Choose from: ['rep_usdt', 'gno_btc', 'xvc_btc', 'pink_btc', 'sys_btc', 
+	'emc2_btc', 'rads_btc', 'note_btc', 'maid_btc', 'bch_btc', 'gnt_btc', 
+	'bcn_btc', 'rep_btc', 'bcy_btc', 'cvc_btc', 'nxt_xmr', 'zec_usdt', 
+	'fct_btc', 'gas_btc', 'pot_btc', 'eth_usdt', 'btc_usdt', 'lbc_btc', 
+	'dcr_btc', 'etc_usdt', 'omg_eth', 'amp_btc', 'xpm_btc', 'nxt_btc', 
+	'vtc_btc', 'steem_eth', 'blk_xmr', 'pasc_btc', 'zec_xmr', 'grc_btc', 
+	'nxc_btc', 'btcd_btc', 'ltc_btc', 'dash_btc', 'naut_btc', 'zec_eth', 
+	'zec_btc', 'burst_btc', 'zrx_eth', 'bela_btc', 'steem_btc', 'etc_btc', 
+	'eth_btc', 'huc_btc', 'strat_btc', 'lsk_btc', 'exp_btc', 'clam_btc', 
+	'rep_eth', 'dash_xmr', 'cvc_eth', 'bch_usdt', 'zrx_btc', 'dash_usdt', 
+	'blk_btc', 'xrp_btc', 'nxt_usdt', 'neos_btc', 'omg_btc', 'bts_btc', 
+	'doge_btc', 'gnt_eth', 'sbd_btc', 'gno_eth', 'xcp_btc', 'ltc_usdt', 
+	'btm_btc', 'xmr_usdt', 'lsk_eth', 'omni_btc', 'nav_btc', 'fldc_btc', 
+	'ppc_btc', 'xbc_btc', 'dgb_btc', 'sc_btc', 'btcd_xmr', 'vrc_btc', 
+	'ric_btc', 'str_btc', 'maid_xmr', 'xmr_btc', 'sjcx_btc', 'via_btc', 
+	'xem_btc', 'nmc_btc', 'etc_eth', 'ltc_xmr', 'ardr_btc', 'gas_eth', 
+	'flo_btc', 'xrp_usdt', 'game_btc', 'bch_eth', 'bcn_xmr', 'str_usdt']
+
+In the example above, exchange Poloniex does not use USD, but uses instead the 
+USDT cryptocurrency asset that is issued on the Bitcoin blockchain via the Omni
+Layer Protocol. Each USDT unit is backed by a U.S Dollar held in the reserves of 
+Tether Limited. USDT can be transferred, stored, and spent, just like bitcoins 
+or any other cryptocurrency. Given its 1:1 mapping to the USD, is a viable alternative.
+
+.. code-block:: bash
+
+   $ catalyst ingest-exchange -x poloniex -i btc_usdt
+
+.. parsed-literal::
+
+	Ingesting exchange bundle poloniex...
+	    [====================================]  Fetching poloniex daily candles: :  100%
+

docs/source/welcome.rst

@@ -0,0 +1,28 @@
+.. image:: https://s3.amazonaws.com/enigmaco-docs/enigma-catalyst.jpg
+|
+Catalyst is a data-driven crypto investment platform. It supports both 
+backtesting and live-trading in a number of different crypto-exchanges. 
+Catalyst empowers users to share and curate data and build profitable, 
+data-driven investment strategies.
+
+Features
+========
+
+-  Ease of use: Catalyst tries to get out of your way so that you can 
+   focus on algorithm development. See 
+   `examples of trading strategies <https://github.com/enigmampc/catalyst/tree/master/catalyst/examples>`_ 
+   provided.
+-  Support for several of the top crypto-exchanges by trading volume:
+   `Bitfinex <https://www.bitfinex.com>`_, `Bittrex <http://www.bittrex.com>`_,
+   and `Poloniex <https://www.poloniex.com>`_. 
+-  Secure: You and only you have access to each exchange API keys for your accounts.
+-  Input of historical pricing data of all crypto-assets by exchange, 
+   with daily and minute resolution. See 
+   `Catalyst Market Coverage Overview <https://www.enigma.co/catalyst/status>`_.
+-  Backtesting and live-trading functionality, with a seamless transition
+   between the two modes.
+-  Output of performance statistics are based on Pandas DataFrames to 
+   integrate nicely into the existing PyData eco-system.
+-  Statistic and machine learning libraries like matplotlib, scipy, 
+   statsmodels, and sklearn support development, analysis, and 
+   visualization of state-of-the-art trading systems.

etc/python2.7-environment.yml

@@ -0,0 +1,60 @@
+name: catalyst
+channels:
+- defaults
+dependencies:
+- certifi=2016.2.28=py27_0
+- libgfortran=3.0.0=1
+- mkl=2017.0.3=0
+- numpy=1.13.1=py27_0
+- openssl=1.0.2l=0
+- pip=9.0.1=py27_1
+- python=2.7.13=0
+- readline=6.2=2
+- scipy=0.19.1=np113py27_0
+- setuptools=36.4.0=py27_1
+- sqlite=3.13.0=0
+- tk=8.5.18=0
+- wheel=0.29.0=py27_0
+- zlib=1.2.11=0
+- pip:
+  - alembic==0.9.6
+  - backports.functools-lru-cache==1.4
+  - bcolz==0.12.1
+  - bottleneck==1.2.1
+  - chardet==3.0.4
+  - click==6.7
+  - contextlib2==0.5.5
+  - cycler==0.10.0
+  - cyordereddict==1.0.0
+  - cython==0.27.1
+  - decorator==4.1.2
+  - empyrical==0.2.1
+  - idna==2.6
+  - intervaltree==2.1.0
+  - logbook==1.1.0
+  - lru-dict==1.1.6
+  - mako==1.0.7
+  - markupsafe==1.0
+  - matplotlib==2.1.0
+  - multipledispatch==0.4.9
+  - networkx==2.0
+  - numexpr==2.6.4
+  - pandas==0.19.2
+  - pandas-datareader==0.5.0
+  - patsy==0.4.1
+  - pyparsing==2.2.0
+  - python-dateutil==2.6.1
+  - python-editor==1.0.3
+  - pytz==2017.2
+  - requests==2.18.4
+  - requests-file==1.4.2
+  - requests-ftp==0.3.1
+  - six==1.11.0
+  - sortedcontainers==1.5.7
+  - sqlalchemy==1.1.14
+  - statsmodels==0.8.0
+  - subprocess32==3.2.7
+  - tables==3.4.2
+  - toolz==0.8.2
+  - urllib3==1.22
+  - enigma-catalyst>=0.3

etc/requirements.txt

@@ -1,7 +1,7 @@
 # Incompatible with earlier PIP versions
 pip>=7.1.0
 # bcolz fails to install if this is not in the build_requires.
-setuptools>18.0
+setuptools>36.0
 
 # Logging
 Logbook==0.12.5
@@ -9,7 +9,9 @@ Logbook==0.12.5
 # Scientific Libraries
 
 pytz==2016.4
-numpy==1.11.1
+
+# FF: Upgraded numpy because of errors with version 1.11
+numpy==1.13.1
 
 # for pandas-datareader
 requests-file==1.4.1
@@ -77,3 +79,4 @@ lru-dict==1.1.4
 empyrical==0.2.1
 
 tables==3.3.0
+

setup.py

@@ -38,6 +38,7 @@ class LazyBuildExtCommandClass(dict):
     Lazy command class that defers operations requiring Cython and numpy until
     they've actually been downloaded and installed by setup_requires.
     """
+
     def __contains__(self, key):
         return (
             key == 'build_ext'
@@ -62,6 +63,7 @@ class LazyBuildExtCommandClass(dict):
             Custom build_ext command that lazily adds numpy's include_dir to
             extensions.
             """
+
             def build_extensions(self):
                 """
                 Lazily append numpy's include directory to Extension includes.
@@ -75,6 +77,7 @@ class LazyBuildExtCommandClass(dict):
                     ext.include_dirs.append(numpy_incl)
 
                 super(build_ext, self).build_extensions()
+
         return build_ext
 
 
@@ -100,7 +103,8 @@ ext_modules = [
     window_specialization('label'),
     Extension('catalyst.lib.rank', ['catalyst/lib/rank.pyx']),
     Extension('catalyst.data._equities', ['catalyst/data/_equities.pyx']),
-    Extension('catalyst.data._adjustments', ['catalyst/data/_adjustments.pyx']),
+    Extension('catalyst.data._adjustments',
+              ['catalyst/data/_adjustments.pyx']),
     Extension('catalyst._protocol', ['catalyst/_protocol.pyx']),
     Extension('catalyst.gens.sim_engine', ['catalyst/gens/sim_engine.pyx']),
     Extension(
@@ -117,7 +121,6 @@ ext_modules = [
     ),
 ]
 
-
 STR_TO_CMP = {
     '<': lt,
     '<=': le,
@@ -212,7 +215,7 @@ def read_requirements(path,
                       conda_format=False,
                       filter_names=None):
     """
-    Read a requirements.txt file, expressed as a path relative to Zipline root.
+    Read a requirements.txt file, expressed as a path relative to Catalyst root.
 
     Returns requirements with the pinned versions as lower bounds
     if `strict_bounds` is falsey.
@@ -264,6 +267,7 @@ def setup_requirements(requirements_path, module_names, strict_bounds,
         )
     return module_lines
 
+
 conda_build = os.path.basename(sys.argv[0]) in ('conda-build',  # unix
                                                 'conda-build-script.py')  # win
 
@@ -295,12 +299,12 @@ setup(
     ext_modules=ext_modules,
     include_package_data=True,
     package_data={root.replace(os.sep, '.'):
-                  ['*.pyi', '*.pyx', '*.pxi', '*.pxd']
+                      ['*.pyi', '*.pyx', '*.pxi', '*.pxd']
                   for root, dirnames, filenames in os.walk('catalyst')
                   if '__pycache__' not in root},
     license='Apache 2.0',
     classifiers=[
-        'Development Status :: 2 - Pre-Alpha',
+        'Development Status :: 3 - Alpha',
         'License :: OSI Approved :: Apache Software License',
         'Natural Language :: English',
         'Programming Language :: Python',

tests/exchange/base.py

@@ -0,0 +1,38 @@
+import unittest
+from abc import ABCMeta, abstractmethod
+
+
+class BaseExchangeTestCase:
+    __metaclass__ = ABCMeta
+
+    @abstractmethod
+    def test_order(self):
+        pass
+
+    @abstractmethod
+    def test_open_orders(self):
+        pass
+
+    @abstractmethod
+    def test_get_order(self):
+        pass
+
+    @abstractmethod
+    def test_cancel_order(self):
+        pass
+
+    @abstractmethod
+    def test_get_candles(self):
+        pass
+
+    @abstractmethod
+    def test_tickers(self):
+        pass
+
+    @abstractmethod
+    def test_get_balances(self):
+        pass
+
+    @abstractmethod
+    def test_get_account(self):
+        pass

tests/exchange/test_bitfinex.py

@@ -0,0 +1,77 @@
+from logbook import Logger
+
+from base import BaseExchangeTestCase
+from catalyst.exchange.bitfinex.bitfinex import Bitfinex
+from catalyst.exchange.exchange_utils import get_exchange_auth
+from catalyst.finance.execution import (LimitOrder)
+
+log = Logger('test_bitfinex')
+
+
+class BitfinexTestCase(BaseExchangeTestCase):
+    @classmethod
+    def setup(self):
+        log.info('creating bitfinex object')
+        auth = get_exchange_auth('bitfinex')
+        self.exchange = Bitfinex(
+            key=auth['key'],
+            secret=auth['secret'],
+            base_currency='usd'
+        )
+
+    def test_order(self):
+        log.info('creating order')
+        asset = self.exchange.get_asset('eth_usd')
+        order_id = self.exchange.order(
+            asset=asset,
+            style=LimitOrder(limit_price=200),
+            limit_price=200,
+            amount=0.5,
+            stop_price=None
+        )
+        log.info('order created {}'.format(order_id))
+        pass
+
+    def test_open_orders(self):
+        log.info('retrieving open orders')
+        orders = self.exchange.get_open_orders()
+        pass
+
+    def test_get_order(self):
+        log.info('retrieving order')
+        pass
+
+    def test_cancel_order(self):
+        log.info('cancel order')
+        pass
+
+    def test_get_candles(self):
+        log.info('retrieving candles')
+        ohlcv_neo = self.exchange.get_candles(
+            data_frequency='1m',
+            assets=self.exchange.get_asset('neo_btc')
+        )
+        pass
+
+    def test_tickers(self):
+        log.info('retrieving tickers')
+        tickers = self.exchange.tickers([
+            self.exchange.get_asset('eth_btc'),
+            self.exchange.get_asset('etc_btc')
+        ])
+        pass
+
+    def test_get_account(self):
+        log.info('retrieving account data')
+        pass
+
+    def test_get_balances(self):
+        log.info('testing exchange balances')
+        balances = self.exchange.get_balances()
+        pass
+
+    def test_orderbook(self):
+        log.info('testing order book for bitfinex')
+        asset = self.exchange.get_asset('eth_btc')
+        orderbook = self.exchange.get_orderbook(asset)
+        pass