dual_vwap: scale volume down by 1000

.gitignore

@@ -78,7 +78,3 @@ zipline.iml
 ./data
 
 TAGS
-
-python2
-python3
-scratch

README.rst

@@ -1,3 +1,196 @@
-All the documentation for `Catalyst <https://github.com/enigmampc/catalyst>`_
+========
-can be found in the 
+Catalyst
-`documentation website <https://enigmampc.github.io/catalyst>`_.
+========
+
+|version status|
+
+Catalyst is an algorithmic trading library for crypto-assets written in Python.
+It allows trading strategies to be easily expressed and backtested against historical data, providing analytics and insights regarding a particular strategy's performance.
+Catalyst will be expanded to support live-trading of crypto-assets in the coming months.
+Please visit `<enigma.co>`_ to learn about Catalyst, or refer to the 
+`whitepaper <https://www.enigma.co/enigma_catalyst.pdf>`_ for further technical details.
+
+Catalyst builds on top of the well-established `Zipline <https://github.com/quantopian/zipline>`_ project.
+We did our best to minimize structural changes to the general API to maximize compatibility with existing trading algorithms, developer knowledge, and tutorials.
+For now, please refer to the `Zipline API Docs <http://zipline.io>`_ as a general reference and bring any other questions you have to our #dev channel on `Slack <https://join.slack.com/enigmacatalyst/shared_invite/MTkzMjQ0MTg1NTczLTE0OTY3MjE3MDEtZGZmMTI5YzI3ZA>`_.
+
+Our primary contributions include the:
+
+- Introduction of an open trading calendar that permits simulation to allow trades on weekends, holidays, and outside of normal business hours.
+- Curation of OHLCV data bundle from `Poloniex's API <https://poloniex.com/support/api/>`_, which contains data in five-minute intervals as early as 2/19/2015.
+- Support for backtesting of daily trading strategies, support for five-minute backtesting is in development.
+- Addition of Bitcoin price (USDT_BTC) as a benchmark asset for comparing performance.
+
+Interested in getting involved?
+`Join us on Slack! <https://join.slack.com/enigmacatalyst/shared_invite/MTkzMjQ0MTg1NTczLTE0OTY3MjE3MDEtZGZmMTI5YzI3ZA>`_
+
+
+Installation
+============
+
+At the moment, Catalyst has some fairly specific and strict depedency requirements.
+We recommend the use of Python virtual environments if you wish to simplify the installation process, or otherwise isolate Catalyst's dependencies from your other projects.
+If you don't have ``virtualenv`` installed, see our later section on Virtual Environments.
+
+.. code-block:: bash
+
+    $ virtualenv catalyst-venv
+    $ source ./catalyst-venv/bin/activate
+    $ pip install enigma-catalyst
+
+**Note:** A successful installation will require several minutes in order to compile dependencies that expose C APIs.
+
+Dependencies
+------------
+
+Catalyst's depedencies can be found in the ``etc/requirements.txt`` file.
+If you need to install them outside of a typical ``pip install``, this is done using:
+
+.. code-block:: bash
+
+    $ pip install -r etc/requirements.txt
+
+Though not required by Catalyst directly, our example algorithms use matplotlib to visually display backtest results.
+If you wish to run any examples or use matplotlib during development, it can be installed using:
+
+.. code-block:: bash
+
+    $ pip install matplotlib
+
+**Note:** If you plan to use matplotlib and virtualenv on Mac OS X, see our later section for additional setup instructions.
+
+Getting Started
+===============
+
+The following code implements a simple buy and hold algorithm. The full source can be found in ``catalyst/examples/buy_and_hodl.py``.
+
+.. code:: python
+
+    import numpy as np
+    
+    from catalyst.api import (
+        order_target_value,
+        symbol,
+        record,
+        cancel_order,
+        get_open_orders,
+    )
+    
+    ASSET = 'USDT_BTC'
+    
+    TARGET_HODL_RATIO = 0.8
+    RESERVE_RATIO = 1.0 - TARGET_HODL_RATIO
+
+    def initialize(context):
+        context.is_buying = True
+        context.asset = symbol(ASSET)
+
+    def handle_data(context, data):
+        cash = context.portfolio.cash
+        target_hodl_value = TARGET_HODL_RATIO * context.portfolio.starting_cash
+        reserve_value = RESERVE_RATIO * context.portfolio.starting_cash
+        
+        # Cancel any outstanding orders from the previous day
+        orders = get_open_orders(context.asset) or []
+        for order in orders:
+            cancel_order(order)
+        
+        # Stop buying after passing reserve threshold
+        if cash <= reserve_value:
+            context.is_buying = False
+        
+        # Retrieve current price from pricing data
+        price = data[context.asset].price
+        
+        # Check if still buying and could (approximately) afford another purchase                    
+        if context.is_buying and cash > price:
+            # Place order to make position in asset equal to target_hodl_value
+            order_target_value(
+                context.asset,
+                target_hodl_value,
+                limit_price=1.1 * price,
+                stop_price=0.9 * price,
+            )
+        
+        # Record any state for later analysis
+        record(
+            price=price,
+            cash=context.portfolio.cash,
+            leverage=context.account.leverage,
+        )
+
+
+You can then run this algorithm using the Catalyst CLI. From the command
+line, run:
+
+.. code:: bash
+
+    $ catalyst ingest
+    $ catalyst run -f buy_and_hodl.py --start 2015-3-1 --end 2017-6-28 --capital-base 100000 -o bah.pickle
+
+This will download the crypto-asset price data from a poloniex bundle
+curated by Enigma in the specified time range and stream it through
+the algorithm and plot the resulting performance using matplotlib.
+
+You can find other examples in the ``catalyst/examples`` directory.
+
+Limitations
+-----------
+
+This project is currently in a pre-alpha state and has some limitations we'd like to address:
+
+- *Minimum Denomination:* The smallest tradable unit in Catalyst is equal to 1/1000th of a full coin. We plan to enable more granular increments, but have capped it at 1/1000th for the time being.
+- *Supported Assets:* Currently the poloniex bundle comes prepopulated with data for all 90 registered trading pairs. However, due to limitations in how portfolios are currently modeled, we recommend sticking to ``USDT_*`` trading pairs. USDT is an independent currency listed on Poloniex whose price is pegged to the US dollar. Currently, this list includes: ``USDT_BTC``, ``USDT_DASH``, ``USDT_ETC``, ``USDT_ETH``, ``USDT_LTC``, ``USDT_NXT``, ``USDT_REP``, ``USDT_STR``, ``USDT_XMR``, ``USDT_XRP``, and ``USDT_ZEC``. We plan to add support for basing your portfolio in arbitrary currencies and provide native support for modeling ForEx trades in the near future!
+
+Virtual Environments
+====================
+
+Here we will provide a brief tutorial for installing ``virtualenv`` and its basic usage.
+For more information regarding ``virtualenv``, please refer to this `virtualenv guide <http://python-guide-pt-br.readthedocs.io/en/latest/dev/virtualenvs/>`_.
+
+The ``virtualenv`` command can be installed using:
+
+.. code-block:: bash
+
+    $ pip install virtualenv
+
+To create a new virtual environment, choose a directory, e.g. ``/path/to/venv-dir``, where project-specific packages and files will be stored.  The environment is created by running:
+
+.. code-block:: bash
+
+    $ virtualenv /path/to/venv-dir
+
+To enter an environment, run the ``bin/activate`` script located in ``/path/to/venv-dir`` using:
+
+.. code-block:: bash
+
+    $ source /path/to/venv-dir/bin/activate
+
+Exiting an environment is accomplished using ``deactivate``, and removing it entirely is done by deleting ``/path/to/venv-dir``.
+
+OS X + virtualenv + matplotlib
+-------------------------------------
+
+A note about using matplotlib in virtual enviroments on OS X: it may be necessary to run
+
+.. code-block:: python
+
+    echo "backend: TkAgg" > ~/.matplotlib/matplotlibrc
+
+in order to override the default ``macosx`` backend for your system, which may not be accessible from inside the virtual environment.
+This will allow Catalyst to open matplotlib charts from within a virtual environment, which is useful for displaying the performance of your backtests.  To learn more about matplotlib backends, please refer to the
+`matplotlib backend documentation <https://matplotlib.org/faq/usage_faq.html#what-is-a-backend>`_.
+
+Disclaimer
+==========
+
+Keep in mind that this project is still under active development, and is not recommended for production use in its current state.
+We are deeply committed to improving the overall user experience, reliability, and feature-set offered by Catalyst.
+If you have any suggestions, feedback, or general improvements regarding any of these topics, please let us know!
+
+Hello World,
+
+The Enigma Team
+
+.. |version status| image:: https://img.shields.io/pypi/pyversions/enigma-catalyst.svg
+   :target: https://pypi.python.org/pypi/enigma-catalyst

catalyst/__main__.py

@@ -8,9 +8,6 @@ import pandas as pd
 from six import text_type
 
 from catalyst.data import bundles as bundles_module
-from catalyst.exchange.exchange_bundle import ExchangeBundle
-from catalyst.exchange.exchange_utils import delete_algo_folder
-from catalyst.exchange.factory import get_exchange
 from catalyst.utils.cli import Date, Timestamp
 from catalyst.utils.run_algo import _run, load_extensions
 
@@ -31,17 +28,16 @@ except NameError:
     '--strict-extensions/--non-strict-extensions',
     is_flag=True,
     help='If --strict-extensions is passed then catalyst will not run if it'
-         ' cannot load all of the specified extensions. If this is not passed or'
+    ' cannot load all of the specified extensions. If this is not passed or'
-         ' --non-strict-extensions is passed then the failure will be logged but'
+    ' --non-strict-extensions is passed then the failure will be logged but'
-         ' execution will continue.',
+    ' execution will continue.',
 )
 @click.option(
     '--default-extension/--no-default-extension',
     is_flag=True,
     default=True,
-    help="Don't load the default catalyst extension.py file in $CATALYST_HOME.",
+    help="Don't load the default catalyst extension.py file in $ZIPLINE_HOME.",
 )
-@click.version_option()
 def main(extension, strict_extensions, default_extension):
     """Top level catalyst entry point.
     """
@@ -68,7 +64,6 @@ def extract_option_object(option):
     option_object : click.Option
         The option object that this decorator will create.
     """
-
     @option
     def opt():
         pass
@@ -100,9 +95,7 @@ def ipython_only(option):
         def _(*args, **kwargs):
             kwargs[argname] = None
             return f(*args, **kwargs)
-
         return _
-
     return d
 
 
@@ -124,13 +117,13 @@ def ipython_only(option):
     '--define',
     multiple=True,
     help="Define a name to be bound in the namespace before executing"
-         " the algotext. For example '-Dname=value'. The value may be any python"
+    " the algotext. For example '-Dname=value'. The value may be any python"
-         " expression. These are evaluated in order so they may refer to previously"
+    " expression. These are evaluated in order so they may refer to previously"
-         " defined names.",
+    " defined names.",
 )
 @click.option(
     '--data-frequency',
-    type=click.Choice({'daily', 'minute'}),
+    type=click.Choice({'daily', '5-minute', 'minute'}),
     default='daily',
     show_default=True,
     help='The data frequency of the simulation.',
@@ -156,7 +149,7 @@ def ipython_only(option):
     default=pd.Timestamp.utcnow(),
     show_default=False,
     help='The date to lookup data on or before.\n'
-         '[default: <current-time>]'
+    '[default: <current-time>]'
 )
 @click.option(
     '-s',
@@ -177,7 +170,7 @@ def ipython_only(option):
     metavar='FILENAME',
     show_default=True,
     help="The location to write the perf data. If this is '-' the perf will"
-         " be written to stdout.",
+    " be written to stdout.",
 )
 @click.option(
     '--print-algo/--no-print-algo',
@@ -191,23 +184,6 @@ def ipython_only(option):
     default=None,
     help='Should the algorithm methods be resolved in the local namespace.'
 ))
-@click.option(
-    '-x',
-    '--exchange-name',
-    type=click.Choice({'bitfinex', 'bittrex', 'poloniex'}),
-    help='The name of the targeted exchange (supported: bitfinex, bittrex, poloniex).',
-)
-@click.option(
-    '-n',
-    '--algo-namespace',
-    help='A label assigned to the algorithm for data storage purposes.'
-)
-@click.option(
-    '-c',
-    '--base-currency',
-    help='The base currency used to calculate statistics '
-         '(e.g. usd, btc, eth).',
-)
 @click.pass_context
 def run(ctx,
         algofile,
@@ -221,19 +197,9 @@ def run(ctx,
         end,
         output,
         print_algo,
-        local_namespace,
+        local_namespace):
-        exchange_name,
-        algo_namespace,
-        base_currency):
     """Run a backtest for the given algorithm.
     """
-
-    if (algotext is not None) == (algofile is not None):
-        ctx.fail(
-            "must specify exactly one of '-f' / '--algofile' or"
-            " '-t' / '--algotext'",
-        )
-
     # check that the start and end dates are passed correctly
     if start is None and end is None:
         # check both at the same time to avoid the case where a user
@@ -247,8 +213,11 @@ def run(ctx,
     if end is None:
         ctx.fail("must specify an end date with '-e' / '--end'")
 
-    if exchange_name is None:
+    if (algotext is not None) == (algofile is not None):
-        ctx.fail("must specify an exchange name '-x'")
+        ctx.fail(
+            "must specify exactly one of '-f' / '--algofile' or"
+            " '-t' / '--algotext'",
+        )
 
     perf = _run(
         initialize=None,
@@ -269,11 +238,6 @@ def run(ctx,
         print_algo=print_algo,
         local_namespace=local_namespace,
         environ=os.environ,
-        live=False,
-        exchange=exchange_name,
-        algo_namespace=algo_namespace,
-        base_currency=base_currency,
-        live_graph=False
     )
 
     if output == '-':
@@ -301,11 +265,11 @@ def catalyst_magic(line, cell=None):
                 '--algotext', cell,
                 '--output', os.devnull,  # don't write the results by default
             ] + ([
-                     # these options are set when running in line magic mode
+                # these options are set when running in line magic mode
-                     # set a non None algo text to use the ipython user_ns
+                # set a non None algo text to use the ipython user_ns
-                     '--algotext', '',
+                '--algotext', '',
-                     '--local-namespace',
+                '--local-namespace',
-                 ] if cell is None else []) + line.split(),
+            ] if cell is None else []) + line.split(),
             '%s%%catalyst' % ((cell or '') and '%'),
             # don't use system exit and propogate errors to the caller
             standalone_mode=False,
@@ -317,280 +281,15 @@ def catalyst_magic(line, cell=None):
             raise ValueError('main returned non-zero status code: %d' % e.code)
 
 
-@main.command()
-@click.option(
-    '-f',
-    '--algofile',
-    default=None,
-    type=click.File('r'),
-    help='The file that contains the algorithm to run.',
-)
-@click.option(
-    '-t',
-    '--algotext',
-    help='The algorithm script to run.',
-)
-@click.option(
-    '-D',
-    '--define',
-    multiple=True,
-    help="Define a name to be bound in the namespace before executing"
-         " the algotext. For example '-Dname=value'. The value may be any python"
-         " expression. These are evaluated in order so they may refer to previously"
-         " defined names.",
-)
-@click.option(
-    '-o',
-    '--output',
-    default='-',
-    metavar='FILENAME',
-    show_default=True,
-    help="The location to write the perf data. If this is '-' the perf will"
-         " be written to stdout.",
-)
-@click.option(
-    '--print-algo/--no-print-algo',
-    is_flag=True,
-    default=False,
-    help='Print the algorithm to stdout.',
-)
-@ipython_only(click.option(
-    '--local-namespace/--no-local-namespace',
-    is_flag=True,
-    default=None,
-    help='Should the algorithm methods be resolved in the local namespace.'
-))
-@click.option(
-    '-x',
-    '--exchange-name',
-    type=click.Choice({'bitfinex', 'bittrex', 'poloniex'}),
-    help='The name of the targeted exchange (supported: bitfinex, bittrex, poloniex).',
-)
-@click.option(
-    '-n',
-    '--algo-namespace',
-    help='A label assigned to the algorithm for data storage purposes.'
-)
-@click.option(
-    '-c',
-    '--base-currency',
-    help='The base currency used to calculate statistics '
-         '(e.g. usd, btc, eth).',
-)
-@click.option(
-    '--live-graph/--no-live-graph',
-    is_flag=True,
-    default=False,
-    help='Display live graph.',
-)
-@click.pass_context
-def live(ctx,
-         algofile,
-         algotext,
-         define,
-         output,
-         print_algo,
-         local_namespace,
-         exchange_name,
-         algo_namespace,
-         base_currency,
-         live_graph):
-    """Trade live with the given algorithm.
-    """
-    if (algotext is not None) == (algofile is not None):
-        ctx.fail(
-            "must specify exactly one of '-f' / '--algofile' or"
-            " '-t' / '--algotext'",
-        )
-
-    if exchange_name is None:
-        ctx.fail("must specify an exchange name '-x'")
-    if algo_namespace is None:
-        ctx.fail("must specify an algorithm name '-n' in live execution mode")
-    if base_currency is None:
-        ctx.fail("must specify a base currency '-c' in live execution mode")
-
-    perf = _run(
-        initialize=None,
-        handle_data=None,
-        before_trading_start=None,
-        analyze=None,
-        algofile=algofile,
-        algotext=algotext,
-        defines=define,
-        data_frequency=None,
-        capital_base=None,
-        data=None,
-        bundle=None,
-        bundle_timestamp=None,
-        start=None,
-        end=None,
-        output=output,
-        print_algo=print_algo,
-        local_namespace=local_namespace,
-        environ=os.environ,
-        live=True,
-        exchange=exchange_name,
-        algo_namespace=algo_namespace,
-        base_currency=base_currency,
-        live_graph=live_graph
-    )
-
-    if output == '-':
-        click.echo(str(perf))
-    elif output != os.devnull:  # make the catalyst magic not write any data
-        perf.to_pickle(output)
-
-    return perf
-
-
-@main.command(name='ingest-exchange')
-@click.option(
-    '-x',
-    '--exchange-name',
-    type=click.Choice({'bitfinex', 'bittrex', 'poloniex'}),
-    help='The name of the exchange bundle to ingest (supported: bitfinex,'
-         ' bittrex, poloniex).',
-)
-@click.option(
-    '-f',
-    '--data-frequency',
-    type=click.Choice({'daily', 'minute', 'daily,minute', 'minute,daily'}),
-    default='daily',
-    show_default=True,
-    help='The data frequency of the desired OHLCV bars.',
-)
-@click.option(
-    '-s',
-    '--start',
-    default=None,
-    type=Date(tz='utc', as_timestamp=True),
-    help='The start date of the data range. (default: one year from end date)',
-)
-@click.option(
-    '-e',
-    '--end',
-    default=None,
-    type=Date(tz='utc', as_timestamp=True),
-    help='The end date of the data range. (default: today)',
-)
-@click.option(
-    '-i',
-    '--include-symbols',
-    default=None,
-    help='A list of symbols to ingest (optional comma separated list)',
-)
-@click.option(
-    '--exclude-symbols',
-    default=None,
-    help='A list of symbols to exclude from the ingestion '
-         '(optional comma separated list)',
-)
-@click.option(
-    '--show-progress/--no-show-progress',
-    default=True,
-    help='Print progress information to the terminal.'
-)
-@click.option(
-    '--verbose/--no-verbose`',
-    default=False,
-    help='Show a progress indicator for every currency pair.'
-)
-@click.option(
-    '--validate/--no-validate`',
-    default=False,
-    help='Report potential anomalies found in data bundles.'
-)
-def ingest_exchange(exchange_name, data_frequency, start, end,
-                    include_symbols, exclude_symbols, show_progress, verbose,
-                    validate):
-    """
-    Ingest data for the given exchange.
-    """
-
-    if exchange_name is None:
-        ctx.fail("must specify an exchange name '-x'")
-
-    exchange = get_exchange(exchange_name)
-    exchange_bundle = ExchangeBundle(exchange)
-
-    click.echo('Ingesting exchange bundle {}...'.format(exchange_name))
-    exchange_bundle.ingest(
-        data_frequency=data_frequency,
-        include_symbols=include_symbols,
-        exclude_symbols=exclude_symbols,
-        start=start,
-        end=end,
-        show_progress=show_progress,
-        show_breakdown=verbose,
-        show_report=validate
-    )
-
-
-@main.command(name='clean-algo')
-@click.option(
-    '-n',
-    '--algo-namespace',
-    help='The label of the algorithm to for which to clean the state.'
-)
-@click.pass_context
-def clean_algo(ctx, algo_namespace):
-    click.echo(
-        'Deleting the state folder of algo: {}...'.format(algo_namespace)
-    )
-    delete_algo_folder(algo_namespace)
-
-
-@main.command(name='clean-exchange')
-@click.option(
-    '-x',
-    '--exchange-name',
-    type=click.Choice({'bitfinex', 'bittrex', 'poloniex'}),
-    help='The name of the exchange bundle to ingest (supported: bitfinex,'
-         ' bittrex, poloniex).',
-)
-@click.option(
-    '-f',
-    '--data-frequency',
-    type=click.Choice({'daily', 'minute'}),
-    default=None,
-    help='The bundle data frequency to remove. If not specified, it will '
-         'remove both daily and minute bundles.',
-)
-@click.pass_context
-def clean_exchange(ctx, exchange_name, data_frequency):
-    """Clean up bundles from 'ingest-exchange'.
-    """
-
-    if exchange_name is None:
-        ctx.fail("must specify an exchange name '-x'")
-
-    exchange = get_exchange(exchange_name)
-    exchange_bundle = ExchangeBundle(exchange)
-
-    click.echo('Cleaning exchange bundle {}...'.format(exchange_name))
-    exchange_bundle.clean(
-        data_frequency=data_frequency,
-    )
-    click.echo('Done')
-
-
 @main.command()
 @click.option(
     '-b',
     '--bundle',
+    default='poloniex',
     metavar='BUNDLE-NAME',
-    default=None,
+    show_default=True,
-    show_default=False,
     help='The data bundle to ingest.',
 )
-@click.option(
-    '-x',
-    '--exchange-name',
-    type=click.Choice({'bitfinex', 'bittrex', 'poloniex'}),
-    help='The name of the exchange bundle to ingest (supported: bitfinex,'
-         ' bittrex, poloniex).',
-)
 @click.option(
     '-c',
     '--compile-locally',
@@ -609,12 +308,9 @@ def clean_exchange(ctx, exchange_name, data_frequency):
     default=True,
     help='Print progress information to the terminal.'
 )
-@click.pass_context
+def ingest(bundle, compile_locally, assets_version, show_progress):
-def ingest(ctx, bundle, exchange_name, compile_locally, assets_version,
-           show_progress):
     """Ingest the data for the given bundle.
     """
-
     bundles_module.ingest(
         bundle,
         os.environ,
@@ -634,26 +330,19 @@ def ingest(ctx, bundle, exchange_name, compile_locally, assets_version,
     show_default=True,
     help='The data bundle to clean.',
 )
-@click.option(
-    '-x',
-    '--exchange_name',
-    metavar='EXCHANGE-NAME',
-    show_default=True,
-    help='The exchange bundle name to clean.',
-)
 @click.option(
     '-e',
     '--before',
     type=Timestamp(),
     help='Clear all data before TIMESTAMP.'
-         ' This may not be passed with -k / --keep-last',
+    ' This may not be passed with -k / --keep-last',
 )
 @click.option(
     '-a',
     '--after',
     type=Timestamp(),
     help='Clear all data after TIMESTAMP'
-         ' This may not be passed with -k / --keep-last',
+    ' This may not be passed with -k / --keep-last',
 )
 @click.option(
     '-k',
@@ -661,10 +350,10 @@ def ingest(ctx, bundle, exchange_name, compile_locally, assets_version,
     type=int,
     metavar='N',
     help='Clear all but the last N downloads.'
-         ' This may not be passed with -e / --before or -a / --after',
+    ' This may not be passed with -e / --before or -a / --after',
 )
 def clean(bundle, before, after, keep_last):
-    """Clean up bundles from 'ingest'.
+    """Clean up data downloaded with the ingest command.
     """
     bundles_module.clean(
         bundle,

catalyst/algorithm.py

@@ -125,7 +125,6 @@ from catalyst.utils.factory import create_simulation_parameters
 from catalyst.utils.math_utils import (
     tolerant_equals,
     round_if_near_integer,
-    round_nearest
 )
 from catalyst.utils.pandas_utils import clear_dataframe_indexer_caches
 from catalyst.utils.preprocess import preprocess
@@ -134,13 +133,15 @@ from catalyst.utils.security_list import SecurityList
 import catalyst.protocol
 from catalyst.sources.requests_csv import PandasRequestsCSV
 
-from catalyst.gens.sim_engine import MinuteSimulationClock
+from catalyst.gens.sim_engine import (
+    MinuteSimulationClock,
+    FiveMinuteSimulationClock,
+)
 from catalyst.sources.benchmark_source import BenchmarkSource
 from catalyst.catalyst_warnings import ZiplineDeprecationWarning
 
-from catalyst.constants import LOG_LEVEL
 
-log = logbook.Logger("CatalystLog", level=LOG_LEVEL)
+log = logbook.Logger("ZiplineLog")
 
 
 class TradingAlgorithm(object):
@@ -172,7 +173,7 @@ class TradingAlgorithm(object):
     algo_filename : str, optional
         The filename for the algoscript. This will be used in exception
         tracebacks. default: '<string>'.
-    data_frequency : {'daily', 'minute'}, optional
+    data_frequency : {'daily', '5-minute', 'minute'}, optional
         The duration of the bars.
     instant_fill : bool, optional
         Whether to fill orders immediately or on next bar. default: False
@@ -225,7 +226,7 @@ class TradingAlgorithm(object):
             script : str
                 Algoscript that contains initialize and
                 handle_data function definition.
-            data_frequency : {'daily', 'minute'}
+            data_frequency : {'daily', '5-minute', 'minute'}
                The duration of the bars.
             capital_base : float <default: 1.0e5>
                How much capital to start with.
@@ -433,6 +434,8 @@ class TradingAlgorithm(object):
         if get_loader is not None:
             if data_frequency == 'daily':
                 all_dates = self.trading_calendar.all_sessions
+            elif data_frequency == '5-minute':
+                all_dates = self.trading_calendar.all_five_minutes
             elif data_frequency == 'minute':
                 all_dates = self.trading_calendar.all_minutes
             else:
@@ -464,7 +467,7 @@ class TradingAlgorithm(object):
         self._in_before_trading_start = True
 
         with handle_non_market_minutes(data) if \
-                self.data_frequency == 'minute' else ExitStack():
+                self.data_frequency in ('minute', '5-minute') else ExitStack():
             self._before_trading_start(self, data)
 
         self._in_before_trading_start = False
@@ -520,10 +523,11 @@ class TradingAlgorithm(object):
         market_closes = trading_o_and_c['market_close']
         minutely_emission = False
 
-        if self.sim_params.data_frequency == 'minute':
+        if self.sim_params.data_frequency in set(('minute', '5-minute')):
             market_opens = trading_o_and_c['market_open']
 
-            minutely_emission = self.sim_params.emission_rate == 'minute'
+            minutely_emission = self.sim_params.emission_rate in \
+                set(('minute', '5-minute'))
         else:
             # in daily mode, we want to have one bar per session, timestamped
             # as the last minute of the session.
@@ -547,6 +551,15 @@ class TradingAlgorithm(object):
             'UTC',
         )
 
+        if self.sim_params.data_frequency == '5-minute':
+            return FiveMinuteSimulationClock(
+                self.sim_params.sessions,
+                execution_opens,
+                execution_closes,
+                before_trading_start_minutes,
+                minute_emission=minutely_emission,
+            )
+
         return MinuteSimulationClock(
             self.sim_params.sessions,
             execution_opens,
@@ -678,6 +691,8 @@ class TradingAlgorithm(object):
                     time_count = times.nunique()
                     if time_count == 1:
                         self.sim_params.data_frequency = 'daily'
+                    elif time_count == 288:
+                        self.sim_params.data_frequency = '5-minute'
                     else:
                         self.sim_params.data_frequency = 'minute'
 
@@ -699,6 +714,8 @@ class TradingAlgorithm(object):
 
                 if self.sim_params.data_frequency == 'daily':
                     equity_reader_arg = 'equity_daily_reader'
+                elif self.sim_params.data_frequency == '5-minute':
+                    equity_daily_reader = 'equity_5_minute_reader'
                 elif self.sim_params.data_frequency == 'minute':
                     equity_reader_arg = 'equity_minute_reader'
                 equity_reader = PanelBarReader(
@@ -725,14 +742,15 @@ class TradingAlgorithm(object):
             for perf in self.get_generator():
                 perfs.append(perf)
 
+          
             # convert perf dict to pandas dataframe
-            daily_stats = self._create_daily_stats(perfs)
+            stats = self._create_daily_stats(perfs)
 
-            self.analyze(daily_stats)
+            self.analyze(stats)
         finally:
             self.data_portal = None
 
-        return daily_stats
+        return stats
 
     def _write_and_map_id_index_to_sids(self, identifiers, as_of_date):
         # Build new Assets for identifiers that can't be resolved as
@@ -942,9 +960,9 @@ class TradingAlgorithm(object):
                   The arena from the simulation parameters. This will normally
                   be ``'backtest'`` but some systems may use this distinguish
                   live trading from backtesting.
-              data_frequency : {'daily', 'minute'}
+              data_frequency : {'daily', '5-minute', 'minute'}
                   data_frequency tells the algorithm if it is running with
-                  daily or minute mode.
+                  daily, minute, or five-minute mode.
               start : datetime
                   The start date for the simulation.
               end : datetime
@@ -1118,12 +1136,17 @@ class TradingAlgorithm(object):
                           'date_rule. You should use keyword argument '
                           'time_rule= when calling schedule_function without '
                           'specifying a date_rule', stacklevel=3)
+        
+        freq = self.sim_params.data_frequency
 
         date_rule = date_rule or date_rules.every_day()
-        time_rule = ((time_rule or time_rules.every_minute())
+        if freq is 'daily':
-                     if self.sim_params.data_frequency == 'minute' else
+            # Ignore any time rules in daily mode.
-                     # If we are in daily mode the time_rule is ignored.
+            # every_minute in daily mode does nothing.
-                     time_rules.every_minute())
+            time_rule = time_rules.every_minute()
+        else:
+            # use provided time rule or default to every minute
+            time_rule = time_rule or time_rules.every_minute()
 
         # Check the type of the algorithm's schedule before pulling calendar
         # Note that the ExchangeTradingSchedule is currently the only
@@ -1147,7 +1170,13 @@ class TradingAlgorithm(object):
             )
 
         self.add_event(
-            make_eventrule(date_rule, time_rule, cal, half_days),
+            make_eventrule(
+                date_rule,
+                time_rule,
+                cal,
+                half_days=half_days,
+                data_frequency=self.data_frequency,
+            ),
             func,
         )
 
@@ -1464,7 +1493,7 @@ class TradingAlgorithm(object):
 
     def _calculate_order(self, asset, amount,
                          limit_price=None, stop_price=None, style=None):
-        amount = self.round_order(amount, asset)
+        amount = self.round_order(amount)
 
         # Raises a ZiplineError if invalid parameters are detected.
         self.validate_order_params(asset,
@@ -1481,13 +1510,16 @@ class TradingAlgorithm(object):
         return amount, style
 
     @staticmethod
-    def round_order(amount, asset):
+    def round_order(amount):
         """
-        Converts the number of shares to the smallest tradable lot size for
+        Convert number of shares to an integer.
-        the asset being ordered.
+
-        
+        By default, truncates to the integer share count that's either within
+        .0001 of amount or closer to zero.
+
+        E.g. 3.9999 -> 4.0; 5.5 -> 5.0; -5.5 -> -5.0
         """
-        return round_nearest(amount, asset.min_trade_size)
+        return int(round_if_near_integer(amount))
 
     def validate_order_params(self,
                               asset,
@@ -1523,6 +1555,7 @@ class TradingAlgorithm(object):
                              self.updated_portfolio(),
                              self.get_datetime(),
                              self.trading_client.current_data)
+
     @staticmethod
     def __convert_order_params_for_blotter(limit_price, stop_price, style):
         """
@@ -1675,12 +1708,12 @@ class TradingAlgorithm(object):
         return dt
 
     @api_method
-    def set_slippage(self, us_equities=None, us_futures=None):
+    def set_slippage(self, equities=None, us_futures=None):
         """Set the slippage models for the simulation.
 
         Parameters
         ----------
-        us_equities : EquitySlippageModel
+        equities : EquitySlippageModel
             The slippage model to use for trading US equities.
         us_futures : FutureSlippageModel
             The slippage model to use for trading US futures.
@@ -1692,14 +1725,14 @@ class TradingAlgorithm(object):
         if self.initialized:
             raise SetSlippagePostInit()
 
-        if us_equities is not None:
+        if equities is not None:
-            if Equity not in us_equities.allowed_asset_types:
+            if Equity not in equities.allowed_asset_types:
                 raise IncompatibleSlippageModel(
                     asset_type='equities',
-                    given_model=us_equities,
+                    given_model=equities,
-                    supported_asset_types=us_equities.allowed_asset_types,
+                    supported_asset_types=equities.allowed_asset_types,
                 )
-            self.blotter.slippage_models[Equity] = us_equities
+            self.blotter.slippage_models[Equity] = equities
 
         if us_futures is not None:
             if Future not in us_futures.allowed_asset_types:
@@ -1711,12 +1744,12 @@ class TradingAlgorithm(object):
             self.blotter.slippage_models[Future] = us_futures
 
     @api_method
-    def set_commission(self, us_equities=None, us_futures=None):
+    def set_commission(self, equities=None, us_futures=None):
         """Sets the commission models for the simulation.
 
         Parameters
         ----------
-        us_equities : EquityCommissionModel
+        equities : EquityCommissionModel
             The commission model to use for trading US equities.
         us_futures : FutureCommissionModel
             The commission model to use for trading US futures.
@@ -1730,14 +1763,14 @@ class TradingAlgorithm(object):
         if self.initialized:
             raise SetCommissionPostInit()
 
-        if us_equities is not None:
+        if equities is not None:
-            if Equity not in us_equities.allowed_asset_types:
+            if Equity not in equities.allowed_asset_types:
                 raise IncompatibleCommissionModel(
                     asset_type='equities',
-                    given_model=us_equities,
+                    given_model=equities,
-                    supported_asset_types=us_equities.allowed_asset_types,
+                    supported_asset_types=equities.allowed_asset_types,
                 )
-            self.blotter.commission_models[Equity] = us_equities
+            self.blotter.commission_models[Equity] = equities
 
         if us_futures is not None:
             if Future not in us_futures.allowed_asset_types:
@@ -1794,7 +1827,7 @@ class TradingAlgorithm(object):
 
     @data_frequency.setter
     def data_frequency(self, value):
-        assert value in ('daily', 'minute')
+        assert value in ('daily', '5-minute', 'minute')
         self.sim_params.data_frequency = value
 
     @api_method

catalyst/assets/_assets.pyx

@@ -17,36 +17,34 @@
 """
 Cythonized Asset object.
 """
-import hashlib
-
 cimport cython
 from cpython.number cimport PyNumber_Index
 from cpython.object cimport (
-Py_EQ,
+    Py_EQ,
-Py_NE,
+    Py_NE,
-Py_GE,
+    Py_GE,
-Py_LE,
+    Py_LE,
-Py_GT,
+    Py_GT,
-Py_LT,
+    Py_LT,
 )
 from cpython cimport bool
 
-import pandas as pd
-from datetime import timedelta
 import numpy as np
 from numpy cimport int64_t
 import warnings
 cimport numpy as np
 
 from catalyst.utils.calendars import get_calendar
-from catalyst.exchange.exchange_errors import InvalidSymbolError, SidHashError
+
 
 # IMPORTANT NOTE: You must change this template if you change
 # Asset.__reduce__, or else we'll attempt to unpickle an old version of this
 # class
 CACHE_FILE_TEMPLATE = '/tmp/.%s-%s.v7.cache'
 
+
 cdef class Asset:
+
     cdef readonly int sid
     # Cached hash of self.sid
     cdef int sid_hash
@@ -61,7 +59,6 @@ cdef class Asset:
 
     cdef readonly object exchange
     cdef readonly object exchange_full
-    cdef readonly object min_trade_size
 
     _kwargnames = frozenset({
         'sid',
@@ -73,20 +70,18 @@ cdef class Asset:
         'auto_close_date',
         'exchange',
         'exchange_full',
-        'min_trade_size',
     })
 
     def __init__(self,
-                 int sid,  # sid is required
+                 int sid, # sid is required
-                 object exchange,  # exchange is required
+                 object exchange, # exchange is required
                  object symbol="",
                  object asset_name="",
                  object start_date=None,
                  object end_date=None,
                  object first_traded=None,
                  object auto_close_date=None,
-                 object exchange_full=None,
+                 object exchange_full=None):
-                 object min_trade_size=None):
 
         self.sid = sid
         self.sid_hash = hash(sid)
@@ -99,7 +94,6 @@ cdef class Asset:
         self.end_date = end_date
         self.first_traded = first_traded
         self.auto_close_date = auto_close_date
-        self.min_trade_size = min_trade_size
 
     def __int__(self):
         return self.sid
@@ -154,8 +148,7 @@ cdef class Asset:
 
     def __repr__(self):
         attrs = ('symbol', 'asset_name', 'exchange',
-                 'start_date', 'end_date', 'first_traded', 'auto_close_date',
+                 'start_date', 'end_date', 'first_traded', 'auto_close_date')
-                 'min_trade_size')
         tuples = ((attr, repr(getattr(self, attr, None)))
                   for attr in attrs)
         strings = ('%s=%s' % (t[0], t[1]) for t in tuples)
@@ -177,8 +170,7 @@ cdef class Asset:
                                  self.end_date,
                                  self.first_traded,
                                  self.auto_close_date,
-                                 self.exchange_full,
+                                 self.exchange_full))
-                                 self.min_trade_size))
 
     cpdef to_dict(self):
         """
@@ -194,7 +186,6 @@ cdef class Asset:
             'auto_close_date': self.auto_close_date,
             'exchange': self.exchange,
             'exchange_full': self.exchange_full,
-            'min_trade_size': self.min_trade_size
         }
 
     @classmethod
@@ -239,11 +230,13 @@ cdef class Asset:
         calendar = get_calendar(self.exchange)
         return calendar.is_open_on_minute(dt_minute)
 
+
 cdef class Equity(Asset):
+
     def __repr__(self):
         attrs = ('symbol', 'asset_name', 'exchange',
                  'start_date', 'end_date', 'first_traded', 'auto_close_date',
-                 'exchange_full', 'min_trade_size')
+                 'exchange_full')
         tuples = ((attr, repr(getattr(self, attr, None)))
                   for attr in attrs)
         strings = ('%s=%s' % (t[0], t[1]) for t in tuples)
@@ -257,8 +250,8 @@ cdef class Equity(Asset):
         """
         def __get__(self):
             warnings.warn("The security_start_date property will soon be "
-                          "retired. Please use the start_date property instead.",
+            "retired. Please use the start_date property instead.",
-                          DeprecationWarning)
+            DeprecationWarning)
             return self.start_date
 
     property security_end_date:
@@ -268,8 +261,8 @@ cdef class Equity(Asset):
         """
         def __get__(self):
             warnings.warn("The security_end_date property will soon be "
-                          "retired. Please use the end_date property instead.",
+            "retired. Please use the end_date property instead.",
-                          DeprecationWarning)
+            DeprecationWarning)
             return self.end_date
 
     property security_name:
@@ -279,11 +272,13 @@ cdef class Equity(Asset):
         """
         def __get__(self):
             warnings.warn("The security_name property will soon be "
-                          "retired. Please use the asset_name property instead.",
+            "retired. Please use the asset_name property instead.",
-                          DeprecationWarning)
+            DeprecationWarning)
             return self.asset_name
 
+
 cdef class Future(Asset):
+
     cdef readonly object root_symbol
     cdef readonly object notice_date
     cdef readonly object expiration_date
@@ -308,8 +303,8 @@ cdef class Future(Asset):
     })
 
     def __init__(self,
-                 int sid,  # sid is required
+                 int sid, # sid is required
-                 object exchange,  # exchange is required
+                 object exchange, # exchange is required
                  object symbol="",
                  object root_symbol="",
                  object asset_name="",
@@ -393,204 +388,6 @@ cdef class Future(Asset):
         super_dict['multiplier'] = self.multiplier
         return super_dict
 
-cdef class TradingPair(Asset):
-    cdef readonly float leverage
-    cdef readonly object market_currency
-    cdef readonly object base_currency
-    cdef readonly object end_daily
-    cdef readonly object end_minute
-    cdef readonly object exchange_symbol
-
-    _kwargnames = frozenset({
-        'sid',
-        'symbol',
-        'asset_name',
-        'start_date',
-        'end_date',
-        'first_traded',
-        'auto_close_date',
-        'exchange',
-        'exchange_full',
-        'leverage',
-        'market_currency',
-        'base_currency',
-        'end_daily',
-        'end_minute',
-        'exchange_symbol',
-        'min_trade_size'
-    })
-    def __init__(self,
-                 object symbol,
-                 object exchange,
-                 object start_date=None,
-                 object asset_name=None,
-                 int sid=0,
-                 float leverage=1.0,
-                 object end_daily=None,
-                 object end_minute=None,
-                 object end_date=None,
-                 object exchange_symbol=None,
-                 object first_traded=None,
-                 object auto_close_date=None,
-                 object exchange_full=None,
-                 object min_trade_size=None):
-        """
-        Replicates the Asset constructor with some built-in conventions
-        and a new 'leverage' attribute.
-
-        Symbol
-        ------
-        Catalyst defines its own set of "universal" symbols to reference
-        trading pairs across exchanges. This is required because exchanges
-        are not adhering to a universal symbolism. For example, Bitfinex
-        uses the BTC symbol for Bitcon while Kraken uses XBT. In addition,
-        pairs are sometimes presented differently. For example, Bitfinex
-        puts the market currency before the base currency without a
-        separator, Bittrex puts the base currency first and uses a dash
-        seperator.
-
-        Here is the Catalyst convention: [Market Currency]_[Base Currency]
-        For example: btc_usd, eth_btc, neo_eth, ltc_eur.
-
-        The symbol for each currency (e.g. btc, eth, ltc) is generally
-        aligned with the Bittrex exchange.
-
-        Sid
-        ---
-        The sid of each asset is calculated based on a numeric hash of the
-        universal symbol. This simple approach avoids maintaining a mapping
-        of sids.
-
-        Leverage
-        --------
-        In contrast with equities, crypto exchanges generally assign
-        leverage values to specific trading pairs. Pairs with the
-        highest volume and market cap generally benefit from high leverage.
-        New currencies from ICO generally cannot be leveraged.
-
-        The leverage value is either None or and integer.
-
-        Leverage allows you to open a larger position with a smaller amount
-        of funds. For example, if you open a $5,000 position in BTC/USD
-        with 5:1 leverage, only one-fifth of this amount, or $1000, will be
-        tied to the position from your balance. Your remaining balance will
-        be available for opening more positions. If you open this same
-        position with 2:1 leverage, $2,500 of your balance will be tied to
-        the position. If you open with 1:1 leverage, $5,000 of your balance
-        will be tied to the position.
-
-        :param symbol:
-        :param exchange:
-        :param start_date:
-        :param asset_name:
-        :param sid:
-        :param leverage:
-        :param end_daily
-        :param end_minute
-        :param end_date:
-        :param exchange_symbol:
-        :param first_traded:
-        :param auto_close_date:
-        :param exchange_full:
-        :param min_trade_size:
-        """
-
-        symbol = symbol.lower()
-        try:
-            self.market_currency, self.base_currency = symbol.split('_')
-        except Exception as e:
-            raise InvalidSymbolError(symbol=symbol, error=e)
-
-        if sid == 0 or sid is None:
-            try:
-                # sid = abs(hash(symbol)) % (10 ** 4)
-                # TODO: try to encode the symbol in the main scope
-                sid = int(
-                    hashlib.sha256(symbol.encode('utf-8')).hexdigest(), 16
-                ) % 10 ** 6
-            except Exception as e:
-                raise SidHashError(symbol=symbol)
-
-        if asset_name is None:
-            asset_name = ' / '.join(symbol.split('_')).upper()
-
-        if start_date is None:
-            start_date = pd.Timestamp.utcnow()
-
-        if end_date is None:
-            end_date = pd.Timestamp.utcnow() + timedelta(days=365)
-
-        super().__init__(
-            sid,
-            exchange,
-            symbol=symbol,
-            asset_name=asset_name,
-            start_date=start_date,
-            end_date=end_date,
-            first_traded=first_traded,
-            auto_close_date=auto_close_date,
-            exchange_full=exchange_full,
-            min_trade_size=min_trade_size
-        )
-
-        self.leverage = leverage
-        self.end_daily = end_daily
-        self.end_minute = end_minute
-        self.exchange_symbol = exchange_symbol
-
-    def __repr__(self):
-        return 'Trading Pair {symbol}({sid}) Exchange: {exchange}, ' \
-               'Introduced On: {start_date}, ' \
-               'Market Currency: {market_currency}, ' \
-               'Base Currency: {base_currency}, ' \
-               'Exchange Leverage: {leverage}, ' \
-               'Minimum Trade Size: {min_trade_size} ' \
-               'Last daily ingestion: {end_daily} ' \
-               'Last minutely ingestion: {end_minute}'.format(
-            symbol=self.symbol,
-            sid=self.sid,
-            exchange=self.exchange,
-            start_date=self.start_date,
-            market_currency=self.market_currency,
-            base_currency=self.base_currency,
-            leverage=self.leverage,
-            min_trade_size=self.min_trade_size,
-            end_daily=self.end_daily,
-            end_minute=self.end_minute
-        )
-
-    def is_exchange_open(self, dt_minute):
-        """
-        Parameters
-        ----------
-        dt_minute: pd.Timestamp (UTC, tz-aware)
-            The minute to check.
-
-        Returns
-        -------
-        boolean: whether the asset's exchange is open at the given minute.
-        """
-        #TODO: consider implementing to spot holds
-        return True
-
-    cpdef __reduce__(self):
-        """
-        Function used by pickle to determine how to serialize/deserialize this
-        class.  Should return a tuple whose first element is self.__class__,
-        and whose second element is a tuple of all the attributes that should
-        be serialized/deserialized during pickling.
-        """
-        return (self.__class__, (self.symbol,
-                                 self.exchange,
-                                 self.start_date,
-                                 self.asset_name,
-                                 self.sid,
-                                 self.leverage,
-                                 self.end_date,
-                                 self.first_traded,
-                                 self.auto_close_date,
-                                 self.exchange_full,
-                                 self.min_trade_size))
 
 def make_asset_array(int size, Asset asset):
     cdef np.ndarray out = np.empty([size], dtype=object)

catalyst/assets/asset_db_schema.py

@@ -39,8 +39,7 @@ equities = sa.Table(
     sa.Column('first_traded', sa.Integer),
     sa.Column('auto_close_date', sa.Integer),
     sa.Column('exchange', sa.Text),
-    sa.Column('exchange_full', sa.Text),
+    sa.Column('exchange_full', sa.Text)
-    sa.Column('min_trade_size', sa.Float)
 )
 
 equity_symbol_mappings = sa.Table(

catalyst/assets/asset_writer.py

@@ -73,7 +73,6 @@ _equities_defaults = {
     'exchange': None,
     # optional, something like "New York Stock Exchange"
     'exchange_full': None,
-    'min_trade_size': 1
 }
 
 # Default values for the futures DataFrame
@@ -391,8 +390,6 @@ class AssetDBWriter(object):
                   The date on which to close any positions in this asset.
               exchange : str
                   The exchange where this asset is traded.
-              min_trade_size: float, optional
-                  The minimum denomination this asset can be traded.
 
             The index of this dataframe should contain the sids.
         futures : pd.DataFrame, optional

catalyst/assets/assets.py

@@ -76,9 +76,7 @@ from catalyst.utils.numpy_utils import as_column
 from catalyst.utils.preprocess import preprocess
 from catalyst.utils.sqlite_utils import group_into_chunks, coerce_string_to_eng
 
-from catalyst.constants import LOG_LEVEL
+log = Logger('assets.py')
-
-log = Logger('assets.py', level=LOG_LEVEL)
 
 # A set of fields that need to be converted to strings before building an
 # Asset to avoid unicode fields

catalyst/constants.py

@@ -1,9 +0,0 @@
-# -*- coding: utf-8 -*-
-
-import logbook
-
-LOG_LEVEL = logbook.INFO
-
-DATE_TIME_FORMAT = '%Y-%m-%d %H:%M'
-
-AUTO_INGEST = False

catalyst/curate/poloniex.py

@@ -1,22 +1,22 @@
 import json, time, csv
 from datetime import datetime
 import pandas as pd
-import os, time, shutil, requests, logbook
+import os
-from catalyst.exchange.exchange_utils import get_exchange_symbols_filename
+import time
+import requests
+import logbook
 
-
+DT_START        = time.mktime(datetime(2010, 1, 1, 0, 0).timetuple())
-DT_START        = int(time.mktime(datetime(2010, 1, 1, 0, 0).timetuple()))
+CSV_OUT_FOLDER  = '/var/tmp/catalyst/data/poloniex/'
-DT_END          = pd.to_datetime('today').value // 10 ** 9
-CSV_OUT_FOLDER  = os.environ.get('CSV_OUT_FOLDER', '/efs/exchanges/poloniex/')
 CONN_RETRIES    = 2
 
 logbook.StderrHandler().push_application()
 log = logbook.Logger(__name__)
 
 class PoloniexCurator(object):
-    '''
+    """
     OHLCV data feed generator for crypto data. Based on Poloniex market data
-    '''
+    """
 
     _api_path       = 'https://poloniex.com/public?'
     currency_pairs  = []
@@ -26,15 +26,10 @@ class PoloniexCurator(object):
             try:
                 os.makedirs(CSV_OUT_FOLDER)
             except Exception as e:
-                log.error('Failed to create data folder: {}'.format(
+                log.error('Failed to create data folder: %s' % CSV_OUT_FOLDER)
-                            CSV_OUT_FOLDER))
                 log.exception(e)
 
-
     def get_currency_pairs(self):
-        '''
-        Retrieves and returns all currency pairs from the exchange 
-        '''
         url = self._api_path + 'command=returnTicker'
 
         try:
@@ -50,327 +45,100 @@ class PoloniexCurator(object):
             self.currency_pairs.append(ticker)
         self.currency_pairs.sort()
 
-        log.debug('Currency pairs retrieved successfully: {}'.format(
+        log.debug('Currency pairs retrieved successfully: %d' % (len(self.currency_pairs)))
-                    len(self.currency_pairs)
-                    ))
 
-
+    def _get_start_date(self, csv_fn):
-
+        ''' Function returns latest appended date, if the file has been previously written
-    def _retrieve_tradeID_date(self, row):
+            the last line is an empty one, so we have to read the second to last line 
-        '''
-        Helper function that reads tradeID and date fields from CSV readline
-        '''
-        tId = int(row.split(',')[0])
-        d   = pd.to_datetime(row.split(',')[1], 
-                             infer_datetime_format=True).value // 10 ** 9
-        return tId, d
-
-
-    def retrieve_trade_history(self, currencyPair, start=DT_START, 
-                               end=DT_END, temp=None):
-        '''
-        Retrieves TradeHistory from exchange for a given currencyPair 
-        between start and end dates. If no start date is provided, uses 
-        a system-wide one (beginning of time for cryptotrading).
-        If no end date is provided, 'now' is used.
-
-        Stores results in CSV file on disk.
-        
-        This function is called recursively to work around the 
-        limitations imposed by the provider API.
-        '''
-        csv_fn = CSV_OUT_FOLDER + 'crypto_trades-' + currencyPair + '.csv'
-
-        '''
-        Check what data we already have on disk, reading first and last 
-        lines from file. Data is stored on file from NEWEST to OLDEST.
         '''
         try:
             with open(csv_fn, 'ab+') as f: 
-                f.seek(0, os.SEEK_END)
+                f.seek(0, os.SEEK_END)              # First check file is not zero size
-                if(f.tell() > 2):                   # Check file size is not 0
+                if(f.tell() > 2):
-                    f.seek(0)                       # Go to start to read 
+                    f.seek(-2, os.SEEK_END)         # Jump to the second last byte.
-                    last_tradeID, end_file = self._retrieve_tradeID_date(f.readline())
-                    f.seek(-2, os.SEEK_END)         # Jump to the 2nd last byte
                     while f.read(1) != b"\n":       # Until EOL is found...
                         f.seek(-2, os.SEEK_CUR)     # ...jump back the read byte plus one more.
-                    first_tradeID, start_file = self._retrieve_tradeID_date(f.readline())
+                    lastrow = f.readline() 
-
+                    return int(lastrow.split(',')[0]) + 300
-                    if( end_file + 3600 * 6 > DT_END and ( first_tradeID == 1
-                        or (currencyPair == 'BTC_HUC' and first_tradeID ==     2) 
-                        or (currencyPair == 'BTC_RIC' and first_tradeID ==     2) 
-                        or (currencyPair == 'BTC_XCP' and first_tradeID ==     2)
-                        or (currencyPair == 'BTC_NAV' and first_tradeID ==  4569)
-                        or (currencyPair == 'BTC_POT' and first_tradeID == 23511) ) ):
-                        return
 
         except Exception as e:
-            log.error('Error opening file: {}'.format(csv_fn))
+            log.error('Error opening file: %s' % csv_fn)
             log.exception(e)
 
-        '''
+        return DT_START
-        Poloniex API limits querying TradeHistory to intervals smaller 
-        than 1 month, so we make sure that start date is never more than
-        1 month apart from end date
-        '''
-        if( end - start > 2419200 ): # 60s/min * 60min/hr * 24hr/day * 28days     
-            newstart = end - 2419200
-        else:
-            newstart = start
 
-        log.debug('{}: Retrieving from {} to {}\t {} - {}'.format(
+    def get_data(self, currencyPair, start, end=9999999999, period=300):
-                    currencyPair, str(newstart), str(end),
+        url = self._api_path + 'command=returnChartData&currencyPair=' + currencyPair + '&start=' + str(start) + '&end=' + str(end) + '&period=' + str(period)
-                    time.ctime(newstart), time.ctime(end)))
 
-        url = '{path}command=returnTradeHistory&currencyPair={pair}' \
+        try:
-              '&start={start}&end={end}'.format(
+            response = requests.get(url)
-                    path = self._api_path,
+        except Exception as e:
-                    pair = currencyPair,
+            log.error('Failed to retrieve candlestick chart data for %s' % currencyPair)
-                    start = str(newstart),
+            log.exception(e)
-                    end = str(end)
-                )
-        print url
-
-        attempts = 0
-        success = 0
-        while attempts < CONN_RETRIES:
-            try:
-                response = requests.get(url)
-            except Exception as e:
-                log.error('Failed to retrieve trade history data for {}'.format(
-                            currencyPair
-                            ))
-                log.exception(e)
-                attempts += 1
-            else:
-                try:
-                    if isinstance(response.json(), dict) and response.json()['error']:
-                        log.error('Failed to to retrieve trade history data '
-                                  'for {}: {}'.format(
-                                    currencyPair,
-                                    response.json()['error']
-                                    ))
-                        attempts += 1
-                except Exception as e:
-                    log.exception(e)
-                    attempts += 1
-                else:
-                    success = 1
-                    break
-
-        if not success:
             return None
 
+        return response.json()
 
-        '''
+    '''
-        If we get to transactionId == 1, and we already have that on 
+    Pulls latest data for a single pair
-        disk, we got to the end of TradeHistory for this coin.
+    '''
-        '''
+    def append_data_single_pair(self, currencyPair, repeat=0):
-        if('first_tradeID' in locals() 
+        log.debug('Getting data for %s' % currencyPair)
-                and response.json()[-1]['tradeID'] == first_tradeID): 
+        csv_fn = CSV_OUT_FOLDER + 'crypto_prices-' + currencyPair + '.csv'
-            return
+        start  = self._get_start_date(csv_fn)
+        # Only fetch data if more than 5min have passed since last fetch
+        if (time.time() > start):
+            data   = self.get_data(currencyPair, start)
+            if data is not None:
+                try: 
+                    with open(csv_fn, 'ab') as csvfile:
+                        csvwriter = csv.writer(csvfile)
+                        for item in data:
+                            if item['date'] == 0:
+                                continue
+                            csvwriter.writerow([
+                                item['date'],
+                                item['open'],
+                                item['high'],
+                                item['low'],
+                                item['close'],
+                                item['volume'],
+                            ])
+                except Exception as e:
+                    log.error('Error opening %s' % csv_fn)
+                    log.exception(e)
+            elif (repeat < CONN_RETRIES):
+                    log.debug('Retrying: attemt %d' % (repeat+1) )
+                    self.append_data_single_pair(currencyPair, repeat + 1)    
 
-        '''
+    '''
-            There are primarily two scenarios:
+    Pulls latest data for all currency pairs
-                a) There is newer data available that we need to add at 
+    '''
-                   the beginning of the file. We'll retrieve all what we 
+    def append_data(self):
-                   need until we get to what we already have, writing it 
+        for currencyPair in self.currency_pairs:
-                   to a temporary file; and we will write that at the 
+            self.append_data_single_pair(currencyPair)
-                   beginning of our existing file.
+            # Rate limit is 6 calls per second, sleep 1sec/6 to be safe
-                b) We are going back in time, appending at the end of 
+            time.sleep(0.17)
-                   our existing TradeHistory until the first transaction 
-                   for this currencyPair
-        '''
-        try: 
-            if( 'end_file' in locals() and end_file + 3600 < end):
-                if (temp is None):
-                    temp = os.tmpfile()
-                tempcsv = csv.writer(temp)
-                for item in response.json():
-                    if( item['tradeID'] <= last_tradeID ):
-                        continue
-                    tempcsv.writerow([
-                        item['tradeID'],
-                        item['date'],
-                        item['type'],
-                        item['rate'],
-                        item['amount'],
-                        item['total'],
-                        item['globalTradeID']        
-                    ])
-                if( response.json()[-1]['tradeID'] > last_tradeID ):
-                    end = pd.to_datetime( response.json()[-1]['date'], 
-                                infer_datetime_format=True).value // 10 ** 9
-                    self.retrieve_trade_history(currencyPair, start, 
-                                                end, temp=temp) 
-                else:
-                    with open(csv_fn,'rb+') as f:
-                        shutil.copyfileobj(f,temp)
-                        f.seek(0)
-                        temp.seek(0)
-                        shutil.copyfileobj(temp,f)
-                    temp.close()
-                    end = start_file
-            else:
-                with open(csv_fn, 'ab') as csvfile:
-                    csvwriter = csv.writer(csvfile)
-                    for item in response.json():
-                        if( 'first_tradeID' in locals() 
-                                and item['tradeID'] >= first_tradeID ):
-                            continue
-                        csvwriter.writerow([
-                            item['tradeID'],
-                            item['date'],
-                            item['type'],
-                            item['rate'],
-                            item['amount'],
-                            item['total'],
-                            item['globalTradeID']
-                        ])
-                end = pd.to_datetime(response.json()[-1]['date'], 
-                            infer_datetime_format=True).value // 10 ** 9
 
-        except Exception as e:
+    '''
-            log.error('Error opening {}'.format(csv_fn))
+    Returns a data frame for all pairs, or for the requests currency pair.
-            log.exception(e)
+    Makes sure data is up to date
+    '''
+    def to_dataframe(self, start, end, currencyPair=None):
+        csv_fn = CSV_OUT_FOLDER + 'crypto_prices-' + currencyPair + '.csv'
+        last_date = self._get_start_date(csv_fn)
+        if last_date + 300 < end or not os.path.exists(csv_fn):
+            # get latest data
+            self.append_data_single_pair(currencyPair)
 
-        '''
+        # CSV holds the latest snapshot
-            If we got here, we aren't done yet. Call recursively with 
+        df = pd.read_csv(csv_fn, names=['date', 'open', 'high', 'low', 'close', 'volume'])
-            'end' times that go sequentially back in time.
+        df['date']=pd.to_datetime(df['date'],unit='s')
-        '''
-        self.retrieve_trade_history(currencyPair, start, end)
-
-
-
-    def generate_ohlcv(self, df):
-        '''
-        Generates OHLCV dataframe from a dataframe containing all TradeHistory
-        by resampling with 1-minute period
-        '''
-        df.set_index('date', inplace=True)               # Index by date
-        vol = df['total'].to_frame('volume')             # set Vol aside
-        df.drop('total', axis=1, inplace=True)           # Drop volume data 
-        ohlc = df.resample('T').ohlc()                   # Resample OHLC 1min
-        ohlc.columns = ohlc.columns.map(lambda t: t[1])  # Raname columns by dropping 'rate'
-        closes = ohlc['close'].fillna(method='pad')      # Pad fwd missing 'close'
-        ohlc = ohlc.apply(lambda x: x.fillna(closes))    # Fill N/A with last close
-        vol = vol.resample('T').sum().fillna(0)          # Add volumes by bin
-        ohlcv = pd.concat([ohlc,vol], axis=1)            # Concatenate OHLC + Vol
-        return ohlcv
-
-
-
-    def write_ohlcv_file(self, currencyPair): 
-        '''
-        Generates OHLCV data file with 1minute bars from TradeHistory on disk
-        '''       
-        csv_trades = CSV_OUT_FOLDER + 'crypto_trades-' + currencyPair + '.csv'
-        csv_1min   = CSV_OUT_FOLDER + 'crypto_1min-' + currencyPair + '.csv'
-        if( os.path.getmtime(csv_1min) > time.time() - 7200 ):
-            log.debug(currencyPair+': 1min data file already up to date. '
-                      'Delete the file if you want to rebuild it.')
-        else:
-            df = pd.read_csv(csv_trades, 
-                                names=['tradeID',
-                                       'date',
-                                       'type',
-                                       'rate',
-                                       'amount',
-                                       'total',
-                                       'globalTradeID'], 
-                                dtype = {'tradeID': int, 
-                                         'date': str, 
-                                         'type': str, 
-                                         'rate': float, 
-                                         'amount': float, 
-                                         'total': float, 
-                                         'globalTradeID': int } 
-                                )
-            df.drop(['tradeID','type','amount','globalTradeID'], 
-                    axis=1, inplace=True)
-            df['date'] = pd.to_datetime(df['date'], infer_datetime_format=True) 
-            ohlcv = self.generate_ohlcv(df)
-            try: 
-                with open(csv_1min, 'w') as csvfile:
-                    csvwriter = csv.writer(csvfile)
-                    for item in ohlcv.itertuples():
-                        if item.Index == 0:
-                            continue
-                        csvwriter.writerow([
-                            item.Index.value // 10 ** 9,
-                            item.open,
-                            item.high,
-                            item.low,
-                            item.close,
-                            item.volume,
-                        ])
-            except Exception as e:
-                log.error('Error opening {}'.format(csv_fn))
-                log.exception(e)
-            log.debug('{}: Generated 1min OHLCV data.'.format(currencyPair))
-
-
-
-    def onemin_to_dataframe(self, currencyPair, start, end):
-        '''
-        Returns a data frame for a given currencyPair from data on disk
-        '''
-        csv_fn     = CSV_OUT_FOLDER + 'crypto_1min-' + currencyPair + '.csv'
-        df         = pd.read_csv(csv_fn, names=['date', 
-                                                'open', 
-                                                'high', 
-                                                'low', 
-                                                'close', 
-                                                'volume']
-                                )
-        df['date'] = pd.to_datetime(df['date'],unit='s')
         df.set_index('date', inplace=True)
-        return df[start : end]
-
-
-    def generate_symbols_json(self, filename=None):
-        '''
-        Generates a symbols.json file with corresponding start_date 
-        for each currencyPair
-        '''
-        symbol_map = {}
-
-        if(filename is None):
-            filename = get_exchange_symbols_filename('poloniex')
-
-        with open(filename, 'w') as symbols:
-            for currencyPair in self.currency_pairs:
-                start = None
-                csv_fn     = '{}crypto_trades-{}.csv'.format(
-                                CSV_OUT_FOLDER, currencyPair)
-                with open(csv_fn, 'r') as f: 
-                    f.seek(0, os.SEEK_END)
-                    if(f.tell() > 2):               # Check file size is not 0
-                        f.seek(-2, os.SEEK_END)     # Jump to 2nd last byte
-                        while f.read(1) != b"\n":   # Until EOL is found...
-                            f.seek(-2, os.SEEK_CUR) # ...jump back the read byte plus one more.
-                        start = pd.to_datetime( f.readline().split(',')[1], 
-                                                infer_datetime_format=True)
-
-                if(start is None):
-                    start = time.gmtime()
-                base, market = currencyPair.lower().split('_')
-                symbol = '{market}_{base}'.format( market=market, base=base )
-                symbol_map[currencyPair] = dict(
-                    symbol = symbol,
-                    start_date = start.strftime("%Y-%m-%d")
-                )
-            json.dump(symbol_map, symbols, sort_keys=True, indent=2, 
-                        separators=(',',':'))    
 
+        return df[datetime.fromtimestamp(start):datetime.fromtimestamp(end-1)]
 
 if __name__ == '__main__':
     pc = PoloniexCurator()
     pc.get_currency_pairs()
-    #pc.generate_symbols_json()
+    pc.append_data()
-    
-    for currencyPair in pc.currency_pairs:
-        pc.retrieve_trade_history(currencyPair)
-        log.debug('{} up to date.'.format(currencyPair))
-        pc.write_ohlcv_file(currencyPair)
-

catalyst/data/_equities.pyx

@@ -215,13 +215,13 @@ cpdef _read_bcolz_data(ctable_t table,
                 else:
                     continue
 
-        if column_name in ['open', 'high', 'low', 'close', 'volume']:
+        if column_name in ['open', 'high', 'low', 'close']:
             where_nan = (outbuf == 0)
-            outbuf_as_float = outbuf.astype(float64) * .000000001
+            outbuf_as_float = outbuf.astype(float64) * .000001
             outbuf_as_float[where_nan] = NAN
             results.append(outbuf_as_float)
-        elif column_name in ['volume']:
+        elif column_name != 'volume':
-            results.append(outbuf.astype(float64) * .000000001)
+            results.append(outbuf.astype(uint32))
         else:
             results.append(outbuf)
     return results

catalyst/data/_minute_bar_internal.pyx

@@ -35,6 +35,17 @@ def minute_value(ndarray[long_t, ndim=1] market_opens,
 
     return market_opens[q] + r
 
+@cython.cdivision(True)
+def five_minute_value(ndarray[long_t, ndim=1] market_opens,
+                      Py_ssize_t pos,
+                      short five_minutes_per_day):
+
+    cdef short q, r
+    q = cython.cdiv(pos, five_minutes_per_day)
+    r = cython.cmod(pos, five_minutes_per_day)
+
+    return market_opens[q] + 5 * r
+
 def find_position_of_minute(ndarray[long_t, ndim=1] market_opens,
                             ndarray[long_t, ndim=1] market_closes,
                             long_t minute_val,
@@ -88,6 +99,30 @@ def find_position_of_minute(ndarray[long_t, ndim=1] market_opens,
 
     return (market_open_loc * minutes_per_day) + delta
 
+def find_position_of_five_minute(ndarray[long_t, ndim=1] market_opens,
+                                 ndarray[long_t, ndim=1] market_closes,
+                                 long_t five_minute_val,
+                                 short five_minutes_per_day,
+                                 bool forward_fill):
+
+    cdef Py_ssize_t market_open_loc, market_open, delta
+
+    market_open_loc = \
+        searchsorted(market_opens, five_minute_val, side='right') - 1
+    market_open = market_opens[market_open_loc]
+    market_close = market_closes[market_open_loc]
+
+    val_open_offset = (five_minute_val - market_open)/5
+    close_open_offset = (market_close - market_open)/5
+
+    if not forward_fill and val_open_offset >= five_minutes_per_day:
+        raise ValueError("Given five minutes is not between an open and a close")
+
+    # clamp offset to close index
+    delta = int_min(val_open_offset, close_open_offset)
+
+    return (market_open_loc * five_minutes_per_day) + delta
+
 def find_last_traded_position_internal(
         ndarray[long_t, ndim=1] market_opens,
         ndarray[long_t, ndim=1] market_closes,
@@ -158,3 +193,50 @@ def find_last_traded_position_internal(
     # found a trade event
     return -1
 
+def find_last_traded_five_minute_position_internal(
+        ndarray[long_t, ndim=1] market_opens,
+        ndarray[long_t, ndim=1] market_closes,
+        long_t end_five_minute,
+        long_t start_five_minute,
+        volumes,
+        short five_minutes_per_day):
+    cdef Py_ssize_t minute_pos, current_minute, q
+
+    five_minute_pos = int_min(
+        find_position_of_five_minute(
+            market_opens,
+            market_closes,
+            end_five_minute,
+            five_minutes_per_day,
+            True,
+        ),
+        len(volumes) - 1,
+    )
+
+    while five_minute_pos >= 0:
+        current_five_minute = five_minute_value(
+            market_opens, five_minute_pos, five_minutes_per_day
+        )
+
+        q = cython.cdiv(five_minute_pos, five_minutes_per_day)
+        if current_five_minute > market_closes[q]:
+            five_minute_pos = find_position_of_five_minute(
+                market_opens,
+                market_closes,
+                market_closes[q],
+                five_minutes_per_day,
+                False,
+            )
+            continue
+
+        if current_five_minute < start_five_minute:
+            return -1
+
+        if volumes[five_minute_pos] != 0:
+            return five_minute_pos
+
+        five_minute_pos -= 1
+
+    # we've gone to the beginning of this asset's range, and still haven't
+    # found a trade event
+    return -1

catalyst/data/bundles/base.py

@@ -30,10 +30,8 @@ from catalyst.utils.cli import (
 )
 from catalyst.utils.memoize import lazyval
 
-from catalyst.constants import LOG_LEVEL
-
 logbook.StderrHandler().push_application()
-log = logbook.Logger(__name__, level=LOG_LEVEL)
+log = logbook.Logger(__name__)
 
 DEFAULT_RETRIES = 5
 
@@ -62,6 +60,10 @@ class BaseBundle(object):
     def minutes_per_day(self):
         raise NotImplementedError()
 
+    @lazyval
+    def five_minutes_per_day(self):
+        raise NotImplementedError()
+
     @lazyval
     def frequencies(self):
         raise NotImplementedError()
@@ -113,6 +115,7 @@ class BaseBundle(object):
                environ,
                asset_db_writer,
                minute_bar_writer,
+               five_minute_bar_writer,
                daily_bar_writer,
                adjustment_writer,
                calendar,
@@ -159,7 +162,7 @@ class BaseBundle(object):
 
                 # Post-process metadata using cached symbol frames, and write to
                 # disk.  This metadata must be written before any attempt to write
-                # minute data.
+                # either minute or 5-minute data.
                 metadata = self._post_process_metadata(
                     raw_metadata,
                     cache,
@@ -167,6 +170,24 @@ class BaseBundle(object):
                 )
                 asset_db_writer.write(metadata)
 
+                # Compile 5-minute symbol data if bundle supports 5-minute mode and
+                # persist the dataset to disk.
+                if '5-minute' in self.frequencies:
+                    five_minute_bar_writer.write(
+                        self._fetch_symbol_iter(
+                            api_key,
+                            cache,
+                            symbol_map,
+                            calendar,
+                            start_session,
+                            end_session,
+                            '5-minute',
+                            retries,
+                        ),
+                        length=len(symbol_map),
+                        show_progress=show_progress,
+                    )
+
                 # Compile minute symbol data if bundle supports minute mode and
                 # persist the dataset to disk.
                 if 'minute' in self.frequencies:
@@ -275,12 +296,14 @@ class BaseBundle(object):
                     except Exception as e:
                         log.exception(
                             'Failed to load metadata from {}. '
-                            'Retrying.'.format(self.name)
+                            'Retrying.'.format(
+                                name=self.name,
+                            )
                         ) 
                 else:
                     raise ValueError(
-                        'Failed to download metadata page {} after {} '
+                        'Failed to download metadata page %d after %d '
-                        'attempts.'.format(page_number, retries)
+                        'attempts.'.format(page_number, retries),
                     )
 
 
@@ -290,8 +313,7 @@ class BaseBundle(object):
 
             # Apply selective asset filtering, useful for benchmark
             # ingestion.
-            if self._asset_filter:
+            raw = raw[raw.symbol.isin(self._asset_filter)]
-                raw = raw[raw.symbol.isin(self._asset_filter)]
 
             # Update cached value for key.
             cache[key] = raw
@@ -468,7 +490,7 @@ class BaseBundle(object):
                     data_frequency,
                 )
                 raw_data.index = pd.to_datetime(raw_data.index, utc=True)
-                #raw_data.index = raw_data.index.tz_localize('UTC')
+                raw_data.index = raw_data.index.tz_localize('UTC')
 
                 # Filter incoming data to fit start and end sessions.
                 raw_data = raw_data[

catalyst/data/bundles/base_pricing.py

@@ -24,7 +24,6 @@ class BasePricingBundle(BaseBundle):
             ('start_date', 'datetime64[ns]'),
             ('end_date', 'datetime64[ns]'),
             ('ac_date', 'datetime64[ns]'),
-            ('min_trade_size', 'float'),
         ]
 
     @lazyval
@@ -47,6 +46,10 @@ class BaseCryptoPricingBundle(BasePricingBundle):
     def minutes_per_day(self):
         return 1440
 
+    @lazyval
+    def five_minutes_per_day(self):
+        return 288
+
     @property
     def splits(self):
         return []
@@ -64,6 +67,10 @@ class BaseEquityPricingBundle(BasePricingBundle):
     def minutes_per_day(self):
         return 390
 
+    @lazyval
+    def five_minutes_per_day(self):
+        return 78
+
     @property
     def splits(self):
         return self._splits

catalyst/data/bundles/core.py

@@ -17,6 +17,10 @@ from ..us_equity_pricing import (
     SQLiteAdjustmentReader,
     SQLiteAdjustmentWriter,
 )
+from ..five_minute_bars import (
+    BcolzFiveMinuteBarReader,
+    BcolzFiveMinuteBarWriter,
+)
 from ..minute_bars import (
     BcolzMinuteBarReader,
     BcolzMinuteBarWriter,
@@ -50,6 +54,11 @@ def minute_path(bundle_name, timestr, environ=None):
         environ=environ,
     )
 
+def five_minute_path(bundle_name, timestr, environ=None):
+    return pth.data_path(
+        five_minute_relative(bundle_name, timestr, environ),
+        environ=environ,
+    )
 
 def daily_path(bundle_name, timestr, environ=None):
     return pth.data_path(
@@ -83,6 +92,8 @@ def cache_relative(bundle_name, timestr, environ=None):
 def daily_relative(bundle_name, timestr, environ=None):
     return bundle_name, timestr, 'daily_equities.bcolz'
 
+def five_minute_relative(bundle_name, timestr, environ=None):
+    return bundle_name, timestr, 'five_minute.bcolz'
 
 def minute_relative(bundle_name, timestr, environ=None):
     return bundle_name, timestr, 'minute_equities.bcolz'
@@ -195,13 +206,14 @@ RegisteredBundle = namedtuple(
      'start_session',
      'end_session',
      'minutes_per_day',
+     'five_minutes_per_day',
      'ingest',
      'create_writers']
 )
 
 BundleData = namedtuple(
     'BundleData',
-    'asset_finder minute_bar_reader daily_bar_reader '
+    'asset_finder minute_bar_reader five_minute_bar_reader daily_bar_reader '
     'adjustment_reader',
 )
 
@@ -291,6 +303,7 @@ def _make_bundle_core():
             bundle.ingest,
             calendar_name=bundle.calendar_name,
             minutes_per_day=bundle.minutes_per_day,
+            five_minutes_per_day=bundle.five_minutes_per_day,
             start_session=start_session,
             end_session=end_session,
             create_writers=create_writers,
@@ -303,6 +316,7 @@ def _make_bundle_core():
                  start_session=None,
                  end_session=None,
                  minutes_per_day=1440,
+                 five_minutes_per_day=288,
                  create_writers=True):
         """Register a data bundle ingest function.
 
@@ -383,6 +397,7 @@ def _make_bundle_core():
             start_session=start_session,
             end_session=end_session,
             minutes_per_day=minutes_per_day,
+            five_minutes_per_day=five_minutes_per_day,
             ingest=f,
             create_writers=create_writers,
         )
@@ -481,6 +496,16 @@ def _make_bundle_core():
                 # that it can compute the adjustment ratios for the dividends.
                 daily_bar_writer.write(())
 
+                five_minute_bar_writer = BcolzFiveMinuteBarWriter(
+                    wd.ensure_dir(*five_minute_relative(
+                        name, timestr, environ=environ)
+                    ),
+                    calendar,
+                    start_session,
+                    end_session,
+                    five_minutes_per_day=bundle.five_minutes_per_day,
+                )
+
                 minute_bar_writer = BcolzMinuteBarWriter(
                     wd.ensure_dir(*minute_relative(
                         name, timestr, environ=environ)
@@ -507,6 +532,7 @@ def _make_bundle_core():
                 )
             else:
                 daily_bar_writer = None
+                five_minute_bar_writer = None
                 minute_bar_writer = None
                 asset_db_writer = None
                 adjustment_db_writer = None
@@ -518,6 +544,7 @@ def _make_bundle_core():
                 environ,
                 asset_db_writer,
                 minute_bar_writer,
+                five_minute_bar_writer,
                 daily_bar_writer,
                 adjustment_db_writer,
                 calendar,
@@ -604,6 +631,9 @@ def _make_bundle_core():
             minute_bar_reader=BcolzMinuteBarReader(
                 minute_path(name, timestr, environ=environ),
             ),
+            five_minute_bar_reader=BcolzFiveMinuteBarReader(
+                five_minute_path(name, timestr, environ=environ),
+            ),
             daily_bar_reader=BcolzDailyBarReader(
                 daily_path(name, timestr, environ=environ),
             ),

catalyst/data/bundles/poloniex.py

@@ -13,8 +13,6 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-import sys
-
 from datetime import datetime
 
 import pandas as pd
@@ -25,8 +23,6 @@ from catalyst.data.bundles.core import register_bundle
 from catalyst.data.bundles.base_pricing import BaseCryptoPricingBundle
 from catalyst.utils.memoize import lazyval
 
-from catalyst.curate.poloniex import PoloniexCurator
-
 class PoloniexBundle(BaseCryptoPricingBundle):
     @lazyval
     def name(self):
@@ -40,13 +36,14 @@ class PoloniexBundle(BaseCryptoPricingBundle):
     def frequencies(self):
         return set((
             'daily',
-            'minute',
+            '5-minute',
         ))
 
     @lazyval
     def tar_url(self):
         return (
-            'https://s3.amazonaws.com/enigmaco/catalyst-bundles/poloniex/poloniex-bundle.tar.gz'
+            'https://www.dropbox.com/s/9naqffawnq8o4r2/'
+            'poloniex-bundle.tar?dl=1'
         )
 
     @lazyval
@@ -79,14 +76,12 @@ class PoloniexBundle(BaseCryptoPricingBundle):
         start_date = sym_data.index[0]
         end_date = sym_data.index[-1]
         ac_date = end_date + pd.Timedelta(days=1)
-        min_trade_size = 0.00000001
 
         return (
             sym_md.symbol,
             start_date,
             end_date,
             ac_date,
-            min_trade_size,
         )
 
     def fetch_raw_symbol_frame(self,
@@ -96,30 +91,19 @@ class PoloniexBundle(BaseCryptoPricingBundle):
                                start_date,
                                end_date,
                                frequency):
+        raw = pd.read_json(
+            self._format_data_url(
+                api_key,
+                symbol,
+                start_date,
+                end_date,
+                frequency,
+            ),
+            orient='records',
+        )
+        raw.set_index('date', inplace=True)
 
-        # TODO: replace this with direct exchange call
+        scale = 1000.0
-        # The end date and frequency should be used to calculate the number of bars
-        if(frequency == 'minute'):
-            pc = PoloniexCurator()
-            raw = pc.onemin_to_dataframe(symbol, start_date, end_date)
-
-        else:
-            raw = pd.read_json(
-                self._format_data_url(
-                    api_key,
-                    symbol,
-                    start_date,
-                    end_date,
-                    frequency,
-                ),
-                orient='records',
-            )
-            raw.set_index('date', inplace=True)
-
-        # BcolzDailyBarReader introduces a 1/1000 factor in the way pricing is stored
-        # on disk, which we compensate here to get the right pricing amounts
-        # ref: data/us_equity_pricing.py
-        scale = 1
         raw.loc[:, 'open'] /= scale
         raw.loc[:, 'high'] /= scale
         raw.loc[:, 'low'] /= scale
@@ -148,6 +132,7 @@ class PoloniexBundle(BaseCryptoPricingBundle):
                          data_frequency):
         period_map = {
             'daily': 86400,
+            '5-minute': 300,
         }
 
         try:
@@ -166,23 +151,8 @@ class PoloniexBundle(BaseCryptoPricingBundle):
         return self._format_polo_query(query_params)
     
     def _format_polo_query(self, query_params):
-        # TODO: got against the exchange object
         return 'https://poloniex.com/public?{query}'.format(
             query=urlencode(query_params),
         )
 
-''' 
+register_bundle(PoloniexBundle, ['USDT_BTC'])
-As a second parameter, you can pass an array of currency pairs 
-that will be processed as an asset_filter to only process that 
-subset of assets in the bundle, such as:
-register_bundle(PoloniexBundle, ['USDT_BTC',])
-
-For a production environment make sure to use (to bundle all pairs):
-register_bundle(PoloniexBundle)
-'''
-
-if 'ingest' in sys.argv and '-c' in sys.argv:
-    register_bundle(PoloniexBundle)
-else:
-    register_bundle(PoloniexBundle, create_writers=False)
-

catalyst/data/bundles/quandl.py

@@ -40,9 +40,7 @@ from catalyst.utils.cli import maybe_show_progress
 
 from . import core as bundles
 
-from catalyst.constants import LOG_LEVEL
+log = Logger(__name__)
-
-log = Logger(__name__, level=LOG_LEVEL)
 seconds_per_call = (pd.Timedelta('10 minutes') / 2000).total_seconds()
 
 class QuandlBundle(BaseEquityPricingBundle):

catalyst/data/data_portal.py

@@ -42,6 +42,7 @@ from catalyst.assets.roll_finder import (
 )
 from catalyst.data.dispatch_bar_reader import (
     AssetDispatchMinuteBarReader,
+    AssetDispatchFiveMinuteBarReader,
     AssetDispatchSessionBarReader
 )
 from catalyst.data.resample import (
@@ -68,9 +69,7 @@ from catalyst.errors import (
     HistoryWindowStartsBeforeData,
 )
 
-from catalyst.constants import LOG_LEVEL
+log = Logger('DataPortal')
-
-log = Logger('DataPortal', level=LOG_LEVEL)
 
 BASE_FIELDS = frozenset([
     "open",
@@ -121,6 +120,10 @@ class DataPortal(object):
         daily data backtests or daily history calls in a minute backetest.
         If a daily bar reader is not provided but a minute bar reader is,
         the minutes will be rolled up to serve the daily requests.
+    five_minute_reader : BcolzFiveMinuteBarReader, optional
+        The five minute bar reader for equities. This will be used to service
+        5-minute data backtests or five-minute history calls.  This can be used
+        to serve daily calls if no daily bar reader is provided.
     minute_reader : BcolzMinuteBarReader, optional
         The minute bar reader for equities. This will be used to service
         minute data backtests or minute history calls. This can be used
@@ -147,6 +150,7 @@ class DataPortal(object):
                  trading_calendar,
                  first_trading_day,
                  daily_reader=None,
+                 five_minute_reader=None,
                  minute_reader=None,
                  future_daily_reader=None,
                  future_minute_reader=None,
@@ -198,6 +202,7 @@ class DataPortal(object):
                 reader.last_available_dt
                 for reader in [
                     minute_reader,
+                    five_minute_reader,
                     future_minute_reader,
                 ]
                 if reader is not None
@@ -209,6 +214,8 @@ class DataPortal(object):
 
         aligned_minute_reader = self._ensure_reader_aligned(
             minute_reader)
+        aligned_five_minute_reader = self._ensure_reader_aligned(
+            five_minute_reader)
         aligned_session_reader = self._ensure_reader_aligned(
             daily_reader)
         aligned_future_minute_reader = self._ensure_reader_aligned(
@@ -222,10 +229,13 @@ class DataPortal(object):
         }
 
         aligned_minute_readers = {}
+        aligned_five_minute_readers = {}
         aligned_session_readers = {}
 
         if aligned_minute_reader is not None:
             aligned_minute_readers[Equity] = aligned_minute_reader
+        if aligned_five_minute_reader is not None:
+            aligned_five_minute_readers[Equity] = aligned_five_minute_reader
         if aligned_session_reader is not None:
             aligned_session_readers[Equity] = aligned_session_reader
 
@@ -257,6 +267,13 @@ class DataPortal(object):
             self._last_available_minute,
         )
 
+        _dispatch_five_minute_reader = AssetDispatchFiveMinuteBarReader(
+            self.trading_calendar,
+            self.asset_finder,
+            aligned_five_minute_readers,
+            self._last_available_minute,
+        )
+
         _dispatch_session_reader = AssetDispatchSessionBarReader(
             self.trading_calendar,
             self.asset_finder,
@@ -266,12 +283,13 @@ class DataPortal(object):
 
         self._pricing_readers = {
             'minute': _dispatch_minute_reader,
+            '5-minute': _dispatch_five_minute_reader,
             'daily': _dispatch_session_reader,
         }
 
         self._daily_aggregator = DailyHistoryAggregator(
             self.trading_calendar.schedule.market_open,
-            _dispatch_minute_reader,
+            _dispatch_session_reader,
             self.trading_calendar
         )
         self._history_loader = DailyHistoryLoader(
@@ -701,6 +719,17 @@ class DataPortal(object):
             spot_value=result
         )
 
+
+    def _get_five_minute_spot_value(self, asset, column, dt, ffill=False):
+        return self._get_minutely_spot_value(
+            asset,
+            column,
+            dt,
+            ffill, 
+            '5-minute',
+        )
+        
+
     def _get_minute_spot_value(self, asset, column, dt, ffill=False):
         return self._get_minutely_spot_value(
             asset,

catalyst/data/dispatch_bar_reader.py

@@ -18,7 +18,6 @@ from numpy import (
     full,
     nan,
     int64,
-    float64,
     zeros
 )
 from six import iteritems, with_metaclass
@@ -71,9 +70,7 @@ class AssetDispatchBarReader(with_metaclass(ABCMeta)):
         return self._dt_window_size(start_dt, end_dt), num_sids
 
     def _make_raw_array_out(self, field, shape):
-        if field == 'volume':
+        if field != 'volume' and field != 'sid':
-            out = zeros(shape, dtype=float64)
-        elif field != 'sid':
             out = full(shape, nan)
         else:
             out = zeros(shape, dtype=int64)
@@ -138,6 +135,12 @@ class AssetDispatchMinuteBarReader(AssetDispatchBarReader):
     def _dt_window_size(self, start_dt, end_dt):
         return len(self.trading_calendar.minutes_in_range(start_dt, end_dt))
 
+
+class AssetDispatchFiveMinuteBarReader(AssetDispatchBarReader):
+
+    def _dt_window_size(self, start_dt, end_dt):
+        return len(self.trading_calendar.five_minutes_in_range(start_dt, end_dt))
+
 class AssetDispatchSessionBarReader(AssetDispatchBarReader):
 
     def _dt_window_size(self, start_dt, end_dt):

catalyst/data/history_loader.py

@@ -38,7 +38,7 @@ from catalyst.utils.numpy_utils import float64_dtype
 from catalyst.utils.pandas_utils import find_in_sorted_index
 
 # Default number of decimal places used for rounding asset prices.
-DEFAULT_ASSET_PRICE_DECIMALS = 9
+DEFAULT_ASSET_PRICE_DECIMALS = 3
 
 
 class HistoryCompatibleUSEquityAdjustmentReader(object):

catalyst/data/loader.py

@@ -12,38 +12,41 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-import datetime
 import os
 from collections import OrderedDict
 
 import logbook
 import pandas as pd
-import pytz
+import numpy as np
 from pandas_datareader.data import DataReader
+import datetime
+import time
+import pytz
 from six import iteritems
 from six.moves.urllib_error import HTTPError
 
-from catalyst.utils.calendars import get_calendar
-from . import treasuries, treasuries_can
 from .benchmarks import get_benchmark_returns
-from ..utils.deprecate import deprecated
+from . import treasuries, treasuries_can
 from ..utils.paths import (
     cache_root,
     data_root,
 )
+from ..utils.deprecate import deprecated
 
-from catalyst.constants import LOG_LEVEL
+from catalyst.data.bundles.poloniex import PoloniexBundle
+from catalyst.utils.calendars import get_calendar
 
-logger = logbook.Logger('Loader', level=LOG_LEVEL)
+
+logger = logbook.Logger('Loader')
 
 # Mapping from index symbol to appropriate bond data
 INDEX_MAPPING = {
     'SPY':
-        (treasuries, 'treasury_curves.csv', 'www.federalreserve.gov'),
+    (treasuries, 'treasury_curves.csv', 'www.federalreserve.gov'),
     '^GSPTSE':
-        (treasuries_can, 'treasury_curves_can.csv', 'bankofcanada.ca'),
+    (treasuries_can, 'treasury_curves_can.csv', 'bankofcanada.ca'),
     '^FTSE':  # use US treasuries until UK bonds implemented
-        (treasuries, 'treasury_curves.csv', 'www.federalreserve.gov'),
+    (treasuries, 'treasury_curves.csv', 'www.federalreserve.gov'),
 }
 
 ONE_HOUR = pd.Timedelta(hours=1)
@@ -90,28 +93,20 @@ def has_data_for_dates(series_or_df, first_date, last_date):
     dts = series_or_df.index
     if not isinstance(dts, pd.DatetimeIndex):
         raise TypeError("Expected a DatetimeIndex, but got %s." % type(dts))
-    first, last = dts[[0, -1]].tz_localize(None)
+    first, last = dts[[0, -1]]
-    return (first <= first_date.tz_localize(None)) and (
+    return (first <= first_date) and (last >= last_date)
-        last >= last_date.tz_localize(None))
 
-
+def load_crypto_market_data(trading_day=None,
-def load_crypto_market_data(trading_day=None, trading_days=None,
+                            trading_days=None,
-                            bm_symbol=None, bundle=None, bundle_data=None,
+                            bm_symbol='USDT_BTC',
-                            environ=None, exchange=None, start_dt=None,
+                            environ=None):
-                            end_dt=None):
     if trading_day is None:
         trading_day = get_calendar('OPEN').trading_day
+    if trading_days is None:
+        trading_days = get_calendar('OPEN').all_sessions
 
-    # TODO: consider making configurable
+    first_date = trading_days[0]
-    bm_symbol = 'btc_usdt'
+    now = pd.Timestamp.utcnow()
-    # if trading_days is None:
-    #    trading_days = get_calendar('OPEN').schedule
-
-    # if start_dt is None:
-    start_dt = get_calendar('OPEN').first_trading_session
-
-    if end_dt is None:
-        end_dt = pd.Timestamp.utcnow()
 
     # We expect to have benchmark and treasury data that's current up until
     # **two** full trading days prior to the most recently completed trading
@@ -127,54 +122,29 @@ def load_crypto_market_data(trading_day=None, trading_days=None,
 
     # We'll attempt to download new data if the latest entry in our cache is
     # before this date.
-    '''
+    last_date = trading_days[trading_days.get_loc(now, method='ffill') - 2]
-    if(bundle_data):
-        # If we are using the bundle to retrieve the cryptobenchmark, find the last
-        # date for which there is trading data in the bundle
-        asset = bundle_data.asset_finder.lookup_symbol(symbol=bm_symbol,as_of_date=None)
-        ix = bundle_data.daily_bar_reader._last_rows[asset.sid]
-        last_date = pd.to_datetime(bundle_data.daily_bar_reader._spot_col('day')[ix],unit='s')
-    else:
-        last_date = trading_days[trading_days.get_loc(now, method='ffill') - 2]
-    '''
-    last_date = trading_days[trading_days.get_loc(end_dt, method='ffill') - 1]
 
-    if exchange is None:
+    br = ensure_crypto_benchmark_data(
-        # This is exceptional, since placing the import at the module scope
-        #  breaks things and it's only needed here
-        from catalyst.exchange.poloniex.poloniex import Poloniex
-        exchange = Poloniex('', '', '')
-
-    benchmark_asset = exchange.get_asset(bm_symbol)
-
-    # exchange.get_history_window() already ensures that we have the right data
-    # for the right dates
-    br = exchange.get_history_window(
-        assets=[benchmark_asset],
-        end_dt=last_date,
-        bar_count=pd.Timedelta(last_date - start_dt).days,
-        frequency='1d',
-        field='close',
-        data_frequency='daily')
-    br.columns = ['close']
-    br = br.pct_change(1).iloc[1:]
-    br.loc[start_dt] = 0
-    br = br.sort_index()
-
-    # Override first_date for treasury data since we have it for many more years
-    # and is independent of crypto data
-    first_date_treasury = pd.Timestamp('1990-01-02', tz='UTC')
-    tc = ensure_treasury_data(
         bm_symbol,
-        first_date_treasury,
+        first_date,
         last_date,
-        end_dt,
+        now,
+        # We need the trading_day to figure out the close prior to the first
+        # date so that we can compute returns for the first date.
+        trading_day,
         environ,
     )
-    benchmark_returns = br[br.index.slice_indexer(start_dt, last_date)]
+    tc = ensure_treasury_data(
-    treasury_curves = tc[
+        bm_symbol,
-        tc.index.slice_indexer(first_date_treasury, last_date)]
+        first_date,
+        last_date,
+        now,
+        environ,
+    )
+    benchmark_returns = br[br.index.slice_indexer(first_date, last_date)]
+    treasury_curves = tc[tc.index.slice_indexer(first_date, last_date)]
     return benchmark_returns, treasury_curves
+    
 
 
 def load_market_data(trading_day=None, trading_days=None, bm_symbol='SPY',
@@ -262,22 +232,20 @@ def load_market_data(trading_day=None, trading_days=None, bm_symbol='SPY',
     treasury_curves = tc[tc.index.slice_indexer(first_date, last_date)]
     return benchmark_returns, treasury_curves
 
-
 def ensure_crypto_benchmark_data(symbol,
                                  first_date,
                                  last_date,
                                  now,
                                  trading_day,
-                                 bundle,
-                                 bundle_data,
                                  environ=None):
+
     filename = get_benchmark_filename(symbol)
 
     logger.info(
         ('Loading benchmark data for {symbol!r} '
-         'from {first_date} to {last_date}'),
+            'from {first_date} to {last_date}'),
         symbol=symbol,
-        first_date=first_date,
+        first_date=first_date - trading_day,
         last_date=last_date
     )
 
@@ -290,65 +258,34 @@ def ensure_crypto_benchmark_data(symbol,
         environ,
     )
 
+
     if data is not None:
         return data
 
     # If no cached data was found or it was missing any dates then download the
     # necessary data.
+    logger.info(
+        ('Downloading benchmark data for {symbol!r} '
+            'from {first_date} to {last_date}'),
+        symbol=symbol,
+        first_date=first_date - trading_day,
+        last_date=last_date
+    )
 
-    if (bundle == 'poloniex'):
+    # Load benchmark symbol from Poloniex API
-        '''
+    try:
-        If we're using the Poloniex bundle, we'll get the benchmark from the bundle
+        bundle = PoloniexBundle()
-        instead of downloading it from Poloniex every time we need it.
+        bench_raw = bundle._fetch_symbol_frame(
-        Poloniex has a captcha for API queries originating from outside the US that 
+            None,
-        prevents users abroad from getting Catalyst to work
+            symbol,
-        '''
+            get_calendar(bundle.calendar_name),
-        logger.info(
+            first_date,
-            (
+            last_date,
-                'Retrieving benchmark data from bundle for {symbol!r} from {first_date} to {last_date}'),
+            'daily',
-            symbol=symbol, first_date=first_date, last_date=last_date)
+        )
-
+    except (OSError, IOError, HTTPError):
-        asset = bundle_data.asset_finder.lookup_symbol(symbol=symbol,
+        logger.exception('Failed to fetch new crypto benchmark returns')
-                                                       as_of_date=None)
+        raise
-        fields = ['day', 'close']
-        raw = bundle_data.daily_bar_reader.load_raw_arrays(
-            columns=fields,
-            start_date=first_date - trading_day,
-            end_date=last_date,
-            assets=[asset, ])
-        bench_raw = pd.concat([pd.DataFrame(raw[0], columns=['date']),
-                               pd.DataFrame(raw[1], columns=['close'])],
-                              axis=1)
-        bench_raw['date'] = pd.to_datetime(bench_raw['date'], unit='s')
-        bench_raw.set_index('date', inplace=True)
-        bench_raw.sort_index(inplace=True)
-        bench_raw = bench_raw[
-                    pd.to_datetime(first_date - trading_day):pd.to_datetime(
-                        last_date)]
-
-    else:
-        # This is how it used to be: downloading the benchmark everytime. 
-        # Leaving this code here to be repurposed in the future for other bundles.
-        logger.info(
-            (
-                'Downloading benchmark data for {symbol!r} from {first_date} to {last_date}'),
-            symbol=symbol, first_date=first_date, last_date=last_date)
-
-        raise DeprecationWarning('poloniex bundle deprecated')
-        # Load benchmark symbol from Poloniex API
-        # try:
-        #     bundle = PoloniexBundle()
-        #     bench_raw = bundle._fetch_symbol_frame(
-        #         None,
-        #         symbol,
-        #         get_calendar(bundle.calendar_name),
-        #         first_date - trading_day,
-        #         last_date,
-        #         'daily',
-        #     )
-        # except (OSError, IOError, HTTPError):
-        #     logger.exception('Failed to fetch new crypto benchmark returns')
-        #     raise
 
     # select close column and compute percent change between days
     daily_close = bench_raw[['close']]
@@ -407,7 +344,7 @@ def ensure_benchmark_data(symbol, first_date, last_date, now, trading_day,
     # necessary data.
     logger.info(
         ('Downloading benchmark data for {symbol!r} '
-         'from {first_date} to {last_date}'),
+            'from {first_date} to {last_date}'),
         symbol=symbol,
         first_date=first_date - trading_day,
         last_date=last_date
@@ -427,7 +364,6 @@ def ensure_benchmark_data(symbol, first_date, last_date, now, trading_day,
         logger.warn("Still don't have expected data after redownload!")
     return data
 
-
 def ensure_benchmark_data(symbol, first_date, last_date, now, trading_day,
                           environ=None):
     """
@@ -468,7 +404,7 @@ def ensure_benchmark_data(symbol, first_date, last_date, now, trading_day,
     # necessary data.
     logger.info(
         ('Downloading benchmark data for {symbol!r} '
-         'from {first_date} to {last_date}'),
+            'from {first_date} to {last_date}'),
         symbol=symbol,
         first_date=first_date - trading_day,
         last_date=last_date
@@ -542,6 +478,11 @@ def ensure_treasury_data(symbol, first_date, last_date, now, environ=None):
 
 def _load_cached_data(filename, first_date, last_date, now, resource_name,
                       environ=None):
+    if resource_name == 'benchmark':
+        from_csv = pd.Series.from_csv
+    else:
+        from_csv = pd.DataFrame.from_csv
+
     # Path for the cache.
     path = get_data_filepath(filename, environ)
 
@@ -549,11 +490,8 @@ def _load_cached_data(filename, first_date, last_date, now, resource_name,
     # yet, so don't try to read from 'path'.
     if os.path.exists(path):
         try:
-            data = pd.DataFrame.from_csv(path)
+            data = from_csv(path)
-            if data.empty:
+            data.index = pd.to_datetime(data.index).tz_localize('UTC')
-                raise ValueError("File is empty.")
-            data.index = pd.to_datetime(data.index, infer_datetime_format=True,
-                                        errors='coerce').tz_localize('UTC')
             if has_data_for_dates(data, first_date, last_date):
                 return data
 
@@ -579,7 +517,7 @@ def _load_cached_data(filename, first_date, last_date, now, resource_name,
             )
 
     logger.info(
-        "Cache at {path} does not have data from {start} to {end}.",
+        "Cache at {path} does not have data from {start} to {end}.\n",
         start=first_date,
         end=last_date,
         path=path,

catalyst/data/minute_bars.py

@@ -39,21 +39,20 @@ from catalyst.data._minute_bar_internal import (
 from catalyst.gens.sim_engine import NANOS_IN_MINUTE
 
 from catalyst.data.bar_reader import BarReader, NoDataOnDate
-from catalyst.data.us_equity_pricing import check_uint64_safe
+from catalyst.data.us_equity_pricing import check_uint32_safe
 from catalyst.utils.calendars import get_calendar
 from catalyst.utils.cli import maybe_show_progress
 from catalyst.utils.memoize import lazyval
 
-from catalyst.constants import LOG_LEVEL
 
-logger = logbook.Logger('MinuteBars', level=LOG_LEVEL)
+logger = logbook.Logger('MinuteBars')
 
 US_EQUITIES_MINUTES_PER_DAY = 390
 FUTURES_MINUTES_PER_DAY = 1440
 
 DEFAULT_EXPECTEDLEN = US_EQUITIES_MINUTES_PER_DAY * 252 * 15
 
-OHLC_RATIO = 100000000
+OHLC_RATIO = 1000
 
 
 class BcolzMinuteOverlappingData(Exception):
@@ -115,15 +114,15 @@ def _sid_subdir_path(sid):
 
 
 def convert_cols(cols, scale_factor, sid, invalid_data_behavior):
-    """Adapt OHLCV columns into uint64 columns.
+    """Adapt OHLCV columns into uint32 columns.
 
     Parameters
     ----------
     cols : dict
         A dict mapping each column name (open, high, low, close, volume)
-        to a float column to convert to uint64.
+        to a float column to convert to uint32.
     scale_factor : int
-        Factor to use to scale float values before converting to uint64.
+        Factor to use to scale float values before converting to uint32.
     sid : int
         Sid of the relevant asset, for logging.
     invalid_data_behavior : str
@@ -136,7 +135,6 @@ def convert_cols(cols, scale_factor, sid, invalid_data_behavior):
     scaled_highs = np.nan_to_num(cols['high']) * scale_factor
     scaled_lows = np.nan_to_num(cols['low']) * scale_factor
     scaled_closes = np.nan_to_num(cols['close']) * scale_factor
-    scaled_volumes = np.nan_to_num(cols['volume']) * scale_factor
 
     exclude_mask = np.zeros_like(scaled_opens, dtype=bool)
 
@@ -145,12 +143,11 @@ def convert_cols(cols, scale_factor, sid, invalid_data_behavior):
         ('high', scaled_highs),
         ('low', scaled_lows),
         ('close', scaled_closes),
-        ('volume', scaled_volumes),
     ]:
         max_val = scaled_col.max()
 
         try:
-            check_uint64_safe(max_val, col_name)
+            check_uint32_safe(max_val, col_name)
         except ValueError:
             if invalid_data_behavior == 'raise':
                 raise
@@ -158,20 +155,20 @@ def convert_cols(cols, scale_factor, sid, invalid_data_behavior):
             if invalid_data_behavior == 'warn':
                 logger.warn(
                     'Values for sid={}, col={} contain some too large for '
-                    'uint64 (max={}), filtering them out',
+                    'uint32 (max={}), filtering them out',
                     sid, col_name, max_val,
                 )
 
             # We want to exclude all rows that have an unsafe value in
             # this column.
-            exclude_mask &= (scaled_col >= np.iinfo(np.uint64).max)
+            exclude_mask &= (scaled_col >= np.iinfo(np.uint32).max)
 
     # Convert all cols to uint32.
-    opens = scaled_opens.astype(np.uint64)
+    opens = scaled_opens.astype(np.uint32)
-    highs = scaled_highs.astype(np.uint64)
+    highs = scaled_highs.astype(np.uint32)
-    lows = scaled_lows.astype(np.uint64)
+    lows = scaled_lows.astype(np.uint32)
-    closes = scaled_closes.astype(np.uint64)
+    closes = scaled_closes.astype(np.uint32)
-    volumes = scaled_volumes.astype(np.uint64)
+    volumes = cols['volume'].astype(np.uint32)
 
     # Exclude rows with unsafe values by setting to zero.
     opens[exclude_mask] = 0
@@ -263,14 +260,14 @@ class BcolzMinuteBarMetadata(object):
             )
 
     def __init__(
-            self,
+        self,
-            default_ohlc_ratio,
+        default_ohlc_ratio,
-            ohlc_ratios_per_sid,
+        ohlc_ratios_per_sid,
-            calendar,
+        calendar,
-            start_session,
+        start_session,
-            end_session,
+        end_session,
-            minutes_per_day,
+        minutes_per_day,
-            version=FORMAT_VERSION,
+        version=FORMAT_VERSION,
     ):
         self.calendar = calendar
         self.start_session = start_session
@@ -291,7 +288,7 @@ class BcolzMinuteBarMetadata(object):
         ohlc_ratio : int
             The default ratio by which to multiply the pricing data to
             convert the floats from floats to an integer to fit within
-            the np.uint64. If ohlc_ratios_per_sid is None or does not
+            the np.uint32. If ohlc_ratios_per_sid is None or does not
             contain a mapping for a given sid, this ratio is used.
         ohlc_ratios_per_sid : dict
              A dict mapping each sid in the output to the factor by
@@ -343,10 +340,10 @@ class BcolzMinuteBarMetadata(object):
             'first_trading_day': str(self.start_session.date()),
             'market_opens': (
                 market_opens.values.astype('datetime64[m]').
-                    astype(np.int64).tolist()),
+                astype(np.int64).tolist()),
             'market_closes': (
                 market_closes.values.astype('datetime64[m]').
-                    astype(np.int64).tolist()),
+                astype(np.int64).tolist()),
         }
         with open(self.metadata_path(rootdir), 'w+') as fp:
             json.dump(metadata, fp)
@@ -375,13 +372,13 @@ class BcolzMinuteBarWriter(object):
         The last trading session in the data set.
     default_ohlc_ratio : int, optional
         The default ratio by which to multiply the pricing data to
-        convert from floats to integers that fit within np.uint64. If
+        convert from floats to integers that fit within np.uint32. If
         ohlc_ratios_per_sid is None or does not contain a mapping for a
-        given sid, this ratio is used. Default is OHLC_RATIO (10^8).
+        given sid, this ratio is used. Default is OHLC_RATIO (1000).
     ohlc_ratios_per_sid : dict, optional
         A dict mapping each sid in the output to the ratio by which to
         multiply the pricing data to convert the floats from floats to
-        an integer to fit within the np.uint64.
+        an integer to fit within the np.uint32.
     expectedlen : int, optional
         The expected length of the dataset, used when creating the initial
         bcolz ctable.
@@ -404,9 +401,11 @@ class BcolzMinuteBarWriter(object):
     Each individual asset's data is stored as a bcolz table with a column for
     each pricing field: (open, high, low, close, volume)
 
-    The open, high, low, close and volume columns are integers which are 10^8 times
+    The open, high, low, and close columns are integers which are 1000 times
     the quoted price, so that the data can represented and stored as an
-    np.uint64, supporting market prices quoted up to the 1/10^8-th place.
+    np.uint32, supporting market prices quoted up to the thousands place.
+
+    volume is a np.uint32 with no mutation of the tens place.
 
     The 'index' for each individual asset are a repeating period of minutes of
     length `minutes_per_day` starting from each market open.
@@ -574,7 +573,7 @@ class BcolzMinuteBarWriter(object):
         if not os.path.exists(sid_containing_dirname):
             # Other sids may have already created the containing directory.
             os.makedirs(sid_containing_dirname)
-        initial_array = np.empty(0, np.uint64)
+        initial_array = np.empty(0, np.uint32)
         table = ctable(
             rootdir=path,
             columns=[
@@ -611,7 +610,7 @@ class BcolzMinuteBarWriter(object):
         minute_offset = len(table) % self._minutes_per_day
         num_to_prepend = numdays * self._minutes_per_day - minute_offset
 
-        prepend_array = np.zeros(num_to_prepend, np.uint64)
+        prepend_array = np.zeros(num_to_prepend, np.uint32)
         # Fill all OHLCV with zeros.
         table.append([prepend_array] * 5)
         table.flush()
@@ -816,11 +815,11 @@ class BcolzMinuteBarWriter(object):
 
         minutes_count = all_minutes_in_window.size
 
-        open_col = np.zeros(minutes_count, dtype=np.uint64)
+        open_col = np.zeros(minutes_count, dtype=np.uint32)
-        high_col = np.zeros(minutes_count, dtype=np.uint64)
+        high_col = np.zeros(minutes_count, dtype=np.uint32)
-        low_col = np.zeros(minutes_count, dtype=np.uint64)
+        low_col = np.zeros(minutes_count, dtype=np.uint32)
-        close_col = np.zeros(minutes_count, dtype=np.uint64)
+        close_col = np.zeros(minutes_count, dtype=np.uint32)
-        vol_col = np.zeros(minutes_count, dtype=np.uint64)
+        vol_col = np.zeros(minutes_count, dtype=np.uint32)
 
         dt_ixs = np.searchsorted(all_minutes_in_window.values,
                                  dts.astype('datetime64[ns]'))
@@ -915,10 +914,10 @@ class BcolzMinuteBarReader(MinuteBarReader):
         )
         self._schedule = self.calendar.schedule[slicer]
         self._market_opens = self._schedule.market_open
-        self._market_open_values = self._market_opens.values. \
+        self._market_open_values = self._market_opens.values.\
             astype('datetime64[m]').astype(np.int64)
         self._market_closes = self._schedule.market_close
-        self._market_close_values = self._market_closes.values. \
+        self._market_close_values = self._market_closes.values.\
             astype('datetime64[m]').astype(np.int64)
 
         self._default_ohlc_inverse = 1.0 / metadata.default_ohlc_ratio
@@ -1126,8 +1125,8 @@ class BcolzMinuteBarReader(MinuteBarReader):
             else:
                 return np.nan
 
-        # if field != 'volume':
+        if field != 'volume':
-        value *= self._ohlc_ratio_inverse_for_sid(sid)
+            value *= self._ohlc_ratio_inverse_for_sid(sid)
         return value
 
     def get_last_traded_dt(self, asset, dt):
@@ -1249,7 +1248,7 @@ class BcolzMinuteBarReader(MinuteBarReader):
             if field != 'volume':
                 out = np.full(shape, np.nan)
             else:
-                out = np.zeros(shape, dtype=np.float64)
+                out = np.zeros(shape, dtype=np.uint32)
 
             for i, sid in enumerate(sids):
                 carray = self._open_minute_file(field, sid)
@@ -1257,17 +1256,17 @@ class BcolzMinuteBarReader(MinuteBarReader):
                 if indices_to_exclude is not None:
                     for excl_start, excl_stop in indices_to_exclude[::-1]:
                         excl_slice = np.s_[
-                                     excl_start - start_idx:excl_stop - start_idx + 1]
+                            excl_start - start_idx:excl_stop - start_idx + 1]
                         values = np.delete(values, excl_slice)
 
                 where = values != 0
                 # first slice down to len(where) because we might not have
                 # written data for all the minutes requested
-                # if field != 'volume':
+                if field != 'volume':
-                out[:len(where), i][where] = (
+                    out[:len(where), i][where] = (
-                    values[where] * self._ohlc_ratio_inverse_for_sid(sid))
+                        values[where] * self._ohlc_ratio_inverse_for_sid(sid))
-                # else:
+                else:
-                #    out[:len(where), i][where] = values[where]
+                    out[:len(where), i][where] = values[where]
 
             results.append(out)
         return results
@@ -1320,9 +1319,9 @@ class H5MinuteBarUpdateWriter(object):
 
     def __init__(self, path, complevel=None, complib=None):
         self._complevel = complevel if complevel \
-                                       is not None else self._COMPLEVEL
+            is not None else self._COMPLEVEL
         self._complib = complib if complib \
-                                   is not None else self._COMPLIB
+            is not None else self._COMPLIB
         self._path = path
 
     def write(self, frames):
@@ -1354,7 +1353,6 @@ class H5MinuteBarUpdateReader(MinuteBarUpdateReader):
     path : str
         The path of the HDF5 file from which to source data.
     """
-
     def __init__(self, path):
         self._panel = pd.read_hdf(path)
 

catalyst/data/resample.py

@@ -156,10 +156,7 @@ class DailyHistoryAggregator(object):
             cache = self._caches[field] = (session, market_open, {})
 
         _, market_open, entries = cache
-        try:
+        market_open = market_open.tz_localize('UTC')
-            market_open = market_open.tz_localize('UTC')
-        except TypeError:
-            market_open = market_open.tz_convert('UTC')
         if dt != market_open:
             prev_dt = dt_value - self._one_min
         else:

catalyst/data/us_equity_pricing.py

@@ -11,9 +11,6 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-
-from __future__ import division     # Python2 req to have division of ints yield float
-
 from errno import ENOENT
 from functools import partial
 from os import remove
@@ -83,9 +80,8 @@ from catalyst.utils.cli import (
 from ._equities import _compute_row_slices, _read_bcolz_data
 from ._adjustments import load_adjustments_from_sqlite
 
-from catalyst.constants import LOG_LEVEL
 
-logger = logbook.Logger('UsEquityPricing', level=LOG_LEVEL)
+logger = logbook.Logger('UsEquityPricing')
 
 OHLC = frozenset(['open', 'high', 'low', 'close'])
 OHLCV = frozenset(['open', 'high', 'low', 'close', 'volume'])
@@ -120,8 +116,6 @@ SQLITE_STOCK_DIVIDEND_PAYOUT_COLUMN_DTYPES = {
 UINT32_MAX = iinfo(uint32).max
 UINT64_MAX = iinfo(uint64).max
 
-PRICE_ADJUSTMENT_FACTOR = 1000000000    # Provides 9 decimals resolution. Also affects _equities.pyx L220
-
 
 def check_uint32_safe(value, colname):
     if value >= UINT32_MAX:
@@ -439,11 +433,11 @@ class BcolzDailyBarWriter(object):
             return raw_data
 
         winsorise_uint64(raw_data, invalid_data_behavior, 'volume', *OHLC)
-        processed = (raw_data[list(OHLC)] * PRICE_ADJUSTMENT_FACTOR).astype('uint64')
+        processed = (raw_data[list(OHLC)] * 1000000).astype('uint64')
         dates = raw_data.index.values.astype('datetime64[s]')
         check_uint32_safe(dates.max().view(np.int64), 'day')
         processed['day'] = dates.astype('uint32')
-        processed['volume'] = (raw_data.volume * PRICE_ADJUSTMENT_FACTOR).astype('uint64')
+        processed['volume'] = raw_data.volume.astype('uint64')
         return ctable.fromdataframe(processed)
 
 
@@ -496,8 +490,9 @@ class BcolzDailyBarReader(SessionBarReader):
 
     The data in these columns is interpreted as follows:
 
-    - Price columns ('open', 'high', 'low', 'close') and Volume are interpreted 
+    - Price columns ('open', 'high', 'low', 'close') are interpreted as 1000 *
-      as 10^9 * as-traded dollar value.
+      as-traded dollar value.
+    - Volume is interpreted as as-traded volume.
     - Day is interpreted as seconds since midnight UTC, Jan 1, 1970.
     - Id is the asset id of the row.
 
@@ -524,6 +519,7 @@ class BcolzDailyBarReader(SessionBarReader):
         # Need to test keeping the entire array in memory for the course of a
         # process first.
         self._spot_cols = {}
+        self.PRICE_ADJUSTMENT_FACTOR = 0.001
         self._read_all_threshold = read_all_threshold
 
     @lazyval
@@ -763,10 +759,13 @@ class BcolzDailyBarReader(SessionBarReader):
         """
         ix = self.sid_day_index(sid, dt)
         price = self._spot_col(field)[ix]
-        if field != 'volume' and price == 0:
+        if field != 'volume':
-            return nan
+            if price == 0:
+                return nan
+            else:
+                return price * 0.000001
         else:
-            return price / PRICE_ADJUSTMENT_FACTOR
+            return price
 
 
 class PanelBarReader(SessionBarReader):

catalyst/examples/arbitrage_with_interface.py

@@ -1,275 +0,0 @@
-from logbook import Logger
-
-from catalyst.api import (
-    record,
-    order,
-    symbol,
-    get_open_orders
-)
-from catalyst.exchange.stats_utils import get_pretty_stats
-from catalyst.utils.run_algo import run_algorithm
-
-algo_namespace = 'arbitrage_eth_btc'
-log = Logger(algo_namespace)
-
-
-def initialize(context):
-    log.info('initializing arbitrage algorithm')
-
-    # The context contains a new "exchanges" attribute which is a dictionary
-    # of exchange objects by exchange name. This allow easy access to the
-    # exchanges.
-    context.buying_exchange = context.exchanges['poloniex']
-    context.selling_exchange = context.exchanges['bitfinex']
-
-    context.trading_pair_symbol = 'eth_btc'
-    context.trading_pairs = dict()
-
-    # Note the second parameter of the symbol() method
-    # Passing the exchange name here returns a TradingPair object including
-    # the exchange information. This allow all other operations using
-    # the TradingPair to target the correct exchange.
-    context.trading_pairs[context.buying_exchange] = \
-        symbol('eth_btc', context.buying_exchange.name)
-
-    context.trading_pairs[context.selling_exchange] = \
-        symbol(context.trading_pair_symbol, context.selling_exchange.name)
-
-    context.entry_points = [
-        dict(gap=0.03, amount=0.05),
-        dict(gap=0.04, amount=0.1),
-        dict(gap=0.05, amount=0.5),
-    ]
-    context.exit_points = [
-        dict(gap=-0.02, amount=0.5),
-    ]
-
-    context.SLIPPAGE_ALLOWED = 0.02
-    pass
-
-
-def place_orders(context, amount, buying_price, selling_price, action):
-    """
-    This method will always place two orders of the same amount to keep
-    the currency position the same as it moves between the two exchanges.
-
-    :param context: TradingAlgorithm
-    :param amount: float
-        The trading pair amount to trade on both exchanges.
-    :param buying_price: float
-        The current trading pair price on the buying exchange.
-    :param selling_price: float
-        The current trading pair price on the selling exchange.
-    :param action: string
-        "enter": buys on the buying exchange and sells on the selling exchange
-        "exit": buys on the selling exchange and sells on the buying exchange
-
-    :return:
-    """
-    if action == 'enter':
-        enter_exchange = context.buying_exchange
-        entry_price = buying_price
-
-        exit_exchange = context.selling_exchange
-        exit_price = selling_price
-
-    elif action == 'exit':
-        enter_exchange = context.selling_exchange
-        entry_price = selling_price
-
-        exit_exchange = context.buying_exchange
-        exit_price = buying_price
-
-    else:
-        raise ValueError('invalid order action')
-
-    base_currency = enter_exchange.base_currency
-    base_currency_amount = enter_exchange.portfolio.cash
-
-    exit_balances = exit_exchange.get_balances()
-    exit_currency = context.trading_pairs[
-        context.selling_exchange].market_currency
-
-    if exit_currency in exit_balances:
-        market_currency_amount = exit_balances[exit_currency]
-    else:
-        log.warn(
-            'the selling exchange {exchange_name} does not hold '
-            'currency {currency}'.format(
-                exchange_name=exit_exchange.name,
-                currency=exit_currency
-            )
-        )
-        return
-
-    if base_currency_amount < (amount * entry_price):
-        adj_amount = base_currency_amount / entry_price
-        log.warn(
-            'not enough {base_currency} ({base_currency_amount}) to buy '
-            '{amount}, adjusting the amount to {adj_amount}'.format(
-                base_currency=base_currency,
-                base_currency_amount=base_currency_amount,
-                amount=amount,
-                adj_amount=adj_amount
-            )
-        )
-        amount = adj_amount
-
-    elif market_currency_amount < amount:
-        log.warn(
-            'not enough {currency} ({currency_amount}) to sell '
-            '{amount}, aborting'.format(
-                currency=exit_currency,
-                currency_amount=market_currency_amount,
-                amount=amount
-            )
-        )
-        return
-
-    adj_buy_price = entry_price * (1 + context.SLIPPAGE_ALLOWED)
-    log.info(
-        'buying {amount} {trading_pair} on {exchange_name} with price '
-        'limit {limit_price}'.format(
-            amount=amount,
-            trading_pair=context.trading_pair_symbol,
-            exchange_name=enter_exchange.name,
-            limit_price=adj_buy_price
-        )
-    )
-    order(
-        asset=context.trading_pairs[enter_exchange],
-        amount=amount,
-        limit_price=adj_buy_price
-    )
-
-    adj_sell_price = exit_price * (1 - context.SLIPPAGE_ALLOWED)
-    log.info(
-        'selling {amount} {trading_pair} on {exchange_name} with price '
-        'limit {limit_price}'.format(
-            amount=-amount,
-            trading_pair=context.trading_pair_symbol,
-            exchange_name=exit_exchange.name,
-            limit_price=adj_sell_price
-        )
-    )
-    order(
-        asset=context.trading_pairs[exit_exchange],
-        amount=-amount,
-        limit_price=adj_sell_price
-    )
-    pass
-
-
-def handle_data(context, data):
-    log.info('handling bar {}'.format(data.current_dt))
-
-    buying_price = data.current(
-        context.trading_pairs[context.buying_exchange], 'price')
-
-    log.info('price on buying exchange {exchange}: {price}'.format(
-        exchange=context.buying_exchange.name.upper(),
-        price=buying_price,
-    ))
-
-    selling_price = data.current(
-        context.trading_pairs[context.selling_exchange], 'price')
-
-    log.info('price on selling exchange {exchange}: {price}'.format(
-        exchange=context.selling_exchange.name.upper(),
-        price=selling_price,
-    ))
-
-    # If for example,
-    #   selling price = 50
-    #   buying price = 25
-    #   expected gap = 1
-
-    # If follows that,
-    #   selling price - buying price / buying price
-    #   50 - 25 / 25 = 1
-    gap = (selling_price - buying_price) / buying_price
-    log.info(
-        'the price gap: {gap} ({gap_percent}%)'.format(
-            gap=gap,
-            gap_percent=gap * 100
-        )
-    )
-    record(buying_price=buying_price, selling_price=selling_price, gap=gap)
-
-    # Waiting for orders to close before initiating new ones
-    for exchange in context.trading_pairs:
-        asset = context.trading_pairs[exchange]
-
-        orders = get_open_orders(asset)
-        if orders:
-            log.info(
-                'found {order_count} open orders on {exchange_name} '
-                'skipping bar until all open orders execute'.format(
-                    order_count=len(orders),
-                    exchange_name=exchange.name
-                )
-            )
-            return
-
-    # Consider the least ambitious entry point first
-    # Override of wider gap is found
-    entry_points = sorted(
-        context.entry_points,
-        key=lambda point: point['gap'],
-    )
-
-    buy_amount = None
-    for entry_point in entry_points:
-        if gap > entry_point['gap']:
-            buy_amount = entry_point['amount']
-
-    if buy_amount:
-        log.info('found buy trigger for amount: {}'.format(buy_amount))
-        place_orders(
-            context=context,
-            amount=buy_amount,
-            buying_price=buying_price,
-            selling_price=selling_price,
-            action='enter'
-        )
-
-    else:
-        # Consider the narrowest exit gap first
-        # Override of wider gap is found
-        exit_points = sorted(
-            context.exit_points,
-            key=lambda point: point['gap'],
-            reverse=True
-        )
-
-        sell_amount = None
-        for exit_point in exit_points:
-            if gap < exit_point['gap']:
-                sell_amount = exit_point['amount']
-
-        if sell_amount:
-            log.info('found sell trigger for amount: {}'.format(sell_amount))
-            place_orders(
-                context=context,
-                amount=sell_amount,
-                buying_price=buying_price,
-                selling_price=selling_price,
-                action='exit'
-            )
-
-
-def analyze(context, stats):
-    log.info('the daily stats:\n{}'.format(get_pretty_stats(stats)))
-    pass
-
-
-run_algorithm(
-    initialize=initialize,
-    handle_data=handle_data,
-    analyze=analyze,
-    exchange_name='poloniex,bitfinex',
-    live=True,
-    algo_namespace=algo_namespace,
-    base_currency='btc',
-    live_graph=False
-)

catalyst/examples/buy_and_hodl.5min.py

@@ -0,0 +1,143 @@
+#!/usr/bin/env python
+#
+# Copyright 2017 Enigma MPC, Inc.
+# Copyright 2015 Quantopian, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from catalyst.finance.slippage import VolumeShareSlippage
+
+from catalyst.api import (
+    order_target_value,
+    symbol,
+    record,
+    cancel_order,
+    get_open_orders,
+    set_slippage,
+)
+
+
+def initialize(context):
+    context.ASSET_NAME = 'USDT_BTC'
+    context.TARGET_HODL_RATIO = 0.8
+    context.RESERVE_RATIO = 1.0 - context.TARGET_HODL_RATIO
+
+    # For all trading pairs in the poloniex bundle, the default denomination
+    # currently supported by Catalyst is 1/1000th of a full coin. Use this
+    # constant to scale the price of up to that of a full coin if desired.
+    context.TICK_SIZE = 1000.0
+
+    context.is_buying = True
+    context.asset = symbol(context.ASSET_NAME)
+
+    context.i = 0
+    
+    set_slippage(equities=VolumeShareSlippage(volume_limit=0.1))
+
+def handle_data(context, data):
+    context.i += 1
+
+    starting_cash = context.portfolio.starting_cash
+    target_hodl_value = context.TARGET_HODL_RATIO * starting_cash
+    reserve_value = context.RESERVE_RATIO * starting_cash
+
+    # Cancel any outstanding orders
+    orders = get_open_orders(context.asset) or []
+    for order in orders:
+        cancel_order(order)
+    
+    # Stop buying after passing the reserve threshold
+    cash = context.portfolio.cash
+    if cash <= reserve_value:
+        context.is_buying = False
+
+    # Retrieve current asset price from pricing data
+    price = data[context.asset].price
+
+    # Check if still buying and could (approximately) afford another purchase
+    if context.is_buying and cash > price:
+        # Place order to make position in asset equal to target_hodl_value
+        order_target_value(
+            context.asset,
+            target_hodl_value,
+            limit_price=price*1.1,
+            stop_price=price*0.9,
+        )
+
+    record(
+        price=price,
+        volume=data[context.asset].volume,
+        cash=cash,
+        starting_cash=context.portfolio.starting_cash,
+        leverage=context.account.leverage,
+    )
+
+def analyze(context=None, results=None):
+    import matplotlib.pyplot as plt
+
+    # Plot the portfolio and asset data.
+    ax1 = plt.subplot(611)
+    results[['portfolio_value']].plot(ax=ax1)
+    ax1.set_ylabel('Portfolio Value (USD)')
+
+    ax2 = plt.subplot(612, sharex=ax1)
+    ax2.set_ylabel('{asset} (USD)'.format(asset=context.ASSET_NAME))
+    (context.TICK_SIZE * results[['price']]).plot(ax=ax2)
+
+    trans = results.ix[[t != [] for t in results.transactions]]
+    buys = trans.ix[
+        [t[0]['amount'] > 0 for t in trans.transactions]
+    ]
+    ax2.plot(
+        buys.index,
+        context.TICK_SIZE * results.price[buys.index],
+        '^',
+        markersize=10,
+        color='g',
+    )
+
+    ax3 = plt.subplot(613, sharex=ax1)
+    results[['leverage', 'alpha', 'beta']].plot(ax=ax3)
+    ax3.set_ylabel('Leverage ')
+
+    ax4 = plt.subplot(614, sharex=ax1)
+    results[['starting_cash', 'cash']].plot(ax=ax4)
+    ax4.set_ylabel('Cash (USD)')
+
+    results[[
+        'treasury',
+        'algorithm',
+        'benchmark',
+    ]] = results[[
+        'treasury_period_return',
+        'algorithm_period_return',
+        'benchmark_period_return',
+    ]]
+
+    ax5 = plt.subplot(615, sharex=ax1)
+    results[[
+        'treasury',
+        'algorithm',
+        'benchmark',
+    ]].plot(ax=ax5)
+    ax5.set_ylabel('Percent Change')
+
+    ax6 = plt.subplot(616, sharex=ax1)
+    (results[['volume']] / context.TICK_SIZE).plot(ax=ax6)
+    ax6.set_ylabel('Volume (mCoins/5min)')
+
+    plt.legend(loc=3)
+
+    # Show the plot.
+    plt.gcf().set_size_inches(18, 8)
+    plt.show()

catalyst/examples/buy_and_hodl.py

@@ -15,6 +15,8 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+from catalyst.finance.slippage import VolumeShareSlippage
+
 from catalyst.api import (
     order_target_value,
     symbol,
@@ -24,7 +26,7 @@ from catalyst.api import (
 )
 
 def initialize(context):
-    context.ASSET_NAME = 'BTC_USDT'
+    context.ASSET_NAME = 'USDT_BTC'
     context.TARGET_HODL_RATIO = 0.8
     context.RESERVE_RATIO = 1.0 - context.TARGET_HODL_RATIO
 
@@ -49,14 +51,14 @@ def handle_data(context, data):
     orders = get_open_orders(context.asset) or []
     for order in orders:
         cancel_order(order)
-
+    
     # Stop buying after passing the reserve threshold
     cash = context.portfolio.cash
     if cash <= reserve_value:
         context.is_buying = False
 
     # Retrieve current asset price from pricing data
-    price = data.current(context.asset, 'price')
+    price = data[context.asset].price
 
     # Check if still buying and could (approximately) afford another purchase
     if context.is_buying and cash > price:
@@ -70,7 +72,7 @@ def handle_data(context, data):
 
     record(
         price=price,
-        volume=data.current(context.asset, 'volume'),
+        volume=data[context.asset].volume,
         cash=cash,
         starting_cash=context.portfolio.starting_cash,
         leverage=context.account.leverage,
@@ -127,11 +129,11 @@ def analyze(context=None, results=None):
     ax5.set_ylabel('Percent Change')
 
     ax6 = plt.subplot(616, sharex=ax1)
-    results[['volume']].plot(ax=ax6)
+    (results[['volume']] / context.TICK_SIZE).plot(ax=ax6)
     ax6.set_ylabel('Volume (mCoins/5min)')
 
     plt.legend(loc=3)
 
     # Show the plot.
     plt.gcf().set_size_inches(18, 8)
-    plt.show()
+    plt.show()

catalyst/examples/buy_btc_simple.py

@@ -1,29 +0,0 @@
-'''
-	This is a very simple example referenced in the beginner's tutorial:
-	https://enigmampc.github.io/catalyst/beginner-tutorial.html
-
-	Run this example, by executing the following from your terminal:
-	catalyst run -f buy_btc_simple.py -x bitfinex --start 2016-1-1 --end 2017-9-30 -o buy_btc_simple_out.pickle
-
-	If you want to run this code using another exchange, make sure that 
-	the asset is available on that exchange. For example, if you were to run 
-	it for exchange Poloniex, you would need to edit the following line:
-
-		context.asset = symbol('btc_usdt')     # note 'usdt' instead of 'usd'
-
-	and specify exchange poloniex as follows:
-
-	catalyst run -f buy_btc_simple.py -x poloniex --start 2016-1-1 --end 2017-9-30 -o buy_btc_simple_out.pickle
-
-	To see which assets are available on each exchange, visit:
-	https://www.enigma.co/catalyst/status
-'''
-
-from catalyst.api import order, record, symbol
-
-def initialize(context):
-    context.asset = symbol('btc_usd')
-
-def handle_data(context, data):
-    order(context.asset, 1)
-    record(btc = data.current(context.asset, 'price'))

catalyst/examples/buy_low_sell_high.py

@@ -1,158 +0,0 @@
-'''
-This algorithm requires an additional library (ta-lib) beyond those required by catalyst.
-Install it first by running: 
-$ pip install TA-Lib
-
-If you get build errors like "fatal error: ta-lib/ta_libc.h: No such file or directory"
-it typically means that it can't find the underlying TA-Lib library and needs to be installed.
-See https://mrjbq7.github.io/ta-lib/install.html for instructions on how to install 
-the required dependencies.
-'''
-
-import talib
-from logbook import Logger
-
-from catalyst.api import (
-    order,
-    order_target_percent,
-    symbol,
-    record,
-    get_open_orders,
-)
-from catalyst.exchange.stats_utils import get_pretty_stats
-
-algo_namespace = 'buy_low_sell_high_xrp'
-log = Logger(algo_namespace)
-
-
-def initialize(context):
-    log.info('initializing algo')
-    context.ASSET_NAME = 'XRP_USDT'
-    context.asset = symbol(context.ASSET_NAME)
-
-    context.TARGET_POSITIONS = 5000
-    context.PROFIT_TARGET = 0.1
-    context.SLIPPAGE_ALLOWED = 0.05
-
-    context.retry_check_open_orders = 10
-    context.retry_update_portfolio = 10
-    context.retry_order = 5
-
-    context.swallow_errors = True
-
-    context.errors = []
-    pass
-
-
-def _handle_data(context, data):
-    prices = data.history(
-        context.asset,
-        fields='price',
-        bar_count=20,
-        frequency='15m'
-    )
-
-    rsi = talib.RSI(prices.values, timeperiod=14)[-1]
-    log.info('got rsi: {}'.format(rsi))
-
-    # Buying more when RSI is low, this should lower our cost basis
-    if rsi <= 30:
-        buy_increment = 50
-    elif rsi <= 40:
-        buy_increment = 20
-    elif rsi <= 70:
-        buy_increment = 5
-    else:
-        buy_increment = None
-
-    cash = context.portfolio.cash
-    log.info('base currency available: {cash}'.format(cash=cash))
-
-    price = data.current(context.asset, 'price')
-    log.info('got price {price}'.format(price=price))
-
-    record(
-        price=price,
-        rsi=rsi,
-    )
-
-    orders = get_open_orders(context.asset)
-    if orders:
-        log.info('skipping bar until all open orders execute')
-        return
-
-    is_buy = False
-    cost_basis = None
-    if context.asset in context.portfolio.positions:
-        position = context.portfolio.positions[context.asset]
-
-        cost_basis = position.cost_basis
-        log.info(
-            'found {amount} positions with cost basis {cost_basis}'.format(
-                amount=position.amount,
-                cost_basis=cost_basis
-            )
-        )
-
-        if position.amount >= context.TARGET_POSITIONS:
-            log.info('reached positions target: {}'.format(position.amount))
-            return
-
-        if price < cost_basis:
-            is_buy = True
-        elif position.amount > 0 and \
-                        price > cost_basis * (1 + context.PROFIT_TARGET):
-            profit = (price * position.amount) - (cost_basis * position.amount)
-            log.info('closing position, taking profit: {}'.format(profit))
-            order_target_percent(
-                asset=context.asset,
-                target=0,
-                limit_price=price * (1 - context.SLIPPAGE_ALLOWED),
-            )
-        else:
-            log.info('no buy or sell opportunity found')
-    else:
-        is_buy = True
-
-    if is_buy:
-        if buy_increment is None:
-            log.info('the rsi is too high to consider buying {}'.format(rsi))
-            return
-
-        if price * buy_increment > cash:
-            log.info('not enough base currency to consider buying')
-            return
-
-        log.info(
-            'buying position cheaper than cost basis {} < {}'.format(
-                price,
-                cost_basis
-            )
-        )
-        order(
-            asset=context.asset,
-            amount=buy_increment,
-            limit_price=price * (1 + context.SLIPPAGE_ALLOWED)
-        )
-
-
-def handle_data(context, data):
-    log.info('handling bar {}'.format(data.current_dt))
-    try:
-        _handle_data(context, data)
-    except Exception as e:
-        log.warn('aborting the bar on error {}'.format(e))
-        context.errors.append(e)
-
-    log.info('completed bar {}, total execution errors {}'.format(
-        data.current_dt,
-        len(context.errors)
-    ))
-
-    if len(context.errors) > 0:
-        log.info('the errors:\n{}'.format(context.errors))
-
-
-def analyze(context, stats):
-    log.info('the daily stats:\n{}'.format(get_pretty_stats(stats)))
-    pass

catalyst/examples/buy_low_sell_high_live.py

@@ -1,168 +0,0 @@
-import talib
-from logbook import Logger
-
-import pandas as pd
-from catalyst.api import (
-    order,
-    order_target_percent,
-    symbol,
-    record,
-    get_open_orders,
-)
-from catalyst.exchange.stats_utils import get_pretty_stats
-from catalyst.utils.run_algo import run_algorithm
-
-algo_namespace = 'buy_the_dip_live'
-log = Logger('buy low sell high')
-
-
-def initialize(context):
-    log.info('initializing algo')
-    context.ASSET_NAME = 'btc_usdt'
-    context.asset = symbol(context.ASSET_NAME)
-
-    context.TARGET_POSITIONS = 30
-    context.PROFIT_TARGET = 0.1
-    context.SLIPPAGE_ALLOWED = 0.02
-
-    context.retry_check_open_orders = 10
-    context.retry_update_portfolio = 10
-    context.retry_order = 5
-
-    context.errors = []
-    pass
-
-
-def _handle_data(context, data):
-    price = data.current(context.asset, 'price')
-    log.info('got price {price}'.format(price=price))
-
-    prices = data.history(
-        context.asset,
-        fields='price',
-        bar_count=20,
-        frequency='1d'
-    )
-    rsi = talib.RSI(prices.values, timeperiod=14)[-1]
-    log.info('got rsi: {}'.format(rsi))
-
-    # Buying more when RSI is low, this should lower our cost basis
-    if rsi <= 30:
-        buy_increment = 1
-    elif rsi <= 40:
-        buy_increment = 0.5
-    elif rsi <= 70:
-        buy_increment = 0.2
-    else:
-        buy_increment = 0.1
-
-    cash = context.portfolio.cash
-    log.info('base currency available: {cash}'.format(cash=cash))
-
-    record(
-        price=price,
-        rsi=rsi,
-    )
-
-    orders = get_open_orders(context.asset)
-    if orders:
-        log.info('skipping bar until all open orders execute')
-        return
-
-    is_buy = False
-    cost_basis = None
-    if context.asset in context.portfolio.positions:
-        position = context.portfolio.positions[context.asset]
-
-        cost_basis = position.cost_basis
-        log.info(
-            'found {amount} positions with cost basis {cost_basis}'.format(
-                amount=position.amount,
-                cost_basis=cost_basis
-            )
-        )
-
-        if position.amount >= context.TARGET_POSITIONS:
-            log.info('reached positions target: {}'.format(position.amount))
-            return
-
-        if price < cost_basis:
-            is_buy = True
-        elif position.amount > 0 and \
-                        price > cost_basis * (1 + context.PROFIT_TARGET):
-            profit = (price * position.amount) - (cost_basis * position.amount)
-            log.info('closing position, taking profit: {}'.format(profit))
-            order_target_percent(
-                asset=context.asset,
-                target=0,
-                limit_price=price * (1 - context.SLIPPAGE_ALLOWED),
-            )
-        else:
-            log.info('no buy or sell opportunity found')
-    else:
-        is_buy = True
-
-    if is_buy:
-        if buy_increment is None:
-            log.info('the rsi is too high to consider buying {}'.format(rsi))
-            return
-
-        if price * buy_increment > cash:
-            log.info('not enough base currency to consider buying')
-            return
-
-        log.info(
-            'buying position cheaper than cost basis {} < {}'.format(
-                price,
-                cost_basis
-            )
-        )
-        order(
-            asset=context.asset,
-            amount=buy_increment,
-            limit_price=price * (1 + context.SLIPPAGE_ALLOWED)
-        )
-
-
-def handle_data(context, data):
-    log.info('handling bar {}'.format(data.current_dt))
-    # try:
-    _handle_data(context, data)
-    # except Exception as e:
-    #     log.warn('aborting the bar on error {}'.format(e))
-    #     context.errors.append(e)
-
-    log.info('completed bar {}, total execution errors {}'.format(
-        data.current_dt,
-        len(context.errors)
-    ))
-
-    if len(context.errors) > 0:
-        log.info('the errors:\n{}'.format(context.errors))
-
-
-def analyze(context, stats):
-    log.info('the daily stats:\n{}'.format(get_pretty_stats(stats)))
-    pass
-
-
-run_algorithm(
-    capital_base=100000,
-    initialize=initialize,
-    handle_data=handle_data,
-    analyze=analyze,
-    exchange_name='poloniex',
-    start=pd.to_datetime('2017-5-01', utc=True),
-    end=pd.to_datetime('2017-10-16', utc=True),
-    base_currency='usdt',
-    data_frequency='daily'
-)
-# run_algorithm(
-#     initialize=initialize,
-#     handle_data=handle_data,
-#     analyze=analyze,
-#     exchange_name='poloniex',
-#     live=True,
-#     algo_namespace=algo_namespace,
-#     base_currency='btc'
-# )

catalyst/examples/dual_vwap.5min.py

@@ -0,0 +1,189 @@
+#!/usr/bin/env python
+#
+# Copyright 2017 Enigma MPC, Inc.
+# Copyright 2014 Quantopian, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from catalyst.api import (
+    order_target_percent,
+    record,
+    symbol,
+    get_open_orders,
+    set_max_leverage,
+    schedule_function,
+    date_rules,
+    time_rules,
+    attach_pipeline,
+    pipeline_output,
+)
+
+from catalyst.pipeline import Pipeline
+from catalyst.pipeline.data import CryptoPricing
+from catalyst.pipeline.factors.crypto import VWAP
+
+
+def initialize(context):
+    context.ASSET_NAME = 'USDT_BTC'
+    context.TARGET_INVESTMENT_RATIO = 0.8
+    context.SHORT_WINDOW = 30 * 288
+    context.LONG_WINDOW = 100 * 288
+
+    # For all trading pairs in the poloniex bundle, the default denomination
+    # currently supported by Catalyst is 1/1000th of a full coin. Use this
+    # constant to scale the price of up to that of a full coin if desired.
+    context.TICK_SIZE = 1000.0
+
+    context.i = 0
+    context.asset = symbol(context.ASSET_NAME)
+
+    set_max_leverage(1.0)
+
+    attach_pipeline(make_pipeline(context), 'vwap_pipeline')
+
+    schedule_function(
+        rebalance,
+        time_rule=time_rules.every_minute(),
+    )
+
+
+def before_trading_start(context, data):
+    context.pipeline_data = pipeline_output('vwap_pipeline')
+
+def make_pipeline(context):
+    return Pipeline(
+        columns={
+            'price': CryptoPricing.open.latest,
+            'volume': CryptoPricing.volume.latest,
+            'short_mavg': VWAP(window_length=context.SHORT_WINDOW),
+            'long_mavg': VWAP(window_length=context.LONG_WINDOW),
+        }
+    )
+
+def rebalance(context, data):
+    context.i += 1
+
+    # skip first LONG_WINDOW bars to fill windows
+    if context.i < context.LONG_WINDOW:
+        return
+
+    # get pipeline data for asset of interest
+    pipeline_data = context.pipeline_data
+    pipeline_data = pipeline_data[pipeline_data.index == context.asset].iloc[0]
+
+    # retrieve long and short moving averages from pipeline
+    short_mavg = pipeline_data.short_mavg
+    long_mavg = pipeline_data.long_mavg
+    price = pipeline_data.price
+    volume = pipeline_data.volume
+
+    # check that order has not already been placed
+    open_orders = get_open_orders()
+    if context.asset not in open_orders:
+        # check that the asset of interest can currently be traded
+        if data.can_trade(context.asset):
+            # adjust portfolio based on comparison of long and short vwap
+            if short_mavg > long_mavg:
+                order_target_percent(
+                    context.asset,
+                    context.TARGET_INVESTMENT_RATIO,
+                )
+            elif short_mavg < long_mavg:
+                order_target_percent(
+                    context.asset,
+                    0.0,
+                )
+
+    record(
+        price=price,
+        cash=context.portfolio.cash,
+        leverage=context.account.leverage,
+        short_mavg=short_mavg,
+        long_mavg=long_mavg,
+        volume=volume,
+    )
+    
+
+
+def analyze(context=None, results=None):
+    import matplotlib.pyplot as plt
+
+    # Plot the portfolio and asset data.
+    ax1 = plt.subplot(611)
+    results[['portfolio_value']].plot(ax=ax1)
+    ax1.set_ylabel('Portfolio value (USD)')
+
+    ax2 = plt.subplot(612, sharex=ax1)
+    ax2.set_ylabel('{asset} (USD)'.format(asset=context.ASSET_NAME))
+    (context.TICK_SIZE*results[['price', 'short_mavg', 'long_mavg']]).plot(ax=ax2)
+
+    trans = results.ix[[t != [] for t in results.transactions]]
+    amounts = [t[0]['amount'] for t in trans.transactions]
+
+    buys = trans.ix[
+        [t[0]['amount'] > 0 for t in trans.transactions]
+    ]
+    sells = trans.ix[
+        [t[0]['amount'] < 0 for t in trans.transactions]
+    ]
+
+    ax2.plot(
+        buys.index,
+        context.TICK_SIZE * results.price[buys.index],
+        '^',
+        markersize=10,
+        color='g',
+    )
+    ax2.plot(
+        sells.index,
+        context.TICK_SIZE * results.price[sells.index],
+        'v',
+        markersize=10,
+        color='r',
+    )
+
+    ax3 = plt.subplot(613, sharex=ax1)
+    results[['leverage', 'alpha', 'beta']].plot(ax=ax3)
+    ax3.set_ylabel('Leverage (USD)')
+
+    ax4 = plt.subplot(614, sharex=ax1)
+    results[['cash']].plot(ax=ax4)
+    ax4.set_ylabel('Cash (USD)')
+
+    results[[
+        'treasury',
+        'algorithm',
+        'benchmark',
+    ]] = results[[
+        'treasury_period_return',
+        'algorithm_period_return',
+        'benchmark_period_return',
+    ]]
+
+    ax5 = plt.subplot(615, sharex=ax1)
+    results[[
+        'treasury',
+        'algorithm',
+        'benchmark',
+    ]].plot(ax=ax5)
+    ax5.set_ylabel('Percent Change')
+
+    ax6 = plt.subplot(616, sharex=ax1)
+    (results[['volume']] / context.TICK_SIZE).plot(ax=ax6)
+    ax6.set_ylabel('Volume (mBTC/day)')
+
+    plt.legend(loc=3)
+
+    # Show the plot.
+    plt.gcf().set_size_inches(18, 8)
+    plt.show()

catalyst/examples/dual_vwap.py

@@ -52,7 +52,7 @@ def initialize(context):
 
     schedule_function(
         rebalance,
-        time_rules=times_rules.every_minute(),
+        date_rule=date_rules.every_day(),
     )
 
 
@@ -178,7 +178,7 @@ def analyze(context=None, results=None):
     ax5.set_ylabel('Percent Change')
 
     ax6 = plt.subplot(616, sharex=ax1)
-    results[['volume']].plot(ax=ax6)
+    (results[['volume']] / context.TICK_SIZE).plot(ax=ax6)
     ax6.set_ylabel('Volume (mBTC/day)')
 
     plt.legend(loc=3)

catalyst/examples/mean_reversion.py

@@ -1,283 +0,0 @@
-# For this example, we're going to write a simple momentum script.  When the 
-# stock goes up quickly, we're going to buy; when it goes down quickly, we're
-# going to sell.  Hopefully we'll ride the waves.
-from datetime import timedelta
-
-import pandas as pd
-import talib
-# To run an algorithm in Catalyst, you need two functions: initialize and
-# handle_data.
-from logbook import Logger
-from talib.common import MA_Type
-
-from catalyst import run_algorithm
-from catalyst.api import symbol, record, order_target_percent, \
-    get_open_orders
-# We give a name to the algorithm which Catalyst will use to persist its state.
-# In this example, Catalyst will create the `.catalyst/data/live_algos`
-# directory. If we stop and start the algorithm, Catalyst will resume its
-# state using the files included in the folder.
-from catalyst.exchange.stats_utils import extract_transactions, trend_direction
-
-algo_namespace = 'momentum'
-log = Logger(algo_namespace)
-
-
-def initialize(context):
-    # This initialize function sets any data or variables that you'll use in
-    # your algorithm.  For instance, you'll want to define the trading pair (or
-    # trading pairs) you want to backtest.  You'll also want to define any
-    # parameters or values you're going to use.
-
-    # In our example, we're looking at Ether in USD Tether.
-    context.eth_btc = symbol('etc_usdt')
-    context.base_price = None
-    context.current_day = None
-    context.trigger = None
-
-
-def handle_data(context, data):
-    # This handle_data function is where the real work is done.  Our data is
-    # minute-level tick data, and each minute is called a frame.  This function
-    # runs on each frame of the data.
-
-    # We flag the first period of each day.
-    # Since cryptocurrencies trade 24/7 the `before_trading_starts` handle
-    # would only execute once. This method works with minute and daily
-    # frequencies.
-    today = data.current_dt.floor('1D')
-    if today != context.current_day:
-        context.traded_today = False
-        context.current_day = today
-
-    # We're computing the volume-weighted-average-price of the security
-    # defined above, in the context.eth_btc variable.  For this example, we're 
-    # using three bars on the 15 min bars.
-
-    # The frequency attribute determine the bar size. We use this convention
-    # for the frequency alias:
-    # http://pandas.pydata.org/pandas-docs/stable/timeseries.html#offset-aliases
-    prices = data.history(
-        context.eth_btc,
-        fields='close',
-        bar_count=50,
-        frequency='15T'
-    )
-
-    # Ta-lib calculates various technical indicator based on price and
-    # volume arrays.
-
-    # In this example, we are comp
-    rsi = talib.RSI(prices.values, timeperiod=14)
-    upper, middle, lower = talib.BBANDS(
-        prices.values,
-        timeperiod=20,
-        nbdevup=2,
-        nbdevdn=2,
-        matype=MA_Type.EMA
-    )
-
-    # We need a variable for the current price of the security to compare to
-    # the average. Since we are requesting two fields, data.current()
-    # returns a DataFrame with
-    current = data.current(context.eth_btc, fields=['close', 'volume'])
-    price = current['close']
-
-    # If base_price is not set, we use the current value. This is the
-    # price at the first bar which we reference to calculate price_change.
-    if context.base_price is None:
-        context.base_price = price
-
-    price_change = (price - context.base_price) / context.base_price
-    cash = context.portfolio.cash
-
-    # Now that we've collected all current data for this frame, we use
-    # the record() method to save it. This data will be available as
-    # a parameter of the analyze() function for further analysis.
-    record(
-        price=price,
-        volume=current['volume'],
-        upper_band=upper[-1],
-        lower_band=lower[-1],
-        price_change=price_change,
-        rsi=rsi[-1],
-        cash=cash
-    )
-
-    # We are trying to avoid over-trading by limiting our trades to
-    # one per day.
-    if context.traded_today:
-        return
-
-    # Since we are using limit orders, some orders may not execute immediately
-    # we wait until all orders are executed before considering more trades.
-    orders = get_open_orders(context.eth_btc)
-    if len(orders) > 0:
-        return
-
-    # Exit if we cannot trade
-    if not data.can_trade(context.eth_btc):
-        return
-
-    # Another powerful built-in feature of the Catalyst backtester is the
-    # portfolio object.  The portfolio object tracks your positions, cash,
-    # cost basis of specific holdings, and more.  In this line, we calculate
-    # how long or short our position is at this minute.   
-    pos_amount = context.portfolio.positions[context.eth_btc].amount
-
-    # In this example, we're using a trigger instead of buying directly after
-    # a signal. Since this is mean reversion, our signals go against the
-    # momentum. Using a trigger allow us to spot the opportunity but trade
-    # only when a trade reversal begins.
-    if context.trigger is not None:
-        # The tread_direction() method determines the trend based on the last
-        # two bars of the series.
-        direction = trend_direction(rsi)
-        if context.trigger[1] == 'buy' and direction == 'up':
-            log.info(
-                '{}: buying - price: {}, rsi: {}, bband: {}'.format(
-                    data.current_dt, price, rsi[-1], lower[-1]
-                )
-            )
-            order_target_percent(context.eth_btc, 1)
-            context.traded_today = True
-            context.trigger = None
-
-        elif context.trigger[1] == 'sell' and direction == 'down':
-            log.info(
-                '{}: selling - price: {}, rsi: {}, bband: {}'.format(
-                    data.current_dt, price, rsi[-1], upper[-1]
-                )
-            )
-            order_target_percent(context.eth_btc, 0)
-            context.traded_today = True
-            context.trigger = None
-
-        # If we found a signal but no trade reversal within two hours, we
-        # reset the trigger.
-        elif context.trigger[0] + timedelta(hours=2) < data.current_dt:
-            context.trigger = None
-
-    else:
-        # Determining the entry and exit signals based on RSI and SMA
-        if rsi[-1] <= 30 and pos_amount == 0:
-            context.trigger = (data.current_dt, 'buy')
-
-        elif rsi[-1] >= 80 and pos_amount > 0:
-            context.trigger = (data.current_dt, 'sell')
-
-
-def analyze(context=None, perf=None):
-    import matplotlib.pyplot as plt
-
-    # The base currency of the algo exchange
-    base_currency = context.exchanges.values()[0].base_currency.upper()
-
-    # Plot the portfolio value over time.
-    ax1 = plt.subplot(611)
-    perf.loc[:, 'portfolio_value'].plot(ax=ax1)
-    ax1.set_ylabel('Portfolio Value ({})'.format(base_currency))
-
-    # Plot the price increase or decrease over time.
-    ax2 = plt.subplot(612, sharex=ax1)
-    perf.loc[:, 'price'].plot(ax=ax2, label='Price')
-    perf.loc[:, 'upper_band'].plot(ax=ax2, label='Upper')
-    perf.loc[:, 'lower_band'].plot(ax=ax2, label='Lower')
-
-    ax2.set_ylabel('{asset} ({base})'.format(
-        asset=context.eth_btc.symbol, base=base_currency
-    ))
-
-    transaction_df = extract_transactions(perf)
-    if not transaction_df.empty:
-        buy_df = transaction_df[transaction_df['amount'] > 0]
-        sell_df = transaction_df[transaction_df['amount'] < 0]
-        ax2.scatter(
-            buy_df.index.to_pydatetime(),
-            perf.loc[buy_df.index, 'price'],
-            marker='^',
-            s=100,
-            c='green',
-            label=''
-        )
-        ax2.scatter(
-            sell_df.index.to_pydatetime(),
-            perf.loc[sell_df.index, 'price'],
-            marker='v',
-            s=100,
-            c='red',
-            label=''
-        )
-
-    ax4 = plt.subplot(613, sharex=ax1)
-    perf.loc[:, 'cash'].plot(
-        ax=ax4, label='Base Currency ({})'.format(base_currency)
-    )
-    ax4.set_ylabel('Cash ({})'.format(base_currency))
-
-    perf['algorithm'] = perf.loc[:, 'algorithm_period_return']
-
-    ax5 = plt.subplot(614, sharex=ax1)
-    perf.loc[:, ['algorithm', 'price_change']].plot(ax=ax5)
-    ax5.set_ylabel('Percent Change')
-
-    ax6 = plt.subplot(615, sharex=ax1)
-    perf.loc[:, 'rsi'].plot(ax=ax6, label='RSI')
-    ax6.axhline(70, color='darkgoldenrod')
-    ax6.axhline(30, color='darkgoldenrod')
-
-    if not transaction_df.empty:
-        ax6.scatter(
-            buy_df.index.to_pydatetime(),
-            perf.loc[buy_df.index, 'rsi'],
-            marker='^',
-            s=100,
-            c='green',
-            label=''
-        )
-        ax6.scatter(
-            sell_df.index.to_pydatetime(),
-            perf.loc[sell_df.index, 'rsi'],
-            marker='v',
-            s=100,
-            c='red',
-            label=''
-        )
-    plt.legend(loc=3)
-
-    # Show the plot.
-    plt.gcf().set_size_inches(18, 8)
-    plt.show()
-    pass
-
-
-if __name__ == '__main__':
-    # The execution mode: backtest or live
-    MODE = 'backtest'
-
-    if MODE == 'backtest':
-        run_algorithm(
-            capital_base=1,
-            data_frequency='minute',
-            initialize=initialize,
-            handle_data=handle_data,
-            analyze=analyze,
-            exchange_name='poloniex',
-            algo_namespace=algo_namespace,
-            base_currency='usdt',
-            start=pd.to_datetime('2017-7-1', utc=True),
-            # end=pd.to_datetime('2017-9-30', utc=True),
-            end=pd.to_datetime('2017-10-31', utc=True),
-        )
-
-    elif MODE == 'live':
-        run_algorithm(
-            initialize=initialize,
-            handle_data=handle_data,
-            analyze=analyze,
-            exchange_name='poloniex',
-            live=True,
-            algo_namespace=algo_namespace,
-            base_currency='usdt',
-            live_graph=True
-        )

catalyst/examples/mean_reversion_simple.py

@@ -1,248 +0,0 @@
-# For this example, we're going to write a simple momentum script.  When the 
-# stock goes up quickly, we're going to buy; when it goes down quickly, we're
-# going to sell.  Hopefully we'll ride the waves.
-from datetime import timedelta
-
-import pandas as pd
-import talib
-# To run an algorithm in Catalyst, you need two functions: initialize and
-# handle_data.
-from logbook import Logger
-from talib.common import MA_Type
-
-from catalyst import run_algorithm
-from catalyst.api import symbol, record, order_target_percent, \
-    get_open_orders
-# We give a name to the algorithm which Catalyst will use to persist its state.
-# In this example, Catalyst will create the `.catalyst/data/live_algos`
-# directory. If we stop and start the algorithm, Catalyst will resume its
-# state using the files included in the folder.
-from catalyst.exchange.stats_utils import extract_transactions, trend_direction
-
-algo_namespace = 'momentum'
-log = Logger(algo_namespace)
-
-
-def initialize(context):
-    # This initialize function sets any data or variables that you'll use in
-    # your algorithm.  For instance, you'll want to define the trading pair (or
-    # trading pairs) you want to backtest.  You'll also want to define any
-    # parameters or values you're going to use.
-
-    # In our example, we're looking at Ether in USD Tether.
-    context.eth_btc = symbol('etc_usdt')
-    context.base_price = None
-    context.current_day = None
-
-
-def handle_data(context, data):
-    # This handle_data function is where the real work is done.  Our data is
-    # minute-level tick data, and each minute is called a frame.  This function
-    # runs on each frame of the data.
-
-    # We flag the first period of each day.
-    # Since cryptocurrencies trade 24/7 the `before_trading_starts` handle
-    # would only execute once. This method works with minute and daily
-    # frequencies.
-    today = data.current_dt.floor('1D')
-    if today != context.current_day:
-        context.traded_today = False
-        context.current_day = today
-
-    # We're computing the volume-weighted-average-price of the security
-    # defined above, in the context.eth_btc variable.  For this example, we're 
-    # using three bars on the 15 min bars.
-
-    # The frequency attribute determine the bar size. We use this convention
-    # for the frequency alias:
-    # http://pandas.pydata.org/pandas-docs/stable/timeseries.html#offset-aliases
-    prices = data.history(
-        context.eth_btc,
-        fields='close',
-        bar_count=50,
-        frequency='15T'
-    )
-
-    # Ta-lib calculates various technical indicator based on price and
-    # volume arrays.
-
-    # In this example, we are comp
-    rsi = talib.RSI(prices.values, timeperiod=14)
-
-    # We need a variable for the current price of the security to compare to
-    # the average. Since we are requesting two fields, data.current()
-    # returns a DataFrame with
-    current = data.current(context.eth_btc, fields=['close', 'volume'])
-    price = current['close']
-
-    # If base_price is not set, we use the current value. This is the
-    # price at the first bar which we reference to calculate price_change.
-    if context.base_price is None:
-        context.base_price = price
-
-    price_change = (price - context.base_price) / context.base_price
-    cash = context.portfolio.cash
-
-    # Now that we've collected all current data for this frame, we use
-    # the record() method to save it. This data will be available as
-    # a parameter of the analyze() function for further analysis.
-    record(
-        price=price,
-        volume=current['volume'],
-        price_change=price_change,
-        rsi=rsi[-1],
-        cash=cash
-    )
-
-    # We are trying to avoid over-trading by limiting our trades to
-    # one per day.
-    if context.traded_today:
-        return
-
-    # Since we are using limit orders, some orders may not execute immediately
-    # we wait until all orders are executed before considering more trades.
-    orders = get_open_orders(context.eth_btc)
-    if len(orders) > 0:
-        return
-
-    # Exit if we cannot trade
-    if not data.can_trade(context.eth_btc):
-        return
-
-    # Another powerful built-in feature of the Catalyst backtester is the
-    # portfolio object.  The portfolio object tracks your positions, cash,
-    # cost basis of specific holdings, and more.  In this line, we calculate
-    # how long or short our position is at this minute.   
-    pos_amount = context.portfolio.positions[context.eth_btc].amount
-
-    if rsi[-1] <= 30 and pos_amount == 0:
-        log.info(
-            '{}: buying - price: {}, rsi: {}'.format(
-                data.current_dt, price, rsi[-1]
-            )
-        )
-        order_target_percent(context.eth_btc, 1)
-        context.traded_today = True
-
-    elif rsi[-1] >= 80 and pos_amount > 0:
-        log.info(
-            '{}: selling - price: {}, rsi: {}'.format(
-                data.current_dt, price, rsi[-1]
-            )
-        )
-        order_target_percent(context.eth_btc, 0)
-        context.traded_today = True
-
-
-def analyze(context=None, perf=None):
-    import matplotlib.pyplot as plt
-
-    # The base currency of the algo exchange
-    base_currency = context.exchanges.values()[0].base_currency.upper()
-
-    # Plot the portfolio value over time.
-    ax1 = plt.subplot(611)
-    perf.loc[:, 'portfolio_value'].plot(ax=ax1)
-    ax1.set_ylabel('Portfolio Value ({})'.format(base_currency))
-
-    # Plot the price increase or decrease over time.
-    ax2 = plt.subplot(612, sharex=ax1)
-    perf.loc[:, 'price'].plot(ax=ax2, label='Price')
-
-    ax2.set_ylabel('{asset} ({base})'.format(
-        asset=context.eth_btc.symbol, base=base_currency
-    ))
-
-    transaction_df = extract_transactions(perf)
-    if not transaction_df.empty:
-        buy_df = transaction_df[transaction_df['amount'] > 0]
-        sell_df = transaction_df[transaction_df['amount'] < 0]
-        ax2.scatter(
-            buy_df.index.to_pydatetime(),
-            perf.loc[buy_df.index, 'price'],
-            marker='^',
-            s=100,
-            c='green',
-            label=''
-        )
-        ax2.scatter(
-            sell_df.index.to_pydatetime(),
-            perf.loc[sell_df.index, 'price'],
-            marker='v',
-            s=100,
-            c='red',
-            label=''
-        )
-
-    ax4 = plt.subplot(613, sharex=ax1)
-    perf.loc[:, 'cash'].plot(
-        ax=ax4, label='Base Currency ({})'.format(base_currency)
-    )
-    ax4.set_ylabel('Cash ({})'.format(base_currency))
-
-    perf['algorithm'] = perf.loc[:, 'algorithm_period_return']
-
-    ax5 = plt.subplot(614, sharex=ax1)
-    perf.loc[:, ['algorithm', 'price_change']].plot(ax=ax5)
-    ax5.set_ylabel('Percent Change')
-
-    ax6 = plt.subplot(615, sharex=ax1)
-    perf.loc[:, 'rsi'].plot(ax=ax6, label='RSI')
-    ax6.axhline(70, color='darkgoldenrod')
-    ax6.axhline(30, color='darkgoldenrod')
-
-    if not transaction_df.empty:
-        ax6.scatter(
-            buy_df.index.to_pydatetime(),
-            perf.loc[buy_df.index, 'rsi'],
-            marker='^',
-            s=100,
-            c='green',
-            label=''
-        )
-        ax6.scatter(
-            sell_df.index.to_pydatetime(),
-            perf.loc[sell_df.index, 'rsi'],
-            marker='v',
-            s=100,
-            c='red',
-            label=''
-        )
-    plt.legend(loc=3)
-
-    # Show the plot.
-    plt.gcf().set_size_inches(18, 8)
-    plt.show()
-    pass
-
-
-if __name__ == '__main__':
-    # The execution mode: backtest or live
-    MODE = 'backtest'
-
-    if MODE == 'backtest':
-        # catalyst run -f catalyst/examples/mean_reversion_simple.py -x poloniex -s 2017-7-1 -e 2017-7-31 -c usdt -n mean-reversion --data-frequency minute --capital-base 10000
-        run_algorithm(
-            capital_base=10000,
-            data_frequency='minute',
-            initialize=initialize,
-            handle_data=handle_data,
-            analyze=analyze,
-            exchange_name='poloniex',
-            algo_namespace=algo_namespace,
-            base_currency='usdt',
-            start=pd.to_datetime('2017-7-1', utc=True),
-            end=pd.to_datetime('2017-7-31', utc=True),
-        )
-
-    elif MODE == 'live':
-        run_algorithm(
-            initialize=initialize,
-            handle_data=handle_data,
-            analyze=analyze,
-            exchange_name='poloniex',
-            live=True,
-            algo_namespace=algo_namespace,
-            base_currency='usdt',
-            live_graph=True
-        )

catalyst/examples/rsi_profit_target.py

@@ -1,276 +0,0 @@
-from datetime import timedelta
-
-import pandas as pd
-import numpy as np
-import talib
-from logbook import Logger
-
-from catalyst.api import (
-    order,
-    symbol,
-    record,
-    get_open_orders,
-)
-from catalyst.exchange.stats_utils import crossover, crossunder
-from catalyst.utils.run_algo import run_algorithm
-
-algo_namespace = 'rsi'
-log = Logger(algo_namespace)
-
-
-def initialize(context):
-    log.info('initializing algo')
-    context.asset = symbol('eth_btc')
-    context.base_price = None
-
-    context.MAX_HOLDINGS = 0.2
-    context.RSI_OVERSOLD = 30
-    context.RSI_OVERSOLD_BBANDS = 45
-    context.RSI_OVERBOUGHT_BBANDS = 55
-    context.SLIPPAGE_ALLOWED = 0.03
-
-    context.TARGET = 0.15
-    context.STOP_LOSS = 0.1
-    context.STOP = 0.03
-    context.position = None
-
-    context.last_bar = None
-
-    context.errors = []
-    pass
-
-
-def _handle_buy_sell_decision(context, data, signal, price):
-    orders = get_open_orders(context.asset)
-    if len(orders) > 0:
-        log.info('skipping bar until all open orders execute')
-        return
-
-    positions = context.portfolio.positions
-    if context.position is None and context.asset in positions:
-        position = positions[context.asset]
-        context.position = dict(
-            cost_basis=position['cost_basis'],
-            amount=position['amount'],
-            stop=None
-        )
-
-    action = None
-    if context.position is not None:
-        cost_basis = context.position['cost_basis']
-        amount = context.position['amount']
-        log.info(
-            'found {amount} positions with cost basis {cost_basis}'.format(
-                amount=amount,
-                cost_basis=cost_basis
-            )
-        )
-        stop = context.position['stop']
-
-        target = cost_basis * (1 + context.TARGET)
-        if price >= target:
-            context.position['cost_basis'] = price
-            context.position['stop'] = context.STOP
-
-        stop_target = context.STOP_LOSS if stop is None else context.STOP
-        if price < cost_basis * (1 - stop_target):
-            log.info('executing stop loss')
-            order(
-                asset=context.asset,
-                amount=-amount,
-                limit_price=price * (1 - context.SLIPPAGE_ALLOWED),
-            )
-            action = 0
-            context.position = None
-
-    else:
-        if signal == 'long':
-            log.info('opening position')
-            buy_amount = context.MAX_HOLDINGS / price
-            order(
-                asset=context.asset,
-                amount=buy_amount,
-                limit_price=price * (1 + context.SLIPPAGE_ALLOWED),
-            )
-            context.position = dict(
-                cost_basis=price,
-                amount=buy_amount,
-                stop=None
-            )
-            action = 0
-
-
-def _handle_data_rsi_only(context, data):
-    price = data.current(context.asset, 'close')
-    log.info('got price {price}'.format(price=price))
-
-    if price is np.nan:
-        log.warn('no pricing data')
-        return
-
-    if context.base_price is None:
-        context.base_price = price
-
-    try:
-        prices = data.history(
-            context.asset,
-            fields='price',
-            bar_count=17,
-            frequency='30T'
-        )
-    except Exception as e:
-        log.warn('historical data not available: '.format(e))
-        return
-
-    rsi = talib.RSI(prices.values, timeperiod=16)[-1]
-    log.info('got rsi {}'.format(rsi))
-
-    signal = None
-    if rsi < context.RSI_OVERSOLD:
-        signal = 'long'
-
-    # Making sure that the price is still current
-    price = data.current(context.asset, 'close')
-    cash = context.portfolio.cash
-    log.info(
-        'base currency available: {cash}, cap: {cap}'.format(
-            cash=cash,
-            cap=context.MAX_HOLDINGS
-        )
-    )
-    volume = data.current(context.asset, 'volume')
-    price_change = (price - context.base_price) / context.base_price
-    record(
-        price=price,
-        price_change=price_change,
-        rsi=rsi,
-        volume=volume,
-        cash=cash,
-        starting_cash=context.portfolio.starting_cash,
-        leverage=context.account.leverage,
-    )
-
-    _handle_buy_sell_decision(context, data, signal, price)
-
-
-def handle_data(context, data):
-    dt = data.current_dt
-
-    if context.last_bar is None or (
-                context.last_bar + timedelta(minutes=15)) <= dt:
-        context.last_bar = dt
-    else:
-        return
-
-    log.info('BAR {}'.format(dt))
-    try:
-        _handle_data_rsi_only(context, data)
-    except Exception as e:
-        log.warn('aborting the bar on error {}'.format(e))
-        context.errors.append(e)
-
-    if len(context.errors) > 0:
-        log.info('the errors:\n{}'.format(context.errors))
-
-
-def analyze(context=None, results=None):
-    import matplotlib.pyplot as plt
-
-    base_currency = context.exchanges.values()[0].base_currency.upper()
-    # Plot the portfolio and asset data.
-    ax1 = plt.subplot(611)
-    results.loc[:, 'portfolio_value'].plot(ax=ax1)
-    ax1.set_ylabel('Portfolio Value ({})'.format(base_currency))
-
-    ax2 = plt.subplot(612, sharex=ax1)
-    results.loc[:, 'price'].plot(ax=ax2)
-    ax2.set_ylabel('{asset} ({base})'.format(
-        asset=context.asset.symbol, base=base_currency
-    ))
-
-    trans = results.loc[[t != [] for t in results.transactions], :]
-    buys = trans.loc[[t[0]['amount'] > 0 for t in trans.transactions], :]
-    sells = trans.loc[[t[0]['amount'] < 0 for t in trans.transactions], :]
-    # buys = results.loc[results['action'] == 1, :]
-    # sells = results.loc[results['action'] == 0, :]
-
-    ax2.plot(
-        buys.index,
-        results.loc[buys.index, 'price'],
-        '^',
-        markersize=10,
-        color='g',
-    )
-    ax2.plot(
-        sells.index,
-        results.loc[sells.index, 'price'],
-        'v',
-        markersize=10,
-        color='r',
-    )
-
-    ax3 = plt.subplot(613, sharex=ax1)
-    results.loc[:, ['alpha', 'beta']].plot(ax=ax3)
-    ax3.set_ylabel('Alpha / Beta ')
-
-    ax4 = plt.subplot(614, sharex=ax1)
-    results.loc[:, ['starting_cash', 'cash']].plot(ax=ax4)
-    ax4.set_ylabel('Base Currency ({})'.format(base_currency))
-
-    results['algorithm'] = results.loc[:, 'algorithm_period_return']
-
-    ax5 = plt.subplot(615, sharex=ax1)
-    results.loc[:, ['algorithm', 'price_change']].plot(ax=ax5)
-    ax5.set_ylabel('Percent Change')
-
-    ax6 = plt.subplot(616, sharex=ax1)
-    results.loc[:, 'rsi'].plot(ax=ax6)
-    ax6.set_ylabel('RSI')
-
-    ax6.plot(
-        buys.index,
-        results.loc[buys.index, 'rsi'],
-        '^',
-        markersize=10,
-        color='g',
-    )
-    ax6.plot(
-        sells.index,
-        results.loc[sells.index, 'rsi'],
-        'v',
-        markersize=10,
-        color='r',
-    )
-
-    plt.legend(loc=3)
-
-    # Show the plot.
-    plt.gcf().set_size_inches(18, 8)
-    plt.show()
-    pass
-
-
-run_algorithm(
-    initialize=initialize,
-    handle_data=handle_data,
-    analyze=analyze,
-    exchange_name='bittrex',
-    live=True,
-    algo_namespace=algo_namespace,
-    base_currency='btc',
-    live_graph=False
-)
-
-# Backtest
-# run_algorithm(
-#     capital_base=0.5,
-#     data_frequency='minute',
-#     initialize=initialize,
-#     handle_data=handle_data,
-#     analyze=analyze,
-#     exchange_name='poloniex',
-#     algo_namespace=algo_namespace,
-#     base_currency='btc',
-#     start=pd.to_datetime('2017-9-1', utc=True),
-#     end=pd.to_datetime('2017-10-1', utc=True),
-# )

catalyst/examples/simple_loop.py

@@ -1,52 +0,0 @@
-import talib
-import pandas as pd
-
-from catalyst import run_algorithm
-from catalyst.api import symbol
-
-
-def initialize(context):
-    print('initializing')
-    context.asset = symbol('swift_btc')
-
-
-def handle_data(context, data):
-    print('handling bar: {}'.format(data.current_dt))
-
-    price = data.current(context.asset, 'close')
-    print('got price {price}'.format(price=price))
-
-    try:
-        prices = data.history(
-            context.asset,
-            fields='price',
-            bar_count=15,
-            frequency='1D'
-        )
-        rsi = talib.RSI(prices.values, timeperiod=14)[-1]
-        print('got rsi: {}'.format(rsi))
-    except Exception as e:
-        print(e)
-
-
-run_algorithm(
-    capital_base=250,
-    start=pd.to_datetime('2015-4-1', utc=True),
-    end=pd.to_datetime('2017-11-1', utc=True),
-    data_frequency='daily',
-    initialize=initialize,
-    handle_data=handle_data,
-    analyze=None,
-    exchange_name='bittrex',
-    algo_namespace='simple_loop',
-    base_currency='btc'
-)
-# run_algorithm(
-#     initialize=initialize,
-#     handle_data=handle_data,
-#     analyze=None,
-#     exchange_name='poloniex',
-#     live=True,
-#     algo_namespace='simple_loop',
-#     base_currency='eth',
-#     live_graph=False

catalyst/examples/simple_universe.py

@@ -1,129 +0,0 @@
-"""
-Requires Catalyst version 0.3.0 or above
-Tested on Catalyst version 0.3.3
-
-These example aims to provide and easy way for users to learn how to collect data from the different exchanges.
-You simply need to specify the exchange and the market that you want to focus on.
-You will all see how to create a universe and filter it base on the exchange and the market you desire.
-
-The example prints out the closing price of all the pairs for a given market-exchange every 30 minutes.
-The example also contains the ohlcv minute data for the past seven days which could be used to create indicators
-Use this as the backbone to create your own trading strategies.
-
-Variables lookback date and date are used to ensure data for a coin existed on the lookback period specified.
-"""
-
-import numpy as np
-import pandas as pd
-from datetime import timedelta
-from catalyst import run_algorithm
-from catalyst.exchange.exchange_utils import get_exchange_symbols
-
-from catalyst.api import (
-    symbols,
-)
-
-
-def initialize(context):
-    context.i = -1  # counts the minutes
-    context.exchange = context.exchanges.values()[0].name.lower()  # exchange name
-    context.base_currency = context.exchanges.values()[0].base_currency.lower()  # market base currency
-
-
-def handle_data(context, data):
-    context.i += 1
-    lookback_days = 7  # 7 days
-
-    # current date formatted into a string
-    today = data.current_dt
-    date, time = today.strftime('%Y-%m-%d %H:%M:%S').split(' ')
-    lookback_date = today - timedelta(days=lookback_days)  # subtract the amount of days specified in lookback
-    lookback_date = lookback_date.strftime('%Y-%m-%d %H:%M:%S').split(' ')[0]  # get only the date as a string
-
-    # update universe everyday
-    new_day = 60 * 24  # assuming data_frequency='minute'
-    if not context.i % new_day:
-        context.universe = universe(context, lookback_date, date)
-
-    # get data every 30 minutes
-    minutes = 30
-    one_day_in_minutes = 1440  # 1440 assumes data_frequency='minute'
-    lookback = one_day_in_minutes / minutes * lookback_days  # get N lookback_days of history data
-    if not context.i % minutes and context.universe:
-        # we iterate for every pair in the current universe
-        for coin in context.coins:
-            pair = str(coin.symbol)
-
-            # 30 minute interval ohlcv data (the standard data required for candlestick or indicators/signals)
-            # 30T means 30 minutes re-sampling of one minute data. change to your desire time interval.
-            opened = fill(data.history(coin, 'open', bar_count=lookback, frequency='30T')).values
-            high = fill(data.history(coin, 'high', bar_count=lookback, frequency='30T')).values
-            low = fill(data.history(coin, 'low', bar_count=lookback, frequency='30T')).values
-            close = fill(data.history(coin, 'price', bar_count=lookback, frequency='30T')).values
-            volume = fill(data.history(coin, 'volume', bar_count=lookback, frequency='30T')).values
-
-            # close[-1] is the equivalent to current price
-            # displays the minute price for each pair every 30 minutes
-            print(today, pair, opened[-1], high[-1], low[-1], close[-1], volume[-1])
-
-            # ----------------------------------------------------------------------------------------------------------
-            # -------------------------------------- Insert Your Strategy Here -----------------------------------------
-            # ----------------------------------------------------------------------------------------------------------
-
-
-def analyze(context=None, results=None):
-    pass
-
-
-# Get the universe for a given exchange and a given base_currency market
-# Example: Poloniex BTC Market
-def universe(context, lookback_date, current_date):
-    json_symbols = get_exchange_symbols(context.exchange)  # get all the pairs for the exchange
-    universe_df = pd.DataFrame.from_dict(json_symbols).transpose().astype(str)  # convert into a dataframe
-    universe_df['base_currency'] = universe_df.apply(lambda row: row.symbol.split('_')[1],
-                                                                       axis=1)
-    universe_df['market_currency'] = universe_df.apply(lambda row: row.symbol.split('_')[0],
-                                                                         axis=1)
-
-    # Filter all the exchange pairs to only the ones for a give base currency
-    universe_df = universe_df[universe_df['base_currency'] == context.base_currency]
-
-    # Filter all the pairs to ensure that pair existed in the current date range
-    universe_df = universe_df[universe_df.start_date < lookback_date]
-    universe_df = universe_df[universe_df.end_daily >= current_date]
-    context.coins = symbols(*universe_df.symbol)  # convert all the pairs to symbols
-
-    # print(universe_df.symbol.tolist())
-    return universe_df.symbol.tolist()
-
-
-# Replace all NA, NAN or infinite values with its nearest value
-def fill(series):
-    if isinstance(series, pd.Series):
-        return series.replace([np.inf, -np.inf], np.nan).ffill().bfill()
-    elif isinstance(series, np.ndarray):
-        return pd.Series(series).replace([np.inf, -np.inf], np.nan).ffill().bfill().values
-    else:
-        return series
-
-
-if __name__ == '__main__':
-    start_date = pd.to_datetime('2017-01-01', utc=True)
-    end_date = pd.to_datetime('2017-11-13', utc=True)
-
-    performance = run_algorithm(start=start_date, end=end_date,
-                                capital_base=100.0,  # amount of base_currency, not always in dollars unless usd
-                                initialize=initialize,
-                                handle_data=handle_data,
-                                analyze=analyze,
-                                exchange_name='poloniex',
-                                data_frequency='minute',
-                                base_currency='btc',
-                                live=False,
-                                live_graph=False,
-                                algo_namespace='simple_universe')
-
-"""
-Run in Terminal (inside catalyst environment):
-python simple_universe.py
-"""

catalyst/exchange/asset_finder_exchange.py

@@ -1,93 +0,0 @@
-from logbook import Logger
-
-from catalyst.constants import LOG_LEVEL
-
-log = Logger('AssetFinderExchange', level=LOG_LEVEL)
-
-
-class AssetFinderExchange(object):
-    def __init__(self):
-        self._asset_cache = {}
-
-    @property
-    def sids(self):
-        """
-        This seems to be used to pre-fetch assets.
-        I don't think that we need this for live-trading.
-        Leaving the list empty.
-        """
-        return list()
-
-    def retrieve_all(self, sids, default_none=False):
-        """
-        Retrieve all assets in `sids`.
-
-        Parameters
-        ----------
-        sids : iterable of int
-            Assets to retrieve.
-        default_none : bool
-            If True, return None for failed lookups.
-            If False, raise `SidsNotFound`.
-
-        Returns
-        -------
-        assets : list[Asset or None]
-            A list of the same length as `sids` containing Assets (or Nones)
-            corresponding to the requested sids.
-
-        Raises
-        ------
-        SidsNotFound
-            When a requested sid is not found and default_none=False.
-        """
-        for sid in sids:
-            if sid in self._asset_cache:
-                log.debug('got asset from cache: {}'.format(sid))
-            else:
-                log.debug('fetching asset: {}'.format(sid))
-        return list()
-
-    def lookup_symbol(self, symbol, exchange, as_of_date=None, fuzzy=False):
-        """Lookup an asset by symbol.
-
-        Parameters
-        ----------
-        symbol : str
-            The ticker symbol to resolve.
-        as_of_date : datetime or None
-            Look up the last owner of this symbol as of this datetime.
-            If ``as_of_date`` is None, then this can only resolve the equity
-            if exactly one equity has ever owned the ticker.
-        fuzzy : bool, optional
-            Should fuzzy symbol matching be used? Fuzzy symbol matching
-            attempts to resolve differences in representations for
-            shareclasses. For example, some people may represent the ``A``
-            shareclass of ``BRK`` as ``BRK.A``, where others could write
-            ``BRK_A``.
-
-        Returns
-        -------
-        equity : Asset
-            The equity that held ``symbol`` on the given ``as_of_date``, or the
-            only equity to hold ``symbol`` if ``as_of_date`` is None.
-
-        Raises
-        ------
-        SymbolNotFound
-            Raised when no equity has ever held the given symbol.
-        MultipleSymbolsFound
-            Raised when no ``as_of_date`` is given and more than one equity
-            has held ``symbol``. This is also raised when ``fuzzy=True`` and
-            there are multiple candidates for the given ``symbol`` on the
-            ``as_of_date``.
-        """
-        log.debug('looking up symbol: {} {}'.format(symbol, exchange.name))
-
-        key = ','.join([exchange.name, symbol])
-        if key in self._asset_cache:
-            return self._asset_cache[key]
-        else:
-            asset = exchange.get_asset(symbol)
-            self._asset_cache[key] = asset
-            return asset

catalyst/exchange/bitfinex/bitfinex.py

@@ -1,700 +0,0 @@
-import base64
-import datetime
-import hashlib
-import hmac
-import json
-import re
-import time
-
-import numpy as np
-import pandas as pd
-import pytz
-import requests
-import six
-from catalyst.assets._assets import TradingPair
-from logbook import Logger
-
-from catalyst.exchange.exchange import Exchange
-from catalyst.exchange.exchange_bundle import ExchangeBundle
-from catalyst.exchange.exchange_errors import (
-    ExchangeRequestError,
-    InvalidHistoryFrequencyError,
-    InvalidOrderStyle, OrderCancelError)
-from catalyst.exchange.exchange_execution import ExchangeLimitOrder, \
-    ExchangeStopLimitOrder, ExchangeStopOrder
-from catalyst.exchange.exchange_utils import get_exchange_symbols_filename, \
-    download_exchange_symbols, get_symbols_string
-from catalyst.finance.order import Order, ORDER_STATUS
-from catalyst.protocol import Account
-
-# Trying to account for REST api instability
-# https://stackoverflow.com/questions/15431044/can-i-set-max-retries-for-requests-request
-requests.adapters.DEFAULT_RETRIES = 20
-
-BITFINEX_URL = 'https://api.bitfinex.com'
-
-from catalyst.constants import LOG_LEVEL
-
-log = Logger('Bitfinex', level=LOG_LEVEL)
-warning_logger = Logger('AlgoWarning')
-
-
-class Bitfinex(Exchange):
-    def __init__(self, key, secret, base_currency, portfolio=None):
-        self.url = BITFINEX_URL
-        self.key = key
-        self.secret = secret.encode('UTF-8')
-        self.name = 'bitfinex'
-        self.color = 'green'
-        self.assets = {}
-        self.load_assets()
-        self.base_currency = base_currency
-        self._portfolio = portfolio
-        self.minute_writer = None
-        self.minute_reader = None
-
-        # The candle limit for each request
-        self.num_candles_limit = 1000
-
-        # Max is 90 but playing it safe
-        # https://www.bitfinex.com/posts/188
-        self.max_requests_per_minute = 80
-        self.request_cpt = dict()
-
-        self.bundle = ExchangeBundle(self)
-
-    def _request(self, operation, data, version='v1'):
-        payload_object = {
-            'request': '/{}/{}'.format(version, operation),
-            'nonce': '{0:f}'.format(time.time() * 1000000),
-            # convert to string
-            'options': {}
-        }
-
-        if data is None:
-            payload_dict = payload_object
-        else:
-            payload_dict = payload_object.copy()
-            payload_dict.update(data)
-
-        payload_json = json.dumps(payload_dict)
-        if six.PY3:
-            payload = base64.b64encode(bytes(payload_json, 'utf-8'))
-        else:
-            payload = base64.b64encode(payload_json)
-
-        m = hmac.new(self.secret, payload, hashlib.sha384)
-        m = m.hexdigest()
-
-        # headers
-        headers = {
-            'X-BFX-APIKEY': self.key,
-            'X-BFX-PAYLOAD': payload,
-            'X-BFX-SIGNATURE': m
-        }
-
-        if data is None:
-            request = requests.get(
-                '{url}/{version}/{operation}'.format(
-                    url=self.url,
-                    version=version,
-                    operation=operation
-                ), data={},
-                headers=headers)
-        else:
-            request = requests.post(
-                '{url}/{version}/{operation}'.format(
-                    url=self.url,
-                    version=version,
-                    operation=operation
-                ),
-                headers=headers)
-
-        return request
-
-    def _get_v2_symbol(self, asset):
-        pair = asset.symbol.split('_')
-        symbol = 't' + pair[0].upper() + pair[1].upper()
-        return symbol
-
-    def _get_v2_symbols(self, assets):
-        """
-        Workaround to support Bitfinex v2
-        TODO: Might require a separate asset dictionary
-
-        :param assets:
-        :return:
-        """
-
-        v2_symbols = []
-        for asset in assets:
-            v2_symbols.append(self._get_v2_symbol(asset))
-
-        return v2_symbols
-
-    def _create_order(self, order_status):
-        """
-        Create a Catalyst order object from a Bitfinex order dictionary
-        :param order_status:
-        :return: Order
-        """
-        if order_status['is_cancelled']:
-            status = ORDER_STATUS.CANCELLED
-        elif not order_status['is_live']:
-            log.info('found executed order {}'.format(order_status))
-            status = ORDER_STATUS.FILLED
-        else:
-            status = ORDER_STATUS.OPEN
-
-        amount = float(order_status['original_amount'])
-        filled = float(order_status['executed_amount'])
-
-        if order_status['side'] == 'sell':
-            amount = -amount
-            filled = -filled
-
-        price = float(order_status['price'])
-        order_type = order_status['type']
-
-        stop_price = None
-        limit_price = None
-
-        # TODO: is this comprehensive enough?
-        if order_type.endswith('limit'):
-            limit_price = price
-        elif order_type.endswith('stop'):
-            stop_price = price
-
-        executed_price = float(order_status['avg_execution_price'])
-
-        # TODO: bitfinex does not specify comission. I could calculate it but not sure if it's worth it.
-        commission = None
-
-        date = pd.Timestamp.utcfromtimestamp(float(order_status['timestamp']))
-        date = pytz.utc.localize(date)
-        order = Order(
-            dt=date,
-            asset=self.assets[order_status['symbol']],
-            amount=amount,
-            stop=stop_price,
-            limit=limit_price,
-            filled=filled,
-            id=str(order_status['id']),
-            commission=commission
-        )
-        order.status = status
-
-        return order, executed_price
-
-    def get_balances(self):
-        log.debug('retrieving wallets balances')
-        try:
-            self.ask_request()
-            response = self._request('balances', None)
-            balances = response.json()
-        except Exception as e:
-            raise ExchangeRequestError(error=e)
-
-        if 'message' in balances:
-            raise ExchangeRequestError(
-                error='unable to fetch balance {}'.format(balances['message'])
-            )
-
-        std_balances = dict()
-        for balance in balances:
-            currency = balance['currency'].lower()
-            std_balances[currency] = float(balance['available'])
-
-        return std_balances
-
-    @property
-    def account(self):
-        account = Account()
-
-        account.settled_cash = None
-        account.accrued_interest = None
-        account.buying_power = None
-        account.equity_with_loan = None
-        account.total_positions_value = None
-        account.total_positions_exposure = None
-        account.regt_equity = None
-        account.regt_margin = None
-        account.initial_margin_requirement = None
-        account.maintenance_margin_requirement = None
-        account.available_funds = None
-        account.excess_liquidity = None
-        account.cushion = None
-        account.day_trades_remaining = None
-        account.leverage = None
-        account.net_leverage = None
-        account.net_liquidation = None
-
-        return account
-
-    @property
-    def time_skew(self):
-        # TODO: research the time skew conditions
-        return pd.Timedelta('0s')
-
-    def get_account(self):
-        # TODO: fetch account data and keep in cache
-        return None
-
-    def get_candles(self, freq, assets, bar_count=None,
-                    start_dt=None, end_dt=None):
-        """
-        Retrieve OHLVC candles from Bitfinex
-
-        :param data_frequency:
-        :param assets:
-        :param bar_count:
-        :return:
-
-        Available Frequencies
-        ---------------------
-        '1m', '5m', '15m', '30m', '1h', '3h', '6h', '12h', '1D', '7D', '14D',
-         '1M'
-        """
-        log.debug(
-            'retrieving {bars} {freq} candles on {exchange} from '
-            '{end_dt} for markets {symbols}, '.format(
-                bars=bar_count,
-                freq=freq,
-                exchange=self.name,
-                end_dt=end_dt,
-                symbols=get_symbols_string(assets)
-            )
-        )
-
-        allowed_frequencies = ['1T', '5T', '15T', '30T', '60T', '180T',
-                               '360T', '720T', '1D', '7D', '14D', '30D']
-        if freq not in allowed_frequencies:
-            raise InvalidHistoryFrequencyError(frequency=freq)
-
-        freq_match = re.match(r'([0-9].*)(T|H|D)', freq, re.M | re.I)
-        if freq_match:
-            number = int(freq_match.group(1))
-            unit = freq_match.group(2)
-
-            if unit == 'T':
-                if number in [60, 180, 360, 720]:
-                    number = number / 60
-                    converted_unit = 'h'
-                else:
-                    converted_unit = 'm'
-            else:
-                converted_unit = unit
-
-            frequency = '{}{}'.format(number, converted_unit)
-
-        else:
-            raise InvalidHistoryFrequencyError(frequency=freq)
-
-        # Making sure that assets are iterable
-        asset_list = [assets] if isinstance(assets, TradingPair) else assets
-        ohlc_map = dict()
-        for asset in asset_list:
-            symbol = self._get_v2_symbol(asset)
-            url = '{url}/v2/candles/trade:{frequency}:{symbol}'.format(
-                url=self.url,
-                frequency=frequency,
-                symbol=symbol
-            )
-
-            if bar_count:
-                is_list = True
-                url += '/hist?limit={}'.format(int(bar_count))
-
-                def get_ms(date):
-                    epoch = datetime.datetime.utcfromtimestamp(0)
-                    epoch = epoch.replace(tzinfo=pytz.UTC)
-
-                    return (date - epoch).total_seconds() * 1000.0
-
-                if start_dt is not None:
-                    start_ms = get_ms(start_dt)
-                    url += '&start={0:f}'.format(start_ms)
-
-                if end_dt is not None:
-                    end_ms = get_ms(end_dt)
-                    url += '&end={0:f}'.format(end_ms)
-
-            else:
-                is_list = False
-                url += '/last'
-
-            try:
-                self.ask_request()
-                response = requests.get(url)
-            except Exception as e:
-                raise ExchangeRequestError(error=e)
-
-            if 'error' in response.content:
-                raise ExchangeRequestError(
-                    error='Unable to retrieve candles: {}'.format(
-                        response.content)
-                )
-
-            candles = response.json()
-
-            def ohlc_from_candle(candle):
-                last_traded = pd.Timestamp.utcfromtimestamp(
-                    candle[0] / 1000.0)
-                last_traded = last_traded.replace(tzinfo=pytz.UTC)
-                ohlc = dict(
-                    open=np.float64(candle[1]),
-                    high=np.float64(candle[3]),
-                    low=np.float64(candle[4]),
-                    close=np.float64(candle[2]),
-                    volume=np.float64(candle[5]),
-                    price=np.float64(candle[2]),
-                    last_traded=last_traded
-                )
-                return ohlc
-
-            if is_list:
-                ohlc_bars = []
-                # We can to list candles from old to new
-                for candle in reversed(candles):
-                    ohlc = ohlc_from_candle(candle)
-                    ohlc_bars.append(ohlc)
-
-                ohlc_map[asset] = ohlc_bars
-
-            else:
-                ohlc = ohlc_from_candle(candles)
-                ohlc_map[asset] = ohlc
-
-        return ohlc_map[assets] \
-            if isinstance(assets, TradingPair) else ohlc_map
-
-    def create_order(self, asset, amount, is_buy, style):
-        """
-        Creating order on the exchange.
-
-        :param asset:
-        :param amount:
-        :param is_buy:
-        :param style:
-        :return:
-        """
-        exchange_symbol = self.get_symbol(asset)
-        if isinstance(style, ExchangeLimitOrder) \
-                or isinstance(style, ExchangeStopLimitOrder):
-            price = style.get_limit_price(is_buy)
-            order_type = 'limit'
-
-        elif isinstance(style, ExchangeStopOrder):
-            price = style.get_stop_price(is_buy)
-            order_type = 'stop'
-
-        else:
-            raise InvalidOrderStyle(exchange=self.name,
-                                    style=style.__class__.__name__)
-
-        req = dict(
-            symbol=exchange_symbol,
-            amount=str(float(abs(amount))),
-            price="{:.20f}".format(float(price)),
-            side='buy' if is_buy else 'sell',
-            type='exchange ' + order_type,  # TODO: support margin trades
-            exchange=self.name,
-            is_hidden=False,
-            is_postonly=False,
-            use_all_available=0,
-            ocoorder=False,
-            buy_price_oco=0,
-            sell_price_oco=0
-        )
-
-        date = pd.Timestamp.utcnow()
-        try:
-            self.ask_request()
-            response = self._request('order/new', req)
-            order_status = response.json()
-        except Exception as e:
-            raise ExchangeRequestError(error=e)
-
-        if 'message' in order_status:
-            raise ExchangeRequestError(
-                error='unable to create Bitfinex order {}'.format(
-                    order_status['message'])
-            )
-
-        order_id = str(order_status['id'])
-        order = Order(
-            dt=date,
-            asset=asset,
-            amount=amount,
-            stop=style.get_stop_price(is_buy),
-            limit=style.get_limit_price(is_buy),
-            id=order_id
-        )
-
-        return order
-
-    def get_open_orders(self, asset=None):
-        """Retrieve all of the current open orders.
-
-        Parameters
-        ----------
-        asset : Asset
-            If passed and not None, return only the open orders for the given
-            asset instead of all open orders.
-
-        Returns
-        -------
-        open_orders : dict[list[Order]] or list[Order]
-            If no asset is passed this will return a dict mapping Assets
-            to a list containing all the open orders for the asset.
-            If an asset is passed then this will return a list of the open
-            orders for this asset.
-        """
-        try:
-            self.ask_request()
-            response = self._request('orders', None)
-            order_statuses = response.json()
-        except Exception as e:
-            raise ExchangeRequestError(error=e)
-
-        if 'message' in order_statuses:
-            raise ExchangeRequestError(
-                error='Unable to retrieve open orders: {}'.format(
-                    order_statuses['message'])
-            )
-
-        orders = []
-        for order_status in order_statuses:
-            order, executed_price = self._create_order(order_status)
-            if asset is None or asset == order.sid:
-                orders.append(order)
-
-        return orders
-
-    def get_order(self, order_id):
-        """Lookup an order based on the order id returned from one of the
-        order functions.
-
-        Parameters
-        ----------
-        order_id : str
-            The unique identifier for the order.
-
-        Returns
-        -------
-        order : Order
-            The order object.
-        """
-        try:
-            self.ask_request()
-            response = self._request(
-                'order/status', {'order_id': int(order_id)})
-            order_status = response.json()
-        except Exception as e:
-            raise ExchangeRequestError(error=e)
-
-        if 'message' in order_status:
-            raise ExchangeRequestError(
-                error='Unable to retrieve order status: {}'.format(
-                    order_status['message'])
-            )
-        return self._create_order(order_status)
-
-    def cancel_order(self, order_param):
-        """Cancel an open order.
-
-        Parameters
-        ----------
-        order_param : str or Order
-            The order_id or order object to cancel.
-        """
-        order_id = order_param.id \
-            if isinstance(order_param, Order) else order_param
-
-        try:
-            self.ask_request()
-            response = self._request('order/cancel', {'order_id': order_id})
-            status = response.json()
-        except Exception as e:
-            raise ExchangeRequestError(error=e)
-
-        if 'message' in status:
-            raise OrderCancelError(
-                order_id=order_id,
-                exchange=self.name,
-                error=status['message']
-            )
-
-    def tickers(self, assets):
-        """
-        Fetch ticket data for assets
-        https://docs.bitfinex.com/v2/reference#rest-public-tickers
-
-        :param assets:
-        :return:
-        """
-        symbols = self._get_v2_symbols(assets)
-        log.debug('fetching tickers {}'.format(symbols))
-
-        try:
-            self.ask_request()
-            response = requests.get(
-                '{url}/v2/tickers?symbols={symbols}'.format(
-                    url=self.url,
-                    symbols=','.join(symbols),
-                )
-            )
-        except Exception as e:
-            raise ExchangeRequestError(error=e)
-
-        if 'error' in response.content:
-            raise ExchangeRequestError(
-                error='Unable to retrieve tickers: {}'.format(
-                    response.content)
-            )
-
-        try:
-            tickers = response.json()
-        except Exception as e:
-            raise ExchangeRequestError(error=e)
-
-        ticks = dict()
-        for index, ticker in enumerate(tickers):
-            if not len(ticker) == 11:
-                raise ExchangeRequestError(
-                    error='Invalid ticker in response: {}'.format(ticker)
-                )
-
-            ticks[assets[index]] = dict(
-                timestamp=pd.Timestamp.utcnow(),
-                bid=ticker[1],
-                ask=ticker[3],
-                last_price=ticker[7],
-                low=ticker[10],
-                high=ticker[9],
-                volume=ticker[8],
-            )
-
-        log.debug('got tickers {}'.format(ticks))
-        return ticks
-
-    def generate_symbols_json(self, filename=None, source_dates=False):
-        symbol_map = {}
-
-        if not source_dates:
-            fn, r = download_exchange_symbols(self.name)
-            with open(fn) as data_file:
-                cached_symbols = json.load(data_file)
-
-        response = self._request('symbols', None)
-
-        for symbol in response.json():
-            if (source_dates):
-                start_date = self.get_symbol_start_date(symbol)
-            else:
-                try:
-                    start_date = cached_symbols[symbol]['start_date']
-                except KeyError as e:
-                    start_date = time.strftime('%Y-%m-%d')
-
-            try:
-                end_daily = cached_symbols[symbol]['end_daily']
-            except KeyError as e:
-                end_daily = 'N/A'
-
-            try:
-                end_minute = cached_symbols[symbol]['end_minute']
-            except KeyError as e:
-                end_minute = 'N/A'
-
-            symbol_map[symbol] = dict(
-                symbol=symbol[:-3] + '_' + symbol[-3:],
-                start_date=start_date,
-                end_daily=end_daily,
-                end_minute=end_minute,
-            )
-
-        if (filename is None):
-            filename = get_exchange_symbols_filename(self.name)
-
-        with open(filename, 'w') as f:
-            json.dump(symbol_map, f, sort_keys=True, indent=2,
-                      separators=(',', ':'))
-
-    def get_symbol_start_date(self, symbol):
-
-        print(symbol)
-        symbol_v2 = 't' + symbol.upper()
-
-        """
-            For each symbol we retrieve candles with Monhtly resolution
-            We get the first month, and query again with daily resolution
-            around that date, and we get the first date
-        """
-        url = '{url}/v2/candles/trade:1M:{symbol}/hist'.format(
-            url=self.url,
-            symbol=symbol_v2
-        )
-
-        try:
-            self.ask_request()
-            response = requests.get(url)
-        except Exception as e:
-            raise ExchangeRequestError(error=e)
-
-        """
-            If we don't get any data back for our monthly-resolution query
-            it means that symbol started trading less than a month ago, so
-            arbitrarily set the ref. date to 15 days ago to be safe with
-            +/- 31 days
-        """
-        if (len(response.json())):
-            startmonth = response.json()[-1][0]
-        else:
-            startmonth = int((time.time() - 15 * 24 * 3600) * 1000)
-
-        """
-            Query again with daily resolution setting the start and end around
-            the startmonth we got above. Avoid end dates greater than now: time.time()
-        """
-        url = '{url}/v2/candles/trade:1D:{symbol}/hist?start={start}&end={end}'.format(
-            url=self.url,
-            symbol=symbol_v2,
-            start=startmonth - 3600 * 24 * 31 * 1000,
-            end=min(startmonth + 3600 * 24 * 31 * 1000,
-                    int(time.time() * 1000))
-        )
-
-        try:
-            self.ask_request()
-            response = requests.get(url)
-        except Exception as e:
-            raise ExchangeRequestError(error=e)
-
-        return time.strftime('%Y-%m-%d',
-                             time.gmtime(int(response.json()[-1][0] / 1000)))
-
-    def get_orderbook(self, asset, order_type='all', limit=100):
-        exchange_symbol = asset.exchange_symbol
-        try:
-            self.ask_request()
-            # TODO: implement limit
-            response = self._request(
-                'book/{}'.format(exchange_symbol), None)
-            data = response.json()
-
-        except Exception as e:
-            raise ExchangeRequestError(error=e)
-
-        # TODO: filter by type
-        result = dict()
-        for order_type in data:
-            result[order_type] = []
-
-            for entry in data[order_type]:
-                result[order_type].append(dict(
-                    rate=float(entry['price']),
-                    quantity=float(entry['amount'])
-                ))
-
-        return result

catalyst/exchange/bitfinex/symbols.json

@@ -1,127 +0,0 @@
-{
-  "neobtc": {
-    "symbol": "neo_btc",
-    "start_date": "2017-09-07",
-    "precision": 5
-  },
-  "neousd": {
-    "symbol": "neo_usd",
-    "start_date": "2017-09-07"
-  },
-  "neoeth": {
-    "symbol": "neo_eth",
-    "start_date": "2017-09-07"
-  },
-  "btcusd": {
-    "symbol": "btc_usd",
-    "start_date": "2010-01-01"
-  },
-  "bchusd": {
-    "symbol": "bch_usd",
-    "start_date": "2010-01-01"
-  },
-  "ltcusd": {
-    "symbol": "ltc_usd",
-    "start_date": "2010-01-01"
-  },
-  "ltcbtc": {
-    "symbol": "ltc_btc",
-    "start_date": "2010-01-01"
-  },
-  "ethusd": {
-    "symbol": "eth_usd",
-    "start_date": "2017-01-01"
-  },
-  "ethbtc": {
-    "symbol": "eth_btc",
-    "start_date": "2017-01-01"
-  },
-  "etcbtc": {
-    "symbol": "etc_btc",
-    "start_date": "2017-01-01"
-  },
-  "etcusd": {
-    "symbol": "etc_usd",
-    "start_date": "2017-01-01"
-  },
-  "rrtusd": {
-    "symbol": "rrt_usd",
-    "start_date": "2010-01-01"
-  },
-  "rrtbtc": {
-    "symbol": "rrt_btc",
-    "start_date": "2010-01-01"
-  },
-  "zecusd": {
-    "symbol": "zec_usd",
-    "start_date": "2010-01-01"
-  },
-  "zecbtc": {
-    "symbol": "zec_btc",
-    "start_date": "2010-01-01"
-  },
-  "xmrusd": {
-    "symbol": "xmr_usd",
-    "start_date": "2010-01-01"
-  },
-  "xmrbtc": {
-    "symbol": "xmr_btc",
-    "start_date": "2010-01-01"
-  },
-  "dshusd": {
-    "symbol": "dsh_usd",
-    "start_date": "2010-01-01"
-  },
-  "dshbtc": {
-    "symbol": "dsh_btc",
-    "start_date": "2010-01-01"
-  },
-  "bccbtc": {
-    "symbol": "bcc_btc",
-    "start_date": "2010-01-01"
-  },
-  "bcubtc": {
-    "symbol": "bcu_btc",
-    "start_date": "2010-01-01"
-  },
-  "bccusd": {
-    "symbol": "bcc_usd",
-    "start_date": "2010-01-01"
-  },
-  "bcuusd": {
-    "symbol": "bcu_usd",
-    "start_date": "2010-01-01"
-  },
-  "xrpusd": {
-    "symbol": "xrp_usd",
-    "start_date": "2010-01-01"
-  },
-  "xrpbtc": {
-    "symbol": "xrp_btc",
-    "start_date": "2010-01-01"
-  },
-  "iotusd": {
-    "symbol": "iot_usd",
-    "start_date": "2010-01-01"
-  },
-  "iotbtc": {
-    "symbol": "iot_btc",
-    "start_date": "2010-01-01"
-  },
-  "ioteth": {
-    "symbol": "iot_eth",
-    "start_date": "2010-01-01"
-  },
-  "eosusd": {
-    "symbol": "eos_usd",
-    "start_date": "2010-01-01"
-  },
-  "eosbtc": {
-    "symbol": "eos_btc",
-    "start_date": "2010-01-01"
-  },
-  "eoseth": {
-    "symbol": "eos_eth",
-    "start_date": "2010-01-01"
-  }
-}

catalyst/exchange/bittrex/bittrex.py

@@ -1,413 +0,0 @@
-import json
-import time
-
-import pandas as pd
-from catalyst.assets._assets import TradingPair
-from logbook import Logger
-from six.moves import urllib
-
-from catalyst.constants import LOG_LEVEL
-from catalyst.exchange.bittrex.bittrex_api import Bittrex_api
-from catalyst.exchange.exchange import Exchange
-from catalyst.exchange.exchange_bundle import ExchangeBundle
-from catalyst.exchange.exchange_errors import InvalidHistoryFrequencyError, \
-    ExchangeRequestError, InvalidOrderStyle, OrderNotFound, OrderCancelError, \
-    CreateOrderError
-from catalyst.exchange.exchange_utils import get_exchange_symbols_filename, \
-    download_exchange_symbols, get_symbols_string
-from catalyst.finance.execution import LimitOrder, StopLimitOrder
-from catalyst.finance.order import Order, ORDER_STATUS
-
-# TODO: consider using this: https://github.com/mondeja/bittrex_v2
-
-log = Logger('Bittrex', level=LOG_LEVEL)
-
-URL2 = 'https://bittrex.com/Api/v2.0'
-
-
-class Bittrex(Exchange):
-    def __init__(self, key, secret, base_currency, portfolio=None):
-        self.api = Bittrex_api(key=key, secret=secret)
-        self.name = 'bittrex'
-        self.color = 'blue'
-        self.base_currency = base_currency
-        self._portfolio = portfolio
-
-        self.num_candles_limit = 2000
-
-        # Not sure what the rate limit is but trying to play it safe
-        # https://bitcoin.stackexchange.com/questions/53778/bittrex-api-rate-limit
-        self.max_requests_per_minute = 60
-        self.request_cpt = dict()
-
-        self.minute_writer = None
-        self.minute_reader = None
-
-        self.assets = dict()
-        self.load_assets()
-
-        self.bundle = ExchangeBundle(self)
-
-    @property
-    def account(self):
-        pass
-
-    @property
-    def time_skew(self):
-        # TODO: research the time skew conditions
-        return pd.Timedelta('0s')
-
-    def sanitize_curency_symbol(self, exchange_symbol):
-        """
-        Helper method used to build the universal pair.
-        Include any symbol mapping here if appropriate.
-
-        :param exchange_symbol:
-        :return universal_symbol:
-        """
-        return exchange_symbol.lower()
-
-    def get_balances(self):
-        balances = self.api.getbalances()
-        try:
-            log.debug('retrieving wallet balances')
-            self.ask_request()
-
-        except Exception as e:
-            raise ExchangeRequestError(error=e)
-
-        std_balances = dict()
-        try:
-            for balance in balances:
-                currency = balance['Currency'].lower()
-                std_balances[currency] = balance['Available']
-
-        except TypeError:
-            raise ExchangeRequestError(error=balances)
-
-        return std_balances
-
-    def create_order(self, asset, amount, is_buy, style):
-        log.info('creating {} order'.format('buy' if is_buy else 'sell'))
-        exchange_symbol = self.get_symbol(asset)
-
-        if isinstance(style, LimitOrder) or isinstance(style, StopLimitOrder):
-            if isinstance(style, StopLimitOrder):
-                log.warn('{} will ignore the stop price'.format(self.name))
-
-            price = style.get_limit_price(is_buy)
-            try:
-                self.ask_request()
-                if is_buy:
-                    order_status = self.api.buylimit(exchange_symbol, amount,
-                                                     price)
-                else:
-                    order_status = self.api.selllimit(exchange_symbol,
-                                                      abs(amount), price)
-            except Exception as e:
-                raise ExchangeRequestError(error=e)
-
-            if 'uuid' in order_status:
-                order_id = order_status['uuid']
-                order = Order(
-                    dt=pd.Timestamp.utcnow(),
-                    asset=asset,
-                    amount=amount,
-                    stop=style.get_stop_price(is_buy),
-                    limit=style.get_limit_price(is_buy),
-                    id=order_id
-                )
-                return order
-            else:
-                if order_status == 'INSUFFICIENT_FUNDS':
-                    log.warn('not enough funds to create order')
-                    return None
-                elif order_status == 'DUST_TRADE_DISALLOWED_MIN_VALUE_50K_SAT':
-                    log.warn('Your order is too small, order at least 50K'
-                             ' Satoshi')
-                    return None
-                else:
-                    raise CreateOrderError(
-                        exchange=self.name,
-                        error=order_status
-                    )
-        else:
-            raise InvalidOrderStyle(exchange=self.name,
-                                    style=style.__class__.__name__)
-
-    def get_open_orders(self, asset):
-        symbol = self.get_symbol(asset)
-        try:
-            self.ask_request()
-            open_orders = self.api.getopenorders(symbol)
-        except Exception as e:
-            raise ExchangeRequestError(error=e)
-
-        orders = list()
-        for order_status in open_orders:
-            order = self._create_order(order_status)
-            orders.append(order)
-
-        return orders
-
-    def _create_order(self, order_status):
-        log.info(
-            'creating catalyst order from Bittrex {}'.format(order_status))
-        if order_status['CancelInitiated']:
-            status = ORDER_STATUS.CANCELLED
-        elif order_status['Closed'] is not None:
-            status = ORDER_STATUS.FILLED
-        else:
-            status = ORDER_STATUS.OPEN
-
-        date = pd.to_datetime(order_status['Opened'], utc=True)
-        amount = order_status['Quantity']
-        filled = amount - order_status['QuantityRemaining']
-        order = Order(
-            dt=date,
-            asset=self.assets[order_status['Exchange']],
-            amount=amount,
-            stop=None,  # Not yet supported by Bittrex
-            limit=order_status['Limit'],
-            filled=filled,
-            id=order_status['OrderUuid'],
-            commission=order_status['CommissionPaid']
-        )
-        order.status = status
-
-        executed_price = order_status['PricePerUnit']
-
-        return order, executed_price
-
-    def get_order(self, order_id):
-        log.info('retrieving order {}'.format(order_id))
-        try:
-            self.ask_request()
-            order_status = self.api.getorder(order_id)
-        except Exception as e:
-            raise ExchangeRequestError(error=e)
-
-        if order_status is None:
-            raise OrderNotFound(order_id=order_id, exchange=self.name)
-
-        return self._create_order(order_status)
-
-    def cancel_order(self, order_param):
-        order_id = order_param.id \
-            if isinstance(order_param, Order) else order_param
-        log.info('cancelling order {}'.format(order_id))
-
-        try:
-            self.ask_request()
-            status = self.api.cancel(order_id)
-        except Exception as e:
-            raise ExchangeRequestError(error=e)
-
-        if 'message' in status:
-            raise OrderCancelError(
-                order_id=order_id,
-                exchange=self.name,
-                error=status['message']
-            )
-
-    def get_candles(self, freq, assets, bar_count=None,
-                    start_dt=None, end_dt=None):
-        """
-        Supported Intervals
-        -------------------
-        day, oneMin, fiveMin, thirtyMin, hour
-
-        :param freq:
-        :param assets:
-        :param bar_count:
-        :param start_dt
-        :param end_dt
-        :return:
-        """
-
-        # TODO: this has no effect at the moment
-        if end_dt is None:
-            end_dt = pd.Timestamp.utcnow()
-
-        log.debug(
-            'retrieving {bars} {freq} candles on {exchange} from '
-            '{end_dt} for markets {symbols}, '.format(
-                bars=bar_count,
-                freq=freq,
-                exchange=self.name,
-                end_dt=end_dt,
-                symbols=get_symbols_string(assets)
-            )
-        )
-
-        if freq == '1T':
-            frequency = 'oneMin'
-        elif freq == '5T':
-            frequency = 'fiveMin'
-        elif freq == '30T':
-            frequency = 'thirtyMin'
-        elif freq == '60T':
-            frequency = 'hour'
-        elif freq == '1D':
-            frequency = 'day'
-        else:
-            raise InvalidHistoryFrequencyError(frequency=freq)
-
-        # Making sure that assets are iterable
-        asset_list = [assets] if isinstance(assets, TradingPair) else assets
-        for asset in asset_list:
-            end = int(time.mktime(end_dt.timetuple()))
-            url = '{url}/pub/market/GetTicks?marketName={symbol}' \
-                  '&tickInterval={frequency}&_={end}'.format(
-                url=URL2,
-                symbol=self.get_symbol(asset),
-                frequency=frequency,
-                end=end
-            )
-
-            try:
-                data = json.loads(urllib.request.urlopen(url).read().decode())
-            except Exception as e:
-                raise ExchangeRequestError(error=e)
-
-            if data['message']:
-                raise ExchangeRequestError(
-                    error='Unable to fetch candles {}'.format(data['message'])
-                )
-
-            candles = data['result']
-
-            def ohlc_from_candle(candle):
-                ohlc = dict(
-                    open=candle['O'],
-                    high=candle['H'],
-                    low=candle['L'],
-                    close=candle['C'],
-                    volume=candle['V'],
-                    price=candle['C'],
-                    last_traded=pd.to_datetime(candle['T'], utc=True)
-                )
-                return ohlc
-
-            ordered_candles = list(reversed(candles))
-            ohlc_map = dict()
-            if bar_count is None:
-                ohlc_map[asset] = ohlc_from_candle(ordered_candles[0])
-            else:
-                # TODO: optimize
-                ohlc_bars = []
-                for candle in ordered_candles[:bar_count]:
-                    ohlc = ohlc_from_candle(candle)
-                    ohlc_bars.append(ohlc)
-
-                ohlc_map[asset] = ohlc_bars
-
-        return ohlc_map[assets] \
-            if isinstance(assets, TradingPair) else ohlc_map
-
-    def tickers(self, assets):
-        """
-        As of v1.1, Bittrex only allows one ticker at the time.
-        So we have to make multiple calls to fetch multiple assets.
-
-        :param assets:
-        :return:
-        """
-        log.info('retrieving tickers')
-
-        ticks = dict()
-        for asset in assets:
-            symbol = self.get_symbol(asset)
-            try:
-                self.ask_request()
-                ticker = self.api.getticker(symbol)
-            except Exception as e:
-                raise ExchangeRequestError(error=e)
-
-            # TODO: catch invalid ticker
-            ticks[asset] = dict(
-                timestamp=pd.Timestamp.utcnow(),
-                bid=ticker['Bid'],
-                ask=ticker['Ask'],
-                last_price=ticker['Last']
-            )
-
-        log.debug('got tickers {}'.format(ticks))
-        return ticks
-
-    def get_account(self):
-        log.info('retrieving account data')
-        pass
-
-    def generate_symbols_json(self, filename=None):
-        symbol_map = {}
-
-        fn, r = download_exchange_symbols(self.name)
-        with open(fn) as data_file:
-            cached_symbols = json.load(data_file)
-
-        markets = self.api.getmarkets()
-        for market in markets:
-            exchange_symbol = market['MarketName']
-            symbol = '{market}_{base}'.format(
-                market=self.sanitize_curency_symbol(market['MarketCurrency']),
-                base=self.sanitize_curency_symbol(market['BaseCurrency'])
-            )
-
-            try:
-                end_daily = cached_symbols[exchange_symbol]['end_daily']
-            except KeyError as e:
-                end_daily = 'N/A'
-
-            try:
-                end_minute = cached_symbols[exchange_symbol]['end_minute']
-            except KeyError as e:
-                end_minute = 'N/A'
-
-            symbol_map[exchange_symbol] = dict(
-                symbol=symbol,
-                start_date=pd.to_datetime(market['Created'],
-                                          utc=True).strftime("%Y-%m-%d"),
-                end_daily=end_daily,
-                end_minute=end_minute,
-            )
-
-        if (filename is None):
-            filename = get_exchange_symbols_filename(self.name)
-
-        with open(filename, 'w') as f:
-            json.dump(symbol_map, f, sort_keys=True, indent=2,
-                      separators=(',', ':'))
-
-    def get_orderbook(self, asset, order_type='all', limit=100):
-        if order_type == 'all':
-            order_type = 'both'
-        elif order_type == 'bid':
-            order_type = 'buy'
-        elif order_type == 'ask':
-            order_type = 'sell'
-        else:
-            raise ValueError('invalid type')
-
-        exchange_symbol = asset.exchange_symbol
-        data = self.api.getorderbook(
-            market=exchange_symbol,
-            type=order_type,
-            depth=100
-        )
-
-        result = dict()
-        for exchange_type in data:
-            if exchange_type == 'buy':
-                order_type = 'bids'
-            elif exchange_type == 'sell':
-                order_type = 'asks'
-
-            result[order_type] = []
-            for entry in data[exchange_type]:
-                result[order_type].append(dict(
-                    rate=entry['Rate'],
-                    quantity=entry['Quantity']
-                ))
-
-        return result

catalyst/exchange/bittrex/bittrex_api.py

@@ -1,132 +0,0 @@
-#!/usr/bin/env python
-import json
-import time
-import hmac
-import hashlib
-import ssl
-
-# Workaround for backwards compatibility
-# https://stackoverflow.com/questions/3745771/urllib-request-in-python-2-7
-from six.moves import urllib
-
-urlopen = urllib.request.urlopen
-
-
-class Bittrex_api(object):
-    def __init__(self, key, secret):
-        self.key = key
-        self.secret = secret
-        self.public = ['getmarkets', 'getcurrencies', 'getticker',
-                       'getmarketsummaries', 'getmarketsummary',
-                       'getorderbook', 'getmarkethistory']
-        self.market = ['buylimit', 'buymarket', 'selllimit', 'sellmarket',
-                       'cancel', 'getopenorders']
-        self.account = ['getbalances', 'getbalance', 'getdepositaddress',
-                        'withdraw', 'getorder', 'getorderhistory',
-                        'getwithdrawalhistory', 'getdeposithistory']
-
-    def query(self, method, values={}):
-        if method in self.public:
-            url = 'https://bittrex.com/api/v1.1/public/'
-        elif method in self.market:
-            url = 'https://bittrex.com/api/v1.1/market/'
-        elif method in self.account:
-            url = 'https://bittrex.com/api/v1.1/account/'
-        else:
-            return 'Something went wrong, sorry.'
-
-        url += method + '?' + urllib.parse.urlencode(values)
-
-        if method not in self.public:
-            url += '&apikey=' + self.key
-            url += '&nonce=' + str(int(time.time()))
-
-            signature = hmac.new(self.secret.encode('utf-8'),
-                                 url.encode('utf-8'),
-                                 hashlib.sha512).hexdigest()
-            headers = {'apisign': signature}
-        else:
-            headers = {}
-
-        req = urllib.request.Request(url, headers=headers)
-        response = json.loads(urlopen(
-            req, context=ssl._create_unverified_context()).read())
-
-        if response["result"]:
-            return response["result"]
-        else:
-            return response["message"]
-
-    def getmarkets(self):
-        return self.query('getmarkets')
-
-    def getcurrencies(self):
-        return self.query('getcurrencies')
-
-    def getticker(self, market):
-        return self.query('getticker', {'market': market})
-
-    def getmarketsummaries(self):
-        return self.query('getmarketsummaries')
-
-    def getmarketsummary(self, market):
-        return self.query('getmarketsummary', {'market': market})
-
-    def getorderbook(self, market, type, depth=20):
-        return self.query('getorderbook',
-                          {'market': market, 'type': type, 'depth': depth})
-
-    def getmarkethistory(self, market, count=20):
-        return self.query('getmarkethistory',
-                          {'market': market, 'count': count})
-
-    def buylimit(self, market, quantity, rate):
-        return self.query('buylimit', {'market': market, 'quantity': quantity,
-                                       'rate': rate})
-
-    def buymarket(self, market, quantity):
-        return self.query('buymarket',
-                          {'market': market, 'quantity': quantity})
-
-    def selllimit(self, market, quantity, rate):
-        return self.query('selllimit', {'market': market, 'quantity': quantity,
-                                        'rate': rate})
-
-    def sellmarket(self, market, quantity):
-        return self.query('sellmarket',
-                          {'market': market, 'quantity': quantity})
-
-    def cancel(self, uuid):
-        return self.query('cancel', {'uuid': uuid})
-
-    def getopenorders(self, market):
-        return self.query('getopenorders', {'market': market})
-
-    def getbalances(self):
-        return self.query('getbalances')
-
-    def getbalance(self, currency):
-        return self.query('getbalance', {'currency': currency})
-
-    def getdepositaddress(self, currency):
-        return self.query('getdepositaddress', {'currency': currency})
-
-    def withdraw(self, currency, quantity, address):
-        return self.query('withdraw',
-                          {'currency': currency, 'quantity': quantity,
-                           'address': address})
-
-    def getorder(self, uuid):
-        return self.query('getorder', {'uuid': uuid})
-
-    def getorderhistory(self, market, count):
-        return self.query('getorderhistory',
-                          {'market': market, 'count': count})
-
-    def getwithdrawalhistory(self, currency, count):
-        return self.query('getwithdrawalhistory',
-                          {'currency': currency, 'count': count})
-
-    def getdeposithistory(self, currency, count):
-        return self.query('getdeposithistory',
-                          {'currency': currency, 'count': count})

catalyst/exchange/bittrex/extensions-example.py

@@ -1,7 +0,0 @@
-from catalyst.data.bundles import register
-from catalyst.exchange.exchange_bundle import exchange_bundle
-
-symbols = (
-    'neo_btc',
-)
-register('exchange_bitfinex', exchange_bundle('bitfinex', symbols))

catalyst/exchange/bundle_utils.py

@@ -1,319 +0,0 @@
-import calendar
-import os
-import tarfile
-from datetime import timedelta, datetime, date
-
-import numpy as np
-import pandas as pd
-import pytz
-
-from catalyst.data.bundles.core import download_without_progress
-from catalyst.exchange.exchange_utils import get_exchange_bundles_folder
-
-EXCHANGE_NAMES = ['bitfinex', 'bittrex', 'poloniex']
-API_URL = 'http://data.enigma.co/api/v1'
-
-
-def get_date_from_ms(ms):
-    """
-    The date from the number of miliseconds from the epoch.
-
-    Parameters
-    ----------
-    ms: int
-
-    Returns
-    -------
-    datetime
-
-    """
-    return datetime.fromtimestamp(ms / 1000.0)
-
-
-def get_seconds_from_date(date):
-    """
-    The number of seconds from the epoch.
-
-    Parameters
-    ----------
-    date: datetime
-
-    Returns
-    -------
-    int
-
-    """
-    epoch = datetime.utcfromtimestamp(0)
-    epoch = epoch.replace(tzinfo=pytz.UTC)
-
-    return int((date - epoch).total_seconds())
-
-
-def get_bcolz_chunk(exchange_name, symbol, data_frequency, period):
-    """
-    Download and extract a bcolz bundle.
-
-    Parameters
-    ----------
-    exchange_name: str
-    symbol: str
-    data_frequency: str
-    period: str
-
-    Returns
-    -------
-    str
-        Filename: bitfinex-daily-neo_eth-2017-10.tar.gz
-
-    """
-    root = get_exchange_bundles_folder(exchange_name)
-    name = '{exchange}-{frequency}-{symbol}-{period}'.format(
-        exchange=exchange_name,
-        frequency=data_frequency,
-        symbol=symbol,
-        period=period
-    )
-    path = os.path.join(root, name)
-
-    if not os.path.isdir(path):
-        url = 'https://s3.amazonaws.com/enigmaco/catalyst-bundles/' \
-              'exchange-{exchange}/{name}.tar.gz'.format(
-            exchange=exchange_name,
-            name=name
-        )
-
-        bytes = download_without_progress(url)
-        with tarfile.open('r', fileobj=bytes) as tar:
-            tar.extractall(path)
-
-    return path
-
-
-def get_delta(periods, data_frequency):
-    """
-    Get a time delta based on the specified data frequency.
-
-    Parameters
-    ----------
-    periods: int
-    data_frequency: str
-
-    Returns
-    -------
-    timedelta
-
-    """
-    return timedelta(minutes=periods) \
-        if data_frequency == 'minute' else timedelta(days=periods)
-
-
-def get_periods_range(start_dt, end_dt, freq):
-    """
-    Get a date range for the specified parameters.
-
-    Parameters
-    ----------
-    start_dt: datetime
-    end_dt: datetime
-    freq: str
-
-    Returns
-    -------
-    DateTimeIndex
-
-    """
-    if freq == 'minute':
-        freq = 'T'
-
-    elif freq == 'daily':
-        freq = 'D'
-
-    return pd.date_range(start_dt, end_dt, freq=freq)
-
-
-def get_periods(start_dt, end_dt, freq):
-    """
-    The number of periods in the specified range.
-
-    Parameters
-    ----------
-    start_dt: datetime
-    end_dt: datetime
-    freq: str
-
-    Returns
-    -------
-    int
-
-    """
-    return len(get_periods_range(start_dt, end_dt, freq))
-
-
-def get_start_dt(end_dt, bar_count, data_frequency, include_first=True):
-    """
-    The start date based on specified end date and data frequency.
-
-    Parameters
-    ----------
-    end_dt: datetime
-    bar_count: int
-    data_frequency: str
-
-    Returns
-    -------
-    datetime
-
-    """
-    periods = bar_count
-    if periods > 1:
-        delta = get_delta(periods, data_frequency)
-        start_dt = end_dt - delta
-
-        if not include_first:
-            start_dt += get_delta(1, data_frequency)
-    else:
-        start_dt = end_dt
-
-    return start_dt
-
-
-def get_period_label(dt, data_frequency):
-    """
-    The period label for the specified date and frequency.
-
-    Parameters
-    ----------
-    dt: datetime
-    data_frequency: str
-
-    Returns
-    -------
-    str
-
-    """
-    return '{}-{:02d}'.format(dt.year, dt.month) if data_frequency == 'minute' \
-        else '{}'.format(dt.year)
-
-
-def get_month_start_end(dt, first_day=None, last_day=None):
-    """
-    The first and last day of the month for the specified date.
-
-    Parameters
-    ----------
-    dt: datetime
-    first_day: datetime
-    last_day: datetime
-
-    Returns
-    -------
-    datetime, datetime
-
-    """
-    month_range = calendar.monthrange(dt.year, dt.month)
-
-    if first_day:
-        month_start = first_day
-    else:
-        month_start = pd.to_datetime(datetime(
-            dt.year, dt.month, 1, 0, 0, 0, 0
-        ), utc=True)
-
-    if last_day:
-        month_end = last_day
-    else:
-        month_end = pd.to_datetime(datetime(
-            dt.year, dt.month, month_range[1], 23, 59, 0, 0
-        ), utc=True)
-
-        if month_end > pd.Timestamp.utcnow():
-            month_end = pd.Timestamp.utcnow().floor('1D')
-
-    return month_start, month_end
-
-
-def get_year_start_end(dt, first_day=None, last_day=None):
-    """
-    The first and last day of the year for the specified date.
-
-    Parameters
-    ----------
-
-    dt: datetime
-    first_day: datetime
-    last_day: datetime
-
-    Returns
-    -------
-    datetime, datetime
-
-    """
-    year_start = first_day if first_day \
-        else pd.to_datetime(date(dt.year, 1, 1), utc=True)
-    year_end = last_day if last_day \
-        else pd.to_datetime(date(dt.year, 12, 31), utc=True)
-
-    if year_end > pd.Timestamp.utcnow():
-        year_end = pd.Timestamp.utcnow().floor('1D')
-
-    return year_start, year_end
-
-
-def get_df_from_arrays(arrays, periods):
-    """
-    A DataFrame from the specified OHCLV arrays.
-
-    Parameters
-    ----------
-    arrays: Object
-    periods: DateTimeIndex
-
-    Returns
-    -------
-    DataFrame
-
-    """
-    ohlcv = dict()
-    for index, field in enumerate(
-            ['open', 'high', 'low', 'close', 'volume']):
-        ohlcv[field] = arrays[index].flatten()
-
-    df = pd.DataFrame(
-        data=ohlcv,
-        index=periods
-    )
-    return df
-
-
-def range_in_bundle(asset, start_dt, end_dt, reader):
-    """
-    Evaluate whether price data of an asset is included has been ingested in
-    the exchange bundle for the given date range.
-
-    Parameters
-    ----------
-    asset: TradingPair
-    start_dt: datetime
-    end_dt: datetime
-    reader: BcolzBarMinuteReader
-
-    Returns
-    -------
-    bool
-
-    """
-    has_data = True
-    dates = [start_dt, end_dt]
-
-    while dates and has_data:
-        try:
-            dt = dates.pop(0)
-            close = reader.get_value(asset.sid, dt, 'close')
-
-            if np.isnan(close):
-                has_data = False
-
-        except Exception as e:
-            has_data = False
-
-    return has_data

catalyst/exchange/exchange.py

@@ -1,950 +0,0 @@
-import abc
-from abc import ABCMeta, abstractmethod, abstractproperty
-from datetime import timedelta
-from time import sleep
-
-import numpy as np
-import pandas as pd
-from catalyst.assets._assets import TradingPair
-from logbook import Logger
-
-from catalyst.constants import LOG_LEVEL
-from catalyst.data.data_portal import BASE_FIELDS
-from catalyst.exchange.bundle_utils import get_start_dt, \
-    get_delta, get_periods, get_periods_range
-from catalyst.exchange.exchange_bundle import ExchangeBundle
-from catalyst.exchange.exchange_errors import MismatchingBaseCurrencies, \
-    InvalidOrderStyle, BaseCurrencyNotFoundError, SymbolNotFoundOnExchange, \
-    PricingDataNotLoadedError, \
-    NoDataAvailableOnExchange
-from catalyst.exchange.exchange_execution import ExchangeStopLimitOrder, \
-    ExchangeLimitOrder, ExchangeStopOrder
-from catalyst.exchange.exchange_portfolio import ExchangePortfolio
-from catalyst.exchange.exchange_utils import get_exchange_symbols, \
-    get_frequency, resample_history_df
-from catalyst.finance.order import ORDER_STATUS
-from catalyst.finance.transaction import Transaction
-from catalyst.utils.deprecate import deprecated
-
-log = Logger('Exchange', level=LOG_LEVEL)
-
-
-class Exchange:
-    __metaclass__ = ABCMeta
-
-    def __init__(self):
-        self.name = None
-        self.assets = {}
-        self._portfolio = None
-        self.minute_writer = None
-        self.minute_reader = None
-        self.base_currency = None
-
-        self.num_candles_limit = None
-        self.max_requests_per_minute = None
-        self.request_cpt = None
-        self.bundle = ExchangeBundle(self)
-
-    @property
-    def positions(self):
-        return self.portfolio.positions
-
-    @property
-    def portfolio(self):
-        """
-        The exchange portfolio
-
-        Returns
-        -------
-        ExchangePortfolio
-        """
-        if self._portfolio is None:
-            self._portfolio = ExchangePortfolio(
-                start_date=pd.Timestamp.utcnow()
-            )
-            self.synchronize_portfolio()
-
-        return self._portfolio
-
-    @abstractproperty
-    def account(self):
-        pass
-
-    @abstractproperty
-    def time_skew(self):
-        pass
-
-    def is_open(self, dt):
-        """
-        Is the exchange open
-
-        Parameters
-        ----------
-        dt: Timestamp
-
-        Returns
-        -------
-        bool
-
-        """
-        # TODO: implement for each exchange.
-        return True
-
-    def ask_request(self):
-        """
-        Asks permission to issue a request to the exchange.
-        The primary purpose is to avoid hitting rate limits.
-
-        The application will pause if the maximum requests per minute
-        permitted by the exchange is exceeded.
-
-        Returns
-        -------
-        bool
-
-        """
-        now = pd.Timestamp.utcnow()
-        if not self.request_cpt:
-            self.request_cpt = dict()
-            self.request_cpt[now] = 0
-            return True
-
-        cpt_date = list(self.request_cpt.keys())[0]
-        cpt = self.request_cpt[cpt_date]
-
-        if now > cpt_date + timedelta(minutes=1):
-            self.request_cpt = dict()
-            self.request_cpt[now] = 0
-            return True
-
-        if cpt >= self.max_requests_per_minute:
-            delta = now - cpt_date
-
-            sleep_period = 60 - delta.total_seconds()
-            sleep(sleep_period)
-
-            now = pd.Timestamp.utcnow()
-            self.request_cpt = dict()
-            self.request_cpt[now] = 0
-            return True
-        else:
-            self.request_cpt[cpt_date] += 1
-
-    def get_symbol(self, asset):
-        """
-        The the exchange specific symbol of the specified market.
-
-        Parameters
-        ----------
-        asset: TradingPair
-
-        Returns
-        -------
-        str
-
-        """
-        symbol = None
-
-        for key in self.assets:
-            if not symbol and self.assets[key].symbol == asset.symbol:
-                symbol = key
-
-        if not symbol:
-            raise ValueError('Currency %s not supported by exchange %s' %
-                             (asset['symbol'], self.name.title()))
-
-        return symbol
-
-    def get_symbols(self, assets):
-        """
-        Get a list of symbols corresponding to each given asset.
-
-        Parameters
-        ----------
-        assets: list[TradingPair]
-
-        Returns
-        -------
-        list[str]
-
-        """
-        symbols = []
-        for asset in assets:
-            symbols.append(self.get_symbol(asset))
-
-        return symbols
-
-    def get_assets(self, symbols=None):
-        """
-        The list of markets for the specified symbols.
-
-        Parameters
-        ----------
-        symbols: list[str]
-
-        Returns
-        -------
-        list[TradingPair]
-
-        """
-        assets = []
-
-        if symbols is not None:
-            for symbol in symbols:
-                asset = self.get_asset(symbol)
-                assets.append(asset)
-        else:
-            for key in self.assets:
-                assets.append(self.assets[key])
-
-        return assets
-
-    def get_asset(self, symbol):
-        """
-        The market for the specified symbol.
-
-        Parameters
-        ----------
-        symbol: str
-
-        Returns
-        -------
-        TradingPair
-
-        """
-        asset = None
-
-        for key in self.assets:
-            if not asset and self.assets[key].symbol.lower() == symbol.lower():
-                asset = self.assets[key]
-
-        if not asset:
-            supported_symbols = [
-                pair.symbol for pair in list(self.assets.values())
-            ]
-
-            raise SymbolNotFoundOnExchange(
-                symbol=symbol,
-                exchange=self.name.title(),
-                supported_symbols=supported_symbols
-            )
-
-        return asset
-
-    def fetch_symbol_map(self):
-        return get_exchange_symbols(self.name)
-
-    def load_assets(self):
-        """
-        Populate the 'assets' attribute with a dictionary of Assets.
-        The key of the resulting dictionary is the exchange specific
-        currency pair symbol. The universal symbol is contained in the
-        'symbol' attribute of each asset.
-
-        Notes
-        -----
-        The sid of each asset is calculated based on a numeric hash of the
-        universal symbol. This simple approach avoids maintaining a mapping
-        of sids.
-
-        This method can be overridden if an exchange offers equivalent data
-        via its api.
-
-        """
-        symbol_map = self.fetch_symbol_map()
-        for exchange_symbol in symbol_map:
-            asset = symbol_map[exchange_symbol]
-
-            if 'start_date' in asset:
-                start_date = pd.to_datetime(asset['start_date'], utc=True)
-            else:
-                start_date = None
-
-            if 'end_date' in asset:
-                end_date = pd.to_datetime(asset['end_date'], utc=True)
-            else:
-                end_date = None
-
-            if 'leverage' in asset:
-                leverage = asset['leverage']
-            else:
-                leverage = 1.0
-
-            if 'asset_name' in asset:
-                asset_name = asset['asset_name']
-            else:
-                asset_name = None
-
-            if 'min_trade_size' in asset:
-                min_trade_size = asset['min_trade_size']
-            else:
-                min_trade_size = 0.0000001
-
-            if 'end_daily' in asset and asset['end_daily'] != 'N/A':
-                end_daily = pd.to_datetime(asset['end_daily'], utc=True)
-            else:
-                end_daily = None
-
-            if 'end_minute' in asset and asset['end_minute'] != 'N/A':
-                end_minute = pd.to_datetime(asset['end_minute'], utc=True)
-            else:
-                end_minute = None
-
-            trading_pair = TradingPair(
-                symbol=asset['symbol'],
-                exchange=self.name,
-                start_date=start_date,
-                end_date=end_date,
-                leverage=leverage,
-                asset_name=asset_name,
-                min_trade_size=min_trade_size,
-                end_daily=end_daily,
-                end_minute=end_minute,
-                exchange_symbol=exchange_symbol
-            )
-
-            self.assets[exchange_symbol] = trading_pair
-
-    def check_open_orders(self):
-        """
-        Loop through the list of open orders in the Portfolio object.
-        For each executed order found, create a transaction and apply to the
-        Portfolio.
-
-        Returns
-        -------
-        list[Transaction]
-
-        """
-        transactions = list()
-        if self.portfolio.open_orders:
-            for order_id in list(self.portfolio.open_orders):
-                log.debug('found open order: {}'.format(order_id))
-
-                order, executed_price = self.get_order(order_id)
-                log.debug('got updated order {} {}'.format(
-                    order, executed_price))
-
-                if order.status == ORDER_STATUS.FILLED:
-                    transaction = Transaction(
-                        asset=order.asset,
-                        amount=order.amount,
-                        dt=pd.Timestamp.utcnow(),
-                        price=executed_price,
-                        order_id=order.id,
-                        commission=order.commission
-                    )
-                    transactions.append(transaction)
-
-                    self.portfolio.execute_order(order, transaction)
-
-                elif order.status == ORDER_STATUS.CANCELLED:
-                    self.portfolio.remove_order(order)
-
-                else:
-                    delta = pd.Timestamp.utcnow() - order.dt
-                    log.info(
-                        'order {order_id} still open after {delta}'.format(
-                            order_id=order_id,
-                            delta=delta
-                        )
-                    )
-        return transactions
-
-    def get_spot_value(self, assets, field, dt=None, data_frequency='minute'):
-        """
-        Public API method that returns a scalar value representing the value
-        of the desired asset's field at either the given dt.
-
-        Parameters
-        ----------
-        assets : Asset, ContinuousFuture, or iterable of same.
-            The asset or assets whose data is desired.
-        field : {'open', 'high', 'low', 'close', 'volume',
-                 'price', 'last_traded'}
-            The desired field of the asset.
-        dt : pd.Timestamp
-            The timestamp for the desired value.
-        data_frequency : str
-            The frequency of the data to query; i.e. whether the data is
-            'daily' or 'minute' bars
-
-        Returns
-        -------
-        value : float, int, or pd.Timestamp
-            The spot value of ``field`` for ``asset`` The return type is based
-            on the ``field`` requested. If the field is one of 'open', 'high',
-            'low', 'close', or 'price', the value will be a float. If the
-            ``field`` is 'volume' the value will be a int. If the ``field`` is
-            'last_traded' the value will be a Timestamp.
-
-        Bitfinex timeframes
-        -------------------
-        Available values: '1m', '5m', '15m', '30m', '1h', '3h', '6h', '12h',
-         '1D', '7D', '14D', '1M'
-        """
-        if field not in BASE_FIELDS:
-            raise KeyError('Invalid column: {}'.format(field))
-
-        values = []
-        for asset in assets:
-            value = self.get_single_spot_value(asset, field, data_frequency)
-            values.append(value)
-
-        return values
-
-    def get_single_spot_value(self, asset, field, data_frequency):
-        """
-        Similar to 'get_spot_value' but for a single asset
-
-        Notes
-        -----
-        We're writing each minute bar to disk using zipline's machinery.
-        This is especially useful when running multiple algorithms
-        concurrently. By using local data when possible, we try to reaching
-        request limits on exchanges.
-
-        Parameters
-        ----------
-        asset: TradingPair
-        field: str
-        data_frequency: str
-
-        Returns
-        -------
-        float
-            The spot value of the given asset / field
-
-        """
-        log.debug(
-            'fetching spot value {field} for symbol {symbol}'.format(
-                symbol=asset.symbol,
-                field=field
-            )
-        )
-
-        freq = '1T' if data_frequency == 'minute' else '1D'
-        ohlc = self.get_candles(freq, asset)
-        if field not in ohlc:
-            raise KeyError('Invalid column: %s' % field)
-
-        value = ohlc[field]
-        log.debug('got spot value: {}'.format(value))
-
-        return value
-
-    def get_series_from_candles(self, candles, start_dt, end_dt,
-                                data_frequency, field, previous_value=None):
-        """
-        Get a series of field data for the specified candles.
-
-        Parameters
-        ----------
-        candles: list[dict[str, float]]
-        start_dt: datetime
-        end_dt: datetime
-        data_frequency: str
-        field: str
-        previous_value: float
-
-        Returns
-        -------
-        Series
-
-        """
-        dates = [candle['last_traded'] for candle in candles]
-        values = [candle[field] for candle in candles]
-        series = pd.Series(values, index=dates)
-
-        periods = get_periods_range(
-            start_dt, end_dt, data_frequency
-        )
-        # TODO: ensure that this working as expected, if not use fillna
-        series = series.reindex(
-            periods,
-            method='ffill',
-            fill_value=previous_value,
-        )
-
-        return series
-
-    @deprecated
-    def get_history_window_direct(self,
-                                  assets,
-                                  end_dt,
-                                  bar_count,
-                                  frequency,
-                                  field,
-                                  data_frequency=None,
-                                  ffill=True):
-
-        """
-        Public API method that returns a dataframe containing the requested
-        history window.  Data is fully adjusted.
-
-        Parameters
-        ----------
-        assets : list[TradingPair]
-            The assets whose data is desired.
-
-        end_dt: datetime
-            The date of the last bar
-
-        bar_count: int
-            The number of bars desired.
-
-        frequency: string
-            "1d" or "1m"
-
-        field: string
-            The desired field of the asset.
-
-        data_frequency: string
-            The frequency of the data to query; i.e. whether the data is
-            'daily' or 'minute' bars.
-
-        # TODO: fill how?
-        ffill: boolean
-            Forward-fill missing values. Only has effect if field
-            is 'price'.
-
-        Returns
-        -------
-        DataFrame
-            A dataframe containing the requested data.
-
-        """
-        start_dt = get_start_dt(end_dt, bar_count, data_frequency)
-
-        # The get_history method supports multiple asset
-        candles = self.get_candles(
-            data_frequency=frequency,
-            assets=assets,
-            bar_count=bar_count,
-            start_dt=start_dt,
-            end_dt=end_dt
-        )
-        candle_series = self.get_series_from_candles(
-            candles=candles,
-            start_dt=start_dt,
-            end_dt=end_dt,
-            data_frequency=frequency,
-            field=field,
-        )
-
-        df = pd.DataFrame(candle_series)
-        return df
-
-    def get_history_window(self,
-                           assets,
-                           end_dt,
-                           bar_count,
-                           frequency,
-                           field,
-                           data_frequency=None,
-                           ffill=True):
-
-        """
-        Public API method that returns a dataframe containing the requested
-        history window.  Data is fully adjusted.
-
-        Parameters
-        ----------
-        assets : list[TradingPair]
-            The assets whose data is desired.
-
-        end_dt: datetime
-            The date of the last bar.
-
-        bar_count: int
-            The number of bars desired.
-
-        frequency: string
-            "1d" or "1m"
-
-        field: string
-            The desired field of the asset.
-
-        data_frequency: string
-            The frequency of the data to query; i.e. whether the data is
-            'daily' or 'minute' bars.
-
-        # TODO: fill how?
-        ffill: boolean
-            Forward-fill missing values. Only has effect if field
-            is 'price'.
-
-        Returns
-        -------
-        DataFrame
-            A dataframe containing the requested data.
-
-        """
-        freq, candle_size, unit, data_frequency = get_frequency(
-            frequency, data_frequency
-        )
-        adj_bar_count = candle_size * bar_count
-        try:
-            series = self.bundle.get_history_window_series_and_load(
-                assets=assets,
-                end_dt=end_dt,
-                bar_count=adj_bar_count,
-                field=field,
-                data_frequency=data_frequency
-            )
-        except (PricingDataNotLoadedError, NoDataAvailableOnExchange):
-            series = dict()
-
-        for asset in assets:
-            if asset not in series or series[asset].index[-1] < end_dt:
-                # Adding bars too recent to be contained in the consolidated
-                # exchanges bundles. We go directly against the exchange
-                # to retrieve the candles.
-                start_dt = get_start_dt(end_dt, adj_bar_count, data_frequency)
-                trailing_dt = \
-                    series[asset].index[-1] + get_delta(1, data_frequency) \
-                        if asset in series else start_dt
-
-                # The get_history method supports multiple asset
-                # Use the original frequency to let each api optimize
-                # the size of result sets
-                trailing_bar_count = get_periods(
-                    trailing_dt, end_dt, freq
-                )
-                candles = self.get_candles(
-                    freq=freq,
-                    assets=asset,
-                    bar_count=trailing_bar_count,
-                    start_dt=start_dt,
-                    end_dt=end_dt
-                )
-
-                last_value = series[asset].iloc(0) if asset in series \
-                    else np.nan
-
-                # Create a series with the common data_frequency, ffill
-                # missing values
-                candle_series = self.get_series_from_candles(
-                    candles=candles,
-                    start_dt=trailing_dt,
-                    end_dt=end_dt,
-                    data_frequency=data_frequency,
-                    field=field,
-                    previous_value=last_value
-                )
-
-                if asset in series:
-                    series[asset].append(candle_series)
-
-                else:
-                    series[asset] = candle_series
-
-        df = resample_history_df(pd.DataFrame(series), freq, field)
-        # TODO: consider this more carefully
-        df.dropna(inplace=True)
-
-        return df
-
-    def synchronize_portfolio(self):
-        """
-        Update the portfolio cash and position balances based on the
-        latest ticker prices.
-
-        """
-        log.debug('synchronizing portfolio with exchange {}'.format(self.name))
-        balances = self.get_balances()
-
-        base_position_available = balances[self.base_currency] \
-            if self.base_currency in balances else None
-
-        if base_position_available is None:
-            raise BaseCurrencyNotFoundError(
-                base_currency=self.base_currency,
-                exchange=self.name.title()
-            )
-
-        portfolio = self._portfolio
-        portfolio.cash = base_position_available
-        log.debug('found base currency balance: {}'.format(portfolio.cash))
-
-        if portfolio.starting_cash is None:
-            portfolio.starting_cash = portfolio.cash
-
-        if portfolio.positions:
-            assets = list(portfolio.positions.keys())
-            tickers = self.tickers(assets)
-
-            portfolio.positions_value = 0.0
-            for asset in tickers:
-                # TODO: convert if the position is not in the base currency
-                ticker = tickers[asset]
-                position = portfolio.positions[asset]
-                position.last_sale_price = ticker['last_price']
-                position.last_sale_date = ticker['timestamp']
-
-                portfolio.positions_value += \
-                    position.amount * position.last_sale_price
-                portfolio.portfolio_value = \
-                    portfolio.positions_value + portfolio.cash
-
-    def order(self, asset, amount, limit_price=None, stop_price=None,
-              style=None):
-        """Place an order.
-
-        Parameters
-        ----------
-        asset : TradingPair
-            The asset that this order is for.
-
-        amount : int
-            The amount of shares to order. If ``amount`` is positive, this is
-            the number of shares to buy or cover. If ``amount`` is negative,
-            this is the number of shares to sell or short.
-
-        limit_price : float, optional
-            The limit price for the order.
-
-        stop_price : float, optional
-            The stop price for the order.
-
-        style : ExecutionStyle, optional
-            The execution style for the order.
-
-        Returns
-        -------
-        order_id : str or None
-            The unique identifier for this order, or None if no order was
-            placed.
-
-        Notes
-        -----
-        The ``limit_price`` and ``stop_price`` arguments provide shorthands for
-        passing common execution styles. Passing ``limit_price=N`` is
-        equivalent to ``style=LimitOrder(N)``. Similarly, passing
-        ``stop_price=M`` is equivalent to ``style=StopOrder(M)``, and passing
-        ``limit_price=N`` and ``stop_price=M`` is equivalent to
-        ``style=StopLimitOrder(N, M)``. It is an error to pass both a ``style``
-        and ``limit_price`` or ``stop_price``.
-
-        See Also
-        --------
-        :class:`catalyst.finance.execution.ExecutionStyle`
-        :func:`catalyst.api.order_value`
-        :func:`catalyst.api.order_percent`
-
-        """
-        if amount == 0:
-            log.warn('skipping order amount of 0')
-            return None
-
-        if asset.base_currency != self.base_currency.lower():
-            raise MismatchingBaseCurrencies(
-                base_currency=asset.base_currency,
-                algo_currency=self.base_currency
-            )
-
-        is_buy = (amount > 0)
-
-        if limit_price is not None and stop_price is not None:
-            style = ExchangeStopLimitOrder(limit_price, stop_price,
-                                           exchange=self.name)
-        elif limit_price is not None:
-            style = ExchangeLimitOrder(limit_price, exchange=self.name)
-
-        elif stop_price is not None:
-            style = ExchangeStopOrder(stop_price, exchange=self.name)
-
-        elif style is not None:
-            raise InvalidOrderStyle(exchange=self.name.title(),
-                                    style=style.__class__.__name__)
-        else:
-            raise ValueError('Incomplete order data.')
-
-        display_price = limit_price if limit_price is not None else stop_price
-        log.debug(
-            'issuing {side} order of {amount} {symbol} for {type}: {price}'.format(
-                side='buy' if is_buy else 'sell',
-                amount=amount,
-                symbol=asset.symbol,
-                type=style.__class__.__name__,
-                price='{}{}'.format(display_price, asset.base_currency)
-            )
-        )
-        order = self.create_order(asset, amount, is_buy, style)
-        if order:
-            self._portfolio.create_order(order)
-            return order.id
-        else:
-            return None
-
-    # The methods below must be implemented for each exchange.
-    @abstractmethod
-    def get_balances(self):
-        """
-        Retrieve wallet balances for the exchange.
-
-        Returns
-        -------
-        dict[TradingPair, float]
-
-        """
-        pass
-
-    @abstractmethod
-    def create_order(self, asset, amount, is_buy, style):
-        """
-        Place an order on the exchange.
-
-        Parameters
-        ----------
-        asset: TradingPair
-            The target market.
-
-        amount: float
-            The amount of shares to order. If ``amount`` is positive, this is
-            the number of shares to buy or cover. If ``amount`` is negative,
-            this is the number of shares to sell or short.
-
-        is_buy: bool
-            Is it a buy order?
-
-        style: ExecutionStyle
-
-        Returns
-        -------
-        Order
-
-        """
-        pass
-
-    @abstractmethod
-    def get_open_orders(self, asset):
-        """Retrieve all of the current open orders.
-
-        Parameters
-        ----------
-        asset : Asset
-            If passed and not None, return only the open orders for the given
-            asset instead of all open orders.
-
-        Returns
-        -------
-        open_orders : dict[list[Order]] or list[Order]
-            If no asset is passed this will return a dict mapping Assets
-            to a list containing all the open orders for the asset.
-            If an asset is passed then this will return a list of the open
-            orders for this asset.
-        """
-        pass
-
-    @abstractmethod
-    def get_order(self, order_id):
-        """Lookup an order based on the order id returned from one of the
-        order functions.
-
-        Parameters
-        ----------
-        order_id : str
-            The unique identifier for the order.
-
-        Returns
-        -------
-        order : Order
-            The order object.
-        execution_price: float
-            The execution price per share of the order
-        """
-        pass
-
-    @abstractmethod
-    def cancel_order(self, order_param):
-        """Cancel an open order.
-
-        Parameters
-        ----------
-        order_param : str or Order
-            The order_id or order object to cancel.
-        """
-        pass
-
-    @abstractmethod
-    def get_candles(self, freq, assets, bar_count=None,
-                    start_dt=None, end_dt=None):
-        """
-        Retrieve OHLCV candles for the given assets
-
-        Parameters
-        ----------
-        freq: str
-            The frequency alias per convention:
-            http://pandas.pydata.org/pandas-docs/stable/timeseries.html#offset-aliases
-
-        assets: list[TradingPair]
-            The targeted assets.
-
-        bar_count: int
-            The number of bar desired. (default 1)
-
-        end_dt: datetime, optional
-            The last bar date.
-
-        start_dt: datetime, optional
-            The first bar date.
-
-        Returns
-        -------
-        dict[TradingPair, dict[str, Object]]
-            A dictionary of OHLCV candles. Each TradingPair instance is
-            mapped to a list of dictionaries with this structure:
-                open: float
-                high: float
-                low: float
-                close: float
-                volume: float
-                last_traded: datetime
-
-            See definition here:
-                http://www.investopedia.com/terms/o/ohlcchart.asp
-        """
-        pass
-
-    @abc.abstractmethod
-    def tickers(self, assets):
-        """
-        Retrieve current tick data for the given assets
-
-        Parameters
-        ----------
-        assets: list[TradingPair]
-
-        Returns
-        -------
-        list[dict[str, float]
-
-        """
-        pass
-
-    @abc.abstractmethod
-    def get_account(self):
-        """
-        Retrieve the account parameters.
-        """
-        pass
-
-    @abc.abstractmethod
-    def get_orderbook(self, asset, order_type, limit):
-        """
-        Retrieve the the orderbook for the given trading pair.
-
-        Parameters
-        ----------
-        asset: TradingPair
-        order_type: str
-            The type of orders: bid, ask or all
-        limit: int
-
-        Returns
-        -------
-        list[dict[str, float]
-        """
-        pass

catalyst/exchange/exchange_algorithm.py

@@ -1,799 +0,0 @@
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-import pickle
-import signal
-import sys
-from collections import deque
-from datetime import timedelta
-from os import listdir
-from os.path import isfile, join
-from time import sleep
-
-import logbook
-import pandas as pd
-from catalyst.assets._assets import TradingPair
-
-import catalyst.protocol as zp
-from catalyst.algorithm import TradingAlgorithm
-from catalyst.constants import LOG_LEVEL
-from catalyst.errors import OrderInBeforeTradingStart
-from catalyst.exchange.exchange_blotter import ExchangeBlotter
-from catalyst.exchange.exchange_errors import (
-    ExchangeRequestError,
-    ExchangePortfolioDataError,
-    ExchangeTransactionError,
-    OrphanOrderError)
-from catalyst.exchange.exchange_execution import ExchangeStopLimitOrder, \
-    ExchangeLimitOrder, ExchangeStopOrder
-from catalyst.exchange.exchange_utils import save_algo_object, get_algo_object, \
-    get_algo_folder, get_algo_df, \
-    save_algo_df
-from catalyst.exchange.live_graph_clock import LiveGraphClock
-from catalyst.exchange.simple_clock import SimpleClock
-from catalyst.exchange.stats_utils import get_pretty_stats
-from catalyst.finance.execution import MarketOrder
-from catalyst.finance.performance.period import calc_period_stats
-from catalyst.gens.tradesimulation import AlgorithmSimulator
-from catalyst.utils.api_support import (
-    api_method,
-    disallowed_in_before_trading_start)
-from catalyst.utils.input_validation import error_keywords, ensure_upper_case, \
-    expect_types
-from catalyst.utils.math_utils import round_nearest
-from catalyst.utils.preprocess import preprocess
-
-log = logbook.Logger('exchange_algorithm', level=LOG_LEVEL)
-
-
-class ExchangeAlgorithmExecutor(AlgorithmSimulator):
-    def __init__(self, *args, **kwargs):
-        super(self.__class__, self).__init__(*args, **kwargs)
-
-
-class ExchangeTradingAlgorithmBase(TradingAlgorithm):
-    def __init__(self, *args, **kwargs):
-        self.exchanges = kwargs.pop('exchanges', None)
-
-        super(ExchangeTradingAlgorithmBase, self).__init__(*args, **kwargs)
-
-    def round_order(self, amount, asset):
-        """
-        We need fractions with cryptocurrencies
-
-        :param amount:
-        :return:
-        """
-        return round_nearest(amount, asset.min_trade_size)
-
-    @api_method
-    @preprocess(symbol_str=ensure_upper_case)
-    def symbol(self, symbol_str, exchange_name=None):
-        """Lookup an Equity by its ticker symbol.
-
-        Parameters
-        ----------
-        symbol_str : str
-            The ticker symbol for the equity to lookup.
-        exchange_name: str
-            The name of the exchange containing the symbol
-
-        Returns
-        -------
-        equity : Equity
-            The equity that held the ticker symbol on the current
-            symbol lookup date.
-
-        Raises
-        ------
-        SymbolNotFound
-            Raised when the symbols was not held on the current lookup date.
-
-        See Also
-        --------
-        :func:`catalyst.api.set_symbol_lookup_date`
-        """
-        # If the user has not set the symbol lookup date,
-        # use the end_session as the date for sybmol->sid resolution.
-
-        _lookup_date = self._symbol_lookup_date \
-            if self._symbol_lookup_date is not None \
-            else self.sim_params.end_session
-
-        if exchange_name is None:
-            exchange = list(self.exchanges.values())[0]
-        else:
-            exchange = self.exchanges[exchange_name]
-
-        return self.asset_finder.lookup_symbol(
-            symbol=symbol_str,
-            exchange=exchange,
-            as_of_date=_lookup_date
-        )
-
-    def prepare_period_stats(self, start_dt, end_dt):
-        """
-        Creates a dictionary representing the state of the tracker.
-
-        Parameters
-        ----------
-        start_dt: datetime
-        end_dt: datetime
-
-        Notes
-        -----
-        I rewrote this in an attempt to better control the stats.
-        I don't want things to happen magically through complex logic
-        pertaining to backtesting.
-
-        """
-        tracker = self.perf_tracker
-        period = tracker.todays_performance
-
-        pos_stats = period.position_tracker.stats()
-        period_stats = calc_period_stats(pos_stats, period.ending_cash)
-
-        stats = dict(
-            period_start=tracker.period_start,
-            period_end=tracker.period_end,
-            capital_base=tracker.capital_base,
-            progress=tracker.progress,
-            ending_value=period.ending_value,
-            ending_exposure=period.ending_exposure,
-            capital_used=period.cash_flow,
-            starting_value=period.starting_value,
-            starting_exposure=period.starting_exposure,
-            starting_cash=period.starting_cash,
-            ending_cash=period.ending_cash,
-            portfolio_value=period.ending_cash + period.ending_value,
-            pnl=period.pnl,
-            returns=period.returns,
-            period_open=period.period_open,
-            period_close=period.period_close,
-            gross_leverage=period_stats.gross_leverage,
-            net_leverage=period_stats.net_leverage,
-            short_exposure=pos_stats.short_exposure,
-            long_exposure=pos_stats.long_exposure,
-            short_value=pos_stats.short_value,
-            long_value=pos_stats.long_value,
-            longs_count=pos_stats.longs_count,
-            shorts_count=pos_stats.shorts_count,
-        )
-
-        # Merging cumulative risk
-        stats.update(tracker.cumulative_risk_metrics.to_dict())
-
-        # Merging latest recorded variables
-        stats.update(self.recorded_vars)
-
-        stats['positions'] = period.position_tracker.get_positions_list()
-
-        # we want the key to be absent, not just empty
-        # Only include transactions for given dt
-        stats['transactions'] = []
-        for date in period.processed_transactions:
-            if start_dt <= date < end_dt:
-                transactions = period.processed_transactions[date]
-                for t in transactions:
-                    stats['transactions'].append(t.to_dict())
-
-        stats['orders'] = []
-        for date in period.orders_by_modified:
-            if start_dt <= date < end_dt:
-                orders = period.orders_by_modified[date]
-                for order in orders:
-                    stats['orders'].append(orders[order].to_dict())
-
-        return stats
-
-
-class ExchangeTradingAlgorithmBacktest(ExchangeTradingAlgorithmBase):
-    def __init__(self, *args, **kwargs):
-        super(ExchangeTradingAlgorithmBacktest, self).__init__(*args, **kwargs)
-
-        self.frame_stats = list()
-        self.blotter = ExchangeBlotter(
-            data_frequency=self.data_frequency,
-            # Default to NeverCancel in catalyst
-            cancel_policy=self.cancel_policy,
-        )
-        log.info('initialized trading algorithm in backtest mode')
-
-    def _calculate_order(self, asset, amount,
-                         limit_price=None, stop_price=None, style=None):
-        # Raises a ZiplineError if invalid parameters are detected.
-        self.validate_order_params(asset,
-                                   amount,
-                                   limit_price,
-                                   stop_price,
-                                   style)
-
-        # Convert deprecated limit_price and stop_price parameters to use
-        # ExecutionStyle objects.
-        style = self.__convert_order_params_for_blotter(limit_price,
-                                                        stop_price,
-                                                        style)
-        return amount, style
-
-    @staticmethod
-    def __convert_order_params_for_blotter(limit_price, stop_price, style):
-        """
-        Helper method for converting deprecated limit_price and stop_price
-        arguments into ExecutionStyle instances.
-
-        This function assumes that either style == None or (limit_price,
-        stop_price) == (None, None).
-        """
-        if style:
-            assert (limit_price, stop_price) == (None, None)
-            return style
-        if limit_price and stop_price:
-            return ExchangeStopLimitOrder(limit_price, stop_price)
-        if limit_price:
-            return ExchangeLimitOrder(limit_price)
-        if stop_price:
-            return ExchangeStopOrder(stop_price)
-        else:
-            return MarketOrder()
-
-    def handle_data(self, data):
-        super(ExchangeTradingAlgorithmBacktest, self).handle_data(data)
-
-        minute_stats = self.prepare_period_stats(
-            data.current_dt, data.current_dt + timedelta(minutes=1))
-        self.frame_stats.append(minute_stats)
-
-    def analyze(self, perf):
-        stats = pd.DataFrame(self.frame_stats)
-        stats.set_index('period_close', inplace=True, drop=False)
-
-        super(ExchangeTradingAlgorithmBacktest, self).analyze(stats)
-
-
-class ExchangeTradingAlgorithmLive(ExchangeTradingAlgorithmBase):
-    def __init__(self, *args, **kwargs):
-        self.algo_namespace = kwargs.pop('algo_namespace', None)
-        self.live_graph = kwargs.pop('live_graph', None)
-
-        self._clock = None
-        self.minute_stats = deque(maxlen=60)
-
-        self.pnl_stats = get_algo_df(self.algo_namespace, 'pnl_stats')
-
-        self.custom_signals_stats = \
-            get_algo_df(self.algo_namespace, 'custom_signals_stats')
-
-        self.exposure_stats = \
-            get_algo_df(self.algo_namespace, 'exposure_stats')
-
-        self.is_running = True
-
-        self.retry_check_open_orders = 5
-        self.retry_synchronize_portfolio = 5
-        self.retry_get_open_orders = 5
-        self.retry_order = 2
-        self.retry_delay = 5
-
-        self.stats_minutes = 5
-
-        super(ExchangeTradingAlgorithmLive, self).__init__(*args, **kwargs)
-
-        signal.signal(signal.SIGINT, self.signal_handler)
-
-        log.info('initialized trading algorithm in live mode')
-
-    def signal_handler(self, signal, frame):
-        """
-        Handles the keyboard interruption signal.
-
-        Parameters
-        ----------
-        signal
-        frame
-
-        Returns
-        -------
-
-        """
-        self.is_running = False
-
-        if self._analyze is None:
-            log.info('Interruption signal detected {}, exiting the '
-                     'algorithm'.format(signal))
-
-        else:
-            log.info('Interruption signal detected {}, calling `analyze()` '
-                     'before exiting the algorithm'.format(signal))
-
-            algo_folder = get_algo_folder(self.algo_namespace)
-            folder = join(algo_folder, 'daily_perf')
-            files = [f for f in listdir(folder) if isfile(join(folder, f))]
-
-            daily_perf_list = []
-            for item in files:
-                filename = join(folder, item)
-                with open(filename, 'rb') as handle:
-                    daily_perf_list.append(pickle.load(handle))
-
-            stats = pd.DataFrame(daily_perf_list)
-
-            self.analyze(stats)
-
-        sys.exit(0)
-
-    @property
-    def clock(self):
-        if self._clock is None:
-            return self._create_clock()
-        else:
-            return self._clock
-
-    def _create_clock(self):
-
-        # The calendar's execution times are the minutes over which we actually
-        # want to run the clock. Typically the execution times simply adhere to
-        # the market open and close times. In the case of the futures calendar,
-        # for example, we only want to simulate over a subset of the full 24
-        # hour calendar, so the execution times dictate a market open time of
-        # 6:31am US/Eastern and a close of 5:00pm US/Eastern.
-
-        # In our case, we are trading around the clock, so the market close
-        # corresponds to the last minute of the day.
-
-        # This method is taken from TradingAlgorithm.
-        # The clock has been replaced to use RealtimeClock
-        # TODO: should we apply a time skew? not sure to understand the utility.
-
-        log.debug('creating clock')
-        if self.live_graph:
-            self._clock = LiveGraphClock(
-                self.sim_params.sessions,
-                context=self
-            )
-        else:
-            self._clock = SimpleClock(
-                self.sim_params.sessions,
-            )
-
-        return self._clock
-
-    def _create_generator(self, sim_params):
-        if self.perf_tracker is None:
-            self.perf_tracker = get_algo_object(
-                algo_name=self.algo_namespace,
-                key='perf_tracker'
-            )
-
-        # Call the simulation trading algorithm for side-effects:
-        # it creates the perf tracker
-        TradingAlgorithm._create_generator(self, sim_params)
-        self.trading_client = ExchangeAlgorithmExecutor(
-            self,
-            sim_params,
-            self.data_portal,
-            self.clock,
-            self._create_benchmark_source(),
-            self.restrictions,
-            universe_func=self._calculate_universe
-        )
-
-        return self.trading_client.transform()
-
-    def updated_portfolio(self):
-        """
-        We skip the entire performance tracker business and update the
-        portfolio directly.
-
-        Returns
-        -------
-        ExchangePortfolio
-
-        """
-        # TODO: build cumulative portfolio
-        return self.perf_tracker.get_portfolio(False)
-
-    def updated_account(self):
-        return self.perf_tracker.get_account(False)
-
-    def _synchronize_portfolio(self, attempt_index=0):
-        try:
-            for exchange_name in self.exchanges:
-                exchange = self.exchanges[exchange_name]
-
-                exchange.synchronize_portfolio()
-
-                # Applying the updated last_sales_price to the positions
-                # in the performance tracker. This seems a bit redundant
-                # but it will make sense when we have multiple exchange portfolios
-                # feeding into the same performance tracker.
-                tracker = self.perf_tracker.todays_performance.position_tracker
-                for asset in exchange.portfolio.positions:
-                    position = exchange.portfolio.positions[asset]
-                    tracker.update_position(
-                        asset=asset,
-                        last_sale_date=position.last_sale_date,
-                        last_sale_price=position.last_sale_price
-                    )
-        except ExchangeRequestError as e:
-            log.warn(
-                'update portfolio attempt {}: {}'.format(attempt_index, e)
-            )
-            if attempt_index < self.retry_synchronize_portfolio:
-                sleep(self.retry_delay)
-                self._synchronize_portfolio(attempt_index + 1)
-            else:
-                raise ExchangePortfolioDataError(
-                    data_type='update-portfolio',
-                    attempts=attempt_index,
-                    error=e
-                )
-
-    def _check_open_orders(self, attempt_index=0):
-        try:
-            orders = list()
-            for exchange_name in self.exchanges:
-                exchange = self.exchanges[exchange_name]
-                exchange_orders = exchange.check_open_orders()
-
-                orders += exchange_orders
-
-            return orders
-        except ExchangeRequestError as e:
-            log.warn(
-                'check open orders attempt {}: {}'.format(attempt_index, e)
-            )
-            if attempt_index < self.retry_check_open_orders:
-                sleep(self.retry_delay)
-                return self._check_open_orders(attempt_index + 1)
-            else:
-                raise ExchangePortfolioDataError(
-                    data_type='order-status',
-                    attempts=attempt_index,
-                    error=e
-                )
-
-    def add_pnl_stats(self, period_stats):
-        """
-        Save p&l stats.
-
-        Parameters
-        ----------
-        period_stats
-
-        Returns
-        -------
-
-        """
-        starting = period_stats['starting_cash']
-        current = period_stats['portfolio_value']
-        appreciation = (current / starting) - 1
-        perc = (appreciation * 100) if current != 0 else 0
-
-        log.debug('adding pnl stats: {:6f}%'.format(perc))
-
-        df = pd.DataFrame(
-            data=[dict(performance=perc)],
-            index=[period_stats['period_close']]
-        )
-        self.pnl_stats = pd.concat([self.pnl_stats, df])
-
-        save_algo_df(self.algo_namespace, 'pnl_stats', self.pnl_stats)
-
-    def add_custom_signals_stats(self, period_stats):
-        """
-        Save custom signals stats.
-
-        Parameters
-        ----------
-        period_stats
-
-        Returns
-        -------
-
-        """
-        log.debug('adding custom signals stats: {}'.format(self.recorded_vars))
-        df = pd.DataFrame(
-            data=[self.recorded_vars],
-            index=[period_stats['period_close']],
-        )
-        self.custom_signals_stats = pd.concat([self.custom_signals_stats, df])
-
-        save_algo_df(self.algo_namespace, 'custom_signals_stats',
-                     self.custom_signals_stats)
-
-    def add_exposure_stats(self, period_stats):
-        """
-        Save exposure stats.
-
-        Parameters
-        ----------
-        period_stats
-
-        Returns
-        -------
-
-        """
-        data = dict(
-            long_exposure=period_stats['long_exposure'],
-            base_currency=period_stats['ending_cash']
-        )
-        log.debug('adding exposure stats: {}'.format(data))
-
-        df = pd.DataFrame(
-            data=[data],
-            index=[period_stats['period_close']],
-        )
-        self.exposure_stats = pd.concat([self.exposure_stats, df])
-
-        save_algo_df(self.algo_namespace, 'exposure_stats',
-                     self.exposure_stats)
-
-    def handle_data(self, data):
-        """
-        Wrapper around the handle_data method of each algo.
-
-        Parameters
-        ----------
-        data
-
-        """
-        if not self.is_running:
-            return
-
-        self._synchronize_portfolio()
-
-        transactions = self._check_open_orders()
-        for transaction in transactions:
-            self.perf_tracker.process_transaction(transaction)
-
-        if self._handle_data:
-            self._handle_data(self, data)
-
-        # Unlike trading controls which remain constant unless placing an
-        # order, account controls can change each bar. Thus, must check
-        # every bar no matter if the algorithm places an order or not.
-        self.validate_account_controls()
-
-        try:
-            # Since the clock runs 24/7, I trying to disable the daily
-            # Performance tracker and keep only minute and cumulative
-            self.perf_tracker.update_performance()
-
-            minute_stats = self.prepare_period_stats(
-                data.current_dt, data.current_dt + timedelta(minutes=1))
-
-            # Saving the last hour in memory
-            self.minute_stats.append(minute_stats)
-
-            self.add_pnl_stats(minute_stats)
-            if self.recorded_vars:
-                self.add_custom_signals_stats(minute_stats)
-                recorded_cols = list(self.recorded_vars.keys())
-            else:
-                recorded_cols = None
-
-            self.add_exposure_stats(minute_stats)
-
-            print_df = pd.DataFrame(list(self.minute_stats))
-            log.info(
-                'statistics for the last {stats_minutes} minutes:\n{stats}'.format(
-                    stats_minutes=self.stats_minutes,
-                    stats=get_pretty_stats(
-                        stats_df=print_df,
-                        recorded_cols=recorded_cols,
-                        num_rows=self.stats_minutes
-                    )
-                ))
-
-            today = pd.to_datetime('today', utc=True)
-            daily_stats = self.prepare_period_stats(
-                start_dt=today,
-                end_dt=pd.Timestamp.utcnow()
-            )
-            save_algo_object(
-                algo_name=self.algo_namespace,
-                key=today.strftime('%Y-%m-%d'),
-                obj=daily_stats,
-                rel_path='daily_perf'
-            )
-
-        except Exception as e:
-            log.warn('unable to calculate performance: {}'.format(e))
-
-        # TODO: pickle does not seem to work in python 3
-        try:
-            save_algo_object(
-                algo_name=self.algo_namespace,
-                key='perf_tracker',
-                obj=self.perf_tracker
-            )
-        except Exception as e:
-            log.warn('unable to save minute perfs to disk: {}'.format(e))
-
-        try:
-            for exchange_name in self.exchanges:
-                exchange = self.exchanges[exchange_name]
-                save_algo_object(
-                    algo_name=self.algo_namespace,
-                    key='portfolio_{}'.format(exchange_name),
-                    obj=exchange.portfolio
-                )
-        except Exception as e:
-            log.warn('unable to save portfolio to disk: {}'.format(e))
-
-    def _order(self,
-               asset,
-               amount,
-               limit_price=None,
-               stop_price=None,
-               style=None,
-               attempt_index=0):
-        try:
-            exchange = self.exchanges[asset.exchange]
-            return exchange.order(asset, amount, limit_price,
-                                  stop_price,
-                                  style)
-        except ExchangeRequestError as e:
-            log.warn(
-                'order attempt {}: {}'.format(attempt_index, e)
-            )
-            if attempt_index < self.retry_order:
-                sleep(self.retry_delay)
-                return self._order(
-                    asset, amount, limit_price, stop_price, style,
-                    attempt_index + 1)
-            else:
-                raise ExchangeTransactionError(
-                    transaction_type='order',
-                    attempts=attempt_index,
-                    error=e
-                )
-
-    @api_method
-    @disallowed_in_before_trading_start(OrderInBeforeTradingStart())
-    @expect_types(asset=TradingPair)
-    def order(self,
-              asset,
-              amount,
-              limit_price=None,
-              stop_price=None,
-              style=None):
-        """
-        We use the exchange specific portfolio to place orders.
-        The cumulative portfolio does not contain open orders but exchange
-        portfolios do.
-
-        Parameters
-        ----------
-        asset: TradingPair
-        amount: float
-        limit_price: float
-        stop_price: float
-        style: Style
-        order: Order
-            The catalyst order object or None
-        """
-        amount, style = self._calculate_order(asset, amount,
-                                              limit_price, stop_price,
-                                              style)
-
-        order_id = self._order(asset, amount, limit_price, stop_price, style)
-
-        exchange = self.exchanges[asset.exchange]
-        exchange_portfolio = exchange.portfolio
-        if order_id is not None:
-
-            if order_id in exchange_portfolio.open_orders:
-                order = exchange_portfolio.open_orders[order_id]
-                self.perf_tracker.process_order(order)
-                return order
-
-            else:
-                raise OrphanOrderError(
-                    order_id=order_id,
-                    exchange=exchange.name
-                )
-        else:
-            log.warn('unable to order {} {} on exchange {}'.format(
-                amount, asset.symbol, asset.exchange))
-            return None
-
-    @api_method
-    def batch_market_order(self, share_counts):
-        raise NotImplementedError()
-
-    def _get_open_orders(self, asset=None, attempt_index=0):
-        try:
-            if asset:
-                exchange = self.exchanges[asset.exchange]
-                return exchange.get_open_orders(asset)
-
-            else:
-                open_orders = []
-                for exchange_name in self.exchanges:
-                    exchange = self.exchanges[exchange_name]
-                    exchange_orders = exchange.get_open_orders()
-                    open_orders.append(exchange_orders)
-
-                return open_orders
-        except ExchangeRequestError as e:
-            log.warn(
-                'open orders attempt {}: {}'.format(attempt_index, e)
-            )
-            if attempt_index < self.retry_get_open_orders:
-                sleep(self.retry_delay)
-                return self._get_open_orders(asset, attempt_index + 1)
-            else:
-                raise ExchangePortfolioDataError(
-                    data_type='open-orders',
-                    attempts=attempt_index,
-                    error=e
-                )
-
-    @error_keywords(sid='Keyword argument `sid` is no longer supported for '
-                        'get_open_orders. Use `asset` instead.')
-    @api_method
-    def get_open_orders(self, asset=None):
-        """Retrieve all of the current open orders.
-
-        Parameters
-        ----------
-        asset : Asset
-            If passed and not None, return only the open orders for the given
-            asset instead of all open orders.
-
-        Returns
-        -------
-        open_orders : dict[list[Order]] or list[Order]
-            If no asset is passed this will return a dict mapping Assets
-            to a list containing all the open orders for the asset.
-            If an asset is passed then this will return a list of the open
-            orders for this asset.
-        """
-        return self._get_open_orders(asset)
-
-    @api_method
-    def get_order(self, order_id, exchange_name):
-        """Lookup an order based on the order id returned from one of the
-        order functions.
-
-        Parameters
-        ----------
-        order_id : str
-            The unique identifier for the order.
-
-        Returns
-        -------
-        order : Order
-            The order object.
-        execution_price: float
-            The execution price per share of the order
-        """
-        exchange = self.exchanges[exchange_name]
-        return exchange.get_order(order_id)
-
-    @api_method
-    def cancel_order(self, order_param, exchange_name):
-        """Cancel an open order.
-
-        Parameters
-        ----------
-        order_param : str or Order
-            The order_id or order object to cancel.
-        """
-        exchange = self.exchanges[exchange_name]
-
-        order_id = order_param
-        if isinstance(order_param, zp.Order):
-            order_id = order_param.id
-
-        exchange.cancel_order(order_id)

catalyst/exchange/exchange_bcolz.py

@@ -1,98 +0,0 @@
-import numpy as np
-
-from catalyst import get_calendar
-from catalyst.data.minute_bars import BcolzMinuteBarReader, \
-    BcolzMinuteBarWriter
-
-
-class BcolzExchangeBarWriter(BcolzMinuteBarWriter):
-    def __init__(self, *args, **kwargs):
-        self._data_frequency = kwargs.pop('data_frequency', None)
-        kwargs.pop('minutes_per_day', None)
-        kwargs.pop('calendar', None)
-
-        end_session = kwargs.pop('end_session', None)
-        if end_session is not None:
-            end_session = end_session.floor('1d')
-
-        minutes_per_day = 1440 if self._data_frequency == 'minute' else 1
-        default_ohlc_ratio = kwargs.pop('default_ohlc_ratio', 100000000)
-        calendar = get_calendar('OPEN')
-
-        super(BcolzExchangeBarWriter, self) \
-            .__init__(*args, **dict(kwargs,
-                                    minutes_per_day=minutes_per_day,
-                                    default_ohlc_ratio=default_ohlc_ratio,
-                                    calendar=calendar,
-                                    end_session=end_session
-                                    ))
-
-
-class BcolzExchangeBarReader(BcolzMinuteBarReader):
-    def __init__(self, *args, **kwargs):
-        self._data_frequency = kwargs.pop('data_frequency', None)
-
-        super(BcolzExchangeBarReader, self).__init__(*args, **kwargs)
-
-    @property
-    def data_frequency(self):
-        return self._data_frequency
-
-    def load_raw_arrays(self, fields, start_dt, end_dt, sids):
-        """
-        Parameters
-        ----------
-        fields : list of str
-           'open', 'high', 'low', 'close', or 'volume'
-        start_dt: Timestamp
-           Beginning of the window range.
-        end_dt: Timestamp
-           End of the window range.
-        sids : list of int
-           The asset identifiers in the window.
-
-        Returns
-        -------
-        list of np.ndarray
-            A list with an entry per field of ndarrays with shape
-            (minutes in range, sids) with a dtype of float64, containing the
-            values for the respective field over start and end dt range.
-        """
-        start_idx = self._find_position_of_minute(start_dt)
-        end_idx = self._find_position_of_minute(end_dt)
-
-        periods = self.calendar.minutes_in_range(start_dt, end_dt) \
-            if self.data_frequency == 'minute' \
-            else self.calendar.sessions_in_range(start_dt, end_dt)
-
-        num_days = len(periods)
-        shape = num_days, len(sids)
-
-        all_fields = fields[:]
-        if len(all_fields) == 1 and all_fields[0] == 'volume':
-            all_fields.insert(0, 'close')
-
-        mask = None
-        data = []
-        for field in all_fields:
-            if field != 'volume':
-                out = np.full(shape, np.nan)
-            else:
-                out = np.zeros(shape, dtype=np.float64)
-
-            for i, sid in enumerate(sids):
-                carray = self._open_minute_file(field, sid)
-                a = carray[start_idx:end_idx + 1]
-
-                if mask is None:
-                    mask = a != 0
-
-                inverse_ratio = self._ohlc_ratio_inverse_for_sid(sid)
-                out[:len(mask), i][mask] = (
-                    a[mask] * inverse_ratio
-                )
-
-            if field in fields:
-                data.append(out)
-
-        return data

catalyst/exchange/exchange_blotter.py

@@ -1,134 +0,0 @@
-from catalyst.assets._assets import TradingPair
-from logbook import Logger
-
-from catalyst.constants import LOG_LEVEL
-from catalyst.finance.blotter import Blotter
-from catalyst.finance.commission import CommissionModel
-from catalyst.finance.slippage import SlippageModel
-from catalyst.finance.transaction import create_transaction
-
-log = Logger('exchange_blotter', level=LOG_LEVEL)
-
-# It seems like we need to accept greater slippage risk in cryptos
-# Orders won't often close at Equity levels.
-# TODO: should work with set_commission and set_slippage
-DEFAULT_SLIPPAGE_SPREAD = 0.0001
-DEFAULT_MAKER_FEE = 0.0015
-DEFAULT_TAKER_FEE = 0.0025
-
-
-class TradingPairFeeSchedule(CommissionModel):
-    """
-    Calculates a commission for a transaction based on a per percentage fee.
-
-    Parameters
-    ----------
-    fee : float, optional
-        The percentage fee.
-    """
-
-    def __init__(self,
-                 maker_fee=DEFAULT_MAKER_FEE,
-                 taker_fee=DEFAULT_TAKER_FEE):
-        self.maker_fee = maker_fee
-        self.taker_fee = taker_fee
-
-    def __repr__(self):
-        return (
-            '{class_name}(maker_fee={maker_fee}, '
-            'taker_fee={taker_fee})'.format(
-                class_name=self.__class__.__name__,
-                maker_fee=self.maker_fee,
-                taker_fee=self.taker_fee,
-            )
-        )
-
-    def calculate(self, order, transaction):
-        """
-        Calculate the final fee based on the order parameters.
-
-        :param order:
-        :param transaction:
-
-        :return float:
-            The total commission.
-        """
-        cost = abs(transaction.amount) * transaction.price
-
-        # Assuming just the taker fee for now
-        fee = cost * self.taker_fee
-        return fee
-
-
-class TradingPairFixedSlippage(SlippageModel):
-    """
-    Model slippage as a fixed spread.
-
-    Parameters
-    ----------
-    spread : float, optional
-        spread / 2 will be added to buys and subtracted from sells.
-    """
-
-    def __init__(self, spread=DEFAULT_SLIPPAGE_SPREAD):
-        super(TradingPairFixedSlippage, self).__init__()
-        self.spread = spread
-
-    def __repr__(self):
-        return '{class_name}(spread={spread})'.format(
-            class_name=self.__class__.__name__, spread=self.spread,
-        )
-
-    def simulate(self, data, asset, orders_for_asset):
-        self._volume_for_bar = 0
-
-        price = data.current(asset, 'close')
-
-        dt = data.current_dt
-        for order in orders_for_asset:
-            if order.open_amount == 0:
-                continue
-
-            order.check_triggers(price, dt)
-            if not order.triggered:
-                log.debug('order has not reached the trigger at current '
-                          'price {}'.format(price))
-                continue
-
-            execution_price, execution_volume = self.process_order(data, order)
-
-            transaction = create_transaction(
-                order, dt, execution_price, execution_volume
-            )
-
-            self._volume_for_bar += abs(transaction.amount)
-            yield order, transaction
-
-    def process_order(self, data, order):
-        price = data.current(order.asset, 'close')
-
-        if order.amount > 0:
-            # Buy order
-            adj_price = price * (1 + self.spread)
-        else:
-            # Sell order
-            adj_price = price * (1 - self.spread)
-
-        log.debug('added slippage to price: {} => {}'.format(price, adj_price))
-
-        return adj_price, order.amount
-
-
-class ExchangeBlotter(Blotter):
-    def __init__(self, *args, **kwargs):
-        super(ExchangeBlotter, self).__init__(*args, **kwargs)
-
-        # Using the equity models for now
-        # We may be able to define more sophisticated models based on the fee
-        # structure of each exchange.
-        self.slippage_models = {
-            TradingPair: TradingPairFixedSlippage()
-        }
-        self.commission_models = {
-            TradingPair: TradingPairFeeSchedule()
-        }

catalyst/exchange/exchange_bundle.py

@@ -1,928 +0,0 @@
-import os
-import shutil
-from functools import partial
-from itertools import chain
-from operator import is_not
-
-import numpy as np
-import pandas as pd
-from catalyst.assets._assets import TradingPair
-from datetime import datetime, timedelta
-from logbook import Logger
-from pytz import UTC
-from six import itervalues
-
-from catalyst import get_calendar
-from catalyst.constants import DATE_TIME_FORMAT, AUTO_INGEST
-from catalyst.constants import LOG_LEVEL
-from catalyst.data.minute_bars import BcolzMinuteOverlappingData, \
-    BcolzMinuteBarMetadata
-from catalyst.exchange.bundle_utils import range_in_bundle, \
-    get_bcolz_chunk, get_month_start_end, \
-    get_year_start_end, get_df_from_arrays, get_start_dt, get_period_label
-from catalyst.exchange.exchange_bcolz import BcolzExchangeBarReader, \
-    BcolzExchangeBarWriter
-from catalyst.exchange.exchange_errors import EmptyValuesInBundleError, \
-    TempBundleNotFoundError, \
-    NoDataAvailableOnExchange, \
-    PricingDataNotLoadedError
-from catalyst.exchange.exchange_utils import get_exchange_folder
-from catalyst.utils.cli import maybe_show_progress
-from catalyst.utils.paths import ensure_directory
-
-log = Logger('exchange_bundle', level=LOG_LEVEL)
-
-BUNDLE_NAME_TEMPLATE = os.path.join('{root}', '{frequency}_bundle')
-
-
-def _cachpath(symbol, type_):
-    return '-'.join([symbol, type_])
-
-
-class ExchangeBundle:
-    def __init__(self, exchange):
-        self.exchange = exchange
-        self.minutes_per_day = 1440
-        self.default_ohlc_ratio = 1000000
-        self._writers = dict()
-        self._readers = dict()
-        self.calendar = get_calendar('OPEN')
-
-    def get_assets(self, include_symbols, exclude_symbols):
-        # TODO: filter exclude symbols assets
-        if include_symbols is not None:
-            include_symbols_list = include_symbols.split(',')
-
-            return self.exchange.get_assets(include_symbols_list)
-
-        else:
-            return self.exchange.get_assets()
-
-    def get_reader(self, data_frequency, path=None):
-        """
-        Get a data writer object, either a new object or from cache
-
-        Returns
-        -------
-        BcolzMinuteBarReader | BcolzDailyBarReader
-
-        """
-        if path is None:
-            root = get_exchange_folder(self.exchange.name)
-            path = BUNDLE_NAME_TEMPLATE.format(
-                root=root,
-                frequency=data_frequency
-            )
-
-        if path in self._readers and self._readers[path] is not None:
-            return self._readers[path]
-
-        try:
-            self._readers[path] = BcolzExchangeBarReader(
-                rootdir=path,
-                data_frequency=data_frequency
-            )
-        except IOError:
-            self._readers[path] = None
-
-        return self._readers[path]
-
-    def update_metadata(self, writer, start_dt, end_dt):
-        pass
-
-    def get_writer(self, start_dt, end_dt, data_frequency):
-        """
-        Get a data writer object, either a new object or from cache
-
-        Returns
-        -------
-        BcolzMinuteBarWriter | BcolzDailyBarWriter
-
-        """
-        root = get_exchange_folder(self.exchange.name)
-        path = BUNDLE_NAME_TEMPLATE.format(
-            root=root,
-            frequency=data_frequency
-        )
-
-        if path in self._writers:
-            return self._writers[path]
-
-        ensure_directory(path)
-
-        if len(os.listdir(path)) > 0:
-
-            metadata = BcolzMinuteBarMetadata.read(path)
-
-            write_metadata = False
-            if start_dt < metadata.start_session:
-                write_metadata = True
-                start_session = start_dt
-            else:
-                start_session = metadata.start_session
-
-            if end_dt > metadata.end_session:
-                write_metadata = True
-
-                end_session = end_dt
-            else:
-                end_session = metadata.end_session
-
-            self._writers[path] = \
-                BcolzExchangeBarWriter(
-                    rootdir=path,
-                    start_session=start_session,
-                    end_session=end_session,
-                    write_metadata=write_metadata,
-                    data_frequency=data_frequency
-                )
-        else:
-            self._writers[path] = BcolzExchangeBarWriter(
-                rootdir=path,
-                start_session=start_dt,
-                end_session=end_dt,
-                write_metadata=True,
-                data_frequency=data_frequency
-            )
-
-        return self._writers[path]
-
-    def filter_existing_assets(self, assets, start_dt, end_dt, data_frequency):
-        """
-        For each asset, get the close on the start and end dates of the chunk.
-            If the data exists, the chunk ingestion is complete.
-            If any data is missing we ingest the data.
-
-        Parameters
-        ----------
-        assets: list[TradingPair]
-            The assets is scope.
-        start_dt: datetime
-            The chunk start date.
-        end_dt: datetime
-            The chunk end date.
-        data_frequency: str
-
-        Returns
-        -------
-        list[TradingPair]
-            The assets missing from the bundle
-        """
-        reader = self.get_reader(data_frequency)
-        missing_assets = []
-        for asset in assets:
-            has_data = range_in_bundle(asset, start_dt, end_dt, reader)
-
-            if not has_data:
-                missing_assets.append(asset)
-
-        return missing_assets
-
-    def _write(self, data, writer, data_frequency):
-        try:
-            writer.write(
-                data=data,
-                show_progress=False,
-                invalid_data_behavior='raise'
-            )
-        except BcolzMinuteOverlappingData as e:
-            log.debug('chunk already exists: {}'.format(e))
-        except Exception as e:
-            log.warn('error when writing data: {}, trying again'.format(e))
-
-            # This is workaround, there is an issue with empty
-            # session_label when using a newly created writer
-            del self._writers[writer._rootdir]
-
-            writer = self.get_writer(writer._start_session,
-                                     writer._end_session, data_frequency)
-            writer.write(
-                data=data,
-                show_progress=False,
-                invalid_data_behavior='raise'
-            )
-
-    def get_calendar_periods_range(self, start_dt, end_dt, data_frequency):
-        """
-        Get a list of dates for the specified range.
-
-        Parameters
-        ----------
-        start_dt: datetime
-        end_dt: datetime
-        data_frequency: str
-
-        Returns
-        -------
-        list[datetime]
-
-        """
-        return self.calendar.minutes_in_range(start_dt, end_dt) \
-            if data_frequency == 'minute' \
-            else self.calendar.sessions_in_range(start_dt, end_dt)
-
-    def _spot_empty_periods(self, ohlcv_df, asset, data_frequency,
-                            empty_rows_behavior):
-        problems = []
-
-        nan_rows = ohlcv_df[ohlcv_df.isnull().T.any().T].index
-        if len(nan_rows) > 0:
-            dates = []
-            for row_date in nan_rows.values:
-                row_date = pd.to_datetime(row_date, utc=True)
-                if row_date > asset.start_date:
-                    dates.append(row_date)
-
-            if len(dates) > 0:
-                end_dt = asset.end_minute if data_frequency == 'minute' \
-                    else asset.end_daily
-
-                problem = '{name} ({start_dt} to {end_dt}) has empty ' \
-                          'periods: {dates}'.format(
-                    name=asset.symbol,
-                    start_dt=asset.start_date.strftime(DATE_TIME_FORMAT),
-                    end_dt=end_dt.strftime(DATE_TIME_FORMAT),
-                    dates=[date.strftime(DATE_TIME_FORMAT) for date in dates]
-                )
-                if empty_rows_behavior == 'warn':
-                    log.warn(problem)
-
-                elif empty_rows_behavior == 'raise':
-                    raise EmptyValuesInBundleError(
-                        name=asset.symbol,
-                        end_minute=end_dt,
-                        dates=dates
-                    )
-
-                else:
-                    ohlcv_df.dropna(inplace=True)
-
-            else:
-                problem = None
-
-            problems.append(problem)
-
-        return problems
-
-    def _spot_duplicates(self, ohlcv_df, asset, data_frequency, threshold):
-        # TODO: work in progress
-        series = ohlcv_df.reset_index().groupby('close')['index'].apply(
-            np.array
-        )
-
-        ref_delta = timedelta(minutes=1) if data_frequency == 'minute' \
-            else timedelta(days=1)
-
-        dups = series.loc[lambda values: [len(x) > 10 for x in values]]
-
-        for index, dates in dups.iteritems():
-            prev_date = None
-            for date in dates:
-                if prev_date is not None:
-                    delta = (date - prev_date) / 1e9
-                    if delta == ref_delta.seconds:
-                        log.info('pex')
-
-                prev_date = date
-
-        problems = []
-        for index, dates in dups.iteritems():
-            end_dt = asset.end_minute if data_frequency == 'minute' \
-                else asset.end_daily
-
-            problem = '{name} ({start_dt} to {end_dt}) has {threshold} ' \
-                      'identical close values on: {dates}'.format(
-                name=asset.symbol,
-                start_dt=asset.start_date.strftime(DATE_TIME_FORMAT),
-                end_dt=end_dt.strftime(DATE_TIME_FORMAT),
-                threshold=threshold,
-                dates=[pd.to_datetime(date).strftime(DATE_TIME_FORMAT)
-                       for date in dates]
-            )
-
-            problems.append(problem)
-
-        return problems
-
-    def ingest_df(self, ohlcv_df, data_frequency, asset, writer,
-                  empty_rows_behavior='warn', duplicates_threshold=None):
-        """
-        Ingest a DataFrame of OHLCV data for a given market.
-
-        Parameters
-        ----------
-        ohlcv_df: DataFrame
-        data_frequency: str
-        asset: TradingPair
-        writer:
-        empty_rows_behavior: str
-
-        """
-        problems = []
-        if empty_rows_behavior is not 'ignore':
-            problems += self._spot_empty_periods(
-                ohlcv_df, asset, data_frequency, empty_rows_behavior
-            )
-
-        # if duplicates_threshold is not None:
-        #     problems += self._spot_duplicates(
-        #         ohlcv_df, asset, data_frequency, duplicates_threshold
-        #     )
-
-        data = []
-        if not ohlcv_df.empty:
-            ohlcv_df.sort_index(inplace=True)
-            data.append((asset.sid, ohlcv_df))
-
-        self._write(data, writer, data_frequency)
-
-        return problems
-
-    def ingest_ctable(self, asset, data_frequency, period,
-                      writer, empty_rows_behavior='strip',
-                      duplicates_threshold=100, cleanup=False):
-        """
-        Merge a ctable bundle chunk into the main bundle for the exchange.
-
-        Parameters
-        ----------
-        asset: TradingPair
-        data_frequency: str
-        period: str
-        writer:
-        empty_rows_behavior: str
-            Ensure that the bundle does not have any missing data.
-
-        cleanup: bool
-            Remove the temp bundle directory after ingestion.
-
-        Returns
-        -------
-        list[str]
-            A list of problems which occurred during ingestion.
-
-        """
-        problems = []
-
-        # Download and extract the bundle
-        path = get_bcolz_chunk(
-            exchange_name=self.exchange.name,
-            symbol=asset.symbol,
-            data_frequency=data_frequency,
-            period=period
-        )
-
-        reader = self.get_reader(data_frequency, path=path)
-        if reader is None:
-            try:
-                log.warn('the reader is unable to use bundle: {}, '
-                         'deleting it.'.format(path))
-                shutil.rmtree(path)
-
-            except Exception as e:
-                log.warn('unable to remove temp bundle: {}'.format(e))
-
-            raise TempBundleNotFoundError(path=path)
-
-        start_dt = reader.first_trading_day
-        end_dt = reader.last_available_dt
-
-        if data_frequency == 'daily':
-            end_dt = end_dt - pd.Timedelta(hours=23, minutes=59)
-
-        arrays = None
-        try:
-            arrays = reader.load_raw_arrays(
-                sids=[asset.sid],
-                fields=['open', 'high', 'low', 'close', 'volume'],
-                start_dt=start_dt,
-                end_dt=end_dt
-            )
-        except Exception as e:
-            log.warn('skipping ctable for {} from {} to {}: {}'.format(
-                asset.symbol, start_dt, end_dt, e
-            ))
-
-        if not arrays:
-            return reader._rootdir
-
-        periods = self.get_calendar_periods_range(
-            start_dt, end_dt, data_frequency
-        )
-        df = get_df_from_arrays(arrays, periods)
-        problems += self.ingest_df(
-            ohlcv_df=df,
-            data_frequency=data_frequency,
-            asset=asset,
-            writer=writer,
-            empty_rows_behavior=empty_rows_behavior,
-            duplicates_threshold=duplicates_threshold
-        )
-
-        if cleanup:
-            log.debug(
-                'removing bundle folder following ingestion: {}'.format(
-                    reader._rootdir)
-            )
-            shutil.rmtree(reader._rootdir)
-
-        return filter(partial(is_not, None), problems)
-
-    def get_adj_dates(self, start, end, assets, data_frequency):
-        """
-        Contains a date range to the trading availability of the specified
-        markets.
-
-        Parameters
-        ----------
-        start: datetime
-        end: datetime
-        assets: list[TradingPair]
-        data_frequency: str
-
-        Returns
-        -------
-        datetime, datetime
-        """
-        earliest_trade = None
-        last_entry = None
-        for asset in assets:
-            if earliest_trade is None or earliest_trade > asset.start_date:
-                if asset.start_date >= self.calendar.first_session:
-                    earliest_trade = asset.start_date
-
-                else:
-                    earliest_trade = self.calendar.first_session
-
-            end_asset = asset.end_minute if data_frequency == 'minute' else \
-                asset.end_daily
-            if end_asset is not None:
-                if last_entry is None or end_asset > last_entry:
-                    last_entry = end_asset
-
-            else:
-                end = None
-                last_entry = None
-
-        if start is None or \
-                (earliest_trade is not None and earliest_trade > start):
-            start = earliest_trade
-
-        if end is None or (last_entry is not None and end > last_entry):
-            end = last_entry
-
-        if end is None or start is None or start > end:
-            raise NoDataAvailableOnExchange(
-                exchange=[asset.exchange for asset in assets],
-                symbol=[asset.symbol for asset in assets],
-                data_frequency=data_frequency,
-            )
-
-        return start, end
-
-    def prepare_chunks(self, assets, data_frequency, start_dt, end_dt):
-        """
-        Split a price data request into chunks corresponding to individual
-        bundles.
-
-        Parameters
-        ----------
-        assets: list[TradingPair]
-        data_frequency: str
-        start_dt: datetime
-        end_dt: datetime
-
-        Returns
-        -------
-        dict[TradingPair, list[dict(str, Object]]]
-
-        """
-        get_start_end = get_month_start_end \
-            if data_frequency == 'minute' else get_year_start_end
-
-        # Get a reader for the main bundle to verify if data exists
-        reader = self.get_reader(data_frequency)
-
-        chunks = dict()
-        for asset in assets:
-            try:
-                # Checking if the the asset has price data in the specified
-                # date range
-                adj_start, adj_end = self.get_adj_dates(
-                    start_dt, end_dt, [asset], data_frequency
-                )
-
-            except NoDataAvailableOnExchange as e:
-                # If not, we continue to the next asset
-                log.debug('skipping {}: {}'.format(asset.symbol, e))
-                continue
-
-            dates = pd.date_range(
-                start=get_period_label(adj_start, data_frequency),
-                end=get_period_label(adj_end, data_frequency),
-                freq='MS' if data_frequency == 'minute' else 'AS',
-                tz=UTC
-            )
-
-            # Adjusting the last date of the range to avoid
-            # going over the asset's trading bounds
-            dates.values[0] = adj_start
-            dates.values[-1] = adj_end
-
-            chunks[asset] = []
-            for index, dt in enumerate(dates):
-                period_start, period_end = get_start_end(
-                    dt=dt,
-                    first_day=dt if index == 0 else None,
-                    last_day=dt if index == len(dates) - 1 else None
-                )
-
-                # Currencies don't always start trading at midnight.
-                # Checking the last minute of the day instead.
-                range_start = period_start.replace(hour=23, minute=59) \
-                    if data_frequency == 'minute' else period_start
-
-                # Checking if the data already exists in the bundle
-                # for the date range of the chunk. If not, we create
-                # a chunk for ingestion.
-                has_data = range_in_bundle(
-                    asset, range_start, period_end, reader
-                )
-                if not has_data:
-                    period = get_period_label(dt, data_frequency)
-                    chunk = dict(
-                        asset=asset,
-                        period=period,
-                    )
-                    chunks[asset].append(chunk)
-
-            # We sort the chunks by end date to ingest most recent data first
-            chunks[asset].sort(
-                key=lambda chunk: pd.to_datetime(chunk['period'])
-            )
-
-        return chunks
-
-    def ingest_assets(self, assets, data_frequency, start_dt=None, end_dt=None,
-                      show_progress=False, show_breakdown=False,
-                      show_report=False):
-        """
-        Determine if data is missing from the bundle and attempt to ingest it.
-
-        Parameters
-        ----------
-        assets: list[TradingPair]
-        data_frequency: str
-        start_dt: datetime
-        end_dt: datetime
-        show_progress: bool
-        show_breakdown: bool
-
-        """
-        if start_dt is None:
-            start_dt = self.calendar.first_session
-
-        if end_dt is None:
-            end_dt = pd.Timestamp.utcnow()
-
-        get_start_end = get_month_start_end \
-            if data_frequency == 'minute' else get_year_start_end
-
-        # Assign the first and last day of the period
-        start_dt, _ = get_start_end(start_dt)
-        _, end_dt = get_start_end(end_dt)
-
-        chunks = self.prepare_chunks(
-            assets=assets,
-            data_frequency=data_frequency,
-            start_dt=start_dt,
-            end_dt=end_dt
-        )
-
-        problems = []
-        # This is the common writer for the entire exchange bundle
-        # we want to give an end_date far in time
-        writer = self.get_writer(start_dt, end_dt, data_frequency)
-        if show_breakdown:
-            for asset in chunks:
-                with maybe_show_progress(
-                        chunks[asset],
-                        show_progress,
-                        label='Ingesting {frequency} price data for '
-                              '{symbol} on {exchange}'.format(
-                            exchange=self.exchange.name,
-                            frequency=data_frequency,
-                            symbol=asset.symbol
-                        )) as it:
-                    for chunk in it:
-                        problems += self.ingest_ctable(
-                            asset=chunk['asset'],
-                            data_frequency=data_frequency,
-                            period=chunk['period'],
-                            writer=writer,
-                            empty_rows_behavior='strip',
-                            cleanup=True
-                        )
-        else:
-            all_chunks = list(chain.from_iterable(itervalues(chunks)))
-
-            # We sort the chunks by end date to ingest most recent data first
-            all_chunks.sort(
-                key=lambda chunk: pd.to_datetime(chunk['period'])
-            )
-            with maybe_show_progress(
-                    all_chunks,
-                    show_progress,
-                    label='Ingesting {frequency} price data on '
-                          '{exchange}'.format(
-                        exchange=self.exchange.name,
-                        frequency=data_frequency,
-                    )) as it:
-                for chunk in it:
-                    problems += self.ingest_ctable(
-                        asset=chunk['asset'],
-                        data_frequency=data_frequency,
-                        period=chunk['period'],
-                        writer=writer,
-                        empty_rows_behavior='strip',
-                        cleanup=True
-                    )
-
-        if show_report and len(problems) > 0:
-            log.info('problems during ingestion:{}\n'.format(
-                '\n'.join(problems)
-            ))
-
-    def ingest(self, data_frequency, include_symbols=None,
-               exclude_symbols=None, start=None, end=None,
-               show_progress=True, show_breakdown=True, show_report=True):
-        """
-        Inject data based on specified parameters.
-
-        Parameters
-        ----------
-        data_frequency: str
-        include_symbols: str
-        exclude_symbols: str
-        start: datetime
-        end: datetime
-        show_progress: bool
-        environ:
-
-        """
-        assets = self.get_assets(include_symbols, exclude_symbols)
-
-        for frequency in data_frequency.split(','):
-            self.ingest_assets(assets, frequency, start, end,
-                               show_progress, show_breakdown, show_report)
-
-    def get_history_window_series_and_load(self,
-                                           assets,
-                                           end_dt,
-                                           bar_count,
-                                           field,
-                                           data_frequency,
-                                           algo_end_dt=None
-                                           ):
-        """
-        Retrieve price data history, ingest missing data.
-
-        Parameters
-        ----------
-        assets: list[TradingPair]
-        end_dt: datetime
-        bar_count: int
-        field: str
-        data_frequency: str
-        algo_end_dt: datetime
-
-        Returns
-        -------
-        Series
-
-        """
-        if AUTO_INGEST:
-            try:
-                series = self.get_history_window_series(
-                    assets=assets,
-                    end_dt=end_dt,
-                    bar_count=bar_count,
-                    field=field,
-                    data_frequency=data_frequency
-                )
-                return pd.DataFrame(series)
-
-            except PricingDataNotLoadedError:
-                start_dt = get_start_dt(end_dt, bar_count, data_frequency)
-                log.info(
-                    'pricing data for {symbol} not found in range '
-                    '{start} to {end}, updating the bundles.'.format(
-                        symbol=[asset.symbol for asset in assets],
-                        start=start_dt,
-                        end=end_dt
-                    )
-                )
-                self.ingest_assets(
-                    assets=assets,
-                    start_dt=start_dt,
-                    end_dt=algo_end_dt,
-                    data_frequency=data_frequency,
-                    show_progress=True,
-                    show_breakdown=True
-                )
-                series = self.get_history_window_series(
-                    assets=assets,
-                    end_dt=end_dt,
-                    bar_count=bar_count,
-                    field=field,
-                    data_frequency=data_frequency,
-                    reset_reader=True
-                )
-                return series
-
-        else:
-            series = self.get_history_window_series(
-                assets=assets,
-                end_dt=end_dt,
-                bar_count=bar_count,
-                field=field,
-                data_frequency=data_frequency
-            )
-            return pd.DataFrame(series)
-
-    def get_spot_values(self,
-                        assets,
-                        field,
-                        dt,
-                        data_frequency,
-                        reset_reader=False
-                        ):
-        """
-        The spot values for the gives assets, field and date. Reads from
-        the exchange data bundle.
-
-        Parameters
-        ----------
-        assets: list[TradingPair]
-        field: str
-        dt: pd.Timestamp
-        data_frequency: str
-        reset_reader:
-
-        Returns
-        -------
-        float
-
-        """
-        values = []
-        try:
-            reader = self.get_reader(data_frequency)
-            if reset_reader:
-                del self._readers[reader._rootdir]
-                reader = self.get_reader(data_frequency)
-
-            for asset in assets:
-                value = reader.get_value(
-                    sid=asset.sid,
-                    dt=dt,
-                    field=field
-                )
-                values.append(value)
-
-            return values
-
-        except Exception:
-            symbols = [asset.symbol for asset in assets]
-            raise PricingDataNotLoadedError(
-                field=field,
-                first_trading_day=min([asset.start_date for asset in assets]),
-                exchange=self.exchange.name,
-                symbols=symbols,
-                symbol_list=','.join(symbols),
-                data_frequency=data_frequency,
-                start_dt=dt,
-                end_dt=dt
-            )
-
-    def get_history_window_series(self,
-                                  assets,
-                                  end_dt,
-                                  bar_count,
-                                  field,
-                                  data_frequency,
-                                  reset_reader=False):
-        start_dt = get_start_dt(end_dt, bar_count, data_frequency, False)
-        start_dt, end_dt = self.get_adj_dates(
-            start_dt, end_dt, assets, data_frequency
-        )
-
-        reader = self.get_reader(data_frequency)
-        if reset_reader:
-            del self._readers[reader._rootdir]
-            reader = self.get_reader(data_frequency)
-
-        if reader is None:
-            symbols = [asset.symbol for asset in assets]
-            raise PricingDataNotLoadedError(
-                field=field,
-                first_trading_day=min([asset.start_date for asset in assets]),
-                exchange=self.exchange.name,
-                symbols=symbols,
-                symbol_list=','.join(symbols),
-                data_frequency=data_frequency,
-                start_dt=start_dt,
-                end_dt=end_dt
-            )
-
-        for asset in assets:
-            asset_start_dt, asset_end_dt = self.get_adj_dates(
-                start_dt, end_dt, assets, data_frequency
-            )
-
-            in_bundle = range_in_bundle(
-                asset, asset_start_dt, asset_end_dt, reader
-            )
-            if not in_bundle:
-                raise PricingDataNotLoadedError(
-                    field=field,
-                    first_trading_day=asset.start_date,
-                    exchange=self.exchange.name,
-                    symbols=asset.symbol,
-                    symbol_list=asset.symbol,
-                    data_frequency=data_frequency,
-                    start_dt=asset_start_dt,
-                    end_dt=asset_end_dt
-                )
-
-        series = dict()
-        try:
-            arrays = reader.load_raw_arrays(
-                sids=[asset.sid for asset in assets],
-                fields=[field],
-                start_dt=start_dt,
-                end_dt=end_dt
-            )
-
-        except Exception:
-            symbols = [asset.symbol.encode('utf-8') for asset in assets]
-            raise PricingDataNotLoadedError(
-                field=field,
-                first_trading_day=min([asset.start_date for asset in assets]),
-                exchange=self.exchange.name,
-                symbols=symbols,
-                symbol_list=','.join(symbols),
-                data_frequency=data_frequency,
-                start_dt=start_dt,
-                end_dt=end_dt
-            )
-
-        periods = self.get_calendar_periods_range(
-            start_dt, end_dt, data_frequency
-        )
-
-        for asset_index, asset in enumerate(assets):
-            asset_values = arrays[asset_index]
-
-            value_series = pd.Series(asset_values.flatten(), index=periods)
-            series[asset] = value_series
-
-        return series
-
-    def clean(self, data_frequency):
-        """
-        Removing the bundle data from the catalyst folder.
-
-        Parameters
-        ----------
-        data_frequency: str
-
-        """
-        log.debug('cleaning exchange {}, frequency {}'.format(
-            self.exchange.name, data_frequency
-        ))
-        root = get_exchange_folder(self.exchange.name)
-
-        symbols = os.path.join(root, 'symbols.json')
-        if os.path.isfile(symbols):
-            os.remove(symbols)
-
-        temp_bundles = os.path.join(root, 'temp_bundles')
-
-        if os.path.isdir(temp_bundles):
-            log.debug('removing folder and content: {}'.format(temp_bundles))
-            shutil.rmtree(temp_bundles)
-            log.debug('{} removed'.format(temp_bundles))
-
-        frequencies = ['daily', 'minute'] if data_frequency is None \
-            else [data_frequency]
-
-        for frequency in frequencies:
-            label = '{}_bundle'.format(frequency)
-            frequency_bundle = os.path.join(root, label)
-
-            if os.path.isdir(frequency_bundle):
-                log.debug(
-                    'removing folder and content: {}'.format(frequency_bundle)
-                )
-                shutil.rmtree(frequency_bundle)
-                log.debug('{} removed'.format(frequency_bundle))

catalyst/exchange/exchange_data_portal.py

@@ -1,405 +0,0 @@
-import abc
-from time import sleep
-
-import numpy as np
-import pandas as pd
-from catalyst.assets._assets import TradingPair
-from logbook import Logger
-
-from catalyst.constants import LOG_LEVEL, AUTO_INGEST
-from catalyst.data.data_portal import DataPortal
-from catalyst.exchange.exchange_bundle import ExchangeBundle
-from catalyst.exchange.exchange_errors import (
-    ExchangeRequestError,
-    ExchangeBarDataError,
-    PricingDataNotLoadedError)
-from catalyst.exchange.exchange_utils import get_frequency, resample_history_df
-
-log = Logger('DataPortalExchange', level=LOG_LEVEL)
-
-
-class DataPortalExchangeBase(DataPortal):
-    def __init__(self, *args, **kwargs):
-
-        self.exchanges = kwargs.pop('exchanges', None)
-        # TODO: put somewhere accessible by each algo
-        self.retry_get_history_window = 5
-        self.retry_get_spot_value = 5
-        self.retry_delay = 5
-
-        super(DataPortalExchangeBase, self).__init__(*args, **kwargs)
-
-    def _get_history_window(self,
-                            assets,
-                            end_dt,
-                            bar_count,
-                            frequency,
-                            field,
-                            data_frequency,
-                            ffill=True,
-                            attempt_index=0):
-        try:
-            exchange_assets = dict()
-            for asset in assets:
-                if asset.exchange not in exchange_assets:
-                    exchange_assets[asset.exchange] = list()
-
-                exchange_assets[asset.exchange].append(asset)
-
-            if len(exchange_assets) > 1:
-                df_list = []
-                for exchange_name in exchange_assets:
-                    exchange = self.exchanges[exchange_name]
-                    assets = exchange_assets[exchange_name]
-
-                    df_exchange = self.get_exchange_history_window(
-                        exchange,
-                        assets,
-                        end_dt,
-                        bar_count,
-                        frequency,
-                        field,
-                        data_frequency,
-                        ffill)
-
-                    df_list.append(df_exchange)
-
-                # Merging the values values of each exchange
-                return pd.concat(df_list)
-
-            else:
-                exchange = self.exchanges[list(exchange_assets.keys())[0]]
-                return self.get_exchange_history_window(
-                    exchange,
-                    assets,
-                    end_dt,
-                    bar_count,
-                    frequency,
-                    field,
-                    data_frequency,
-                    ffill)
-
-        except ExchangeRequestError as e:
-            log.warn(
-                'get history attempt {}: {}'.format(attempt_index, e)
-            )
-            if attempt_index < self.retry_get_history_window:
-                sleep(self.retry_delay)
-                return self._get_history_window(assets,
-                                                end_dt,
-                                                bar_count,
-                                                frequency,
-                                                field,
-                                                data_frequency,
-                                                ffill,
-                                                attempt_index + 1)
-            else:
-                raise ExchangeBarDataError(
-                    data_type='history',
-                    attempts=attempt_index,
-                    error=e
-                )
-
-    def get_history_window(self,
-                           assets,
-                           end_dt,
-                           bar_count,
-                           frequency,
-                           field,
-                           data_frequency=None,
-                           ffill=True):
-
-        if field == 'price':
-            field = 'close'
-
-        return self._get_history_window(assets,
-                                        end_dt,
-                                        bar_count,
-                                        frequency,
-                                        field,
-                                        data_frequency,
-                                        ffill)
-
-    @abc.abstractmethod
-    def get_exchange_history_window(self,
-                                    exchange,
-                                    assets,
-                                    end_dt,
-                                    bar_count,
-                                    frequency,
-                                    field,
-                                    data_frequency,
-                                    ffill=True):
-        pass
-
-    def _get_spot_value(self, assets, field, dt, data_frequency,
-                        attempt_index=0):
-        try:
-            if isinstance(assets, TradingPair):
-                exchange = self.exchanges[assets.exchange]
-                spot_values = self.get_exchange_spot_value(
-                    exchange, [assets], field, dt, data_frequency)
-
-                if not spot_values:
-                    return np.nan
-
-                return spot_values[0]
-
-            else:
-                exchange_assets = dict()
-                for asset in assets:
-                    if asset.exchange not in exchange_assets:
-                        exchange_assets[asset.exchange] = list()
-
-                    exchange_assets[asset.exchange].append(asset)
-
-                if len(list(exchange_assets.keys())) == 1:
-                    exchange = self.exchanges[list(exchange_assets.keys())[0]]
-                    return self.get_exchange_spot_value(
-                        exchange, assets, field, dt, data_frequency)
-
-                else:
-                    spot_values = []
-                    for exchange_name in exchange_assets:
-                        exchange = self.exchanges[exchange_name]
-                        assets = exchange_assets[exchange_name]
-                        exchange_spot_values = self.get_exchange_spot_value(
-                            exchange,
-                            assets,
-                            field,
-                            dt,
-                            data_frequency
-                        )
-                        if len(assets) == 1:
-                            spot_values.append(exchange_spot_values)
-                        else:
-                            spot_values += exchange_spot_values
-
-                    return spot_values
-
-        except ExchangeRequestError as e:
-            log.warn(
-                'get spot value attempt {}: {}'.format(attempt_index, e)
-            )
-            if attempt_index < self.retry_get_spot_value:
-                sleep(self.retry_delay)
-                return self._get_spot_value(assets, field, dt, data_frequency,
-                                            attempt_index + 1)
-            else:
-                raise ExchangeBarDataError(
-                    data_type='spot',
-                    attempts=attempt_index,
-                    error=e
-                )
-
-    def get_spot_value(self, assets, field, dt, data_frequency):
-        if field == 'price':
-            field = 'close'
-
-        return self._get_spot_value(assets, field, dt, data_frequency)
-
-    @abc.abstractmethod
-    def get_exchange_spot_value(self, exchange, assets, field, dt,
-                                data_frequency):
-        return
-
-    def get_adjusted_value(self, asset, field, dt,
-                           perspective_dt,
-                           data_frequency,
-                           spot_value=None):
-        # TODO: does this pertain to cryptocurrencies?
-        log.warn('get_adjusted_value is not implemented yet!')
-        return spot_value
-
-
-class DataPortalExchangeLive(DataPortalExchangeBase):
-    def __init__(self, *args, **kwargs):
-        super(DataPortalExchangeLive, self).__init__(*args, **kwargs)
-
-    def get_exchange_history_window(self,
-                                    exchange,
-                                    assets,
-                                    end_dt,
-                                    bar_count,
-                                    frequency,
-                                    field,
-                                    data_frequency,
-                                    ffill=True):
-        """
-        Fetching price history window from the exchange.
-
-        Parameters
-        ----------
-        exchange: Exchange
-        assets: list[TradingPair]
-        end_dt: datetime
-        bar_count: int
-        frequency: str
-        field: str
-        data_frequency: str
-        ffill: bool
-
-        Returns
-        -------
-        DataFrame
-
-        """
-        df = exchange.get_history_window(
-            assets,
-            end_dt,
-            bar_count,
-            frequency,
-            field,
-            data_frequency,
-            ffill)
-        return df
-
-    def get_exchange_spot_value(self, exchange, assets, field, dt,
-                                data_frequency):
-        """
-        A spot value for the exchange.
-
-        Parameters
-        ----------
-        exchange: Exchange
-        assets: list[TradingPair]
-        field: str
-        dt: datetime
-        data_frequency: str
-
-        Returns
-        -------
-        float
-
-        """
-        exchange_spot_values = exchange.get_spot_value(
-            assets, field, dt, data_frequency)
-
-        return exchange_spot_values
-
-
-class DataPortalExchangeBacktest(DataPortalExchangeBase):
-    def __init__(self, *args, **kwargs):
-        super(DataPortalExchangeBacktest, self).__init__(*args, **kwargs)
-
-        self.exchange_bundles = dict()
-
-        self.history_loaders = dict()
-        self.minute_history_loaders = dict()
-
-        for exchange_name in self.exchanges:
-            exchange = self.exchanges[exchange_name]
-            self.exchange_bundles[exchange_name] = ExchangeBundle(exchange)
-
-    def _get_first_trading_day(self, assets):
-        first_date = None
-        for asset in assets:
-            if first_date is None or asset.start_date > first_date:
-                first_date = asset.start_date
-        return first_date
-
-    def get_exchange_history_window(self,
-                                    exchange,
-                                    assets,
-                                    end_dt,
-                                    bar_count,
-                                    frequency,
-                                    field,
-                                    data_frequency,
-                                    ffill=True):
-        """
-        Fetching price history window from the exchange bundle.
-
-        Parameters
-        ----------
-        exchange: Exchange
-        assets: list[TradingPair]
-        end_dt: datetime
-        bar_count: int
-        frequency: str
-        field: str
-        data_frequency: str
-        ffill: bool
-
-        Returns
-        -------
-        DataFrame
-
-        """
-        bundle = self.exchange_bundles[exchange.name]  # type: ExchangeBundle
-
-        freq, candle_size, unit, adj_data_frequency = get_frequency(
-            frequency, data_frequency
-        )
-        adj_bar_count = candle_size * bar_count
-
-        if data_frequency == 'minute' and adj_data_frequency == 'daily':
-            end_dt = end_dt.floor('1D')
-
-        series = bundle.get_history_window_series_and_load(
-            assets=assets,
-            end_dt=end_dt,
-            bar_count=adj_bar_count,
-            field=field,
-            data_frequency=adj_data_frequency,
-            algo_end_dt=self._last_available_session,
-        )
-
-        df = resample_history_df(pd.DataFrame(series), freq, field)
-        return df
-
-    def get_exchange_spot_value(self,
-                                exchange,
-                                assets,
-                                field,
-                                dt,
-                                data_frequency
-                                ):
-        """
-        A spot value for the exchange bundle. Try to ingest data if not in
-        the bundle.
-
-        Parameters
-        ----------
-        exchange: Exchange
-        assets: list[TradingPair]
-        field: str
-        dt: datetime
-        data_frequency: str
-
-        Returns
-        -------
-        float
-
-        """
-        bundle = self.exchange_bundles[exchange.name]
-        if data_frequency == 'daily':
-            dt = dt.floor('1D')
-        else:
-            dt = dt.floor('1 min')
-
-        if AUTO_INGEST:
-            try:
-                return bundle.get_spot_values(
-                    assets, field, dt, data_frequency
-                )
-            except PricingDataNotLoadedError:
-                log.info(
-                    'pricing data for {symbol} not found on {dt}'
-                    ', updating the bundles.'.format(
-                        symbol=[asset.symbol for asset in assets],
-                        dt=dt
-                    )
-                )
-                bundle.ingest_assets(
-                    assets=assets,
-                    start_dt=self._first_trading_day,
-                    end_dt=self._last_available_session,
-                    data_frequency=data_frequency,
-                    show_progress=True
-                )
-                return bundle.get_spot_values(
-                    assets, field, dt, data_frequency, True
-                )
-        else:
-            return bundle.get_spot_values(assets, field, dt, data_frequency)

catalyst/exchange/exchange_errors.py

@@ -1,229 +0,0 @@
-import sys
-import traceback
-
-from catalyst.errors import ZiplineError
-
-
-def silent_except_hook(exctype, excvalue, exctraceback):
-    if exctype in [PricingDataBeforeTradingError, PricingDataNotLoadedError,
-                   SymbolNotFoundOnExchange, NoDataAvailableOnExchange,
-                   ExchangeAuthEmpty]:
-        fn = traceback.extract_tb(exctraceback)[-1][0]
-        ln = traceback.extract_tb(exctraceback)[-1][1]
-        print("Error traceback: {1} (line {2})\n"
-              "{0.__name__}:  {3}".format(exctype, fn, ln, excvalue))
-    else:
-        sys.__excepthook__(exctype, excvalue, exctraceback)
-
-
-sys.excepthook = silent_except_hook
-
-
-class ExchangeRequestError(ZiplineError):
-    msg = (
-        'Request failed: {error}'
-    ).strip()
-
-
-class ExchangeRequestErrorTooManyAttempts(ZiplineError):
-    msg = (
-        'Request failed: {error}, giving up after {attempts} attempts'
-    ).strip()
-
-
-class ExchangeBarDataError(ZiplineError):
-    msg = (
-        'Unable to retrieve bar data: {data_type}, ' +
-        'giving up after {attempts} attempts: {error}'
-    ).strip()
-
-
-class ExchangePortfolioDataError(ZiplineError):
-    msg = (
-        'Unable to retrieve portfolio data: {data_type}, ' +
-        'giving up after {attempts} attempts: {error}'
-    ).strip()
-
-
-class ExchangeTransactionError(ZiplineError):
-    msg = (
-        'Unable to execute transaction: {transaction_type}, ' +
-        'giving up after {attempts} attempts: {error}'
-    ).strip()
-
-
-class ExchangeNotFoundError(ZiplineError):
-    msg = (
-        'Exchange {exchange_name} not found. Please specify exchanges '
-        'supported by Catalyst and verify spelling for accuracy.'
-    ).strip()
-
-
-class ExchangeAuthNotFound(ZiplineError):
-    msg = (
-        'Please create an auth.json file containing the api token and key for '
-        'exchange {exchange}. Place the file here: {filename}'
-    ).strip()
-
-
-class ExchangeAuthEmpty(ZiplineError):
-    msg = (
-        'Please enter your API token key and secret for exchange {exchange} '
-        'in the following file: {filename}'
-    ).strip()
-
-
-class ExchangeSymbolsNotFound(ZiplineError):
-    msg = (
-        'Unable to download or find a local copy of symbols.json for exchange '
-        '{exchange}. The file should be here: {filename}'
-    ).strip()
-
-
-class AlgoPickleNotFound(ZiplineError):
-    msg = (
-        'Pickle not found for algo {algo} in path {filename}'
-    ).strip()
-
-
-class InvalidHistoryFrequencyAlias(ZiplineError):
-    msg = (
-        'Invalid frequency alias {freq}. Valid suffixes are M (minute) '
-        'and D (day). For example, these aliases would be valid '
-        '1M, 5M, 1D.'
-    ).strip()
-
-
-class InvalidHistoryFrequencyError(ZiplineError):
-    msg = (
-        'Frequency {frequency} not supported by the exchange.'
-    ).strip()
-
-
-class MismatchingFrequencyError(ZiplineError):
-    msg = (
-        'Bar aggregate frequency {frequency} not compatible with '
-        'data frequency {data_frequency}.'
-    ).strip()
-
-
-class InvalidSymbolError(ZiplineError):
-    msg = (
-        'Invalid trading pair symbol: {symbol}. '
-        'Catalyst symbols must follow this convention: '
-        '[Market Currency]_[Base Currency]. For example: eth_usd, btc_usd, '
-        'neo_eth, ubq_btc. Error details: {error}'
-    ).strip()
-
-
-class InvalidOrderStyle(ZiplineError):
-    msg = (
-        'Order style {style} not supported by exchange {exchange}.'
-    ).strip()
-
-
-class CreateOrderError(ZiplineError):
-    msg = (
-        'Unable to create order on exchange {exchange} {error}.'
-    ).strip()
-
-
-class OrderNotFound(ZiplineError):
-    msg = (
-        'Order {order_id} not found on exchange {exchange}.'
-    ).strip()
-
-
-class OrphanOrderError(ZiplineError):
-    msg = (
-        'Order {order_id} found in exchange {exchange} but not tracked by '
-        'the algorithm.'
-    ).strip()
-
-
-class OrphanOrderReverseError(ZiplineError):
-    msg = (
-        'Order {order_id} tracked by algorithm, but not found in exchange {exchange}.'
-    ).strip()
-
-
-class OrderCancelError(ZiplineError):
-    msg = (
-        'Unable to cancel order {order_id} on exchange {exchange} {error}.'
-    ).strip()
-
-
-class SidHashError(ZiplineError):
-    msg = (
-        'Unable to hash sid from symbol {symbol}.'
-    ).strip()
-
-
-class BaseCurrencyNotFoundError(ZiplineError):
-    msg = (
-        'Algorithm base currency {base_currency} not found in exchange '
-        '{exchange}.'
-    ).strip()
-
-
-class MismatchingBaseCurrencies(ZiplineError):
-    msg = (
-        'Unable to trade with base currency {base_currency} when the '
-        'algorithm uses {algo_currency}.'
-    ).strip()
-
-
-class MismatchingBaseCurrenciesExchanges(ZiplineError):
-    msg = (
-        'Unable to trade with base currency {base_currency} when the '
-        'exchange {exchange_name} users {exchange_currency}.'
-    ).strip()
-
-
-class SymbolNotFoundOnExchange(ZiplineError):
-    """
-    Raised when a symbol() call contains a non-existent symbol.
-    """
-    msg = ('Symbol {symbol} not found on exchange {exchange}. '
-           'Choose from: {supported_symbols}').strip()
-
-
-class BundleNotFoundError(ZiplineError):
-    msg = ('Unable to find bundle data for exchange {exchange} and '
-           'data frequency {data_frequency}.'
-           'Please ingest some price data.'
-           'See `catalyst ingest-exchange --help` for details.').strip()
-
-
-class TempBundleNotFoundError(ZiplineError):
-    msg = ('Temporary bundle not found in: {path}.').strip()
-
-
-class EmptyValuesInBundleError(ZiplineError):
-    msg = ('{name} with end minute {end_minute} has empty rows '
-           'in ranges: {dates}').strip()
-
-
-class PricingDataBeforeTradingError(ZiplineError):
-    msg = ('Pricing data for trading pairs {symbols} on exchange {exchange} '
-           'starts on {first_trading_day}, but you are either trying to trade or '
-           'retrieve pricing data on {dt}. Adjust your dates accordingly.').strip()
-
-
-class PricingDataNotLoadedError(ZiplineError):
-    msg = ('Missing data for {exchange} {symbols} in date range '
-           '[{start_dt} - {end_dt}]'
-           '\nPlease run: `catalyst ingest-exchange -x {exchange} -f '
-           '{data_frequency} -i {symbol_list}`. See catalyst documentation '
-           'for details.').strip()
-
-
-class ApiCandlesError(ZiplineError):
-    msg = ('Unable to fetch candles from the remote API: {error}.').strip()
-
-
-class NoDataAvailableOnExchange(ZiplineError):
-    msg = (
-        'Requested data for trading pair {symbol} is not available on exchange {exchange} '
-        'in `{data_frequency}` frequency at this time. '
-        'Check `http://enigma.co/catalyst/status` for market coverage.').strip()

catalyst/exchange/exchange_execution.py

@@ -1,67 +0,0 @@
-from catalyst.finance.execution import LimitOrder, StopOrder, StopLimitOrder
-
-
-class ExchangeLimitOrder(LimitOrder):
-    def get_limit_price(self, is_buy):
-        """
-        We may be trading Satoshis with 8 decimals, we cannot round numbers.
-
-        Parameters
-        ----------
-        is_buy: bool
-
-        Returns
-        -------
-        float
-
-        """
-        return self.limit_price
-
-
-class ExchangeStopOrder(StopOrder):
-    def get_stop_price(self, is_buy):
-        """
-        We may be trading Satoshis with 8 decimals, we cannot round numbers.
-
-        Parameters
-        ----------
-        is_buy: bool
-
-        Returns
-        -------
-        float
-
-        """
-        return self.stop_price
-
-
-class ExchangeStopLimitOrder(StopLimitOrder):
-    def get_limit_price(self, is_buy):
-        """
-        We may be trading Satoshis with 8 decimals, we cannot round numbers.
-
-        Parameters
-        ----------
-        is_buy: bool
-
-        Returns
-        -------
-        float
-
-        """
-        return self.limit_price
-
-    def get_stop_price(self, is_buy):
-        """
-        We may be trading Satoshis with 8 decimals, we cannot round numbers.
-
-        Parameters
-        ----------
-        is_buy: bool
-
-        Returns
-        -------
-        float
-
-        """
-        return self.stop_price

catalyst/exchange/exchange_portfolio.py

@@ -1,140 +0,0 @@
-import numpy as np
-from logbook import Logger
-
-from catalyst.constants import LOG_LEVEL
-from catalyst.protocol import Portfolio, Positions, Position
-from catalyst.utils.deprecate import deprecated
-
-log = Logger('ExchangePortfolio', level=LOG_LEVEL)
-
-
-class ExchangePortfolio(Portfolio):
-    """
-    Since the goal is to support multiple exchanges, it makes sense to
-    include additional stats in the portfolio object.
-
-    Instead of relying on the performance tracker, each exchange portfolio
-    tracks its own holding. This offers a separation between tracking an
-    exchange and the statistics of the algorithm.
-    """
-
-    def __init__(self, start_date, starting_cash=None):
-        self.capital_used = 0.0
-        self.starting_cash = starting_cash
-        self.portfolio_value = starting_cash
-        self.pnl = 0.0
-        self.returns = 0.0
-        self.cash = starting_cash
-        self.positions = Positions()
-        self.start_date = start_date
-        self.positions_value = 0.0
-        self.open_orders = dict()
-
-    def create_order(self, order):
-        """
-        Create an open order and store in memory.
-
-        Parameters
-        ----------
-        order: Order
-
-        """
-        log.debug('creating order {}'.format(order.id))
-        self.open_orders[order.id] = order
-
-        order_position = self.positions[order.asset] \
-            if order.asset in self.positions else None
-
-        if order_position is None:
-            order_position = Position(order.asset)
-            self.positions[order.asset] = order_position
-
-        order_position.amount += order.amount
-        log.debug('open order added to portfolio')
-
-    def execute_order(self, order, transaction):
-        """
-        Update the open orders and positions to apply an executed order.
-
-        Unlike with backtesting, we do not need to add slippage and fees.
-        The executed price includes transaction fees.
-
-        Parameters
-        ----------
-        order: Order
-        transaction: Transaction
-
-        """
-        log.debug('executing order {}'.format(order.id))
-        del self.open_orders[order.id]
-
-        order_position = self.positions[order.asset] \
-            if order.asset in self.positions else None
-
-        if order_position is None:
-            raise ValueError(
-                'Trying to execute order for a position not held: %s' % order.id
-            )
-
-        self.capital_used += order.amount * transaction.price
-
-        if order.amount > 0:
-            if order_position.cost_basis > 0:
-                order_position.cost_basis = np.average(
-                    [order_position.cost_basis, transaction.price],
-                    weights=[order_position.amount, order.amount]
-                )
-            else:
-                order_position.cost_basis = transaction.price
-
-        log.debug('updated portfolio with executed order')
-
-    @deprecated
-    def execute_transaction(self, transaction):
-        # TODO: almost duplicate of execute_order. Not sure why Poloniex needs this.
-        log.debug('executing transaction {}'.format(transaction.order_id))
-
-        order_position = self.positions[transaction.asset] \
-            if transaction.asset in self.positions else None
-
-        if order_position is None:
-            raise ValueError(
-                'Trying to execute transaction for a position not held: %s' % transaction.order_id
-            )
-
-        self.capital_used += transaction.amount * transaction.price
-
-        if transaction.amount > 0:
-            if order_position.cost_basis > 0:
-                order_position.cost_basis = np.average(
-                    [order_position.cost_basis, transaction.price],
-                    weights=[order_position.amount, transaction.amount]
-                )
-            else:
-                order_position.cost_basis = transaction.price
-
-        log.debug('updated portfolio with executed order')
-
-    def remove_order(self, order):
-        """
-        Removing an open order.
-
-        Parameters
-        ----------
-        order: Order
-
-        """
-        log.info('removing cancelled order {}'.format(order.id))
-        del self.open_orders[order.id]
-
-        order_position = self.positions[order.asset] \
-            if order.asset in self.positions else None
-
-        if order_position is None:
-            raise ValueError(
-                'Trying to remove order for a position not held: %s' % order.id
-            )
-
-        order_position.amount -= order.amount
-
-        log.debug('removed order from portfolio')

catalyst/exchange/exchange_utils.py

@@ -1,503 +0,0 @@
-import json
-import os
-import pickle
-import re
-import shutil
-from datetime import date, datetime
-
-import pandas as pd
-from catalyst.assets._assets import TradingPair
-from six.moves.urllib import request
-
-from catalyst.exchange.exchange_errors import ExchangeSymbolsNotFound, \
-    InvalidHistoryFrequencyError, InvalidHistoryFrequencyAlias
-from catalyst.utils.paths import data_root, ensure_directory, \
-    last_modified_time
-
-SYMBOLS_URL = 'https://s3.amazonaws.com/enigmaco/catalyst-exchanges/' \
-              '{exchange}/symbols.json'
-
-
-def get_exchange_folder(exchange_name, environ=None):
-    """
-    The root path of an exchange folder.
-
-    Parameters
-    ----------
-    exchange_name: str
-    environ:
-
-    Returns
-    -------
-    str
-
-    """
-    if not environ:
-        environ = os.environ
-
-    root = data_root(environ)
-    exchange_folder = os.path.join(root, 'exchanges', exchange_name)
-    ensure_directory(exchange_folder)
-
-    return exchange_folder
-
-
-def get_exchange_symbols_filename(exchange_name, environ=None):
-    """
-    The absolute path of the exchange's symbol.json file.
-
-    Parameters
-    ----------
-    exchange_name:
-    environ:
-
-    Returns
-    -------
-    str
-
-    """
-    exchange_folder = get_exchange_folder(exchange_name, environ)
-    return os.path.join(exchange_folder, 'symbols.json')
-
-
-def download_exchange_symbols(exchange_name, environ=None):
-    """
-    Downloads the exchange's symbols.json from the repository.
-
-    Parameters
-    ----------
-    exchange_name: str
-    environ:
-
-    Returns
-    -------
-    str
-
-    """
-    filename = get_exchange_symbols_filename(exchange_name)
-    url = SYMBOLS_URL.format(exchange=exchange_name)
-    response = request.urlretrieve(url=url, filename=filename)
-    return response
-
-
-def get_exchange_symbols(exchange_name, environ=None):
-    """
-    The de-serialized content of the exchange's symbols.json.
-
-    Parameters
-    ----------
-    exchange_name: str
-    environ:
-
-    Returns
-    -------
-    Object
-
-    """
-    filename = get_exchange_symbols_filename(exchange_name)
-
-    if not os.path.isfile(filename) or \
-                    pd.Timedelta(pd.Timestamp('now',
-                                              tz='UTC') - last_modified_time(
-                        filename)).days > 1:
-        download_exchange_symbols(exchange_name, environ)
-
-    if os.path.isfile(filename):
-        with open(filename) as data_file:
-            data = json.load(data_file)
-            return data
-    else:
-        raise ExchangeSymbolsNotFound(
-            exchange=exchange_name,
-            filename=filename
-        )
-
-
-def get_symbols_string(assets):
-    """
-    A concatenated string of symbols from a list of assets.
-
-    Parameters
-    ----------
-    assets: list[TradingPair]
-
-    Returns
-    -------
-    str
-
-    """
-    array = [assets] if isinstance(assets, TradingPair) else assets
-    return ', '.join([asset.symbol for asset in array])
-
-
-def get_exchange_auth(exchange_name, environ=None):
-    """
-    The de-serialized contend of the exchange's auth.json file.
-
-    Parameters
-    ----------
-    exchange_name: str
-    environ:
-
-    Returns
-    -------
-    Object
-
-    """
-    exchange_folder = get_exchange_folder(exchange_name, environ)
-    filename = os.path.join(exchange_folder, 'auth.json')
-
-    if os.path.isfile(filename):
-        with open(filename) as data_file:
-            data = json.load(data_file)
-            return data
-    else:
-        data = dict(name=exchange_name, key='', secret='')
-        with open(filename, 'w') as f:
-            json.dump(data, f, sort_keys=False, indent=2,
-                      separators=(',', ':'))
-            return data
-
-
-def delete_algo_folder(algo_name, environ=None):
-    """
-    Delete the folder containing the algo state.
-
-    Parameters
-    ----------
-    algo_name: str
-    environ:
-
-    Returns
-    -------
-    str
-
-    """
-    folder = get_algo_folder(algo_name, environ)
-    shutil.rmtree(folder)
-
-
-def get_algo_folder(algo_name, environ=None):
-    """
-    The algorithm root folder of the algorithm.
-
-    Parameters
-    ----------
-    algo_name: str
-    environ:
-
-    Returns
-    -------
-    str
-
-    """
-    if not environ:
-        environ = os.environ
-
-    root = data_root(environ)
-    algo_folder = os.path.join(root, 'live_algos', algo_name)
-    ensure_directory(algo_folder)
-
-    return algo_folder
-
-
-def get_algo_object(algo_name, key, environ=None, rel_path=None):
-    """
-    The de-serialized object of the algo name and key.
-
-    Parameters
-    ----------
-    algo_name: str
-    key: str
-    environ:
-    rel_path: str
-
-    Returns
-    -------
-    Object
-
-    """
-    if algo_name is None:
-        return None
-
-    folder = get_algo_folder(algo_name, environ)
-
-    if rel_path is not None:
-        folder = os.path.join(folder, rel_path)
-
-    filename = os.path.join(folder, key + '.p')
-
-    if os.path.isfile(filename):
-        try:
-            with open(filename, 'rb') as handle:
-                return pickle.load(handle)
-        except Exception as e:
-            return None
-    else:
-        return None
-
-
-def save_algo_object(algo_name, key, obj, environ=None, rel_path=None):
-    """
-    Serialize and save an object by algo name and key.
-
-    Parameters
-    ----------
-    algo_name: str
-    key: str
-    obj: Object
-    environ:
-    rel_path: str
-
-    """
-    folder = get_algo_folder(algo_name, environ)
-
-    if rel_path is not None:
-        folder = os.path.join(folder, rel_path)
-        ensure_directory(folder)
-
-    filename = os.path.join(folder, key + '.p')
-
-    with open(filename, 'wb') as handle:
-        pickle.dump(obj, handle, protocol=pickle.HIGHEST_PROTOCOL)
-
-
-def get_algo_df(algo_name, key, environ=None, rel_path=None):
-    """
-    The de-serialized DataFrame of an algo name and key.
-
-    Parameters
-    ----------
-    algo_name: str
-    key: str
-    environ:
-    rel_path: str
-
-    Returns
-    -------
-    DataFrame
-
-    """
-    folder = get_algo_folder(algo_name, environ)
-
-    if rel_path is not None:
-        folder = os.path.join(folder, rel_path)
-
-    filename = os.path.join(folder, key + '.csv')
-
-    if os.path.isfile(filename):
-        try:
-            with open(filename, 'rb') as handle:
-                return pd.read_csv(handle, index_col=0, parse_dates=True)
-        except IOError:
-            return pd.DataFrame()
-    else:
-        return pd.DataFrame()
-
-
-def save_algo_df(algo_name, key, df, environ=None, rel_path=None):
-    """
-    Serialize to csv and save a DataFrame by algo name and key.
-
-    Parameters
-    ----------
-    algo_name: str
-    key: str
-    df: pd.DataFrame
-    environ:
-    rel_path: str
-
-    """
-    folder = get_algo_folder(algo_name, environ)
-    if rel_path is not None:
-        folder = os.path.join(folder, rel_path)
-        ensure_directory(folder)
-
-    filename = os.path.join(folder, key + '.csv')
-
-    with open(filename, 'wt') as handle:
-        df.to_csv(handle, encoding='UTF_8')
-
-
-def get_exchange_minute_writer_root(exchange_name, environ=None):
-    """
-    The minute writer folder for the exchange.
-
-    Parameters
-    ----------
-    exchange_name: str
-    environ:
-
-    Returns
-    -------
-    BcolzExchangeBarWriter
-
-    """
-    exchange_folder = get_exchange_folder(exchange_name, environ)
-
-    minute_data_folder = os.path.join(exchange_folder, 'minute_data')
-    ensure_directory(minute_data_folder)
-
-    return minute_data_folder
-
-
-def get_exchange_bundles_folder(exchange_name, environ=None):
-    """
-    The temp folder for bundle downloads by algo name.
-
-    Parameters
-    ----------
-    exchange_name: str
-    environ:
-
-    Returns
-    -------
-    str
-
-    """
-    exchange_folder = get_exchange_folder(exchange_name, environ)
-
-    temp_bundles = os.path.join(exchange_folder, 'temp_bundles')
-    ensure_directory(temp_bundles)
-
-    return temp_bundles
-
-
-def perf_serial(obj):
-    """
-    JSON serializer for objects not serializable by default json code
-
-    Parameters
-    ----------
-    obj: Object
-
-    Returns
-    -------
-    str
-
-    """
-    if isinstance(obj, (datetime, date)):
-        return obj.isoformat()
-
-    raise TypeError("Type %s not serializable" % type(obj))
-
-
-def get_common_assets(exchanges):
-    """
-    The assets available in all specified exchanges.
-
-    Parameters
-    ----------
-    exchanges: list[Exchange]
-
-    Returns
-    -------
-    list[TradingPair]
-
-    """
-    symbols = []
-    for exchange_name in exchanges:
-        s = [asset.symbol for asset in exchanges[exchange_name].get_assets()]
-        symbols.append(s)
-
-    inter_symbols = set.intersection(*map(set, symbols))
-
-    assets = []
-    for symbol in inter_symbols:
-        for exchange_name in exchanges:
-            asset = exchanges[exchange_name].get_asset(symbol)
-            assets.append(asset)
-
-    return assets
-
-
-def get_frequency(freq, data_frequency):
-    """
-    Get the frequency parameters.
-
-    Notes
-    -----
-    We're trying to use Pandas convention for frequency aliases.
-
-    Parameters
-    ----------
-    freq: str
-    data_frequency: str
-
-    Returns
-    -------
-    str, int, str, str
-
-    """
-    if freq == 'minute':
-        unit = 'T'
-        candle_size = 1
-
-    elif freq == 'daily':
-        unit = 'D'
-        candle_size = 1
-
-    else:
-        freq_match = re.match(r'([0-9].*)?(m|M|d|D|h|H|T)', freq, re.M | re.I)
-        if freq_match:
-            candle_size = int(freq_match.group(1)) if freq_match.group(1) \
-                else 1
-            unit = freq_match.group(2)
-
-        else:
-            raise InvalidHistoryFrequencyError(frequency=freq)
-
-    if unit.lower() == 'd':
-        alias = '{}D'.format(candle_size)
-
-        if data_frequency == 'minute':
-            data_frequency = 'daily'
-
-    elif unit.lower() == 'm' or unit == 'T':
-        alias = '{}T'.format(candle_size)
-
-        if data_frequency == 'daily':
-            data_frequency = 'minute'
-
-    # elif unit.lower() == 'h':
-    #     candle_size = candle_size * 60
-    #
-    #     alias = '{}T'.format(candle_size)
-    #     if data_frequency == 'daily':
-    #         data_frequency = 'minute'
-
-    else:
-        raise InvalidHistoryFrequencyAlias(freq=freq)
-
-    return alias, candle_size, unit, data_frequency
-
-
-def resample_history_df(df, freq, field):
-    """
-    Resample the OHCLV DataFrame using the specified frequency.
-
-    Parameters
-    ----------
-    df: DataFrame
-    freq: str
-    field: str
-
-    Returns
-    -------
-    DataFrame
-
-    """
-    if field == 'open':
-        agg = 'first'
-    elif field == 'high':
-        agg = 'max'
-    elif field == 'low':
-        agg = 'min'
-    elif field == 'close':
-        agg = 'last'
-    elif field == 'volume':
-        agg = 'sum'
-    else:
-        raise ValueError('Invalid field.')
-
-    return df.resample(freq).agg(agg)

catalyst/exchange/factory.py

@@ -1,43 +0,0 @@
-from catalyst.exchange.bitfinex.bitfinex import Bitfinex
-from catalyst.exchange.bittrex.bittrex import Bittrex
-from catalyst.exchange.exchange_errors import ExchangeNotFoundError
-from catalyst.exchange.exchange_utils import get_exchange_auth
-from catalyst.exchange.poloniex.poloniex import Poloniex
-
-
-def get_exchange(exchange_name, base_currency=None):
-    exchange_auth = get_exchange_auth(exchange_name)
-    if exchange_name == 'bitfinex':
-        return Bitfinex(
-            key=exchange_auth['key'],
-            secret=exchange_auth['secret'],
-            base_currency=base_currency,
-            portfolio=None
-        )
-
-    elif exchange_name == 'bittrex':
-        return Bittrex(
-            key=exchange_auth['key'],
-            secret=exchange_auth['secret'],
-            base_currency=base_currency,
-            portfolio=None
-        )
-
-    elif exchange_name == 'poloniex':
-        return Poloniex(
-            key=exchange_auth['key'],
-            secret=exchange_auth['secret'],
-            base_currency=base_currency,
-            portfolio=None
-        )
-
-    else:
-        raise ExchangeNotFoundError(exchange_name=exchange_name)
-
-
-def get_exchanges(exchange_names):
-    exchanges = dict()
-    for exchange_name in exchange_names:
-        exchanges[exchange_name] = get_exchange(exchange_name)
-
-    return exchanges

catalyst/exchange/live_graph_clock.py

@@ -1,233 +0,0 @@
-import pandas as pd
-from catalyst.gens.sim_engine import (
-    BAR,
-    SESSION_START
-)
-from logbook import Logger
-
-from catalyst.constants import LOG_LEVEL
-from catalyst.exchange.exchange_errors import \
-    MismatchingBaseCurrenciesExchanges
-
-log = Logger('LiveGraphClock', level=LOG_LEVEL)
-
-
-class LiveGraphClock(object):
-    """Realtime clock for live trading.
-
-    This class is a drop-in replacement for
-    :class:`zipline.gens.sim_engine.MinuteSimulationClock`.
-
-    This mixes the clock with a live graph.
-
-    Notes
-    -----
-    This seemingly awkward approach allows us to run the program using a single
-    thread. This is important because Matplotlib does not play nice with
-    multi-threaded environments. Zipline probably does not either.
-
-
-    Matplotlib has a pause() method which is a wrapper around time.sleep()
-    used in the SimpleClock. The key difference is that users
-    can still interact with the chart during the pause cycles. This is
-    what enables us to keep a single thread. This is also why we are not using
-    the 'animate' callback of Matplotlib. We need to direct access to the
-    __iter__ method in order to yield events to Zipline.
-
-    The :param:`time_skew` parameter represents the time difference between
-    the exchange and the live trading machine's clock. It's not used currently.
-    """
-
-    def __init__(self, sessions, context, time_skew=pd.Timedelta('0s')):
-
-        global mdates, plt  # TODO: Could be cleaner
-        import matplotlib.dates as mdates
-        from matplotlib import pyplot as plt
-        from matplotlib import style
-
-        self.sessions = sessions
-        self.time_skew = time_skew
-        self._last_emit = None
-        self._before_trading_start_bar_yielded = True
-        self.context = context
-        self.fmt = mdates.DateFormatter('%Y-%m-%d %H:%M')
-
-        style.use('dark_background')
-
-        fig = plt.figure()
-        fig.canvas.set_window_title('Enigma Catalyst: {}'.format(
-            self.context.algo_namespace))
-
-        self.ax_pnl = fig.add_subplot(311)
-
-        self.ax_custom_signals = fig.add_subplot(312, sharex=self.ax_pnl)
-
-        self.ax_exposure = fig.add_subplot(313, sharex=self.ax_pnl)
-
-        if len(context.minute_stats) > 0:
-            self.draw_pnl()
-            self.draw_custom_signals()
-            self.draw_exposure()
-
-        # rotates and right aligns the x labels, and moves the bottom of the
-        # axes up to make room for them
-        fig.autofmt_xdate()
-        fig.subplots_adjust(hspace=0.5)
-
-        plt.tight_layout()
-        plt.ion()
-        plt.show()
-
-    def format_ax(self, ax):
-        """
-        Trying to assign reasonable parameters to the time axis.
-
-        Parameters
-        ----------
-        ax:
-
-        """
-        # TODO: room for improvement
-        ax.xaxis.set_major_locator(mdates.DayLocator(interval=1))
-        ax.xaxis.set_major_formatter(self.fmt)
-
-        locator = mdates.HourLocator(interval=4)
-        locator.MAXTICKS = 5000
-        ax.xaxis.set_minor_locator(locator)
-
-        datemin = pd.Timestamp.utcnow()
-        ax.set_xlim(datemin)
-
-        ax.grid(True)
-
-    def set_legend(self, ax):
-        """
-        Set legend on the chart.
-
-        Parameters
-        ----------
-        ax
-
-        """
-        ax.legend(loc='upper left', ncol=1, fontsize=10, numpoints=1)
-
-    def draw_pnl(self):
-        """
-        Draw p&l line on the chart.
-
-        """
-        ax = self.ax_pnl
-        df = self.context.pnl_stats
-
-        ax.clear()
-        ax.set_title('Performance')
-        ax.plot(df.index, df['performance'], '-',
-                color='green',
-                linewidth=1.0,
-                label='Performance'
-                )
-
-        def perc(val):
-            return '{:2f}'.format(val)
-
-        ax.format_ydata = perc
-
-        self.set_legend(ax)
-        self.format_ax(ax)
-
-    def draw_custom_signals(self):
-        """
-        Draw custom signals on the chart.
-
-        """
-        ax = self.ax_custom_signals
-        df = self.context.custom_signals_stats
-
-        colors = ['blue', 'green', 'red', 'black', 'orange', 'yellow', 'pink']
-
-        ax.clear()
-        ax.set_title('Custom Signals')
-        for index, column in enumerate(df.columns.values.tolist()):
-            ax.plot(df.index, df[column], '-',
-                    color=colors[index],
-                    linewidth=1.0,
-                    label=column
-                    )
-
-        self.set_legend(ax)
-        self.format_ax(ax)
-
-    def draw_exposure(self):
-        """
-        Draw exposure line on the chart.
-
-        """
-        ax = self.ax_exposure
-        context = self.context
-        df = context.exposure_stats
-
-        # TODO: list exchanges in graph
-        base_currency = None
-        positions = []
-        for exchange_name in context.exchanges:
-            exchange = context.exchanges[exchange_name]
-
-            if not base_currency:
-                base_currency = exchange.base_currency
-            elif base_currency != exchange.base_currency:
-                raise MismatchingBaseCurrenciesExchanges(
-                    base_currency=base_currency,
-                    exchange_name=exchange.name,
-                    exchange_currency=exchange.base_currency
-                )
-
-            positions += exchange.portfolio.positions
-
-        ax.clear()
-        ax.set_title('Exposure')
-        ax.plot(df.index, df['base_currency'], '-',
-                color='green',
-                linewidth=1.0,
-                label='Base Currency: {}'.format(base_currency.upper())
-                )
-
-        symbols = []
-        for position in positions:
-            symbols.append(position.symbol)
-
-        ax.plot(df.index, df['long_exposure'], '-',
-                color='blue',
-                linewidth=1.0,
-                label='Long Exposure: {}'.format(', '.join(symbols).upper()))
-
-        self.set_legend(ax)
-        self.format_ax(ax)
-
-    def __iter__(self):
-        yield pd.Timestamp.utcnow(), SESSION_START
-
-        while True:
-            current_time = pd.Timestamp.utcnow()
-            current_minute = current_time.floor('1 min')
-
-            if self._last_emit is None or current_minute > self._last_emit:
-                log.debug('emitting minutely bar: {}'.format(current_minute))
-
-                self._last_emit = current_minute
-                yield current_minute, BAR
-
-                try:
-                    self.draw_pnl()
-                    self.draw_custom_signals()
-                    self.draw_exposure()
-
-                    plt.draw()
-                except Exception as e:
-                    log.warn('Unable to update the graph: {}'.format(e))
-
-            else:
-                # I can't use the "animate" reactive approach here because
-                # I need to yield from the main loop.
-
-                # Workaround: https://stackoverflow.com/a/33050617/814633
-                plt.pause(1)

catalyst/exchange/poloniex/poloniex.py

@@ -1,650 +0,0 @@
-import json
-import json
-import time
-from collections import defaultdict
-
-import numpy as np
-import pandas as pd
-import pytz
-from catalyst.assets._assets import TradingPair
-from logbook import Logger
-# import six
-from six import iteritems
-
-from catalyst.constants import LOG_LEVEL
-# from websocket import create_connection
-from catalyst.exchange.exchange import Exchange
-from catalyst.exchange.exchange_bundle import ExchangeBundle
-from catalyst.exchange.exchange_errors import (
-    ExchangeRequestError,
-    InvalidHistoryFrequencyError,
-    InvalidOrderStyle, OrphanOrderReverseError)
-from catalyst.exchange.exchange_execution import ExchangeLimitOrder, \
-    ExchangeStopLimitOrder
-from catalyst.exchange.exchange_utils import get_exchange_symbols_filename, \
-    download_exchange_symbols, get_symbols_string
-from catalyst.exchange.poloniex.poloniex_api import Poloniex_api
-from catalyst.finance.order import Order, ORDER_STATUS
-from catalyst.finance.transaction import Transaction
-from catalyst.protocol import Account
-
-log = Logger('Poloniex', level=LOG_LEVEL)
-
-
-class Poloniex(Exchange):
-    def __init__(self, key, secret, base_currency, portfolio=None):
-        self.api = Poloniex_api(key=key, secret=secret)
-        self.name = 'poloniex'
-        self.assets = {}
-        self.load_assets()
-        self.base_currency = base_currency
-        self._portfolio = portfolio
-        self.minute_writer = None
-        self.minute_reader = None
-        self.transactions = defaultdict(list)
-
-        self.num_candles_limit = 2000
-        self.max_requests_per_minute = 60
-        self.request_cpt = dict()
-
-        self.bundle = ExchangeBundle(self)
-
-    def sanitize_curency_symbol(self, exchange_symbol):
-        """
-        Helper method used to build the universal pair.
-        Include any symbol mapping here if appropriate.
-
-        :param exchange_symbol:
-        :return universal_symbol:
-        """
-        return exchange_symbol.lower()
-
-    def _create_order(self, order_status):
-        """
-        Create a Catalyst order object from the Exchange order dictionary
-        :param order_status:
-        :return: Order
-        """
-        # if order_status['is_cancelled']:
-        #    status = ORDER_STATUS.CANCELLED
-        # elif not order_status['is_live']:
-        #    log.info('found executed order {}'.format(order_status))
-        #    status = ORDER_STATUS.FILLED
-        # else:
-        status = ORDER_STATUS.OPEN
-
-        amount = float(order_status['amount'])
-        # filled = float(order_status['executed_amount'])
-        filled = None
-
-        if order_status['type'] == 'sell':
-            amount = -amount
-            # filled = -filled
-
-        price = float(order_status['rate'])
-        order_type = order_status['type']
-
-        stop_price = None
-        limit_price = None
-
-        # TODO: is this comprehensive enough?
-        # if order_type.endswith('limit'):
-        #    limit_price = price
-        # elif order_type.endswith('stop'):
-        #    stop_price = price
-
-        # executed_price = float(order_status['avg_execution_price'])
-        executed_price = price
-
-        # TODO: bitfinex does not specify comission. I could calculate it but not sure if it's worth it.
-        commission = None
-
-        # date = pd.Timestamp.utcfromtimestamp(float(order_status['timestamp']))
-        # date = pytz.utc.localize(date)
-        date = None
-
-        order = Order(
-            dt=date,
-            asset=self.assets[order_status['symbol']],
-            # No such field in Poloniex
-            amount=amount,
-            stop=stop_price,
-            limit=limit_price,
-            filled=filled,
-            id=str(order_status['orderNumber']),
-            commission=commission
-        )
-        order.status = status
-
-        return order, executed_price
-
-    def get_balances(self):
-        balances = self.api.returnbalances()
-        try:
-            log.debug('retrieving wallets balances')
-        except Exception as e:
-            log.debug(e)
-            raise ExchangeRequestError(error=e)
-
-        if 'error' in balances:
-            raise ExchangeRequestError(
-                error='unable to fetch balance {}'.format(balances['error'])
-            )
-
-        std_balances = dict()
-        for (key, value) in iteritems(balances):
-            currency = key.lower()
-            std_balances[currency] = float(value)
-
-        return std_balances
-
-    @property
-    def account(self):
-        account = Account()
-
-        account.settled_cash = None
-        account.accrued_interest = None
-        account.buying_power = None
-        account.equity_with_loan = None
-        account.total_positions_value = None
-        account.total_positions_exposure = None
-        account.regt_equity = None
-        account.regt_margin = None
-        account.initial_margin_requirement = None
-        account.maintenance_margin_requirement = None
-        account.available_funds = None
-        account.excess_liquidity = None
-        account.cushion = None
-        account.day_trades_remaining = None
-        account.leverage = None
-        account.net_leverage = None
-        account.net_liquidation = None
-
-        return account
-
-    @property
-    def time_skew(self):
-        # TODO: research the time skew conditions
-        return pd.Timedelta('0s')
-
-    def get_account(self):
-        # TODO: fetch account data and keep in cache
-        return None
-
-    def get_candles(self, freq, assets, bar_count=None,
-                    start_dt=None, end_dt=None):
-        """
-        Retrieve OHLVC candles from Poloniex
-
-        :param freq:
-        :param assets:
-        :param bar_count:
-        :return:
-
-        Available Frequencies
-        ---------------------
-        '5m', '15m', '30m', '2h', '4h', '1D'
-        """
-
-        if end_dt is None:
-            end_dt = pd.Timestamp.utcnow()
-
-        log.debug(
-            'retrieving {bars} {freq} candles on {exchange} from '
-            '{end_dt} for markets {symbols}, '.format(
-                bars=bar_count,
-                freq=freq,
-                exchange=self.name,
-                end_dt=end_dt,
-                symbols=get_symbols_string(assets)
-            )
-        )
-
-        if freq == '1T' and (bar_count == 1 or bar_count is None):
-            # TODO: use the order book instead
-            # We use the 5m to fetch the last bar
-            frequency = 300
-        elif freq == '5T':
-            frequency = 300
-        elif freq == '15T':
-            frequency = 900
-        elif freq == '30T':
-            frequency = 1800
-        elif freq == '120T':
-            frequency = 7200
-        elif freq == '240T':
-            frequency = 14400
-        elif freq == '1D':
-            frequency = 86400
-        else:
-            # Poloniex does not offer 1m data candles
-            # It is likely to error out there frequently
-            raise InvalidHistoryFrequencyError(frequency=freq)
-
-        # Making sure that assets are iterable
-        asset_list = [assets] if isinstance(assets, TradingPair) else assets
-        ohlc_map = dict()
-
-        for asset in asset_list:
-            delta = end_dt - pd.to_datetime('1970-1-1', utc=True)
-            end = int(delta.total_seconds())
-
-            if bar_count is None:
-                start = end - 2 * frequency
-            else:
-                start = end - bar_count * frequency
-
-            try:
-                response = self.api.returnchartdata(
-                    self.get_symbol(asset), frequency, start, end
-                )
-            except Exception as e:
-                raise ExchangeRequestError(error=e)
-
-            if 'error' in response:
-                raise ExchangeRequestError(
-                    error='Unable to retrieve candles: {}'.format(
-                        response.content)
-                )
-
-            def ohlc_from_candle(candle):
-                last_traded = pd.Timestamp.utcfromtimestamp(candle['date'])
-                last_traded = last_traded.replace(tzinfo=pytz.UTC)
-
-                ohlc = dict(
-                    open=np.float64(candle['open']),
-                    high=np.float64(candle['high']),
-                    low=np.float64(candle['low']),
-                    close=np.float64(candle['close']),
-                    volume=np.float64(candle['volume']),
-                    price=np.float64(candle['close']),
-                    last_traded=last_traded
-                )
-
-                return ohlc
-
-            if bar_count is None:
-                ohlc_map[asset] = ohlc_from_candle(response[0])
-            else:
-                ohlc_bars = []
-                for candle in response:
-                    ohlc = ohlc_from_candle(candle)
-                    ohlc_bars.append(ohlc)
-                ohlc_map[asset] = ohlc_bars
-
-        return ohlc_map[assets] \
-            if isinstance(assets, TradingPair) else ohlc_map
-
-    def create_order(self, asset, amount, is_buy, style):
-        """
-        Creating order on the exchange.
-
-        :param asset:
-        :param amount:
-        :param is_buy:
-        :param style:
-        :return:
-        """
-        exchange_symbol = self.get_symbol(asset)
-
-        if isinstance(style, ExchangeLimitOrder) or isinstance(style,
-                                                               ExchangeStopLimitOrder):
-            if isinstance(style, ExchangeStopLimitOrder):
-                log.warn('{} will ignore the stop price'.format(self.name))
-
-            price = style.get_limit_price(is_buy)
-
-            try:
-                if (is_buy):
-                    response = self.api.buy(exchange_symbol, amount, price)
-                else:
-                    response = self.api.sell(exchange_symbol, -amount, price)
-            except Exception as e:
-                raise ExchangeRequestError(error=e)
-
-            date = pd.Timestamp.utcnow()
-
-            if ('orderNumber' in response):
-                order_id = str(response['orderNumber'])
-                order = Order(
-                    dt=date,
-                    asset=asset,
-                    amount=amount,
-                    stop=style.get_stop_price(is_buy),
-                    limit=style.get_limit_price(is_buy),
-                    id=order_id
-                )
-                return order
-            else:
-                log.warn(
-                    '{} order failed: {}'.format('buy' if is_buy else 'sell',
-                                                 response['error']))
-                return None
-        else:
-            raise InvalidOrderStyle(exchange=self.name,
-                                    style=style.__class__.__name__)
-
-    def get_open_orders(self, asset='all'):
-        """Retrieve all of the current open orders.
-
-        Parameters
-        ----------
-        asset : Asset
-            If passed and not 'all', return only the open orders for the given
-            asset instead of all open orders.
-
-        Returns
-        -------
-        open_orders : dict[list[Order]] or list[Order]
-            If 'all' is passed this will return a dict mapping Assets
-            to a list containing all the open orders for the asset.
-            If an asset is passed then this will return a list of the open
-            orders for this asset.
-        """
-
-        return self.portfolio.open_orders
-
-        """
-            TODO: Why going to the exchange if we already have this info locally?
-                  And why creating all these Orders if we later discard them?
-        """
-
-        try:
-            if (asset == 'all'):
-                response = self.api.returnopenorders('all')
-            else:
-                response = self.api.returnopenorders(self.get_symbol(asset))
-        except Exception as e:
-            raise ExchangeRequestError(error=e)
-
-        if 'error' in response:
-            raise ExchangeRequestError(
-                error='Unable to retrieve open orders: {}'.format(
-                    order_statuses['message'])
-            )
-
-        print(self.portfolio.open_orders)
-
-        # TODO: Need to handle openOrders for 'all'
-        orders = list()
-        for order_status in response:
-            order, executed_price = self._create_order(
-                order_status)  # will Throw error b/c Polo doesn't track order['symbol']
-            if asset is None or asset == order.sid:
-                orders.append(order)
-
-        return orders
-
-    def get_order(self, order_id):
-        """Lookup an order based on the order id returned from one of the
-        order functions.
-
-        Parameters
-        ----------
-        order_id : str
-            The unique identifier for the order.
-
-        Returns
-        -------
-        order : Order
-            The order object.
-        """
-
-        try:
-            order = self._portfolio.open_orders[order_id]
-        except Exception as e:
-            raise OrphanOrderError(order_id=order_id, exchange=self.name)
-
-        return order
-
-        # TODO: Need to decide whether we fetch orders locally or from exchnage
-        # The code below is ignored
-
-        try:
-            response = self.api.returnopenorders(self.get_symbol(order.sid))
-        except Exception as e:
-            raise ExchangeRequestError(error=e)
-
-        for o in response:
-            if (int(o['orderNumber']) == int(order_id)):
-                return order
-
-        return None
-
-    def cancel_order(self, order_param):
-        """Cancel an open order.
-
-        Parameters
-        ----------
-        order_param : str or Order
-            The order_id or order object to cancel.
-        """
-
-        if (isinstance(order_param, Order)):
-            order = order_param
-        else:
-            order = self._portfolio.open_orders[order_param]
-
-        try:
-            response = self.api.cancelorder(order.id)
-        except Exception as e:
-            raise ExchangeRequestError(error=e)
-
-        if 'error' in response:
-            log.info(
-                'Unable to cancel order {order_id} on exchange {exchange} {error}.'.format(
-                    order_id=order.id,
-                    exchange=self.name,
-                    error=response['error']
-                ))
-
-            # raise OrderCancelError(
-            #    order_id=order.id,
-            #    exchange=self.name,
-            #    error=response['error']
-            # )
-
-        self.portfolio.remove_order(order)
-
-    def tickers(self, assets):
-        """
-        Fetch ticket data for assets
-        https://docs.bitfinex.com/v2/reference#rest-public-tickers
-
-        :param assets:
-        :return:
-        """
-        symbols = self.get_symbols(assets)
-
-        log.debug('fetching tickers {}'.format(symbols))
-
-        try:
-            response = self.api.returnticker()
-        except Exception as e:
-            raise ExchangeRequestError(error=e)
-
-        if 'error' in response:
-            raise ExchangeRequestError(
-                error='Unable to retrieve tickers: {}'.format(
-                    response['error'])
-            )
-
-        ticks = dict()
-
-        for index, symbol in enumerate(symbols):
-            ticks[assets[index]] = dict(
-                timestamp=pd.Timestamp.utcnow(),
-                bid=float(response[symbol]['highestBid']),
-                ask=float(response[symbol]['lowestAsk']),
-                last_price=float(response[symbol]['last']),
-                low=float(response[symbol]['lowestAsk']),
-                # TODO: Polo does not provide low
-                high=float(response[symbol]['highestBid']),
-                # TODO: Polo does not provide high
-                volume=float(response[symbol]['baseVolume']),
-            )
-
-        log.debug('got tickers {}'.format(ticks))
-        return ticks
-
-    def generate_symbols_json(self, filename=None, source_dates=False):
-        symbol_map = {}
-
-        if not source_dates:
-            fn, r = download_exchange_symbols(self.name)
-            with open(fn) as data_file:
-                cached_symbols = json.load(data_file)
-
-        response = self.api.returnticker()
-
-        for exchange_symbol in response:
-            base, market = self.sanitize_curency_symbol(exchange_symbol).split(
-                '_')
-            symbol = '{market}_{base}'.format(market=market, base=base)
-
-            if (source_dates):
-                start_date = self.get_symbol_start_date(exchange_symbol)
-            else:
-                try:
-                    start_date = cached_symbols[exchange_symbol]['start_date']
-                except KeyError as e:
-                    start_date = time.strftime('%Y-%m-%d')
-
-            try:
-                end_daily = cached_symbols[exchange_symbol]['end_daily']
-            except KeyError as e:
-                end_daily = 'N/A'
-
-            try:
-                end_minute = cached_symbols[exchange_symbol]['end_minute']
-            except KeyError as e:
-                end_minute = 'N/A'
-
-            symbol_map[exchange_symbol] = dict(
-                symbol=symbol,
-                start_date=start_date,
-                end_daily=end_daily,
-                end_minute=end_minute,
-            )
-
-        if (filename is None):
-            filename = get_exchange_symbols_filename(self.name)
-
-        with open(filename, 'w') as f:
-            json.dump(symbol_map, f, sort_keys=True, indent=2,
-                      separators=(',', ':'))
-
-    def get_symbol_start_date(self, symbol):
-        try:
-            r = self.api.returnchartdata(symbol, 86400, pd.to_datetime(
-                '2010-1-1').value // 10 ** 9)
-        except Exception as e:
-            raise ExchangeRequestError(error=e)
-
-        return time.strftime('%Y-%m-%d', time.gmtime(int(r[0]['date'])))
-
-    def check_open_orders(self):
-        """
-        Need to override this function for Poloniex:
-
-        Loop through the list of open orders in the Portfolio object.
-        Check if any transactions have been executed:
-            If so, create a transaction and apply to the Portfolio.
-        Check if the order is still open:
-            If not, remove it from open orders
-
-        :return:
-        transactions: Transaction[]
-        """
-        transactions = list()
-        if self.portfolio.open_orders:
-            for order_id in list(self.portfolio.open_orders):
-
-                order = self._portfolio.open_orders[order_id]
-                log.debug('found open order: {}'.format(order_id))
-
-                try:
-                    order_open = self.get_order(order_id)
-                except Exception as e:
-                    raise ExchangeRequestError(error=e)
-
-                if (order_open):
-                    delta = pd.Timestamp.utcnow() - order.dt
-                    log.info(
-                        'order {order_id} still open after {delta}'.format(
-                            order_id=order_id,
-                            delta=delta)
-                    )
-
-                try:
-                    response = self.api.returnordertrades(order_id)
-                except Exception as e:
-                    raise ExchangeRequestError(error=e)
-
-                if ('error' in response):
-                    if (not order_open):
-                        raise OrphanOrderReverseError(order_id=order_id,
-                                                      exchange=self.name)
-                else:
-                    for tx in response:
-                        """
-                            We maintain a list of dictionaries of transactions that correspond to
-                            partially filled orders, indexed by order_id. Every time we query 
-                            executed transactions from the exchange, we check if we had that 
-                            transaction for that order already. If not, we process it.
-
-                            When an order if fully filled, we flush the dict of transactions  
-                            associated with that order.
-                        """
-                        if (not filter(
-                                lambda item: item['order_id'] == tx['tradeID'],
-                                self.transactions[order_id])):
-                            log.debug(
-                                'Got new transaction for order {}: amount {}, price {}'.format(
-                                    order_id, tx['amount'], tx['rate']))
-                            tx['amount'] = float(tx['amount'])
-                            if (tx['type'] == 'sell'):
-                                tx['amount'] = -tx['amount']
-                            transaction = Transaction(
-                                asset=order.asset,
-                                amount=tx['amount'],
-                                dt=pd.to_datetime(tx['date'], utc=True),
-                                price=float(tx['rate']),
-                                order_id=tx['tradeID'],
-                                # it's a misnomer, but keeping it for compatibility
-                                commission=float(tx['fee'])
-                            )
-                            self.transactions[order_id].append(transaction)
-                            self.portfolio.execute_transaction(transaction)
-                            transactions.append(transaction)
-
-                    if (not order_open):
-                        """
-                            Since transactions have been executed individually
-                            the only thing left to do is remove them from list of open_orders
-                        """
-                        del self.portfolio.open_orders[order_id]
-                        del self.transactions[order_id]
-
-        return transactions
-
-    def get_orderbook(self, asset, order_type='all'):
-        exchange_symbol = asset.exchange_symbol
-        data = self.api.returnOrderBook(market=exchange_symbol)
-
-        result = dict()
-        for order_type in data:
-            # TODO: filter by type
-            if order_type != 'asks' and order_type != 'bids':
-                continue
-
-            result[order_type] = []
-            for entry in data[order_type]:
-                if len(entry) == 2:
-                    result[order_type].append(
-                        dict(
-                            rate=float(entry[0]),
-                            quantity=float(entry[1])
-                        )
-                    )
-        return result

catalyst/exchange/poloniex/poloniex_api.py

@@ -1,215 +0,0 @@
-#!/usr/bin/env python
-import json
-import time
-import hmac
-import hashlib
-import ssl
-
-from six.moves import urllib
-
-# Workaround for backwards compatibility
-# https://stackoverflow.com/questions/3745771/urllib-request-in-python-2-7
-urlopen = urllib.request.urlopen
-
-
-class Poloniex_api(object):
-    def __init__(self, key, secret):
-        self.key = key
-        self.secret = secret
-
-        self.max_requests_per_second = 6
-        self.request_cpt = dict()
-
-        self.public = ['returnTicker', 'return24Volume', 'returnOrderBook',
-                       'returnTradeHistory', 'returnChartData',
-                       'returnCurrencies', 'returnLoanOrders']
-        self.trading = ['returnBalances', 'returnCompleteBalances',
-                        'returnDepositAddresses',
-                        'generateNewAddress', 'returnDepositsWithdrawals',
-                        'returnOpenOrders',
-                        'returnTradeHistory', 'returnOrderTrades',
-                        'buy', 'sell', 'cancelOrder', 'moveOrder',
-                        'withdraw', 'returnFeeInfo',
-                        'returnAvailableAccountBalances',
-                        'returnTradableBalances', 'transferBalance',
-                        'returnMarginAccountSummary', 'marginBuy',
-                        'marginSell',
-                        'getMarginPosition', 'closeMarginPosition',
-                        'createLoanOffer',
-                        'cancelLoanOffer', 'returnOpenLoanOffers',
-                        'returnActiveLoans',
-                        'returnLendingHistory', 'toggleAutoRenew']
-
-    def ask_request(self):
-        """
-        Asks permission to issue a request to the exchange.
-        The primary purpose is to avoid hitting rate limits.
-
-        The application will pause if the maximum requests per minute
-        permitted by the exchange is exceeded.
-
-        :return boolean:
-
-        """
-        now = time.time()
-        if not self.request_cpt:
-            self.request_cpt = dict()
-            self.request_cpt[now] = 0
-            return True
-
-        cpt_date = list(self.request_cpt.keys())[0]
-        cpt = self.request_cpt[cpt_date]
-
-        if now > cpt_date + 1:
-            self.request_cpt = dict()
-            self.request_cpt[now] = 0
-            return True
-
-        if cpt >= self.max_requests_per_second:
-
-            time.sleep(1)
-
-            now = time.time()
-            self.request_cpt = dict()
-            self.request_cpt[now] = 0
-            return True
-        else:
-            self.request_cpt[cpt_date] += 1
-
-    def query(self, method, req={}):
-
-        if method in self.public:
-            url = 'https://poloniex.com/public?command=' + method + '&' + \
-                  urllib.parse.urlencode(req)
-            headers = {}
-            post_data = None
-        elif method in self.trading:
-            url = 'https://poloniex.com/tradingApi'
-            req['command'] = method
-            req['nonce'] = int(time.time() * 1000)
-            post_data = urllib.parse.urlencode(req)
-
-            signature = hmac.new(self.secret.encode('utf-8'),
-                                 post_data.encode('utf-8'),
-                                 hashlib.sha512).hexdigest()
-            headers = {'Sign': signature, 'Key': self.key}
-
-            post_data = post_data.encode('utf-8')
-        else:
-            raise ValueError(
-                'Method "' + method + '" not found in neither the Public API '
-                                      'or Trading API endpoints'
-            )
-
-        self.ask_request()
-        req = urllib.request.Request(
-            url,
-            data=post_data,
-            headers=headers,
-        )
-        return json.loads(
-            urlopen(req, context=ssl._create_unverified_context()).read())
-
-    def returnticker(self):
-        return self.query('returnTicker', {})
-
-    def return24volume(self):
-        return self.query('return24Volume', {})
-
-    def returnOrderBook(self, market='all'):
-        return self.query('returnOrderBook', {'currencyPair': market})
-
-    def returntradehistory(self, market, start=None, end=None):
-        if (start is not None and end is not None):
-            return self.query('returntradehistory',
-                              {'currencyPair': market, 'start': start,
-                               'end': end})
-        else:
-            return self.query('returntradehistory', {'currencyPair': market})
-
-    def returnchartdata(self, market, period, start, end=9999999999):
-        return self.query('returnChartData',
-                          {'currencyPair': market, 'period': period,
-                           'start': start, 'end': end})
-
-    def returncurrencies(self):
-        return self.query('returnCurrencies', {})
-
-    def returnloadorders(self, market):
-        return self.query('returnLoanOrders', {'currency': market})
-
-    def returnbalances(self):
-        return self.query('returnBalances')
-
-    def returncompletebalances(self, account):
-        if (account):
-            return self.query('returnCompleteBalances', {'account': account})
-        else:
-            return self.query('returnCompleteBalances')
-
-    def returndepositaddresses(self):
-        return self.query('returnDepositAddresses')
-
-    def generatenewaddress(self, currency):
-        return self.query('generateNewAddress', {'currency': currency})
-
-    def returnDepositsWithdrawals(self, start, end):
-        return self.query('returnDepositsWithdrawals',
-                          {'start': start, 'end': end})
-
-    def returnopenorders(self, market):
-        return self.query('returnOpenOrders', {'currencyPair': market})
-
-    def returntradehistory(self, market):
-        # TODO: optional start and/or end and limit
-        return self.query('returnTradeHistory', {'currencyPair': market})
-
-    def returnordertrades(self, ordernumber):
-        return self.query('returnOrderTrades', {'orderNumber': ordernumber})
-
-    def buy(self, market, amount, rate, fillorkill=0, immediateorcancel=0,
-            postonly=0):
-        if (fillorkill):
-            return self.query('buy', {'currencyPair': market, 'rate': rate,
-                                      'amount': amount,
-                                      'fillOrKill': fillorkill, })
-        elif (immediateorcancel):
-            return self.query('buy', {'currencyPair': market, 'rate': rate,
-                                      'amount': amount,
-                                      'immediateOrCancel': immediateorcancel, })
-        elif (postonly):
-            return self.query('buy', {'currencyPair': market, 'rate': rate,
-                                      'amount': amount,
-                                      'postOnly': postonly, })
-        else:
-            return self.query('buy', {'currencyPair': market, 'rate': rate,
-                                      'amount': amount, })
-
-    def sell(self, market, amount, rate, fillorkill=0, immediateorcancel=0,
-             postonly=0):
-        if (fillorkill):
-            return self.query('sell', {'currencyPair': market, 'rate': rate,
-                                       'amount': amount,
-                                       'fillOrKill': fillorkill, })
-        elif (immediateorcancel):
-            return self.query('sell', {'currencyPair': market, 'rate': rate,
-                                       'amount': amount,
-                                       'immediateOrCancel': immediateorcancel, })
-        elif (postonly):
-            return self.query('sell', {'currencyPair': market, 'rate': rate,
-                                       'amount': amount,
-                                       'postOnly': postonly, })
-        else:
-            return self.query('sell', {'currencyPair': market, 'rate': rate,
-                                       'amount': amount, })
-
-    def cancelorder(self, ordernumber):
-        return self.query('cancelOrder', {'orderNumber': ordernumber})
-
-    def withdraw(self, currency, quantity, address):
-        return self.query('withdraw',
-                          {'currency': currency, 'amount': quantity,
-                           'address': address})
-
-    def returnfeeinfo(self):
-        return self.query('returnFeeInfo')

catalyst/exchange/simple_clock.py

@@ -1,60 +0,0 @@
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from time import sleep
-
-import pandas as pd
-from catalyst.gens.sim_engine import (
-    BAR,
-    SESSION_START
-)
-from logbook import Logger
-
-from catalyst.constants import LOG_LEVEL
-
-log = Logger('ExchangeClock', level=LOG_LEVEL)
-
-
-class SimpleClock(object):
-    """Realtime clock for live trading.
-
-    This class is a drop-in replacement for
-    :class:`zipline.gens.sim_engine.MinuteSimulationClock`.
-
-    This is a stripped down version because crypto exchanges run around the clock.
-
-    The :param:`time_skew` parameter represents the time difference between
-    the Broker and the live trading machine's clock.
-    """
-
-    def __init__(self, sessions, time_skew=pd.Timedelta("0s")):
-
-        self.sessions = sessions
-        self.time_skew = time_skew
-        self._last_emit = None
-        self._before_trading_start_bar_yielded = True
-
-    def __iter__(self):
-        yield pd.Timestamp.utcnow(), SESSION_START
-
-        while True:
-            current_time = pd.Timestamp.utcnow()
-            current_minute = current_time.floor('1 min')
-
-            if self._last_emit is None or current_minute > self._last_emit:
-                log.debug('emitting minutely bar: {}'.format(current_minute))
-
-                self._last_emit = current_minute
-                yield current_minute, BAR
-            else:
-                sleep(1)

catalyst/exchange/stats_utils.py

@@ -1,210 +0,0 @@
-import numbers
-
-import numpy as np
-import pandas as pd
-
-
-def trend_direction(series):
-    if series[-1] is np.nan or series[-1] is np.nan:
-        return None
-
-    if series[-1] > series[-2]:
-        return 'up'
-    else:
-        return 'down'
-
-
-def crossover(source, target):
-    """
-    The `x`-series is defined as having crossed over `y`-series if the value
-    of `x` is greater than the value of `y` and the value of `x` was less than
-    the value of `y` on the bar immediately preceding the current bar.
-
-    Parameters
-    ----------
-    source: Series
-    target: Series
-
-    Returns
-    -------
-    bool
-
-    """
-    if source[-1] is np.nan or source[-2] is np.nan \
-            or target[-1] is np.nan or target[-2] is np.nan:
-        return False
-
-    if source[-1] > target[-1] and source[-2] < target[-2]:
-        return True
-    else:
-        return False
-
-
-def crossunder(source, target):
-    """
-    The `x`-series is defined as having crossed under `y`-series if the value
-    of `x` is less than the value of `y` and the value of `x` was greater than
-    the value of `y` on the bar immediately preceding the current bar.
-
-    Parameters
-    ----------
-    source: Series
-    target: Series
-
-    Returns
-    -------
-    bool
-
-    """
-    if isinstance(target, numbers.Number):
-        if source[-1] is np.nan or source[-2] is np.nan \
-                or target is np.nan:
-            return False
-
-        if source[-1] < target <= source[-2]:
-            return True
-        else:
-            return False
-    else:
-        if source[-1] is np.nan or source[-2] is np.nan \
-                or target[-1] is np.nan or target[-2] is np.nan:
-            return False
-
-        if source[-1] < target[-1] and source[-2] >= target[-2]:
-            return True
-        else:
-            return False
-
-
-def vwap(df):
-    """
-    Volume-weighted average price (VWAP) is a ratio generally used by
-    institutional investors and mutual funds to make buys and sells so as not
-    to disturb the market prices with large orders. It is the average share
-    price of a stock weighted against its trading volume within a particular
-    time frame, generally one day.
-
-    Read more: Volume Weighted Average Price - VWAP
-    https://www.investopedia.com/terms/v/vwap.asp#ixzz4xt922daE
-
-    Parameters
-    ----------
-    df: pd.DataFrame
-
-    Returns
-    -------
-
-    """
-    if 'close' not in df.columns or 'volume' not in df.columns:
-        raise ValueError('price data must include `volume` and `close`')
-
-    vol_sum = np.nansum(df['volume'].values)
-
-    try:
-        ret = np.nansum(df['close'].values * df['volume'].values) / vol_sum
-    except ZeroDivisionError:
-        ret = np.nan
-
-    return ret
-
-
-def get_pretty_stats(stats_df, recorded_cols=None, num_rows=10):
-    """
-    Format and print the last few rows of a statistics DataFrame.
-    See the pyfolio project for the data structure.
-
-    Parameters
-    ----------
-    stats_df: DataFrame
-    num_rows: int
-
-    Returns
-    -------
-    str
-
-    """
-    stats_df.set_index('period_close', drop=True, inplace=True)
-    stats_df.dropna(axis=1, how='all', inplace=True)
-
-    pd.set_option('display.expand_frame_repr', False)
-    pd.set_option('precision', 3)
-    pd.set_option('display.width', 1000)
-    pd.set_option('display.max_colwidth', 1000)
-
-    columns = ['starting_cash', 'ending_cash', 'portfolio_value',
-               'pnl', 'long_exposure', 'short_exposure', 'orders',
-               'transactions', 'positions']
-
-    if recorded_cols is not None:
-        for column in recorded_cols:
-            columns.append(column)
-
-    def format_positions(positions):
-        parts = []
-        for position in positions:
-            msg = '{amount:.2f}{market} cost basis {cost_basis:.4f}{base}'.format(
-                amount=position['amount'],
-                market=position['sid'].market_currency,
-                cost_basis=position['cost_basis'],
-                base=position['sid'].base_currency
-            )
-            parts.append(msg)
-        return ', '.join(parts)
-
-    formatters = {
-        'orders': lambda orders: len(orders),
-        'transactions': lambda transactions: len(transactions),
-        'returns': lambda returns: "{0:.4f}".format(returns),
-        'positions': format_positions
-    }
-
-    return stats_df.tail(num_rows).to_string(
-        columns=columns,
-        formatters=formatters
-    )
-
-
-def df_to_string(df):
-    """
-    Create a formatted str representation of the DataFrame.
-
-    Parameters
-    ----------
-    df: DataFrame
-
-    Returns
-    -------
-    str
-
-    """
-    pd.set_option('display.expand_frame_repr', False)
-    pd.set_option('precision', 8)
-    pd.set_option('display.width', 1000)
-    pd.set_option('display.max_colwidth', 1000)
-
-    return df.to_string()
-
-
-def extract_transactions(perf):
-    """
-    Compute indexes for buy and sell transactions
-
-    Parameters
-    ----------
-    perf: DataFrame
-        The algo performance DataFrame.
-
-    Returns
-    -------
-    DataFrame
-        A DataFrame of transactions.
-
-    """
-    trans_list = perf.transactions.values
-    all_trans = [t for sublist in trans_list for t in sublist]
-    all_trans.sort(key=lambda t: t['dt'])
-
-    transactions = pd.DataFrame(all_trans)
-    if not transactions.empty:
-        transactions.set_index('dt', inplace=True, drop=True)
-    return transactions

catalyst/finance/blotter.py

@@ -34,9 +34,7 @@ from catalyst.finance.commission import (
 from catalyst.finance.cancel_policy import NeverCancel
 from catalyst.utils.input_validation import expect_types
 
-from catalyst.constants import LOG_LEVEL
+log = Logger('Blotter')
-
-log = Logger('Blotter', level=LOG_LEVEL)
 warning_logger = Logger('AlgoWarning')
 
 

catalyst/finance/controls.py

@@ -24,9 +24,7 @@ from catalyst.errors import (
     TradingControlViolation,
 )
 
-from catalyst.constants import LOG_LEVEL
+log = logbook.Logger('TradingControl')
-
-log = logbook.Logger('TradingControl', level=LOG_LEVEL)
 
 
 class TradingControl(with_metaclass(abc.ABCMeta)):

catalyst/finance/execution.py

@@ -77,7 +77,6 @@ class LimitOrder(ExecutionStyle):
     Execution style representing an order to be executed at a price equal to or
     better than a specified limit price.
     """
-
     def __init__(self, limit_price, exchange=None):
         """
         Store the given price.
@@ -100,7 +99,6 @@ class StopOrder(ExecutionStyle):
     Execution style representing an order to be placed once the market price
     reaches a specified stop price.
     """
-
     def __init__(self, stop_price, exchange=None):
         """
         Store the given price.
@@ -123,7 +121,6 @@ class StopLimitOrder(ExecutionStyle):
     Execution style representing a limit order to be placed with a specified
     limit price once the market reaches a specified stop price.
     """
-
     def __init__(self, limit_price, stop_price, exchange=None):
         """
         Store the given prices
@@ -147,20 +144,31 @@ class StopLimitOrder(ExecutionStyle):
 def asymmetric_round_price_to_penny(price, prefer_round_down,
                                     diff=(0.0095 - .005)):
     """
-    Modified the original function because we do not want to round
+    Asymmetric rounding function for adjusting prices to two places in a way
-    prices on crypto exchange.
+    that "improves" the price.  For limit prices, this means preferring to
+    round down on buys and preferring to round up on sells.  For stop prices,
+    it means the reverse.
 
-    Parameters
+    If prefer_round_down == True:
-    ----------
+        When .05 below to .95 above a penny, use that penny.
-    price: float
+    If prefer_round_down == False:
-
+        When .95 below to .05 above a penny, use that penny.
-    Returns
-    -------
-    float
 
+    In math-speak:
+    If prefer_round_down: [<X-1>.0095, X.0195) -> round to X.01.
+    If not prefer_round_down: (<X-1>.0005, X.0105] -> round to X.01.
     """
-    # TODO: consider overriding outside of the original function
+    # Subtracting an epsilon from diff to enforce the open-ness of the upper
-    return price
+    # bound on buys and the lower bound on sells.  Using the actual system
+    # epsilon doesn't quite get there, so use a slightly less epsilon-ey value.
+    epsilon = float_info.epsilon * 10
+    diff = diff - epsilon
+
+    # relies on rounding half away from zero, unlike numpy's bankers' rounding
+    rounded = round(price - (diff if prefer_round_down else -diff), 2)
+    if zp_math.tolerant_equals(rounded, 0.0):
+        return 0.0
+    return rounded
 
 
 def check_stoplimit_prices(price, label):

catalyst/finance/performance/period.py

@@ -88,10 +88,7 @@ from six import itervalues, iteritems
 
 import catalyst.protocol as zp
 
-from catalyst.constants import LOG_LEVEL
+log = logbook.Logger('Performance')
-
-log = logbook.Logger('Performance', level=LOG_LEVEL)
-
 TRADE_TYPE = zp.DATASOURCE_TYPE.TRADE
 
 

catalyst/finance/performance/position.py

@@ -40,9 +40,7 @@ import logbook
 from catalyst.assets import Future, Asset
 from catalyst.utils.input_validation import expect_types
 
-from catalyst.constants import LOG_LEVEL
+log = logbook.Logger('Performance')
-
-log = logbook.Logger('Performance', level=LOG_LEVEL)
 
 
 class Position(object):

catalyst/finance/performance/position_tracker.py

@@ -32,9 +32,7 @@ from catalyst.assets import (
 )
 from . position import positiondict
 
-from catalyst.constants import LOG_LEVEL
+log = logbook.Logger('Performance')
-
-log = logbook.Logger('Performance', level=LOG_LEVEL)
 
 
 PositionStats = namedtuple('PositionStats',

catalyst/finance/performance/tracker.py

@@ -70,9 +70,7 @@ import catalyst.finance.risk as risk
 
 from . position_tracker import PositionTracker
 
-from catalyst.constants import LOG_LEVEL
+log = logbook.Logger('Performance')
-
-log = logbook.Logger('Performance', level=LOG_LEVEL)
 
 
 class PerformanceTracker(object):
@@ -113,11 +111,27 @@ class PerformanceTracker(object):
                     self.treasury_curves,
                     self.trading_calendar
                 )
+        elif self.emission_rate == '5-minute':
+            self.all_benchmark_returns = pd.Series(
+                index=pd.date_range(
+                    self.sim_params.first_open,
+                    self.sim_params.last_close,
+                    freq='5min'
+                ),
+            )
+            self.cumulative_risk_metrics = \
+                risk.RiskMetricsCumulative(
+                    self.sim_params,
+                    self.treasury_curves,
+                    self.trading_calendar,
+                    create_first_day_stats=True,
+                )
         elif self.emission_rate == 'minute':
             self.all_benchmark_returns = pd.Series(index=pd.date_range(
                 self.sim_params.first_open, self.sim_params.last_close,
                 freq='Min')
             )
+
             self.cumulative_risk_metrics = \
                 risk.RiskMetricsCumulative(
                     self.sim_params,
@@ -175,14 +189,14 @@ class PerformanceTracker(object):
 
     @property
     def progress(self):
-        if self.emission_rate == 'minute':
+        if self.emission_rate in set(('minute', '5-minute')):
             # Fake a value
             return 1.0
         elif self.emission_rate == 'daily':
             return self.session_count / self.total_session_count
 
     def set_date(self, date):
-        if self.emission_rate == 'minute':
+        if self.emission_rate in set(('minute', '5-minute')):
             self.saved_dt = date
             self.todays_performance.period_close = self.saved_dt
 
@@ -356,7 +370,9 @@ class PerformanceTracker(object):
                                             bench_since_open,
                                             account.leverage)
 
+        assert self.emission_rate in set(('minute', '5-minute'))
         minute_packet = self.to_dict(emission_type='minute')
+
         return minute_packet
 
     def handle_market_close(self, dt, data_portal):

catalyst/finance/risk/cumulative.py

@@ -22,7 +22,7 @@ from pandas.tseries.tools import normalize_date
 
 from six import iteritems
 
-from .risk import (
+from . risk import (
     check_entry,
     choose_treasury
 )
@@ -37,10 +37,9 @@ from empyrical import (
     sharpe_ratio,
     sortino_ratio,
 )
-import warnings
-from catalyst.constants import LOG_LEVEL
 
-log = logbook.Logger('Risk Cumulative', level=LOG_LEVEL)
+log = logbook.Logger('Risk Cumulative')
+
 
 choose_treasury = functools.partial(choose_treasury, lambda *args: '10year',
                                     compound=False)
@@ -144,8 +143,6 @@ class RiskMetricsCumulative(object):
         self.num_trading_days = 0
 
     def update(self, dt, algorithm_returns, benchmark_returns, leverage):
-        warnings.filterwarnings('error')
-
         # Keep track of latest dt for use in to_dict and other methods
         # that report current state.
         self.latest_dt = dt
@@ -192,12 +189,9 @@ class RiskMetricsCumulative(object):
             if len(self.benchmark_returns) == 1:
                 self.benchmark_returns = np.append(0.0, self.benchmark_returns)
 
-        try:
+        self.benchmark_cumulative_returns[dt_loc] = cum_returns(
-            self.benchmark_cumulative_returns[dt_loc] = cum_returns(
+            self.benchmark_returns
-                self.benchmark_returns
+        )[-1]
-            )[-1]
-        except Exception as e:
-            log.debug('cumulative returns error: {}'.format(e))
 
         benchmark_cumulative_returns_to_date = \
             self.benchmark_cumulative_returns[:dt_loc + 1]
@@ -272,15 +266,10 @@ algorithm_returns ({algo_count}) in range {start} : {end} on {dt}"
         self.downside_risk[dt_loc] = downside_risk(
             self.algorithm_returns
         )
-
+        self.sortino[dt_loc] = sortino_ratio(
-        try:
+            self.algorithm_returns,
-            self.sortino[dt_loc] = sortino_ratio(
+            _downside_risk=self.downside_risk[dt_loc]
-                self.algorithm_returns,
+        )
-                _downside_risk=self.downside_risk[dt_loc]
-            )
-        except Exception as e:
-            log.debug('sortino ratio error: {}'.format(e))
-
         self.information[dt_loc] = information_ratio(
             self.algorithm_returns,
             self.benchmark_returns,
@@ -292,8 +281,6 @@ algorithm_returns ({algo_count}) in range {start} : {end} on {dt}"
         self.max_leverage = self.calculate_max_leverage()
         self.max_leverages[dt_loc] = self.max_leverage
 
-        warnings.resetwarnings()
-
     def to_dict(self):
         """
         Creates a dictionary representing the state of the risk report.
@@ -305,18 +292,18 @@ algorithm_returns ({algo_count}) in range {start} : {end} on {dt}"
         rval = {
             'trading_days': self.num_trading_days,
             'benchmark_volatility':
-                self.benchmark_volatility[dt_loc],
+            self.benchmark_volatility[dt_loc],
             'algo_volatility':
-                self.algorithm_volatility[dt_loc],
+            self.algorithm_volatility[dt_loc],
             'treasury_period_return': self.treasury_period_return,
             # Though the two following keys say period return,
             # they would be more accurately called the cumulative return.
             # However, the keys need to stay the same, for now, for backwards
             # compatibility with existing consumers.
             'algorithm_period_return':
-                self.algorithm_cumulative_returns[dt_loc],
+            self.algorithm_cumulative_returns[dt_loc],
             'benchmark_period_return':
-                self.benchmark_cumulative_returns[dt_loc],
+            self.benchmark_cumulative_returns[dt_loc],
             'beta': self.beta[dt_loc],
             'alpha': self.alpha[dt_loc],
             'sharpe': self.sharpe[dt_loc],

catalyst/finance/risk/period.py

@@ -36,9 +36,7 @@ from empyrical import (
     sortino_ratio
 )
 
-from catalyst.constants import LOG_LEVEL
+log = logbook.Logger('Risk Period')
-
-log = logbook.Logger('Risk Period', level=LOG_LEVEL)
 
 choose_treasury = functools.partial(risk.choose_treasury,
                                     risk.select_treasury_duration)

catalyst/finance/risk/report.py

@@ -63,9 +63,7 @@ from dateutil.relativedelta import relativedelta
 
 from . period import RiskMetricsPeriod
 
-from catalyst.constants import LOG_LEVEL
+log = logbook.Logger('Risk Report')
-
-log = logbook.Logger('Risk Report', level=LOG_LEVEL)
 
 
 class RiskReport(object):

catalyst/finance/risk/risk.py

@@ -61,9 +61,7 @@ Risk Report
 import logbook
 import numpy as np
 
-from catalyst.constants import LOG_LEVEL
+log = logbook.Logger('Risk')
-
-log = logbook.Logger('Risk', level=LOG_LEVEL)
 
 
 TREASURY_DURATIONS = [
@@ -160,7 +158,7 @@ def choose_treasury(select_treasury, treasury_curves, start_session,
                 )
                 break
 
-        if search_day and trading_calendar.name != 'OPEN':          # Supress warning for 'OPEN' calendar
+        if search_day:
             if (search_dist is None or search_dist > 1) and \
                     search_days[0] <= end_session <= search_days[-1]:
                 message = "No rate within 1 trading day of end date = \

catalyst/finance/slippage.py

@@ -41,7 +41,6 @@ DEFAULT_EQUITY_VOLUME_SLIPPAGE_BAR_LIMIT = 0.025
 DEFAULT_FUTURE_VOLUME_SLIPPAGE_BAR_LIMIT = 0.05
 
 
-
 class LiquidityExceeded(Exception):
     pass
 
@@ -206,22 +205,20 @@ class VolumeShareSlippage(SlippageModel):
     def process_order(self, data, order):
         volume = data.current(order.asset, "volume")
 
-        min_trade_size = order.asset.min_trade_size
-
         max_volume = self.volume_limit * volume
 
         # price impact accounts for the total volume of transactions
         # created against the current minute bar
         remaining_volume = max_volume - self.volume_for_bar
-        if remaining_volume < min_trade_size:
+        if remaining_volume < 1:
             # we can't fill any more transactions
             raise LiquidityExceeded()
 
         # the current order amount will be the min of the
         # volume available in the bar or the open amount.
-        cur_volume = min(remaining_volume, abs(order.open_amount))
+        cur_volume = int(min(remaining_volume, abs(order.open_amount)))
 
-        if cur_volume < min_trade_size:
+        if cur_volume < 1:
             return None, None
 
         # tally the current amount into our total amount ordered.

catalyst/finance/trading.py

@@ -26,9 +26,7 @@ from catalyst.data.loader import load_market_data
 from catalyst.utils.calendars import get_calendar
 from catalyst.utils.memoize import remember_last
 
-from catalyst.constants import LOG_LEVEL
+log = logbook.Logger('Trading')
-
-log = logbook.Logger('Trading', level=LOG_LEVEL)
 
 
 DEFAULT_CAPITAL_BASE = 1e5

catalyst/finance/transaction.py

@@ -65,10 +65,14 @@ def create_transaction(order, dt, price, amount):
     # floor the amount to protect against non-whole number orders
     # TODO: Investigate whether we can add a robust check in blotter
     # and/or tradesimulation, as well.
+    amount_magnitude = int(abs(amount))
+
+    if amount_magnitude < 1:
+        raise Exception("Transaction magnitude must be at least 1.")
 
     transaction = Transaction(
         asset=order.asset,
-        amount=amount,
+        amount=int(amount),
         dt=dt,
         price=price,
         order_id=order.id

catalyst/gens/sim_engine.pyx

@@ -20,7 +20,9 @@ cimport cython
 from cpython cimport bool
 
 cdef np.int64_t _nanos_in_minute = 60000000000
+cdef np.int64_t _nanos_in_five_minutes = 5 * _nanos_in_minute
 NANOS_IN_MINUTE = _nanos_in_minute
+NANOS_IN_FIVE_MINUTES = _nanos_in_five_minutes
 
 cpdef enum:
     BAR = 0
@@ -115,3 +117,24 @@ cdef class MinuteSimulationClock:
             yield minute, BAR
             if minute_emission:
                 yield minute, MINUTE_END
+
+cdef class FiveMinuteSimulationClock(MinuteSimulationClock):
+    @cython.boundscheck(False)
+    @cython.wraparound(False)
+    cdef dict calc_minutes_by_session(self):
+        cdef dict five_minutes_by_session
+        cdef int session_idx
+        cdef np.int64_t session_nano
+        cdef np.ndarray[np.int64_t, ndim=1] five_minutes_nanos
+
+        five_minutes_by_session = {}
+        for session_idx, session_nano in enumerate(self.sessions_nanos):
+            five_minutes_nanos = np.arange(
+                self.market_opens_nanos[session_idx],
+                self.market_closes_nanos[session_idx],
+                _nanos_in_five_minutes
+            )
+            five_minutes_by_session[session_nano] = pd.to_datetime(
+                five_minutes_nanos, utc=True, box=True
+            )
+        return five_minutes_by_session

catalyst/gens/tradesimulation.py

@@ -27,15 +27,14 @@ from catalyst.gens.sim_engine import (
     BEFORE_TRADING_START_BAR
 )
 
-from catalyst.constants import LOG_LEVEL
+log = Logger('Trade Simulation')
-
-log = Logger('Trade Simulation', level=LOG_LEVEL)
 
 
 class AlgorithmSimulator(object):
 
     EMISSION_TO_PERF_KEY_MAP = {
         'minute': 'minute_perf',
+        '5-minute': '5_minute_perf',
         'daily': 'daily_perf'
     }
 
@@ -203,7 +202,7 @@ class AlgorithmSimulator(object):
             stack.enter_context(self.processor)
             stack.enter_context(ZiplineAPI(self.algo))
 
-            if algo.data_frequency == 'minute':
+            if algo.data_frequency in set(('minute', '5-minute')):
                 def execute_order_cancellation_policy():
                     algo.blotter.execute_cancel_policy(SESSION_END)
 

catalyst/pipeline/loaders/crypto_pricing_loader.py

@@ -41,7 +41,11 @@ class CryptoPricingLoader(PipelineLoader):
             reader = bundle.daily_bar_reader
             all_sessions = cal.all_sessions
 
-        elif data_frequency == 'minute':
+        elif data_frequency == '5-minute':
+            reader = bundle.five_minute_bar_reader
+            all_sessions = cal.all_five_minutes
+
+        elif daily_bar_reader == 'minute':
             reader = bundle.minute_bar_reader
             all_sessions = cal.all_minutes
 
@@ -53,6 +57,7 @@ class CryptoPricingLoader(PipelineLoader):
         self.raw_price_loader = reader
         self._columns = dataset.columns
         self._all_sessions = all_sessions
+        self._data_frequency = data_frequency
 
     @classmethod
     def from_files(cls, pricing_path):
@@ -102,7 +107,6 @@ class CryptoPricingLoader(PipelineLoader):
 
 
 def _shift_dates(dates, start_date, end_date, shift):
-
     try:
         start = dates.get_loc(start_date)
     except KeyError:

catalyst/pipeline/loaders/equity_pricing_loader.py

@@ -40,6 +40,8 @@ class USEquityPricingLoader(PipelineLoader):
 
         if data_frequency == 'daily':
             reader = bundle.daily_bar_reader
+        elif data_frequency == '5-minute':
+            reader = bundle.five_minute_bar_reader
         elif daily_bar_reader == 'minute':
             reader = bundle.minute_bar_reader
         else:
@@ -51,6 +53,9 @@ class USEquityPricingLoader(PipelineLoader):
 
         if data_frequency == 'daily':
             all_sessions = cal.all_sessions
+        elif data_frequency == '5-minute':
+            reader = bundle.five_minute_bar_reader
+            all_sessions = cal.all_five_minutes
         elif daily_bar_reader == 'minute':
             reader = bundle.minute_bar_reader
             all_sessions = cal.all_minutes

catalyst/sources/benchmark_source.py

@@ -51,7 +51,10 @@ class BenchmarkSource(object):
         elif benchmark_returns is not None:
             daily_series = benchmark_returns[sessions[0]:sessions[-1]]
 
+            print 'BENCHMARK_RETURNS'
+
             if self.emission_rate == "minute":
+                print 'BENCHMARK_RETURNS minute'
                 # we need to take the env's benchmark returns, which are daily,
                 # and resample them to minute
                 minutes = trading_calendar.minutes_for_sessions_in_range(
@@ -65,20 +68,29 @@ class BenchmarkSource(object):
                 )
 
                 self._precalculated_series = minute_series
+            elif self.emission_rate == '5-minute':
+                print 'BENCHMARK_RETURNS 5-minute'
+                five_minutes = \
+                    trading_calendar.five_minutes_for_sessions_in_range(
+                        sessions[0],
+                        sessions[-1],
+                    )
+
+                five_minute_series = daily_series.reindex(
+                    index=five_minutes,
+                    method='ffill',
+                )
+                
+                self._precalculated_series = five_minute_series
             else:
+                print 'BENCHMARK_RETURNS daily'
                 self._precalculated_series = daily_series
         else:
             raise Exception("Must provide either benchmark_asset or "
                             "benchmark_returns.")
 
     def get_value(self, dt):
-        try:
+        return self._precalculated_series.loc[dt]
-            series = self._precalculated_series
-            value = series.loc[dt]
-            return value
-        except Exception:
-            # TODO: workaround, find permanent fix
-            return 0
 
     def get_range(self, start_dt, end_dt):
         return self._precalculated_series.loc[start_dt:end_dt]
@@ -161,8 +173,24 @@ class BenchmarkSource(object):
                 ffill=True
             )[asset]
 
+            return benchmark_series.pct_change()[1:]
+        elif self.emission_rate == '5-minute':
+            five_minutes = trading_calendar.five_minutes_for_sessions_in_range(
+                self.sessions[0], self.sessions[-1]
+            )
+            benchmark_series = data_portal.get_history_window(
+                [asset],
+                five_minutes[-1],
+                bar_count=len(five_minutes) + 1,
+                frequency='5m',
+                field='price',
+                data_frequency=self.emission_rate,
+                ffill=True,
+            )[asset]
+
             return benchmark_series.pct_change()[1:]
         else:
+            print '----------------------------------------'
             start_date = asset.start_date
             if start_date < trading_days[0]:
                 # get the window of close prices for benchmark_asset from the

catalyst/sources/requests_csv.py

@@ -23,9 +23,7 @@ from catalyst.protocol import (
 )
 from catalyst.assets import Equity
 
-from catalyst.constants import LOG_LEVEL
+logger = Logger('Requests Source Logger')
-
-logger = Logger('Requests Source Logger', level=LOG_LEVEL)
 
 
 def roll_dts_to_midnight(dts, trading_day):

catalyst/support/issue_44.py

@@ -1,109 +0,0 @@
-import pandas as pd
-from catalyst import run_algorithm
-from catalyst.exchange.exchange_utils import get_exchange_symbols
-
-from catalyst.api import (
-    symbols,
-)
-
-
-def initialize(context):
-    context.i = -1
-    context.base_currency = 'btc'
-
-
-def handle_data(context, data):
-    lookback = 60 * 24 * 7  # (minutes, hours, days)
-    context.i += 1
-    if context.i < lookback:
-        return
-
-    today = context.blotter.current_dt.strftime('%Y-%m-%d %H:%M:%S')
-
-    try:
-        # update universe everyday
-        new_day = 60 * 24
-        if not context.i % new_day:
-            context.universe = universe(context, today)
-
-        # get data every 30 minutes
-        minutes = 30
-        if not context.i % minutes and context.universe:
-            for coin in context.coins:
-                pair = str(coin.symbol)
-
-                # ohlcv data
-                open = data.history(coin, 'open', lookback,
-                                    '1m').ffill().bfill().resample(
-                    '30T').first()
-                high = data.history(coin, 'high', lookback,
-                                    '1m').ffill().bfill().resample('30T').max()
-                low = data.history(coin, 'low', lookback,
-                                   '1m').ffill().bfill().resample('30T').min()
-                close = data.history(coin, 'price', lookback,
-                                     '1m').ffill().bfill().resample(
-                    '30T').last()
-                volume = data.history(coin, 'volume', lookback,
-                                      '1m').ffill().bfill().resample(
-                    '30T').sum()
-
-                print(today, pair, close[-1])
-
-    except Exception as e:
-        print(e)
-
-
-def analyze(context=None, results=None):
-    pass
-
-
-def universe(context, today):
-    json_symbols = get_exchange_symbols('poloniex')
-    poloniex_universe_df = pd.DataFrame.from_dict(
-        json_symbols).transpose().astype(str)
-    poloniex_universe_df['base_currency'] = poloniex_universe_df.apply(
-        lambda row: row.symbol.split('_')[1],
-        axis=1)
-    poloniex_universe_df['market_currency'] = poloniex_universe_df.apply(
-        lambda row: row.symbol.split('_')[0],
-        axis=1)
-    poloniex_universe_df = poloniex_universe_df[
-        poloniex_universe_df['base_currency'] == context.base_currency]
-    poloniex_universe_df = poloniex_universe_df[
-        poloniex_universe_df.symbol != 'gas_btc']
-
-    # Markets currently not working on Catalyst 0.3.1
-    # 2017-01-01
-    # poloniex_universe_df = poloniex_universe_df[poloniex_universe_df.symbol != 'bcn_btc']
-    # poloniex_universe_df = poloniex_universe_df[poloniex_universe_df.symbol != 'burst_btc']
-    # poloniex_universe_df = poloniex_universe_df[poloniex_universe_df.symbol != 'dgb_btc']
-    # poloniex_universe_df = poloniex_universe_df[poloniex_universe_df.symbol != 'doge_btc']
-    # poloniex_universe_df = poloniex_universe_df[poloniex_universe_df.symbol != 'emc2_btc']
-    # poloniex_universe_df = poloniex_universe_df[poloniex_universe_df.symbol != 'pink_btc']
-    # poloniex_universe_df = poloniex_universe_df[poloniex_universe_df.symbol != 'sc_btc']
-    print(poloniex_universe_df.head())
-
-    date = str(today).split(' ')[0]
-
-    poloniex_universe_df = poloniex_universe_df[
-        poloniex_universe_df.start_date < date]
-    context.coins = symbols(*poloniex_universe_df.symbol)
-    print(len(poloniex_universe_df))
-    return poloniex_universe_df.symbol.tolist()
-
-
-if __name__ == '__main__':
-    start_date = pd.to_datetime('2017-01-01', utc=True)
-    end_date = pd.to_datetime('2017-10-15', utc=True)
-
-    performance = run_algorithm(start=start_date, end=end_date,
-                                capital_base=10000.0,
-                                initialize=initialize,
-                                handle_data=handle_data,
-                                analyze=analyze,
-                                exchange_name='poloniex',
-                                data_frequency='minute',
-                                base_currency='btc',
-                                live=False,
-                                live_graph=False,
-                                algo_namespace='test')

catalyst/support/issue_47.py

@@ -1,140 +0,0 @@
-"""
-Requires Catalyst version 0.3.0 or above
-Tested on Catalyst version 0.3.2
-
-These example aims to provide and easy way for users to learn how to collect data from the different exchanges.
-You simply need to specify the exchange and the market that you want to focus on.
-You will all see how to create a universe and filter it base on the exchange and the market you desire.
-
-The example prints out the closing price of all the pairs for a given market-exchange every 30 minutes.
-The example also contains the ohlcv minute data for the past seven days which could be used to create indicators
-Use this as the backbone to create your own trading strategies.
-
-Variables lookback date and date are used to ensure data for a coin existed on the lookback period specified.
-"""
-
-import numpy as np
-import pandas as pd
-from datetime import timedelta
-from catalyst import run_algorithm
-from catalyst.exchange.exchange_utils import get_exchange_symbols
-
-from catalyst.api import (
-    symbols,
-)
-
-
-def initialize(context):
-    context.i = -1  # counts the minutes
-    context.exchange = 'poloniex'  # must match the exchange specified in run_algorithm
-    context.base_currency = 'eth'  # must match the base currency specified in run_algorithm
-
-
-def handle_data(context, data):
-    lookback = 60 * 24 * 7  # (minutes, hours, days) of how far to lookback in the data history
-    context.i += 1
-
-    # current date formatted into a string
-    today = context.blotter.current_dt
-    date, time = today.strftime('%Y-%m-%d %H:%M:%S').split(' ')
-    lookback_date = today - timedelta(days=(
-    lookback / (60 * 24)))  # subtract the amount of days specified in lookback
-    lookback_date = lookback_date.strftime('%Y-%m-%d %H:%M:%S').split(' ')[
-        0]  # get only the date as a string
-
-    # update universe everyday
-    new_day = 60 * 24
-    if not context.i % new_day:
-        context.universe = universe(context, lookback_date, date)
-
-    # get data every 30 minutes
-    minutes = 30
-    if not context.i % minutes and context.universe:
-        # we iterate for every pair in the current universe
-        for coin in context.coins:
-            pair = str(coin.symbol)
-
-            # 30 minute interval ohlcv data (the standard data required for candlestick or indicators/signals)
-            # 30T means 30 minutes re-sampling of one minute data. change to your desire time interval.
-            open = fill(data.history(coin, 'open', bar_count=lookback,
-                                     frequency='1m')).resample('30T').first()
-            high = fill(data.history(coin, 'high', bar_count=lookback,
-                                     frequency='1m')).resample('30T').max()
-            low = fill(data.history(coin, 'low', bar_count=lookback,
-                                    frequency='1m')).resample('30T').min()
-            close = fill(data.history(coin, 'price', bar_count=lookback,
-                                      frequency='1m')).resample('30T').last()
-            volume = fill(data.history(coin, 'volume', bar_count=lookback,
-                                       frequency='1m')).resample('30T').sum()
-
-            # close[-1] is the equivalent to current price
-            # displays the minute price for each pair every 30 minutes
-            print(
-            today, pair, open[-1], high[-1], low[-1], close[-1], volume[-1])
-
-            # ----------------------------------------------------------------------------------------------------------
-            # -------------------------------------- Insert Your Strategy Here -----------------------------------------
-            # ----------------------------------------------------------------------------------------------------------
-
-
-def analyze(context=None, results=None):
-    pass
-
-
-# Get the universe for a given exchange and a given base_currency market
-# Example: Poloniex BTC Market
-def universe(context, lookback_date, current_date):
-    json_symbols = get_exchange_symbols(
-        context.exchange)  # get all the pairs for the exchange
-    universe_df = pd.DataFrame.from_dict(json_symbols).transpose().astype(
-        str)  # convert into a dataframe
-    universe_df['base_currency'] = universe_df.apply(
-        lambda row: row.symbol.split('_')[1],
-        axis=1)
-    universe_df['market_currency'] = universe_df.apply(
-        lambda row: row.symbol.split('_')[0],
-        axis=1)
-    # Filter all the exchange pairs to only the ones for a give base currency
-    universe_df = universe_df[
-        universe_df['base_currency'] == context.base_currency]
-
-    # Filter all the pairs to ensure that pair existed in the current date range
-    universe_df = universe_df[universe_df.start_date < lookback_date]
-    universe_df = universe_df[universe_df.end_daily >= current_date]
-    context.coins = symbols(
-        *universe_df.symbol)  # convert all the pairs to symbols
-    print(universe_df.head(), len(universe_df))
-    return universe_df.symbol.tolist()
-
-
-# Replace all NA, NAN or infinite values with its nearest value
-def fill(series):
-    if isinstance(series, pd.Series):
-        return series.replace([np.inf, -np.inf], np.nan).ffill().bfill()
-    elif isinstance(series, np.ndarray):
-        return pd.Series(series).replace([np.inf, -np.inf],
-                                         np.nan).ffill().bfill().values
-    else:
-        return series
-
-
-if __name__ == '__main__':
-    start_date = pd.to_datetime('2017-01-01', utc=True)
-    end_date = pd.to_datetime('2017-10-15', utc=True)
-
-    performance = run_algorithm(start=start_date, end=end_date,
-                                capital_base=10000.0,
-                                initialize=initialize,
-                                handle_data=handle_data,
-                                analyze=analyze,
-                                exchange_name='poloniex',
-                                data_frequency='minute',
-                                base_currency='eth',
-                                live=False,
-                                live_graph=False,
-                                algo_namespace='simple_universe')
-
-"""
-Run in Terminal (inside catalyst environment):
-python simple_universe.py
-"""

catalyst/support/issue_55.py

@@ -1,42 +0,0 @@
-import talib
-import pandas as pd
-
-from catalyst import run_algorithm
-from catalyst.api import symbol
-
-
-def initialize(context):
-    print('initializing')
-    context.asset = symbol('xcp_btc')
-
-
-def handle_data(context, data):
-    print('handling bar: {}'.format(data.current_dt))
-
-    price = data.current(context.asset, 'close')
-    print('got price {price}'.format(price=price))
-
-    try:
-        prices = data.history(
-            context.asset,
-            fields='close',
-            bar_count=1,
-            frequency='1D'
-        )
-        print('got {} price entries\n'.format(len(prices), prices))
-    except Exception as e:
-        print(e)
-
-
-run_algorithm(
-    capital_base=1,
-    start=pd.to_datetime('2015-3-2', utc=True),
-    end=pd.to_datetime('2017-8-31', utc=True),
-    data_frequency='daily',
-    initialize=initialize,
-    handle_data=handle_data,
-    analyze=None,
-    exchange_name='poloniex',
-    algo_namespace='issue_55',
-    base_currency='btc'
-)

catalyst/support/issue_57.py

@@ -1,46 +0,0 @@
-import talib
-import pandas as pd
-
-from catalyst import run_algorithm
-from catalyst.api import symbol
-
-
-def initialize(context):
-    print('initializing')
-    context.asset = symbol('btc_usdt')
-
-
-def handle_data(context, data):
-    print('handling bar: {}'.format(data.current_dt))
-
-    price = data.current(context.asset, 'close')
-    print('got price {price}'.format(price=price))
-
-    try:
-        prices = data.history(
-            context.asset,
-            fields='close',
-            bar_count=60,
-            frequency='1D'
-        )
-        print('got {} price entries\n'.format(len(prices), prices))
-    except Exception as e:
-        print(e)
-
-
-run_algorithm(
-    capital_base=1,
-    start=pd.to_datetime('2016-2-11', utc=True),
-    end=pd.to_datetime('2017-8-31', utc=True),
-    data_frequency='daily',
-    initialize=initialize,
-    handle_data=handle_data,
-    analyze=None,
-    exchange_name='bittrex',
-    algo_namespace='issue_57',
-    base_currency='btc'
-<<<<<<< HEAD
-)
-=======
-)
->>>>>>> develop

catalyst/support/rodrigo_1.py

@@ -1,153 +0,0 @@
-import pandas as pd
-from logbook import Logger, DEBUG
-
-from catalyst import run_algorithm
-from catalyst.api import (schedule_function, order_target_percent, symbol,
-                          date_rules, get_open_orders, cancel_order, record,
-                          set_commission, set_slippage)
-
-log = Logger('rodrigo_1', level=DEBUG)
-"""
-The initialize function sets any data or variables that 
-you'll use in your algorithm. 
-It's only called once at the beginning of your algorithm.
-"""
-
-
-def initialize(context):
-    # Select asset of interest
-    context.asset = symbol('BTC_USD')
-
-    # set_commission(TradingPairFeeSchedule(maker_fee=0.5, taker_fee=0.5))
-    # set_slippage(TradingPairFixedSlippage(spread=0.5))
-    # Set up a rebalance method to run every day
-    schedule_function(rebalance, date_rule=date_rules.every_day())
-
-
-"""
-Rebalance function scheduled to run once per day.
-"""
-
-
-def rebalance(context, data):
-    # To make market decisions, we're calculating the token's
-    # moving average for the last 5 days.
-
-    # We get the price history for the last 5 days.
-    price_history = data.history(context.asset, fields='price', bar_count=5,
-                                 frequency='1d')
-
-    # Then we take an average of those 5 days.
-    average_price = price_history.mean()
-
-    # We also get the coin's current price.
-    price = data.current(context.asset, 'price')
-
-    # Cancel any outstanding orders
-    orders = get_open_orders(context.asset) or []
-    for order in orders:
-        cancel_order(order)
-
-    # If our coin is currently listed on a major exchange
-    if data.can_trade(context.asset):
-        # If the current price is 1% above the 5-day average price,
-        # we open a long position. If the current price is below the
-        # average price, then we want to close our position to 0 shares.
-        if price > (1.01 * average_price):
-            # Place the buy order (positive means buy, negative means sell)
-            order_target_percent(context.asset, .99)
-            log.info("Buying %s" % (context.asset.symbol))
-        elif price < average_price:
-            # Sell all of our shares by setting the target position to zero
-            order_target_percent(context.asset, 0)
-            log.info("Selling %s" % (context.asset.symbol))
-
-    # Use the record() method to track up to five custom signals.
-    # Record Apple's current price and the average price over the last
-    # five days.
-    cash = context.portfolio.cash
-    leverage = context.account.leverage
-
-    record(price=price, average_price=average_price, cash=cash,
-           leverage=leverage)
-
-
-def analyze(context=None, results=None):
-    import matplotlib.pyplot as plt
-
-    # Plot the portfolio and asset data.
-    ax1 = plt.subplot(511)
-    results[['portfolio_value']].plot(ax=ax1)
-    ax1.set_ylabel('Portfolio Value (USD)')
-
-    ax2 = plt.subplot(512, sharex=ax1)
-    ax2.set_ylabel('{asset} (USD)'.format(asset=context.asset))
-    (results[[
-        'price',
-    ]]).plot(ax=ax2)
-
-    trans = results.ix[[t != [] for t in results.transactions]]
-    buys = trans.ix[
-        [t[0]['amount'] > 0 for t in trans.transactions]
-    ]
-    sells = trans.ix[
-        [t[0]['amount'] < 0 for t in trans.transactions]
-    ]
-
-    ax2.plot(
-        buys.index,
-        results.price[buys.index],
-        '^',
-        markersize=10,
-        color='g',
-    )
-    ax2.plot(
-        sells.index,
-        results.price[sells.index],
-        'v',
-        markersize=10,
-        color='r',
-    )
-
-    ax3 = plt.subplot(513, sharex=ax1)
-    results[['leverage']].plot(ax=ax3)
-    ax3.set_ylabel('Leverage ')
-
-    ax4 = plt.subplot(514, sharex=ax1)
-    results[['cash']].plot(ax=ax4)
-    ax4.set_ylabel('Cash (USD)')
-
-    results[[
-        'algorithm',
-        'benchmark',
-    ]] = results[[
-        'algorithm_period_return',
-        'benchmark_period_return',
-    ]]
-
-    ax5 = plt.subplot(515, sharex=ax1)
-    results[[
-        'algorithm',
-        'benchmark',
-    ]].plot(ax=ax5)
-    ax5.set_ylabel('Percent Change')
-
-    plt.legend(loc=3)
-
-    # Show the plot.
-    plt.gcf().set_size_inches(18, 8)
-    plt.show()
-
-
-run_algorithm(
-    capital_base=100000,
-    start=pd.to_datetime('2017-1-1', utc=True),
-    end=pd.to_datetime('2017-10-22', utc=True),
-    data_frequency='minute',
-    initialize=initialize,
-    handle_data=None,
-    analyze=analyze,
-    exchange_name='bitfinex',
-    algo_namespace='rodrigo_1',
-    base_currency='usd'
-)

catalyst/utils/calendars/exchange_calendar_open.py

@@ -1,7 +1,6 @@
 from datetime import time
 from pytz import timezone
 
-from pandas import Timestamp
 from pandas.tseries.offsets import DateOffset
 
 from catalyst.utils.memoize import lazyval
@@ -29,6 +28,3 @@ class OpenExchangeCalendar(TradingCalendar):
     @lazyval
     def day(self):
         return DateOffset(days=1)
-
-    def __init__(self, *args, **kwargs):
-        super(OpenExchangeCalendar, self).__init__(start=Timestamp('2015-3-1', tz='UTC'), **kwargs)

catalyst/utils/calendars/trading_calendar.py

@@ -117,6 +117,9 @@ class TradingCalendar(with_metaclass(ABCMeta)):
 
         self._trading_minutes_nanos = self.all_minutes.values.\
             astype(np.int64)
+        
+        self._trading_five_minutes_nanos = self.all_five_minutes.values.\
+            astype(np.int64)
 
         self.first_trading_session = _all_days[0]
         self.last_trading_session = _all_days[-1]
@@ -179,6 +182,18 @@ class TradingCalendar(with_metaclass(ABCMeta)):
         """
         return int(self._minutes_per_session[start_session:end_session].sum())
 
+    @lazyval
+    def _five_minutes_per_session(self):
+        diff = self.schedule.market_close - self.schedule.market_open
+        diff = diff.astype('timedelta64[m]')
+        return (diff + 1) // 5
+
+    def five_minutes_count_for_sessions_in_range(self,
+                                                 start_session,
+                                                 end_session):
+        five_mins = self._five_minutes_per_session[start_session:end_session]
+        return int(five_mins.sum())
+
     @property
     def regular_holidays(self):
         """
@@ -371,6 +386,10 @@ class TradingCalendar(with_metaclass(ABCMeta)):
         idx = next_divider_idx(self._trading_minutes_nanos, dt.value)
         return self.all_minutes[idx]
 
+    def next_five_minute(self, dt):
+        idx = next_divider_idx(self._trading_five_minutes_nanos, dt.values)
+        return self.all_five_mintutes[idx]
+
     def previous_minute(self, dt):
         """
         Given a dt, return the previous exchange minute.
@@ -465,6 +484,12 @@ class TradingCalendar(with_metaclass(ABCMeta)):
             end_minute=self.schedule.at[session_label, 'market_close'],
         )
 
+    def five_minutes_for_session(self, session_label):
+        return self.five_minutes_in_range(
+            start_five_minute=self.schedule.at[session_label, 'market_open'],
+            end_five_minute=self.schedule.at[session_label, 'market_close'],
+        )
+
     def minutes_window(self, start_dt, count):
         start_dt_nanos = start_dt.value
         all_minutes_nanos = self._trading_minutes_nanos
@@ -566,6 +591,20 @@ class TradingCalendar(with_metaclass(ABCMeta)):
 
         return abs(end_idx - start_idx)
 
+    def five_minutes_in_range(self, start_five_minute, end_five_minute):
+        start_idx = searchsorted(self._trading_five_minutes_nanos,
+                                 start_five_minute.value)
+
+        end_idx = searchsorted(self._trading_five_minutes_nanos,
+                               end_five_minute.value)
+
+        if end_five_minute.value == self._trading_five_minutes_nanos[end_idx]:
+            # if the end minute is a market minute, increase by 1
+            end_idx += 1
+
+        return self.all_five_minutes[start_idx:end_idx]
+        
+
     def minutes_in_range(self, start_minute, end_minute):
         """
         Given start and end minutes, return all the calendar minutes
@@ -623,6 +662,15 @@ class TradingCalendar(with_metaclass(ABCMeta)):
 
         return self.minutes_in_range(first_minute, last_minute)
 
+    def five_minutes_for_sessions_in_range(self,
+                                           start_session_label,
+                                           end_session_label):
+
+        first_minute, _ = self.open_and_close_for_session(start_session_label)
+        _, last_minute = self.open_and_close_for_session(end_session_label)
+
+        return self.five_minutes_in_range(first_minute, last_minute)
+
     def open_and_close_for_session(self, session_label):
         """
         Returns a tuple of timestamps of the open and close of the session
@@ -729,6 +777,13 @@ class TradingCalendar(with_metaclass(ABCMeta)):
 
         return DatetimeIndex(all_minutes).tz_localize("UTC")
 
+    @lazyval
+    def all_five_minutes(self):
+        """
+        Returns a DatetimeIndex representing all the five minutes in this calendar.
+        """
+        return self._all_minutes_with_interval(5) 
+
     @lazyval
     def all_minutes(self):
         """

catalyst/utils/events.py

@@ -47,6 +47,8 @@ __all__ = [
     'NDaysBeforeLastTradingDayOfMonth',
     'StatefulRule',
     'OncePerDay',
+    'OncePerFiveMinutes',
+    'OncePerMinute',
 
     # Factory API
     'date_rules',
@@ -552,15 +554,18 @@ class StatefulRule(EventRule):
         """
         self.should_trigger = callable_
 
-
+class OncePerInterval(StatefulRule):
-class OncePerDay(StatefulRule):
     def __init__(self, rule=None):
         self.triggered = False
 
         self.date = None
         self.next_date = None
 
-        super(OncePerDay, self).__init__(rule)
+        super(OncePerInterval, self).__init__(rule)
+
+    @lazyval
+    def interval(self):
+        raise NotImplementedError
 
     def should_trigger(self, dt):
         if self.date is None or dt >= self.next_date:
@@ -570,11 +575,28 @@ class OncePerDay(StatefulRule):
 
             # record the timestamp for the next day, so that we can use it
             # to know if we've moved to the next day
-            self.next_date = dt + pd.Timedelta(1, unit="d")
+            self.next_date = dt + self.interval
 
         if not self.triggered and self.rule.should_trigger(dt):
             self.triggered = True
             return True
+        
+
+
+class OncePerDay(OncePerInterval):
+    @lazyval
+    def interval(self):
+        return pd.Timedelta(1, unit='d')
+
+class OncePerFiveMinutes(OncePerInterval):
+    @lazyval
+    def interval(self):
+        return pd.Timedelta(5, unit='m')
+
+class OncePerMinute(OncePerInterval):
+    @lazyval
+    def interval(self):
+        return pd.Timedelta(1, unit='m')
 
 
 # Factory API
@@ -602,6 +624,7 @@ class date_rules(object):
 class time_rules(object):
     market_open = AfterOpen
     market_close = BeforeClose
+    every_5_minutes = Always
     every_minute = Always
 
 
@@ -611,7 +634,11 @@ class calendars(object):
     US_FUTURES = sentinel('US_FUTURES')
 
 
-def make_eventrule(date_rule, time_rule, cal, half_days=True):
+def make_eventrule(date_rule,
+                   time_rule,
+                   cal,
+                   half_days=True,
+                   data_frequency=None):
     """
     Constructs an event rule from the factory api.
     """
@@ -627,4 +654,15 @@ def make_eventrule(date_rule, time_rule, cal, half_days=True):
         nhd_rule.cal = cal
         inner_rule = date_rule & time_rule & nhd_rule
 
-    return OncePerDay(rule=inner_rule)
+    if data_frequency == 'daily':
+        return OncePerDay(rule=inner_rule)
+    elif data_frequency == '5-minute':
+        return OncePerFiveMinutes(rule=inner_rule)
+    elif data_frequency == 'minute':
+        return OncePerMinute(rule=inner_rule)
+    else:
+        raise ValueError(
+            'Cannot make event rule for data frequency: {}'.format(
+                data_frequency,
+            )
+        )

catalyst/utils/math_utils.py

@@ -17,8 +17,6 @@ import math
 
 from numpy import isnan
 
-def round_nearest(x, a):
-    return round(round(x / a) * a, -int(math.floor(math.log10(a))))
 
 def tolerant_equals(a, b, atol=10e-7, rtol=10e-7, equal_nan=False):
     """Check if a and b are equal with some tolerance.

catalyst/utils/paths.py

@@ -126,7 +126,7 @@ def catalyst_root(environ=None):
 
     root = environ.get('ZIPLINE_ROOT', None)
     if root is None:
-        root = os.path.join(expanduser('~'),'.catalyst')
+        root = expanduser('~/.catalyst')
 
     return root
 

catalyst/utils/run_algo.py

@@ -1,52 +1,34 @@
 import os
+import re
+from runpy import run_path
 import sys
 import warnings
-from datetime import timedelta
-from runpy import run_path
-from time import sleep
 
 import click
-import pandas as pd
-
-from catalyst.exchange.bittrex.bittrex import Bittrex
-from catalyst.exchange.bitfinex.bitfinex import Bitfinex
-from catalyst.exchange.poloniex.poloniex import Poloniex
-
 try:
     from pygments import highlight
     from pygments.lexers import PythonLexer
     from pygments.formatters import TerminalFormatter
-
     PYGMENTS = True
 except:
     PYGMENTS = False
 from toolz import valfilter, concatv
 from functools import partial
 
+from catalyst.algorithm import TradingAlgorithm
+from catalyst.data.bundles.core import load
+from catalyst.data.data_portal import DataPortal
+from catalyst.data.loader import load_crypto_market_data
 from catalyst.finance.trading import TradingEnvironment
+from catalyst.pipeline.data import USEquityPricing, CryptoPricing
+from catalyst.pipeline.loaders import (
+    USEquityPricingLoader,
+    CryptoPricingLoader,
+)
 from catalyst.utils.calendars import get_calendar
 from catalyst.utils.factory import create_simulation_parameters
-from catalyst.data.loader import load_crypto_market_data
 import catalyst.utils.paths as pth
 
-from catalyst.exchange.exchange_algorithm import ExchangeTradingAlgorithmLive, \
-    ExchangeTradingAlgorithmBacktest
-from catalyst.exchange.exchange_data_portal import DataPortalExchangeLive, \
-    DataPortalExchangeBacktest
-from catalyst.exchange.asset_finder_exchange import AssetFinderExchange
-from catalyst.exchange.exchange_portfolio import ExchangePortfolio
-from catalyst.exchange.exchange_errors import (
-    ExchangeRequestError, ExchangeAuthEmpty,
-    ExchangeRequestErrorTooManyAttempts,
-    BaseCurrencyNotFoundError, ExchangeNotFoundError)
-from catalyst.exchange.exchange_utils import get_exchange_auth, \
-    get_algo_object, get_exchange_folder
-from logbook import Logger
-
-from catalyst.constants import LOG_LEVEL
-
-log = Logger('run_algo', level=LOG_LEVEL)
-
 
 class _RunAlgoError(click.ClickException, ValueError):
     """Signal an error that should have a different message if invoked from
@@ -86,12 +68,7 @@ def _run(handle_data,
          output,
          print_algo,
          local_namespace,
-         environ,
+         environ):
-         live,
-         exchange,
-         algo_namespace,
-         base_currency,
-         live_graph):
     """Run a backtest for the given algorithm.
 
     This is shared between the cli and :func:`catalyst.run_algo`.
@@ -140,188 +117,108 @@ def _run(handle_data,
         else:
             click.echo(algotext)
 
-    mode = 'live' if live else 'backtest'
+    if bundle is not None:
-    log.info('running algo in {mode} mode'.format(mode=mode))
+        bundles = bundle.split(',')
 
-    exchange_name = exchange
+        def get_trading_env_and_data(bundles):
-    if exchange_name is None:
+            env = data = None
-        raise ValueError('Please specify at least one exchange.')
 
-    exchange_list = [x.strip().lower() for x in exchange.split(',')]
+            b = 'poloniex'
+            if len(bundles) == 0:
+                return env, data
+            elif len(bundles) == 1:
+                b = bundles[0]
 
-    exchanges = dict()
+            bundle_data = load(
-    for exchange_name in exchange_list:
+                b,
-
+                environ,
-        # Looking for the portfolio from the cache first
+                bundle_timestamp,
-        portfolio = get_algo_object(
-            algo_name=algo_namespace,
-            key='portfolio_{}'.format(exchange_name),
-            environ=environ
-        )
-
-        if portfolio is None:
-            portfolio = ExchangePortfolio(
-                start_date=pd.Timestamp.utcnow()
             )
 
-        # This corresponds to the json file containing api token info
+            prefix, connstr = re.split(
-        exchange_auth = get_exchange_auth(exchange_name)
+                r'sqlite:///',
-
+                str(bundle_data.asset_finder.engine.url),
-        if live and (exchange_auth['key'] == '' or exchange_auth['secret'] == ''):
+                maxsplit=1,
-            raise ExchangeAuthEmpty(
-                    exchange=exchange_name.title(), 
-                    filename=os.path.join(get_exchange_folder(exchange_name, environ), 'auth.json') )
-
-        if exchange_name == 'bitfinex':
-            exchanges[exchange_name] = Bitfinex(
-                key=exchange_auth['key'],
-                secret=exchange_auth['secret'],
-                base_currency=base_currency,
-                portfolio=portfolio
             )
-        elif exchange_name == 'bittrex':
+            if prefix:
-            exchanges[exchange_name] = Bittrex(
+                raise ValueError(
-                key=exchange_auth['key'],
+                    "invalid url %r, must begin with 'sqlite:///'" %
-                secret=exchange_auth['secret'],
+                    str(bundle_data.asset_finder.engine.url),
-                base_currency=base_currency,
-                portfolio=portfolio
-            )
-        elif exchange_name == 'poloniex':
-            exchanges[exchange_name] = Poloniex(
-                key=exchange_auth['key'],
-                secret=exchange_auth['secret'],
-                base_currency=base_currency,
-                portfolio=portfolio
-            )
-        else:
-            raise ExchangeNotFoundError(exchange_name=exchange_name)
-
-    open_calendar = get_calendar('OPEN')
-
-    env = TradingEnvironment(
-        load=partial(
-            load_crypto_market_data,
-            environ=environ,
-            start_dt=start,
-            end_dt=end
-        ),
-        environ=environ,
-        exchange_tz='UTC',
-        asset_db_path=None  # We don't need an asset db, we have exchanges
-    )
-    env.asset_finder = AssetFinderExchange()
-    choose_loader = None  # TODO: use the DataPortal for in the algorithm class for this
-
-    if live:
-        start = pd.Timestamp.utcnow()
-
-        # TODO: fix the end data.
-        end = start + timedelta(hours=8760)
-
-        data = DataPortalExchangeLive(
-            exchanges=exchanges,
-            asset_finder=env.asset_finder,
-            trading_calendar=open_calendar,
-            first_trading_day=pd.to_datetime('today', utc=True)
-        )
-
-        def fetch_capital_base(exchange, attempt_index=0):
-            """
-            Fetch the base currency amount required to bootstrap
-            the algorithm against the exchange.
-
-            The algorithm cannot continue without this value.
-
-            :param exchange: the targeted exchange
-            :param attempt_index:
-            :return capital_base: the amount of base currency available for
-            trading
-            """
-            try:
-                log.debug('retrieving capital base in {} to bootstrap '
-                          'exchange {}'.format(base_currency, exchange_name))
-                balances = exchange.get_balances()
-            except ExchangeRequestError as e:
-                if attempt_index < 20:
-                    log.warn(
-                        'could not retrieve balances on {}: {}'.format(
-                            exchange.name, e
-                        )
-                    )
-                    sleep(5)
-                    return fetch_capital_base(exchange, attempt_index + 1)
-
-                else:
-                    raise ExchangeRequestErrorTooManyAttempts(
-                        attempts=attempt_index,
-                        error=e
-                    )
-
-            if base_currency in balances:
-                return balances[base_currency]
-            else:
-                raise BaseCurrencyNotFoundError(
-                    base_currency=base_currency,
-                    exchange=exchange_name
                 )
 
-        capital_base = 0
+            open_calendar = get_calendar('OPEN')
-        for exchange_name in exchanges:
-            exchange = exchanges[exchange_name]
-            capital_base += fetch_capital_base(exchange)
 
-        sim_params = create_simulation_parameters(
+            env = TradingEnvironment(
-            start=start,
+                load=partial(load_crypto_market_data, environ=environ),
-            end=end,
+                bm_symbol='USDT_BTC',
-            capital_base=capital_base,
+                trading_calendar=open_calendar,
-            emission_rate='minute',
+                asset_db_path=connstr,
-            data_frequency='minute'
+                environ=environ,
-        )
+            )
 
-        # TODO: use the constructor instead
+            first_trading_day = bundle_data.minute_bar_reader.first_trading_day
-        sim_params._arena = 'live'
+
+            data = DataPortal(
+                env.asset_finder,
+                open_calendar,
+                first_trading_day=first_trading_day,
+                minute_reader=bundle_data.minute_bar_reader,
+                five_minute_reader=bundle_data.five_minute_bar_reader,
+                daily_reader=bundle_data.daily_bar_reader,
+                adjustment_reader=bundle_data.adjustment_reader,
+            )
+
+            return env, data
+
+        def get_loader_for_bundle(b):
+            bundle_data = load(
+                b,
+                environ,
+                bundle_timestamp,
+            )
+
+            if b == 'poloniex':
+                return CryptoPricingLoader(
+                           bundle_data,
+                           data_frequency,
+                           CryptoPricing,
+                       )
+            elif b == 'quandl':
+                return USEquityPricingLoader(
+                           bundle_data,
+                           data_frequency,
+                           USEquityPricing,
+                       )
+            raise ValueError(
+                "No PipelineLoader registered for bundle %s." % b
+            )
+
+        loaders = [get_loader_for_bundle(b) for b in bundles]
+        env, data = get_trading_env_and_data(bundles)
+
+        def choose_loader(column):
+            for loader in loaders:
+                if column in loader.columns:
+                    return loader
+            raise ValueError(
+                "No PipelineLoader registered for column %s." % column
+            )
 
-        algorithm_class = partial(
-            ExchangeTradingAlgorithmLive,
-            exchanges=exchanges,
-            algo_namespace=algo_namespace,
-            live_graph=live_graph
-        )
     else:
-        # Removed the existing Poloniex fork to keep things simple
+        env = TradingEnvironment(environ=environ)
-        # We can add back the complexity if required.
+        choose_loader = None
 
-        # I don't think that we should have arbitrary price data bundles
+    perf = TradingAlgorithm(
-        # Instead, we should center this data around exchanges.
+        namespace=namespace,
-        # We still need to support bundles for other misc data, but we
+        env=env,
-        # can handle this later.
+        get_pipeline_loader=choose_loader,
-
+        sim_params=create_simulation_parameters(
-        data = DataPortalExchangeBacktest(
-            exchanges=exchanges,
-            asset_finder=None,
-            trading_calendar=open_calendar,
-            first_trading_day=start,
-            last_available_session=end
-        )
-
-        sim_params = create_simulation_parameters(
             start=start,
             end=end,
             capital_base=capital_base,
             data_frequency=data_frequency,
             emission_rate=data_frequency,
-        )
+        ),
-
-        algorithm_class = partial(
-            ExchangeTradingAlgorithmBacktest,
-            exchanges=exchanges
-        )
-
-    perf = algorithm_class(
-        namespace=namespace,
-        env=env,
-        get_pipeline_loader=choose_loader,
-        sim_params=sim_params,
         **{
             'initialize': initialize,
             'handle_data': handle_data,
@@ -397,10 +294,10 @@ def load_extensions(default, extensions, strict, environ, reload=False):
             _loaded_extensions.add(ext)
 
 
-def run_algorithm(initialize,
+def run_algorithm(start,
-                  capital_base=None,
+                  end,
-                  start=None,
+                  initialize,
-                  end=None,
+                  capital_base,
                   handle_data=None,
                   before_trading_start=None,
                   analyze=None,
@@ -411,12 +308,7 @@ def run_algorithm(initialize,
                   default_extension=True,
                   extensions=(),
                   strict_extensions=True,
-                  environ=os.environ,
+                  environ=os.environ):
-                  live=False,
-                  exchange_name=None,
-                  base_currency=None,
-                  algo_namespace=None,
-                  live_graph=False):
     """Run a trading algorithm.
 
     Parameters
@@ -470,12 +362,6 @@ def run_algorithm(initialize,
     environ : mapping[str -> str], optional
         The os environment to use. Many extensions use this to get parameters.
         This defaults to ``os.environ``.
-    live: execute live trading
-    exchange_conn: The exchange connection parameters
-
-    Supported Exchanges
-    -------------------
-    bitfinex
 
     Returns
     -------
@@ -488,30 +374,25 @@ def run_algorithm(initialize,
     """
     load_extensions(default_extension, extensions, strict_extensions, environ)
 
-    # I'm not sure that we need this since the modified DataPortal
+    non_none_data = valfilter(bool, {
-    # does not require extensions to be explicitly loaded.
+        'data': data is not None,
+        'bundle': bundle is not None,
+    })
+    if not non_none_data:
+        # if neither data nor bundle are passed use 'quantopian-quandl'
+        bundle = 'quantopian-quandl'
 
-    # This will be useful for arbitrary non-pricing bundles but we may
+    elif len(non_none_data) != 1:
-    # need to modify the logic.
+        raise ValueError(
-    if not live:
+            'must specify one of `data`, `data_portal`, or `bundle`,'
-        non_none_data = valfilter(bool, {
+            ' got: %r' % non_none_data,
-            'data': data is not None,
+        )
-            'bundle': bundle is not None,
-        })
-        if not non_none_data:
-            # if neither data nor bundle are passed use 'quantopian-quandl'
-            bundle = 'quantopian-quandl'
 
-        elif len(non_none_data) != 1:
+    elif 'bundle' not in non_none_data and bundle_timestamp is not None:
-            raise ValueError(
+        raise ValueError(
-                'must specify one of `data`, `data_portal`, or `bundle`,'
+            'cannot specify `bundle_timestamp` without passing `bundle`',
-                ' got: %r' % non_none_data,
+        )
-            )
 
-        elif 'bundle' not in non_none_data and bundle_timestamp is not None:
-            raise ValueError(
-                'cannot specify `bundle_timestamp` without passing `bundle`',
-            )
     return _run(
         handle_data=handle_data,
         initialize=initialize,
@@ -531,9 +412,4 @@ def run_algorithm(initialize,
         print_algo=False,
         local_namespace=False,
         environ=environ,
-        live=live,
-        exchange=exchange_name,
-        algo_namespace=algo_namespace,
-        base_currency=base_currency,
-        live_graph=live_graph
     )

docs/CNAME

@@ -1 +1 @@
-enigma-catalyst.readthedocs.io
+www.zipline.io

docs/live-trading-blueprint.md

@@ -1,207 +0,0 @@
-<h1>Live Trading Blueprint</h1>
-The purpose of this document is to allow project contributors navigate
-through the ongoing live trading implementation.
-
-<h2>Components</h2>
-At a high level, the following components have been implemented to coerce
-zipline into live trading.
-
-<h3>Exchange</h3>
-
-*catalyst/exchange*
-
-Exchange is a new package introducing cryptocurrency
-exchanges to zipline. The package contains mostly new implementations
-of existing components, adapted to characteristics of exchanges.
-
-Here are some key characteristics which make cryptocurrency exchanges
-exchanges different compared to equity brokers.
-* They trade around the clock.
-* Currency symbols are inconsistent across exchanges.
-* They trade currency pairs (i.e. the base currency is not always be USD).
-This is a paradigm shift in context of zipline. Additional
-business logic will be required to manage the portfolio data and orders.
-* The price of a single asset might vary across exchanges. This means
-arbitrage opportunities. Consequently, to extract maximum alpha, the
-platform should not only support multiple exchanges, but also multiple
-exchanges per algorithm.
-* The fee model is usually more complex than that of an equity broker.
-It can vary drastically between exchanges.
-* There are no splits, mergers, etc. to worry about.
-* A complete order book is usually available, the platform should
-offer access to it order to help traders reduce slippage.
-
-<h3>New Components</h3>
-These components of the exchange package were added to the zipline
-sources.
-
-<h4>Exchange</h4>
-
-*catalyst/exchange/exchange.py*
-
-Abstract class which acts as an interface for the implementation of
-various exchanges. It also contains logic common to all exchanges.
-
-<h4>Bitfinex</h4>
-
-*catalyst/exchange/bitfinex.py*
-
-The Bitfinex exchange implementation. It extends the Exchange class.
-
-<h4>DataPortalExchange</h4>
-
-*catalyst/exchange/data_portal_exchange.py*
-
-Extends the zipline DataPortal to route spot data to the exchange.
-This is critical because it allows the algoritm to request data in
-real-time.
-
-For example, `data.current(asset, 'price')` retrieves the current price
-of the asset, not the price at the time of yielding the bar this
-is critical to minimize slippage.
-
-At the time of writing, it only supports spot data but I believe that
-it should be extended to historical data as well. Some exchanges
-have better historical data APIs than others. This will need to
-be considered during each individual implementation.
-
-<h4>ExchangeClock</h4>
-
-*catalyst/exchange/exchange_clock.py*
-
-An implementation to the zipline Clock which runs 24/7. It yields a
-bar every minute.
-
-<h4>AssetFinderExchange</h4>
-
-*catalyst/exchange/asset_finder_exchange.py*
-
-An alternate implementation of AssetFinder which locates each asset
-against the exchanges instead of bundle databases.
-
-For example, `symbol('eth_usd')` should return an Ethereum/USD asset
-regardless of currency notation of the target exchange.
-
-To acheive this, I have created a dictionary of currencies for the
-Bitfinex exchange. Here is what it looks like.
-* Each key represents the exchange specific symbol.
-* The symbol attribute represents the abstract symbol common across
-all exchanges for the given currency.
-* The start_date attribute should correspond to its first trading day
-on the exchange.
-
-```json
-{
-  "btcusd": {
-    "symbol": "btc_usd",
-    "start_date": "2010-01-01"
-  },
-  "ltcusd": {
-    "symbol": "ltc_usd",
-    "start_date": "2010-01-01"
-  },
-  "ltcbtc": {
-    "symbol": "ltc_btc",
-    "start_date": "2010-01-01"
-  },
-  "ethusd": {
-    "symbol": "eth_usd",
-    "start_date": "2010-01-01"
-  },
-  "ethbtc": {
-    "symbol": "eth_btc",
-    "start_date": "2010-01-01"
-  }
-}
-```
-
-<h4>ExchangeTradingAlgorithm</h4>
-
-*catalyst/exchange/algorithm_exchange.py*
-
-Extends the TradingAlgorithm class which orchestrates the api
-operations. This class brings together most of the components
-described above.
-
-<h3>Modified Components</h3>
-
-The following components have been modified to include conditional
-business logic to enable live trading.
-
-<h4>run_algorithm</h4>
-
-*catalyst/utils/run_algo.py*
-
-The run_algorithm interface is an entry point to execute an
-algorithm in zipline. This component was already modified for
-the catalyst concurrency bundles. I added conditional logic
-which should not interfere with backtesting.
-
-In a nutshell, the run_algorithm method now contains three additional
-parameters:
-* live: If True, zipline will attempt to trade live. If False or not
-specified, it will run a backtest as normal.
-* algo_namespace: An arbitrary namespace for the current algorithm.
-It will be used to persist data between runs.
-* exchange_conn: A dictionary containing the attributes required
-to instantiate an exchange. Here is an example for Bitfinex:
-
-```python
-exchange_conn = dict(
-    name='bitfinex',
-    key='',
-    secret=b'',
-    base_currency='usd'
-)
-```
-
-The following sample algorithm uses the run_algorithm interface:
-
-*catalyst/examples/buy_and_hold_live.py*
-
-<h2>Portfolio Management</h2>
-
-Zipline has a Portfolio class containing key metrics used by zipline
-for, but not only, these reasons:
-
-* Placing orders: When placing orders (e.g. order_target_percent),
-zipline queries the portfolio to assess the size of current positions,
-cash available, etc.
-* Measuring performance: The portfolio contains attributes like
-cost basis of each asset, p&l, etc. which zipline uses to compute all
-of its performance criteria.
-
-When backtesting, zipline automatically updates the Portfolio object
-of its corresponding algorithm. When live trading, these updates should
-be the responsibility of the exchange as it holds the truth for:
-
-* Executed price of each order (including fees and slippage)
-* Partial / failed orders
-* Cash (i.e. base currency) available
-* Cost basis of each position
-
-If each exchange account had a one-to-one relationship with an
-algorithm, portfolio metrics could be retrieved directly from the
-exchange without persisting any data to the algorithm. However,
-doing this would have at least the following drawbacks:
-
-* It may not be reasonable to ask users to dedicate an
-exchange account to a single algorithm. Exchanges are not easy
-to partition.
-* If an exchange account contains existing positions, the calculated
-cost basis would correspond to all positions, not just those
-initiated by the algorithm.
-* It would not be possible impose trading limits on algorithms.
-
-It follows that Portfolio metrics should be calculated using a strategic
-combination of the exchange data and algorithm activity. While tracking
-the activity of an algorithm works well in backtesting, it is more
-challenging during live trading. A live algorithm might run over
-several months. It might have to stop and start for many reasons.
-This means that the platform should have the ability to persist
-algorithm activity in order to be reliable.
-
-In the interest of time, I will start by persisting algorithm
-activity in memory. Data will be lost when the algorithm execution stops.
-The intent it to offer a simple basis from which to implement data
-persistence strategies in the future.