Merge branch 'rc-0.1.dev7'

Normalize timestamps before comparison.

README.rst

@@ -1,196 +1 @@
-========
+All the documentation for `Catalyst <https://github.com/enigmampc/catalyst>`_ can be found in the `catalyst-docs wiki <https://github.com/enigmampc/catalyst-docs/wiki>`_.
-Catalyst
-========
-
-|version status|
-
-Catalyst is an algorithmic trading library for crypto-assets written in Python.
-It allows trading strategies to be easily expressed and backtested against historical data, providing analytics and insights regarding a particular strategy's performance.
-Catalyst will be expanded to support live-trading of crypto-assets in the coming months.
-Please visit `<enigma.co>`_ to learn about Catalyst, or refer to the 
-`whitepaper <https://www.enigma.co/enigma_catalyst.pdf>`_ for further technical details.
-
-Catalyst builds on top of the well-established `Zipline <https://github.com/quantopian/zipline>`_ project.
-We did our best to minimize structural changes to the general API to maximize compatibility with existing trading algorithms, developer knowledge, and tutorials.
-For now, please refer to the `Zipline API Docs <http://zipline.io>`_ as a general reference and bring any other questions you have to our #dev channel on `Slack <https://join.slack.com/enigmacatalyst/shared_invite/MTkzMjQ0MTg1NTczLTE0OTY3MjE3MDEtZGZmMTI5YzI3ZA>`_.
-
-Our primary contributions include the:
-
-- Introduction of an open trading calendar that permits simulation to allow trades on weekends, holidays, and outside of normal business hours.
-- Curation of OHLCV data bundle from `Poloniex's API <https://poloniex.com/support/api/>`_, which contains data in five-minute intervals as early as 2/19/2015.
-- Support for backtesting of daily trading strategies, support for five-minute backtesting is in development.
-- Addition of Bitcoin price (USDT_BTC) as a benchmark asset for comparing performance.
-
-Interested in getting involved?
-`Join us on Slack! <https://join.slack.com/enigmacatalyst/shared_invite/MTkzMjQ0MTg1NTczLTE0OTY3MjE3MDEtZGZmMTI5YzI3ZA>`_
-
-
-Installation
-============
-
-At the moment, Catalyst has some fairly specific and strict depedency requirements.
-We recommend the use of Python virtual environments if you wish to simplify the installation process, or otherwise isolate Catalyst's dependencies from your other projects.
-If you don't have ``virtualenv`` installed, see our later section on Virtual Environments.
-
-.. code-block:: bash
-
-    $ virtualenv catalyst-venv
-    $ source ./catalyst-venv/bin/activate
-    $ pip install enigma-catalyst
-
-**Note:** A successful installation will require several minutes in order to compile dependencies that expose C APIs.
-
-Dependencies
-------------
-
-Catalyst's depedencies can be found in the ``etc/requirements.txt`` file.
-If you need to install them outside of a typical ``pip install``, this is done using:
-
-.. code-block:: bash
-
-    $ pip install -r etc/requirements.txt
-
-Though not required by Catalyst directly, our example algorithms use matplotlib to visually display backtest results.
-If you wish to run any examples or use matplotlib during development, it can be installed using:
-
-.. code-block:: bash
-
-    $ pip install matplotlib
-
-**Note:** If you plan to use matplotlib and virtualenv on Mac OS X, see our later section for additional setup instructions.
-
-Getting Started
-===============
-
-The following code implements a simple buy and hold algorithm. The full source can be found in ``catalyst/examples/buy_and_hodl.py``.
-
-.. code:: python
-
-    import numpy as np
-    
-    from catalyst.api import (
-        order_target_value,
-        symbol,
-        record,
-        cancel_order,
-        get_open_orders,
-    )
-    
-    ASSET = 'USDT_BTC'
-    
-    TARGET_HODL_RATIO = 0.8
-    RESERVE_RATIO = 1.0 - TARGET_HODL_RATIO
-
-    def initialize(context):
-        context.is_buying = True
-        context.asset = symbol(ASSET)
-
-    def handle_data(context, data):
-        cash = context.portfolio.cash
-        target_hodl_value = TARGET_HODL_RATIO * context.portfolio.starting_cash
-        reserve_value = RESERVE_RATIO * context.portfolio.starting_cash
-        
-        # Cancel any outstanding orders from the previous day
-        orders = get_open_orders(context.asset) or []
-        for order in orders:
-            cancel_order(order)
-        
-        # Stop buying after passing reserve threshold
-        if cash <= reserve_value:
-            context.is_buying = False
-        
-        # Retrieve current price from pricing data
-        price = data[context.asset].price
-        
-        # Check if still buying and could (approximately) afford another purchase                    
-        if context.is_buying and cash > price:
-            # Place order to make position in asset equal to target_hodl_value
-            order_target_value(
-                context.asset,
-                target_hodl_value,
-                limit_price=1.1 * price,
-                stop_price=0.9 * price,
-            )
-        
-        # Record any state for later analysis
-        record(
-            price=price,
-            cash=context.portfolio.cash,
-            leverage=context.account.leverage,
-        )
-
-
-You can then run this algorithm using the Catalyst CLI. From the command
-line, run:
-
-.. code:: bash
-
-    $ catalyst ingest
-    $ catalyst run -f buy_and_hodl.py --start 2015-3-1 --end 2017-6-28 --capital-base 100000 -o bah.pickle
-
-This will download the crypto-asset price data from a poloniex bundle
-curated by Enigma in the specified time range and stream it through
-the algorithm and plot the resulting performance using matplotlib.
-
-You can find other examples in the ``catalyst/examples`` directory.
-
-Limitations
------------
-
-This project is currently in a pre-alpha state and has some limitations we'd like to address:
-
-- *Minimum Denomination:* The smallest tradable unit in Catalyst is equal to 1/1000th of a full coin. We plan to enable more granular increments, but have capped it at 1/1000th for the time being.
-- *Supported Assets:* Currently the poloniex bundle comes prepopulated with data for all 90 registered trading pairs. However, due to limitations in how portfolios are currently modeled, we recommend sticking to ``USDT_*`` trading pairs. USDT is an independent currency listed on Poloniex whose price is pegged to the US dollar. Currently, this list includes: ``USDT_BTC``, ``USDT_DASH``, ``USDT_ETC``, ``USDT_ETH``, ``USDT_LTC``, ``USDT_NXT``, ``USDT_REP``, ``USDT_STR``, ``USDT_XMR``, ``USDT_XRP``, and ``USDT_ZEC``. We plan to add support for basing your portfolio in arbitrary currencies and provide native support for modeling ForEx trades in the near future!
-
-Virtual Environments
-====================
-
-Here we will provide a brief tutorial for installing ``virtualenv`` and its basic usage.
-For more information regarding ``virtualenv``, please refer to this `virtualenv guide <http://python-guide-pt-br.readthedocs.io/en/latest/dev/virtualenvs/>`_.
-
-The ``virtualenv`` command can be installed using:
-
-.. code-block:: bash
-
-    $ pip install virtualenv
-
-To create a new virtual environment, choose a directory, e.g. ``/path/to/venv-dir``, where project-specific packages and files will be stored.  The environment is created by running:
-
-.. code-block:: bash
-
-    $ virtualenv /path/to/venv-dir
-
-To enter an environment, run the ``bin/activate`` script located in ``/path/to/venv-dir`` using:
-
-.. code-block:: bash
-
-    $ source /path/to/venv-dir/bin/activate
-
-Exiting an environment is accomplished using ``deactivate``, and removing it entirely is done by deleting ``/path/to/venv-dir``.
-
-OS X + virtualenv + matplotlib
--------------------------------------
-
-A note about using matplotlib in virtual enviroments on OS X: it may be necessary to run
-
-.. code-block:: python
-
-    echo "backend: TkAgg" > ~/.matplotlib/matplotlibrc
-
-in order to override the default ``macosx`` backend for your system, which may not be accessible from inside the virtual environment.
-This will allow Catalyst to open matplotlib charts from within a virtual environment, which is useful for displaying the performance of your backtests.  To learn more about matplotlib backends, please refer to the
-`matplotlib backend documentation <https://matplotlib.org/faq/usage_faq.html#what-is-a-backend>`_.
-
-Disclaimer
-==========
-
-Keep in mind that this project is still under active development, and is not recommended for production use in its current state.
-We are deeply committed to improving the overall user experience, reliability, and feature-set offered by Catalyst.
-If you have any suggestions, feedback, or general improvements regarding any of these topics, please let us know!
-
-Hello World,
-
-The Enigma Team
-
-.. |version status| image:: https://img.shields.io/pypi/pyversions/enigma-catalyst.svg
-   :target: https://pypi.python.org/pypi/enigma-catalyst

catalyst/algorithm.py

@@ -444,9 +444,6 @@ class TradingAlgorithm(object):
                     'data frequency: {}'.format(data_frequency)
                 )
 
-            print 'first_dates:', all_dates[:10]
-            print 'last_dates:', all_dates[:-10]
-
             self.engine = SimplePipelineEngine(
                 get_loader,
                 all_dates,

catalyst/data/bundles/base.py

@@ -172,6 +172,7 @@ class BaseBundle(object):
 
                 # Compile 5-minute symbol data if bundle supports 5-minute mode and
                 # persist the dataset to disk.
+                '''
                 if '5-minute' in self.frequencies:
                     five_minute_bar_writer.write(
                         self._fetch_symbol_iter(
@@ -187,6 +188,7 @@ class BaseBundle(object):
                         length=len(symbol_map),
                         show_progress=show_progress,
                     )
+                '''
 
                 # Compile minute symbol data if bundle supports minute mode and
                 # persist the dataset to disk.
@@ -296,14 +298,12 @@ class BaseBundle(object):
                     except Exception as e:
                         log.exception(
                             'Failed to load metadata from {}. '
-                            'Retrying.'.format(
+                            'Retrying.'.format(self.name)
-                                name=self.name,
-                            )
                         ) 
                 else:
                     raise ValueError(
-                        'Failed to download metadata page %d after %d '
+                        'Failed to download metadata page {} after {} '
-                        'attempts.'.format(page_number, retries),
+                        'attempts.'.format(page_number, retries)
                     )
 
 
@@ -313,7 +313,8 @@ class BaseBundle(object):
 
             # Apply selective asset filtering, useful for benchmark
             # ingestion.
-            raw = raw[raw.symbol.isin(self._asset_filter)]
+            if self._asset_filter:
+                raw = raw[raw.symbol.isin(self._asset_filter)]
 
             # Update cached value for key.
             cache[key] = raw

catalyst/data/bundles/core.py

@@ -90,13 +90,13 @@ def cache_relative(bundle_name, timestr, environ=None):
 
 
 def daily_relative(bundle_name, timestr, environ=None):
-    return bundle_name, timestr, 'daily.bcolz'
+    return bundle_name, timestr, 'daily_equities.bcolz'
 
 def five_minute_relative(bundle_name, timestr, environ=None):
     return bundle_name, timestr, 'five_minute.bcolz'
 
 def minute_relative(bundle_name, timestr, environ=None):
-    return bundle_name, timestr, 'minute.bcolz'
+    return bundle_name, timestr, 'minute_equities.bcolz'
 
 
 def asset_db_relative(bundle_name, timestr, environ=None, db_version=None):

catalyst/data/bundles/poloniex.py

@@ -36,7 +36,7 @@ class PoloniexBundle(BaseCryptoPricingBundle):
     def frequencies(self):
         return set((
             'daily',
-            '5-minute',
+            #'5-minute',
         ))
 
     @lazyval
@@ -103,7 +103,7 @@ class PoloniexBundle(BaseCryptoPricingBundle):
         )
         raw.set_index('date', inplace=True)
 
-        scale = 1000.0
+        scale = 1
         raw.loc[:, 'open'] /= scale
         raw.loc[:, 'high'] /= scale
         raw.loc[:, 'low'] /= scale
@@ -132,7 +132,7 @@ class PoloniexBundle(BaseCryptoPricingBundle):
                          data_frequency):
         period_map = {
             'daily': 86400,
-            '5-minute': 300,
+#            '5-minute': 300,
         }
 
         try:
@@ -155,4 +155,13 @@ class PoloniexBundle(BaseCryptoPricingBundle):
             query=urlencode(query_params),
         )
 
-register_bundle(PoloniexBundle, ['USDT_BTC'])
+''' 
+As a second parameter, you can pass an array of currency pairs 
+that will be processed as an asset_filter to only process that 
+subset of assets in the bundle, such as:
+register_bundle(PoloniexBundle, ['USDT_BTC',])
+
+For a production environment make sure to use (to bundle all pairs):
+register_bundle(PoloniexBundle)
+'''
+register_bundle(PoloniexBundle)

catalyst/data/loader.py

@@ -93,8 +93,8 @@ def has_data_for_dates(series_or_df, first_date, last_date):
     dts = series_or_df.index
     if not isinstance(dts, pd.DatetimeIndex):
         raise TypeError("Expected a DatetimeIndex, but got %s." % type(dts))
-    first, last = dts[[0, -1]]
+    first, last = dts[[0, -1]].tz_localize(None)
-    return (first <= first_date) and (last >= last_date)
+    return (first <= first_date.tz_localize(None)) and (last >= last_date.tz_localize(None))
 
 def load_crypto_market_data(trading_day=None,
                             trading_days=None,
@@ -134,19 +134,21 @@ def load_crypto_market_data(trading_day=None,
         trading_day,
         environ,
     )
+    # Override first_date for treasury data since we have it for many more years
+    # and is independent of crypto data
+    first_date_treasury = pd.Timestamp('1990-01-01', tz='UTC') 
     tc = ensure_treasury_data(
         bm_symbol,
-        first_date,
+        first_date_treasury,
         last_date,
         now,
         environ,
     )
     benchmark_returns = br[br.index.slice_indexer(first_date, last_date)]
-    treasury_curves = tc[tc.index.slice_indexer(first_date, last_date)]
+    treasury_curves = tc[tc.index.slice_indexer(first_date_treasury, last_date)]
     return benchmark_returns, treasury_curves
 
 
-
 def load_market_data(trading_day=None, trading_days=None, bm_symbol='SPY',
                      environ=None):
     """
@@ -232,6 +234,7 @@ def load_market_data(trading_day=None, trading_days=None, bm_symbol='SPY',
     treasury_curves = tc[tc.index.slice_indexer(first_date, last_date)]
     return benchmark_returns, treasury_curves
 
+
 def ensure_crypto_benchmark_data(symbol,
                                  first_date,
                                  last_date,
@@ -279,7 +282,7 @@ def ensure_crypto_benchmark_data(symbol,
             None,
             symbol,
             get_calendar(bundle.calendar_name),
-            first_date,
+            first_date - trading_day,
             last_date,
             'daily',
         )
@@ -364,6 +367,7 @@ def ensure_benchmark_data(symbol, first_date, last_date, now, trading_day,
         logger.warn("Still don't have expected data after redownload!")
     return data
 
+
 def ensure_benchmark_data(symbol, first_date, last_date, now, trading_day,
                           environ=None):
     """
@@ -478,11 +482,6 @@ def ensure_treasury_data(symbol, first_date, last_date, now, environ=None):
 
 def _load_cached_data(filename, first_date, last_date, now, resource_name,
                       environ=None):
-    if resource_name == 'benchmark':
-        from_csv = pd.Series.from_csv
-    else:
-        from_csv = pd.DataFrame.from_csv
-
     # Path for the cache.
     path = get_data_filepath(filename, environ)
 
@@ -490,8 +489,10 @@ def _load_cached_data(filename, first_date, last_date, now, resource_name,
     # yet, so don't try to read from 'path'.
     if os.path.exists(path):
         try:
-            data = from_csv(path)
+            data = pd.DataFrame.from_csv(path)
-            data.index = pd.to_datetime(data.index).tz_localize('UTC')
+            if data.empty:
+                raise ValueError("File is empty.")
+            data.index = pd.to_datetime(data.index, infer_datetime_format=True, errors='coerce' ).tz_localize('UTC')
             if has_data_for_dates(data, first_date, last_date):
                 return data
 

catalyst/finance/risk/risk.py

@@ -158,7 +158,7 @@ def choose_treasury(select_treasury, treasury_curves, start_session,
                 )
                 break
 
-        if search_day:
+        if search_day and trading_calendar.name != 'OPEN':          # Supress warning for 'OPEN' calendar
             if (search_dist is None or search_dist > 1) and \
                     search_days[0] <= end_session <= search_days[-1]:
                 message = "No rate within 1 trading day of end date = \

catalyst/pipeline/loaders/crypto_pricing_loader.py

@@ -45,7 +45,7 @@ class CryptoPricingLoader(PipelineLoader):
             reader = bundle.five_minute_bar_reader
             all_sessions = cal.all_five_minutes
 
-        elif daily_bar_reader == 'minute':
+        elif data_frequency == 'minute':
             reader = bundle.minute_bar_reader
             all_sessions = cal.all_minutes
 
@@ -106,12 +106,6 @@ class CryptoPricingLoader(PipelineLoader):
 
 
 def _shift_dates(dates, start_date, end_date, shift):
-    print 'dates.head:\n', dates[:10]
-    print 'dates.tail:\n', dates[:-10]
-
-    print 'start_date:', start_date
-    print 'end_date:', end_date
-    print 'shift:', shift
 
     try:
         start = dates.get_loc(start_date)

catalyst/sources/benchmark_source.py

@@ -51,10 +51,7 @@ class BenchmarkSource(object):
         elif benchmark_returns is not None:
             daily_series = benchmark_returns[sessions[0]:sessions[-1]]
 
-            print 'BENCHMARK_RETURNS'
-
             if self.emission_rate == "minute":
-                print 'BENCHMARK_RETURNS minute'
                 # we need to take the env's benchmark returns, which are daily,
                 # and resample them to minute
                 minutes = trading_calendar.minutes_for_sessions_in_range(
@@ -69,7 +66,6 @@ class BenchmarkSource(object):
 
                 self._precalculated_series = minute_series
             elif self.emission_rate == '5-minute':
-                print 'BENCHMARK_RETURNS 5-minute'
                 five_minutes = \
                     trading_calendar.five_minutes_for_sessions_in_range(
                         sessions[0],
@@ -83,7 +79,6 @@ class BenchmarkSource(object):
                 
                 self._precalculated_series = five_minute_series
             else:
-                print 'BENCHMARK_RETURNS daily'
                 self._precalculated_series = daily_series
         else:
             raise Exception("Must provide either benchmark_asset or "
@@ -190,7 +185,6 @@ class BenchmarkSource(object):
 
             return benchmark_series.pct_change()[1:]
         else:
-            print '----------------------------------------'
             start_date = asset.start_date
             if start_date < trading_days[0]:
                 # get the window of close prices for benchmark_asset from the

catalyst/utils/calendars/exchange_calendar_open.py

@@ -1,6 +1,7 @@
 from datetime import time
 from pytz import timezone
 
+from pandas import Timestamp
 from pandas.tseries.offsets import DateOffset
 
 from catalyst.utils.memoize import lazyval
@@ -28,3 +29,6 @@ class OpenExchangeCalendar(TradingCalendar):
     @lazyval
     def day(self):
         return DateOffset(days=1)
+
+    def __init__(self, *args, **kwargs):
+        super(OpenExchangeCalendar, self).__init__(start=Timestamp('2015-03-01', tz='UTC'), **kwargs)