From 15aadbb10119294573c6c025a567e6dd4e3a24fc Mon Sep 17 00:00:00 2001 From: izokay Date: Mon, 29 Jan 2018 17:14:15 -0500 Subject: [PATCH 01/87] documentation for PyCharm --- docs/source/beginner-tutorial.rst | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/docs/source/beginner-tutorial.rst b/docs/source/beginner-tutorial.rst index 3bcfc4cf..2ca5d283 100644 --- a/docs/source/beginner-tutorial.rst +++ b/docs/source/beginner-tutorial.rst @@ -16607,7 +16607,30 @@ NaN +Catalyst using PyCharm +~~~~~~~~~~~~~~~~~~~~~~ +Install +^^^^^^^ +Install PyCharm from their 'website '_ . + +Setup +^^^^^ + +1. Create a new project folder for your scripts or open your existing folder in PyCharm. + +2. Once your project is open, go to File -> Settings -> Project:'NAME_OF_PROJECT' -> Project Interpreter. + Click the gear box next to the project interpreter and select 'add local'. + + Then select 'Conda Environment' -> 'Existing environment'. Click the '...' button and + navigate to your catalyst env located in the Anaconda2 folder to select the 'python.exe' file + (for example: 'C:\Users\user\Anaconda2\envs\catalyst\python.exe'). Select OK, then apply and click OK again. + +3. Next, click on the dropdown menu on the top right of PyCharm and select 'Edit Configurations'. + Select the '+' button. + Set the script Path to the location of your script and make sure the interpreter is correct, then hit ok. + +You should now be able to run your script in PyCharm. Next steps ~~~~~~~~~~ From 929e0a9a016235aa71b86d08b286134064625df9 Mon Sep 17 00:00:00 2001 From: izokay Date: Mon, 29 Jan 2018 17:18:16 -0500 Subject: [PATCH 02/87] Update beginner-tutorial.rst --- docs/source/beginner-tutorial.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/beginner-tutorial.rst b/docs/source/beginner-tutorial.rst index 2ca5d283..7a918478 100644 --- a/docs/source/beginner-tutorial.rst +++ b/docs/source/beginner-tutorial.rst @@ -16612,7 +16612,7 @@ Catalyst using PyCharm Install ^^^^^^^ -Install PyCharm from their 'website '_ . +Install PyCharm from their ('website ') . Setup ^^^^^ From 1cadaf1ed0936de4489c9a6a7e6657e6bdd76254 Mon Sep 17 00:00:00 2001 From: izokay Date: Mon, 29 Jan 2018 17:19:44 -0500 Subject: [PATCH 03/87] Update beginner-tutorial.rst --- docs/source/beginner-tutorial.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/beginner-tutorial.rst b/docs/source/beginner-tutorial.rst index 7a918478..a77b9d1b 100644 --- a/docs/source/beginner-tutorial.rst +++ b/docs/source/beginner-tutorial.rst @@ -16612,7 +16612,7 @@ Catalyst using PyCharm Install ^^^^^^^ -Install PyCharm from their ('website ') . +('Install PyCharm from their website '_ .) Setup ^^^^^ From 5417881a721182d160cea5995000190e659b1ca7 Mon Sep 17 00:00:00 2001 From: izokay Date: Mon, 29 Jan 2018 17:21:53 -0500 Subject: [PATCH 04/87] Update beginner-tutorial.rst --- docs/source/beginner-tutorial.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/beginner-tutorial.rst b/docs/source/beginner-tutorial.rst index a77b9d1b..43084cd3 100644 --- a/docs/source/beginner-tutorial.rst +++ b/docs/source/beginner-tutorial.rst @@ -16612,7 +16612,7 @@ Catalyst using PyCharm Install ^^^^^^^ -('Install PyCharm from their website '_ .) +Install PyCharm from their 'website'_ . Setup ^^^^^ From 5b77f5eebaa7fe16894fe8eb4cc4ca51ea19b50f Mon Sep 17 00:00:00 2001 From: izokay Date: Mon, 29 Jan 2018 17:23:00 -0500 Subject: [PATCH 05/87] Update beginner-tutorial.rst --- docs/source/beginner-tutorial.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/beginner-tutorial.rst b/docs/source/beginner-tutorial.rst index 43084cd3..c02febe4 100644 --- a/docs/source/beginner-tutorial.rst +++ b/docs/source/beginner-tutorial.rst @@ -16612,7 +16612,7 @@ Catalyst using PyCharm Install ^^^^^^^ -Install PyCharm from their 'website'_ . +Install PyCharm from their 'website ' . Setup ^^^^^ From e9a3bcf3e9002203f361eea6edbe0f08bcb991ec Mon Sep 17 00:00:00 2001 From: izokay Date: Mon, 29 Jan 2018 17:23:22 -0500 Subject: [PATCH 06/87] Update beginner-tutorial.rst --- docs/source/beginner-tutorial.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/beginner-tutorial.rst b/docs/source/beginner-tutorial.rst index c02febe4..e8e0252e 100644 --- a/docs/source/beginner-tutorial.rst +++ b/docs/source/beginner-tutorial.rst @@ -16612,7 +16612,7 @@ Catalyst using PyCharm Install ^^^^^^^ -Install PyCharm from their 'website ' . +Install PyCharm from their 'website '__ . Setup ^^^^^ From 426fde40b1b3331ba52905744474f05c4660805b Mon Sep 17 00:00:00 2001 From: izokay Date: Mon, 29 Jan 2018 17:24:21 -0500 Subject: [PATCH 07/87] Update beginner-tutorial.rst --- docs/source/beginner-tutorial.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/beginner-tutorial.rst b/docs/source/beginner-tutorial.rst index e8e0252e..82da1900 100644 --- a/docs/source/beginner-tutorial.rst +++ b/docs/source/beginner-tutorial.rst @@ -16612,7 +16612,7 @@ Catalyst using PyCharm Install ^^^^^^^ -Install PyCharm from their 'website '__ . +Install PyCharm from their `Website `__. Setup ^^^^^ From 4a4e32846b7c3ddc5eca0b7ed4a2d3b265ae41a5 Mon Sep 17 00:00:00 2001 From: izokay Date: Mon, 29 Jan 2018 17:27:14 -0500 Subject: [PATCH 08/87] Update beginner-tutorial.rst --- docs/source/beginner-tutorial.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/beginner-tutorial.rst b/docs/source/beginner-tutorial.rst index 82da1900..0c2a8564 100644 --- a/docs/source/beginner-tutorial.rst +++ b/docs/source/beginner-tutorial.rst @@ -16624,7 +16624,7 @@ Setup Then select 'Conda Environment' -> 'Existing environment'. Click the '...' button and navigate to your catalyst env located in the Anaconda2 folder to select the 'python.exe' file - (for example: 'C:\Users\user\Anaconda2\envs\catalyst\python.exe'). Select OK, then apply and click OK again. + (for example: 'C:"\\"Users"\\"user"\\"Anaconda2"\\"envs"\\"catalyst"\\"python.exe'). Select OK, then apply and click OK again. 3. Next, click on the dropdown menu on the top right of PyCharm and select 'Edit Configurations'. Select the '+' button. From cbae2465d2de5651c96892657598f4cb263dc733 Mon Sep 17 00:00:00 2001 From: izokay Date: Mon, 29 Jan 2018 17:27:58 -0500 Subject: [PATCH 09/87] Update beginner-tutorial.rst --- docs/source/beginner-tutorial.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/beginner-tutorial.rst b/docs/source/beginner-tutorial.rst index 0c2a8564..8cad3596 100644 --- a/docs/source/beginner-tutorial.rst +++ b/docs/source/beginner-tutorial.rst @@ -16624,7 +16624,7 @@ Setup Then select 'Conda Environment' -> 'Existing environment'. Click the '...' button and navigate to your catalyst env located in the Anaconda2 folder to select the 'python.exe' file - (for example: 'C:"\\"Users"\\"user"\\"Anaconda2"\\"envs"\\"catalyst"\\"python.exe'). Select OK, then apply and click OK again. + (for example: 'C:\\Users\\user\\Anaconda2\\envs\\catalyst\\python.exe'). Select OK, then apply and click OK again. 3. Next, click on the dropdown menu on the top right of PyCharm and select 'Edit Configurations'. Select the '+' button. From 1f29f7fdf142057df7a9fee55b938c6806067381 Mon Sep 17 00:00:00 2001 From: izokay Date: Mon, 29 Jan 2018 17:31:32 -0500 Subject: [PATCH 10/87] Update beginner-tutorial.rst --- docs/source/beginner-tutorial.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/beginner-tutorial.rst b/docs/source/beginner-tutorial.rst index 8cad3596..c2a037d2 100644 --- a/docs/source/beginner-tutorial.rst +++ b/docs/source/beginner-tutorial.rst @@ -16628,7 +16628,7 @@ Setup 3. Next, click on the dropdown menu on the top right of PyCharm and select 'Edit Configurations'. Select the '+' button. - Set the script Path to the location of your script and make sure the interpreter is correct, then hit ok. + Set the script Path to the path of your script and make sure the interpreter is correct, then hit ok. You should now be able to run your script in PyCharm. From 262dffd4bf9101571458c561a73d4f5a70ef5e90 Mon Sep 17 00:00:00 2001 From: westurner <@westurner> Date: Fri, 9 Feb 2018 01:46:44 -0500 Subject: [PATCH 11/87] DOC: live-trading.rst: add newline before ul --- docs/source/live-trading.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/source/live-trading.rst b/docs/source/live-trading.rst index a2898d61..d0d7f721 100644 --- a/docs/source/live-trading.rst +++ b/docs/source/live-trading.rst @@ -38,6 +38,7 @@ that you should test any new algorithm. Once you are confident with the simulations that you have obtained with your algorithm in backtesting, you may switch to live trading, where you have two different modes: + * *Paper Trading*: The simulated algorithm runs in real time, and fetches pricing data in real time from the exchange, but the orders never reach the exchange, and are instead kept within Catalyst and simulated. No real currency From a8cfcead70cab6d276adfefacbdfd0c9af1a3a81 Mon Sep 17 00:00:00 2001 From: Victor Grau Serrat Date: Fri, 9 Feb 2018 10:13:52 -0800 Subject: [PATCH 12/87] DOC: live-trading: improved upon #221 --- docs/source/live-trading.rst | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/docs/source/live-trading.rst b/docs/source/live-trading.rst index 7576b798..cb881a76 100644 --- a/docs/source/live-trading.rst +++ b/docs/source/live-trading.rst @@ -30,8 +30,8 @@ Paper Trading vs Live Trading modes ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Catalyst currently supports three different modes in which you can execute your -trading algorithm. The first is backtesting, which is covered extensively in the -tutorial, and uses historical data to run your algorithm. There is no +trading algorithm. The first is **backtesting**, which is covered extensively in +the tutorial, and uses historical data to run your algorithm. There is no interaction with the exchange in backtesting mode, and this is the first mode that you should test any new algorithm. @@ -39,14 +39,15 @@ Once you are confident with the simulations that you have obtained with your algorithm in backtesting, you may switch to live trading, where you have two different modes: -* *Paper Trading*: The simulated algorithm runs in real time, and fetches -pricing data in real time from the exchange, but the orders never reach the -exchange, and are instead kept within Catalyst and simulated. No real currency -is bought or sold. Think of it as a `backtesting happening in real time`. -* *Live Trading*: This is the proper live trading mode in which an algorithm -runs in real time, fetching pricing data from live exchanges and placing orders -against the exchange. Real currency is transacted on the exchange driven by the -algorithm. +* **Paper Trading**: The simulated algorithm runs in real time, and fetches + pricing data in real time from the exchange, but the orders never reach the + exchange, and are instead kept within Catalyst and simulated. No real currency + is bought or sold. Think of it as a `backtesting happening in real time`. + +* **Live Trading**: This is the proper live trading mode in which an algorithm + runs in real time, fetching pricing data from live exchanges and placing + orders against the exchange. Real currency is transacted on the exchange + driven by the algorithm. These three modes are controlled by the following variables: From 2c1f81016366a1541c9eafe9d470d3a1b3037c2d Mon Sep 17 00:00:00 2001 From: Victor Grau Serrat Date: Fri, 9 Feb 2018 11:10:32 -0800 Subject: [PATCH 13/87] DOC: pycharm: improved upon #195 --- docs/source/beginner-tutorial.rst | 45 ++++++++++++++++++++++--------- 1 file changed, 32 insertions(+), 13 deletions(-) diff --git a/docs/source/beginner-tutorial.rst b/docs/source/beginner-tutorial.rst index 43cda572..41afe3c0 100644 --- a/docs/source/beginner-tutorial.rst +++ b/docs/source/beginner-tutorial.rst @@ -16609,30 +16609,49 @@ NaN -Catalyst using PyCharm -~~~~~~~~~~~~~~~~~~~~~~ +PyCharm IDE +~~~~~~~~~~~ + +PyCharm is an Integrated Development Environment (IDE) used in computer +programming, specifically for the Python language. It streamlines the continuos +development of Python code, and among other things includes a debugger that +comes in handy to see the inner workings of Catalyst, and your trading +algorithms. Install ^^^^^^^ -Install PyCharm from their `Website `__. +Install PyCharm from their `Website `__. +There is a free and open-source **Community** version. Setup ^^^^^ -1. Create a new project folder for your scripts or open your existing folder in PyCharm. +1. When creating a new project in PyCharm, right under you specify the Location, + click on **Project Interpreter** to display a drop down menu -2. Once your project is open, go to File -> Settings -> Project:'NAME_OF_PROJECT' -> Project Interpreter. - Click the gear box next to the project interpreter and select 'add local'. +2. Select **Existing interpreter**, click the gear box right next to it and + select 'add local'. Depending on your installation, select either + "*Virtual Environemnt*" or "*Conda Environment" and click the '...' button to + navigate to your catalyst env and select the Python binary file: + ``bin/python`` for Linux/MacOS installations or 'python.exe' for Windows + installs (for example: 'C:\\Users\\user\\Anaconda2\\envs\\catalyst\\python.exe'). + Select OK. You may want to click on *Make available to all projects* for your + future reference. Click OK again, and create your new environment using the + set up of your virtual environment. - Then select 'Conda Environment' -> 'Existing environment'. Click the '...' button and - navigate to your catalyst env located in the Anaconda2 folder to select the 'python.exe' file - (for example: 'C:\\Users\\user\\Anaconda2\\envs\\catalyst\\python.exe'). Select OK, then apply and click OK again. +Alternatively, if you already have your project created, in Windows do: -3. Next, click on the dropdown menu on the top right of PyCharm and select 'Edit Configurations'. - Select the '+' button. - Set the script Path to the path of your script and make sure the interpreter is correct, then hit ok. +1. File -> Default Settings -> Project Interpreter. Click the gear box next to + the project interpreter and select ‘add local’, and follow the steps from the + second step above. -You should now be able to run your script in PyCharm. +On MacOS: + +1. PyCharm -> Preferences -> Settings -> Project:’NAME_OF_PROJECT’ -> + Project Interpreter. Click the gear box next to the project interpreter + and select ‘add local’, and follow the steps from the second step above. + +You should now be able to run your project/scripts in PyCharm. Next steps ~~~~~~~~~~ From f35444fabc4d1c6763e0eb3371fe23bd565398a0 Mon Sep 17 00:00:00 2001 From: Victor Grau Serrat Date: Fri, 9 Feb 2018 15:16:14 -0800 Subject: [PATCH 14/87] DOC: updated algo as per #201 --- catalyst/examples/portfolio_optimization.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/catalyst/examples/portfolio_optimization.py b/catalyst/examples/portfolio_optimization.py index 37f8a55d..26c957b6 100644 --- a/catalyst/examples/portfolio_optimization.py +++ b/catalyst/examples/portfolio_optimization.py @@ -146,4 +146,5 @@ if __name__ == '__main__': start=start, end=end, exchange_name='poloniex', - capital_base=100000, ) + capital_base=100000, + base_currency='usdt', ) From e9745be0e883157964c9ba816fff532e2cefc1f2 Mon Sep 17 00:00:00 2001 From: AvishaiW Date: Sun, 11 Feb 2018 13:53:39 +0200 Subject: [PATCH 15/87] DOC: added use of end_date & start_date in live mode. issues #225 & #213 --- docs/source/live-trading.rst | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/docs/source/live-trading.rst b/docs/source/live-trading.rst index cb881a76..f2b6873a 100644 --- a/docs/source/live-trading.rst +++ b/docs/source/live-trading.rst @@ -176,6 +176,13 @@ Here is the breakdown of the new arguments: - ``simulate_orders``: Enables the paper trading mode, in which orders are simulated in Catalyst instead of processed on the exchange. It defaults to ``True``. +- ``end_date``: When setting the end_date to a time in the **future**, + it will schedule the live algo to finish gracefully at the specified date. +- ``start_date``: (**Will be implemented in the future**) + The live algo starts by default in the present, as mentioned above. + by setting the start_date to a time in the future, the algorithm would + essentially sleep and when the predefined time comes, it would start executing. + Here is a complete algorithm for reference: `Buy Low and Sell High `_ From 5345bfc43d3eb57b64a4c547794ffbc66e8b0bcf Mon Sep 17 00:00:00 2001 From: Frederic Fortier Date: Mon, 12 Feb 2018 13:59:19 -0500 Subject: [PATCH 16/87] BLD: remove extra candle at the beginning of history bars after resampling --- catalyst/exchange/exchange_data_portal.py | 8 +++-- catalyst/exchange/utils/exchange_utils.py | 9 +++--- .../exchange/test_suites/test_suite_bundle.py | 30 +++++++++++-------- 3 files changed, 28 insertions(+), 19 deletions(-) diff --git a/catalyst/exchange/exchange_data_portal.py b/catalyst/exchange/exchange_data_portal.py index 8f9665dc..c6523326 100644 --- a/catalyst/exchange/exchange_data_portal.py +++ b/catalyst/exchange/exchange_data_portal.py @@ -9,8 +9,9 @@ from catalyst.exchange.exchange_bundle import ExchangeBundle from catalyst.exchange.exchange_errors import ( ExchangeRequestError, PricingDataNotLoadedError) -from catalyst.exchange.utils.exchange_utils import resample_history_df, group_assets_by_exchange -from catalyst.exchange.utils.datetime_utils import get_frequency +from catalyst.exchange.utils.exchange_utils import resample_history_df, \ + group_assets_by_exchange +from catalyst.exchange.utils.datetime_utils import get_frequency, get_start_dt from logbook import Logger from redo import retry @@ -311,7 +312,8 @@ class DataPortalExchangeBacktest(DataPortalExchangeBase): algo_end_dt=self._last_available_session, ) - df = resample_history_df(pd.DataFrame(series), freq, field) + start_dt = get_start_dt(end_dt, adj_bar_count, data_frequency) + df = resample_history_df(pd.DataFrame(series), freq, field, start_dt) return df def get_exchange_spot_value(self, diff --git a/catalyst/exchange/utils/exchange_utils.py b/catalyst/exchange/utils/exchange_utils.py index 132845bb..ac48728e 100644 --- a/catalyst/exchange/utils/exchange_utils.py +++ b/catalyst/exchange/utils/exchange_utils.py @@ -126,8 +126,8 @@ def get_exchange_symbols(exchange_name, is_local=False, environ=None): filename = get_exchange_symbols_filename(exchange_name, is_local) if not is_local and (not os.path.isfile(filename) or pd.Timedelta( - pd.Timestamp('now', tz='UTC') - last_modified_time( - filename)).days > 1): + pd.Timestamp('now', tz='UTC') - last_modified_time( + filename)).days > 1): try: download_exchange_symbols(exchange_name, environ) except Exception as e: @@ -512,7 +512,7 @@ def get_common_assets(exchanges): return assets -def resample_history_df(df, freq, field): +def resample_history_df(df, freq, field, start_dt=None): """ Resample the OHCLV DataFrame using the specified frequency. @@ -541,6 +541,7 @@ def resample_history_df(df, freq, field): raise ValueError('Invalid field.') resampled_df = df.resample(freq, closed='left', label='left').agg(agg) + resampled_df = resampled_df[resampled_df.index >= start_dt] return resampled_df @@ -567,7 +568,7 @@ def mixin_market_params(exchange_name, params, market): params['taker'] = 0.002 elif 'maker' in market and 'taker' in market \ - and market['maker'] is not None and market['taker'] is not None: + and market['maker'] is not None and market['taker'] is not None: params['maker'] = market['maker'] params['taker'] = market['taker'] diff --git a/tests/exchange/test_suites/test_suite_bundle.py b/tests/exchange/test_suites/test_suite_bundle.py index 15d9cbcd..0bf8b1d2 100644 --- a/tests/exchange/test_suites/test_suite_bundle.py +++ b/tests/exchange/test_suites/test_suite_bundle.py @@ -37,7 +37,7 @@ class TestSuiteBundle: return data_portal def compare_bundle_with_exchange(self, exchange, assets, end_dt, bar_count, - freq, data_frequency, data_portal): + freq, data_frequency, data_portal, field): """ Creates DataFrames from the bundle and exchange for the specified data set. @@ -62,8 +62,8 @@ class TestSuiteBundle: with log_catcher: symbols = [asset.symbol for asset in assets] print( - 'comparing data for {}/{} with {} timeframe until {}'.format( - exchange.name, symbols, freq, end_dt + 'comparing {} for {}/{} with {} timeframe until {}'.format( + field, exchange.name, symbols, freq, end_dt ) ) data['bundle'] = data_portal.get_history_window( @@ -71,13 +71,13 @@ class TestSuiteBundle: end_dt=end_dt, bar_count=bar_count, frequency=freq, - field='close', + field=field, data_frequency=data_frequency, ) set_print_settings() print( - 'the bundle first / last row:\n{}'.format( - data['bundle'].iloc[[-1, 0]] + 'the bundle data:\n{}'.format( + data['bundle'] ) ) candles = exchange.get_candles( @@ -88,14 +88,14 @@ class TestSuiteBundle: ) data['exchange'] = get_candles_df( candles=candles, - field='close', + field=field, freq=freq, bar_count=bar_count, end_dt=end_dt, ) print( - 'the exchange first / last row:\n{}'.format( - data['exchange'].iloc[[-1, 0]] + 'the exchange data:\n{}'.format( + data['exchange'] ) ) for source in data: @@ -118,8 +118,10 @@ class TestSuiteBundle: check_less_precise=min([a.decimals for a in assets]), ) except Exception as e: - print('Some differences were found within a 1 decimal point ' - 'interval of confidence: {}'.format(e)) + print( + 'Some differences were found within a 1 decimal point ' + 'interval of confidence: {}'.format(e) + ) with open(os.path.join(folder, 'compare.txt'), 'w+') as handle: handle.write(e.args[0]) @@ -203,8 +205,11 @@ class TestSuiteBundle: frequencies = exchange.get_candle_frequencies(data_frequency) freq = random.sample(frequencies, 1)[0] + rnd = random.SystemRandom() + # field = rnd.choice(['open', 'high', 'low', 'close', 'volume']) + field = rnd.choice(['close']) - bar_count = random.randint(1, 10) + bar_count = random.randint(3, 6) assets = select_random_assets( exchange.assets, asset_population @@ -229,6 +234,7 @@ class TestSuiteBundle: freq=freq, data_frequency=data_frequency, data_portal=data_portal, + field=field, ) pass From d64d6f191f5a7c7165ef626c6d1460377bdad1ac Mon Sep 17 00:00:00 2001 From: Frederic Fortier Date: Mon, 12 Feb 2018 16:44:25 -0500 Subject: [PATCH 17/87] BLD: upgraded CCXT --- etc/python2.7-environment.yml | 2 +- etc/requirements.txt | 2 +- tests/exchange/test_suites/test_suite_bundle.py | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/etc/python2.7-environment.yml b/etc/python2.7-environment.yml index 4f06bbae..0aa916d6 100644 --- a/etc/python2.7-environment.yml +++ b/etc/python2.7-environment.yml @@ -20,7 +20,7 @@ dependencies: - bcolz==0.12.1 - bottleneck==1.2.1 - chardet==3.0.4 - - ccxt==1.10.1049 + - ccxt==1.10.1094 - web3==4.0.0b7 - requests-toolbelt==0.8.0 - click==6.7 diff --git a/etc/requirements.txt b/etc/requirements.txt index 5d890d2e..bac9666d 100644 --- a/etc/requirements.txt +++ b/etc/requirements.txt @@ -81,7 +81,7 @@ empyrical==0.2.1 tables==3.3.0 #Catalyst dependencies -ccxt==1.10.1049 +ccxt==1.10.1094 boto3==1.4.8 redo==1.6 web3==4.0.0b7 diff --git a/tests/exchange/test_suites/test_suite_bundle.py b/tests/exchange/test_suites/test_suite_bundle.py index 0bf8b1d2..d338045d 100644 --- a/tests/exchange/test_suites/test_suite_bundle.py +++ b/tests/exchange/test_suites/test_suite_bundle.py @@ -207,7 +207,7 @@ class TestSuiteBundle: freq = random.sample(frequencies, 1)[0] rnd = random.SystemRandom() # field = rnd.choice(['open', 'high', 'low', 'close', 'volume']) - field = rnd.choice(['close']) + field = rnd.choice(['volume']) bar_count = random.randint(3, 6) From 67400f048bc717ad3b5e14430c6be16da41b1b43 Mon Sep 17 00:00:00 2001 From: Frederic Fortier Date: Mon, 12 Feb 2018 22:39:03 -0500 Subject: [PATCH 18/87] BLD: fix issue with removing extra candle in resampling --- catalyst/exchange/utils/exchange_utils.py | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/catalyst/exchange/utils/exchange_utils.py b/catalyst/exchange/utils/exchange_utils.py index ac48728e..3f8d6121 100644 --- a/catalyst/exchange/utils/exchange_utils.py +++ b/catalyst/exchange/utils/exchange_utils.py @@ -540,8 +540,16 @@ def resample_history_df(df, freq, field, start_dt=None): else: raise ValueError('Invalid field.') - resampled_df = df.resample(freq, closed='left', label='left').agg(agg) - resampled_df = resampled_df[resampled_df.index >= start_dt] + resampled_df = df.resample( + freq, closed='left', label='left' + ).agg(agg) # type: pd.DataFrame + + # Because the samples are closed left, we get one more candle at + # the beginning then the requested number for bars. Removing this + # candle to avoid confusion. + if start_dt and not resampled_df.empty: + resampled_df = resampled_df[resampled_df.index >= start_dt] + return resampled_df From f45d673d94d0c19ed583998a0a80415978ec950b Mon Sep 17 00:00:00 2001 From: Frederic Fortier Date: Tue, 13 Feb 2018 00:11:20 -0500 Subject: [PATCH 19/87] BUG: fixed issue #227 by allowing H frequency --- catalyst/exchange/utils/datetime_utils.py | 14 +++++--------- 1 file changed, 5 insertions(+), 9 deletions(-) diff --git a/catalyst/exchange/utils/datetime_utils.py b/catalyst/exchange/utils/datetime_utils.py index 2a8cb886..5f3a2e04 100644 --- a/catalyst/exchange/utils/datetime_utils.py +++ b/catalyst/exchange/utils/datetime_utils.py @@ -302,16 +302,12 @@ def get_frequency(freq, data_frequency=None): elif unit.lower() == 'm' or unit == 'T': unit = 'T' alias = '{}T'.format(candle_size) + data_frequency = 'minute' - if data_frequency == 'daily': - data_frequency = 'minute' - - # elif unit.lower() == 'h': - # candle_size = candle_size * 60 - # - # alias = '{}T'.format(candle_size) - # if data_frequency == 'daily': - # data_frequency = 'minute' + elif unit.lower() == 'h': + candle_size = candle_size * 60 + alias = '{}T'.format(candle_size) + data_frequency = 'minute' else: raise InvalidHistoryFrequencyAlias(freq=freq) From 89bf6742e36f72d1195f5e18081c9d147aad7c5b Mon Sep 17 00:00:00 2001 From: lenak25 Date: Tue, 13 Feb 2018 16:57:03 +0200 Subject: [PATCH 20/87] MAINT: remove unnecessary Binance exclusion code --- catalyst/exchange/ccxt/ccxt_exchange.py | 25 +++++++++---------------- 1 file changed, 9 insertions(+), 16 deletions(-) diff --git a/catalyst/exchange/ccxt/ccxt_exchange.py b/catalyst/exchange/ccxt/ccxt_exchange.py index c5fee495..28a90a69 100644 --- a/catalyst/exchange/ccxt/ccxt_exchange.py +++ b/catalyst/exchange/ccxt/ccxt_exchange.py @@ -426,25 +426,18 @@ class CCXT(Exchange): ) if start_dt is None: - # TODO: determine why binance is failing - if end_dt is None and self.name not in ['binance']: + if end_dt is None: end_dt = pd.Timestamp.utcnow() - if end_dt is not None: - dt_range = get_periods_range( - end_dt=end_dt, - periods=bar_count, - freq=freq, - ) - start_dt = dt_range[0] + dt_range = get_periods_range( + end_dt=end_dt, + periods=bar_count, + freq=freq, + ) + start_dt = dt_range[0] - if start_dt is not None: - # Convert out start date to a UNIX timestamp, then translate to - # milliseconds - delta = start_dt - get_epoch() - since = int(delta.total_seconds()) * 1000 - else: - since = None + delta = start_dt - get_epoch() + since = int(delta.total_seconds()) * 1000 candles = dict() for index, asset in enumerate(assets): From 6812aa0c5bf31c1341432560d1e0eb10a2980916 Mon Sep 17 00:00:00 2001 From: AvishaiW Date: Tue, 13 Feb 2018 19:39:38 +0200 Subject: [PATCH 21/87] BLD: added password into credentials needed for exchanges such as 'gdax' --- catalyst/exchange/utils/factory.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/catalyst/exchange/utils/factory.py b/catalyst/exchange/utils/factory.py index 77b2d708..67294f95 100644 --- a/catalyst/exchange/utils/factory.py +++ b/catalyst/exchange/utils/factory.py @@ -33,6 +33,8 @@ def get_exchange(exchange_name, base_currency=None, must_authenticate=False, exchange_name=exchange_name, key=exchange_auth['key'], secret=exchange_auth['secret'], + password=exchange_auth['password'] if 'password' + in exchange_auth.keys() else '', base_currency=base_currency, ) exchange_cache[key] = exchange From f3183b267aa4fcaeaa3ff48d38e26751a433ff80 Mon Sep 17 00:00:00 2001 From: AvishaiW Date: Tue, 13 Feb 2018 19:41:37 +0200 Subject: [PATCH 22/87] BLD: added password into credentials needed for exchanges such as 'gdax' --- catalyst/exchange/ccxt/ccxt_exchange.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/catalyst/exchange/ccxt/ccxt_exchange.py b/catalyst/exchange/ccxt/ccxt_exchange.py index c5fee495..1c81c90f 100644 --- a/catalyst/exchange/ccxt/ccxt_exchange.py +++ b/catalyst/exchange/ccxt/ccxt_exchange.py @@ -43,7 +43,8 @@ SUPPORTED_EXCHANGES = dict( class CCXT(Exchange): - def __init__(self, exchange_name, key, secret, base_currency): + def __init__(self, exchange_name, key, + secret, password, base_currency): log.debug( 'finding {} in CCXT exchanges:\n{}'.format( exchange_name, ccxt.exchanges @@ -60,6 +61,7 @@ class CCXT(Exchange): self.api = exchange_attr({ 'apiKey': key, 'secret': secret, + 'password': password, }) self.api.enableRateLimit = True From d494de7551e43ad0797a52265c73d02d338e59ee Mon Sep 17 00:00:00 2001 From: Victor Grau Serrat Date: Tue, 13 Feb 2018 12:04:49 -0700 Subject: [PATCH 23/87] MAINT: conda env for Python3 --- etc/python2.7-environment.yml | 2 +- etc/python3.6-environment.yml | 95 +++++++++++++++++++++++++++++++++++ 2 files changed, 96 insertions(+), 1 deletion(-) create mode 100644 etc/python3.6-environment.yml diff --git a/etc/python2.7-environment.yml b/etc/python2.7-environment.yml index 0aa916d6..2037515b 100644 --- a/etc/python2.7-environment.yml +++ b/etc/python2.7-environment.yml @@ -59,4 +59,4 @@ dependencies: - tables==3.4.2 - toolz==0.8.2 - urllib3==1.22 - - enigma-catalyst>=0.3 + - enigma-catalyst>=0.5 diff --git a/etc/python3.6-environment.yml b/etc/python3.6-environment.yml new file mode 100644 index 00000000..446198e0 --- /dev/null +++ b/etc/python3.6-environment.yml @@ -0,0 +1,95 @@ +name: catalyst +channels: +- defaults +dependencies: +- ca-certificates=2017.08.26=ha1e5d58_0 +- certifi=2018.1.18=py36_0 +- intel-openmp=2018.0.0=h8158457_8 +- libcxx=4.0.1=h579ed51_0 +- libcxxabi=4.0.1=hebd6815_0 +- libedit=3.1=hb4e282d_0 +- libffi=3.2.1=h475c297_4 +- libgfortran=3.0.1=h93005f0_2 +- mkl=2018.0.1=hfbd8650_4 +- ncurses=6.0=hd04f020_2 +- numpy=1.14.0=py36h8a80b8c_1 +- openssl=1.0.2n=hdbc3d79_0 +- pip=9.0.1=py36h1555ced_4 +- python=3.6.4=hc167b69_1 +- readline=7.0=hc1231fa_4 +- scipy=1.0.0=py36h1de22e9_0 +- setuptools=38.4.0=py36_0 +- sqlite=3.22.0=h3efe00b_0 +- tk=8.6.7=h35a86e2_3 +- wheel=0.30.0=py36h5eb2c71_1 +- xz=5.2.3=h0278029_2 +- zlib=1.2.11=hf3cbc9b_2 +- pip: + - aiodns==1.1.1 + - aiohttp==3.0.1 + - alembic==0.9.7 + - async-timeout==2.0.0 + - attrdict==2.0.0 + - attrs==17.4.0 + - bcolz==0.12.1 + - boto3==1.5.27 + - botocore==1.8.41 + - bottleneck==1.2.1 + - cchardet==2.1.1 + - ccxt==1.10.1102 + - chardet==3.0.4 + - click==6.7 + - contextlib2==0.5.5 + - cyordereddict==1.0.0 + - cython==0.27.3 + - cytoolz==0.9.0 + - decorator==4.2.1 + - docutils==0.14 + - empyrical==0.2.1 + - enigma-catalyst>=0.5.3 + - eth-abi==1.0.0b0 + - eth-account==0.1.0a2 + - eth-keyfile==0.5.1 + - eth-keys==0.2.0b1 + - eth-rlp==0.1.0a2 + - eth-utils==1.0.0b1 + - hexbytes==0.1.0b0 + - idna==2.6 + - idna-ssl==1.0.0 + - intervaltree==2.1.0 + - jmespath==0.9.3 + - logbook==1.2.1 + - lru-dict==1.1.6 + - lxml==4.1.1 + - mako==1.0.7 + - markupsafe==1.0 + - multidict==4.1.0 + - multipledispatch==0.4.9 + - networkx==2.1 + - numexpr==2.6.4 + - pandas==0.19.2 + - pandas-datareader==0.6.0 + - patsy==0.5.0 + - pycares==2.3.0 + - pycryptodome==3.4.11 + - pysha3==1.0.2 + - python-dateutil==2.6.1 + - python-editor==1.0.3 + - pytz==2018.3 + - redo==1.6 + - requests==2.18.4 + - requests-file==1.4.3 + - requests-ftp==0.3.1 + - requests-toolbelt==0.8.0 + - rlp==0.6.0 + - s3transfer==0.1.12 + - six==1.11.0 + - sortedcontainers==1.5.9 + - sqlalchemy==1.2.2 + - statsmodels==0.8.0 + - tables==3.4.2 + - toolz==0.9.0 + - urllib3==1.22 + - web3==4.0.0b9 + - wrapt==1.10.11 + - yarl==1.1.0 From 92d3b98448a1c2d804479d11bb9329ef1b38fd00 Mon Sep 17 00:00:00 2001 From: Victor Grau Serrat Date: Tue, 13 Feb 2018 12:14:11 -0700 Subject: [PATCH 24/87] MAINT: updated conda install instructions for Python3 --- docs/source/install.rst | 23 +++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-) diff --git a/docs/source/install.rst b/docs/source/install.rst index 7459f97a..59279683 100644 --- a/docs/source/install.rst +++ b/docs/source/install.rst @@ -47,8 +47,10 @@ you can install MiniConda, which is a smaller footprint (fewer packages and smaller size) than its big brother Anaconda, but it still contains all the main packages needed. To install MiniConda, you can follow these steps: -1. Download `MiniConda `_. Select Python 2.7 - for your Operating System. +1. Download `MiniConda `_. Select either + Python 3.6 (recommended) or Python 2.7 for your Operating System. The + `Enigma Data Marketplace `_ will + require Python3, that's why we are recommending to opt for the newer version. 2. Install MiniConda. See the `Installation Instructions `_ if you need help. 3. Ensure the correct installation by running ``conda list`` in a Terminal @@ -64,21 +66,30 @@ main packages needed. To install MiniConda, you can follow these steps: Once either Conda or MiniConda has been set up you can install Catalyst: -1. Download the file `python2.7-environment.yml - `_. +1. Download the file `python3.6-environment.yml + `_ + (recommended) or `python2.7-environment.yml + `_ + matching your Conda installation from step #1 above. To download, simply click on the 'Raw' button and save the file locally to a folder you can remember. Make sure that the file gets saved with the ``.yml`` extension, and nothing like a ``.txt`` file or anything else. 2. Open a Terminal window and enter [``cd/dir``] into the directory where you - saved the above ``python2.7-environment.yml`` file. + saved the above ``.yml`` file. 3. Install using this file. This step can take about 5-10 minutes to install. .. code-block:: bash - conda env create -f python2.7-environment.yml + conda env create -f python3.6-environment.yml + + or + + .. code-block:: bash + + conda env create -f python2.7-environment.yml 4. Activate the environment (which you need to do every time you start a new session to run Catalyst): From 8072ded532afcc3a82c3a96e54fafe30a37a41a0 Mon Sep 17 00:00:00 2001 From: Frederic Fortier Date: Wed, 14 Feb 2018 12:25:44 -0500 Subject: [PATCH 25/87] BLD: clarified open order message --- catalyst/exchange/exchange_blotter.py | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/catalyst/exchange/exchange_blotter.py b/catalyst/exchange/exchange_blotter.py index 6ded5adf..c82b2f00 100644 --- a/catalyst/exchange/exchange_blotter.py +++ b/catalyst/exchange/exchange_blotter.py @@ -238,9 +238,12 @@ class ExchangeBlotter(Blotter): else: delta = pd.Timestamp.utcnow() - order.dt log.info( - 'order {order_id} still open after {delta}'.format( + '{exchange} order {order_id} for {symbol} still open ' + 'after {delta}'.format( + exchange=exchange.name, order_id=order.id, - delta=delta + delta=delta, + symbol=order.asset.symbol, ) ) From 041665dee1ede9ea3ccccb2c00444675994870e7 Mon Sep 17 00:00:00 2001 From: Frederic Fortier Date: Wed, 14 Feb 2018 13:37:51 -0500 Subject: [PATCH 26/87] BUG: fixed issue with incremental ingestion --- catalyst/marketplace/utils/bundle_utils.py | 10 +++++----- tests/marketplace/test_marketplace.py | 2 +- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/catalyst/marketplace/utils/bundle_utils.py b/catalyst/marketplace/utils/bundle_utils.py index b58595ac..014887a6 100644 --- a/catalyst/marketplace/utils/bundle_utils.py +++ b/catalyst/marketplace/utils/bundle_utils.py @@ -2,6 +2,7 @@ import os import shutil import bcolz +import pandas as pd def merge_bundles(zsource, ztarget): @@ -18,14 +19,13 @@ def merge_bundles(zsource, ztarget): """ # TODO: find a way to do this iteratively instead of in-memory df_source = zsource.todataframe() - df_source.set_index('date', drop=False, inplace=True) df_target = ztarget.todataframe() - df_target.set_index('date', drop=False, inplace=True) - df = df_target.merge( - right=df_source, - how='right', + df = pd.concat( + [df_source, df_target], ignore_index=True ) # type: pd.DataFrame + df.drop_duplicates(inplace=True) + df.set_index(['date', 'symbol'], drop=False, inplace=True) dirname = os.path.basename(ztarget.rootdir) bak_dir = ztarget.rootdir.replace(dirname, '.{}'.format(dirname)) diff --git a/tests/marketplace/test_marketplace.py b/tests/marketplace/test_marketplace.py index 59564a5e..c0895b5a 100644 --- a/tests/marketplace/test_marketplace.py +++ b/tests/marketplace/test_marketplace.py @@ -21,7 +21,7 @@ class TestMarketplace(WithLogger, ZiplineTestCase): def test_ingest(self): marketplace = Marketplace() - ds_def = marketplace.ingest('marketcap1234') + ds_def = marketplace.ingest('github') pass def test_publish(self): From 2ba825db9bf8a375e61434f55cd7e0ff6e0edfcf Mon Sep 17 00:00:00 2001 From: Frederic Fortier Date: Wed, 14 Feb 2018 20:26:41 -0500 Subject: [PATCH 27/87] BUG: for issue #227, made more mappings for hourly frequency --- catalyst/examples/simple_loop.py | 4 ++-- catalyst/exchange/exchange.py | 5 +++-- catalyst/exchange/utils/datetime_utils.py | 15 ++++++++++----- 3 files changed, 15 insertions(+), 9 deletions(-) diff --git a/catalyst/examples/simple_loop.py b/catalyst/examples/simple_loop.py index 0de91d3d..c95726fa 100644 --- a/catalyst/examples/simple_loop.py +++ b/catalyst/examples/simple_loop.py @@ -26,7 +26,7 @@ def handle_data(context, data): context.asset, fields='price', bar_count=20, - frequency='30T' + frequency='1H' ) last_traded = prices.index[-1] log.info('last candle date: {}'.format(last_traded)) @@ -114,7 +114,7 @@ def analyze(context, perf): if __name__ == '__main__': - mode = 'backtest' + mode = 'live' if mode == 'backtest': run_algorithm( diff --git a/catalyst/exchange/exchange.py b/catalyst/exchange/exchange.py index f32d9a2b..c1b64f7c 100644 --- a/catalyst/exchange/exchange.py +++ b/catalyst/exchange/exchange.py @@ -502,7 +502,7 @@ class Exchange: """ freq, candle_size, unit, data_frequency = get_frequency( - frequency, data_frequency + frequency, data_frequency, supported_freqs=['T', 'D', 'H'] ) # The get_history method supports multiple asset candles = self.get_candles( @@ -523,8 +523,9 @@ class Exchange: field=field, ) + delta_candle_size = candle_size * 60 if unit == 'H' else candle_size # Checking to make sure that the dates match - delta = get_delta(candle_size, data_frequency) + delta = get_delta(delta_candle_size, data_frequency) adj_end_dt = end_dt - delta last_traded = asset_series.index[-1] diff --git a/catalyst/exchange/utils/datetime_utils.py b/catalyst/exchange/utils/datetime_utils.py index 5f3a2e04..c34b0c69 100644 --- a/catalyst/exchange/utils/datetime_utils.py +++ b/catalyst/exchange/utils/datetime_utils.py @@ -92,7 +92,7 @@ def get_periods_range(freq, start_dt=None, end_dt=None, periods=None): adj_periods = periods * unit_periods # TODO: standardize time aliases to avoid any mapping - unit = 'd' if unit == 'D' else 'm' + unit = 'd' if unit == 'D' else 'h' if unit == 'H' else 'm' delta = pd.Timedelta(adj_periods, unit) if start_dt is not None: @@ -248,7 +248,7 @@ def get_year_start_end(dt, first_day=None, last_day=None): return year_start, year_end -def get_frequency(freq, data_frequency=None): +def get_frequency(freq, data_frequency=None, supported_freqs=['D', 'T']): """ Get the frequency parameters. @@ -305,9 +305,14 @@ def get_frequency(freq, data_frequency=None): data_frequency = 'minute' elif unit.lower() == 'h': - candle_size = candle_size * 60 - alias = '{}T'.format(candle_size) - data_frequency = 'minute' + if 'H' in supported_freqs: + unit = 'H' + alias = '{}H'.format(candle_size) + + else: + candle_size = candle_size * 60 + alias = '{}T'.format(candle_size) + data_frequency = 'minute' else: raise InvalidHistoryFrequencyAlias(freq=freq) From 866b0a215c7321b7ce9f371fa04ecf8f32861ae5 Mon Sep 17 00:00:00 2001 From: Frederic Fortier Date: Wed, 14 Feb 2018 23:30:53 -0500 Subject: [PATCH 28/87] BUG: for issue #227, made more mappings for hourly frequency --- catalyst/examples/simple_loop.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/catalyst/examples/simple_loop.py b/catalyst/examples/simple_loop.py index c95726fa..99822b70 100644 --- a/catalyst/examples/simple_loop.py +++ b/catalyst/examples/simple_loop.py @@ -26,7 +26,7 @@ def handle_data(context, data): context.asset, fields='price', bar_count=20, - frequency='1H' + frequency='2H' ) last_traded = prices.index[-1] log.info('last candle date: {}'.format(last_traded)) From a61857b37d87de8c029857cdfd5899feb2f9597c Mon Sep 17 00:00:00 2001 From: AvishaiW Date: Thu, 15 Feb 2018 20:22:15 +0200 Subject: [PATCH 29/87] BUG: changed the data, analyze gets in live - fixed the bug following #229 - at the end of each day the stores the daily stats to local directory - removes the stats folder at the begining of each run to avoid overloading the disk. - removes old data (over a month) during the run to avoid overloading the disk --- catalyst/exchange/exchange_algorithm.py | 99 ++++++++++++++++++----- catalyst/exchange/utils/exchange_utils.py | 70 +++++++++++++++- 2 files changed, 148 insertions(+), 21 deletions(-) diff --git a/catalyst/exchange/exchange_algorithm.py b/catalyst/exchange/exchange_algorithm.py index aeff9e4e..5c9f8771 100644 --- a/catalyst/exchange/exchange_algorithm.py +++ b/catalyst/exchange/exchange_algorithm.py @@ -16,7 +16,7 @@ import signal import sys from datetime import timedelta from os import listdir -from os.path import isfile, join +from os.path import isfile, join, exists import catalyst.protocol as zp import logbook @@ -36,9 +36,11 @@ from catalyst.exchange.utils.exchange_utils import ( get_algo_folder, get_algo_df, save_algo_df, + clear_frame_stats_directory, + remove_old_files, group_assets_by_exchange, ) -from catalyst.exchange.utils.stats_utils import get_pretty_stats, stats_to_s3, \ - stats_to_algo_folder +from catalyst.exchange.utils.stats_utils import \ + get_pretty_stats, stats_to_s3, stats_to_algo_folder from catalyst.finance.execution import MarketOrder from catalyst.finance.performance import PerformanceTracker from catalyst.finance.performance.period import calc_period_stats @@ -67,8 +69,8 @@ class ExchangeTradingAlgorithmBase(TradingAlgorithm): self.current_day = None - if self.simulate_orders is None \ - and self.sim_params.arena == 'backtest': + if self.simulate_orders is None and \ + self.sim_params.arena == 'backtest': self.simulate_orders = True # Operations with retry features @@ -118,7 +120,7 @@ class ExchangeTradingAlgorithmBase(TradingAlgorithm): # be in-line with CXXT and many exchanges. We'll consider # adding more order types in the future. if not isinstance(style, ExchangeLimitOrder) or \ - not isinstance(style, MarketOrder): + not isinstance(style, MarketOrder): raise OrderTypeNotSupported( order_type=style.__class__.__name__ ) @@ -368,6 +370,11 @@ class ExchangeTradingAlgorithmLive(ExchangeTradingAlgorithmBase): self._clock = None self.frame_stats = list() + # erase the frame_stats folder to avoid overloading the disk + error = clear_frame_stats_directory(self.algo_namespace) + if error: + log.warning(error) + self.pnl_stats = get_algo_df(self.algo_namespace, 'pnl_stats') self.custom_signals_stats = \ @@ -392,6 +399,19 @@ class ExchangeTradingAlgorithmLive(ExchangeTradingAlgorithmBase): "Exit should be handled by the user.") def interrupt_algorithm(self): + """ + + when algorithm comes to an end this function is called. + extracts the stats and calls analyze. + after finishing, it exits the run. + + Parameters + ---------- + + Returns + ------- + + """ self.is_running = False if self._analyze is None: @@ -401,21 +421,31 @@ class ExchangeTradingAlgorithmLive(ExchangeTradingAlgorithmBase): log.info('Exiting the algorithm. Calling `analyze()` ' 'before exiting the algorithm.') + # add the last day stats which is not saved in the directory + current_stats = pd.DataFrame(self.frame_stats) + current_stats.set_index('period_close', drop=False, inplace=True) + + # get the location of the directory algo_folder = get_algo_folder(self.algo_namespace) - folder = join(algo_folder, 'daily_performance') - files = [f for f in listdir(folder) if isfile(join(folder, f))] + folder = join(algo_folder, 'frame_stats') - daily_perf_list = [] - for item in files: - filename = join(folder, item) + if exists(folder): + files = [f for f in listdir(folder) if isfile(join(folder, f))] - with open(filename, 'rb') as handle: - perf_period = pickle.load(handle) - perf_period_dict = perf_period.to_dict() - daily_perf_list.append(perf_period_dict) + period_stats_list = [] + for item in files: + filename = join(folder, item) - stats = pd.DataFrame(daily_perf_list) - stats.set_index('period_close', drop=False, inplace=True) + with open(filename, 'rb') as handle: + perf_period = pickle.load(handle) + period_stats_list.extend(perf_period) + + stats = pd.DataFrame(period_stats_list) + stats.set_index('period_close', drop=False, inplace=True) + + stats = pd.concat([stats, current_stats]) + else: + stats = current_stats self.analyze(stats) @@ -709,6 +739,37 @@ class ExchangeTradingAlgorithmLive(ExchangeTradingAlgorithmBase): self.algo_namespace, 'exposure_stats', self.exposure_stats ) + def nullify_frame_stats(self, now): + """ + + Save all period_stats to local directory + erase old files from the folder and nullify + self.frame_stats + + Parameters + ---------- + now: Timestamp + + Returns + ------- + + """ + save_algo_object( + algo_name=self.algo_namespace, + key=now.floor('1D').strftime('%Y-%m-%d'), + obj=self.frame_stats, + rel_path='frame_stats' + ) + error = remove_old_files( + algo_name=self.algo_namespace, + today=now, + rel_path='frame_stats' + ) + if error: + log.warning(error) + + self.frame_stats = list() + def handle_data(self, data): """ Wrapper around the handle_data method of each algo. @@ -728,7 +789,7 @@ class ExchangeTradingAlgorithmLive(ExchangeTradingAlgorithmBase): # Resetting the frame stats every day to minimize memory footprint today = data.current_dt.floor('1D') if self.current_day is not None and today > self.current_day: - self.frame_stats = list() + self.nullify_frame_stats(now=data.current_dt) self.performance_needs_update = False orders = list(self.perf_tracker.todays_performance.orders_by_id.keys()) @@ -808,6 +869,8 @@ class ExchangeTradingAlgorithmLive(ExchangeTradingAlgorithmBase): # Saving the last hour in memory self.frame_stats.append(frame_stats) + # creating and saving the pnl_stats into the local + # directory self.add_pnl_stats(frame_stats) if self.recorded_vars: self.add_custom_signals_stats(frame_stats) diff --git a/catalyst/exchange/utils/exchange_utils.py b/catalyst/exchange/utils/exchange_utils.py index 3f8d6121..fb24f1c8 100644 --- a/catalyst/exchange/utils/exchange_utils.py +++ b/catalyst/exchange/utils/exchange_utils.py @@ -130,7 +130,7 @@ def get_exchange_symbols(exchange_name, is_local=False, environ=None): filename)).days > 1): try: download_exchange_symbols(exchange_name, environ) - except Exception as e: + except Exception: pass if os.path.isfile(filename): @@ -273,6 +273,7 @@ def get_algo_object(algo_name, key, environ=None, rel_path=None, how='pickle'): key: str environ: rel_path: str + how: str Returns ------- @@ -316,6 +317,7 @@ def save_algo_object(algo_name, key, obj, environ=None, rel_path=None, obj: Object environ: rel_path: str + how: str """ folder = get_algo_folder(algo_name, environ) @@ -392,6 +394,67 @@ def save_algo_df(algo_name, key, df, environ=None, rel_path=None): df.to_csv(handle, encoding='UTF_8') +def clear_frame_stats_directory(algo_name): + """ + remove the outdated directory + to avoid overloading the disk + + Parameters + ---------- + algo_name: str + + Returns + ------- + error: str + + """ + error = None + algo_folder = get_algo_folder(algo_name) + folder = os.path.join(algo_folder, 'frame_stats') + if os.path.exists(folder): + try: + shutil.rmtree(folder) + except OSError: + error = 'unable to remove {}, the analyze ' \ + 'data will be inconsistent'.format(folder) + return error + + +def remove_old_files(algo_name, today, rel_path): + """ + remove old files from a directory + to avoid overloading the disk + + Parameters + ---------- + algo_name: str + today: Timestamp + rel_path: str + + Returns + ------- + error: str + + """ + error = None + algo_folder = get_algo_folder(algo_name) + folder = os.path.join(algo_folder, rel_path) + + # run on all files in the folder + for f in os.listdir(folder): + creation_unix = os.path.getctime(f) + creation_time = pd.to_datetime(creation_unix, unit='s', ) + + # if the file is older than 30 days erase it + if today - pd.DateOffset(30) > creation_time: + try: + os.unlink(f) + except OSError: + error = 'unable to erase files in {}'.format(folder) + + return error + + def get_exchange_minute_writer_root(exchange_name, environ=None): """ The minute writer folder for the exchange. @@ -575,8 +638,9 @@ def mixin_market_params(exchange_name, params, market): params['maker'] = 0.001 params['taker'] = 0.002 - elif 'maker' in market and 'taker' in market \ - and market['maker'] is not None and market['taker'] is not None: + elif 'maker' in market and 'taker' in market and \ + market['maker'] is not None and market['taker'] is not None: + params['maker'] = market['maker'] params['taker'] = market['taker'] From bd88ba2277f6067a41ac00f388c754a9091b68b2 Mon Sep 17 00:00:00 2001 From: Victor Grau Serrat Date: Fri, 16 Feb 2018 11:48:52 -0700 Subject: [PATCH 30/87] DOC: marketplace code examples --- .../examples/marketplace/github-research.py | 70 +++++++++++++++++++ .../mean_reversion_by_marketcap.py | 0 2 files changed, 70 insertions(+) create mode 100644 catalyst/examples/marketplace/github-research.py rename catalyst/examples/{ => marketplace}/mean_reversion_by_marketcap.py (100%) diff --git a/catalyst/examples/marketplace/github-research.py b/catalyst/examples/marketplace/github-research.py new file mode 100644 index 00000000..b2443597 --- /dev/null +++ b/catalyst/examples/marketplace/github-research.py @@ -0,0 +1,70 @@ +import pandas as pd +import matplotlib.pyplot as plt + +from catalyst import run_algorithm +from catalyst.api import symbol, get_dataset + +START = '2017-01-01' +END = '2017-12-31' + + +def initialize(context): + pass + + +def handle_data(context, data): + context.github = get_dataset('github') + context.github.sort_index(level=0, inplace=True) + + context.zec = data.history(symbol('zec_usdt'), + ['price', ], + bar_count=365, + frequency="1d") + context.xmr = data.history(symbol('xmr_usdt'), + ['price', ], + bar_count=365, + frequency="1d") + + +def analyze(context=None, results=None): + ax1 = plt.subplot(211) + idx = pd.IndexSlice + df = context.github.loc[START:END].loc[ + idx[:, [b'ZEC']], ['commits']].reset_index( + level='symbol', drop=True) + df.plot(ax=ax1, color='blue') + ax1.legend(loc=2) + ax1.set_title('Zcash') + ax2 = ax1.twinx() + context.zec['price'].loc[START:END].plot(ax=ax2, color='green') + ax2.legend(loc=1) + + ax3 = plt.subplot(212) + idx = pd.IndexSlice + df = context.github.loc[START:END].loc[ + idx[:, [b'XMR']], ['commits']].reset_index( + level='symbol', drop=True) + df.plot(ax=ax3, color='blue') + ax3.legend(loc=2) + ax3.set_title('Monero') + ax4 = ax3.twinx() + context.xmr['price'].loc[START:END].plot(ax=ax4, color='green') + ax4.legend(loc=1) + + plt.show() + + +if __name__ == '__main__': + run_algorithm( + capital_base=1000, + data_frequency='daily', + initialize=initialize, + handle_data=handle_data, + analyze=analyze, + exchange_name='poloniex', + algo_namespace='algo-github', + base_currency='usdt', + live=False, + start=pd.to_datetime(END, utc=True), + end=pd.to_datetime(END, utc=True), + ) diff --git a/catalyst/examples/mean_reversion_by_marketcap.py b/catalyst/examples/marketplace/mean_reversion_by_marketcap.py similarity index 100% rename from catalyst/examples/mean_reversion_by_marketcap.py rename to catalyst/examples/marketplace/mean_reversion_by_marketcap.py From 2becc4157adc5ce8336a7aeca77670a9d72abebf Mon Sep 17 00:00:00 2001 From: lenak25 Date: Tue, 20 Feb 2018 14:51:25 +0200 Subject: [PATCH 31/87] MAINT: cosmetics --- catalyst/marketplace/utils/auth_utils.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/catalyst/marketplace/utils/auth_utils.py b/catalyst/marketplace/utils/auth_utils.py index 4979b6f3..ab3c668d 100644 --- a/catalyst/marketplace/utils/auth_utils.py +++ b/catalyst/marketplace/utils/auth_utils.py @@ -47,11 +47,11 @@ def get_key_secret(pubAddr, wallet='mew'): if wallet == 'mew': print('\nObtaining a key/secret pair to streamline all future ' 'requests with the authentication server.\n' - 'Visit https://www.myetherwallet.com/signmsg.html and sign the' + 'Visit https://www.myetherwallet.com/signmsg.html and sign the ' 'following message:\n{}'.format(nonce)) signature = input('Copy and Paste the "sig" field from ' 'the signature here (without the double quotes, ' - 'only the HEX value:\n') + 'only the HEX value):\n') else: raise MarketplaceWalletNotSupported(wallet=wallet) From ab644bd73239fd85d87b4fb36b5c448c152312b0 Mon Sep 17 00:00:00 2001 From: AvishaiW Date: Tue, 20 Feb 2018 19:17:03 +0200 Subject: [PATCH 32/87] BUG: modified dual_moving_average example modified long_window to be larger than short_window --- catalyst/examples/dual_moving_average.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/catalyst/examples/dual_moving_average.py b/catalyst/examples/dual_moving_average.py index 363edba1..00eb01af 100644 --- a/catalyst/examples/dual_moving_average.py +++ b/catalyst/examples/dual_moving_average.py @@ -21,7 +21,7 @@ def initialize(context): def handle_data(context, data): # define the windows for the moving averages short_window = 2 - long_window = 2 + long_window = 3 # Skip as many bars as long_window to properly compute the average context.i += 1 From 9936b38e09b07a829755be8cac6bc56ec4c2dada Mon Sep 17 00:00:00 2001 From: Victor Grau Serrat Date: Wed, 21 Feb 2018 08:51:59 -0700 Subject: [PATCH 33/87] DOC: updated Visual C++ instructions for Windows & Python 3 --- docs/source/install.rst | 22 ++++++++++++++++------ 1 file changed, 16 insertions(+), 6 deletions(-) diff --git a/docs/source/install.rst b/docs/source/install.rst index 59279683..5c24159f 100644 --- a/docs/source/install.rst +++ b/docs/source/install.rst @@ -454,12 +454,22 @@ about matplotlib backends, please refer to the Windows Requirements -------------------- -In Windows, you will first need to install the `Microsoft Visual C++ Compiler -for Python 2.7 -`_. This -package contains the compiler and the set of system headers necessary for -producing binary wheels for Python 2.7 packages. If it's not already in your -system, download it and install it before proceeding to the next step. +In Windows, you will first need to install the Microsoft Visual C++ Compiler, +which is different depending on the version of Python that you plan to use: + +* Python 3.5, 3.6: `Visual C++ 2015 Build Tools + `_, + which installs Visual C++ version 14.0. **This is the recommended version** + +* Python 2.7: `Microsoft Visual C++ Compiler for Python 2.7 + `_, which + installs version Visual C++ version 9.0 + +This package contains the compiler and the set of system headers necessary for +producing binary wheels for Python packages. If it's not already in your +system, download it and install it before proceeding to the next step. If you +need additional help, or are looking for other versions of Visual C++ for +Windows (only advanced users), follow `this link `_. Once you have the above compiler installed, the easiest and best supported way to install Catalyst in Windows is to use :ref:`Conda `. If you didn't From 7ee875ee7029cade5fada75e99831b4f6d5c05e0 Mon Sep 17 00:00:00 2001 From: Frederic Fortier Date: Wed, 21 Feb 2018 13:06:57 -0500 Subject: [PATCH 34/87] BLD: adjusted sample algo --- catalyst/examples/mean_reversion_simple.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/catalyst/examples/mean_reversion_simple.py b/catalyst/examples/mean_reversion_simple.py index 81a3b182..c697a88a 100644 --- a/catalyst/examples/mean_reversion_simple.py +++ b/catalyst/examples/mean_reversion_simple.py @@ -37,8 +37,8 @@ def initialize(context): context.base_price = None context.current_day = None - context.RSI_OVERSOLD = 40 - context.RSI_OVERBOUGHT = 60 + context.RSI_OVERSOLD = 60 + context.RSI_OVERBOUGHT = 70 context.CANDLE_SIZE = '15T' context.start_time = time.time() From 20f8a75f4a8906641bbb1b198d05cc15148fff02 Mon Sep 17 00:00:00 2001 From: AvishaiW Date: Wed, 21 Feb 2018 20:42:39 +0200 Subject: [PATCH 35/87] BUG: for issue #237, update positions before checking balances --- catalyst/exchange/exchange.py | 11 ++++++++--- catalyst/exchange/exchange_algorithm.py | 21 ++++++++++----------- 2 files changed, 18 insertions(+), 14 deletions(-) diff --git a/catalyst/exchange/exchange.py b/catalyst/exchange/exchange.py index c1b64f7c..3b57256b 100644 --- a/catalyst/exchange/exchange.py +++ b/catalyst/exchange/exchange.py @@ -656,16 +656,21 @@ class Exchange: return df - def _check_low_balance(self, currency, balances, amount): + def _check_low_balance(self, currency, balances, amount, open_orders=None): free = balances[currency]['free'] if currency in balances else 0.0 + if open_orders: + # TODO: make sure that this works + free += sum([order.amount for order in open_orders]) + if free < amount: return free, True else: return free, False - def sync_positions(self, positions, cash=None, check_balances=False): + def sync_positions(self, positions, open_orders=None, cash=None, + check_balances=False): """ Update the portfolio cash and position balances based on the latest ticker prices. @@ -694,7 +699,7 @@ class Exchange: balances=balances, amount=cash, ) - if is_lower: + if is_lower and not open_orders: raise NotEnoughCashError( currency=self.base_currency, exchange=self.name, diff --git a/catalyst/exchange/exchange_algorithm.py b/catalyst/exchange/exchange_algorithm.py index 5c9f8771..c827af4a 100644 --- a/catalyst/exchange/exchange_algorithm.py +++ b/catalyst/exchange/exchange_algorithm.py @@ -631,23 +631,12 @@ class ExchangeTradingAlgorithmLive(ExchangeTradingAlgorithmBase): if base_currency is None: base_currency = exchange.base_currency - # Don't check the cash if there are open orders. This could - # results in false positives. orders = [] for asset in self.blotter.open_orders: asset_orders = self.blotter.open_orders[asset] if asset_orders: orders += asset_orders - required_cash = self.portfolio.cash if not orders else None - cash, positions_value = exchange.sync_positions( - positions=exchange_positions, - check_balances=check_balances, - cash=required_cash, - ) - total_cash += cash - total_positions_value += positions_value - # Applying modifications to the original positions for position in exchange_positions: tracker.update_position( @@ -657,6 +646,16 @@ class ExchangeTradingAlgorithmLive(ExchangeTradingAlgorithmBase): last_sale_price=position.last_sale_price, ) + required_cash = self.portfolio.cash if not orders else None + cash, positions_value = exchange.sync_positions( + positions=exchange_positions, + open_orders=orders, + check_balances=check_balances, + cash=required_cash, + ) + total_cash += cash + total_positions_value += positions_value + if not check_balances: total_cash = self.portfolio.cash From 4337abd60a4bfa2f8335dba2c8c3bb9173f38cda Mon Sep 17 00:00:00 2001 From: Victor Grau Serrat Date: Wed, 21 Feb 2018 15:51:28 -0700 Subject: [PATCH 36/87] DOC: linking example_algo to their sources --- docs/source/beginner-tutorial.rst | 158 +----- docs/source/example-algos.rst | 898 +----------------------------- 2 files changed, 19 insertions(+), 1037 deletions(-) diff --git a/docs/source/beginner-tutorial.rst b/docs/source/beginner-tutorial.rst index 41afe3c0..291dd205 100644 --- a/docs/source/beginner-tutorial.rst +++ b/docs/source/beginner-tutorial.rst @@ -580,162 +580,8 @@ which you can skim through for now. A copy of this algorithm is available in the ``examples`` directory: `dual_moving_average.py `_. -.. code-block:: python - - import numpy as np - import pandas as pd - from logbook import Logger - import matplotlib.pyplot as plt - - from catalyst import run_algorithm - from catalyst.api import (order, record, symbol, order_target_percent, - get_open_orders) - from catalyst.exchange.utils.stats_utils import extract_transactions - - NAMESPACE = 'dual_moving_average' - log = Logger(NAMESPACE) - - def initialize(context): - context.i = 0 - context.asset = symbol('ltc_usd') - context.base_price = None - - - def handle_data(context, data): - # define the windows for the moving averages - short_window = 50 - long_window = 200 - - # Skip as many bars as long_window to properly compute the average - context.i += 1 - if context.i < long_window: - return - - # Compute moving averages calling data.history() for each - # moving average with the appropriate parameters. We choose to use - # minute bars for this simulation -> freq="1m" - # Returns a pandas dataframe. - short_mavg = data.history(context.asset, 'price', - bar_count=short_window, frequency="1m").mean() - long_mavg = data.history(context.asset, 'price', - bar_count=long_window, frequency="1m").mean() - - # Let's keep the price of our asset in a more handy variable - price = data.current(context.asset, 'price') - - # If base_price is not set, we use the current value. This is the - # price at the first bar which we reference to calculate price_change. - if context.base_price is None: - context.base_price = price - price_change = (price - context.base_price) / context.base_price - - # Save values for later inspection - record(price=price, - cash=context.portfolio.cash, - price_change=price_change, - short_mavg=short_mavg, - long_mavg=long_mavg) - - # Since we are using limit orders, some orders may not execute immediately - # we wait until all orders are executed before considering more trades. - orders = get_open_orders(context.asset) - if len(orders) > 0: - return - - # Exit if we cannot trade - if not data.can_trade(context.asset): - return - - # We check what's our position on our portfolio and trade accordingly - pos_amount = context.portfolio.positions[context.asset].amount - - # Trading logic - if short_mavg > long_mavg and pos_amount == 0: - # we buy 100% of our portfolio for this asset - order_target_percent(context.asset, 1) - elif short_mavg < long_mavg and pos_amount > 0: - # we sell all our positions for this asset - order_target_percent(context.asset, 0) - - - def analyze(context, perf): - - # Get the base_currency that was passed as a parameter to the simulation - exchange = list(context.exchanges.values())[0] - base_currency = exchange.base_currency.upper() - - # First chart: Plot portfolio value using base_currency - ax1 = plt.subplot(411) - perf.loc[:, ['portfolio_value']].plot(ax=ax1) - ax1.legend_.remove() - ax1.set_ylabel('Portfolio Value\n({})'.format(base_currency)) - start, end = ax1.get_ylim() - ax1.yaxis.set_ticks(np.arange(start, end, (end-start)/5)) - - # Second chart: Plot asset price, moving averages and buys/sells - ax2 = plt.subplot(412, sharex=ax1) - perf.loc[:, ['price','short_mavg','long_mavg']].plot(ax=ax2, label='Price') - ax2.legend_.remove() - ax2.set_ylabel('{asset}\n({base})'.format( - asset = context.asset.symbol, - base = base_currency - )) - start, end = ax2.get_ylim() - ax2.yaxis.set_ticks(np.arange(start, end, (end-start)/5)) - - transaction_df = extract_transactions(perf) - if not transaction_df.empty: - buy_df = transaction_df[transaction_df['amount'] > 0] - sell_df = transaction_df[transaction_df['amount'] < 0] - ax2.scatter( - buy_df.index.to_pydatetime(), - perf.loc[buy_df.index, 'price'], - marker='^', - s=100, - c='green', - label='' - ) - ax2.scatter( - sell_df.index.to_pydatetime(), - perf.loc[sell_df.index, 'price'], - marker='v', - s=100, - c='red', - label='' - ) - - # Third chart: Compare percentage change between our portfolio - # and the price of the asset - ax3 = plt.subplot(413, sharex=ax1) - perf.loc[:, ['algorithm_period_return', 'price_change']].plot(ax=ax3) - ax3.legend_.remove() - ax3.set_ylabel('Percent Change') - start, end = ax3.get_ylim() - ax3.yaxis.set_ticks(np.arange(start, end, (end-start)/5)) - - # Fourth chart: Plot our cash - ax4 = plt.subplot(414, sharex=ax1) - perf.cash.plot(ax=ax4) - ax4.set_ylabel('Cash\n({})'.format(base_currency)) - start, end = ax4.get_ylim() - ax4.yaxis.set_ticks(np.arange(0, end, end/5)) - - plt.show() - - - if __name__ == '__main__': - run_algorithm( - capital_base=1000, - data_frequency='minute', - initialize=initialize, - handle_data=handle_data, - analyze=analyze, - exchange_name='bitfinex', - algo_namespace=NAMESPACE, - base_currency='usd', - start=pd.to_datetime('2017-9-22', utc=True), - end=pd.to_datetime('2017-9-23', utc=True), - ) +.. literalinclude:: ../../catalyst/examples/dual_moving_average.py + :language: python In order to run the code above, you have to ingest the needed data first: diff --git a/docs/source/example-algos.rst b/docs/source/example-algos.rst index 0136b899..ec5b74a0 100644 --- a/docs/source/example-algos.rst +++ b/docs/source/example-algos.rst @@ -52,35 +52,8 @@ Buy BTC Simple Algorithm Source code: `examples/buy_btc_simple.py `_ -.. code-block:: python - - ''' - Run this example, by executing the following from your terminal: - catalyst ingest-exchange -x bitfinex -f daily -i btc_usdt - catalyst run -f buy_btc_simple.py -x bitfinex --start 2016-1-1 --end 2017-9-30 -o buy_btc_simple_out.pickle - - If you want to run this code using another exchange, make sure that - the asset is available on that exchange. For example, if you were to run - it for exchange Poloniex, you would need to edit the following line: - - context.asset = symbol('btc_usdt') # note 'usdt' instead of 'usd' - - and specify exchange poloniex as follows: - catalyst ingest-exchange -x poloniex -f daily -i btc_usdt - catalyst run -f buy_btc_simple.py -x poloniex --start 2016-1-1 --end 2017-9-30 -o buy_btc_simple_out.pickle - - To see which assets are available on each exchange, visit: - https://www.enigma.co/catalyst/status - ''' - - from catalyst.api import order, record, symbol - - def initialize(context): - context.asset = symbol('btc_usd') - - def handle_data(context, data): - order(context.asset, 1) - record(btc = data.current(context.asset, 'price')) +.. literalinclude:: ../../catalyst/examples/buy_btc_simple.py + :language: python This simple algorithm does not produce any output nor displays any chart. @@ -90,8 +63,6 @@ This simple algorithm does not produce any output nor displays any chart. Buy and Hodl Algorithm ~~~~~~~~~~~~~~~~~~~~~~ -Source code: `examples/buy_and_hodl.py `_ - First ingest the historical pricing data needed to run this algorithm: .. code-block:: bash @@ -119,157 +90,10 @@ that 2015-3-1 is the earliest date that Catalyst supports (if you choose an earlier date, you'll get an error), and the most recent date you can choose is one day prior to the current date. +Source code: `examples/buy_and_hodl.py `_ -.. code-block:: python - - #!/usr/bin/env python - # - # Copyright 2017 Enigma MPC, Inc. - # Copyright 2015 Quantopian, Inc. - # - # Licensed under the Apache License, Version 2.0 (the "License"); - # you may not use this file except in compliance with the License. - # You may obtain a copy of the License at - # - # http://www.apache.org/licenses/LICENSE-2.0 - # - # Unless required by applicable law or agreed to in writing, software - # distributed under the License is distributed on an "AS IS" BASIS, - # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - # See the License for the specific language governing permissions and - # limitations under the License. - import pandas as pd - import matplotlib.pyplot as plt - - from catalyst import run_algorithm - from catalyst.api import (order_target_value, symbol, record, - cancel_order, get_open_orders, ) - - - def initialize(context): - context.ASSET_NAME = 'btc_usd' - context.TARGET_HODL_RATIO = 0.8 - context.RESERVE_RATIO = 1.0 - context.TARGET_HODL_RATIO - - context.is_buying = True - context.asset = symbol(context.ASSET_NAME) - - context.i = 0 - - - def handle_data(context, data): - context.i += 1 - - starting_cash = context.portfolio.starting_cash - target_hodl_value = context.TARGET_HODL_RATIO * starting_cash - reserve_value = context.RESERVE_RATIO * starting_cash - - # Cancel any outstanding orders - orders = get_open_orders(context.asset) or [] - for order in orders: - cancel_order(order) - - # Stop buying after passing the reserve threshold - cash = context.portfolio.cash - if cash <= reserve_value: - context.is_buying = False - - # Retrieve current asset price from pricing data - price = data.current(context.asset, 'price') - - # Check if still buying and could (approximately) afford another purchase - if context.is_buying and cash > price: - print('buying') - # Place order to make position in asset equal to target_hodl_value - order_target_value( - context.asset, - target_hodl_value, - limit_price=price * 1.1, - ) - - record( - price=price, - volume=data.current(context.asset, 'volume'), - cash=cash, - starting_cash=context.portfolio.starting_cash, - leverage=context.account.leverage, - ) - - - def analyze(context=None, results=None): - - # Plot the portfolio and asset data. - ax1 = plt.subplot(611) - results[['portfolio_value']].plot(ax=ax1) - ax1.set_ylabel('Portfolio Value (USD)') - - ax2 = plt.subplot(612, sharex=ax1) - ax2.set_ylabel('{asset} (USD)'.format(asset=context.ASSET_NAME)) - results[['price']].plot(ax=ax2) - - trans = results.ix[[t != [] for t in results.transactions]] - buys = trans.ix[ - [t[0]['amount'] > 0 for t in trans.transactions] - ] - ax2.scatter( - buys.index.to_pydatetime(), - results.price[buys.index], - marker='^', - s=100, - c='g', - label='' - ) - - ax3 = plt.subplot(613, sharex=ax1) - results[['leverage', 'alpha', 'beta']].plot(ax=ax3) - ax3.set_ylabel('Leverage ') - - ax4 = plt.subplot(614, sharex=ax1) - results[['starting_cash', 'cash']].plot(ax=ax4) - ax4.set_ylabel('Cash (USD)') - - results[[ - 'treasury', - 'algorithm', - 'benchmark', - ]] = results[[ - 'treasury_period_return', - 'algorithm_period_return', - 'benchmark_period_return', - ]] - - ax5 = plt.subplot(615, sharex=ax1) - results[[ - 'treasury', - 'algorithm', - 'benchmark', - ]].plot(ax=ax5) - ax5.set_ylabel('Percent Change') - - ax6 = plt.subplot(616, sharex=ax1) - results[['volume']].plot(ax=ax6) - ax6.set_ylabel('Volume (mCoins/5min)') - - plt.legend(loc=3) - - # Show the plot. - plt.gcf().set_size_inches(18, 8) - plt.show() - - - if __name__ == '__main__': - run_algorithm( - capital_base=10000, - data_frequency='daily', - initialize=initialize, - handle_data=handle_data, - analyze=analyze, - exchange_name='bitfinex', - algo_namespace='buy_and_hodl', - base_currency='usd', - start=pd.to_datetime('2015-03-01', utc=True), - end=pd.to_datetime('2017-10-31', utc=True), - ) +.. literalinclude:: ../../catalyst/examples/buy_and_hodl.py + :language: python .. image:: https://s3.amazonaws.com/enigmaco-docs/github.io/example_buy_and_hodl.png @@ -278,166 +102,13 @@ one day prior to the current date. Dual Moving Average Crossover ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Source Code: `examples/dual_moving_average.py `_ - This strategy is covered in detail in the last part of `this tutorial `_. -.. code-block:: python +Source Code: `examples/dual_moving_average.py `_ - import numpy as np - import pandas as pd - from logbook import Logger - import matplotlib.pyplot as plt - - from catalyst import run_algorithm - from catalyst.api import (order, record, symbol, order_target_percent, - get_open_orders) - from catalyst.exchange.stats_utils import extract_transactions - - NAMESPACE = 'dual_moving_average' - log = Logger(NAMESPACE) - - def initialize(context): - context.i = 0 - context.asset = symbol('ltc_usd') - context.base_price = None - - - def handle_data(context, data): - # define the windows for the moving averages - short_window = 50 - long_window = 200 - - # Skip as many bars as long_window to properly compute the average - context.i += 1 - if context.i < long_window: - return - - # Compute moving averages calling data.history() for each - # moving average with the appropriate parameters. We choose to use - # minute bars for this simulation -> freq="1m" - # Returns a pandas dataframe. - short_mavg = data.history(context.asset, 'price', - bar_count=short_window, frequency="1m").mean() - long_mavg = data.history(context.asset, 'price', - bar_count=long_window, frequency="1m").mean() - - # Let's keep the price of our asset in a more handy variable - price = data.current(context.asset, 'price') - - # If base_price is not set, we use the current value. This is the - # price at the first bar which we reference to calculate price_change. - if context.base_price is None: - context.base_price = price - price_change = (price - context.base_price) / context.base_price - - # Save values for later inspection - record(price=price, - cash=context.portfolio.cash, - price_change=price_change, - short_mavg=short_mavg, - long_mavg=long_mavg) - - # Since we are using limit orders, some orders may not execute immediately - # we wait until all orders are executed before considering more trades. - orders = get_open_orders(context.asset) - if len(orders) > 0: - return - - # Exit if we cannot trade - if not data.can_trade(context.asset): - return - - # We check what's our position on our portfolio and trade accordingly - pos_amount = context.portfolio.positions[context.asset].amount - - # Trading logic - if short_mavg > long_mavg and pos_amount == 0: - # we buy 100% of our portfolio for this asset - order_target_percent(context.asset, 1) - elif short_mavg < long_mavg and pos_amount > 0: - # we sell all our positions for this asset - order_target_percent(context.asset, 0) - - - def analyze(context, perf): - - # Get the base_currency that was passed as a parameter to the simulation - base_currency = context.exchanges.values()[0].base_currency.upper() - - # First chart: Plot portfolio value using base_currency - ax1 = plt.subplot(411) - perf.loc[:, ['portfolio_value']].plot(ax=ax1) - ax1.legend_.remove() - ax1.set_ylabel('Portfolio Value\n({})'.format(base_currency)) - start, end = ax1.get_ylim() - ax1.yaxis.set_ticks(np.arange(start, end, (end-start)/5)) - - # Second chart: Plot asset price, moving averages and buys/sells - ax2 = plt.subplot(412, sharex=ax1) - perf.loc[:, ['price','short_mavg','long_mavg']].plot(ax=ax2, label='Price') - ax2.legend_.remove() - ax2.set_ylabel('{asset}\n({base})'.format( - asset = context.asset.symbol, - base = base_currency - )) - start, end = ax2.get_ylim() - ax2.yaxis.set_ticks(np.arange(start, end, (end-start)/5)) - - transaction_df = extract_transactions(perf) - if not transaction_df.empty: - buy_df = transaction_df[transaction_df['amount'] > 0] - sell_df = transaction_df[transaction_df['amount'] < 0] - ax2.scatter( - buy_df.index.to_pydatetime(), - perf.loc[buy_df.index, 'price'], - marker='^', - s=100, - c='green', - label='' - ) - ax2.scatter( - sell_df.index.to_pydatetime(), - perf.loc[sell_df.index, 'price'], - marker='v', - s=100, - c='red', - label='' - ) - - # Third chart: Compare percentage change between our portfolio - # and the price of the asset - ax3 = plt.subplot(413, sharex=ax1) - perf.loc[:, ['algorithm_period_return', 'price_change']].plot(ax=ax3) - ax3.legend_.remove() - ax3.set_ylabel('Percent Change') - start, end = ax3.get_ylim() - ax3.yaxis.set_ticks(np.arange(start, end, (end-start)/5)) - - # Fourth chart: Plot our cash - ax4 = plt.subplot(414, sharex=ax1) - perf.cash.plot(ax=ax4) - ax4.set_ylabel('Cash\n({})'.format(base_currency)) - start, end = ax4.get_ylim() - ax4.yaxis.set_ticks(np.arange(0, end, end/5)) - - plt.show() - - - if __name__ == '__main__': - run_algorithm( - capital_base=1000, - data_frequency='minute', - initialize=initialize, - handle_data=handle_data, - analyze=analyze, - exchange_name='bitfinex', - algo_namespace=NAMESPACE, - base_currency='usd', - start=pd.to_datetime('2017-9-22', utc=True), - end=pd.to_datetime('2017-9-23', utc=True), - ) +.. literalinclude:: ../../catalyst/examples/dual_moving_average.py + :language: python .. image:: https://s3.amazonaws.com/enigmaco-docs/github.io/tutorial_dual_moving_average.png @@ -447,8 +118,6 @@ This strategy is covered in detail in the last part of Mean Reversion Algorithm ~~~~~~~~~~~~~~~~~~~~~~~~ -Source code: `examples/mean_reversion_simple.py `_ - This algorithm is based on a simple momentum strategy. When the cryptoasset goes up quickly, we're going to buy; when it goes down quickly, we're going to sell. Hopefully, we'll ride the waves. @@ -469,284 +138,10 @@ lines 218-245, so in order to run the algorithm we just type: python mean_reversion_simple.py -.. code-block:: python +Source code: `examples/mean_reversion_simple.py `_ - import os - import tempfile - import time - - import numpy as np - import pandas as pd - import talib - from logbook import Logger - - from catalyst import run_algorithm - from catalyst.api import symbol, record, order_target_percent, get_open_orders - from catalyst.exchange.stats_utils import extract_transactions - # We give a name to the algorithm which Catalyst will use to persist its state. - # In this example, Catalyst will create the `.catalyst/data/live_algos` - # directory. If we stop and start the algorithm, Catalyst will resume its - # state using the files included in the folder. - from catalyst.utils.paths import ensure_directory - - NAMESPACE = 'mean_reversion_simple' - log = Logger(NAMESPACE) - - - # To run an algorithm in Catalyst, you need two functions: initialize and - # handle_data. - - def initialize(context): - # This initialize function sets any data or variables that you'll use in - # your algorithm. For instance, you'll want to define the trading pair (or - # trading pairs) you want to backtest. You'll also want to define any - # parameters or values you're going to use. - - # In our example, we're looking at Neo in USD. - context.neo_eth = symbol('neo_usd') - context.base_price = None - context.current_day = None - - context.RSI_OVERSOLD = 30 - context.RSI_OVERBOUGHT = 80 - context.CANDLE_SIZE = '15T' - - context.start_time = time.time() - - - def handle_data(context, data): - # This handle_data function is where the real work is done. Our data is - # minute-level tick data, and each minute is called a frame. This function - # runs on each frame of the data. - - # We flag the first period of each day. - # Since cryptocurrencies trade 24/7 the `before_trading_starts` handle - # would only execute once. This method works with minute and daily - # frequencies. - today = data.current_dt.floor('1D') - if today != context.current_day: - context.traded_today = False - context.current_day = today - - # We're computing the volume-weighted-average-price of the security - # defined above, in the context.neo_eth variable. For this example, we're - # using three bars on the 15 min bars. - - # The frequency attribute determine the bar size. We use this convention - # for the frequency alias: - # http://pandas.pydata.org/pandas-docs/stable/timeseries.html#offset-aliases - prices = data.history( - context.neo_eth, - fields='close', - bar_count=50, - frequency=context.CANDLE_SIZE - ) - - # Ta-lib calculates various technical indicator based on price and - # volume arrays. - - # In this example, we are comp - rsi = talib.RSI(prices.values, timeperiod=14) - - # We need a variable for the current price of the security to compare to - # the average. Since we are requesting two fields, data.current() - # returns a DataFrame with - current = data.current(context.neo_eth, fields=['close', 'volume']) - price = current['close'] - - # If base_price is not set, we use the current value. This is the - # price at the first bar which we reference to calculate price_change. - if context.base_price is None: - context.base_price = price - - price_change = (price - context.base_price) / context.base_price - cash = context.portfolio.cash - - # Now that we've collected all current data for this frame, we use - # the record() method to save it. This data will be available as - # a parameter of the analyze() function for further analysis. - record( - price=price, - volume=current['volume'], - price_change=price_change, - rsi=rsi[-1], - cash=cash - ) - - # We are trying to avoid over-trading by limiting our trades to - # one per day. - if context.traded_today: - return - - # Since we are using limit orders, some orders may not execute immediately - # we wait until all orders are executed before considering more trades. - orders = get_open_orders(context.neo_eth) - if len(orders) > 0: - return - - # Exit if we cannot trade - if not data.can_trade(context.neo_eth): - return - - # Another powerful built-in feature of the Catalyst backtester is the - # portfolio object. The portfolio object tracks your positions, cash, - # cost basis of specific holdings, and more. In this line, we calculate - # how long or short our position is at this minute. - pos_amount = context.portfolio.positions[context.neo_eth].amount - - if rsi[-1] <= context.RSI_OVERSOLD and pos_amount == 0: - log.info( - '{}: buying - price: {}, rsi: {}'.format( - data.current_dt, price, rsi[-1] - ) - ) - # Set a style for limit orders, - limit_price = price * 1.005 - order_target_percent( - context.neo_eth, 1, limit_price=limit_price - ) - context.traded_today = True - - elif rsi[-1] >= context.RSI_OVERBOUGHT and pos_amount > 0: - log.info( - '{}: selling - price: {}, rsi: {}'.format( - data.current_dt, price, rsi[-1] - ) - ) - limit_price = price * 0.995 - order_target_percent( - context.neo_eth, 0, limit_price=limit_price - ) - context.traded_today = True - - - def analyze(context=None, perf=None): - end = time.time() - log.info('elapsed time: {}'.format(end - context.start_time)) - - import matplotlib.pyplot as plt - # The base currency of the algo exchange - base_currency = context.exchanges.values()[0].base_currency.upper() - - # Plot the portfolio value over time. - ax1 = plt.subplot(611) - perf.loc[:, 'portfolio_value'].plot(ax=ax1) - ax1.set_ylabel('Portfolio\nValue\n({})'.format(base_currency)) - - # Plot the price increase or decrease over time. - ax2 = plt.subplot(612, sharex=ax1) - perf.loc[:, 'price'].plot(ax=ax2, label='Price') - - ax2.set_ylabel('{asset}\n({base})'.format( - asset=context.neo_eth.symbol, base=base_currency - )) - - transaction_df = extract_transactions(perf) - if not transaction_df.empty: - buy_df = transaction_df[transaction_df['amount'] > 0] - sell_df = transaction_df[transaction_df['amount'] < 0] - ax2.scatter( - buy_df.index.to_pydatetime(), - perf.loc[buy_df.index.floor('1 min'), 'price'], - marker='^', - s=100, - c='green', - label='' - ) - ax2.scatter( - sell_df.index.to_pydatetime(), - perf.loc[sell_df.index.floor('1 min'), 'price'], - marker='v', - s=100, - c='red', - label='' - ) - - ax4 = plt.subplot(613, sharex=ax1) - perf.loc[:, 'cash'].plot( - ax=ax4, label='Base Currency ({})'.format(base_currency) - ) - ax4.set_ylabel('Cash\n({})'.format(base_currency)) - - perf['algorithm'] = perf.loc[:, 'algorithm_period_return'] - - ax5 = plt.subplot(614, sharex=ax1) - perf.loc[:, ['algorithm', 'price_change']].plot(ax=ax5) - ax5.set_ylabel('Percent\nChange') - - ax6 = plt.subplot(615, sharex=ax1) - perf.loc[:, 'rsi'].plot(ax=ax6, label='RSI') - ax6.set_ylabel('RSI') - ax6.axhline(context.RSI_OVERBOUGHT, color='darkgoldenrod') - ax6.axhline(context.RSI_OVERSOLD, color='darkgoldenrod') - - if not transaction_df.empty: - ax6.scatter( - buy_df.index.to_pydatetime(), - perf.loc[buy_df.index.floor('1 min'), 'rsi'], - marker='^', - s=100, - c='green', - label='' - ) - ax6.scatter( - sell_df.index.to_pydatetime(), - perf.loc[sell_df.index.floor('1 min'), 'rsi'], - marker='v', - s=100, - c='red', - label='' - ) - plt.legend(loc=3) - start, end = ax6.get_ylim() - ax6.yaxis.set_ticks(np.arange(0, end, end/5)) - - # Show the plot. - plt.gcf().set_size_inches(18, 8) - plt.show() - pass - - - if __name__ == '__main__': - # The execution mode: backtest or live - MODE = 'backtest' - - if MODE == 'backtest': - folder = os.path.join( - tempfile.gettempdir(), 'catalyst', NAMESPACE - ) - ensure_directory(folder) - - timestr = time.strftime('%Y%m%d-%H%M%S') - out = os.path.join(folder, '{}.p'.format(timestr)) - # catalyst run -f catalyst/examples/mean_reversion_simple.py -x bitfinex -s 2017-10-1 -e 2017-11-10 -c usdt -n mean-reversion --data-frequency minute --capital-base 10000 - run_algorithm( - capital_base=10000, - data_frequency='minute', - initialize=initialize, - handle_data=handle_data, - analyze=analyze, - exchange_name='bitfinex', - algo_namespace=NAMESPACE, - base_currency='usd', - start=pd.to_datetime('2017-10-01', utc=True), - end=pd.to_datetime('2017-11-10', utc=True), - output=out - ) - log.info('saved perf stats: {}'.format(out)) - - elif MODE == 'live': - run_algorithm( - capital_base=0.5, - initialize=initialize, - handle_data=handle_data, - analyze=analyze, - exchange_name='bittrex', - live=True, - algo_namespace=NAMESPACE, - base_currency='usd', - live_graph=False - ) +.. literalinclude:: ../../catalyst/examples/mean_reversion_simple.py + :language: python .. image:: https://s3.amazonaws.com/enigmaco-docs/github.io/example_mean_reversion_simple.png @@ -763,8 +158,6 @@ strategy. Simple Universe ~~~~~~~~~~~~~~~ -Source code: `examples/simple_universe.py `_ - This example aims to provide an easy way for users to learn how to collect data from any given exchange and select a subset of the available currency pairs for trading. You simply need to specify the exchange and @@ -791,142 +184,10 @@ of the file: catalyst ingest-exchange -x bitfinex -f minute -.. code-block:: bash - - python simple_universe.py - -Credits: This code was originally submitted by `Abner Ayala-Acevedo -`_. Thank you! - -.. code-block:: python - - from datetime import timedelta - - import numpy as np - import pandas as pd - - from catalyst import run_algorithm - from catalyst.exchange.utils.exchange_utils import get_exchange_symbols - from catalyst.api import (symbols, ) - - - def initialize(context): - context.i = -1 # minute counter - context.exchange = context.exchanges.values()[0].name.lower() - context.base_currency = context.exchanges.values()[0].base_currency.lower() - - - def handle_data(context, data): - context.i += 1 - lookback_days = 7 # 7 days - - # current date & time in each iteration formatted into a string - now = data.current_dt - date, time = now.strftime('%Y-%m-%d %H:%M:%S').split(' ') - lookback_date = now - timedelta(days=lookback_days) - # keep only the date as a string, discard the time - lookback_date = lookback_date.strftime('%Y-%m-%d %H:%M:%S').split(' ')[0] - - one_day_in_minutes = 1440 # 60 * 24 assumes data_frequency='minute' - # update universe everyday at midnight - if not context.i % one_day_in_minutes: - context.universe = universe(context, lookback_date, date) - - # get data every 30 minutes - minutes = 30 - # get lookback_days of history data: that is 'lookback' number of bins - lookback = one_day_in_minutes / minutes * lookback_days - if not context.i % minutes and context.universe: - # we iterate for every pair in the current universe - for coin in context.coins: - pair = str(coin.symbol) - - # Get 30 minute interval OHLCV data. This is the standard data - # required for candlestick or indicators/signals. Return Pandas - # DataFrames. 30T means 30-minute re-sampling of one minute data. - # Adjust it to your desired time interval as needed. - opened = fill(data.history(coin, 'open', - bar_count=lookback, frequency='30T')).values - high = fill(data.history(coin, 'high', - bar_count=lookback, frequency='30T')).values - low = fill(data.history(coin, 'low', - bar_count=lookback, frequency='30T')).values - close = fill(data.history(coin, 'price', - bar_count=lookback, frequency='30T')).values - volume = fill(data.history(coin, 'volume', - bar_count=lookback, frequency='30T')).values - - # close[-1] is the last value in the set, which is the equivalent - # to current price (as in the most recent value) - # displays the minute price for each pair every 30 minutes - print('{now}: {pair} -\tO:{o},\tH:{h},\tL:{c},\tC{c},\tV:{v}'.format( - now=now, - pair=pair, - o=opened[-1], - h=high[-1], - l=low[-1], - c=close[-1], - v=volume[-1], - )) - - # ------------------------------------------------------------- - # --------------- Insert Your Strategy Here ------------------- - # ------------------------------------------------------------- - - - def analyze(context=None, results=None): - pass - - - # Get the universe for a given exchange and a given base_currency market - # Example: Poloniex BTC Market - def universe(context, lookback_date, current_date): - # get all the pairs for the given exchange - json_symbols = get_exchange_symbols(context.exchange) - # convert into a DataFrame for easier processing - df = pd.DataFrame.from_dict(json_symbols).transpose().astype(str) - df['base_currency'] = df.apply(lambda row: row.symbol.split('_')[1],axis=1) - df['market_currency'] = df.apply(lambda row: row.symbol.split('_')[0],axis=1) - - # Filter all the pairs to get only the ones for a given base_currency - df = df[df['base_currency'] == context.base_currency] - - # Filter all the pairs to ensure that pair existed in the current date range - df = df[df.start_date < lookback_date] - df = df[df.end_daily >= current_date] - context.coins = symbols(*df.symbol) # convert all the pairs to symbols - - return df.symbol.tolist() - - - # Replace all NA, NAN or infinite values with its nearest value - def fill(series): - if isinstance(series, pd.Series): - return series.replace([np.inf, -np.inf], np.nan).ffill().bfill() - elif isinstance(series, np.ndarray): - return pd.Series(series).replace( - [np.inf, -np.inf], np.nan - ).ffill().bfill().values - else: - return series - - - if __name__ == '__main__': - start_date = pd.to_datetime('2017-11-10', utc=True) - end_date = pd.to_datetime('2017-11-13', utc=True) - - performance = run_algorithm(start=start_date, end=end_date, - capital_base=100.0, # amount of base_currency - initialize=initialize, - handle_data=handle_data, - analyze=analyze, - exchange_name='bitfinex', - data_frequency='minute', - base_currency='btc', - live=False, - live_graph=False, - algo_namespace='simple_universe') +Source code: `examples/simple_universe.py `_ +.. literalinclude:: ../../catalyst/examples/simple_universe.py + :language: python .. _portfolio_optimization: @@ -940,135 +201,10 @@ use 180 days of historical data and rebalance every 30 days. This code was used in writting the following article: `Markowitz Portfolio Optimization for Cryptocurrencies `_. -.. code-block:: python +Source code: `examples/simple_universe.py `_ - ''' - You can run this code using the Python interpreter: - - $ python portfolio_optimization.py - ''' - - from __future__ import division - import os - import pytz - import numpy as np - import pandas as pd - from scipy.optimize import minimize - import matplotlib.pyplot as plt - from datetime import datetime - - from catalyst.api import record, symbol, symbols, order_target_percent - from catalyst.utils.run_algo import run_algorithm - - np.set_printoptions(threshold='nan', suppress=True) - - - def initialize(context): - # Portfolio assets list - context.assets = symbols('btc_usdt', 'eth_usdt', 'ltc_usdt', 'dash_usdt', - 'xmr_usdt') - context.nassets = len(context.assets) - # Set the time window that will be used to compute expected return - # and asset correlations - context.window = 180 - # Set the number of days between each portfolio rebalancing - context.rebalance_period = 30 - context.i = 0 - - - def handle_data(context, data): - # Only rebalance at the beggining of the algorithm execution and - # every multiple of the rebalance period - if context.i == 0 or context.i%context.rebalance_period == 0: - n = context.window - prices = data.history(context.assets, fields='price', - bar_count=n+1, frequency='1d') - pr = np.asmatrix(prices) - t_prices = prices.iloc[1:n+1] - t_val = t_prices.values - tminus_prices = prices.iloc[0:n] - tminus_val = tminus_prices.values - # Compute daily returns (r) - r = np.asmatrix(t_val/tminus_val-1) - # Compute the expected returns of each asset with the average - # daily return for the selected time window - m = np.asmatrix(np.mean(r, axis=0)) - # ### - stds = np.std(r, axis=0) - # Compute excess returns matrix (xr) - xr = r - m - # Matrix algebra to get variance-covariance matrix - cov_m = np.dot(np.transpose(xr),xr)/n - # Compute asset correlation matrix (informative only) - corr_m = cov_m/np.dot(np.transpose(stds),stds) - - # Define portfolio optimization parameters - n_portfolios = 50000 - results_array = np.zeros((3+context.nassets,n_portfolios)) - for p in xrange(n_portfolios): - weights = np.random.random(context.nassets) - weights /= np.sum(weights) - w = np.asmatrix(weights) - p_r = np.sum(np.dot(w,np.transpose(m)))*365 - p_std = np.sqrt(np.dot(np.dot(w,cov_m),np.transpose(w)))*np.sqrt(365) - - #store results in results array - results_array[0,p] = p_r - results_array[1,p] = p_std - #store Sharpe Ratio (return / volatility) - risk free rate element - #excluded for simplicity - results_array[2,p] = results_array[0,p] / results_array[1,p] - i = 0 - for iw in weights: - results_array[3+i,p] = weights[i] - i += 1 - - #convert results array to Pandas DataFrame - results_frame = pd.DataFrame(np.transpose(results_array), - columns=['r','stdev','sharpe']+context.assets) - #locate position of portfolio with highest Sharpe Ratio - max_sharpe_port = results_frame.iloc[results_frame['sharpe'].idxmax()] - #locate positon of portfolio with minimum standard deviation - min_vol_port = results_frame.iloc[results_frame['stdev'].idxmin()] - - #order optimal weights for each asset - for asset in context.assets: - if data.can_trade(asset): - order_target_percent(asset, max_sharpe_port[asset]) - - #create scatter plot coloured by Sharpe Ratio - plt.scatter(results_frame.stdev,results_frame.r,c=results_frame.sharpe,cmap='RdYlGn') - plt.xlabel('Volatility') - plt.ylabel('Returns') - plt.colorbar() - #plot red star to highlight position of portfolio with highest Sharpe Ratio - plt.scatter(max_sharpe_port[1],max_sharpe_port[0],marker='o',color='b',s=200) - #plot green star to highlight position of minimum variance portfolio - plt.show() - print(max_sharpe_port) - record(pr=pr,r=r, m=m, stds=stds ,max_sharpe_port=max_sharpe_port, corr_m=corr_m) - context.i += 1 - - - def analyze(context=None, results=None): - # Form DataFrame with selected data - data = results[['pr','r','m','stds','max_sharpe_port','corr_m','portfolio_value']] - - # Save results in CSV file - filename = os.path.splitext(os.path.basename(__file__))[0] - data.to_csv(filename + '.csv') - - - # Bitcoin data is available from 2015-3-2. Dates vary for other tokens. - start = datetime(2017, 1, 1, 0, 0, 0, 0, pytz.utc) - end = datetime(2017, 8, 16, 0, 0, 0, 0, pytz.utc) - results = run_algorithm(initialize=initialize, - handle_data=handle_data, - analyze=analyze, - start=start, - end=end, - exchange_name='poloniex', - capital_base=100000, ) +.. literalinclude:: ../../catalyst/examples/portfolio_optimization.py + :language: python .. image:: https://cdn-images-1.medium.com/max/1600/0*EjjiKZHlYF3sn7yQ. :align: center From d40585f56e286d923479a7b160dcac463e7720fa Mon Sep 17 00:00:00 2001 From: embaral Date: Thu, 22 Feb 2018 14:34:00 +0200 Subject: [PATCH 37/87] DOC: added an option "catalyst live --help" to the documentation. --- docs/source/live-trading.rst | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/docs/source/live-trading.rst b/docs/source/live-trading.rst index a2898d61..df449c05 100644 --- a/docs/source/live-trading.rst +++ b/docs/source/live-trading.rst @@ -175,5 +175,14 @@ Here is the breakdown of the new arguments: simulated in Catalyst instead of processed on the exchange. It defaults to ``True``. + +The `catalyst live` command offers additional parameters. +You can learn more by running the following from the command line: + +.. code-block:: bash + +catalyst live --help + + Here is a complete algorithm for reference: `Buy Low and Sell High `_ From 127878413e1a439ceb43864b5988de45a4f0a318 Mon Sep 17 00:00:00 2001 From: embaral Date: Thu, 22 Feb 2018 14:45:14 +0200 Subject: [PATCH 38/87] DOC: added an option "catalyst live --help" to the documentation. --- docs/source/live-trading.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/live-trading.rst b/docs/source/live-trading.rst index e65796f5..7d5f2394 100644 --- a/docs/source/live-trading.rst +++ b/docs/source/live-trading.rst @@ -190,7 +190,7 @@ You can learn more by running the following from the command line: .. code-block:: bash -catalyst live --help + catalyst live --help Here is a complete algorithm for reference: From fea2ed104e8aaf9796f815be984ef95a04ccbb5f Mon Sep 17 00:00:00 2001 From: lenak25 Date: Thu, 22 Feb 2018 16:50:27 +0200 Subject: [PATCH 39/87] BUG: fix issue #236: handle properly empty candles received from exchanges --- catalyst/exchange/exchange.py | 48 ++++++++++++++++++++--------------- catalyst/support/issue_236.py | 32 +++++++++++++++++++++++ 2 files changed, 59 insertions(+), 21 deletions(-) create mode 100644 catalyst/support/issue_236.py diff --git a/catalyst/exchange/exchange.py b/catalyst/exchange/exchange.py index 3b57256b..a0a247fb 100644 --- a/catalyst/exchange/exchange.py +++ b/catalyst/exchange/exchange.py @@ -1,4 +1,5 @@ import abc +import pytz from abc import ABCMeta, abstractmethod, abstractproperty from datetime import timedelta from time import sleep @@ -514,32 +515,37 @@ class Exchange: series = dict() for asset in candles: - first_candle = candles[asset][0] - asset_series = self.get_series_from_candles( - candles=candles[asset], - start_dt=first_candle['last_traded'], - end_dt=end_dt, - data_frequency=frequency, - field=field, - ) - - delta_candle_size = candle_size * 60 if unit == 'H' else candle_size - # Checking to make sure that the dates match - delta = get_delta(delta_candle_size, data_frequency) - adj_end_dt = end_dt - delta - last_traded = asset_series.index[-1] - - if last_traded < adj_end_dt: - raise LastCandleTooEarlyError( - last_traded=last_traded, - end_dt=adj_end_dt, - exchange=self.name, + if candles[asset]: + first_candle = candles[asset][0] + asset_series = self.get_series_from_candles( + candles=candles[asset], + start_dt=first_candle['last_traded'], + end_dt=end_dt, + data_frequency=frequency, + field=field, ) + delta_candle_size = candle_size * 60 if unit == 'H' else candle_size + # Checking to make sure that the dates match + delta = get_delta(delta_candle_size, data_frequency) + adj_end_dt = end_dt - delta + last_traded = asset_series.index[-1] + + if last_traded < adj_end_dt: + raise LastCandleTooEarlyError( + last_traded=last_traded, + end_dt=adj_end_dt, + exchange=self.name, + ) + else: # empty candle received + # because other assets are tz-aware, we need its tz to be set as well + asset_series = pd.Series([], index=pd.DatetimeIndex([], tz=pytz.utc)) + + series[asset] = asset_series df = pd.DataFrame(series) - df.dropna(inplace=True) + #df.dropna(inplace=True) # commented out due to issue 236 return df diff --git a/catalyst/support/issue_236.py b/catalyst/support/issue_236.py new file mode 100644 index 00000000..c3a437a9 --- /dev/null +++ b/catalyst/support/issue_236.py @@ -0,0 +1,32 @@ +from catalyst.api import symbol +from catalyst.utils.run_algo import run_algorithm + +coins = ['dash', 'btc', 'dash', 'etc', 'eth', 'ltc', 'nxt', 'rep', 'str', 'xmr', 'xrp', 'zec'] +symbols = None + + +def initialize(context): + pass + + +def _handle_data(context, data): + global symbols + if symbols is None: symbols = [symbol(c + '_usdt') for c in coins] + + print'getting history for: %s' % [s.symbol for s in symbols] + history = data.history(symbols, + ['close', 'volume'], + bar_count=1, # EXCEPTION, Change to 2 + frequency='5T') + #print 'history: %s' % history.shape + +run_algorithm(initialize=initialize, + handle_data=_handle_data, + analyze=lambda _, results: True, + exchange_name='poloniex', + base_currency='usdt', + algo_namespace='issue-236', + live=True, + data_frequency='minute', + capital_base=3000, + simulate_orders=True) \ No newline at end of file From 50310576f90bfb87a789114685ac0177ec794650 Mon Sep 17 00:00:00 2001 From: lenak25 Date: Thu, 22 Feb 2018 17:42:14 +0200 Subject: [PATCH 40/87] BUG:fix an issue with wrong timestamps seen at tests.exchange.test_suites.test_suite_bundle.TestSuiteBundle#test_validate_bundles (which issue #230 uncovered) --- catalyst/exchange/utils/exchange_utils.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/catalyst/exchange/utils/exchange_utils.py b/catalyst/exchange/utils/exchange_utils.py index fb24f1c8..4f82356c 100644 --- a/catalyst/exchange/utils/exchange_utils.py +++ b/catalyst/exchange/utils/exchange_utils.py @@ -722,12 +722,14 @@ def get_candles_df(candles, field, freq, bar_count, end_dt, values = [candle[field] for candle in candles[asset]] series = pd.Series(values, index=dates) + """ series = series.reindex( periods, method='ffill', fill_value=previous_value, ) series.sort_index(inplace=True) + """ all_series[asset] = series df = pd.DataFrame(all_series) From bfd7e4b2dd98a9878c20291469894bb0f07347fc Mon Sep 17 00:00:00 2001 From: Victor Date: Wed, 14 Feb 2018 09:27:53 -0700 Subject: [PATCH 41/87] Update python3.6-environment.yml --- etc/python3.6-environment.yml | 42 +++++++++++++++++------------------ 1 file changed, 21 insertions(+), 21 deletions(-) diff --git a/etc/python3.6-environment.yml b/etc/python3.6-environment.yml index 446198e0..0e5eabbf 100644 --- a/etc/python3.6-environment.yml +++ b/etc/python3.6-environment.yml @@ -2,28 +2,28 @@ name: catalyst channels: - defaults dependencies: -- ca-certificates=2017.08.26=ha1e5d58_0 -- certifi=2018.1.18=py36_0 -- intel-openmp=2018.0.0=h8158457_8 -- libcxx=4.0.1=h579ed51_0 -- libcxxabi=4.0.1=hebd6815_0 -- libedit=3.1=hb4e282d_0 -- libffi=3.2.1=h475c297_4 -- libgfortran=3.0.1=h93005f0_2 -- mkl=2018.0.1=hfbd8650_4 -- ncurses=6.0=hd04f020_2 -- numpy=1.14.0=py36h8a80b8c_1 -- openssl=1.0.2n=hdbc3d79_0 -- pip=9.0.1=py36h1555ced_4 -- python=3.6.4=hc167b69_1 -- readline=7.0=hc1231fa_4 -- scipy=1.0.0=py36h1de22e9_0 +- ca-certificates=2017.08.26 +- certifi=2018.1.18 +- intel-openmp=2018.0.0 +- libcxx=4.0.1 +- libcxxabi=4.0.1 +- libedit=3.1 +- libffi=3.2.1 +- libgfortran=3.0.1 +- mkl=2018.0.1 +- ncurses=6.0 +- numpy=1.14.0 +- openssl=1.0.2n +- pip=9.0.1 +- python=3.6.4 +- readline=7.0 +- scipy=1.0.0 - setuptools=38.4.0=py36_0 -- sqlite=3.22.0=h3efe00b_0 -- tk=8.6.7=h35a86e2_3 -- wheel=0.30.0=py36h5eb2c71_1 -- xz=5.2.3=h0278029_2 -- zlib=1.2.11=hf3cbc9b_2 +- sqlite=3.22.0 +- tk=8.6.7 +- wheel=0.30.0 +- xz=5.2.3 +- zlib=1.2.11 - pip: - aiodns==1.1.1 - aiohttp==3.0.1 From 8fe3ab344e89dade33ba306c15708aa8cf3f5f96 Mon Sep 17 00:00:00 2001 From: Victor Grau Serrat Date: Thu, 22 Feb 2018 12:54:29 -0700 Subject: [PATCH 42/87] MAINT: conda environment updates --- etc/python2.7-environment.yml | 2 ++ etc/python3.6-environment.yml | 9 ++------- 2 files changed, 4 insertions(+), 7 deletions(-) diff --git a/etc/python2.7-environment.yml b/etc/python2.7-environment.yml index 2037515b..b2e2486c 100644 --- a/etc/python2.7-environment.yml +++ b/etc/python2.7-environment.yml @@ -1,9 +1,11 @@ name: catalyst channels: - defaults +- conda-forge dependencies: - certifi=2016.2.28=py27_0 - mkl=2017.0.3 +- matplotlib=2.1.2=py36_0 - numpy=1.13.1=py27_0 - openssl=1.0.2l - pip=9.0.1=py27_1 diff --git a/etc/python3.6-environment.yml b/etc/python3.6-environment.yml index 0e5eabbf..c93f3c82 100644 --- a/etc/python3.6-environment.yml +++ b/etc/python3.6-environment.yml @@ -1,22 +1,17 @@ name: catalyst channels: - defaults +- conda-forge dependencies: - ca-certificates=2017.08.26 - certifi=2018.1.18 - intel-openmp=2018.0.0 -- libcxx=4.0.1 -- libcxxabi=4.0.1 -- libedit=3.1 -- libffi=3.2.1 -- libgfortran=3.0.1 - mkl=2018.0.1 -- ncurses=6.0 - numpy=1.14.0 - openssl=1.0.2n +- matplotlib=2.1.2=py36_0 - pip=9.0.1 - python=3.6.4 -- readline=7.0 - scipy=1.0.0 - setuptools=38.4.0=py36_0 - sqlite=3.22.0 From b4bd557273dd253b9adf4e02d68389a082d81f8f Mon Sep 17 00:00:00 2001 From: AvishaiW Date: Fri, 23 Feb 2018 00:38:51 +0200 Subject: [PATCH 43/87] BUG: fixes for issues #204 #237 -modified parameters for cancel_orders -update portfolio after any change in the orders before sync --- catalyst/examples/dual_moving_average.py | 7 +++---- catalyst/exchange/ccxt/ccxt_exchange.py | 6 ++++-- catalyst/exchange/exchange.py | 6 ++++-- catalyst/exchange/exchange_algorithm.py | 24 ++++++++++++++++++------ 4 files changed, 29 insertions(+), 14 deletions(-) diff --git a/catalyst/examples/dual_moving_average.py b/catalyst/examples/dual_moving_average.py index 00eb01af..88a6fad5 100644 --- a/catalyst/examples/dual_moving_average.py +++ b/catalyst/examples/dual_moving_average.py @@ -4,8 +4,7 @@ import pandas as pd from logbook import Logger from catalyst import run_algorithm -from catalyst.api import (record, symbol, order_target_percent, - get_open_orders) +from catalyst.api import (record, symbol, order_target_percent,) from catalyst.exchange.utils.stats_utils import extract_transactions NAMESPACE = 'dual_moving_average' @@ -21,7 +20,7 @@ def initialize(context): def handle_data(context, data): # define the windows for the moving averages short_window = 2 - long_window = 3 + long_window = 5 # Skip as many bars as long_window to properly compute the average context.i += 1 @@ -63,7 +62,7 @@ def handle_data(context, data): # Since we are using limit orders, some orders may not execute immediately # we wait until all orders are executed before considering more trades. - orders = get_open_orders(context.asset) + orders = context.blotter.open_orders if len(orders) > 0: return diff --git a/catalyst/exchange/ccxt/ccxt_exchange.py b/catalyst/exchange/ccxt/ccxt_exchange.py index 40f4b4d3..875661e9 100644 --- a/catalyst/exchange/ccxt/ccxt_exchange.py +++ b/catalyst/exchange/ccxt/ccxt_exchange.py @@ -980,7 +980,8 @@ class CCXT(Exchange): ) raise ExchangeRequestError(error=e) - def cancel_order(self, order_param, asset_or_symbol=None): + def cancel_order(self, order_param, + asset_or_symbol=None, params={}): order_id = order_param.id \ if isinstance(order_param, Order) else order_param @@ -992,7 +993,8 @@ class CCXT(Exchange): try: symbol = self.get_symbol(asset_or_symbol) \ if asset_or_symbol is not None else None - self.api.cancel_order(id=order_id, symbol=symbol) + self.api.cancel_order(id=order_id, + symbol=symbol, params= params) except (ExchangeError, NetworkError) as e: log.warn( diff --git a/catalyst/exchange/exchange.py b/catalyst/exchange/exchange.py index a0a247fb..20cbe967 100644 --- a/catalyst/exchange/exchange.py +++ b/catalyst/exchange/exchange.py @@ -705,7 +705,7 @@ class Exchange: balances=balances, amount=cash, ) - if is_lower and not open_orders: + if is_lower: raise NotEnoughCashError( currency=self.base_currency, exchange=self.name, @@ -932,7 +932,8 @@ class Exchange: """ @abstractmethod - def cancel_order(self, order_param, symbol_or_asset=None): + def cancel_order(self, order_param, + symbol_or_asset=None, params={}): """Cancel an open order. Parameters @@ -941,6 +942,7 @@ class Exchange: The order_id or order object to cancel. symbol_or_asset: str|TradingPair The catalyst symbol, some exchanges need this + params: """ pass diff --git a/catalyst/exchange/exchange_algorithm.py b/catalyst/exchange/exchange_algorithm.py index c827af4a..5bf103fa 100644 --- a/catalyst/exchange/exchange_algorithm.py +++ b/catalyst/exchange/exchange_algorithm.py @@ -388,6 +388,7 @@ class ExchangeTradingAlgorithmLive(ExchangeTradingAlgorithmBase): self.stats_minutes = 1 self._last_orders = [] + self._last_open_orders = [] self.trading_client = None super(ExchangeTradingAlgorithmLive, self).__init__(*args, **kwargs) @@ -791,12 +792,17 @@ class ExchangeTradingAlgorithmLive(ExchangeTradingAlgorithmBase): self.nullify_frame_stats(now=data.current_dt) self.performance_needs_update = False - orders = list(self.perf_tracker.todays_performance.orders_by_id.keys()) - if orders != self._last_orders: + last_orders_list = list(self.blotter.orders.keys()) + open_orders_list = list(self.blotter.open_orders.keys()) + + if last_orders_list != self._last_orders or \ + open_orders_list != self._last_open_orders: self.performance_needs_update = True - # Saving current orders to detect changes in the next frame - self._last_orders = copy.deepcopy(orders) + # Saving current order positions + # to detect changes in the next frame + self._last_orders = copy.deepcopy(last_orders_list) + self._last_open_orders = copy.deepcopy(open_orders_list) if self.performance_needs_update: self.perf_tracker.update_performance() @@ -1011,13 +1017,19 @@ class ExchangeTradingAlgorithmLive(ExchangeTradingAlgorithmBase): args=(order_id,)) @api_method - def cancel_order(self, order_param, exchange_name): + def cancel_order(self, order_param, exchange_name, + symbol=None, params={}): """Cancel an open order. Parameters ---------- order_param : str or Order The order_id or order object to cancel. + + exchange_name: name of exchange from + which you want to cancel the order + symbol: + params: """ exchange = self.exchanges[exchange_name] @@ -1031,4 +1043,4 @@ class ExchangeTradingAlgorithmLive(ExchangeTradingAlgorithmBase): sleeptime=self.attempts['retry_sleeptime'], retry_exceptions=(ExchangeRequestError,), cleanup=lambda: log.warn('cancelling order again.'), - args=(order_id,)) + args=(order_id, symbol, params)) From 388535b09c64c95f565397b2736b8576a7c73cab Mon Sep 17 00:00:00 2001 From: Victor Grau Serrat Date: Thu, 22 Feb 2018 22:09:51 -0700 Subject: [PATCH 44/87] BUG: reverts changed introduced in 00f232e2d7426b8593295ee3e4e6488a39a4866b --- catalyst/examples/dual_moving_average.py | 39 +++++++++--------------- 1 file changed, 14 insertions(+), 25 deletions(-) diff --git a/catalyst/examples/dual_moving_average.py b/catalyst/examples/dual_moving_average.py index 88a6fad5..f11e6b77 100644 --- a/catalyst/examples/dual_moving_average.py +++ b/catalyst/examples/dual_moving_average.py @@ -19,8 +19,8 @@ def initialize(context): def handle_data(context, data): # define the windows for the moving averages - short_window = 2 - long_window = 5 + short_window = 50 + long_window = 200 # Skip as many bars as long_window to properly compute the average context.i += 1 @@ -149,27 +149,16 @@ def analyze(context, perf): if __name__ == '__main__': + run_algorithm( - capital_base=1000, - data_frequency='minute', - initialize=initialize, - handle_data=handle_data, - analyze=analyze, - exchange_name='bitfinex', - algo_namespace=NAMESPACE, - base_currency='usd', - simulate_orders=True, - live=True, - ) - # run_algorithm( - # capital_base=1000, - # data_frequency='minute', - # initialize=initialize, - # handle_data=handle_data, - # analyze=analyze, - # exchange_name='bitfinex', - # algo_namespace=NAMESPACE, - # base_currency='usd', - # start=pd.to_datetime('2017-9-22', utc=True), - # end=pd.to_datetime('2017-9-23', utc=True), - # ) + capital_base=1000, + data_frequency='minute', + initialize=initialize, + handle_data=handle_data, + analyze=analyze, + exchange_name='bitfinex', + algo_namespace=NAMESPACE, + base_currency='usd', + start=pd.to_datetime('2017-9-22', utc=True), + end=pd.to_datetime('2017-9-23', utc=True), + ) From 8587fee0cea0c8bcd51ccbfc36fc68fce2a6b2f1 Mon Sep 17 00:00:00 2001 From: AvishaiW Date: Sun, 25 Feb 2018 11:29:26 +0200 Subject: [PATCH 45/87] BUG: revert previous changes #249 --- catalyst/exchange/exchange.py | 8 ++------ catalyst/exchange/exchange_algorithm.py | 19 +++++++++---------- 2 files changed, 11 insertions(+), 16 deletions(-) diff --git a/catalyst/exchange/exchange.py b/catalyst/exchange/exchange.py index 20cbe967..310d12fc 100644 --- a/catalyst/exchange/exchange.py +++ b/catalyst/exchange/exchange.py @@ -662,20 +662,16 @@ class Exchange: return df - def _check_low_balance(self, currency, balances, amount, open_orders=None): + def _check_low_balance(self, currency, balances, amount): free = balances[currency]['free'] if currency in balances else 0.0 - if open_orders: - # TODO: make sure that this works - free += sum([order.amount for order in open_orders]) - if free < amount: return free, True else: return free, False - def sync_positions(self, positions, open_orders=None, cash=None, + def sync_positions(self, positions, cash=None, check_balances=False): """ Update the portfolio cash and position balances based on the diff --git a/catalyst/exchange/exchange_algorithm.py b/catalyst/exchange/exchange_algorithm.py index 5bf103fa..98966a82 100644 --- a/catalyst/exchange/exchange_algorithm.py +++ b/catalyst/exchange/exchange_algorithm.py @@ -638,6 +638,15 @@ class ExchangeTradingAlgorithmLive(ExchangeTradingAlgorithmBase): if asset_orders: orders += asset_orders + required_cash = self.portfolio.cash if not orders else None + cash, positions_value = exchange.sync_positions( + positions=exchange_positions, + check_balances=check_balances, + cash=required_cash, + ) + total_cash += cash + total_positions_value += positions_value + # Applying modifications to the original positions for position in exchange_positions: tracker.update_position( @@ -647,16 +656,6 @@ class ExchangeTradingAlgorithmLive(ExchangeTradingAlgorithmBase): last_sale_price=position.last_sale_price, ) - required_cash = self.portfolio.cash if not orders else None - cash, positions_value = exchange.sync_positions( - positions=exchange_positions, - open_orders=orders, - check_balances=check_balances, - cash=required_cash, - ) - total_cash += cash - total_positions_value += positions_value - if not check_balances: total_cash = self.portfolio.cash From e5870ea60a9b99aae256fa006620d84063a25dad Mon Sep 17 00:00:00 2001 From: AvishaiW Date: Mon, 26 Feb 2018 09:21:00 +0200 Subject: [PATCH 46/87] BUG: fix #252 #253 and split state into paper and live --- catalyst/exchange/exchange_algorithm.py | 51 ++++++++++++++++------- catalyst/exchange/utils/exchange_utils.py | 23 +++++----- catalyst/exchange/utils/stats_utils.py | 6 ++- 3 files changed, 53 insertions(+), 27 deletions(-) diff --git a/catalyst/exchange/exchange_algorithm.py b/catalyst/exchange/exchange_algorithm.py index 98966a82..b9c319f1 100644 --- a/catalyst/exchange/exchange_algorithm.py +++ b/catalyst/exchange/exchange_algorithm.py @@ -375,13 +375,23 @@ class ExchangeTradingAlgorithmLive(ExchangeTradingAlgorithmBase): if error: log.warning(error) - self.pnl_stats = get_algo_df(self.algo_namespace, 'pnl_stats') + # in order to save paper & live files separately + self.mode_name = 'paper' if kwargs['simulate_orders'] else 'live' - self.custom_signals_stats = \ - get_algo_df(self.algo_namespace, 'custom_signals_stats') + self.pnl_stats = get_algo_df( + self.algo_namespace, + 'pnl_stats_{}'.format(self.mode_name), + ) - self.exposure_stats = \ - get_algo_df(self.algo_namespace, 'exposure_stats') + self.custom_signals_stats = get_algo_df( + self.algo_namespace, + 'custom_signals_stats_{}'.format(self.mode_name) + ) + + self.exposure_stats = get_algo_df( + self.algo_namespace, + 'exposure_stats_{}'.format(self.mode_name) + ) self.is_running = True @@ -515,7 +525,7 @@ class ExchangeTradingAlgorithmLive(ExchangeTradingAlgorithmBase): """ self.state = get_algo_object( algo_name=self.algo_namespace, - key='context.state', + key='context.state_{}'.format(self.mode_name), ) if self.state is None: self.state = {} @@ -538,7 +548,7 @@ class ExchangeTradingAlgorithmLive(ExchangeTradingAlgorithmBase): # Unpacking the perf_tracker and positions if available cum_perf = get_algo_object( algo_name=self.algo_namespace, - key='cumulative_performance', + key='cumulative_performance_{}'.format(self.mode_name), ) if cum_perf is not None: tracker.cumulative_performance = cum_perf @@ -549,7 +559,7 @@ class ExchangeTradingAlgorithmLive(ExchangeTradingAlgorithmBase): todays_perf = get_algo_object( algo_name=self.algo_namespace, key=today.strftime('%Y-%m-%d'), - rel_path='daily_performance', + rel_path='daily_performance_{}'.format(self.mode_name), ) if todays_perf is not None: # Ensure single common position tracker @@ -686,7 +696,11 @@ class ExchangeTradingAlgorithmLive(ExchangeTradingAlgorithmBase): ) self.pnl_stats = pd.concat([self.pnl_stats, df]) - save_algo_df(self.algo_namespace, 'pnl_stats', self.pnl_stats) + save_algo_df( + self.algo_namespace, + 'pnl_stats_{}'.format(self.mode_name), + self.pnl_stats, + ) def add_custom_signals_stats(self, period_stats): """ @@ -707,8 +721,11 @@ class ExchangeTradingAlgorithmLive(ExchangeTradingAlgorithmBase): ) self.custom_signals_stats = pd.concat([self.custom_signals_stats, df]) - save_algo_df(self.algo_namespace, 'custom_signals_stats', - self.custom_signals_stats) + save_algo_df( + self.algo_namespace, + 'custom_signals_stats_{}'.format(self.mode_name), + self.custom_signals_stats, + ) def add_exposure_stats(self, period_stats): """ @@ -735,7 +752,9 @@ class ExchangeTradingAlgorithmLive(ExchangeTradingAlgorithmBase): self.exposure_stats = pd.concat([self.exposure_stats, df]) save_algo_df( - self.algo_namespace, 'exposure_stats', self.exposure_stats + self.algo_namespace, + 'exposure_stats_{}'.format(self.mode_name), + self.exposure_stats ) def nullify_frame_stats(self, now): @@ -759,6 +778,7 @@ class ExchangeTradingAlgorithmLive(ExchangeTradingAlgorithmBase): obj=self.frame_stats, rel_path='frame_stats' ) + error = remove_old_files( algo_name=self.algo_namespace, today=now, @@ -843,7 +863,7 @@ class ExchangeTradingAlgorithmLive(ExchangeTradingAlgorithmBase): log.debug('saving cumulative performance object') save_algo_object( algo_name=self.algo_namespace, - key='cumulative_performance', + key='cumulative_performance_{}'.format(self.mode_name), obj=self.perf_tracker.cumulative_performance, ) log.debug('saving todays performance object') @@ -851,12 +871,12 @@ class ExchangeTradingAlgorithmLive(ExchangeTradingAlgorithmBase): algo_name=self.algo_namespace, key=today.strftime('%Y-%m-%d'), obj=self.perf_tracker.todays_performance, - rel_path='daily_performance' + rel_path='daily_performance_{}'.format(self.mode_name) ) log.debug('saving context.state object') save_algo_object( algo_name=self.algo_namespace, - key='context.state', + key='context.state_{}'.format(self.mode_name), obj=self.state) def _process_stats(self, data): @@ -912,6 +932,7 @@ class ExchangeTradingAlgorithmLive(ExchangeTradingAlgorithmBase): csv_bytes = stats_to_algo_folder( stats=self.frame_stats, algo_namespace=self.algo_namespace, + folder_name='stats_{}'.format(self.mode_name), recorded_cols=recorded_cols, ) except Exception as e: diff --git a/catalyst/exchange/utils/exchange_utils.py b/catalyst/exchange/utils/exchange_utils.py index 4f82356c..5c40a26d 100644 --- a/catalyst/exchange/utils/exchange_utils.py +++ b/catalyst/exchange/utils/exchange_utils.py @@ -420,7 +420,7 @@ def clear_frame_stats_directory(algo_name): return error -def remove_old_files(algo_name, today, rel_path): +def remove_old_files(algo_name, today, rel_path, environ=None): """ remove old files from a directory to avoid overloading the disk @@ -430,27 +430,30 @@ def remove_old_files(algo_name, today, rel_path): algo_name: str today: Timestamp rel_path: str + environ: Returns ------- error: str """ + error = None - algo_folder = get_algo_folder(algo_name) + algo_folder = get_algo_folder(algo_name, environ) folder = os.path.join(algo_folder, rel_path) + ensure_directory(folder) # run on all files in the folder for f in os.listdir(folder): - creation_unix = os.path.getctime(f) - creation_time = pd.to_datetime(creation_unix, unit='s', ) + try: + creation_unix = os.path.getctime(os.path.join(folder, f)) + creation_time = pd.to_datetime(creation_unix, unit='s', ) - # if the file is older than 30 days erase it - if today - pd.DateOffset(30) > creation_time: - try: - os.unlink(f) - except OSError: - error = 'unable to erase files in {}'.format(folder) + # if the file is older than 30 days erase it + if today - pd.DateOffset(30) > creation_time: + os.unlink(f) + except OSError: + error = 'unable to erase files in {}'.format(folder) return error diff --git a/catalyst/exchange/utils/stats_utils.py b/catalyst/exchange/utils/stats_utils.py index 6e2aab0b..3db79d3b 100644 --- a/catalyst/exchange/utils/stats_utils.py +++ b/catalyst/exchange/utils/stats_utils.py @@ -396,7 +396,8 @@ def email_error(algo_name, dt, e, environ=None): )}) -def stats_to_algo_folder(stats, algo_namespace, recorded_cols=None): +def stats_to_algo_folder(stats, algo_namespace, + folder_name, recorded_cols=None): """ Saves the performance stats to the algo local folder. @@ -404,6 +405,7 @@ def stats_to_algo_folder(stats, algo_namespace, recorded_cols=None): ---------- stats: list[Object] algo_namespace: str + folder_name: str recorded_cols: list[str] Returns @@ -416,7 +418,7 @@ def stats_to_algo_folder(stats, algo_namespace, recorded_cols=None): timestr = time.strftime('%Y%m%d') folder = get_algo_folder(algo_namespace) - stats_folder = os.path.join(folder, 'stats') + stats_folder = os.path.join(folder, folder_name) ensure_directory(stats_folder) filename = os.path.join(stats_folder, '{}.csv'.format(timestr)) From 497212383abf3d3730fd45c586e0df87f8d64ab7 Mon Sep 17 00:00:00 2001 From: Matt Bornski Date: Mon, 26 Feb 2018 15:44:32 -0800 Subject: [PATCH 47/87] Python 3 returns bytes, the parsing functions are looking for strings --- catalyst/marketplace/marketplace.py | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/catalyst/marketplace/marketplace.py b/catalyst/marketplace/marketplace.py index a1ac263c..37ed2ec6 100644 --- a/catalyst/marketplace/marketplace.py +++ b/catalyst/marketplace/marketplace.py @@ -40,6 +40,13 @@ else: log = logbook.Logger('Marketplace', level=LOG_LEVEL) +def first_line_from_url(url): + request = urllib.urlopen(url) + first_line = request.readline() + if sys.version_info.major >= 3: + charset = request.info().get_content_charset() + first_line = first_line.decode(charset) + return first_line.strip() class Marketplace: def __init__(self): @@ -57,10 +64,8 @@ class Marketplace: self.web3 = Web3(HTTPProvider(ETH_REMOTE_NODE)) - contract_url = urllib.urlopen(MARKETPLACE_CONTRACT) - self.mkt_contract_address = Web3.toChecksumAddress( - contract_url.readline().strip()) + first_line_from_url(MARKETPLACE_CONTRACT)) abi_url = urllib.urlopen(MARKETPLACE_CONTRACT_ABI) abi = json.load(abi_url) @@ -70,10 +75,8 @@ class Marketplace: abi=abi, ) - contract_url = urllib.urlopen(ENIGMA_CONTRACT) - self.eng_contract_address = Web3.toChecksumAddress( - contract_url.readline().strip()) + first_line_from_url(ENIGMA_CONTRACT)) abi_url = urllib.urlopen(ENIGMA_CONTRACT_ABI) abi = json.load(abi_url) From cfafafb8fc481ef937ad1e55f40a27c1ee3052a4 Mon Sep 17 00:00:00 2001 From: Avishai Weingarten <33716232+AvishaiW@users.noreply.github.com> Date: Tue, 27 Feb 2018 09:47:02 +0200 Subject: [PATCH 48/87] BUG #252 fixed utc time and file erased --- catalyst/exchange/utils/exchange_utils.py | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/catalyst/exchange/utils/exchange_utils.py b/catalyst/exchange/utils/exchange_utils.py index 5c40a26d..658a7002 100644 --- a/catalyst/exchange/utils/exchange_utils.py +++ b/catalyst/exchange/utils/exchange_utils.py @@ -446,12 +446,13 @@ def remove_old_files(algo_name, today, rel_path, environ=None): # run on all files in the folder for f in os.listdir(folder): try: - creation_unix = os.path.getctime(os.path.join(folder, f)) - creation_time = pd.to_datetime(creation_unix, unit='s', ) + file_path = os.path.join(folder, f) + creation_unix = os.path.getctime(file_path) + creation_time = pd.to_datetime(creation_unix, unit='s', utc=True) # if the file is older than 30 days erase it if today - pd.DateOffset(30) > creation_time: - os.unlink(f) + os.unlink(file_path) except OSError: error = 'unable to erase files in {}'.format(folder) From 46e1a87a3def5e27584819f4e3f72b46cf145078 Mon Sep 17 00:00:00 2001 From: Avishai Weingarten <33716232+AvishaiW@users.noreply.github.com> Date: Tue, 27 Feb 2018 18:47:43 +0200 Subject: [PATCH 49/87] DOC: added troubleshooting for python3 --- docs/source/install.rst | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/docs/source/install.rst b/docs/source/install.rst index 5c24159f..6ef366b5 100644 --- a/docs/source/install.rst +++ b/docs/source/install.rst @@ -89,7 +89,7 @@ Once either Conda or MiniConda has been set up you can install Catalyst: .. code-block:: bash - conda env create -f python2.7-environment.yml + conda env create -f python2.7-environment.yml 4. Activate the environment (which you need to do every time you start a new session to run Catalyst): @@ -132,10 +132,19 @@ with the following steps: conda env remove --name catalyst 2. Create the environment: + + for python 2.7: .. code-block:: bash conda create --name catalyst python=2.7 scipy zlib + + or for python 3.6: + + .. code-block:: bash + + conda create --name catalyst python=3.6 scipy zlib + 3. Activate the environment: From 37c1057ab65643e0ac7da9b7ad1b357e237227b9 Mon Sep 17 00:00:00 2001 From: AvishaiW Date: Thu, 1 Mar 2018 01:09:35 +0200 Subject: [PATCH 50/87] BLD: added a cmd for running on the cloud (WIP) --- catalyst/__main__.py | 174 +++++++++++++++++++++++++++++++++++ catalyst/utils/run_server.py | 65 +++++++++++++ 2 files changed, 239 insertions(+) create mode 100644 catalyst/utils/run_server.py diff --git a/catalyst/__main__.py b/catalyst/__main__.py index 1b08f69d..6df67fde 100644 --- a/catalyst/__main__.py +++ b/catalyst/__main__.py @@ -14,6 +14,7 @@ from catalyst.exchange.exchange_bundle import ExchangeBundle from catalyst.exchange.utils.exchange_utils import delete_algo_folder from catalyst.utils.cli import Date, Timestamp from catalyst.utils.run_algo import _run, load_extensions +from catalyst.utils.run_server import run_server try: __IPYTHON__ @@ -505,6 +506,179 @@ def live(ctx, return perf +@main.command(name='serve-live') +@click.option( + '-f', + '--algofile', + default=None, + type=click.File('r'), + help='The file that contains the algorithm to run.', +) +@click.option( + '--capital-base', + type=float, + show_default=True, + help='The amount of capital (in base_currency) allocated to trading.', +) +@click.option( + '-t', + '--algotext', + help='The algorithm script to run.', +) +@click.option( + '-D', + '--define', + multiple=True, + help="Define a name to be bound in the namespace before executing" + " the algotext. For example '-Dname=value'. The value may be" + " any python expression. These are evaluated in order so they" + " may refer to previously defined names.", +) +@click.option( + '-o', + '--output', + default='-', + metavar='FILENAME', + show_default=True, + help="The location to write the perf data. If this is '-' the perf will" + " be written to stdout.", +) +@click.option( + '--print-algo/--no-print-algo', + is_flag=True, + default=False, + help='Print the algorithm to stdout.', +) +@ipython_only(click.option( + '--local-namespace/--no-local-namespace', + is_flag=True, + default=None, + help='Should the algorithm methods be resolved in the local namespace.' +)) +@click.option( + '-x', + '--exchange-name', + help='The name of the targeted exchange.', +) +@click.option( + '-n', + '--algo-namespace', + help='A label assigned to the algorithm for data storage purposes.' +) +@click.option( + '-c', + '--base-currency', + help='The base currency used to calculate statistics ' + '(e.g. usd, btc, eth).', +) +@click.option( + '-e', + '--end', + type=Date(tz='utc', as_timestamp=True), + help='An optional end date at which to stop the execution.', +) +@click.option( + '--live-graph/--no-live-graph', + is_flag=True, + default=False, + help='Display live graph.', +) +@click.option( + '--simulate-orders/--no-simulate-orders', + is_flag=True, + default=True, + help='Simulating orders enable the paper trading mode. No orders will be ' + 'sent to the exchange unless set to false.', +) +@click.option( + '--auth-aliases', + default=None, + help='Authentication file aliases for the specified exchanges. By default,' + 'each exchange uses the "auth.json" file in the exchange folder. ' + 'Specifying an "auth2" alias would use "auth2.json". It should be ' + 'specified like this: "[exchange_name],[alias],..." For example, ' + '"binance,auth2" or "binance,auth2,bittrex,auth2".', +) +@click.pass_context +def serve_live(ctx, + algofile, + capital_base, + algotext, + define, + output, + print_algo, + local_namespace, + exchange_name, + algo_namespace, + base_currency, + end, + live_graph, + auth_aliases, + simulate_orders): + """Trade live with the given algorithm on the server. + """ + if (algotext is not None) == (algofile is not None): + ctx.fail( + "must specify exactly one of '-f' / '--algofile' or" + " '-t' / '--algotext'", + ) + + if exchange_name is None: + ctx.fail("must specify an exchange name '-x'") + + if algo_namespace is None: + ctx.fail("must specify an algorithm name '-n' in live execution mode") + + if base_currency is None: + ctx.fail("must specify a base currency '-c' in live execution mode") + + if capital_base is None: + ctx.fail("must specify a capital base with '--capital-base'") + + if simulate_orders: + click.echo('Running in paper trading mode.', sys.stdout) + + else: + click.echo('Running in live trading mode.', sys.stdout) + + perf = run_server( + initialize=None, + handle_data=None, + before_trading_start=None, + analyze=None, + algofile=algofile, + algotext=algotext, + defines=define, + data_frequency=None, + capital_base=capital_base, + data=None, + bundle=None, + bundle_timestamp=None, + start=None, + end=end, + output=output, + print_algo=print_algo, + local_namespace=local_namespace, + environ=os.environ, + live=True, + exchange=exchange_name, + algo_namespace=algo_namespace, + base_currency=base_currency, + live_graph=live_graph, + analyze_live=None, + simulate_orders=simulate_orders, + auth_aliases=auth_aliases, + stats_output=None, + ) + + if output == '-': + click.echo(str(perf), sys.stdout) + elif output != os.devnull: # make the catalyst magic not write any data + perf.to_pickle(output) + + return perf + + @main.command(name='ingest-exchange') @click.option( '-x', diff --git a/catalyst/utils/run_server.py b/catalyst/utils/run_server.py new file mode 100644 index 00000000..2d4e7d02 --- /dev/null +++ b/catalyst/utils/run_server.py @@ -0,0 +1,65 @@ +import requests +import base64 +import json + + +def run_server(handle_data, + initialize, + before_trading_start, + analyze, + algofile, + algotext, + defines, + data_frequency, + capital_base, + data, + bundle, + bundle_timestamp, + start, + end, + output, + print_algo, + local_namespace, + environ, + live, + exchange, + algo_namespace, + base_currency, + live_graph, + analyze_live, + simulate_orders, + auth_aliases, + stats_output): + + json_file = {'arguments': { + 'handle_data': handle_data, + 'initialize': initialize, + 'before_trading_start': before_trading_start, + 'analyze': analyze, + 'algofile': base64.b64encode(algofile.read()), + 'algotext': algotext, + 'defines': defines, + 'data_frequency': data_frequency, + 'capital_base': capital_base, + 'data': data, + 'bundle': bundle, + 'bundle_timestamp': bundle_timestamp, + 'start': start, + 'end': end, + 'output': output, + 'print_algo': print_algo, + 'local_namespace': local_namespace, + 'environ': None, + 'live': False, + 'exchange': exchange, + 'algo_namespace': algo_namespace, + 'base_currency': base_currency, + 'live_graph': live_graph, + 'analyze_live': analyze_live, + 'simulate_orders': simulate_orders, + 'auth_aliases': auth_aliases, + 'stats_output': stats_output, + }} + + url = 'http://127.0.0.1:5000/todo/api/v1.0/tasks' + response = requests.post(url, json=json_file) From b85219d5b4f8ad5b99774bbf7a6dd2e6ff18a6d5 Mon Sep 17 00:00:00 2001 From: Victor Grau Serrat Date: Wed, 28 Feb 2018 18:06:36 -0700 Subject: [PATCH 51/87] MAINT: undoing last 2 unwanted commits --- catalyst/__main__.py | 174 ---------------------- catalyst/exchange/utils/exchange_utils.py | 7 +- catalyst/utils/run_server.py | 65 -------- docs/source/install.rst | 11 +- 4 files changed, 4 insertions(+), 253 deletions(-) delete mode 100644 catalyst/utils/run_server.py diff --git a/catalyst/__main__.py b/catalyst/__main__.py index 6df67fde..1b08f69d 100644 --- a/catalyst/__main__.py +++ b/catalyst/__main__.py @@ -14,7 +14,6 @@ from catalyst.exchange.exchange_bundle import ExchangeBundle from catalyst.exchange.utils.exchange_utils import delete_algo_folder from catalyst.utils.cli import Date, Timestamp from catalyst.utils.run_algo import _run, load_extensions -from catalyst.utils.run_server import run_server try: __IPYTHON__ @@ -506,179 +505,6 @@ def live(ctx, return perf -@main.command(name='serve-live') -@click.option( - '-f', - '--algofile', - default=None, - type=click.File('r'), - help='The file that contains the algorithm to run.', -) -@click.option( - '--capital-base', - type=float, - show_default=True, - help='The amount of capital (in base_currency) allocated to trading.', -) -@click.option( - '-t', - '--algotext', - help='The algorithm script to run.', -) -@click.option( - '-D', - '--define', - multiple=True, - help="Define a name to be bound in the namespace before executing" - " the algotext. For example '-Dname=value'. The value may be" - " any python expression. These are evaluated in order so they" - " may refer to previously defined names.", -) -@click.option( - '-o', - '--output', - default='-', - metavar='FILENAME', - show_default=True, - help="The location to write the perf data. If this is '-' the perf will" - " be written to stdout.", -) -@click.option( - '--print-algo/--no-print-algo', - is_flag=True, - default=False, - help='Print the algorithm to stdout.', -) -@ipython_only(click.option( - '--local-namespace/--no-local-namespace', - is_flag=True, - default=None, - help='Should the algorithm methods be resolved in the local namespace.' -)) -@click.option( - '-x', - '--exchange-name', - help='The name of the targeted exchange.', -) -@click.option( - '-n', - '--algo-namespace', - help='A label assigned to the algorithm for data storage purposes.' -) -@click.option( - '-c', - '--base-currency', - help='The base currency used to calculate statistics ' - '(e.g. usd, btc, eth).', -) -@click.option( - '-e', - '--end', - type=Date(tz='utc', as_timestamp=True), - help='An optional end date at which to stop the execution.', -) -@click.option( - '--live-graph/--no-live-graph', - is_flag=True, - default=False, - help='Display live graph.', -) -@click.option( - '--simulate-orders/--no-simulate-orders', - is_flag=True, - default=True, - help='Simulating orders enable the paper trading mode. No orders will be ' - 'sent to the exchange unless set to false.', -) -@click.option( - '--auth-aliases', - default=None, - help='Authentication file aliases for the specified exchanges. By default,' - 'each exchange uses the "auth.json" file in the exchange folder. ' - 'Specifying an "auth2" alias would use "auth2.json". It should be ' - 'specified like this: "[exchange_name],[alias],..." For example, ' - '"binance,auth2" or "binance,auth2,bittrex,auth2".', -) -@click.pass_context -def serve_live(ctx, - algofile, - capital_base, - algotext, - define, - output, - print_algo, - local_namespace, - exchange_name, - algo_namespace, - base_currency, - end, - live_graph, - auth_aliases, - simulate_orders): - """Trade live with the given algorithm on the server. - """ - if (algotext is not None) == (algofile is not None): - ctx.fail( - "must specify exactly one of '-f' / '--algofile' or" - " '-t' / '--algotext'", - ) - - if exchange_name is None: - ctx.fail("must specify an exchange name '-x'") - - if algo_namespace is None: - ctx.fail("must specify an algorithm name '-n' in live execution mode") - - if base_currency is None: - ctx.fail("must specify a base currency '-c' in live execution mode") - - if capital_base is None: - ctx.fail("must specify a capital base with '--capital-base'") - - if simulate_orders: - click.echo('Running in paper trading mode.', sys.stdout) - - else: - click.echo('Running in live trading mode.', sys.stdout) - - perf = run_server( - initialize=None, - handle_data=None, - before_trading_start=None, - analyze=None, - algofile=algofile, - algotext=algotext, - defines=define, - data_frequency=None, - capital_base=capital_base, - data=None, - bundle=None, - bundle_timestamp=None, - start=None, - end=end, - output=output, - print_algo=print_algo, - local_namespace=local_namespace, - environ=os.environ, - live=True, - exchange=exchange_name, - algo_namespace=algo_namespace, - base_currency=base_currency, - live_graph=live_graph, - analyze_live=None, - simulate_orders=simulate_orders, - auth_aliases=auth_aliases, - stats_output=None, - ) - - if output == '-': - click.echo(str(perf), sys.stdout) - elif output != os.devnull: # make the catalyst magic not write any data - perf.to_pickle(output) - - return perf - - @main.command(name='ingest-exchange') @click.option( '-x', diff --git a/catalyst/exchange/utils/exchange_utils.py b/catalyst/exchange/utils/exchange_utils.py index 658a7002..5c40a26d 100644 --- a/catalyst/exchange/utils/exchange_utils.py +++ b/catalyst/exchange/utils/exchange_utils.py @@ -446,13 +446,12 @@ def remove_old_files(algo_name, today, rel_path, environ=None): # run on all files in the folder for f in os.listdir(folder): try: - file_path = os.path.join(folder, f) - creation_unix = os.path.getctime(file_path) - creation_time = pd.to_datetime(creation_unix, unit='s', utc=True) + creation_unix = os.path.getctime(os.path.join(folder, f)) + creation_time = pd.to_datetime(creation_unix, unit='s', ) # if the file is older than 30 days erase it if today - pd.DateOffset(30) > creation_time: - os.unlink(file_path) + os.unlink(f) except OSError: error = 'unable to erase files in {}'.format(folder) diff --git a/catalyst/utils/run_server.py b/catalyst/utils/run_server.py deleted file mode 100644 index 2d4e7d02..00000000 --- a/catalyst/utils/run_server.py +++ /dev/null @@ -1,65 +0,0 @@ -import requests -import base64 -import json - - -def run_server(handle_data, - initialize, - before_trading_start, - analyze, - algofile, - algotext, - defines, - data_frequency, - capital_base, - data, - bundle, - bundle_timestamp, - start, - end, - output, - print_algo, - local_namespace, - environ, - live, - exchange, - algo_namespace, - base_currency, - live_graph, - analyze_live, - simulate_orders, - auth_aliases, - stats_output): - - json_file = {'arguments': { - 'handle_data': handle_data, - 'initialize': initialize, - 'before_trading_start': before_trading_start, - 'analyze': analyze, - 'algofile': base64.b64encode(algofile.read()), - 'algotext': algotext, - 'defines': defines, - 'data_frequency': data_frequency, - 'capital_base': capital_base, - 'data': data, - 'bundle': bundle, - 'bundle_timestamp': bundle_timestamp, - 'start': start, - 'end': end, - 'output': output, - 'print_algo': print_algo, - 'local_namespace': local_namespace, - 'environ': None, - 'live': False, - 'exchange': exchange, - 'algo_namespace': algo_namespace, - 'base_currency': base_currency, - 'live_graph': live_graph, - 'analyze_live': analyze_live, - 'simulate_orders': simulate_orders, - 'auth_aliases': auth_aliases, - 'stats_output': stats_output, - }} - - url = 'http://127.0.0.1:5000/todo/api/v1.0/tasks' - response = requests.post(url, json=json_file) diff --git a/docs/source/install.rst b/docs/source/install.rst index 6ef366b5..5c24159f 100644 --- a/docs/source/install.rst +++ b/docs/source/install.rst @@ -89,7 +89,7 @@ Once either Conda or MiniConda has been set up you can install Catalyst: .. code-block:: bash - conda env create -f python2.7-environment.yml + conda env create -f python2.7-environment.yml 4. Activate the environment (which you need to do every time you start a new session to run Catalyst): @@ -132,19 +132,10 @@ with the following steps: conda env remove --name catalyst 2. Create the environment: - - for python 2.7: .. code-block:: bash conda create --name catalyst python=2.7 scipy zlib - - or for python 3.6: - - .. code-block:: bash - - conda create --name catalyst python=3.6 scipy zlib - 3. Activate the environment: From 6c4f7afaea3d4a33c141dc603b0a14de16d673f7 Mon Sep 17 00:00:00 2001 From: AvishaiW Date: Thu, 1 Mar 2018 09:42:07 +0200 Subject: [PATCH 52/87] BUG: fixed removing files- check the path, not the file --- catalyst/exchange/utils/exchange_utils.py | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/catalyst/exchange/utils/exchange_utils.py b/catalyst/exchange/utils/exchange_utils.py index 5c40a26d..658a7002 100644 --- a/catalyst/exchange/utils/exchange_utils.py +++ b/catalyst/exchange/utils/exchange_utils.py @@ -446,12 +446,13 @@ def remove_old_files(algo_name, today, rel_path, environ=None): # run on all files in the folder for f in os.listdir(folder): try: - creation_unix = os.path.getctime(os.path.join(folder, f)) - creation_time = pd.to_datetime(creation_unix, unit='s', ) + file_path = os.path.join(folder, f) + creation_unix = os.path.getctime(file_path) + creation_time = pd.to_datetime(creation_unix, unit='s', utc=True) # if the file is older than 30 days erase it if today - pd.DateOffset(30) > creation_time: - os.unlink(f) + os.unlink(file_path) except OSError: error = 'unable to erase files in {}'.format(folder) From c4b10bae392ec39c61fd2ea7e2c0514c753a3281 Mon Sep 17 00:00:00 2001 From: AvishaiW Date: Thu, 1 Mar 2018 09:47:18 +0200 Subject: [PATCH 53/87] DOC: added- creating a virtual env for 3.6 --- docs/source/install.rst | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/docs/source/install.rst b/docs/source/install.rst index 5c24159f..0bd3ff48 100644 --- a/docs/source/install.rst +++ b/docs/source/install.rst @@ -89,7 +89,7 @@ Once either Conda or MiniConda has been set up you can install Catalyst: .. code-block:: bash - conda env create -f python2.7-environment.yml + conda env create -f python2.7-environment.yml 4. Activate the environment (which you need to do every time you start a new session to run Catalyst): @@ -133,6 +133,14 @@ with the following steps: 2. Create the environment: + for python 2.7: + + .. code-block:: bash + + conda create --name catalyst python=2.7 scipy zlib + + or for python 3.6: + .. code-block:: bash conda create --name catalyst python=2.7 scipy zlib From b95cf465fc5855eac1138c5ea66564abf98d5b43 Mon Sep 17 00:00:00 2001 From: lenak25 Date: Thu, 1 Mar 2018 15:51:54 +0200 Subject: [PATCH 54/87] BLD:updating the forward fill to set volume to zero and others values to the previous close value --- catalyst/exchange/utils/exchange_utils.py | 35 ++++---- tests/exchange/test_exchange_utils.py | 105 ++++++++++++++++++++++ 2 files changed, 125 insertions(+), 15 deletions(-) create mode 100644 tests/exchange/test_exchange_utils.py diff --git a/catalyst/exchange/utils/exchange_utils.py b/catalyst/exchange/utils/exchange_utils.py index 658a7002..3366c396 100644 --- a/catalyst/exchange/utils/exchange_utils.py +++ b/catalyst/exchange/utils/exchange_utils.py @@ -716,25 +716,30 @@ def save_asset_data(folder, df, decimals=8): ) -def get_candles_df(candles, field, freq, bar_count, end_dt, - previous_value=None): +def forward_fill_df_if_needed(df, periods): + df = df.reindex(periods) + df['volume'] = df['volume'].fillna(0.0)# volume should always be 0 (if there were no trades in this interval) + df['close'] = df.fillna(method='pad') # ie pull the last close into this close + # now copy the close that was pulled down from the last timestep into this row, across into o/h/l + df['open'] = df['open'].fillna(df['close']) + df['low'] = df['low'].fillna(df['close']) + df['high'] = df['high'].fillna(df['close']) + return df + + +def transform_candles_to_df(candles): + return pd.DataFrame(candles).set_index('last_traded') + + +def get_candles_df(candles, field, freq, bar_count, end_dt=None): all_series = dict() + for asset in candles: + asset_df = transform_candles_to_df(candles[asset]) periods = pd.date_range(end=end_dt, periods=bar_count, freq=freq) + asset_df = forward_fill_df_if_needed(asset_df, periods) - dates = [candle['last_traded'] for candle in candles[asset]] - values = [candle[field] for candle in candles[asset]] - series = pd.Series(values, index=dates) - - """ - series = series.reindex( - periods, - method='ffill', - fill_value=previous_value, - ) - series.sort_index(inplace=True) - """ - all_series[asset] = series + all_series[asset] = pd.Series(asset_df[field]) df = pd.DataFrame(all_series) df.dropna(inplace=True) diff --git a/tests/exchange/test_exchange_utils.py b/tests/exchange/test_exchange_utils.py new file mode 100644 index 00000000..deebc9f1 --- /dev/null +++ b/tests/exchange/test_exchange_utils.py @@ -0,0 +1,105 @@ +from catalyst.exchange.utils.exchange_utils import transform_candles_to_df, forward_fill_df_if_needed, get_candles_df + +from catalyst.testing.fixtures import WithLogger, ZiplineTestCase +from pandas import Timestamp, Series, DataFrame + +import numpy as np + + +class TestExchangeUtils(WithLogger, ZiplineTestCase): + @classmethod + def get_specific_field_from_df(cls, df, field, asset): + new_df = DataFrame(df[field]) + new_df.columns = [asset] + new_df.index.name = None + return new_df + + def test_transform_candles_to_series(self): + asset = 'btc_usdt' + + candles = [{'high': 595, 'volume': 10, 'low': 594, + 'close': 595, 'open': 594, + 'last_traded': Timestamp('2018-03-01 09:45:00+0000', tz='UTC')}, + {'high': 594, 'volume': 108, 'low': 592, + 'close': 593, 'open': 592, + 'last_traded': Timestamp('2018-03-01 09:50:00+0000', tz='UTC')}] + + expected = [{'high': 595.0, 'volume': 10.0, 'low': 594.0, + 'close': 595.0, 'open': 594.0, + 'last_traded': Timestamp('2018-03-01 09:45:00+0000', tz='UTC')}, + {'high': 594.0, 'volume': 108.0, 'low': 592.0, + 'close': 593.0, 'open': 592.0, + 'last_traded': Timestamp('2018-03-01 09:50:00+0000', tz='UTC')}, + {'high': 593.0, 'volume': 0.0, 'low': 593.0, + 'close': 593.0, 'open': 593.0, + 'last_traded': Timestamp('2018-03-01 09:55:00+0000', tz='UTC')} + ] + + periods = [Timestamp('2018-03-01 09:45:00+0000', tz='UTC'), + Timestamp('2018-03-01 09:50:00+0000', tz='UTC'), + Timestamp('2018-03-01 09:55:00+0000', tz='UTC')] + + observed_df = forward_fill_df_if_needed(transform_candles_to_df(candles), periods) + expected_df = transform_candles_to_df(expected) + + assert (expected_df.equals(observed_df)) + + for field in ['volume', 'open', 'close', 'high', 'low']: + assert(self.get_specific_field_from_df(observed_df, field, asset).equals( + get_candles_df({asset:candles}, field, '5T', 3, end_dt=periods[2]))) + + candles = [{'high': 595, 'volume': 10, 'low': 594, + 'close': 595, 'open': 594, + 'last_traded': Timestamp('2018-03-01 09:45:00+0000', tz='UTC')}, + {'high': 594, 'volume': 108, 'low': 592, + 'close': 593, 'open': 592, + 'last_traded': Timestamp('2018-03-01 09:55:00+0000', tz='UTC')}] + + expected = [{'high': 595.0, 'volume': 10.0, 'low': 594.0, + 'close': 595.0, 'open': 594.0, + 'last_traded': Timestamp('2018-03-01 09:45:00+0000', tz='UTC')}, + {'high': 595.0, 'volume': 0.0, 'low': 595.0, + 'close': 595.0, 'open': 595.0, + 'last_traded': Timestamp('2018-03-01 09:50:00+0000', tz='UTC')}, + {'high': 594.0, 'volume': 108.0, 'low': 592.0, + 'close': 593.0, 'open': 592.0, + 'last_traded': Timestamp('2018-03-01 09:55:00+0000', tz='UTC')} + ] + + df = transform_candles_to_df(candles) + observed_df = forward_fill_df_if_needed(df, periods) + + assert (transform_candles_to_df(expected).equals(observed_df)) + + for field in ['volume', 'open', 'close', 'high', 'low']: + assert(self.get_specific_field_from_df(observed_df, field, asset).equals( + get_candles_df({asset:candles}, field, '5T', 3, end_dt=periods[2]))) + + candles = [{'high': 595, 'volume': 10, 'low': 594, + 'close': 595, 'open': 594, + 'last_traded': Timestamp('2018-03-01 09:50:00+0000', tz='UTC')}, + {'high': 594, 'volume': 108, 'low': 592, + 'close': 593, 'open': 592, + 'last_traded': Timestamp('2018-03-01 09:55:00+0000', tz='UTC')}] + + expected = [{'high': np.NaN, 'volume': 0.0, 'low': np.NaN, + 'close': np.NaN, 'open': np.NaN, + 'last_traded': Timestamp('2018-03-01 09:45:00+0000', tz='UTC')}, + {'high': 595, 'volume': 10, 'low': 594, + 'close': 595, 'open': 594, + 'last_traded': Timestamp('2018-03-01 09:50:00+0000', tz='UTC')}, + {'high': 594, 'volume': 108, 'low': 592, + 'close': 593, 'open': 592, + 'last_traded': Timestamp('2018-03-01 09:55:00+0000', tz='UTC')} + ] + + df = transform_candles_to_df(candles) + observed_df = forward_fill_df_if_needed(df, periods) + + assert (transform_candles_to_df(expected).equals(observed_df)) + # Not the same due to dropna - commenting out for now + """ + for field in ['volume', 'open', 'close', 'high', 'low']: + assert(self.get_specific_field_from_df(observed_df, field, asset).equals( + get_candles_df({asset:candles}, field, '5T', 3, end_dt=periods[2]))) + """ \ No newline at end of file From 868f17fd9d97e1522e4dd4ebd89c7d083fcebb7f Mon Sep 17 00:00:00 2001 From: lenak25 Date: Thu, 1 Mar 2018 16:39:42 +0200 Subject: [PATCH 55/87] BLD:flake fixes --- catalyst/exchange/utils/exchange_utils.py | 9 +- tests/exchange/test_exchange_utils.py | 110 +++++++++++++++------- 2 files changed, 82 insertions(+), 37 deletions(-) diff --git a/catalyst/exchange/utils/exchange_utils.py b/catalyst/exchange/utils/exchange_utils.py index 3366c396..b88a4483 100644 --- a/catalyst/exchange/utils/exchange_utils.py +++ b/catalyst/exchange/utils/exchange_utils.py @@ -718,9 +718,12 @@ def save_asset_data(folder, df, decimals=8): def forward_fill_df_if_needed(df, periods): df = df.reindex(periods) - df['volume'] = df['volume'].fillna(0.0)# volume should always be 0 (if there were no trades in this interval) - df['close'] = df.fillna(method='pad') # ie pull the last close into this close - # now copy the close that was pulled down from the last timestep into this row, across into o/h/l + # volume should always be 0 (if there were no trades in this interval) + df['volume'] = df['volume'].fillna(0.0) + # ie pull the last close into this close + df['close'] = df.fillna(method='pad') + # now copy the close that was pulled down from the last timestep + # into this row, across into o/h/l df['open'] = df['open'].fillna(df['close']) df['low'] = df['low'].fillna(df['close']) df['high'] = df['high'].fillna(df['close']) diff --git a/tests/exchange/test_exchange_utils.py b/tests/exchange/test_exchange_utils.py index deebc9f1..ae5c07dc 100644 --- a/tests/exchange/test_exchange_utils.py +++ b/tests/exchange/test_exchange_utils.py @@ -1,7 +1,8 @@ -from catalyst.exchange.utils.exchange_utils import transform_candles_to_df, forward_fill_df_if_needed, get_candles_df +from catalyst.exchange.utils.exchange_utils import transform_candles_to_df, \ + forward_fill_df_if_needed, get_candles_df from catalyst.testing.fixtures import WithLogger, ZiplineTestCase -from pandas import Timestamp, Series, DataFrame +from pandas import Timestamp, DataFrame import numpy as np @@ -19,52 +20,76 @@ class TestExchangeUtils(WithLogger, ZiplineTestCase): candles = [{'high': 595, 'volume': 10, 'low': 594, 'close': 595, 'open': 594, - 'last_traded': Timestamp('2018-03-01 09:45:00+0000', tz='UTC')}, + 'last_traded': Timestamp('2018-03-01 09:45:00+0000', + tz='UTC') + }, {'high': 594, 'volume': 108, 'low': 592, 'close': 593, 'open': 592, - 'last_traded': Timestamp('2018-03-01 09:50:00+0000', tz='UTC')}] + 'last_traded': Timestamp('2018-03-01 09:50:00+0000', + tz='UTC') + }] expected = [{'high': 595.0, 'volume': 10.0, 'low': 594.0, - 'close': 595.0, 'open': 594.0, - 'last_traded': Timestamp('2018-03-01 09:45:00+0000', tz='UTC')}, - {'high': 594.0, 'volume': 108.0, 'low': 592.0, - 'close': 593.0, 'open': 592.0, - 'last_traded': Timestamp('2018-03-01 09:50:00+0000', tz='UTC')}, - {'high': 593.0, 'volume': 0.0, 'low': 593.0, - 'close': 593.0, 'open': 593.0, - 'last_traded': Timestamp('2018-03-01 09:55:00+0000', tz='UTC')} - ] + 'close': 595.0, 'open': 594.0, + 'last_traded': Timestamp('2018-03-01 09:45:00+0000', + tz='UTC') + }, + {'high': 594.0, 'volume': 108.0, 'low': 592.0, + 'close': 593.0, 'open': 592.0, + 'last_traded': Timestamp('2018-03-01 09:50:00+0000', + tz='UTC') + }, + {'high': 593.0, 'volume': 0.0, 'low': 593.0, + 'close': 593.0, 'open': 593.0, + 'last_traded': Timestamp('2018-03-01 09:55:00+0000', + tz='UTC') + }] periods = [Timestamp('2018-03-01 09:45:00+0000', tz='UTC'), Timestamp('2018-03-01 09:50:00+0000', tz='UTC'), Timestamp('2018-03-01 09:55:00+0000', tz='UTC')] - observed_df = forward_fill_df_if_needed(transform_candles_to_df(candles), periods) + observed_df = forward_fill_df_if_needed( + transform_candles_to_df(candles), + periods) expected_df = transform_candles_to_df(expected) assert (expected_df.equals(observed_df)) for field in ['volume', 'open', 'close', 'high', 'low']: - assert(self.get_specific_field_from_df(observed_df, field, asset).equals( - get_candles_df({asset:candles}, field, '5T', 3, end_dt=periods[2]))) + field_dt = self.get_specific_field_from_df(observed_df, + field, + asset) + assert (field_dt.equals(get_candles_df({asset: candles}, + field, '5T', 3, + end_dt=periods[2]))) candles = [{'high': 595, 'volume': 10, 'low': 594, 'close': 595, 'open': 594, - 'last_traded': Timestamp('2018-03-01 09:45:00+0000', tz='UTC')}, + 'last_traded': Timestamp('2018-03-01 09:45:00+0000', + tz='UTC') + }, {'high': 594, 'volume': 108, 'low': 592, 'close': 593, 'open': 592, - 'last_traded': Timestamp('2018-03-01 09:55:00+0000', tz='UTC')}] + 'last_traded': Timestamp('2018-03-01 09:55:00+0000', + tz='UTC') + }] expected = [{'high': 595.0, 'volume': 10.0, 'low': 594.0, 'close': 595.0, 'open': 594.0, - 'last_traded': Timestamp('2018-03-01 09:45:00+0000', tz='UTC')}, + 'last_traded': Timestamp('2018-03-01 09:45:00+0000', + tz='UTC') + }, {'high': 595.0, 'volume': 0.0, 'low': 595.0, 'close': 595.0, 'open': 595.0, - 'last_traded': Timestamp('2018-03-01 09:50:00+0000', tz='UTC')}, + 'last_traded': Timestamp('2018-03-01 09:50:00+0000', + tz='UTC') + }, {'high': 594.0, 'volume': 108.0, 'low': 592.0, 'close': 593.0, 'open': 592.0, - 'last_traded': Timestamp('2018-03-01 09:55:00+0000', tz='UTC')} - ] + 'last_traded': Timestamp('2018-03-01 09:55:00+0000', + tz='UTC') + }] df = transform_candles_to_df(candles) observed_df = forward_fill_df_if_needed(df, periods) @@ -72,26 +97,39 @@ class TestExchangeUtils(WithLogger, ZiplineTestCase): assert (transform_candles_to_df(expected).equals(observed_df)) for field in ['volume', 'open', 'close', 'high', 'low']: - assert(self.get_specific_field_from_df(observed_df, field, asset).equals( - get_candles_df({asset:candles}, field, '5T', 3, end_dt=periods[2]))) + field_dt = self.get_specific_field_from_df(observed_df, + field, + asset) + assert(field_dt.equals(get_candles_df({asset: candles}, + field, '5T', 3, + end_dt=periods[2]))) candles = [{'high': 595, 'volume': 10, 'low': 594, 'close': 595, 'open': 594, - 'last_traded': Timestamp('2018-03-01 09:50:00+0000', tz='UTC')}, + 'last_traded': Timestamp('2018-03-01 09:50:00+0000', + tz='UTC') + }, {'high': 594, 'volume': 108, 'low': 592, 'close': 593, 'open': 592, - 'last_traded': Timestamp('2018-03-01 09:55:00+0000', tz='UTC')}] + 'last_traded': Timestamp('2018-03-01 09:55:00+0000', + tz='UTC') + }] expected = [{'high': np.NaN, 'volume': 0.0, 'low': np.NaN, 'close': np.NaN, 'open': np.NaN, - 'last_traded': Timestamp('2018-03-01 09:45:00+0000', tz='UTC')}, + 'last_traded': Timestamp('2018-03-01 09:45:00+0000', + tz='UTC') + }, {'high': 595, 'volume': 10, 'low': 594, 'close': 595, 'open': 594, - 'last_traded': Timestamp('2018-03-01 09:50:00+0000', tz='UTC')}, + 'last_traded': Timestamp('2018-03-01 09:50:00+0000', + tz='UTC') + }, {'high': 594, 'volume': 108, 'low': 592, 'close': 593, 'open': 592, - 'last_traded': Timestamp('2018-03-01 09:55:00+0000', tz='UTC')} - ] + 'last_traded': Timestamp('2018-03-01 09:55:00+0000', + tz='UTC') + }] df = transform_candles_to_df(candles) observed_df = forward_fill_df_if_needed(df, periods) @@ -99,7 +137,11 @@ class TestExchangeUtils(WithLogger, ZiplineTestCase): assert (transform_candles_to_df(expected).equals(observed_df)) # Not the same due to dropna - commenting out for now """ - for field in ['volume', 'open', 'close', 'high', 'low']: - assert(self.get_specific_field_from_df(observed_df, field, asset).equals( - get_candles_df({asset:candles}, field, '5T', 3, end_dt=periods[2]))) - """ \ No newline at end of file + for field in ['volume', 'open', 'close', 'high', 'low']: + field_dt = self.get_specific_field_from_df(observed_df, + field, + asset) + assert(field_dt.equals(get_candles_df({asset:candles}, + field, '5T', 3, + end_dt=periods[2]))) + """ From 8be0626fc97741bf1ed3982bd0cb9ef28b7192dc Mon Sep 17 00:00:00 2001 From: lenak25 Date: Thu, 1 Mar 2018 18:16:57 +0200 Subject: [PATCH 56/87] BLD:refine unit-test --- tests/exchange/test_exchange_utils.py | 37 +++++++++++++++++++-------- 1 file changed, 26 insertions(+), 11 deletions(-) diff --git a/tests/exchange/test_exchange_utils.py b/tests/exchange/test_exchange_utils.py index ae5c07dc..ddc15cc9 100644 --- a/tests/exchange/test_exchange_utils.py +++ b/tests/exchange/test_exchange_utils.py @@ -2,7 +2,7 @@ from catalyst.exchange.utils.exchange_utils import transform_candles_to_df, \ forward_fill_df_if_needed, get_candles_df from catalyst.testing.fixtures import WithLogger, ZiplineTestCase -from pandas import Timestamp, DataFrame +from pandas import Timestamp, DataFrame, concat import numpy as np @@ -15,9 +15,11 @@ class TestExchangeUtils(WithLogger, ZiplineTestCase): new_df.index.name = None return new_df - def test_transform_candles_to_series(self): + def test_get_candles_df(self): asset = 'btc_usdt' + asset2 = 'eth_usdt' + # test forward fill in the end candles = [{'high': 595, 'volume': 10, 'low': 594, 'close': 595, 'open': 594, 'last_traded': Timestamp('2018-03-01 09:45:00+0000', @@ -57,13 +59,14 @@ class TestExchangeUtils(WithLogger, ZiplineTestCase): assert (expected_df.equals(observed_df)) for field in ['volume', 'open', 'close', 'high', 'low']: - field_dt = self.get_specific_field_from_df(observed_df, + field_dt = self.get_specific_field_from_df(expected_df, field, asset) assert (field_dt.equals(get_candles_df({asset: candles}, field, '5T', 3, end_dt=periods[2]))) + # test forward fill in the middle candles = [{'high': 595, 'volume': 10, 'low': 594, 'close': 595, 'open': 594, 'last_traded': Timestamp('2018-03-01 09:45:00+0000', @@ -93,17 +96,28 @@ class TestExchangeUtils(WithLogger, ZiplineTestCase): df = transform_candles_to_df(candles) observed_df = forward_fill_df_if_needed(df, periods) + expected_df = transform_candles_to_df(expected) - assert (transform_candles_to_df(expected).equals(observed_df)) + assert (expected_df.equals(observed_df)) for field in ['volume', 'open', 'close', 'high', 'low']: - field_dt = self.get_specific_field_from_df(observed_df, - field, - asset) - assert(field_dt.equals(get_candles_df({asset: candles}, - field, '5T', 3, - end_dt=periods[2]))) + # test several assets as well + observed_df = get_candles_df({asset: candles, + asset2: candles}, + field, '5T', 3, + end_dt=periods[2]) + field_dt_a1 = self.get_specific_field_from_df(expected_df, + field, + asset) + field_dt_a2 = self.get_specific_field_from_df(expected_df, + field, + asset2) + + assert(observed_df.equals(concat([field_dt_a1, field_dt_a2], + axis=1))) + + # test "forward fill" at the beginning candles = [{'high': 595, 'volume': 10, 'low': 594, 'close': 595, 'open': 594, 'last_traded': Timestamp('2018-03-01 09:50:00+0000', @@ -133,8 +147,9 @@ class TestExchangeUtils(WithLogger, ZiplineTestCase): df = transform_candles_to_df(candles) observed_df = forward_fill_df_if_needed(df, periods) + expected_df = transform_candles_to_df(expected) - assert (transform_candles_to_df(expected).equals(observed_df)) + assert (expected_df.equals(observed_df)) # Not the same due to dropna - commenting out for now """ for field in ['volume', 'open', 'close', 'high', 'low']: From 3159ec7dc2ca9e93fc0b2f2f2a09567039934730 Mon Sep 17 00:00:00 2001 From: lenak25 Date: Fri, 2 Mar 2018 00:36:13 +0200 Subject: [PATCH 57/87] BLD: fix periods calculations and bundle unit-test --- catalyst/exchange/utils/exchange_utils.py | 17 ++++++++++++++++- tests/exchange/test_suites/test_suite_bundle.py | 8 ++++---- 2 files changed, 20 insertions(+), 5 deletions(-) diff --git a/catalyst/exchange/utils/exchange_utils.py b/catalyst/exchange/utils/exchange_utils.py index b88a4483..ac16ebdc 100644 --- a/catalyst/exchange/utils/exchange_utils.py +++ b/catalyst/exchange/utils/exchange_utils.py @@ -16,6 +16,7 @@ from catalyst.exchange.utils.serialization_utils import ExchangeJSONEncoder, \ ExchangeJSONDecoder from catalyst.utils.paths import data_root, ensure_directory, \ last_modified_time +from catalyst.exchange.utils.datetime_utils import get_periods_range def get_sid(symbol): @@ -739,7 +740,21 @@ def get_candles_df(candles, field, freq, bar_count, end_dt=None): for asset in candles: asset_df = transform_candles_to_df(candles[asset]) - periods = pd.date_range(end=end_dt, periods=bar_count, freq=freq) + rounded_end_dt = end_dt.round(freq) + + periods = get_periods_range( + start_dt=None, end_dt=rounded_end_dt, + freq=freq, periods=bar_count + ) + + if rounded_end_dt > end_dt: + periods = periods[:-1] + elif rounded_end_dt <= end_dt: + periods = periods[1:] + + print rounded_end_dt + print periods + # periods = pd.date_range(end=end_dt, periods=bar_count, freq=freq) asset_df = forward_fill_df_if_needed(asset_df, periods) all_series[asset] = pd.Series(asset_df[field]) diff --git a/tests/exchange/test_suites/test_suite_bundle.py b/tests/exchange/test_suites/test_suite_bundle.py index d338045d..023b9ab8 100644 --- a/tests/exchange/test_suites/test_suite_bundle.py +++ b/tests/exchange/test_suites/test_suite_bundle.py @@ -107,14 +107,14 @@ class TestSuiteBundle: print('saved {} test results: {}'.format(end_dt, folder)) assert_frame_equal( - right=data['bundle'], - left=data['exchange'], + right=data['bundle'][:-1], + left=data['exchange'][:-1], check_less_precise=1, ) try: assert_frame_equal( - right=data['bundle'], - left=data['exchange'], + right=data['bundle'][:-1], + left=data['exchange'][:-1], check_less_precise=min([a.decimals for a in assets]), ) except Exception as e: From 5d3a1c2f8b916e69ecc7bef0a0c2091ed8a8ea5e Mon Sep 17 00:00:00 2001 From: lenak25 Date: Fri, 2 Mar 2018 00:39:28 +0200 Subject: [PATCH 58/87] BLD: cosmetics --- catalyst/exchange/utils/exchange_utils.py | 2 -- 1 file changed, 2 deletions(-) diff --git a/catalyst/exchange/utils/exchange_utils.py b/catalyst/exchange/utils/exchange_utils.py index ac16ebdc..9a681068 100644 --- a/catalyst/exchange/utils/exchange_utils.py +++ b/catalyst/exchange/utils/exchange_utils.py @@ -752,8 +752,6 @@ def get_candles_df(candles, field, freq, bar_count, end_dt=None): elif rounded_end_dt <= end_dt: periods = periods[1:] - print rounded_end_dt - print periods # periods = pd.date_range(end=end_dt, periods=bar_count, freq=freq) asset_df = forward_fill_df_if_needed(asset_df, periods) From f990ecf14d58c79233eab7d122078b3182c480b4 Mon Sep 17 00:00:00 2001 From: Victor Date: Fri, 2 Mar 2018 11:34:18 -0700 Subject: [PATCH 59/87] BUG: fix incompatibility with web3==4.0.0b11 --- catalyst/marketplace/marketplace.py | 13 ++++--------- 1 file changed, 4 insertions(+), 9 deletions(-) diff --git a/catalyst/marketplace/marketplace.py b/catalyst/marketplace/marketplace.py index 37ed2ec6..b188d6d6 100644 --- a/catalyst/marketplace/marketplace.py +++ b/catalyst/marketplace/marketplace.py @@ -40,13 +40,6 @@ else: log = logbook.Logger('Marketplace', level=LOG_LEVEL) -def first_line_from_url(url): - request = urllib.urlopen(url) - first_line = request.readline() - if sys.version_info.major >= 3: - charset = request.info().get_content_charset() - first_line = first_line.decode(charset) - return first_line.strip() class Marketplace: def __init__(self): @@ -65,7 +58,8 @@ class Marketplace: self.web3 = Web3(HTTPProvider(ETH_REMOTE_NODE)) self.mkt_contract_address = Web3.toChecksumAddress( - first_line_from_url(MARKETPLACE_CONTRACT)) + contract_url.readline().decode( + contract_url.info().get_content_charset()).strip()) abi_url = urllib.urlopen(MARKETPLACE_CONTRACT_ABI) abi = json.load(abi_url) @@ -76,7 +70,8 @@ class Marketplace: ) self.eng_contract_address = Web3.toChecksumAddress( - first_line_from_url(ENIGMA_CONTRACT)) + contract_url.readline().decode( + contract_url.info().get_content_charset()).strip()) abi_url = urllib.urlopen(ENIGMA_CONTRACT_ABI) abi = json.load(abi_url) From b4e5b699bd77eb78863208b76786f56290d72e74 Mon Sep 17 00:00:00 2001 From: Victor Date: Fri, 2 Mar 2018 11:37:22 -0700 Subject: [PATCH 60/87] BUG: fix2 incompatibility with web3==4.0.0b11 --- catalyst/marketplace/marketplace.py | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/catalyst/marketplace/marketplace.py b/catalyst/marketplace/marketplace.py index b188d6d6..731f116a 100644 --- a/catalyst/marketplace/marketplace.py +++ b/catalyst/marketplace/marketplace.py @@ -57,6 +57,8 @@ class Marketplace: self.web3 = Web3(HTTPProvider(ETH_REMOTE_NODE)) + contract_url = urllib.urlopen(MARKETPLACE_CONTRACT) + self.mkt_contract_address = Web3.toChecksumAddress( contract_url.readline().decode( contract_url.info().get_content_charset()).strip()) @@ -69,6 +71,8 @@ class Marketplace: abi=abi, ) + contract_url = urllib.urlopen(ENIGMA_CONTRACT) + self.eng_contract_address = Web3.toChecksumAddress( contract_url.readline().decode( contract_url.info().get_content_charset()).strip()) From fa0e9332bfadcc33a7c5440885f75281ff6051a9 Mon Sep 17 00:00:00 2001 From: Victor Grau Serrat Date: Fri, 2 Mar 2018 11:43:53 -0700 Subject: [PATCH 61/87] MAINT: CLI info on marketplace cmds --- catalyst/__main__.py | 18 +++++++++++++++++- etc/requirements.txt | 2 +- 2 files changed, 18 insertions(+), 2 deletions(-) diff --git a/catalyst/__main__.py b/catalyst/__main__.py index 1b08f69d..03669486 100644 --- a/catalyst/__main__.py +++ b/catalyst/__main__.py @@ -767,12 +767,18 @@ def bundles(): @main.group() @click.pass_context def marketplace(ctx): + """Access the Enigma Data Marketplace to:\n + - Register and Publish new datasets (seller-side)\n + - Subscribe and Ingest premium datasets (buyer-side)\n + """ pass @marketplace.command() @click.pass_context def ls(ctx): + """List all available datasets. + """ click.echo('Listing of available data sources on the marketplace:', sys.stdout) marketplace = Marketplace() @@ -787,6 +793,8 @@ def ls(ctx): ) @click.pass_context def subscribe(ctx, dataset): + """Subscribe to an exisiting dataset. + """ if dataset is None: ctx.fail("must specify a dataset to subscribe to with '--dataset'\n" "List available dataset on the marketplace with " @@ -825,6 +833,8 @@ def subscribe(ctx, dataset): ) @click.pass_context def ingest(ctx, dataset, data_frequency, start, end): + """Ingest a dataset (requires subscription). + """ if dataset is None: ctx.fail("must specify a dataset to clean with '--dataset'\n" "List available dataset on the marketplace with " @@ -842,8 +852,10 @@ def ingest(ctx, dataset, data_frequency, start, end): ) @click.pass_context def clean(ctx, dataset): + """Clean/Remove local data for a given dataset. + """ if dataset is None: - ctx.fail("must specify a dataset to ingest with '--dataset'\n" + ctx.fail("must specify a dataset to clean up with '--dataset'\n" "List available dataset on the marketplace with " "'catalyst marketplace ls'") click.echo('Cleaning data source: {}'.format(dataset), sys.stdout) @@ -855,6 +867,8 @@ def clean(ctx, dataset): @marketplace.command() @click.pass_context def register(ctx): + """Register a new dataset. + """ marketplace = Marketplace() marketplace.register() @@ -878,6 +892,8 @@ def register(ctx): ) @click.pass_context def publish(ctx, dataset, datadir, watch): + """Publish data for a registered dataset. + """ marketplace = Marketplace() if dataset is None: ctx.fail("must specify a dataset to publish data for " diff --git a/etc/requirements.txt b/etc/requirements.txt index bac9666d..73573e45 100644 --- a/etc/requirements.txt +++ b/etc/requirements.txt @@ -84,5 +84,5 @@ tables==3.3.0 ccxt==1.10.1094 boto3==1.4.8 redo==1.6 -web3==4.0.0b7 +web3==4.0.0b11 requests-toolbelt==0.8.0 From d7b6cb8490ab45ee88dddaafa45ac3fbbf2a907d Mon Sep 17 00:00:00 2001 From: Victor Grau Serrat Date: Fri, 2 Mar 2018 12:18:48 -0700 Subject: [PATCH 62/87] BUG: marketplace typo --- catalyst/marketplace/marketplace.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/catalyst/marketplace/marketplace.py b/catalyst/marketplace/marketplace.py index 731f116a..68088c5c 100644 --- a/catalyst/marketplace/marketplace.py +++ b/catalyst/marketplace/marketplace.py @@ -623,7 +623,7 @@ class Marketplace: ) except Exception as e: - print('Unable to subscribe to data source: {}'.format(e)) + print('Unable to register the requested dataset: {}'.format(e)) return self.check_transaction(tx_hash) From 5197ab6cc2ed9df66508b9565f256aebda9258ce Mon Sep 17 00:00:00 2001 From: Victor Grau Serrat Date: Fri, 2 Mar 2018 12:50:46 -0700 Subject: [PATCH 63/87] BUG: isolated Python3 depedency to the marketplace --- catalyst/marketplace/marketplace.py | 39 ++++++++++++---------- catalyst/marketplace/marketplace_errors.py | 11 +++++- etc/python2.7-environment.yml | 4 ++- etc/requirements.txt | 2 +- etc/requirements_marketplace.txt | 2 -- 5 files changed, 35 insertions(+), 23 deletions(-) delete mode 100644 etc/requirements_marketplace.txt diff --git a/catalyst/marketplace/marketplace.py b/catalyst/marketplace/marketplace.py index 68088c5c..e972b750 100644 --- a/catalyst/marketplace/marketplace.py +++ b/catalyst/marketplace/marketplace.py @@ -23,7 +23,7 @@ from catalyst.exchange.utils.stats_utils import set_print_settings from catalyst.marketplace.marketplace_errors import ( MarketplacePubAddressEmpty, MarketplaceDatasetNotFound, MarketplaceNoAddressMatch, MarketplaceHTTPRequest, - MarketplaceNoCSVFiles) + MarketplaceNoCSVFiles, MarketplaceRequiresPython3) from catalyst.marketplace.utils.auth_utils import get_key_secret, \ get_signed_headers from catalyst.marketplace.utils.bundle_utils import merge_bundles @@ -44,7 +44,10 @@ log = logbook.Logger('Marketplace', level=LOG_LEVEL) class Marketplace: def __init__(self): global Web3 - from web3 import Web3, HTTPProvider + try: + from web3 import Web3, HTTPProvider + except ImportError: + raise MarketplaceRequiresPython3() self.addresses = get_user_pubaddr() @@ -58,7 +61,7 @@ class Marketplace: self.web3 = Web3(HTTPProvider(ETH_REMOTE_NODE)) contract_url = urllib.urlopen(MARKETPLACE_CONTRACT) - + self.mkt_contract_address = Web3.toChecksumAddress( contract_url.readline().decode( contract_url.info().get_content_charset()).strip()) @@ -72,7 +75,7 @@ class Marketplace: ) contract_url = urllib.urlopen(ENIGMA_CONTRACT) - + self.eng_contract_address = Web3.toChecksumAddress( contract_url.readline().decode( contract_url.info().get_content_charset()).strip()) @@ -150,13 +153,13 @@ class Marketplace: 'Gas Price:\t\t[Accept the default value]\n' 'Nonce:\t\t\t{nonce}\n' 'Data:\t\t\t{data}\n'.format( - _from=from_address, - to=tx['to'], - value=tx['value'], - gas=tx['gas'], - nonce=tx['nonce'], - data=tx['data'], ) - ) + _from=from_address, + to=tx['to'], + value=tx['value'], + gas=tx['gas'], + nonce=tx['nonce'], + data=tx['data'], ) + ) signed_tx = input('Copy and Paste the "Signed Transaction" ' 'field here:\n') @@ -261,14 +264,14 @@ class Marketplace: 'buy: {} ENG. Get enough ENG to cover the costs of the ' 'monthly\nsubscription for what you are trying to buy, ' 'and try again.'.format( - address, from_grains(balance), price)) + address, from_grains(balance), price)) return while True: agree_pay = input('Please confirm that you agree to pay {} ENG ' 'for a monthly subscription to the dataset "{}" ' 'starting today. [default: Y] '.format( - price, dataset)) or 'y' + price, dataset)) or 'y' if agree_pay.lower() not in ('y', 'n'): print("Please answer Y or N.") else: @@ -371,7 +374,7 @@ class Marketplace: 'You can now ingest this dataset anytime during the ' 'next month by running the following command:\n' 'catalyst marketplace ingest --dataset={}'.format( - dataset, address, dataset)) + dataset, address, dataset)) def process_temp_bundle(self, ds_name, path): """ @@ -428,10 +431,10 @@ class Marketplace: print('Your subscription to dataset "{}" expired on {} UTC.' 'Please renew your subscription by running:\n' 'catalyst marketplace subscribe --dataset={}'.format( - ds_name, - pd.to_datetime(check_sub[4], unit='s', utc=True), - ds_name) - ) + ds_name, + pd.to_datetime(check_sub[4], unit='s', utc=True), + ds_name) + ) if 'key' in self.addresses[address_i]: key = self.addresses[address_i]['key'] diff --git a/catalyst/marketplace/marketplace_errors.py b/catalyst/marketplace/marketplace_errors.py index b6be1c3b..488c204f 100644 --- a/catalyst/marketplace/marketplace_errors.py +++ b/catalyst/marketplace/marketplace_errors.py @@ -9,7 +9,8 @@ def silent_except_hook(exctype, excvalue, exctraceback): MarketplaceNoAddressMatch, MarketplaceHTTPRequest, MarketplaceNoCSVFiles, MarketplaceContractDataNoMatch, MarketplaceSubscriptionExpired, MarketplaceJSONError, - MarketplaceWalletNotSupported, MarketplaceEmptySignature]: + MarketplaceWalletNotSupported, MarketplaceEmptySignature, + MarketplaceRequiresPython3]: fn = traceback.extract_tb(exctraceback)[-1][0] ln = traceback.extract_tb(exctraceback)[-1][1] print("Error traceback: {1} (line {2})\n" @@ -86,3 +87,11 @@ class MarketplaceJSONError(ZiplineError): 'The configuration file {file} is malformed. Please correct ' 'the following error:\n{error}' ) + + +class MarketplaceRequiresPython3(ZiplineError): + msg = ( + '\nCatalyst requires Python3 to access the Enigma Data Marketplace.\n' + 'If you want to use the Data Marketplace, you need to reinstall ' + 'Catalyst\nwith Python3. See the documentation website for additional ' + 'information.') diff --git a/etc/python2.7-environment.yml b/etc/python2.7-environment.yml index b2e2486c..3835d7d4 100644 --- a/etc/python2.7-environment.yml +++ b/etc/python2.7-environment.yml @@ -23,7 +23,9 @@ dependencies: - bottleneck==1.2.1 - chardet==3.0.4 - ccxt==1.10.1094 - - web3==4.0.0b7 +# The Enigma Data Marketplace requires Python3 because it depends on +# web3, which requires Python3, as building its dependencies breaks in Python2 +# - web3==4.0.0b7 - requests-toolbelt==0.8.0 - click==6.7 - contextlib2==0.5.5 diff --git a/etc/requirements.txt b/etc/requirements.txt index 73573e45..5be47e3d 100644 --- a/etc/requirements.txt +++ b/etc/requirements.txt @@ -84,5 +84,5 @@ tables==3.3.0 ccxt==1.10.1094 boto3==1.4.8 redo==1.6 -web3==4.0.0b11 +web3==4.0.0b11; python_version > '3.4' requests-toolbelt==0.8.0 diff --git a/etc/requirements_marketplace.txt b/etc/requirements_marketplace.txt deleted file mode 100644 index 2a56c200..00000000 --- a/etc/requirements_marketplace.txt +++ /dev/null @@ -1,2 +0,0 @@ -web3==4.0.0b7 -requests-toolbelt==0.8.0 From e59f46dfd653220a77fdf1ea1698e9137a14d2b9 Mon Sep 17 00:00:00 2001 From: lenak25 Date: Sun, 4 Mar 2018 13:44:04 +0200 Subject: [PATCH 64/87] BLD: improve periods calculation --- catalyst/exchange/utils/exchange_utils.py | 18 +--- tests/exchange/test_exchange_utils.py | 109 ++++++++++++---------- 2 files changed, 65 insertions(+), 62 deletions(-) diff --git a/catalyst/exchange/utils/exchange_utils.py b/catalyst/exchange/utils/exchange_utils.py index 9a681068..28185091 100644 --- a/catalyst/exchange/utils/exchange_utils.py +++ b/catalyst/exchange/utils/exchange_utils.py @@ -16,7 +16,6 @@ from catalyst.exchange.utils.serialization_utils import ExchangeJSONEncoder, \ ExchangeJSONDecoder from catalyst.utils.paths import data_root, ensure_directory, \ last_modified_time -from catalyst.exchange.utils.datetime_utils import get_periods_range def get_sid(symbol): @@ -740,19 +739,10 @@ def get_candles_df(candles, field, freq, bar_count, end_dt=None): for asset in candles: asset_df = transform_candles_to_df(candles[asset]) - rounded_end_dt = end_dt.round(freq) - - periods = get_periods_range( - start_dt=None, end_dt=rounded_end_dt, - freq=freq, periods=bar_count - ) - - if rounded_end_dt > end_dt: - periods = periods[:-1] - elif rounded_end_dt <= end_dt: - periods = periods[1:] - - # periods = pd.date_range(end=end_dt, periods=bar_count, freq=freq) + rounded_end_dt = end_dt.floor(freq) + periods = pd.date_range(end=rounded_end_dt, + periods=bar_count, + freq=freq) asset_df = forward_fill_df_if_needed(asset_df, periods) all_series[asset] = pd.Series(asset_df[field]) diff --git a/tests/exchange/test_exchange_utils.py b/tests/exchange/test_exchange_utils.py index ddc15cc9..2d3d1efe 100644 --- a/tests/exchange/test_exchange_utils.py +++ b/tests/exchange/test_exchange_utils.py @@ -2,6 +2,7 @@ from catalyst.exchange.utils.exchange_utils import transform_candles_to_df, \ forward_fill_df_if_needed, get_candles_df from catalyst.testing.fixtures import WithLogger, ZiplineTestCase +from datetime import timedelta from pandas import Timestamp, DataFrame, concat import numpy as np @@ -15,9 +16,59 @@ class TestExchangeUtils(WithLogger, ZiplineTestCase): new_df.index.name = None return new_df + @classmethod + def verify_forward_fill_df_if_needed(cls, candles, periods, expected_df): + observed_df = forward_fill_df_if_needed( + transform_candles_to_df(candles), + periods) + assert (expected_df.equals(observed_df)) + + @classmethod + def verify_get_candles_df(cls, assets, candles, end_fixed_dt, + expected_df, check_next_candle=False): + # run on all the fields + for field in ['volume', 'open', 'close', 'high', 'low']: + + field_dt = cls.get_specific_field_from_df(expected_df, + field, + assets[0]) + # run on several timestamps + for delta in range(5): + end_dt = end_fixed_dt + timedelta(minutes=delta) + assert (field_dt.equals(get_candles_df({assets[0]: candles}, + field, '5T', 3, + end_dt=end_dt))) + + field_dt_a1 = cls.get_specific_field_from_df(expected_df, + field, + assets[0]) + field_dt_a2 = cls.get_specific_field_from_df(expected_df, + field, + assets[1]) + observed_df = get_candles_df({assets[0]: candles, + assets[1]: candles}, + field, '5T', 3, + end_dt=end_dt) + + assert (observed_df.equals(concat([field_dt_a1, field_dt_a2], + axis=1))) + + if check_next_candle: + # one candle forward + end_dt = end_fixed_dt + timedelta(minutes=6) + observed_df = get_candles_df({assets[0]: candles, + assets[1]: candles}, + field, '5T', 3, + end_dt=end_dt) + + assert (not observed_df.equals(concat([field_dt_a1, + field_dt_a2], + axis=1))) + assert (concat([field_dt_a1, field_dt_a2], + axis=1)[1:].equals(observed_df[:-1])) + def test_get_candles_df(self): - asset = 'btc_usdt' - asset2 = 'eth_usdt' + assets = ['btc_usdt', 'eth_usdt'] # test forward fill in the end candles = [{'high': 595, 'volume': 10, 'low': 594, @@ -51,20 +102,12 @@ class TestExchangeUtils(WithLogger, ZiplineTestCase): Timestamp('2018-03-01 09:50:00+0000', tz='UTC'), Timestamp('2018-03-01 09:55:00+0000', tz='UTC')] - observed_df = forward_fill_df_if_needed( - transform_candles_to_df(candles), - periods) expected_df = transform_candles_to_df(expected) - assert (expected_df.equals(observed_df)) - - for field in ['volume', 'open', 'close', 'high', 'low']: - field_dt = self.get_specific_field_from_df(expected_df, - field, - asset) - assert (field_dt.equals(get_candles_df({asset: candles}, - field, '5T', 3, - end_dt=periods[2]))) + self.verify_forward_fill_df_if_needed(candles, periods, + expected_df) + self.verify_get_candles_df(assets, candles, periods[2], + expected_df, True) # test forward fill in the middle candles = [{'high': 595, 'volume': 10, 'low': 594, @@ -94,28 +137,9 @@ class TestExchangeUtils(WithLogger, ZiplineTestCase): tz='UTC') }] - df = transform_candles_to_df(candles) - observed_df = forward_fill_df_if_needed(df, periods) expected_df = transform_candles_to_df(expected) - - assert (expected_df.equals(observed_df)) - - for field in ['volume', 'open', 'close', 'high', 'low']: - # test several assets as well - observed_df = get_candles_df({asset: candles, - asset2: candles}, - field, '5T', 3, - end_dt=periods[2]) - - field_dt_a1 = self.get_specific_field_from_df(expected_df, - field, - asset) - field_dt_a2 = self.get_specific_field_from_df(expected_df, - field, - asset2) - - assert(observed_df.equals(concat([field_dt_a1, field_dt_a2], - axis=1))) + self.verify_forward_fill_df_if_needed(candles, periods, expected_df) + self.verify_get_candles_df(assets, candles, periods[2], expected_df) # test "forward fill" at the beginning candles = [{'high': 595, 'volume': 10, 'low': 594, @@ -145,18 +169,7 @@ class TestExchangeUtils(WithLogger, ZiplineTestCase): tz='UTC') }] - df = transform_candles_to_df(candles) - observed_df = forward_fill_df_if_needed(df, periods) expected_df = transform_candles_to_df(expected) - - assert (expected_df.equals(observed_df)) + self.verify_forward_fill_df_if_needed(candles, periods, expected_df) # Not the same due to dropna - commenting out for now - """ - for field in ['volume', 'open', 'close', 'high', 'low']: - field_dt = self.get_specific_field_from_df(observed_df, - field, - asset) - assert(field_dt.equals(get_candles_df({asset:candles}, - field, '5T', 3, - end_dt=periods[2]))) - """ + # self.verify_get_candles_df(assets, candles, periods[2], expected_df) From aa520d5a8bc99b45704623088d76724cbdae6523 Mon Sep 17 00:00:00 2001 From: AvishaiW Date: Sun, 4 Mar 2018 17:23:12 +0200 Subject: [PATCH 65/87] STY: pep8 change in exchange_blotter --- catalyst/exchange/exchange_blotter.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/catalyst/exchange/exchange_blotter.py b/catalyst/exchange/exchange_blotter.py index c82b2f00..3d85149e 100644 --- a/catalyst/exchange/exchange_blotter.py +++ b/catalyst/exchange/exchange_blotter.py @@ -68,7 +68,7 @@ class TradingPairFeeSchedule(CommissionModel): multiplier = maker \ if ((order.amount > 0 and order.limit < transaction.price) or (order.amount < 0 and order.limit > transaction.price)) \ - and order.limit_reached else taker + and order.limit_reached else taker fee = cost * multiplier return fee From ed406a30ffb8caed1f372a02236dca131c819320 Mon Sep 17 00:00:00 2001 From: lenak25 Date: Sun, 4 Mar 2018 17:42:32 +0200 Subject: [PATCH 66/87] BLD: fix issue #260 - always request more data to avoid empty bars and always give the exact bar number --- catalyst/exchange/exchange.py | 73 ++++++++++++----------- catalyst/exchange/exchange_errors.py | 7 +++ catalyst/exchange/utils/exchange_utils.py | 2 +- 3 files changed, 46 insertions(+), 36 deletions(-) diff --git a/catalyst/exchange/exchange.py b/catalyst/exchange/exchange.py index 310d12fc..e830b90a 100644 --- a/catalyst/exchange/exchange.py +++ b/catalyst/exchange/exchange.py @@ -1,5 +1,4 @@ import abc -import pytz from abc import ABCMeta, abstractmethod, abstractproperty from datetime import timedelta from time import sleep @@ -12,13 +11,14 @@ from catalyst.exchange.exchange_bundle import ExchangeBundle from catalyst.exchange.exchange_errors import MismatchingBaseCurrencies, \ SymbolNotFoundOnExchange, \ PricingDataNotLoadedError, \ - NoDataAvailableOnExchange, NoValueForField, LastCandleTooEarlyError, \ + NoDataAvailableOnExchange, NoValueForField, \ + NoCandlesReceivedFromExchange, \ TickerNotFoundError, NotEnoughCashError from catalyst.exchange.utils.datetime_utils import get_delta, \ get_periods_range, \ get_periods, get_start_dt, get_frequency from catalyst.exchange.utils.exchange_utils import get_exchange_symbols, \ - resample_history_df, has_bundle + resample_history_df, has_bundle, get_candles_df from logbook import Logger log = Logger('Exchange', level=LOG_LEVEL) @@ -256,7 +256,8 @@ class Exchange: elif data_frequency is not None: applies = ( ( - data_frequency == 'minute' and a.end_minute is not None) + data_frequency == 'minute' and + a.end_minute is not None) or ( data_frequency == 'daily' and a.end_daily is not None) ) @@ -505,49 +506,50 @@ class Exchange: freq, candle_size, unit, data_frequency = get_frequency( frequency, data_frequency, supported_freqs=['T', 'D', 'H'] ) + + # we want to avoid receiving empty candles + # so we request more than needed + requested_bar_count = bar_count + 30 # The get_history method supports multiple asset candles = self.get_candles( freq=freq, assets=assets, - bar_count=bar_count, + bar_count=requested_bar_count, end_dt=end_dt if not is_current else None, ) - series = dict() + # candles sanity check - verify no empty candles were received: for asset in candles: - if candles[asset]: - first_candle = candles[asset][0] - asset_series = self.get_series_from_candles( - candles=candles[asset], - start_dt=first_candle['last_traded'], + if not candles[asset]: + raise NoCandlesReceivedFromExchange( + bar_count=requested_bar_count, end_dt=end_dt, - data_frequency=frequency, - field=field, - ) + asset=asset, + exchange=self.name) - delta_candle_size = candle_size * 60 if unit == 'H' else candle_size - # Checking to make sure that the dates match - delta = get_delta(delta_candle_size, data_frequency) - adj_end_dt = end_dt - delta - last_traded = asset_series.index[-1] + series = get_candles_df(candles=candles, + field=field, + freq=frequency, + bar_count=requested_bar_count, + end_dt=end_dt) - if last_traded < adj_end_dt: - raise LastCandleTooEarlyError( - last_traded=last_traded, - end_dt=adj_end_dt, - exchange=self.name, - ) - else: # empty candle received - # because other assets are tz-aware, we need its tz to be set as well - asset_series = pd.Series([], index=pd.DatetimeIndex([], tz=pytz.utc)) - - - series[asset] = asset_series + # TODO: consider how to approach this edge case + # delta_candle_size = candle_size * 60 if unit == 'H' else candle_size + # Checking to make sure that the dates match + # delta = get_delta(delta_candle_size, data_frequency) + # adj_end_dt = end_dt - delta + # last_traded = asset_series.index[-1] + # if last_traded < adj_end_dt: + # raise LastCandleTooEarlyError( + # last_traded=last_traded, + # end_dt=adj_end_dt, + # exchange=self.name, + # ) df = pd.DataFrame(series) - #df.dropna(inplace=True) # commented out due to issue 236 + df.dropna(inplace=True) - return df + return df.tail(bar_count) def get_history_window_with_bundle(self, assets, @@ -595,7 +597,8 @@ class Exchange: A dataframe containing the requested data. """ - # TODO: this function needs some work, we're currently using it just for benchmark data + # TODO: this function needs some work, + # we're currently using it just for benchmark data freq, candle_size, unit, data_frequency = get_frequency( frequency, data_frequency ) @@ -621,7 +624,7 @@ class Exchange: start_dt = get_start_dt(end_dt, adj_bar_count, data_frequency) trailing_dt = \ series[asset].index[-1] + get_delta(1, data_frequency) \ - if asset in series else start_dt + if asset in series else start_dt # The get_history method supports multiple asset # Use the original frequency to let each api optimize diff --git a/catalyst/exchange/exchange_errors.py b/catalyst/exchange/exchange_errors.py index d5af87c4..1d38cd18 100644 --- a/catalyst/exchange/exchange_errors.py +++ b/catalyst/exchange/exchange_errors.py @@ -322,3 +322,10 @@ class BalanceTooLowError(ZiplineError): 'add positions to hold a free amount greater than {amount}, or clean ' 'the state of this algo and restart.' ).strip() + + +class NoCandlesReceivedFromExchange(ZiplineError): + msg = ( + 'Although requesting {bar_count} candles until {end_dt} of asset {asset}, ' + 'an empty list of candles was received for {exchange}.' + ).strip() diff --git a/catalyst/exchange/utils/exchange_utils.py b/catalyst/exchange/utils/exchange_utils.py index 28185091..50e5124a 100644 --- a/catalyst/exchange/utils/exchange_utils.py +++ b/catalyst/exchange/utils/exchange_utils.py @@ -734,7 +734,7 @@ def transform_candles_to_df(candles): return pd.DataFrame(candles).set_index('last_traded') -def get_candles_df(candles, field, freq, bar_count, end_dt=None): +def get_candles_df(candles, field, freq, bar_count, end_dt): all_series = dict() for asset in candles: From 09068a4c37800da69c14d423f5220251710a352a Mon Sep 17 00:00:00 2001 From: lenak25 Date: Sun, 4 Mar 2018 18:20:14 +0200 Subject: [PATCH 67/87] BLD: adjust the example to Python 3 --- catalyst/examples/portfolio_optimization.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/catalyst/examples/portfolio_optimization.py b/catalyst/examples/portfolio_optimization.py index 26c957b6..c6da1f1f 100644 --- a/catalyst/examples/portfolio_optimization.py +++ b/catalyst/examples/portfolio_optimization.py @@ -66,7 +66,7 @@ def handle_data(context, data): # Define portfolio optimization parameters n_portfolios = 50000 results_array = np.zeros((3 + context.nassets, n_portfolios)) - for p in xrange(n_portfolios): + for p in range(n_portfolios): weights = np.random.random(context.nassets) weights /= np.sum(weights) w = np.asmatrix(weights) From 60924711807f62dde30da9b3f6dc23f6128d779e Mon Sep 17 00:00:00 2001 From: lenak25 Date: Mon, 5 Mar 2018 20:42:14 +0200 Subject: [PATCH 68/87] DOC: add some commented TODOs --- catalyst/exchange/exchange.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/catalyst/exchange/exchange.py b/catalyst/exchange/exchange.py index e830b90a..5af13e22 100644 --- a/catalyst/exchange/exchange.py +++ b/catalyst/exchange/exchange.py @@ -509,6 +509,8 @@ class Exchange: # we want to avoid receiving empty candles # so we request more than needed + # TODO: consider defining a const per asset + # and/or some retry mechanism (in each iteration request more data) requested_bar_count = bar_count + 30 # The get_history method supports multiple asset candles = self.get_candles( From cb4668f09350fcd74a020e227f9e2bf392a897d4 Mon Sep 17 00:00:00 2001 From: AvishaiW Date: Tue, 6 Mar 2018 00:05:29 +0200 Subject: [PATCH 69/87] BUG: #243 added a function which reduces open orders amount from calculated target/amount for target orders --- catalyst/exchange/exchange_algorithm.py | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/catalyst/exchange/exchange_algorithm.py b/catalyst/exchange/exchange_algorithm.py index b9c319f1..abaaea14 100644 --- a/catalyst/exchange/exchange_algorithm.py +++ b/catalyst/exchange/exchange_algorithm.py @@ -163,6 +163,25 @@ class ExchangeTradingAlgorithmBase(TradingAlgorithm): style) return amount, style + def _calculate_order_target_amount(self, asset, target): + """ + removes order amounts so we won't run into issues + when two orders are placed one after the other. + it then proceeds to removing positions amount at TradingAlgorithm + :param asset: + :param target: + :return: target + """ + if asset in self.blotter.open_orders: + for open_order in self.blotter.open_orders[asset]: + current_amount = open_order.amount + target -= current_amount + + target = super(ExchangeTradingAlgorithmBase, self). \ + _calculate_order_target_amount(asset, target) + + return target + def round_order(self, amount, asset): """ We need fractions with cryptocurrencies From 4a794aa035ba0e677ab8f81feca9779b5b407a7a Mon Sep 17 00:00:00 2001 From: Victor Grau Serrat Date: Mon, 5 Mar 2018 21:59:18 -0700 Subject: [PATCH 70/87] BLD: show catalyst version at runtime --- catalyst/utils/run_algo.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/catalyst/utils/run_algo.py b/catalyst/utils/run_algo.py index 2e9981fa..224921d0 100644 --- a/catalyst/utils/run_algo.py +++ b/catalyst/utils/run_algo.py @@ -10,6 +10,7 @@ import click import pandas as pd from six import string_types +import catalyst from catalyst.data.bundles import load from catalyst.data.data_portal import DataPortal from catalyst.exchange.exchange_pricing_loader import ExchangePricingLoader, \ @@ -23,7 +24,7 @@ try: from pygments.formatters import TerminalFormatter PYGMENTS = True -except: +except ImportError: PYGMENTS = False from toolz import valfilter, concatv from functools import partial @@ -151,6 +152,7 @@ def _run(handle_data, 'We encourage you to report any issue on GitHub: ' 'https://github.com/enigmampc/catalyst/issues' ) + log.info('Catalyst version {}'.format(catalyst.__version__)) sleep(3) if live: From 89c080fce7a0eed09291492974d69b908c211524 Mon Sep 17 00:00:00 2001 From: Victor Grau Serrat Date: Wed, 7 Mar 2018 16:05:38 -0700 Subject: [PATCH 71/87] BLD: marketplace: dataset param to subscribe optional, fix "from" tx --- catalyst/__main__.py | 4 -- catalyst/marketplace/marketplace.py | 68 ++++++++++++++++++++++------- 2 files changed, 52 insertions(+), 20 deletions(-) diff --git a/catalyst/__main__.py b/catalyst/__main__.py index 03669486..1529d5d4 100644 --- a/catalyst/__main__.py +++ b/catalyst/__main__.py @@ -795,10 +795,6 @@ def ls(ctx): def subscribe(ctx, dataset): """Subscribe to an exisiting dataset. """ - if dataset is None: - ctx.fail("must specify a dataset to subscribe to with '--dataset'\n" - "List available dataset on the marketplace with " - "'catalyst marketplace ls'") marketplace = Marketplace() marketplace.subscribe(dataset) diff --git a/catalyst/marketplace/marketplace.py b/catalyst/marketplace/marketplace.py index e972b750..d3e7383e 100644 --- a/catalyst/marketplace/marketplace.py +++ b/catalyst/marketplace/marketplace.py @@ -7,6 +7,7 @@ import re import shutil import sys import time +import webbrowser import bcolz import logbook @@ -141,10 +142,10 @@ class Marketplace: return address, address_i - def sign_transaction(self, from_address, tx): + def sign_transaction(self, tx): - print('\nVisit https://www.myetherwallet.com/#offline-transaction and ' - 'enter the following parameters:\n\n' + url = 'https://www.myetherwallet.com/#offline-transaction' + print('\nVisit {url} and enter the following parameters:\n\n' 'From Address:\t\t{_from}\n' '\n\tClick the "Generate Information" button\n\n' 'To Address:\t\t{to}\n' @@ -153,7 +154,8 @@ class Marketplace: 'Gas Price:\t\t[Accept the default value]\n' 'Nonce:\t\t\t{nonce}\n' 'Data:\t\t\t{data}\n'.format( - _from=from_address, + url=url, + _from=tx['from'], to=tx['to'], value=tx['value'], gas=tx['gas'], @@ -161,6 +163,8 @@ class Marketplace: data=tx['data'], ) ) + webbrowser.open_new(url) + signed_tx = input('Copy and Paste the "Signed Transaction" ' 'field here:\n') @@ -180,8 +184,7 @@ class Marketplace: print('\nYou can check the outcome of your transaction here:\n' '{}\n\n'.format(etherscan)) - def list(self): - + def _list(self): data_sources = self.mkt_contract.functions.getAllProviders().call() data = [] @@ -193,15 +196,45 @@ class Marketplace: dataset=self.to_text(data_source) ) ) + return pd.DataFrame(data) + + def list(self): + df = self._list() - df = pd.DataFrame(data) set_print_settings() if df.empty: print('There are no datasets available yet.') else: print(df) - def subscribe(self, dataset): + def subscribe(self, dataset=None): + + if dataset is None: + + df_sets = self._list() + if df_sets.empty: + print('There are no datasets available yet.') + return + + set_print_settings() + while True: + print(df_sets) + dataset_num = input('Choose the dataset you want to ' + 'subscribe to [0..{}]: '.format( + df_sets.size-1)) + try: + dataset_num = int(dataset_num) + except ValueError: + print('Enter a number between 0 and {}'.format( + df_sets.size-1)) + else: + if dataset_num not in range(0, df_sets.size): + print('Enter a number between 0 and {}'.format( + df_sets.size-1)) + else: + dataset = df_sets.iloc[dataset_num]['dataset'] + print(dataset) + break dataset = dataset.lower() @@ -292,13 +325,14 @@ class Marketplace: self.mkt_contract_address, grains, ).buildTransaction( - {'nonce': self.web3.eth.getTransactionCount(address)} + {'from': address, + 'nonce': self.web3.eth.getTransactionCount(address)} ) if 'ropsten' in ETH_REMOTE_NODE: tx['gas'] = min(int(tx['gas'] * 1.5), 4700000) - signed_tx = self.sign_transaction(address, tx) + signed_tx = self.sign_transaction(tx) try: tx_hash = '0x{}'.format( bin_hex(self.web3.eth.sendRawTransaction(signed_tx)) @@ -332,14 +366,15 @@ class Marketplace: 'Now processing second transaction.') tx = self.mkt_contract.functions.subscribe( - Web3.toHex(dataset), - ).buildTransaction( - {'nonce': self.web3.eth.getTransactionCount(address)}) + Web3.toHex(dataset), + ).buildTransaction({ + 'from': address, + 'nonce': self.web3.eth.getTransactionCount(address)}) if 'ropsten' in ETH_REMOTE_NODE: tx['gas'] = min(int(tx['gas'] * 1.5), 4700000) - signed_tx = self.sign_transaction(address, tx) + signed_tx = self.sign_transaction(tx) try: tx_hash = '0x{}'.format(bin_hex( @@ -609,13 +644,14 @@ class Marketplace: grains, address, ).buildTransaction( - {'nonce': self.web3.eth.getTransactionCount(address)} + {'from': address, + 'nonce': self.web3.eth.getTransactionCount(address)} ) if 'ropsten' in ETH_REMOTE_NODE: tx['gas'] = min(int(tx['gas'] * 1.5), 4700000) - signed_tx = self.sign_transaction(address, tx) + signed_tx = self.sign_transaction(tx) try: tx_hash = '0x{}'.format( From 218dc0bafd12213ebd02c6028159725b5449ad38 Mon Sep 17 00:00:00 2001 From: Victor Grau Serrat Date: Wed, 7 Mar 2018 22:47:27 -0700 Subject: [PATCH 72/87] BUG: marketplace os.rename -> shutil.move --- catalyst/marketplace/utils/bundle_utils.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/catalyst/marketplace/utils/bundle_utils.py b/catalyst/marketplace/utils/bundle_utils.py index 014887a6..086e6b2c 100644 --- a/catalyst/marketplace/utils/bundle_utils.py +++ b/catalyst/marketplace/utils/bundle_utils.py @@ -29,7 +29,7 @@ def merge_bundles(zsource, ztarget): dirname = os.path.basename(ztarget.rootdir) bak_dir = ztarget.rootdir.replace(dirname, '.{}'.format(dirname)) - os.rename(ztarget.rootdir, bak_dir) + shutil.move(ztarget.rootdir, bak_dir) z = bcolz.ctable.fromdataframe(df=df, rootdir=ztarget.rootdir) shutil.rmtree(bak_dir) From 586d7f2954243c820d10e74f4e30db740795da61 Mon Sep 17 00:00:00 2001 From: AvishaiW Date: Thu, 8 Mar 2018 18:34:25 +0200 Subject: [PATCH 73/87] DOC: added warning that its not possible to start & end at a specific time --- catalyst/utils/run_algo.py | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/catalyst/utils/run_algo.py b/catalyst/utils/run_algo.py index 224921d0..fcf25f15 100644 --- a/catalyst/utils/run_algo.py +++ b/catalyst/utils/run_algo.py @@ -263,6 +263,15 @@ def _run(handle_data, # We still need to support bundles for other misc data, but we # can handle this later. + if start != pd.tslib.normalize_date(start) or \ + end != pd.tslib.normalize_date(end): + # todo: add to Sim_Params the option to start & end at specific times + log.warn( + "Catalyst currently starts and ends on the start and " + "end of the dates specified, respectively. We hope to " + "Modify this and support specific times in a future release." + ) + data = DataPortalExchangeBacktest( exchange_names=[exchange_name for exchange_name in exchanges], asset_finder=None, From 5de89549eef3fba162b0b5be149eed16948f8921 Mon Sep 17 00:00:00 2001 From: Victor Grau Serrat Date: Thu, 8 Mar 2018 10:33:10 -0700 Subject: [PATCH 74/87] MAINT: [mktplace] ingest --dataset parameter now optional --- catalyst/__main__.py | 5 ----- catalyst/marketplace/marketplace.py | 29 +++++++++++++++++++++++++++-- 2 files changed, 27 insertions(+), 7 deletions(-) diff --git a/catalyst/__main__.py b/catalyst/__main__.py index 1529d5d4..74b65819 100644 --- a/catalyst/__main__.py +++ b/catalyst/__main__.py @@ -831,11 +831,6 @@ def subscribe(ctx, dataset): def ingest(ctx, dataset, data_frequency, start, end): """Ingest a dataset (requires subscription). """ - if dataset is None: - ctx.fail("must specify a dataset to clean with '--dataset'\n" - "List available dataset on the marketplace with " - "'catalyst marketplace ls'") - click.echo('Ingesting data: {}'.format(dataset), sys.stdout) marketplace = Marketplace() marketplace.ingest(dataset, data_frequency, start, end) diff --git a/catalyst/marketplace/marketplace.py b/catalyst/marketplace/marketplace.py index d3e7383e..225b2459 100644 --- a/catalyst/marketplace/marketplace.py +++ b/catalyst/marketplace/marketplace.py @@ -233,7 +233,6 @@ class Marketplace: df_sets.size-1)) else: dataset = df_sets.iloc[dataset_num]['dataset'] - print(dataset) break dataset = dataset.lower() @@ -437,7 +436,33 @@ class Marketplace: pass - def ingest(self, ds_name, start=None, end=None, force_download=False): + def ingest(self, ds_name=None, start=None, end=None, force_download=False): + + if ds_name is None: + + df_sets = self._list() + if df_sets.empty: + print('There are no datasets available yet.') + return + + set_print_settings() + while True: + print(df_sets) + dataset_num = input('Choose the dataset you want to ' + 'ingest [0..{}]: '.format( + df_sets.size-1)) + try: + dataset_num = int(dataset_num) + except ValueError: + print('Enter a number between 0 and {}'.format( + df_sets.size-1)) + else: + if dataset_num not in range(0, df_sets.size): + print('Enter a number between 0 and {}'.format( + df_sets.size-1)) + else: + ds_name = df_sets.iloc[dataset_num]['dataset'] + break # ds_name = ds_name.lower() From fc9837b67812a78b6564e182f1eb1fb5966c5e93 Mon Sep 17 00:00:00 2001 From: Frederic Fortier Date: Thu, 8 Mar 2018 17:27:25 -0500 Subject: [PATCH 75/87] BUG: fixed an issue with extracting bundles --- catalyst/marketplace/marketplace.py | 56 ++++++++++++---------- catalyst/marketplace/utils/bundle_utils.py | 54 +++++++++++++++++++++ tests/marketplace/test_marketplace.py | 5 +- 3 files changed, 86 insertions(+), 29 deletions(-) diff --git a/catalyst/marketplace/marketplace.py b/catalyst/marketplace/marketplace.py index 225b2459..22344c1b 100644 --- a/catalyst/marketplace/marketplace.py +++ b/catalyst/marketplace/marketplace.py @@ -33,6 +33,7 @@ from catalyst.marketplace.utils.eth_utils import bin_hex, from_grains, \ from catalyst.marketplace.utils.path_utils import get_bundle_folder, \ get_data_source_folder, get_marketplace_folder, \ get_user_pubaddr, get_temp_bundles_folder, extract_bundle +from catalyst.utils.paths import ensure_directory if sys.version_info.major < 3: import urllib @@ -154,14 +155,14 @@ class Marketplace: 'Gas Price:\t\t[Accept the default value]\n' 'Nonce:\t\t\t{nonce}\n' 'Data:\t\t\t{data}\n'.format( - url=url, - _from=tx['from'], - to=tx['to'], - value=tx['value'], - gas=tx['gas'], - nonce=tx['nonce'], - data=tx['data'], ) - ) + url=url, + _from=tx['from'], + to=tx['to'], + value=tx['value'], + gas=tx['gas'], + nonce=tx['nonce'], + data=tx['data'], ) + ) webbrowser.open_new(url) @@ -221,16 +222,16 @@ class Marketplace: print(df_sets) dataset_num = input('Choose the dataset you want to ' 'subscribe to [0..{}]: '.format( - df_sets.size-1)) + df_sets.size - 1)) try: dataset_num = int(dataset_num) except ValueError: print('Enter a number between 0 and {}'.format( - df_sets.size-1)) + df_sets.size - 1)) else: if dataset_num not in range(0, df_sets.size): print('Enter a number between 0 and {}'.format( - df_sets.size-1)) + df_sets.size - 1)) else: dataset = df_sets.iloc[dataset_num]['dataset'] break @@ -296,14 +297,14 @@ class Marketplace: 'buy: {} ENG. Get enough ENG to cover the costs of the ' 'monthly\nsubscription for what you are trying to buy, ' 'and try again.'.format( - address, from_grains(balance), price)) + address, from_grains(balance), price)) return while True: agree_pay = input('Please confirm that you agree to pay {} ENG ' 'for a monthly subscription to the dataset "{}" ' 'starting today. [default: Y] '.format( - price, dataset)) or 'y' + price, dataset)) or 'y' if agree_pay.lower() not in ('y', 'n'): print("Please answer Y or N.") else: @@ -365,10 +366,10 @@ class Marketplace: 'Now processing second transaction.') tx = self.mkt_contract.functions.subscribe( - Web3.toHex(dataset), - ).buildTransaction({ - 'from': address, - 'nonce': self.web3.eth.getTransactionCount(address)}) + Web3.toHex(dataset), + ).buildTransaction({ + 'from': address, + 'nonce': self.web3.eth.getTransactionCount(address)}) if 'ropsten' in ETH_REMOTE_NODE: tx['gas'] = min(int(tx['gas'] * 1.5), 4700000) @@ -408,7 +409,7 @@ class Marketplace: 'You can now ingest this dataset anytime during the ' 'next month by running the following command:\n' 'catalyst marketplace ingest --dataset={}'.format( - dataset, address, dataset)) + dataset, address, dataset)) def process_temp_bundle(self, ds_name, path): """ @@ -425,7 +426,10 @@ class Marketplace: """ tmp_bundle = extract_bundle(path) - bundle_folder = get_data_source_folder(ds_name) + bundle_folder = os.path.join( + get_data_source_folder(ds_name), 'bundle' + ) + ensure_directory(bundle_folder) if os.listdir(bundle_folder): zsource = bcolz.ctable(rootdir=tmp_bundle, mode='r') ztarget = bcolz.ctable(rootdir=bundle_folder, mode='r') @@ -450,16 +454,16 @@ class Marketplace: print(df_sets) dataset_num = input('Choose the dataset you want to ' 'ingest [0..{}]: '.format( - df_sets.size-1)) + df_sets.size - 1)) try: dataset_num = int(dataset_num) except ValueError: print('Enter a number between 0 and {}'.format( - df_sets.size-1)) + df_sets.size - 1)) else: if dataset_num not in range(0, df_sets.size): print('Enter a number between 0 and {}'.format( - df_sets.size-1)) + df_sets.size - 1)) else: ds_name = df_sets.iloc[dataset_num]['dataset'] break @@ -491,10 +495,10 @@ class Marketplace: print('Your subscription to dataset "{}" expired on {} UTC.' 'Please renew your subscription by running:\n' 'catalyst marketplace subscribe --dataset={}'.format( - ds_name, - pd.to_datetime(check_sub[4], unit='s', utc=True), - ds_name) - ) + ds_name, + pd.to_datetime(check_sub[4], unit='s', utc=True), + ds_name) + ) if 'key' in self.addresses[address_i]: key = self.addresses[address_i]['key'] diff --git a/catalyst/marketplace/utils/bundle_utils.py b/catalyst/marketplace/utils/bundle_utils.py index 086e6b2c..3fc9a45a 100644 --- a/catalyst/marketplace/utils/bundle_utils.py +++ b/catalyst/marketplace/utils/bundle_utils.py @@ -1,8 +1,12 @@ import os +import random +import re import shutil import bcolz +import numpy as np import pandas as pd +from six import string_types def merge_bundles(zsource, ztarget): @@ -27,6 +31,8 @@ def merge_bundles(zsource, ztarget): df.drop_duplicates(inplace=True) df.set_index(['date', 'symbol'], drop=False, inplace=True) + sanitize_df(df) + dirname = os.path.basename(ztarget.rootdir) bak_dir = ztarget.rootdir.replace(dirname, '.{}'.format(dirname)) shutil.move(ztarget.rootdir, bak_dir) @@ -34,3 +40,51 @@ def merge_bundles(zsource, ztarget): z = bcolz.ctable.fromdataframe(df=df, rootdir=ztarget.rootdir) shutil.rmtree(bak_dir) return z + + +def sanitize_df(df): + # Using a sampling method to identify dates for efficiency with + # large datasets + if len(df) > 100: + indexes = random.sample(range(0, len(df) - 1), 100) + else: + indexes = range(0, len(df) - 1) + + for column in df.columns: + is_date = False + for index in indexes: + value = df[column].iloc[index] + if not isinstance(value, string_types): + continue + + # TODO: assuming that the date is at least daily + exp = re.compile(r'^\d{4}-\d{2}-\d{2}.*$') + matches = exp.findall(value) + + if matches: + is_date = True + break + + if is_date: + df[column] = pd.to_datetime(df[column]) + + else: + try: + ser = safely_reduce_dtype(df[column]) + df[column] = ser + except Exception: + pass + + return df + + +def safely_reduce_dtype(ser): # pandas.Series or numpy.array + orig_dtype = "".join( + [x for x in ser.dtype.name if x.isalpha()]) # float/int + mx = 1 + for val in ser.values: + new_itemsize = np.min_scalar_type(val).itemsize + if mx < new_itemsize: + mx = new_itemsize + new_dtype = orig_dtype + str(mx * 8) + return ser.astype(new_dtype) diff --git a/tests/marketplace/test_marketplace.py b/tests/marketplace/test_marketplace.py index c0895b5a..017f4e86 100644 --- a/tests/marketplace/test_marketplace.py +++ b/tests/marketplace/test_marketplace.py @@ -1,6 +1,5 @@ from catalyst.marketplace.marketplace import Marketplace from catalyst.testing.fixtures import WithLogger, ZiplineTestCase -import pandas as pd class TestMarketplace(WithLogger, ZiplineTestCase): @@ -16,12 +15,12 @@ class TestMarketplace(WithLogger, ZiplineTestCase): def test_subscribe(self): marketplace = Marketplace() - marketplace.subscribe('marketcap2222') + marketplace.subscribe('marketcap') pass def test_ingest(self): marketplace = Marketplace() - ds_def = marketplace.ingest('github') + ds_def = marketplace.ingest('marketcap') pass def test_publish(self): From b0dce13672e23d7acd8f6e9dc7c6c2ed6bbfc0c5 Mon Sep 17 00:00:00 2001 From: AvishaiW Date: Fri, 9 Mar 2018 11:01:19 +0200 Subject: [PATCH 76/87] BUG: removed get_open_orders from buy_low_sell_high --- catalyst/examples/buy_low_sell_high.py | 1 - 1 file changed, 1 deletion(-) diff --git a/catalyst/examples/buy_low_sell_high.py b/catalyst/examples/buy_low_sell_high.py index 075e2f71..ed5e211c 100644 --- a/catalyst/examples/buy_low_sell_high.py +++ b/catalyst/examples/buy_low_sell_high.py @@ -7,7 +7,6 @@ from catalyst.api import ( order_target_percent, symbol, record, - get_open_orders, ) from catalyst.exchange.utils.stats_utils import get_pretty_stats from catalyst.utils.run_algo import run_algorithm From f0606b5ea4f8ae32804e058b6a235eac834e269b Mon Sep 17 00:00:00 2001 From: Victor Grau Serrat Date: Fri, 9 Mar 2018 09:50:20 -0700 Subject: [PATCH 77/87] BLD: [mktplace] clean --dataset param optional --- catalyst/__main__.py | 6 --- catalyst/marketplace/marketplace.py | 69 ++++++++++++++++++++--------- 2 files changed, 48 insertions(+), 27 deletions(-) diff --git a/catalyst/__main__.py b/catalyst/__main__.py index 74b65819..f6505054 100644 --- a/catalyst/__main__.py +++ b/catalyst/__main__.py @@ -845,14 +845,8 @@ def ingest(ctx, dataset, data_frequency, start, end): def clean(ctx, dataset): """Clean/Remove local data for a given dataset. """ - if dataset is None: - ctx.fail("must specify a dataset to clean up with '--dataset'\n" - "List available dataset on the marketplace with " - "'catalyst marketplace ls'") - click.echo('Cleaning data source: {}'.format(dataset), sys.stdout) marketplace = Marketplace() marketplace.clean(dataset) - click.echo('Done', sys.stdout) @marketplace.command() diff --git a/catalyst/marketplace/marketplace.py b/catalyst/marketplace/marketplace.py index 22344c1b..ec2627f4 100644 --- a/catalyst/marketplace/marketplace.py +++ b/catalyst/marketplace/marketplace.py @@ -155,14 +155,14 @@ class Marketplace: 'Gas Price:\t\t[Accept the default value]\n' 'Nonce:\t\t\t{nonce}\n' 'Data:\t\t\t{data}\n'.format( - url=url, - _from=tx['from'], - to=tx['to'], - value=tx['value'], - gas=tx['gas'], - nonce=tx['nonce'], - data=tx['data'], ) - ) + url=url, + _from=tx['from'], + to=tx['to'], + value=tx['value'], + gas=tx['gas'], + nonce=tx['nonce'], + data=tx['data'], ) + ) webbrowser.open_new(url) @@ -222,7 +222,7 @@ class Marketplace: print(df_sets) dataset_num = input('Choose the dataset you want to ' 'subscribe to [0..{}]: '.format( - df_sets.size - 1)) + df_sets.size - 1)) try: dataset_num = int(dataset_num) except ValueError: @@ -297,14 +297,14 @@ class Marketplace: 'buy: {} ENG. Get enough ENG to cover the costs of the ' 'monthly\nsubscription for what you are trying to buy, ' 'and try again.'.format( - address, from_grains(balance), price)) + address, from_grains(balance), price)) return while True: agree_pay = input('Please confirm that you agree to pay {} ENG ' 'for a monthly subscription to the dataset "{}" ' 'starting today. [default: Y] '.format( - price, dataset)) or 'y' + price, dataset)) or 'y' if agree_pay.lower() not in ('y', 'n'): print("Please answer Y or N.") else: @@ -409,7 +409,7 @@ class Marketplace: 'You can now ingest this dataset anytime during the ' 'next month by running the following command:\n' 'catalyst marketplace ingest --dataset={}'.format( - dataset, address, dataset)) + dataset, address, dataset)) def process_temp_bundle(self, ds_name, path): """ @@ -429,6 +429,7 @@ class Marketplace: bundle_folder = os.path.join( get_data_source_folder(ds_name), 'bundle' ) + ensure_directory(bundle_folder) if os.listdir(bundle_folder): zsource = bcolz.ctable(rootdir=tmp_bundle, mode='r') @@ -454,7 +455,7 @@ class Marketplace: print(df_sets) dataset_num = input('Choose the dataset you want to ' 'ingest [0..{}]: '.format( - df_sets.size - 1)) + df_sets.size - 1)) try: dataset_num = int(dataset_num) except ValueError: @@ -495,10 +496,10 @@ class Marketplace: print('Your subscription to dataset "{}" expired on {} UTC.' 'Please renew your subscription by running:\n' 'catalyst marketplace subscribe --dataset={}'.format( - ds_name, - pd.to_datetime(check_sub[4], unit='s', utc=True), - ds_name) - ) + ds_name, + pd.to_datetime(check_sub[4], unit='s', utc=True), + ds_name) + ) if 'key' in self.addresses[address_i]: key = self.addresses[address_i]['key'] @@ -562,14 +563,40 @@ class Marketplace: return df - def clean(self, data_source_name, data_frequency=None): - data_source_name = data_source_name.lower() + def clean(self, ds_name=None, data_frequency=None): + + if ds_name is None: + mktplace_root = get_marketplace_folder() + folders = [os.path.basename(f.rstrip('/')) + for f in glob.glob('{}/*/'.format(mktplace_root)) + if 'temp_bundles' not in f] + + while True: + for idx, f in enumerate(folders): + print('{}\t{}'.format(idx, f)) + dataset_num = input('Choose the dataset you want to ' + 'clean [0..{}]: '.format( + len(folders) - 1)) + try: + dataset_num = int(dataset_num) + except ValueError: + print('Enter a number between 0 and {}'.format( + len(folders) - 1)) + else: + if dataset_num not in range(0, len(folders)): + print('Enter a number between 0 and {}'.format( + len(folders) - 1)) + else: + ds_name = folders[dataset_num] + break + + ds_name = ds_name.lower() if data_frequency is None: - folder = get_data_source_folder(data_source_name) + folder = get_data_source_folder(ds_name) else: - folder = get_bundle_folder(data_source_name, data_frequency) + folder = get_bundle_folder(ds_name, data_frequency) shutil.rmtree(folder) pass From 4694372496a6e1d008dd1bad540162e596729ecf Mon Sep 17 00:00:00 2001 From: Victor Grau Serrat Date: Fri, 9 Mar 2018 09:56:09 -0700 Subject: [PATCH 78/87] BUG: [mktplace] ingest mismatch dest. folder --- catalyst/marketplace/marketplace.py | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/catalyst/marketplace/marketplace.py b/catalyst/marketplace/marketplace.py index ec2627f4..fbaaed3a 100644 --- a/catalyst/marketplace/marketplace.py +++ b/catalyst/marketplace/marketplace.py @@ -426,10 +426,7 @@ class Marketplace: """ tmp_bundle = extract_bundle(path) - bundle_folder = os.path.join( - get_data_source_folder(ds_name), 'bundle' - ) - + bundle_folder = get_data_source_folder(ds_name) ensure_directory(bundle_folder) if os.listdir(bundle_folder): zsource = bcolz.ctable(rootdir=tmp_bundle, mode='r') From 93ca4e990d73b1ef0bfdc7746e1aa185631c0b28 Mon Sep 17 00:00:00 2001 From: Frederic Fortier Date: Fri, 9 Mar 2018 13:20:50 -0500 Subject: [PATCH 79/87] BUG: fixed a dependency issue --- tests/exchange/test_bundle.py | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/tests/exchange/test_bundle.py b/tests/exchange/test_bundle.py index c66fcfb4..d88723b2 100644 --- a/tests/exchange/test_bundle.py +++ b/tests/exchange/test_bundle.py @@ -11,7 +11,7 @@ from catalyst.exchange.exchange_bundle import ExchangeBundle, \ BUNDLE_NAME_TEMPLATE from catalyst.exchange.utils.bundle_utils import get_bcolz_chunk, \ get_df_from_arrays -from exchange.utils.datetime_utils import get_start_dt +from catalyst.exchange.utils.datetime_utils import get_start_dt from catalyst.exchange.utils.exchange_utils import get_exchange_folder from catalyst.exchange.utils.factory import get_exchange from catalyst.exchange.utils.stats_utils import df_to_string @@ -42,7 +42,7 @@ class TestExchangeBundle: def test_ingest_minute(self): data_frequency = 'minute' - exchange_name = 'poloniex' + exchange_name = 'binance' exchange = get_exchange(exchange_name) exchange_bundle = ExchangeBundle(exchange) @@ -50,8 +50,8 @@ class TestExchangeBundle: exchange.get_asset('eth_btc') ] - start = pd.to_datetime('2016-03-01', utc=True) - end = pd.to_datetime('2017-11-1', utc=True) + start = pd.to_datetime('2018-03-01', utc=True) + end = pd.to_datetime('2018-03-8', utc=True) log.info('ingesting exchange bundle {}'.format(exchange_name)) exchange_bundle.ingest( @@ -101,7 +101,7 @@ class TestExchangeBundle: # data_frequency = 'daily' # include_symbols = 'neo_btc,bch_btc,eth_btc' - exchange_name = 'bitfinex' + exchange_name = 'binance' data_frequency = 'minute' exchange = get_exchange(exchange_name) From 8f7d67817056e6ec9fba2a292f2c43ecb215c49c Mon Sep 17 00:00:00 2001 From: Victor Grau Serrat Date: Fri, 9 Mar 2018 12:44:01 -0700 Subject: [PATCH 80/87] BUG: [mktplace]: fix sanitize_df to handle 1-row DFs --- catalyst/marketplace/utils/bundle_utils.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/catalyst/marketplace/utils/bundle_utils.py b/catalyst/marketplace/utils/bundle_utils.py index 3fc9a45a..e489c0b5 100644 --- a/catalyst/marketplace/utils/bundle_utils.py +++ b/catalyst/marketplace/utils/bundle_utils.py @@ -47,8 +47,10 @@ def sanitize_df(df): # large datasets if len(df) > 100: indexes = random.sample(range(0, len(df) - 1), 100) - else: + elif len(df) > 1: indexes = range(0, len(df) - 1) + else: + indexes = [0, ] for column in df.columns: is_date = False From 0a37cdec5b4438cbe9202bf8a42f7571ba806a2a Mon Sep 17 00:00:00 2001 From: lenak25 Date: Sun, 11 Mar 2018 19:36:28 +0200 Subject: [PATCH 81/87] BLD: fix 'on the clock' candles fetch and request extra candles using a fixed time interval --- catalyst/exchange/exchange.py | 15 ++++++++--- catalyst/exchange/utils/datetime_utils.py | 31 +++++++++++++++++++++++ 2 files changed, 43 insertions(+), 3 deletions(-) diff --git a/catalyst/exchange/exchange.py b/catalyst/exchange/exchange.py index 5af13e22..c86b8cdd 100644 --- a/catalyst/exchange/exchange.py +++ b/catalyst/exchange/exchange.py @@ -16,7 +16,8 @@ from catalyst.exchange.exchange_errors import MismatchingBaseCurrencies, \ TickerNotFoundError, NotEnoughCashError from catalyst.exchange.utils.datetime_utils import get_delta, \ get_periods_range, \ - get_periods, get_start_dt, get_frequency + get_periods, get_start_dt, get_frequency, \ + get_candles_number_from_minutes from catalyst.exchange.utils.exchange_utils import get_exchange_symbols, \ resample_history_df, has_bundle, get_candles_df from logbook import Logger @@ -511,7 +512,12 @@ class Exchange: # so we request more than needed # TODO: consider defining a const per asset # and/or some retry mechanism (in each iteration request more data) - requested_bar_count = bar_count + 30 + kExtra_minutes_candles = 150 + requested_bar_count = bar_count + \ + get_candles_number_from_minutes(unit, + candle_size, + kExtra_minutes_candles) + # The get_history method supports multiple asset candles = self.get_candles( freq=freq, @@ -529,11 +535,14 @@ class Exchange: asset=asset, exchange=self.name) + # for avoiding unnecessary forward fill end_dt is taken back one second + forward_fill_till_dt = end_dt - timedelta(seconds=1) + series = get_candles_df(candles=candles, field=field, freq=frequency, bar_count=requested_bar_count, - end_dt=end_dt) + end_dt=forward_fill_till_dt) # TODO: consider how to approach this edge case # delta_candle_size = candle_size * 60 if unit == 'H' else candle_size diff --git a/catalyst/exchange/utils/datetime_utils.py b/catalyst/exchange/utils/datetime_utils.py index c34b0c69..b5a03c49 100644 --- a/catalyst/exchange/utils/datetime_utils.py +++ b/catalyst/exchange/utils/datetime_utils.py @@ -1,4 +1,5 @@ import calendar +import math import re from datetime import datetime, timedelta, date @@ -326,3 +327,33 @@ def from_ms_timestamp(ms): def get_epoch(): return pd.to_datetime('1970-1-1', utc=True) + + +def get_candles_number_from_minutes(unit, candle_size, minutes): + """ + Get the number of bars needed for the given time interval + in minutes. + + Notes + ----- + Supports only "T", "D" and "H" units + + Parameters + ---------- + unit: str + candle_size : int + minutes: int + + Returns + ------- + int + + """ + if unit == "T": + res = (float(minutes) / candle_size) + elif unit == "H": + res = (minutes / 60.0) / candle_size + else: # unit == "D" + res = (minutes / 1440.0) / candle_size + + return int(math.ceil(res)) From 8d86a5548f5f1be68a02f6cbca74255f309f5c1a Mon Sep 17 00:00:00 2001 From: lenak25 Date: Mon, 12 Mar 2018 18:04:38 +0200 Subject: [PATCH 82/87] DOC: add ta_lib troubleshooting to the docs --- docs/source/install.rst | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/docs/source/install.rst b/docs/source/install.rst index 0bd3ff48..e4ae99dd 100644 --- a/docs/source/install.rst +++ b/docs/source/install.rst @@ -314,6 +314,16 @@ Troubleshooting ``pip`` Install $ sudo apt-get install python-dev +---- + +**Issue**: + Missing TA_Lib + +**Solution**: + Follow `these instructions + `_ to install the TA_Lib Python wrapper + (and if needed, its underlying C library as well). + .. _pipenv: Installing with ``pipenv`` From 127d779eb10885da3dea5afc01947ac9a012067e Mon Sep 17 00:00:00 2001 From: Victor Grau Serrat Date: Mon, 12 Mar 2018 16:52:54 -0600 Subject: [PATCH 83/87] BUG: fix sanitize_df to min of int32 --- catalyst/marketplace/utils/bundle_utils.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/catalyst/marketplace/utils/bundle_utils.py b/catalyst/marketplace/utils/bundle_utils.py index e489c0b5..5a0b2f6c 100644 --- a/catalyst/marketplace/utils/bundle_utils.py +++ b/catalyst/marketplace/utils/bundle_utils.py @@ -88,5 +88,7 @@ def safely_reduce_dtype(ser): # pandas.Series or numpy.array new_itemsize = np.min_scalar_type(val).itemsize if mx < new_itemsize: mx = new_itemsize + if orig_dtype == 'int': + mx = max(mx, 4) new_dtype = orig_dtype + str(mx * 8) return ser.astype(new_dtype) From 69731b653d3362634556394677a7b76cdf81cde7 Mon Sep 17 00:00:00 2001 From: lenak25 Date: Tue, 13 Mar 2018 18:42:28 +0200 Subject: [PATCH 84/87] BLD: revert hourly freq support reported at issue #227 --- catalyst/examples/simple_loop.py | 2 +- catalyst/exchange/exchange.py | 5 +++++ 2 files changed, 6 insertions(+), 1 deletion(-) diff --git a/catalyst/examples/simple_loop.py b/catalyst/examples/simple_loop.py index 99822b70..bc356e0a 100644 --- a/catalyst/examples/simple_loop.py +++ b/catalyst/examples/simple_loop.py @@ -26,7 +26,7 @@ def handle_data(context, data): context.asset, fields='price', bar_count=20, - frequency='2H' + frequency='30T' ) last_traded = prices.index[-1] log.info('last candle date: {}'.format(last_traded)) diff --git a/catalyst/exchange/exchange.py b/catalyst/exchange/exchange.py index c86b8cdd..73593c6d 100644 --- a/catalyst/exchange/exchange.py +++ b/catalyst/exchange/exchange.py @@ -13,6 +13,7 @@ from catalyst.exchange.exchange_errors import MismatchingBaseCurrencies, \ PricingDataNotLoadedError, \ NoDataAvailableOnExchange, NoValueForField, \ NoCandlesReceivedFromExchange, \ + InvalidHistoryFrequencyAlias, \ TickerNotFoundError, NotEnoughCashError from catalyst.exchange.utils.datetime_utils import get_delta, \ get_periods_range, \ @@ -508,6 +509,10 @@ class Exchange: frequency, data_frequency, supported_freqs=['T', 'D', 'H'] ) + if unit == 'H': + raise InvalidHistoryFrequencyAlias( + freq=frequency) + # we want to avoid receiving empty candles # so we request more than needed # TODO: consider defining a const per asset From 3e69449a6bf98af1cf93dd638ef5aba1a391d571 Mon Sep 17 00:00:00 2001 From: Victor Grau Serrat Date: Tue, 13 Mar 2018 22:30:45 -0600 Subject: [PATCH 85/87] BLD: marketplace switch to rinkeby post-audit --- catalyst/constants.py | 13 ++++++------- .../marketplace/contract_enigma_address.txt | 2 +- .../marketplace/contract_marketplace_abi.json | 2 +- .../contract_marketplace_address.txt | 2 +- catalyst/marketplace/marketplace.py | 17 +++++------------ 5 files changed, 14 insertions(+), 22 deletions(-) diff --git a/catalyst/constants.py b/catalyst/constants.py index b29d6f62..2bac11c2 100644 --- a/catalyst/constants.py +++ b/catalyst/constants.py @@ -25,22 +25,21 @@ AUTO_INGEST = False AUTH_SERVER = 'https://data.enigma.co' # TODO: switch to mainnet -ETH_REMOTE_NODE = 'https://ropsten.infura.io/' - +ETH_REMOTE_NODE = 'https://rinkeby.infura.io/' MARKETPLACE_CONTRACT = 'https://raw.githubusercontent.com/enigmampc/' \ - 'catalyst/master/catalyst/marketplace/' \ + 'catalyst/rinkeby/catalyst/marketplace/' \ 'contract_marketplace_address.txt' MARKETPLACE_CONTRACT_ABI = 'https://raw.githubusercontent.com/enigmampc/' \ - 'catalyst/master/catalyst/marketplace/' \ + 'catalyst/rinkeby/catalyst/marketplace/' \ 'contract_marketplace_abi.json' # TODO: switch to mainnet -ENIGMA_CONTRACT = 'https://raw.githubusercontent.com/enigmampc/catalyst/' \ - 'master/catalyst/marketplace/' \ +ENIGMA_CONTRACT = 'https://raw.githubusercontent.com/enigmampc/' \ + 'catalyst/rinkeby/catalyst/marketplace/' \ 'contract_enigma_address.txt' ENIGMA_CONTRACT_ABI = 'https://raw.githubusercontent.com/enigmampc/' \ - 'catalyst/master/catalyst/marketplace/' \ + 'catalyst/rinkeby/catalyst/marketplace/' \ 'contract_enigma_abi.json' diff --git a/catalyst/marketplace/contract_enigma_address.txt b/catalyst/marketplace/contract_enigma_address.txt index 90fcd06a..8ce1f421 100644 --- a/catalyst/marketplace/contract_enigma_address.txt +++ b/catalyst/marketplace/contract_enigma_address.txt @@ -1 +1 @@ -0x7fAec9aaE31BE428DeAAE1be8195dF609079Fd10 \ No newline at end of file +0x39a54f480d922a58c963de8091a6c9afc69db2cf diff --git a/catalyst/marketplace/contract_marketplace_abi.json b/catalyst/marketplace/contract_marketplace_abi.json index 4f0e2460..220a09cd 100644 --- a/catalyst/marketplace/contract_marketplace_abi.json +++ b/catalyst/marketplace/contract_marketplace_abi.json @@ -1 +1 @@ -[{"constant":true,"inputs":[{"name":"_dataSourceName","type":"bytes32"}],"name":"isActiveDataSource","outputs":[{"name":"isActive","type":"bool"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":true,"inputs":[],"name":"mBegin","outputs":[{"name":"","type":"bytes32"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":true,"inputs":[],"name":"FIXED_SUBSCRIPTION_PERIOD","outputs":[{"name":"","type":"uint256"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":false,"inputs":[{"name":"_dataSourceName","type":"bytes32"},{"name":"_isPunished","type":"bool"}],"name":"setPunishProvider","outputs":[{"name":"success","type":"bool"}],"payable":false,"stateMutability":"nonpayable","type":"function"},{"constant":true,"inputs":[{"name":"_dataSourceName","type":"bytes32"}],"name":"getDataProviderInfo","outputs":[{"name":"owner","type":"address"},{"name":"price","type":"uint256"},{"name":"volume","type":"uint256"},{"name":"subscriptionsNum","type":"uint256"},{"name":"isProvider","type":"bool"},{"name":"isActive","type":"bool"},{"name":"isPunished","type":"bool"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":false,"inputs":[{"name":"_dataSourceName","type":"bytes32"}],"name":"subscribe","outputs":[{"name":"success","type":"bool"}],"payable":false,"stateMutability":"nonpayable","type":"function"},{"constant":true,"inputs":[],"name":"mProvidersSize","outputs":[{"name":"","type":"uint256"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":false,"inputs":[{"name":"_dataSourceName","type":"bytes32"},{"name":"_newPrice","type":"uint256"}],"name":"updateDataSourcePrice","outputs":[{"name":"success","type":"bool"}],"payable":false,"stateMutability":"nonpayable","type":"function"},{"constant":false,"inputs":[{"name":"_dataSourceName","type":"bytes32"}],"name":"withdrawProvider","outputs":[{"name":"success","type":"bool"}],"payable":false,"stateMutability":"nonpayable","type":"function"},{"constant":true,"inputs":[],"name":"getAllProviders","outputs":[{"name":"","type":"bytes32[]"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":true,"inputs":[],"name":"getMarketplaceTotalBalance","outputs":[{"name":"totalBalance","type":"uint256"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":false,"inputs":[{"name":"_dataSourceName","type":"bytes32"}],"name":"refundSubscriber","outputs":[{"name":"success","type":"bool"}],"payable":false,"stateMutability":"nonpayable","type":"function"},{"constant":false,"inputs":[{"name":"_dataSourceName","type":"bytes32"},{"name":"_price","type":"uint256"},{"name":"_dataOwner","type":"address"}],"name":"register","outputs":[{"name":"success","type":"bool"}],"payable":false,"stateMutability":"nonpayable","type":"function"},{"constant":true,"inputs":[],"name":"mCurrent","outputs":[{"name":"","type":"bytes32"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":true,"inputs":[{"name":"_dataSourceName","type":"bytes32"}],"name":"getWithdrawAmount","outputs":[{"name":"withdrawAmount","type":"uint256"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":true,"inputs":[{"name":"_subscriber","type":"address"},{"name":"_dataSourceName","type":"bytes32"}],"name":"checkAddressSubscription","outputs":[{"name":"subscriber","type":"address"},{"name":"dataSourceName","type":"bytes32"},{"name":"price","type":"uint256"},{"name":"startTime","type":"uint256"},{"name":"endTime","type":"uint256"},{"name":"isUnExpired","type":"bool"},{"name":"isPaid","type":"bool"},{"name":"isPunishedProvider","type":"bool"},{"name":"isOrder","type":"bool"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":false,"inputs":[{"name":"_dataSourceName","type":"bytes32"},{"name":"_isActive","type":"bool"}],"name":"changeDataSourceActivityStatus","outputs":[{"name":"success","type":"bool"}],"payable":false,"stateMutability":"nonpayable","type":"function"},{"constant":true,"inputs":[{"name":"","type":"bytes32"}],"name":"mProviders","outputs":[{"name":"owner","type":"address"},{"name":"volume","type":"uint256"},{"name":"subscriptionsNum","type":"uint256"},{"name":"name","type":"bytes32"},{"name":"price","type":"uint256"},{"name":"isPunished","type":"bool"},{"name":"punishTimeStamp","type":"uint256"},{"name":"isProvider","type":"bool"},{"name":"isActive","type":"bool"},{"name":"nextProvider","type":"bytes32"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":true,"inputs":[{"name":"","type":"bytes32"},{"name":"","type":"uint256"}],"name":"mOrders","outputs":[{"name":"dataSourceName","type":"bytes32"},{"name":"subscriber","type":"address"},{"name":"provider","type":"address"},{"name":"price","type":"uint256"},{"name":"startTime","type":"uint256"},{"name":"endTime","type":"uint256"},{"name":"isPaid","type":"bool"},{"name":"isOrder","type":"bool"},{"name":"isRefundPaid","type":"bool"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":true,"inputs":[],"name":"mToken","outputs":[{"name":"","type":"address"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":true,"inputs":[{"name":"_dataSourceName","type":"bytes32"}],"name":"getOwnerFromName","outputs":[{"name":"owner","type":"address"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":true,"inputs":[{"name":"_subscriber","type":"address"},{"name":"_dataSourceName","type":"bytes32"}],"name":"getRefundAmount","outputs":[{"name":"refundAmount","type":"uint256"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":true,"inputs":[],"name":"mOwner","outputs":[{"name":"","type":"address"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":true,"inputs":[],"name":"MARKETPLACE_VERSION","outputs":[{"name":"","type":"string"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":true,"inputs":[{"name":"_subscriber","type":"address"},{"name":"_dataSourceName","type":"bytes32"}],"name":"isExpiredSubscription","outputs":[{"name":"isExpired","type":"bool"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":false,"inputs":[{"name":"_newOwner","type":"address"}],"name":"transferOwnership","outputs":[],"payable":false,"stateMutability":"nonpayable","type":"function"},{"inputs":[{"name":"_tokenAddress","type":"address"}],"payable":false,"stateMutability":"nonpayable","type":"constructor"},{"anonymous":false,"inputs":[{"indexed":true,"name":"previousOwner","type":"address"},{"indexed":true,"name":"newOwner","type":"address"}],"name":"OwnershipTransferred","type":"event"},{"anonymous":false,"inputs":[{"indexed":true,"name":"editor","type":"address"},{"indexed":true,"name":"dataSourceName","type":"bytes32"},{"indexed":false,"name":"newPrice","type":"uint256"}],"name":"PriceUpdate","type":"event"},{"anonymous":false,"inputs":[{"indexed":true,"name":"editor","type":"address"},{"indexed":true,"name":"dataSourceName","type":"bytes32"},{"indexed":false,"name":"newStatus","type":"bool"}],"name":"ActivityUpdate","type":"event"},{"anonymous":false,"inputs":[{"indexed":true,"name":"dataOwner","type":"address"},{"indexed":true,"name":"dataSourceName","type":"bytes32"},{"indexed":false,"name":"price","type":"uint256"},{"indexed":false,"name":"success","type":"bool"}],"name":"Registered","type":"event"},{"anonymous":false,"inputs":[{"indexed":true,"name":"from","type":"address"},{"indexed":true,"name":"to","type":"address"},{"indexed":false,"name":"value","type":"uint256"}],"name":"SubscriptionDeposited","type":"event"},{"anonymous":false,"inputs":[{"indexed":true,"name":"subscriber","type":"address"},{"indexed":true,"name":"dataSourceName","type":"bytes32"},{"indexed":true,"name":"dataOwner","type":"address"},{"indexed":false,"name":"price","type":"uint256"},{"indexed":false,"name":"success","type":"bool"}],"name":"Subscribed","type":"event"},{"anonymous":false,"inputs":[{"indexed":true,"name":"dataOwner","type":"address"},{"indexed":true,"name":"dataSourceName","type":"bytes32"},{"indexed":false,"name":"amount","type":"uint256"}],"name":"TransferToProvider","type":"event"},{"anonymous":false,"inputs":[{"indexed":true,"name":"dataOwner","type":"address"},{"indexed":true,"name":"dataSourceName","type":"bytes32"},{"indexed":false,"name":"amount","type":"uint256"}],"name":"ProviderWithdraw","type":"event"},{"anonymous":false,"inputs":[{"indexed":true,"name":"dataOwner","type":"address"},{"indexed":true,"name":"dataSourceName","type":"bytes32"},{"indexed":false,"name":"isPunished","type":"bool"}],"name":"ProviderPunishStatus","type":"event"},{"anonymous":false,"inputs":[{"indexed":true,"name":"subscriber","type":"address"},{"indexed":true,"name":"dataSourceName","type":"bytes32"},{"indexed":false,"name":"refundAmount","type":"uint256"}],"name":"SubscriberRefund","type":"event"}] \ No newline at end of file +[{"constant":true,"inputs":[{"name":"_dataSourceName","type":"bytes32"}],"name":"isActiveDataSource","outputs":[{"name":"isActive","type":"bool"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":true,"inputs":[],"name":"mBegin","outputs":[{"name":"","type":"bytes32"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":true,"inputs":[{"name":"_index","type":"uint256"}],"name":"getNameAt","outputs":[{"name":"_dataSourceName","type":"bytes32"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":false,"inputs":[{"name":"_dataSourceName","type":"bytes32"},{"name":"_index","type":"uint256"}],"name":"refundSubscriberAt","outputs":[{"name":"success","type":"bool"}],"payable":false,"stateMutability":"nonpayable","type":"function"},{"constant":true,"inputs":[],"name":"FIXED_SUBSCRIPTION_PERIOD","outputs":[{"name":"","type":"uint256"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":true,"inputs":[{"name":"_dataSourceName","type":"bytes32"},{"name":"_index","type":"uint256"}],"name":"getWithdrawAmountAt","outputs":[{"name":"withdrawAmount","type":"uint256"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":false,"inputs":[{"name":"_dataSourceName","type":"bytes32"},{"name":"_index","type":"uint256"}],"name":"withrawProviderAt","outputs":[{"name":"success","type":"bool"}],"payable":false,"stateMutability":"nonpayable","type":"function"},{"constant":false,"inputs":[{"name":"_dataSourceName","type":"bytes32"},{"name":"_isPunished","type":"bool"}],"name":"setPunishProvider","outputs":[{"name":"success","type":"bool"}],"payable":false,"stateMutability":"nonpayable","type":"function"},{"constant":true,"inputs":[{"name":"_dataSourceName","type":"bytes32"}],"name":"getDataProviderInfo","outputs":[{"name":"owner","type":"address"},{"name":"price","type":"uint256"},{"name":"volume","type":"uint256"},{"name":"subscriptionsNum","type":"uint256"},{"name":"isProvider","type":"bool"},{"name":"isActive","type":"bool"},{"name":"isPunished","type":"bool"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":false,"inputs":[{"name":"_dataSourceName","type":"bytes32"}],"name":"subscribe","outputs":[{"name":"success","type":"bool"}],"payable":false,"stateMutability":"nonpayable","type":"function"},{"constant":true,"inputs":[],"name":"mProvidersSize","outputs":[{"name":"","type":"uint256"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":false,"inputs":[{"name":"_dataSourceName","type":"bytes32"},{"name":"_newPrice","type":"uint256"}],"name":"updateDataSourcePrice","outputs":[{"name":"success","type":"bool"}],"payable":false,"stateMutability":"nonpayable","type":"function"},{"constant":false,"inputs":[{"name":"_dataSourceName","type":"bytes32"}],"name":"withdrawProvider","outputs":[{"name":"success","type":"bool"}],"payable":false,"stateMutability":"nonpayable","type":"function"},{"constant":true,"inputs":[],"name":"getAllProviders","outputs":[{"name":"","type":"bytes32[]"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":true,"inputs":[],"name":"getMarketplaceTotalBalance","outputs":[{"name":"totalBalance","type":"uint256"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":false,"inputs":[{"name":"_dataSourceName","type":"bytes32"}],"name":"refundSubscriber","outputs":[{"name":"success","type":"bool"}],"payable":false,"stateMutability":"nonpayable","type":"function"},{"constant":false,"inputs":[{"name":"_dataSourceName","type":"bytes32"},{"name":"_price","type":"uint256"},{"name":"_dataOwner","type":"address"}],"name":"register","outputs":[{"name":"success","type":"bool"}],"payable":false,"stateMutability":"nonpayable","type":"function"},{"constant":true,"inputs":[],"name":"mCurrent","outputs":[{"name":"","type":"bytes32"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":true,"inputs":[{"name":"_dataSourceName","type":"bytes32"}],"name":"getWithdrawAmount","outputs":[{"name":"withdrawAmount","type":"uint256"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":true,"inputs":[{"name":"_dataSourceName","type":"bytes32"},{"name":"_index","type":"uint256"}],"name":"getRefundAmountAt","outputs":[{"name":"refundAmount","type":"uint256"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":true,"inputs":[],"name":"getProviderNamesSize","outputs":[{"name":"size","type":"uint256"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":true,"inputs":[{"name":"_subscriber","type":"address"},{"name":"_dataSourceName","type":"bytes32"}],"name":"checkAddressSubscription","outputs":[{"name":"subscriber","type":"address"},{"name":"dataSourceName","type":"bytes32"},{"name":"price","type":"uint256"},{"name":"startTime","type":"uint256"},{"name":"endTime","type":"uint256"},{"name":"isUnExpired","type":"bool"},{"name":"isPaid","type":"bool"},{"name":"isPunishedProvider","type":"bool"},{"name":"isOrder","type":"bool"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":false,"inputs":[{"name":"_dataSourceName","type":"bytes32"},{"name":"_isActive","type":"bool"}],"name":"changeDataSourceActivityStatus","outputs":[{"name":"success","type":"bool"}],"payable":false,"stateMutability":"nonpayable","type":"function"},{"constant":true,"inputs":[{"name":"_dataSourceName","type":"bytes32"},{"name":"_index","type":"uint256"}],"name":"isExpiredSubscriptionAt","outputs":[{"name":"isExpired","type":"bool"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":true,"inputs":[{"name":"","type":"uint256"}],"name":"mNames","outputs":[{"name":"","type":"bytes32"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":true,"inputs":[{"name":"","type":"bytes32"}],"name":"mProviders","outputs":[{"name":"owner","type":"address"},{"name":"volume","type":"uint256"},{"name":"subscriptionsNum","type":"uint256"},{"name":"name","type":"bytes32"},{"name":"price","type":"uint256"},{"name":"isPunished","type":"bool"},{"name":"punishTimeStamp","type":"uint256"},{"name":"isProvider","type":"bool"},{"name":"isActive","type":"bool"},{"name":"nextProvider","type":"bytes32"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":true,"inputs":[{"name":"","type":"bytes32"},{"name":"","type":"uint256"}],"name":"mOrders","outputs":[{"name":"dataSourceName","type":"bytes32"},{"name":"subscriber","type":"address"},{"name":"provider","type":"address"},{"name":"price","type":"uint256"},{"name":"startTime","type":"uint256"},{"name":"endTime","type":"uint256"},{"name":"isPaid","type":"bool"},{"name":"isOrder","type":"bool"},{"name":"isRefundPaid","type":"bool"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":true,"inputs":[],"name":"mToken","outputs":[{"name":"","type":"address"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":true,"inputs":[{"name":"_dataSourceName","type":"bytes32"}],"name":"getOwnerFromName","outputs":[{"name":"owner","type":"address"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":true,"inputs":[{"name":"_subscriber","type":"address"},{"name":"_dataSourceName","type":"bytes32"}],"name":"getRefundAmount","outputs":[{"name":"refundAmount","type":"uint256"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":true,"inputs":[{"name":"_dataSourceName","type":"bytes32"}],"name":"getSubscriptionsSize","outputs":[{"name":"size","type":"uint256"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":true,"inputs":[],"name":"mOwner","outputs":[{"name":"","type":"address"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":true,"inputs":[],"name":"MARKETPLACE_VERSION","outputs":[{"name":"","type":"string"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":true,"inputs":[{"name":"_subscriber","type":"address"},{"name":"_dataSourceName","type":"bytes32"}],"name":"isExpiredSubscription","outputs":[{"name":"isExpired","type":"bool"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":false,"inputs":[{"name":"_newOwner","type":"address"}],"name":"transferOwnership","outputs":[],"payable":false,"stateMutability":"nonpayable","type":"function"},{"constant":true,"inputs":[{"name":"_dataSourceName","type":"bytes32"},{"name":"_index","type":"uint256"}],"name":"checkSubscriptionAt","outputs":[{"name":"subscriber","type":"address"},{"name":"dataSourceName","type":"bytes32"},{"name":"price","type":"uint256"},{"name":"startTime","type":"uint256"},{"name":"endTime","type":"uint256"},{"name":"isUnExpired","type":"bool"},{"name":"isPaid","type":"bool"},{"name":"isPunishedProvider","type":"bool"},{"name":"isOrder","type":"bool"}],"payable":false,"stateMutability":"view","type":"function"},{"inputs":[{"name":"_tokenAddress","type":"address"}],"payable":false,"stateMutability":"nonpayable","type":"constructor"},{"anonymous":false,"inputs":[{"indexed":true,"name":"previousOwner","type":"address"},{"indexed":true,"name":"newOwner","type":"address"}],"name":"OwnershipTransferred","type":"event"},{"anonymous":false,"inputs":[{"indexed":true,"name":"editor","type":"address"},{"indexed":true,"name":"dataSourceName","type":"bytes32"},{"indexed":false,"name":"newPrice","type":"uint256"}],"name":"PriceUpdate","type":"event"},{"anonymous":false,"inputs":[{"indexed":true,"name":"editor","type":"address"},{"indexed":true,"name":"dataSourceName","type":"bytes32"},{"indexed":false,"name":"newStatus","type":"bool"}],"name":"ActivityUpdate","type":"event"},{"anonymous":false,"inputs":[{"indexed":true,"name":"dataOwner","type":"address"},{"indexed":true,"name":"dataSourceName","type":"bytes32"},{"indexed":false,"name":"price","type":"uint256"},{"indexed":false,"name":"success","type":"bool"}],"name":"Registered","type":"event"},{"anonymous":false,"inputs":[{"indexed":true,"name":"from","type":"address"},{"indexed":true,"name":"to","type":"address"},{"indexed":false,"name":"value","type":"uint256"}],"name":"SubscriptionDeposited","type":"event"},{"anonymous":false,"inputs":[{"indexed":true,"name":"subscriber","type":"address"},{"indexed":true,"name":"dataSourceName","type":"bytes32"},{"indexed":true,"name":"dataOwner","type":"address"},{"indexed":false,"name":"price","type":"uint256"},{"indexed":false,"name":"success","type":"bool"}],"name":"Subscribed","type":"event"},{"anonymous":false,"inputs":[{"indexed":true,"name":"dataOwner","type":"address"},{"indexed":true,"name":"dataSourceName","type":"bytes32"},{"indexed":false,"name":"amount","type":"uint256"}],"name":"TransferToProvider","type":"event"},{"anonymous":false,"inputs":[{"indexed":true,"name":"dataOwner","type":"address"},{"indexed":true,"name":"dataSourceName","type":"bytes32"},{"indexed":false,"name":"amount","type":"uint256"}],"name":"ProviderWithdraw","type":"event"},{"anonymous":false,"inputs":[{"indexed":true,"name":"dataOwner","type":"address"},{"indexed":true,"name":"dataSourceName","type":"bytes32"},{"indexed":false,"name":"isPunished","type":"bool"}],"name":"ProviderPunishStatus","type":"event"},{"anonymous":false,"inputs":[{"indexed":true,"name":"subscriber","type":"address"},{"indexed":true,"name":"dataSourceName","type":"bytes32"},{"indexed":false,"name":"refundAmount","type":"uint256"}],"name":"SubscriberRefund","type":"event"}] \ No newline at end of file diff --git a/catalyst/marketplace/contract_marketplace_address.txt b/catalyst/marketplace/contract_marketplace_address.txt index 577adaa0..6f0728fa 100644 --- a/catalyst/marketplace/contract_marketplace_address.txt +++ b/catalyst/marketplace/contract_marketplace_address.txt @@ -1 +1 @@ -0x3985f5de8fddf2e8f7705cd360b498bf35ebfbc4 \ No newline at end of file +0xa2b37c6cd52f60fd4eb46ca59fafcf22d081aebc \ No newline at end of file diff --git a/catalyst/marketplace/marketplace.py b/catalyst/marketplace/marketplace.py index fbaaed3a..f72bd661 100644 --- a/catalyst/marketplace/marketplace.py +++ b/catalyst/marketplace/marketplace.py @@ -177,10 +177,12 @@ class Marketplace: def check_transaction(self, tx_hash): if 'ropsten' in ETH_REMOTE_NODE: - etherscan = 'https://ropsten.etherscan.io/tx/{}'.format( - tx_hash) + etherscan = 'https://ropsten.etherscan.io/tx/' + elif 'rinkeby' in ETH_REMOTE_NODE: + etherscan = 'https://rinkeby.etherscan.io/tx/' else: - etherscan = 'https://etherscan.io/tx/{}'.format(tx_hash) + etherscan = 'https://etherscan.io/tx/' + etherscan = '{}{}'.format(etherscan, tx_hash) print('\nYou can check the outcome of your transaction here:\n' '{}\n\n'.format(etherscan)) @@ -329,9 +331,6 @@ class Marketplace: 'nonce': self.web3.eth.getTransactionCount(address)} ) - if 'ropsten' in ETH_REMOTE_NODE: - tx['gas'] = min(int(tx['gas'] * 1.5), 4700000) - signed_tx = self.sign_transaction(tx) try: tx_hash = '0x{}'.format( @@ -371,9 +370,6 @@ class Marketplace: 'from': address, 'nonce': self.web3.eth.getTransactionCount(address)}) - if 'ropsten' in ETH_REMOTE_NODE: - tx['gas'] = min(int(tx['gas'] * 1.5), 4700000) - signed_tx = self.sign_transaction(tx) try: @@ -701,9 +697,6 @@ class Marketplace: 'nonce': self.web3.eth.getTransactionCount(address)} ) - if 'ropsten' in ETH_REMOTE_NODE: - tx['gas'] = min(int(tx['gas'] * 1.5), 4700000) - signed_tx = self.sign_transaction(tx) try: From dbf3b6e6b24cbd5bfab4ea70eb26dfc80d0beb29 Mon Sep 17 00:00:00 2001 From: Victor Grau Serrat Date: Tue, 13 Mar 2018 23:30:34 -0600 Subject: [PATCH 86/87] MAINT: typo in marketplace help --- catalyst/__main__.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/catalyst/__main__.py b/catalyst/__main__.py index f6505054..39b5e277 100644 --- a/catalyst/__main__.py +++ b/catalyst/__main__.py @@ -793,7 +793,7 @@ def ls(ctx): ) @click.pass_context def subscribe(ctx, dataset): - """Subscribe to an exisiting dataset. + """Subscribe to an existing dataset. """ marketplace = Marketplace() marketplace.subscribe(dataset) From 5bb7eed072e61b6cd46546be8c65d8391d28f253 Mon Sep 17 00:00:00 2001 From: Victor Grau Serrat Date: Wed, 14 Mar 2018 00:51:49 -0600 Subject: [PATCH 87/87] MAINT: ref. mktplace to master, updated release notes 0.5.4 --- catalyst/constants.py | 8 ++++---- docs/source/releases.rst | 37 +++++++++++++++++++++++++++++++++++++ 2 files changed, 41 insertions(+), 4 deletions(-) diff --git a/catalyst/constants.py b/catalyst/constants.py index 2bac11c2..3369e412 100644 --- a/catalyst/constants.py +++ b/catalyst/constants.py @@ -28,18 +28,18 @@ AUTH_SERVER = 'https://data.enigma.co' ETH_REMOTE_NODE = 'https://rinkeby.infura.io/' MARKETPLACE_CONTRACT = 'https://raw.githubusercontent.com/enigmampc/' \ - 'catalyst/rinkeby/catalyst/marketplace/' \ + 'catalyst/master/catalyst/marketplace/' \ 'contract_marketplace_address.txt' MARKETPLACE_CONTRACT_ABI = 'https://raw.githubusercontent.com/enigmampc/' \ - 'catalyst/rinkeby/catalyst/marketplace/' \ + 'catalyst/master/catalyst/marketplace/' \ 'contract_marketplace_abi.json' # TODO: switch to mainnet ENIGMA_CONTRACT = 'https://raw.githubusercontent.com/enigmampc/' \ - 'catalyst/rinkeby/catalyst/marketplace/' \ + 'catalyst/master/catalyst/marketplace/' \ 'contract_enigma_address.txt' ENIGMA_CONTRACT_ABI = 'https://raw.githubusercontent.com/enigmampc/' \ - 'catalyst/rinkeby/catalyst/marketplace/' \ + 'catalyst/master/catalyst/marketplace/' \ 'contract_enigma_abi.json' diff --git a/docs/source/releases.rst b/docs/source/releases.rst index e49ef423..dc8a0e8a 100644 --- a/docs/source/releases.rst +++ b/docs/source/releases.rst @@ -2,6 +2,43 @@ Release Notes ============= +Version 0.5.4 +^^^^^^^^^^^^^ +**Release Date**: 2018-03-14 + +Build +~~~~~ +- Switched Data Marketplace from Ropstein testnet to Rinkeby testnet after + incorporating changes resulting from the marketplace contract audit +- Several usability improvements of the Data Marketplace that make the + `--dataset` parameter optional. If it is not included in the command line, + will list available datasets, and let you choose interactively. + +Bug Fixes +~~~~~~~~~ +- Fix Binance requirement of symbol to be included in the cancelled order + :issue:`204` +- Fix `notenoughcasherror` when an open order is filled minutes later + :issue:`237` +- Properly handle of empty candles received from exchanges :issue:`236` +- Added a function to reduce open orders amount from calculated target/amount + for target orders :issue:`243` +- Fix missing file in live trading mode on date change :issue:`252`, + :issue:`253` +- Upgraded Data Marketplace to Web3==4.0.0b11, which was breaking some + functionality from prior version 4.0.0b7 :issue:`257` +- Always request more data to avoid empty bars and always give the exact bar + number :issue:`260` + +Documentation +~~~~~~~~~~~~~ +- PyCharm documentation :issue:`195` +- Added TA-Lib troubleshooting instructions +- Added instructions on how to create a Conda environment for Python 3.6, and + updated Visual C++ instructions for Windows and Python 3 +- Linking example algorithms in the documentation to their sources + + Version 0.5.3 ^^^^^^^^^^^^^ **Release Date**: 2018-02-09