Compare commits

..
24 Commits
Author SHA1 Message Date
Victor Grau Serrat 2fbb6f5134 DOC: release notes 0.5.8 2018-03-29 14:13:42 -06:00
lenak25 af7b211035 BLD: fix bundles last partial candle for minute mode - as reported in issue #266 2018-03-29 18:47:30 +03:00
Avishai WeingartenandGitHub 4eaec97614 DOC: modified bug fixes for 0.5.7 2018-03-29 17:39:59 +03:00
Victor Grau Serrat c29b1ef3c1 BLD: deployment of marketplace on mainnet 2018-03-29 08:26:09 -06:00
Victor Grau Serrat 17f9906df4 DOC: updated release notes for 0.5.7 2018-03-29 08:01:58 -06:00
Victor Grau Serrat af85ee31c9 BUG: [mktplace] progress counters start at 1, not 0 2018-03-27 18:12:08 -06:00
Victor Grau Serrat 132dffd239 BLD: [mktplace] progress indicator for publishing data 2018-03-27 13:22:30 -06:00
VictorandGitHub 887a7cc825 BUG: [mktplace] removing whitespace from datasets to register 2018-03-26 17:22:53 -06:00
AvishaiW 22249506e6 BUG: #214 #287 added arguments to the _reduce_ function in tha Asset class 2018-03-25 22:53:01 +03:00
Victor Grau Serrat b979ffd123 BLD: [mktplace] json handling improvement 2018-03-23 12:38:37 -06:00
Victor Grau Serrat 0db9950347 DOC: link to forum from install doc 2018-03-22 15:24:50 -06:00
Victor 7c4467d800 DOC: added forum button in README 2018-03-22 15:21:17 -06:00
Victor 30dbeaa5fb DOC: Updated README with link to the forum. 2018-03-22 15:21:13 -06:00
Victor Grau Serrat f5cb6e38d6 DOC: fixed formatting of release notes 2018-03-22 12:13:55 -06:00
Victor Grau Serrat c768b207bc MAINT: [marketplace] output formatting address list 2018-03-22 12:01:21 -06:00
VictorandGitHub e18686d5c5 Merge pull request #278 from izokay/develop
BUG: Error when ingesting marketcap on windows
2018-03-22 12:23:29 -05:00
VictorandGitHub b7779cf363 MAINT: general bug fix for existing path across OS 2018-03-22 11:23:06 -06:00
Victor Grau Serrat 28819b8a32 DOC: updated release notes for 0.5.6 2018-03-21 22:29:21 -06:00
Victor Grau Serrat a56d7f34c7 BLD: [mktplace] support for most wallets, switch to mycrypto 2018-03-21 20:35:18 -05:00
AvishaiW 9f0b3303f1 BUG: #285 #271 changed benchmark to be constant, so it wouldn't ingest data at all, for now 2018-03-21 21:19:40 +02:00
EmbarAlmog 9d7a35658b ENH: when ingesting data of non-existing pair it is now throwing log warning. 2018-03-20 16:12:13 +02:00
Victor Grau Serrat c58cebd1eb ENH: progress on marketplace bundle ingestion 2018-03-19 11:50:39 -06:00
izokayandGitHub 9eb649371b BUG: Error when ingesting on windows
Error message: Cannot create a file when that file already exists: '.catalyst\\data\\marketplace\\temp_bundles\\marketcap-hourly-2018' -> '.catalyst\\data\\marketplace\\marketcap'
2018-03-16 16:50:31 -04:00
izokayandGitHub 7f2ded65bc Merge pull request #2 from enigmampc/develop
Develop
2018-03-16 16:45:03 -04:00
16 changed files with 260 additions and 104 deletions
+9 -3
View File
@@ -5,6 +5,7 @@
|version tag| |version tag|
|version status| |version status|
|forum|
|discord| |discord|
|twitter| |twitter|
@@ -22,9 +23,11 @@ visit `enigma.co <https://www.enigma.co>`_ to learn more about Catalyst.
Catalyst builds on top of the well-established Catalyst builds on top of the well-established
`Zipline <https://github.com/quantopian/zipline>`_ project. We did our best to `Zipline <https://github.com/quantopian/zipline>`_ project. We did our best to
minimize structural changes to the general API to maximize compatibility with minimize structural changes to the general API to maximize compatibility with
existing trading algorithms, developer knowledge, and tutorials. Join us on existing trading algorithms, developer knowledge, and tutorials. Join us on the
`Discord <https://discord.gg/SJK32GY>`_ where we have a *#catalyst_dev* channel `Catalyst Forum <https://catalyst.enigma.co/>`_ for questions around Catalyst,
for questions around Catalyst, algorithmic trading and technical support. algorithmic trading and technical support. We also have a
`Discord <https://discord.gg/SJK32GY>`_ group with the *#catalyst_dev* and
*#catalyst_setup* dedicated channels.
Overview Overview
======== ========
@@ -61,6 +64,9 @@ Go to our `Documentation Website <https://enigmampc.github.io/catalyst/>`_.
.. |version status| image:: https://img.shields.io/pypi/pyversions/enigma-catalyst.svg .. |version status| image:: https://img.shields.io/pypi/pyversions/enigma-catalyst.svg
:target: https://pypi.python.org/pypi/enigma-catalyst :target: https://pypi.python.org/pypi/enigma-catalyst
.. |forum| image:: https://img.shields.io/badge/forum-join-green.svg
:target: https://catalyst.enigma.co/
.. |discord| image:: https://img.shields.io/badge/discord-join%20chat-green.svg .. |discord| image:: https://img.shields.io/badge/discord-join%20chat-green.svg
:target: https://discordapp.com/invite/SJK32GY :target: https://discordapp.com/invite/SJK32GY
+1 -1
View File
@@ -580,7 +580,7 @@ def ingest_exchange(ctx, exchange_name, data_frequency, start, end,
exchange_bundle = ExchangeBundle(exchange_name) exchange_bundle = ExchangeBundle(exchange_name)
click.echo('Ingesting exchange bundle {}...'.format(exchange_name), click.echo('Trying to ingest exchange bundle {}...'.format(exchange_name),
sys.stdout) sys.stdout)
exchange_bundle.ingest( exchange_bundle.ingest(
data_frequency=data_frequency, data_frequency=data_frequency,
+8 -3
View File
@@ -630,23 +630,28 @@ cdef class TradingPair(Asset):
and whose second element is a tuple of all the attributes that should and whose second element is a tuple of all the attributes that should
be serialized/deserialized during pickling. be serialized/deserialized during pickling.
""" """
#TODO: make sure that all fields set there # added arguments for catalyst
return (self.__class__, (self.symbol, return (self.__class__, (self.symbol,
self.exchange, self.exchange,
self.start_date, self.start_date,
self.asset_name, self.asset_name,
self.sid, self.sid,
self.leverage, self.leverage,
self.end_daily,
self.end_minute,
self.end_date, self.end_date,
self.exchange_symbol,
self.first_traded, self.first_traded,
self.auto_close_date, self.auto_close_date,
self.exchange_full, self.exchange_full,
self.min_trade_size, self.min_trade_size,
self.max_trade_size, self.max_trade_size,
self.maker,
self.taker,
self.lot, self.lot,
self.decimals, self.decimals,
self.taker, self.trading_state,
self.maker)) self.data_source))
def make_asset_array(int size, Asset asset): def make_asset_array(int size, Asset asset):
cdef np.ndarray out = np.empty([size], dtype=object) cdef np.ndarray out = np.empty([size], dtype=object)
+3
View File
@@ -43,3 +43,6 @@ ENIGMA_CONTRACT = 'https://raw.githubusercontent.com/enigmampc/' \
ENIGMA_CONTRACT_ABI = 'https://raw.githubusercontent.com/enigmampc/' \ ENIGMA_CONTRACT_ABI = 'https://raw.githubusercontent.com/enigmampc/' \
'catalyst/master/catalyst/marketplace/' \ 'catalyst/master/catalyst/marketplace/' \
'contract_enigma_abi.json' 'contract_enigma_abi.json'
SUPPORTED_WALLETS = ['metamask', 'ledger', 'trezor', 'bitbox', 'keystore',
'key']
+2 -6
View File
@@ -199,12 +199,8 @@ class Exchange:
) )
assets.append(asset) assets.append(asset)
except SymbolNotFoundOnExchange: except SymbolNotFoundOnExchange as e:
log.debug( log.warn(e)
'skipping non-existent market {} {}'.format(
self.name, symbol
)
)
return assets return assets
def get_asset(self, symbol, data_frequency=None, is_exchange_symbol=False, def get_asset(self, symbol, data_frequency=None, is_exchange_symbol=False,
+45 -44
View File
@@ -22,7 +22,7 @@ from catalyst.exchange.exchange_errors import EmptyValuesInBundleError, \
PricingDataNotLoadedError, DataCorruptionError, PricingDataValueError PricingDataNotLoadedError, DataCorruptionError, PricingDataValueError
from catalyst.exchange.utils.bundle_utils import range_in_bundle, \ from catalyst.exchange.utils.bundle_utils import range_in_bundle, \
get_bcolz_chunk, get_df_from_arrays, get_assets get_bcolz_chunk, get_df_from_arrays, get_assets
from catalyst.exchange.utils.datetime_utils import get_delta, get_start_dt, \ from catalyst.exchange.utils.datetime_utils import get_start_dt, \
get_period_label, get_month_start_end, get_year_start_end get_period_label, get_month_start_end, get_year_start_end
from catalyst.exchange.utils.exchange_utils import get_exchange_folder, \ from catalyst.exchange.utils.exchange_utils import get_exchange_folder, \
save_exchange_symbols, mixin_market_params, get_catalyst_symbol save_exchange_symbols, mixin_market_params, get_catalyst_symbol
@@ -232,12 +232,12 @@ class ExchangeBundle:
problem = '{name} ({start_dt} to {end_dt}) has empty ' \ problem = '{name} ({start_dt} to {end_dt}) has empty ' \
'periods: {dates}'.format( 'periods: {dates}'.format(
name=asset.symbol, name=asset.symbol,
start_dt=asset.start_date.strftime( start_dt=asset.start_date.strftime(
DATE_TIME_FORMAT), DATE_TIME_FORMAT),
end_dt=end_dt.strftime(DATE_TIME_FORMAT), end_dt=end_dt.strftime(DATE_TIME_FORMAT),
dates=[date.strftime( dates=[date.strftime(
DATE_TIME_FORMAT) for date in dates]) DATE_TIME_FORMAT) for date in dates])
if empty_rows_behavior == 'warn': if empty_rows_behavior == 'warn':
log.warn(problem) log.warn(problem)
@@ -286,12 +286,12 @@ class ExchangeBundle:
problem = '{name} ({start_dt} to {end_dt}) has {threshold} ' \ problem = '{name} ({start_dt} to {end_dt}) has {threshold} ' \
'identical close values on: {dates}'.format( 'identical close values on: {dates}'.format(
name=asset.symbol, name=asset.symbol,
start_dt=asset.start_date.strftime(DATE_TIME_FORMAT), start_dt=asset.start_date.strftime(DATE_TIME_FORMAT),
end_dt=end_dt.strftime(DATE_TIME_FORMAT), end_dt=end_dt.strftime(DATE_TIME_FORMAT),
threshold=threshold, threshold=threshold,
dates=[pd.to_datetime(date).strftime(DATE_TIME_FORMAT) dates=[pd.to_datetime(date).strftime(DATE_TIME_FORMAT)
for date in dates]) for date in dates])
problems.append(problem) problems.append(problem)
@@ -458,7 +458,7 @@ class ExchangeBundle:
last_entry = None last_entry = None
if start is None or \ if start is None or \
(earliest_trade is not None and earliest_trade > start): (earliest_trade is not None and earliest_trade > start):
start = earliest_trade start = earliest_trade
if last_entry is not None and (end is None or end > last_entry): if last_entry is not None and (end is None or end > last_entry):
@@ -598,16 +598,41 @@ class ExchangeBundle:
# we want to give an end_date far in time # we want to give an end_date far in time
writer = self.get_writer(start_dt, end_dt, data_frequency) writer = self.get_writer(start_dt, end_dt, data_frequency)
if show_breakdown: if show_breakdown:
for asset in chunks: if chunks:
for asset in chunks:
with maybe_show_progress(
chunks[asset],
show_progress,
label='Ingesting {frequency} price data for '
'{symbol} on {exchange}'.format(
exchange=self.exchange_name,
frequency=data_frequency,
symbol=asset.symbol
)) as it:
for chunk in it:
problems += self.ingest_ctable(
asset=chunk['asset'],
data_frequency=data_frequency,
period=chunk['period'],
writer=writer,
empty_rows_behavior='strip',
cleanup=True
)
else:
all_chunks = list(chain.from_iterable(itervalues(chunks)))
# We sort the chunks by end date to ingest most recent data first
if all_chunks:
all_chunks.sort(
key=lambda chunk: pd.to_datetime(chunk['period'])
)
with maybe_show_progress( with maybe_show_progress(
chunks[asset], all_chunks,
show_progress, show_progress,
label='Ingesting {frequency} price data for ' label='Ingesting {frequency} price data on '
'{symbol} on {exchange}'.format( '{exchange}'.format(
exchange=self.exchange_name, exchange=self.exchange_name,
frequency=data_frequency, frequency=data_frequency,
symbol=asset.symbol )) as it:
)) as it:
for chunk in it: for chunk in it:
problems += self.ingest_ctable( problems += self.ingest_ctable(
asset=chunk['asset'], asset=chunk['asset'],
@@ -617,30 +642,6 @@ class ExchangeBundle:
empty_rows_behavior='strip', empty_rows_behavior='strip',
cleanup=True cleanup=True
) )
else:
all_chunks = list(chain.from_iterable(itervalues(chunks)))
# We sort the chunks by end date to ingest most recent data first
all_chunks.sort(
key=lambda chunk: pd.to_datetime(chunk['period'])
)
with maybe_show_progress(
all_chunks,
show_progress,
label='Ingesting {frequency} price data on '
'{exchange}'.format(
exchange=self.exchange_name,
frequency=data_frequency,
)) as it:
for chunk in it:
problems += self.ingest_ctable(
asset=chunk['asset'],
data_frequency=data_frequency,
period=chunk['period'],
writer=writer,
empty_rows_behavior='strip',
cleanup=True
)
if show_report and len(problems) > 0: if show_report and len(problems) > 0:
log.info('problems during ingestion:{}\n'.format( log.info('problems during ingestion:{}\n'.format(
+19 -4
View File
@@ -1,4 +1,5 @@
import abc import abc
import datetime
import numpy as np import numpy as np
import pandas as pd import pandas as pd
@@ -300,20 +301,34 @@ class DataPortalExchangeBacktest(DataPortalExchangeBase):
) )
adj_bar_count = candle_size * bar_count adj_bar_count = candle_size * bar_count
if data_frequency == 'minute' and adj_data_frequency == 'daily': if data_frequency == "minute":
end_dt = end_dt.floor('1D') # for minute frequency always request data until the
# current minute (do not include the current minute)
last_dt_for_series = end_dt - datetime.timedelta(minutes=1)
# read the minute bundles for daily frequency to
# support last partial candle
# TODO: optimize this by applying this logic only for the last day
if adj_data_frequency == 'daily':
adj_data_frequency = 'minute'
adj_bar_count = adj_bar_count * 1440
else: # data_frequency == "daily":
last_dt_for_series = end_dt
series = bundle.get_history_window_series_and_load( series = bundle.get_history_window_series_and_load(
assets=assets, assets=assets,
end_dt=end_dt, end_dt=last_dt_for_series,
bar_count=adj_bar_count, bar_count=adj_bar_count,
field=field, field=field,
data_frequency=adj_data_frequency, data_frequency=adj_data_frequency,
algo_end_dt=self._last_available_session, algo_end_dt=self._last_available_session,
) )
start_dt = get_start_dt(end_dt, adj_bar_count, adj_data_frequency) start_dt = get_start_dt(last_dt_for_series, adj_bar_count,
adj_data_frequency, False)
df = resample_history_df(pd.DataFrame(series), freq, field, start_dt) df = resample_history_df(pd.DataFrame(series), freq, field, start_dt)
return df return df
def get_exchange_spot_value(self, def get_exchange_spot_value(self,
+18 -5
View File
@@ -95,11 +95,24 @@ class TradingEnvironment(object):
if not trading_calendar: if not trading_calendar:
trading_calendar = get_calendar("NYSE") trading_calendar = get_calendar("NYSE")
self.benchmark_returns, self.treasury_curves = load( # todo: uncomment and add a well defined benchmark
trading_calendar.day, # self.benchmark_returns, self.treasury_curves = load(
trading_calendar.schedule.index, # trading_calendar.day,
self.bm_symbol, # trading_calendar.schedule.index,
) # self.bm_symbol,
# exchange=exchange,
# )
start_data = get_calendar('OPEN').first_trading_session
end_data = pd.Timestamp.utcnow()
treasure_cols = ['1month', '3month', '6month', '1year', '2year',
'3year', '5year', '7year', '10year', '20year', '30year']
self.benchmark_returns = pd.DataFrame(data=0.001,
index=pd.date_range(start_data, end_data),
columns=['close'])
self.treasury_curves = pd.DataFrame(data=0.001,
index=pd.date_range(start_data, end_data),
columns=treasure_cols)
self.exchange_tz = exchange_tz self.exchange_tz = exchange_tz
@@ -1 +1 @@
0x39a54f480d922a58c963de8091a6c9afc69db2cf 0xf0ee6b27b759c9893ce4f094b49ad28fd15a23e4
@@ -1 +1 @@
0xa2b37c6cd52f60fd4eb46ca59fafcf22d081aebc 0xa64927358a82254be92eb1f1cb01de68d1787004
+46 -24
View File
@@ -69,7 +69,10 @@ class Marketplace:
contract_url.info().get_content_charset()).strip()) contract_url.info().get_content_charset()).strip())
abi_url = urllib.urlopen(MARKETPLACE_CONTRACT_ABI) abi_url = urllib.urlopen(MARKETPLACE_CONTRACT_ABI)
abi = json.load(abi_url) abi_url = abi_url.read().decode(
abi_url.info().get_content_charset())
abi = json.loads(abi_url)
self.mkt_contract = self.web3.eth.contract( self.mkt_contract = self.web3.eth.contract(
self.mkt_contract_address, self.mkt_contract_address,
@@ -83,7 +86,10 @@ class Marketplace:
contract_url.info().get_content_charset()).strip()) contract_url.info().get_content_charset()).strip())
abi_url = urllib.urlopen(ENIGMA_CONTRACT_ABI) abi_url = urllib.urlopen(ENIGMA_CONTRACT_ABI)
abi = json.load(abi_url) abi_url = abi_url.read().decode(
abi_url.info().get_content_charset())
abi = json.loads(abi_url)
self.eng_contract = self.web3.eth.contract( self.eng_contract = self.web3.eth.contract(
self.eng_contract_address, self.eng_contract_address,
@@ -126,9 +132,10 @@ class Marketplace:
else: else:
while True: while True:
for i in range(0, len(self.addresses)): for i in range(0, len(self.addresses)):
print('{}\t{}\t{}'.format( print('{}\t{}\t{}\t{}'.format(
i, i,
self.addresses[i]['pubAddr'], self.addresses[i]['pubAddr'],
self.addresses[i]['wallet'].ljust(10),
self.addresses[i]['desc']) self.addresses[i]['desc'])
) )
address_i = int(input('Choose your address associated with ' address_i = int(input('Choose your address associated with '
@@ -145,7 +152,7 @@ class Marketplace:
def sign_transaction(self, tx): def sign_transaction(self, tx):
url = 'https://www.myetherwallet.com/#offline-transaction' url = 'https://www.mycrypto.com/#offline-transaction'
print('\nVisit {url} and enter the following parameters:\n\n' print('\nVisit {url} and enter the following parameters:\n\n'
'From Address:\t\t{_from}\n' 'From Address:\t\t{_from}\n'
'\n\tClick the "Generate Information" button\n\n' '\n\tClick the "Generate Information" button\n\n'
@@ -430,10 +437,9 @@ class Marketplace:
merge_bundles(zsource, ztarget) merge_bundles(zsource, ztarget)
else: else:
shutil.rmtree(bundle_folder, ignore_errors=True)
os.rename(tmp_bundle, bundle_folder) os.rename(tmp_bundle, bundle_folder)
pass
def ingest(self, ds_name=None, start=None, end=None, force_download=False): def ingest(self, ds_name=None, start=None, end=None, force_download=False):
if ds_name is None: if ds_name is None:
@@ -498,20 +504,29 @@ class Marketplace:
key = self.addresses[address_i]['key'] key = self.addresses[address_i]['key']
secret = self.addresses[address_i]['secret'] secret = self.addresses[address_i]['secret']
else: else:
key, secret = get_key_secret(address) key, secret = get_key_secret(address,
self.addresses[address_i]['wallet'])
headers = get_signed_headers(ds_name, key, secret) headers = get_signed_headers(ds_name, key, secret)
log.debug('Starting download of dataset for ingestion...') log.info('Starting download of dataset for ingestion...')
r = requests.post( r = requests.post(
'{}/marketplace/ingest'.format(AUTH_SERVER), '{}/marketplace/ingest'.format(AUTH_SERVER),
headers=headers, headers=headers,
stream=True, stream=True,
) )
if r.status_code == 200: if r.status_code == 200:
log.info('Dataset downloaded successfully. Processing dataset...')
target_path = get_temp_bundles_folder() target_path = get_temp_bundles_folder()
try: try:
decoder = MultipartDecoder.from_response(r) decoder = MultipartDecoder.from_response(r)
# with maybe_show_progress(
# iter(decoder.parts),
# True,
# label='Processing files') as part:
counter = 1
for part in decoder.parts: for part in decoder.parts:
log.info("Processing file {} of {}".format(
counter, len(decoder.parts)))
h = part.headers[b'Content-Disposition'].decode('utf-8') h = part.headers[b'Content-Disposition'].decode('utf-8')
# Extracting the filename from the header # Extracting the filename from the header
name = re.search(r'filename="(.*)"', h).group(1) name = re.search(r'filename="(.*)"', h).group(1)
@@ -525,6 +540,7 @@ class Marketplace:
f.write(part.content) f.write(part.content)
self.process_temp_bundle(ds_name, filename) self.process_temp_bundle(ds_name, filename)
counter += 1
except NonMultipartContentTypeException: except NonMultipartContentTypeException:
response = r.json() response = r.json()
@@ -592,7 +608,6 @@ class Marketplace:
folder = get_bundle_folder(ds_name, data_frequency) folder = get_bundle_folder(ds_name, data_frequency)
shutil.rmtree(folder) shutil.rmtree(folder)
pass
def create_metadata(self, key, secret, ds_name, data_frequency, desc, def create_metadata(self, key, secret, ds_name, data_frequency, desc,
has_history=True, has_live=True): has_history=True, has_live=True):
@@ -628,7 +643,7 @@ class Marketplace:
def register(self): def register(self):
while True: while True:
desc = input('Enter the name of the dataset to register: ') desc = input('Enter the name of the dataset to register: ')
dataset = desc.lower() dataset = desc.lower().strip()
provider_info = self.mkt_contract.functions.getDataProviderInfo( provider_info = self.mkt_contract.functions.getDataProviderInfo(
Web3.toHex(dataset) Web3.toHex(dataset)
).call() ).call()
@@ -684,7 +699,8 @@ class Marketplace:
key = self.addresses[address_i]['key'] key = self.addresses[address_i]['key']
secret = self.addresses[address_i]['secret'] secret = self.addresses[address_i]['secret']
else: else:
key, secret = get_key_secret(address) key, secret = get_key_secret(address,
self.addresses[address_i]['wallet'])
grains = to_grains(price) grains = to_grains(price)
@@ -765,28 +781,34 @@ class Marketplace:
key = match['key'] key = match['key']
secret = match['secret'] secret = match['secret']
else: else:
key, secret = get_key_secret(provider_info[0]) key, secret = get_key_secret(provider_info[0], match['wallet'])
headers = get_signed_headers(dataset, key, secret)
filenames = glob.glob(os.path.join(datadir, '*.csv')) filenames = glob.glob(os.path.join(datadir, '*.csv'))
if not filenames: if not filenames:
raise MarketplaceNoCSVFiles(datadir=datadir) raise MarketplaceNoCSVFiles(datadir=datadir)
files = [] files = []
for file in filenames: for idx, file in enumerate(filenames):
log.info('Uploading file {} of {}: {}'.format(
idx+1, len(filenames), file))
files = []
files.append(('file', open(file, 'rb'))) files.append(('file', open(file, 'rb')))
r = requests.post('{}/marketplace/publish'.format(AUTH_SERVER), headers = get_signed_headers(dataset, key, secret)
files=files, r = requests.post('{}/marketplace/publish'.format(AUTH_SERVER),
headers=headers) files=files,
headers=headers)
if r.status_code != 200: if r.status_code != 200:
raise MarketplaceHTTPRequest(request='upload file', raise MarketplaceHTTPRequest(request='upload file',
error=r.status_code) error=r.status_code)
if 'error' in r.json(): if 'error' in r.json():
raise MarketplaceHTTPRequest(request='upload file', raise MarketplaceHTTPRequest(request='upload file',
error=r.json()['error']) error=r.json()['error'])
print('Dataset {} uploaded successfully.'.format(dataset)) log.info('File processed successfully.')
print('\nDataset {} uploaded and processed successfully.'.format(
dataset))
+11 -9
View File
@@ -10,10 +10,10 @@ from catalyst.marketplace.marketplace_errors import (
MarketplaceEmptySignature) MarketplaceEmptySignature)
from catalyst.marketplace.utils.path_utils import ( from catalyst.marketplace.utils.path_utils import (
get_user_pubaddr, save_user_pubaddr) get_user_pubaddr, save_user_pubaddr)
from catalyst.constants import AUTH_SERVER from catalyst.constants import AUTH_SERVER, SUPPORTED_WALLETS
def get_key_secret(pubAddr, wallet='mew'): def get_key_secret(pubAddr, wallet):
""" """
Obtain a new key/secret pair from authentication server Obtain a new key/secret pair from authentication server
@@ -43,21 +43,22 @@ def get_key_secret(pubAddr, wallet='mew'):
auth_type, auth_info = header.split(None, 1) auth_type, auth_info = header.split(None, 1)
d = requests.utils.parse_dict_header(auth_info) d = requests.utils.parse_dict_header(auth_info)
nonce = '0x{}'.format(d['nonce']) nonce = 'Catalyst nonce: 0x{}'.format(d['nonce'])
if wallet == 'mew': if wallet in SUPPORTED_WALLETS:
url = 'https://www.myetherwallet.com/signmsg.html' url = 'https://www.mycrypto.com/signmsg.html'
print('\nObtaining a key/secret pair to streamline all future ' print('\nObtaining a key/secret pair to streamline all future '
'requests with the authentication server.\n' 'requests with the authentication server.\n'
'Visit {url} and sign the ' 'Visit {url} and sign the '
'following message:\n{nonce}'.format( 'following message (copy the entire line, without the '
'line break at the end):\n\n{nonce}'.format(
url=url, url=url,
nonce=nonce)) nonce=nonce))
webbrowser.open_new(url) webbrowser.open_new(url)
signature = input('Copy and Paste the "sig" field from ' signature = input('\nCopy and Paste the "sig" field from '
'the signature here (without the double quotes, ' 'the signature here (without the double quotes, '
'only the HEX value):\n') 'only the HEX value):\n')
else: else:
@@ -91,7 +92,8 @@ def get_key_secret(pubAddr, wallet='mew'):
addresses = get_user_pubaddr() addresses = get_user_pubaddr()
match = next((l for l in addresses if match = next((l for l in addresses if
l['pubAddr'] == pubAddr), None) l['pubAddr'].lower() == pubAddr.lower()), None)
match['key'] = response.json()['key'] match['key'] = response.json()['key']
match['secret'] = response.json()['secret'] match['secret'] = response.json()['secret']
@@ -121,7 +123,7 @@ def get_signed_headers(ds_name, key, secret):
------- -------
""" """
nonce = str(int(time.time())) nonce = str(int(time.time() * 1000))
signature = hmac.new( signature = hmac.new(
secret.encode('utf-8'), secret.encode('utf-8'),
+49 -2
View File
@@ -2,6 +2,7 @@ import os
import json import json
import tarfile import tarfile
from catalyst.constants import SUPPORTED_WALLETS
from catalyst.utils.deprecate import deprecated from catalyst.utils.deprecate import deprecated
from catalyst.utils.paths import data_root, ensure_directory from catalyst.utils.paths import data_root, ensure_directory
from catalyst.marketplace.marketplace_errors import MarketplaceJSONError from catalyst.marketplace.marketplace_errors import MarketplaceJSONError
@@ -131,17 +132,63 @@ def get_user_pubaddr(environ=None):
try: try:
d = data[0]['pubAddr'] d = data[0]['pubAddr']
except Exception as e: except Exception as e:
return [data, ] data = [data, ]
changed = False
for idx, d in enumerate(data):
try:
if d['wallet'] not in SUPPORTED_WALLETS:
data[idx]['wallet'] = _choose_wallet(
d['pubAddr'], False)
changed = True
except KeyError:
data[idx]['wallet'] = _choose_wallet(
d['pubAddr'], True)
changed = True
if changed:
save_user_pubaddr(data)
return data return data
else: else:
data = [] data = []
data.append(dict(pubAddr='', desc='')) data.append(dict(pubAddr='', desc='', wallet=''))
with open(filename, 'w') as f: with open(filename, 'w') as f:
json.dump(data, f, sort_keys=False, indent=2, json.dump(data, f, sort_keys=False, indent=2,
separators=(',', ':')) separators=(',', ':'))
return data return data
def _choose_wallet(pubAddr, missing):
while True:
if missing:
print('\nYou need to specify a wallet for address '
'{}.'.format(pubAddr))
else:
print('\nThe wallet specified for address {} is not '
'supported.'.format(pubAddr))
print('Please choose among the following options:')
for idx, wallet in enumerate(SUPPORTED_WALLETS):
print('{}\t{}'.format(idx, wallet))
lw = len(SUPPORTED_WALLETS)-1
w = input('Choose a number between 0 and {}: '.format(
lw))
try:
w = int(w)
except ValueError:
print('Enter a number between 0 and {}'.format(lw))
else:
if w not in range(0, lw+1):
print('Enter a number between 0 and '
'{}'.format(lw))
else:
return SUPPORTED_WALLETS[w]
def save_user_pubaddr(data, environ=None): def save_user_pubaddr(data, environ=None):
""" """
Saves the user's public addresses and their related metadata in Saves the user's public addresses and their related metadata in
+4
View File
@@ -562,6 +562,10 @@ If after following the instructions above, and going through the
*Troubleshooting* sections, you still experience problems installing Catalyst, *Troubleshooting* sections, you still experience problems installing Catalyst,
you can seek additional help through the following channels: you can seek additional help through the following channels:
- Join our `Catalyst Forum <https://catalyst.enigma.co/>`_, and browse a variety
of topics and conversations around common issues that others face when using
Catalyst, and how to resolve them. And join the conversation!
- Join our `Discord community <https://discord.gg/SJK32GY>`_, and head over - Join our `Discord community <https://discord.gg/SJK32GY>`_, and head over
the #catalyst_dev channel where many other users (as well as the project the #catalyst_dev channel where many other users (as well as the project
developers) hang out, and can assist you with your particular issue. The developers) hang out, and can assist you with your particular issue. The
+41
View File
@@ -2,6 +2,47 @@
Release Notes Release Notes
============= =============
Version 0.5.8
^^^^^^^^^^^^^
**Release Date**: 2018-03-29
Bug Fixes
~~~~~~~~~
- Fix Data Marketplace release on mainnet
Version 0.5.7
^^^^^^^^^^^^^
**Release Date**: 2018-03-29
Build
~~~~~
- Data Marketplace deployed on mainnet.
- Added progress indicators for publishing data, and made the data publishing
synchronous to provide feedback to the publisher.
Bug Fixes
~~~~~~~~~
- fixes in storing and loading the state :issue:`214`,
:issue:`287`
Version 0.5.6
^^^^^^^^^^^^^
**Release Date**: 2018-03-22
Build
~~~~~
- Data Marketplace: ensures compatibility across wallets, now fully supporting
``ledger``, ``trezor``, ``keystore``, ``private key``. Partial support for
``metamask`` (includes sign_msg, but not sign_tx). Current support for
``Digital Bitbox`` is unknown, but believed to be supported.
- Data Marketplace: Switched online provider from MyEtherWallet to MyCrypto.
- Data Marketplace: Added progress indicator for data ingestion.
Bug Fixes
~~~~~~~~~
- Changed benchmark to be constant, so it doesn't ingest data at all. Temporary
fix for :issue:`271`, :issue:`285`
Version 0.5.5 Version 0.5.5
^^^^^^^^^^^^^ ^^^^^^^^^^^^^
**Release Date**: 2018-03-19 **Release Date**: 2018-03-19
+2 -1
View File
@@ -5,6 +5,7 @@ channels:
dependencies: dependencies:
- certifi=2016.2.28=py27_0 - certifi=2016.2.28=py27_0
- mkl=2017.0.3 - mkl=2017.0.3
- matplotlib=2.1.2=py36_0
- numpy=1.13.1=py27_0 - numpy=1.13.1=py27_0
- openssl=1.0.2l - openssl=1.0.2l
- pip=9.0.1=py27_1 - pip=9.0.1=py27_1
@@ -39,7 +40,7 @@ dependencies:
- lru-dict==1.1.6 - lru-dict==1.1.6
- mako==1.0.7 - mako==1.0.7
- markupsafe==1.0 - markupsafe==1.0
- matplotlib==2.1.2 - matplotlib==2.1.0
- multipledispatch==0.4.9 - multipledispatch==0.4.9
- networkx==2.0 - networkx==2.0
- numexpr==2.6.4 - numexpr==2.6.4