mirror of
https://github.com/wassname/catalyst.git
synced 2026-07-22 12:40:30 +08:00
Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
423e30da1e | ||
|
|
c29b1ef3c1 | ||
|
|
17f9906df4 | ||
|
|
41b5135ed4 | ||
|
|
af85ee31c9 | ||
|
|
132dffd239 | ||
|
|
887a7cc825 | ||
|
|
22249506e6 | ||
|
|
b979ffd123 | ||
|
|
0db9950347 | ||
|
|
7c4467d800 | ||
|
|
30dbeaa5fb | ||
|
|
4d6837b5d6 | ||
|
|
098a4c4fc6 | ||
|
|
f5cb6e38d6 | ||
|
|
1bd65397b6 | ||
|
|
c768b207bc | ||
|
|
e18686d5c5 | ||
|
|
b7779cf363 | ||
|
|
28819b8a32 | ||
|
|
a56d7f34c7 | ||
|
|
9f0b3303f1 | ||
|
|
9d7a35658b | ||
|
|
c58cebd1eb | ||
|
|
9eb649371b | ||
|
|
7f2ded65bc | ||
|
|
2d5d2b21ee |
+2
-2
@@ -1,11 +1,11 @@
|
|||||||
#
|
#
|
||||||
# Dockerfile for an image with the currently checked out version of catalyst installed. To build:
|
# Dockerfile for an image with the currently checked out version of catalyst installed. To build:
|
||||||
#
|
#
|
||||||
# docker build -t quantopian/catalyst .
|
# docker build -t enigmampc/catalyst .
|
||||||
#
|
#
|
||||||
# To run the container:
|
# To run the container:
|
||||||
#
|
#
|
||||||
# docker run -v /path/to/your/notebooks:/projects -v ~/.catalyst:/root/.catalyst -p 8888:8888/tcp --name catalyst -it quantopian/catalyst
|
# docker run -v /path/to/your/notebooks:/projects -v ~/.catalyst:/root/.catalyst -p 8888:8888/tcp --name catalyst -it enigmampc/catalyst
|
||||||
#
|
#
|
||||||
# To access Jupyter when running docker locally (you may need to add NAT rules):
|
# To access Jupyter when running docker locally (you may need to add NAT rules):
|
||||||
#
|
#
|
||||||
|
|||||||
+5
-5
@@ -1,15 +1,15 @@
|
|||||||
#
|
#
|
||||||
# Dockerfile for an image with the currently checked out version of catalyst installed. To build:
|
# Dockerfile for an image with the currently checked out version of catalyst installed. To build:
|
||||||
#
|
#
|
||||||
# docker build -t quantopian/catalystdev -f Dockerfile-dev .
|
# docker build -t enigmampc/catalystdev -f Dockerfile-dev .
|
||||||
#
|
#
|
||||||
# Note: the dev build requires a quantopian/catalyst image, which you can build as follows:
|
# Note: the dev build requires a enigmampc/catalyst image, which you can build as follows:
|
||||||
#
|
#
|
||||||
# docker build -t quantopian/catalyst -f Dockerfile .
|
# docker build -t enigmampc/catalyst -f Dockerfile .
|
||||||
#
|
#
|
||||||
# To run the container:
|
# To run the container:
|
||||||
#
|
#
|
||||||
# docker run -v /path/to/your/notebooks:/projects -v ~/.catalyst:/root/.catalyst -p 8888:8888/tcp --name catalystdev -it quantopian/catalystdev
|
# docker run -v /path/to/your/notebooks:/projects -v ~/.catalyst:/root/.catalyst -p 8888:8888/tcp --name catalystdev -it enigmampc/catalystdev
|
||||||
#
|
#
|
||||||
# To access Jupyter when running docker locally (you may need to add NAT rules):
|
# To access Jupyter when running docker locally (you may need to add NAT rules):
|
||||||
#
|
#
|
||||||
@@ -25,7 +25,7 @@
|
|||||||
#
|
#
|
||||||
# docker exec -it catalystdev catalyst run -f /projects/my_algo.py --start 2015-1-1 --end 2016-1-1 /projects/result.pickle
|
# docker exec -it catalystdev catalyst run -f /projects/my_algo.py --start 2015-1-1 --end 2016-1-1 /projects/result.pickle
|
||||||
#
|
#
|
||||||
FROM quantopian/catalyst
|
FROM enigmampc/catalyst
|
||||||
|
|
||||||
WORKDIR /catalyst
|
WORKDIR /catalyst
|
||||||
|
|
||||||
|
|||||||
+9
-3
@@ -5,6 +5,7 @@
|
|||||||
|
|
||||||
|version tag|
|
|version tag|
|
||||||
|version status|
|
|version status|
|
||||||
|
|forum|
|
||||||
|discord|
|
|discord|
|
||||||
|twitter|
|
|twitter|
|
||||||
|
|
||||||
@@ -22,9 +23,11 @@ visit `enigma.co <https://www.enigma.co>`_ to learn more about Catalyst.
|
|||||||
Catalyst builds on top of the well-established
|
Catalyst builds on top of the well-established
|
||||||
`Zipline <https://github.com/quantopian/zipline>`_ project. We did our best to
|
`Zipline <https://github.com/quantopian/zipline>`_ project. We did our best to
|
||||||
minimize structural changes to the general API to maximize compatibility with
|
minimize structural changes to the general API to maximize compatibility with
|
||||||
existing trading algorithms, developer knowledge, and tutorials. Join us on
|
existing trading algorithms, developer knowledge, and tutorials. Join us on the
|
||||||
`Discord <https://discord.gg/SJK32GY>`_ where we have a *#catalyst_dev* channel
|
`Catalyst Forum <https://catalyst.enigma.co/>`_ for questions around Catalyst,
|
||||||
for questions around Catalyst, algorithmic trading and technical support.
|
algorithmic trading and technical support. We also have a
|
||||||
|
`Discord <https://discord.gg/SJK32GY>`_ group with the *#catalyst_dev* and
|
||||||
|
*#catalyst_setup* dedicated channels.
|
||||||
|
|
||||||
Overview
|
Overview
|
||||||
========
|
========
|
||||||
@@ -61,6 +64,9 @@ Go to our `Documentation Website <https://enigmampc.github.io/catalyst/>`_.
|
|||||||
.. |version status| image:: https://img.shields.io/pypi/pyversions/enigma-catalyst.svg
|
.. |version status| image:: https://img.shields.io/pypi/pyversions/enigma-catalyst.svg
|
||||||
:target: https://pypi.python.org/pypi/enigma-catalyst
|
:target: https://pypi.python.org/pypi/enigma-catalyst
|
||||||
|
|
||||||
|
.. |forum| image:: https://img.shields.io/badge/forum-join-green.svg
|
||||||
|
:target: https://catalyst.enigma.co/
|
||||||
|
|
||||||
.. |discord| image:: https://img.shields.io/badge/discord-join%20chat-green.svg
|
.. |discord| image:: https://img.shields.io/badge/discord-join%20chat-green.svg
|
||||||
:target: https://discordapp.com/invite/SJK32GY
|
:target: https://discordapp.com/invite/SJK32GY
|
||||||
|
|
||||||
|
|||||||
@@ -580,7 +580,7 @@ def ingest_exchange(ctx, exchange_name, data_frequency, start, end,
|
|||||||
|
|
||||||
exchange_bundle = ExchangeBundle(exchange_name)
|
exchange_bundle = ExchangeBundle(exchange_name)
|
||||||
|
|
||||||
click.echo('Ingesting exchange bundle {}...'.format(exchange_name),
|
click.echo('Trying to ingest exchange bundle {}...'.format(exchange_name),
|
||||||
sys.stdout)
|
sys.stdout)
|
||||||
exchange_bundle.ingest(
|
exchange_bundle.ingest(
|
||||||
data_frequency=data_frequency,
|
data_frequency=data_frequency,
|
||||||
|
|||||||
@@ -630,23 +630,28 @@ cdef class TradingPair(Asset):
|
|||||||
and whose second element is a tuple of all the attributes that should
|
and whose second element is a tuple of all the attributes that should
|
||||||
be serialized/deserialized during pickling.
|
be serialized/deserialized during pickling.
|
||||||
"""
|
"""
|
||||||
#TODO: make sure that all fields set there
|
# added arguments for catalyst
|
||||||
return (self.__class__, (self.symbol,
|
return (self.__class__, (self.symbol,
|
||||||
self.exchange,
|
self.exchange,
|
||||||
self.start_date,
|
self.start_date,
|
||||||
self.asset_name,
|
self.asset_name,
|
||||||
self.sid,
|
self.sid,
|
||||||
self.leverage,
|
self.leverage,
|
||||||
|
self.end_daily,
|
||||||
|
self.end_minute,
|
||||||
self.end_date,
|
self.end_date,
|
||||||
|
self.exchange_symbol,
|
||||||
self.first_traded,
|
self.first_traded,
|
||||||
self.auto_close_date,
|
self.auto_close_date,
|
||||||
self.exchange_full,
|
self.exchange_full,
|
||||||
self.min_trade_size,
|
self.min_trade_size,
|
||||||
self.max_trade_size,
|
self.max_trade_size,
|
||||||
|
self.maker,
|
||||||
|
self.taker,
|
||||||
self.lot,
|
self.lot,
|
||||||
self.decimals,
|
self.decimals,
|
||||||
self.taker,
|
self.trading_state,
|
||||||
self.maker))
|
self.data_source))
|
||||||
|
|
||||||
def make_asset_array(int size, Asset asset):
|
def make_asset_array(int size, Asset asset):
|
||||||
cdef np.ndarray out = np.empty([size], dtype=object)
|
cdef np.ndarray out = np.empty([size], dtype=object)
|
||||||
|
|||||||
@@ -43,3 +43,6 @@ ENIGMA_CONTRACT = 'https://raw.githubusercontent.com/enigmampc/' \
|
|||||||
ENIGMA_CONTRACT_ABI = 'https://raw.githubusercontent.com/enigmampc/' \
|
ENIGMA_CONTRACT_ABI = 'https://raw.githubusercontent.com/enigmampc/' \
|
||||||
'catalyst/master/catalyst/marketplace/' \
|
'catalyst/master/catalyst/marketplace/' \
|
||||||
'contract_enigma_abi.json'
|
'contract_enigma_abi.json'
|
||||||
|
|
||||||
|
SUPPORTED_WALLETS = ['metamask', 'ledger', 'trezor', 'bitbox', 'keystore',
|
||||||
|
'key']
|
||||||
|
|||||||
@@ -199,12 +199,8 @@ class Exchange:
|
|||||||
)
|
)
|
||||||
assets.append(asset)
|
assets.append(asset)
|
||||||
|
|
||||||
except SymbolNotFoundOnExchange:
|
except SymbolNotFoundOnExchange as e:
|
||||||
log.debug(
|
log.warn(e)
|
||||||
'skipping non-existent market {} {}'.format(
|
|
||||||
self.name, symbol
|
|
||||||
)
|
|
||||||
)
|
|
||||||
return assets
|
return assets
|
||||||
|
|
||||||
def get_asset(self, symbol, data_frequency=None, is_exchange_symbol=False,
|
def get_asset(self, symbol, data_frequency=None, is_exchange_symbol=False,
|
||||||
|
|||||||
@@ -22,7 +22,7 @@ from catalyst.exchange.exchange_errors import EmptyValuesInBundleError, \
|
|||||||
PricingDataNotLoadedError, DataCorruptionError, PricingDataValueError
|
PricingDataNotLoadedError, DataCorruptionError, PricingDataValueError
|
||||||
from catalyst.exchange.utils.bundle_utils import range_in_bundle, \
|
from catalyst.exchange.utils.bundle_utils import range_in_bundle, \
|
||||||
get_bcolz_chunk, get_df_from_arrays, get_assets
|
get_bcolz_chunk, get_df_from_arrays, get_assets
|
||||||
from catalyst.exchange.utils.datetime_utils import get_delta, get_start_dt, \
|
from catalyst.exchange.utils.datetime_utils import get_start_dt, \
|
||||||
get_period_label, get_month_start_end, get_year_start_end
|
get_period_label, get_month_start_end, get_year_start_end
|
||||||
from catalyst.exchange.utils.exchange_utils import get_exchange_folder, \
|
from catalyst.exchange.utils.exchange_utils import get_exchange_folder, \
|
||||||
save_exchange_symbols, mixin_market_params, get_catalyst_symbol
|
save_exchange_symbols, mixin_market_params, get_catalyst_symbol
|
||||||
@@ -232,12 +232,12 @@ class ExchangeBundle:
|
|||||||
|
|
||||||
problem = '{name} ({start_dt} to {end_dt}) has empty ' \
|
problem = '{name} ({start_dt} to {end_dt}) has empty ' \
|
||||||
'periods: {dates}'.format(
|
'periods: {dates}'.format(
|
||||||
name=asset.symbol,
|
name=asset.symbol,
|
||||||
start_dt=asset.start_date.strftime(
|
start_dt=asset.start_date.strftime(
|
||||||
DATE_TIME_FORMAT),
|
DATE_TIME_FORMAT),
|
||||||
end_dt=end_dt.strftime(DATE_TIME_FORMAT),
|
end_dt=end_dt.strftime(DATE_TIME_FORMAT),
|
||||||
dates=[date.strftime(
|
dates=[date.strftime(
|
||||||
DATE_TIME_FORMAT) for date in dates])
|
DATE_TIME_FORMAT) for date in dates])
|
||||||
|
|
||||||
if empty_rows_behavior == 'warn':
|
if empty_rows_behavior == 'warn':
|
||||||
log.warn(problem)
|
log.warn(problem)
|
||||||
@@ -286,12 +286,12 @@ class ExchangeBundle:
|
|||||||
|
|
||||||
problem = '{name} ({start_dt} to {end_dt}) has {threshold} ' \
|
problem = '{name} ({start_dt} to {end_dt}) has {threshold} ' \
|
||||||
'identical close values on: {dates}'.format(
|
'identical close values on: {dates}'.format(
|
||||||
name=asset.symbol,
|
name=asset.symbol,
|
||||||
start_dt=asset.start_date.strftime(DATE_TIME_FORMAT),
|
start_dt=asset.start_date.strftime(DATE_TIME_FORMAT),
|
||||||
end_dt=end_dt.strftime(DATE_TIME_FORMAT),
|
end_dt=end_dt.strftime(DATE_TIME_FORMAT),
|
||||||
threshold=threshold,
|
threshold=threshold,
|
||||||
dates=[pd.to_datetime(date).strftime(DATE_TIME_FORMAT)
|
dates=[pd.to_datetime(date).strftime(DATE_TIME_FORMAT)
|
||||||
for date in dates])
|
for date in dates])
|
||||||
|
|
||||||
problems.append(problem)
|
problems.append(problem)
|
||||||
|
|
||||||
@@ -458,7 +458,7 @@ class ExchangeBundle:
|
|||||||
last_entry = None
|
last_entry = None
|
||||||
|
|
||||||
if start is None or \
|
if start is None or \
|
||||||
(earliest_trade is not None and earliest_trade > start):
|
(earliest_trade is not None and earliest_trade > start):
|
||||||
start = earliest_trade
|
start = earliest_trade
|
||||||
|
|
||||||
if last_entry is not None and (end is None or end > last_entry):
|
if last_entry is not None and (end is None or end > last_entry):
|
||||||
@@ -598,16 +598,41 @@ class ExchangeBundle:
|
|||||||
# we want to give an end_date far in time
|
# we want to give an end_date far in time
|
||||||
writer = self.get_writer(start_dt, end_dt, data_frequency)
|
writer = self.get_writer(start_dt, end_dt, data_frequency)
|
||||||
if show_breakdown:
|
if show_breakdown:
|
||||||
for asset in chunks:
|
if chunks:
|
||||||
|
for asset in chunks:
|
||||||
|
with maybe_show_progress(
|
||||||
|
chunks[asset],
|
||||||
|
show_progress,
|
||||||
|
label='Ingesting {frequency} price data for '
|
||||||
|
'{symbol} on {exchange}'.format(
|
||||||
|
exchange=self.exchange_name,
|
||||||
|
frequency=data_frequency,
|
||||||
|
symbol=asset.symbol
|
||||||
|
)) as it:
|
||||||
|
for chunk in it:
|
||||||
|
problems += self.ingest_ctable(
|
||||||
|
asset=chunk['asset'],
|
||||||
|
data_frequency=data_frequency,
|
||||||
|
period=chunk['period'],
|
||||||
|
writer=writer,
|
||||||
|
empty_rows_behavior='strip',
|
||||||
|
cleanup=True
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
all_chunks = list(chain.from_iterable(itervalues(chunks)))
|
||||||
|
# We sort the chunks by end date to ingest most recent data first
|
||||||
|
if all_chunks:
|
||||||
|
all_chunks.sort(
|
||||||
|
key=lambda chunk: pd.to_datetime(chunk['period'])
|
||||||
|
)
|
||||||
with maybe_show_progress(
|
with maybe_show_progress(
|
||||||
chunks[asset],
|
all_chunks,
|
||||||
show_progress,
|
show_progress,
|
||||||
label='Ingesting {frequency} price data for '
|
label='Ingesting {frequency} price data on '
|
||||||
'{symbol} on {exchange}'.format(
|
'{exchange}'.format(
|
||||||
exchange=self.exchange_name,
|
exchange=self.exchange_name,
|
||||||
frequency=data_frequency,
|
frequency=data_frequency,
|
||||||
symbol=asset.symbol
|
)) as it:
|
||||||
)) as it:
|
|
||||||
for chunk in it:
|
for chunk in it:
|
||||||
problems += self.ingest_ctable(
|
problems += self.ingest_ctable(
|
||||||
asset=chunk['asset'],
|
asset=chunk['asset'],
|
||||||
@@ -617,30 +642,6 @@ class ExchangeBundle:
|
|||||||
empty_rows_behavior='strip',
|
empty_rows_behavior='strip',
|
||||||
cleanup=True
|
cleanup=True
|
||||||
)
|
)
|
||||||
else:
|
|
||||||
all_chunks = list(chain.from_iterable(itervalues(chunks)))
|
|
||||||
|
|
||||||
# We sort the chunks by end date to ingest most recent data first
|
|
||||||
all_chunks.sort(
|
|
||||||
key=lambda chunk: pd.to_datetime(chunk['period'])
|
|
||||||
)
|
|
||||||
with maybe_show_progress(
|
|
||||||
all_chunks,
|
|
||||||
show_progress,
|
|
||||||
label='Ingesting {frequency} price data on '
|
|
||||||
'{exchange}'.format(
|
|
||||||
exchange=self.exchange_name,
|
|
||||||
frequency=data_frequency,
|
|
||||||
)) as it:
|
|
||||||
for chunk in it:
|
|
||||||
problems += self.ingest_ctable(
|
|
||||||
asset=chunk['asset'],
|
|
||||||
data_frequency=data_frequency,
|
|
||||||
period=chunk['period'],
|
|
||||||
writer=writer,
|
|
||||||
empty_rows_behavior='strip',
|
|
||||||
cleanup=True
|
|
||||||
)
|
|
||||||
|
|
||||||
if show_report and len(problems) > 0:
|
if show_report and len(problems) > 0:
|
||||||
log.info('problems during ingestion:{}\n'.format(
|
log.info('problems during ingestion:{}\n'.format(
|
||||||
|
|||||||
@@ -95,11 +95,24 @@ class TradingEnvironment(object):
|
|||||||
if not trading_calendar:
|
if not trading_calendar:
|
||||||
trading_calendar = get_calendar("NYSE")
|
trading_calendar = get_calendar("NYSE")
|
||||||
|
|
||||||
self.benchmark_returns, self.treasury_curves = load(
|
# todo: uncomment and add a well defined benchmark
|
||||||
trading_calendar.day,
|
# self.benchmark_returns, self.treasury_curves = load(
|
||||||
trading_calendar.schedule.index,
|
# trading_calendar.day,
|
||||||
self.bm_symbol,
|
# trading_calendar.schedule.index,
|
||||||
)
|
# self.bm_symbol,
|
||||||
|
# exchange=exchange,
|
||||||
|
# )
|
||||||
|
|
||||||
|
start_data = get_calendar('OPEN').first_trading_session
|
||||||
|
end_data = pd.Timestamp.utcnow()
|
||||||
|
treasure_cols = ['1month', '3month', '6month', '1year', '2year',
|
||||||
|
'3year', '5year', '7year', '10year', '20year', '30year']
|
||||||
|
self.benchmark_returns = pd.DataFrame(data=0.001,
|
||||||
|
index=pd.date_range(start_data, end_data),
|
||||||
|
columns=['close'])
|
||||||
|
self.treasury_curves = pd.DataFrame(data=0.001,
|
||||||
|
index=pd.date_range(start_data, end_data),
|
||||||
|
columns=treasure_cols)
|
||||||
|
|
||||||
self.exchange_tz = exchange_tz
|
self.exchange_tz = exchange_tz
|
||||||
|
|
||||||
|
|||||||
@@ -1 +1 @@
|
|||||||
0x39a54f480d922a58c963de8091a6c9afc69db2cf
|
0xf0ee6b27b759c9893ce4f094b49ad28fd15a23e4
|
||||||
@@ -1 +1 @@
|
|||||||
0xa2b37c6cd52f60fd4eb46ca59fafcf22d081aebc
|
0xa64927358a82254be92eb1f1cb01de68d1787004
|
||||||
@@ -69,7 +69,10 @@ class Marketplace:
|
|||||||
contract_url.info().get_content_charset()).strip())
|
contract_url.info().get_content_charset()).strip())
|
||||||
|
|
||||||
abi_url = urllib.urlopen(MARKETPLACE_CONTRACT_ABI)
|
abi_url = urllib.urlopen(MARKETPLACE_CONTRACT_ABI)
|
||||||
abi = json.load(abi_url)
|
abi_url = abi_url.read().decode(
|
||||||
|
abi_url.info().get_content_charset())
|
||||||
|
|
||||||
|
abi = json.loads(abi_url)
|
||||||
|
|
||||||
self.mkt_contract = self.web3.eth.contract(
|
self.mkt_contract = self.web3.eth.contract(
|
||||||
self.mkt_contract_address,
|
self.mkt_contract_address,
|
||||||
@@ -83,7 +86,10 @@ class Marketplace:
|
|||||||
contract_url.info().get_content_charset()).strip())
|
contract_url.info().get_content_charset()).strip())
|
||||||
|
|
||||||
abi_url = urllib.urlopen(ENIGMA_CONTRACT_ABI)
|
abi_url = urllib.urlopen(ENIGMA_CONTRACT_ABI)
|
||||||
abi = json.load(abi_url)
|
abi_url = abi_url.read().decode(
|
||||||
|
abi_url.info().get_content_charset())
|
||||||
|
|
||||||
|
abi = json.loads(abi_url)
|
||||||
|
|
||||||
self.eng_contract = self.web3.eth.contract(
|
self.eng_contract = self.web3.eth.contract(
|
||||||
self.eng_contract_address,
|
self.eng_contract_address,
|
||||||
@@ -126,9 +132,10 @@ class Marketplace:
|
|||||||
else:
|
else:
|
||||||
while True:
|
while True:
|
||||||
for i in range(0, len(self.addresses)):
|
for i in range(0, len(self.addresses)):
|
||||||
print('{}\t{}\t{}'.format(
|
print('{}\t{}\t{}\t{}'.format(
|
||||||
i,
|
i,
|
||||||
self.addresses[i]['pubAddr'],
|
self.addresses[i]['pubAddr'],
|
||||||
|
self.addresses[i]['wallet'].ljust(10),
|
||||||
self.addresses[i]['desc'])
|
self.addresses[i]['desc'])
|
||||||
)
|
)
|
||||||
address_i = int(input('Choose your address associated with '
|
address_i = int(input('Choose your address associated with '
|
||||||
@@ -145,7 +152,7 @@ class Marketplace:
|
|||||||
|
|
||||||
def sign_transaction(self, tx):
|
def sign_transaction(self, tx):
|
||||||
|
|
||||||
url = 'https://www.myetherwallet.com/#offline-transaction'
|
url = 'https://www.mycrypto.com/#offline-transaction'
|
||||||
print('\nVisit {url} and enter the following parameters:\n\n'
|
print('\nVisit {url} and enter the following parameters:\n\n'
|
||||||
'From Address:\t\t{_from}\n'
|
'From Address:\t\t{_from}\n'
|
||||||
'\n\tClick the "Generate Information" button\n\n'
|
'\n\tClick the "Generate Information" button\n\n'
|
||||||
@@ -430,10 +437,9 @@ class Marketplace:
|
|||||||
merge_bundles(zsource, ztarget)
|
merge_bundles(zsource, ztarget)
|
||||||
|
|
||||||
else:
|
else:
|
||||||
|
shutil.rmtree(bundle_folder, ignore_errors=True)
|
||||||
os.rename(tmp_bundle, bundle_folder)
|
os.rename(tmp_bundle, bundle_folder)
|
||||||
|
|
||||||
pass
|
|
||||||
|
|
||||||
def ingest(self, ds_name=None, start=None, end=None, force_download=False):
|
def ingest(self, ds_name=None, start=None, end=None, force_download=False):
|
||||||
|
|
||||||
if ds_name is None:
|
if ds_name is None:
|
||||||
@@ -498,20 +504,29 @@ class Marketplace:
|
|||||||
key = self.addresses[address_i]['key']
|
key = self.addresses[address_i]['key']
|
||||||
secret = self.addresses[address_i]['secret']
|
secret = self.addresses[address_i]['secret']
|
||||||
else:
|
else:
|
||||||
key, secret = get_key_secret(address)
|
key, secret = get_key_secret(address,
|
||||||
|
self.addresses[address_i]['wallet'])
|
||||||
|
|
||||||
headers = get_signed_headers(ds_name, key, secret)
|
headers = get_signed_headers(ds_name, key, secret)
|
||||||
log.debug('Starting download of dataset for ingestion...')
|
log.info('Starting download of dataset for ingestion...')
|
||||||
r = requests.post(
|
r = requests.post(
|
||||||
'{}/marketplace/ingest'.format(AUTH_SERVER),
|
'{}/marketplace/ingest'.format(AUTH_SERVER),
|
||||||
headers=headers,
|
headers=headers,
|
||||||
stream=True,
|
stream=True,
|
||||||
)
|
)
|
||||||
if r.status_code == 200:
|
if r.status_code == 200:
|
||||||
|
log.info('Dataset downloaded successfully. Processing dataset...')
|
||||||
target_path = get_temp_bundles_folder()
|
target_path = get_temp_bundles_folder()
|
||||||
try:
|
try:
|
||||||
decoder = MultipartDecoder.from_response(r)
|
decoder = MultipartDecoder.from_response(r)
|
||||||
|
# with maybe_show_progress(
|
||||||
|
# iter(decoder.parts),
|
||||||
|
# True,
|
||||||
|
# label='Processing files') as part:
|
||||||
|
counter = 1
|
||||||
for part in decoder.parts:
|
for part in decoder.parts:
|
||||||
|
log.info("Processing file {} of {}".format(
|
||||||
|
counter, len(decoder.parts)))
|
||||||
h = part.headers[b'Content-Disposition'].decode('utf-8')
|
h = part.headers[b'Content-Disposition'].decode('utf-8')
|
||||||
# Extracting the filename from the header
|
# Extracting the filename from the header
|
||||||
name = re.search(r'filename="(.*)"', h).group(1)
|
name = re.search(r'filename="(.*)"', h).group(1)
|
||||||
@@ -525,6 +540,7 @@ class Marketplace:
|
|||||||
f.write(part.content)
|
f.write(part.content)
|
||||||
|
|
||||||
self.process_temp_bundle(ds_name, filename)
|
self.process_temp_bundle(ds_name, filename)
|
||||||
|
counter += 1
|
||||||
|
|
||||||
except NonMultipartContentTypeException:
|
except NonMultipartContentTypeException:
|
||||||
response = r.json()
|
response = r.json()
|
||||||
@@ -592,7 +608,6 @@ class Marketplace:
|
|||||||
folder = get_bundle_folder(ds_name, data_frequency)
|
folder = get_bundle_folder(ds_name, data_frequency)
|
||||||
|
|
||||||
shutil.rmtree(folder)
|
shutil.rmtree(folder)
|
||||||
pass
|
|
||||||
|
|
||||||
def create_metadata(self, key, secret, ds_name, data_frequency, desc,
|
def create_metadata(self, key, secret, ds_name, data_frequency, desc,
|
||||||
has_history=True, has_live=True):
|
has_history=True, has_live=True):
|
||||||
@@ -628,7 +643,7 @@ class Marketplace:
|
|||||||
def register(self):
|
def register(self):
|
||||||
while True:
|
while True:
|
||||||
desc = input('Enter the name of the dataset to register: ')
|
desc = input('Enter the name of the dataset to register: ')
|
||||||
dataset = desc.lower()
|
dataset = desc.lower().strip()
|
||||||
provider_info = self.mkt_contract.functions.getDataProviderInfo(
|
provider_info = self.mkt_contract.functions.getDataProviderInfo(
|
||||||
Web3.toHex(dataset)
|
Web3.toHex(dataset)
|
||||||
).call()
|
).call()
|
||||||
@@ -684,7 +699,8 @@ class Marketplace:
|
|||||||
key = self.addresses[address_i]['key']
|
key = self.addresses[address_i]['key']
|
||||||
secret = self.addresses[address_i]['secret']
|
secret = self.addresses[address_i]['secret']
|
||||||
else:
|
else:
|
||||||
key, secret = get_key_secret(address)
|
key, secret = get_key_secret(address,
|
||||||
|
self.addresses[address_i]['wallet'])
|
||||||
|
|
||||||
grains = to_grains(price)
|
grains = to_grains(price)
|
||||||
|
|
||||||
@@ -765,28 +781,34 @@ class Marketplace:
|
|||||||
key = match['key']
|
key = match['key']
|
||||||
secret = match['secret']
|
secret = match['secret']
|
||||||
else:
|
else:
|
||||||
key, secret = get_key_secret(provider_info[0])
|
key, secret = get_key_secret(provider_info[0], match['wallet'])
|
||||||
|
|
||||||
headers = get_signed_headers(dataset, key, secret)
|
|
||||||
filenames = glob.glob(os.path.join(datadir, '*.csv'))
|
filenames = glob.glob(os.path.join(datadir, '*.csv'))
|
||||||
|
|
||||||
if not filenames:
|
if not filenames:
|
||||||
raise MarketplaceNoCSVFiles(datadir=datadir)
|
raise MarketplaceNoCSVFiles(datadir=datadir)
|
||||||
|
|
||||||
files = []
|
files = []
|
||||||
for file in filenames:
|
for idx, file in enumerate(filenames):
|
||||||
|
log.info('Uploading file {} of {}: {}'.format(
|
||||||
|
idx+1, len(filenames), file))
|
||||||
|
files = []
|
||||||
files.append(('file', open(file, 'rb')))
|
files.append(('file', open(file, 'rb')))
|
||||||
|
|
||||||
r = requests.post('{}/marketplace/publish'.format(AUTH_SERVER),
|
headers = get_signed_headers(dataset, key, secret)
|
||||||
files=files,
|
r = requests.post('{}/marketplace/publish'.format(AUTH_SERVER),
|
||||||
headers=headers)
|
files=files,
|
||||||
|
headers=headers)
|
||||||
|
|
||||||
if r.status_code != 200:
|
if r.status_code != 200:
|
||||||
raise MarketplaceHTTPRequest(request='upload file',
|
raise MarketplaceHTTPRequest(request='upload file',
|
||||||
error=r.status_code)
|
error=r.status_code)
|
||||||
|
|
||||||
if 'error' in r.json():
|
if 'error' in r.json():
|
||||||
raise MarketplaceHTTPRequest(request='upload file',
|
raise MarketplaceHTTPRequest(request='upload file',
|
||||||
error=r.json()['error'])
|
error=r.json()['error'])
|
||||||
|
|
||||||
print('Dataset {} uploaded successfully.'.format(dataset))
|
log.info('File processed successfully.')
|
||||||
|
|
||||||
|
print('\nDataset {} uploaded and processed successfully.'.format(
|
||||||
|
dataset))
|
||||||
|
|||||||
@@ -10,10 +10,10 @@ from catalyst.marketplace.marketplace_errors import (
|
|||||||
MarketplaceEmptySignature)
|
MarketplaceEmptySignature)
|
||||||
from catalyst.marketplace.utils.path_utils import (
|
from catalyst.marketplace.utils.path_utils import (
|
||||||
get_user_pubaddr, save_user_pubaddr)
|
get_user_pubaddr, save_user_pubaddr)
|
||||||
from catalyst.constants import AUTH_SERVER
|
from catalyst.constants import AUTH_SERVER, SUPPORTED_WALLETS
|
||||||
|
|
||||||
|
|
||||||
def get_key_secret(pubAddr, wallet='mew'):
|
def get_key_secret(pubAddr, wallet):
|
||||||
"""
|
"""
|
||||||
Obtain a new key/secret pair from authentication server
|
Obtain a new key/secret pair from authentication server
|
||||||
|
|
||||||
@@ -43,21 +43,22 @@ def get_key_secret(pubAddr, wallet='mew'):
|
|||||||
auth_type, auth_info = header.split(None, 1)
|
auth_type, auth_info = header.split(None, 1)
|
||||||
d = requests.utils.parse_dict_header(auth_info)
|
d = requests.utils.parse_dict_header(auth_info)
|
||||||
|
|
||||||
nonce = '0x{}'.format(d['nonce'])
|
nonce = 'Catalyst nonce: 0x{}'.format(d['nonce'])
|
||||||
|
|
||||||
if wallet == 'mew':
|
if wallet in SUPPORTED_WALLETS:
|
||||||
url = 'https://www.myetherwallet.com/signmsg.html'
|
url = 'https://www.mycrypto.com/signmsg.html'
|
||||||
|
|
||||||
print('\nObtaining a key/secret pair to streamline all future '
|
print('\nObtaining a key/secret pair to streamline all future '
|
||||||
'requests with the authentication server.\n'
|
'requests with the authentication server.\n'
|
||||||
'Visit {url} and sign the '
|
'Visit {url} and sign the '
|
||||||
'following message:\n{nonce}'.format(
|
'following message (copy the entire line, without the '
|
||||||
|
'line break at the end):\n\n{nonce}'.format(
|
||||||
url=url,
|
url=url,
|
||||||
nonce=nonce))
|
nonce=nonce))
|
||||||
|
|
||||||
webbrowser.open_new(url)
|
webbrowser.open_new(url)
|
||||||
|
|
||||||
signature = input('Copy and Paste the "sig" field from '
|
signature = input('\nCopy and Paste the "sig" field from '
|
||||||
'the signature here (without the double quotes, '
|
'the signature here (without the double quotes, '
|
||||||
'only the HEX value):\n')
|
'only the HEX value):\n')
|
||||||
else:
|
else:
|
||||||
@@ -91,7 +92,8 @@ def get_key_secret(pubAddr, wallet='mew'):
|
|||||||
addresses = get_user_pubaddr()
|
addresses = get_user_pubaddr()
|
||||||
|
|
||||||
match = next((l for l in addresses if
|
match = next((l for l in addresses if
|
||||||
l['pubAddr'] == pubAddr), None)
|
l['pubAddr'].lower() == pubAddr.lower()), None)
|
||||||
|
|
||||||
match['key'] = response.json()['key']
|
match['key'] = response.json()['key']
|
||||||
match['secret'] = response.json()['secret']
|
match['secret'] = response.json()['secret']
|
||||||
|
|
||||||
@@ -121,7 +123,7 @@ def get_signed_headers(ds_name, key, secret):
|
|||||||
-------
|
-------
|
||||||
|
|
||||||
"""
|
"""
|
||||||
nonce = str(int(time.time()))
|
nonce = str(int(time.time() * 1000))
|
||||||
|
|
||||||
signature = hmac.new(
|
signature = hmac.new(
|
||||||
secret.encode('utf-8'),
|
secret.encode('utf-8'),
|
||||||
|
|||||||
@@ -2,6 +2,7 @@ import os
|
|||||||
import json
|
import json
|
||||||
import tarfile
|
import tarfile
|
||||||
|
|
||||||
|
from catalyst.constants import SUPPORTED_WALLETS
|
||||||
from catalyst.utils.deprecate import deprecated
|
from catalyst.utils.deprecate import deprecated
|
||||||
from catalyst.utils.paths import data_root, ensure_directory
|
from catalyst.utils.paths import data_root, ensure_directory
|
||||||
from catalyst.marketplace.marketplace_errors import MarketplaceJSONError
|
from catalyst.marketplace.marketplace_errors import MarketplaceJSONError
|
||||||
@@ -131,17 +132,63 @@ def get_user_pubaddr(environ=None):
|
|||||||
try:
|
try:
|
||||||
d = data[0]['pubAddr']
|
d = data[0]['pubAddr']
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
return [data, ]
|
data = [data, ]
|
||||||
|
|
||||||
|
changed = False
|
||||||
|
|
||||||
|
for idx, d in enumerate(data):
|
||||||
|
try:
|
||||||
|
if d['wallet'] not in SUPPORTED_WALLETS:
|
||||||
|
data[idx]['wallet'] = _choose_wallet(
|
||||||
|
d['pubAddr'], False)
|
||||||
|
changed = True
|
||||||
|
except KeyError:
|
||||||
|
data[idx]['wallet'] = _choose_wallet(
|
||||||
|
d['pubAddr'], True)
|
||||||
|
changed = True
|
||||||
|
|
||||||
|
if changed:
|
||||||
|
save_user_pubaddr(data)
|
||||||
|
|
||||||
return data
|
return data
|
||||||
|
|
||||||
else:
|
else:
|
||||||
data = []
|
data = []
|
||||||
data.append(dict(pubAddr='', desc=''))
|
data.append(dict(pubAddr='', desc='', wallet=''))
|
||||||
with open(filename, 'w') as f:
|
with open(filename, 'w') as f:
|
||||||
json.dump(data, f, sort_keys=False, indent=2,
|
json.dump(data, f, sort_keys=False, indent=2,
|
||||||
separators=(',', ':'))
|
separators=(',', ':'))
|
||||||
return data
|
return data
|
||||||
|
|
||||||
|
|
||||||
|
def _choose_wallet(pubAddr, missing):
|
||||||
|
while True:
|
||||||
|
if missing:
|
||||||
|
print('\nYou need to specify a wallet for address '
|
||||||
|
'{}.'.format(pubAddr))
|
||||||
|
else:
|
||||||
|
print('\nThe wallet specified for address {} is not '
|
||||||
|
'supported.'.format(pubAddr))
|
||||||
|
|
||||||
|
print('Please choose among the following options:')
|
||||||
|
for idx, wallet in enumerate(SUPPORTED_WALLETS):
|
||||||
|
print('{}\t{}'.format(idx, wallet))
|
||||||
|
|
||||||
|
lw = len(SUPPORTED_WALLETS)-1
|
||||||
|
w = input('Choose a number between 0 and {}: '.format(
|
||||||
|
lw))
|
||||||
|
try:
|
||||||
|
w = int(w)
|
||||||
|
except ValueError:
|
||||||
|
print('Enter a number between 0 and {}'.format(lw))
|
||||||
|
else:
|
||||||
|
if w not in range(0, lw+1):
|
||||||
|
print('Enter a number between 0 and '
|
||||||
|
'{}'.format(lw))
|
||||||
|
else:
|
||||||
|
return SUPPORTED_WALLETS[w]
|
||||||
|
|
||||||
|
|
||||||
def save_user_pubaddr(data, environ=None):
|
def save_user_pubaddr(data, environ=None):
|
||||||
"""
|
"""
|
||||||
Saves the user's public addresses and their related metadata in
|
Saves the user's public addresses and their related metadata in
|
||||||
|
|||||||
@@ -562,6 +562,10 @@ If after following the instructions above, and going through the
|
|||||||
*Troubleshooting* sections, you still experience problems installing Catalyst,
|
*Troubleshooting* sections, you still experience problems installing Catalyst,
|
||||||
you can seek additional help through the following channels:
|
you can seek additional help through the following channels:
|
||||||
|
|
||||||
|
- Join our `Catalyst Forum <https://catalyst.enigma.co/>`_, and browse a variety
|
||||||
|
of topics and conversations around common issues that others face when using
|
||||||
|
Catalyst, and how to resolve them. And join the conversation!
|
||||||
|
|
||||||
- Join our `Discord community <https://discord.gg/SJK32GY>`_, and head over
|
- Join our `Discord community <https://discord.gg/SJK32GY>`_, and head over
|
||||||
the #catalyst_dev channel where many other users (as well as the project
|
the #catalyst_dev channel where many other users (as well as the project
|
||||||
developers) hang out, and can assist you with your particular issue. The
|
developers) hang out, and can assist you with your particular issue. The
|
||||||
|
|||||||
@@ -2,6 +2,39 @@
|
|||||||
Release Notes
|
Release Notes
|
||||||
=============
|
=============
|
||||||
|
|
||||||
|
Version 0.5.7
|
||||||
|
^^^^^^^^^^^^^
|
||||||
|
**Release Date**: 2018-03-29
|
||||||
|
|
||||||
|
Build
|
||||||
|
~~~~~
|
||||||
|
- Data Marketplace deployed on mainnet.
|
||||||
|
- Added progress indicators for publishing data, and made the data publishing
|
||||||
|
synchronous to provide feedback to the publisher.
|
||||||
|
|
||||||
|
Bug Fixes
|
||||||
|
~~~~~~~~~
|
||||||
|
- Added arguments to the ``reduce`` function in tha Asset class :issue:`214`,
|
||||||
|
:issue:`287`
|
||||||
|
|
||||||
|
Version 0.5.6
|
||||||
|
^^^^^^^^^^^^^
|
||||||
|
**Release Date**: 2018-03-22
|
||||||
|
|
||||||
|
Build
|
||||||
|
~~~~~
|
||||||
|
- Data Marketplace: ensures compatibility across wallets, now fully supporting
|
||||||
|
``ledger``, ``trezor``, ``keystore``, ``private key``. Partial support for
|
||||||
|
``metamask`` (includes sign_msg, but not sign_tx). Current support for
|
||||||
|
``Digital Bitbox`` is unknown, but believed to be supported.
|
||||||
|
- Data Marketplace: Switched online provider from MyEtherWallet to MyCrypto.
|
||||||
|
- Data Marketplace: Added progress indicator for data ingestion.
|
||||||
|
|
||||||
|
Bug Fixes
|
||||||
|
~~~~~~~~~
|
||||||
|
- Changed benchmark to be constant, so it doesn't ingest data at all. Temporary
|
||||||
|
fix for :issue:`271`, :issue:`285`
|
||||||
|
|
||||||
Version 0.5.5
|
Version 0.5.5
|
||||||
^^^^^^^^^^^^^
|
^^^^^^^^^^^^^
|
||||||
**Release Date**: 2018-03-19
|
**Release Date**: 2018-03-19
|
||||||
|
|||||||
Reference in New Issue
Block a user