Error while ingesting data


#1

Hi,

I’m getting the following error when I try to ingest all of the binance data. Am I doing something wrong or perhaps its due to the amount of data I’m trying to ingest?

(catalyst) xyz@xyz:~/catalyst$ catalyst ingest-exchange -x binance -f minute
Trying to ingest exchange bundle binance…
[======================= ] Ingesting minute price data on binance: 64% 0d 00:21:11
Traceback (most recent call last):
File “/home/xyz/catalyst/catalyst/bin/catalyst”, line 11, in
sys.exit(main())
File “/home/xyz/catalyst/catalyst/local/lib/python2.7/site-packages/click/core.py”, line 722, in call
return self.main(*args, **kwargs)
File “/home/xyz/catalyst/catalyst/local/lib/python2.7/site-packages/click/core.py”, line 697, in main
rv = self.invoke(ctx)
File “/home/xyz/catalyst/catalyst/local/lib/python2.7/site-packages/click/core.py”, line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File “/home/xyz/catalyst/catalyst/local/lib/python2.7/site-packages/click/core.py”, line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File “/home/xyz/catalyst/catalyst/local/lib/python2.7/site-packages/click/core.py”, line 535, in invoke
return callback(*args, **kwargs)
File “/home/xyz/catalyst/catalyst/local/lib/python2.7/site-packages/click/decorators.py”, line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File “/home/xyz/catalyst/catalyst/local/lib/python2.7/site-packages/catalyst/main.py”, line 609, in ingest_exchange
csv=csv
File “/home/xyz/catalyst/catalyst/local/lib/python2.7/site-packages/catalyst/exchange/exchange_bundle.py”, line 824, in ingest
show_report=show_report
File “/home/xyz/catalyst/catalyst/local/lib/python2.7/site-packages/catalyst/exchange/exchange_bundle.py”, line 643, in ingest_assets
cleanup=True
File “/home/xyz/catalyst/catalyst/local/lib/python2.7/site-packages/catalyst/exchange/exchange_bundle.py”, line 365, in ingest_ctable
period=period
File “/home/xyz/catalyst/catalyst/local/lib/python2.7/site-packages/catalyst/exchange/utils/bundle_utils.py”, line 47, in get_bcolz_chunk
bytes = download_without_progress(url)
File “/home/xyz/catalyst/catalyst/local/lib/python2.7/site-packages/catalyst/data/bundles/core.py”, line 190, in download_without_progress
resp.raise_for_status()
File “/home/xyz/catalyst/catalyst/local/lib/python2.7/site-packages/requests/models.py”, line 935, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://s3.amazonaws.com/enigmaco/catalyst-bundles/exchange-binance/binance-minute-nuls_btc-2018-06.tar.gz

Thanks in advance


#2

Hi Papan,

Ingesting all Binance minute data can be challenging, as there are 387 available trading pairs.
Have you tried ingesting minute data of a specific pair?

Thanks,
Lena