Ingesting Data fails silently


#1

I’m trying to get started with enigma catalyst but running into issues at the data ingestion step:

pipenv run catalyst ingest-exchange -x binance -i btc_usdt
Trying to ingest exchange bundle binance…

The script just fails silently without loading a progress bar. Same in verbose mode. I’ve tried it with binance, bittrex, bitfinex, poloniex at the 1m and daily timeframes. The status page says everything is up.

When I try to run the backtest directly without ingesting I get the following error:

pipenv run catalyst run -f my_first_algorithm.py -x bitfinex --start 2016-1-1 --end 2017-9-30 -c usd --capital-base 100000 -o buy_btc_simple_out.pickle
catalyst.exchange.exchange_errors.NoDataAvailableOnExchange: Requested data for trading pair BTC/USD is not available on exchange bitfinex in daily frequency at this time. Check http://enigma.co/catalyst/status for market coverage.

Has anyone run into this issue before? Thanks

I installed catalyst using pipenv - here are the details:

python version: 3.6.1
Mac OS HighSierra 10.13.4
enigma-catalyst==0.5.18

  • alembic [required: ==0.9.7, installed: 0.9.7]
    • Mako [required: Any, installed: 1.0.7]
      • MarkupSafe [required: >=0.9.2, installed: 1.0]
    • python-dateutil [required: Any, installed: 2.7.3]
      • six [required: >=1.5, installed: 1.11.0]
    • python-editor [required: >=0.3, installed: 1.0.3]
    • SQLAlchemy [required: >=0.7.6, installed: 1.2.2]
  • bcolz [required: ==1.2.1, installed: 1.2.1]
    • numpy [required: >=1.7, installed: 1.14.0]
  • boto3 [required: ==1.5.27, installed: 1.5.27]
    • botocore [required: >=1.8.41,<1.9.0, installed: 1.8.50]
      • docutils [required: >=0.10, installed: 0.14]
      • jmespath [required: >=0.7.1,<1.0.0, installed: 0.9.3]
      • python-dateutil [required: >=2.1,<3.0.0, installed: 2.7.3]
        • six [required: >=1.5, installed: 1.11.0]
    • jmespath [required: >=0.7.1,<1.0.0, installed: 0.9.3]
    • s3transfer [required: >=0.1.10,<0.2.0, installed: 0.1.13]
      • botocore [required: >=1.3.0,<2.0.0, installed: 1.8.50]
        • docutils [required: >=0.10, installed: 0.14]
        • jmespath [required: >=0.7.1,<1.0.0, installed: 0.9.3]
        • python-dateutil [required: >=2.1,<3.0.0, installed: 2.7.3]
          • six [required: >=1.5, installed: 1.11.0]
  • bottleneck [required: ==1.2.1, installed: 1.2.1]
    • numpy [required: Any, installed: 1.14.0]
  • ccxt [required: ==1.12.131, installed: 1.12.131]
    • aiodns [required: ==1.1.1, installed: 1.1.1]
      • pycares [required: >=1.0.0, installed: 2.3.0]
    • aiohttp [required: ==3.0.1, installed: 3.0.1]
      • async-timeout [required: >=1.2,<3.0, installed: 2.0.1]
      • attrs [required: >=17.4.0, installed: 18.1.0]
      • chardet [required: >=2.0,<4.0, installed: 3.0.4]
      • idna-ssl [required: >=1.0, installed: 1.1.0]
        • idna [required: >=2.0, installed: 2.6]
      • multidict [required: >=4.0,<5.0, installed: 4.3.1]
      • yarl [required: >=1.0,<2.0, installed: 1.1.0]
        • idna [required: >=2.0, installed: 2.6]
        • multidict [required: >=4.0, installed: 4.3.1]
    • cchardet [required: ==2.1.1, installed: 2.1.1]
    • certifi [required: ==2018.1.18, installed: 2018.1.18]
    • requests [required: ==2.18.4, installed: 2.18.4]
      • certifi [required: >=2017.4.17, installed: 2018.1.18]
      • chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
      • idna [required: >=2.5,<2.7, installed: 2.6]
      • urllib3 [required: >=1.21.1,<1.23, installed: 1.22]
    • setuptools [required: ==38.5.1, installed: 38.5.1]
    • yarl [required: ==1.1.0, installed: 1.1.0]
      • idna [required: >=2.0, installed: 2.6]
      • multidict [required: >=4.0, installed: 4.3.1]
  • certifi [required: ==2018.1.18, installed: 2018.1.18]
  • click [required: ==6.7, installed: 6.7]
  • contextlib2 [required: ==0.5.5, installed: 0.5.5]
  • cyordereddict [required: ==1.0.0, installed: 1.0.0]
  • Cython [required: ==0.27.3, installed: 0.27.3]
  • decorator [required: ==4.3.0, installed: 4.3.0]
  • empyrical [required: ==0.2.1, installed: 0.2.1]
    • bottleneck [required: >=1.0.0, installed: 1.2.1]
      • numpy [required: Any, installed: 1.14.0]
    • numpy [required: >=1.9.2, installed: 1.14.0]
    • pandas [required: >=0.16.1, installed: 0.19.2]
      • numpy [required: >=1.7.0, installed: 1.14.0]
      • python-dateutil [required: >=2, installed: 2.7.3]
        • six [required: >=1.5, installed: 1.11.0]
      • pytz [required: >=2011k, installed: 2016.4]
    • scipy [required: >=0.15.1, installed: 1.0.0]
      • numpy [required: >=1.8.2, installed: 1.14.0]
  • intervaltree [required: ==2.1.0, installed: 2.1.0]
    • sortedcontainers [required: Any, installed: 1.5.9]
  • Logbook [required: ==0.12.5, installed: 0.12.5]
  • lru-dict [required: ==1.1.6, installed: 1.1.6]
  • Mako [required: ==1.0.7, installed: 1.0.7]
    • MarkupSafe [required: >=0.9.2, installed: 1.0]
  • MarkupSafe [required: ==1.0, installed: 1.0]
  • multipledispatch [required: ==0.4.9, installed: 0.4.9]
  • networkx [required: ==2.1, installed: 2.1]
    • decorator [required: >=4.1.0, installed: 4.3.0]
  • numexpr [required: ==2.6.4, installed: 2.6.4]
    • numpy [required: >=1.7, installed: 1.14.0]
  • numpy [required: ==1.14.0, installed: 1.14.0]
  • pandas [required: ==0.19.2, installed: 0.19.2]
    • numpy [required: >=1.7.0, installed: 1.14.0]
    • python-dateutil [required: >=2, installed: 2.7.3]
      • six [required: >=1.5, installed: 1.11.0]
    • pytz [required: >=2011k, installed: 2016.4]
  • pandas-datareader [required: ==0.6.0, installed: 0.6.0]
    • lxml [required: Any, installed: 4.2.4]
    • pandas [required: >=0.19.2, installed: 0.19.2]
      • numpy [required: >=1.7.0, installed: 1.14.0]
      • python-dateutil [required: >=2, installed: 2.7.3]
        • six [required: >=1.5, installed: 1.11.0]
      • pytz [required: >=2011k, installed: 2016.4]
    • requests [required: >=2.3.0, installed: 2.18.4]
      • certifi [required: >=2017.4.17, installed: 2018.1.18]
      • chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
      • idna [required: >=2.5,<2.7, installed: 2.6]
      • urllib3 [required: >=1.21.1,<1.23, installed: 1.22]
    • requests-file [required: Any, installed: 1.4.3]
      • requests [required: >=1.0.0, installed: 2.18.4]
        • certifi [required: >=2017.4.17, installed: 2018.1.18]
        • chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
        • idna [required: >=2.5,<2.7, installed: 2.6]
        • urllib3 [required: >=1.21.1,<1.23, installed: 1.22]
      • six [required: Any, installed: 1.11.0]
    • requests-ftp [required: Any, installed: 0.3.1]
      • requests [required: Any, installed: 2.18.4]
        • certifi [required: >=2017.4.17, installed: 2018.1.18]
        • chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
        • idna [required: >=2.5,<2.7, installed: 2.6]
        • urllib3 [required: >=1.21.1,<1.23, installed: 1.22]
    • wrapt [required: Any, installed: 1.10.11]
  • patsy [required: ==0.5.0, installed: 0.5.0]
    • numpy [required: >=1.4, installed: 1.14.0]
    • six [required: Any, installed: 1.11.0]
  • pip [required: >=7.1.0, installed: 18.0]
  • python-dateutil [required: ==2.7.3, installed: 2.7.3]
    • six [required: >=1.5, installed: 1.11.0]
  • pytz [required: ==2016.4, installed: 2016.4]
  • redo [required: ==1.6, installed: 1.6]
  • requests [required: ==2.18.4, installed: 2.18.4]
    • certifi [required: >=2017.4.17, installed: 2018.1.18]
    • chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
    • idna [required: >=2.5,<2.7, installed: 2.6]
    • urllib3 [required: >=1.21.1,<1.23, installed: 1.22]
  • requests-file [required: ==1.4.3, installed: 1.4.3]
    • requests [required: >=1.0.0, installed: 2.18.4]
      • certifi [required: >=2017.4.17, installed: 2018.1.18]
      • chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
      • idna [required: >=2.5,<2.7, installed: 2.6]
      • urllib3 [required: >=1.21.1,<1.23, installed: 1.22]
    • six [required: Any, installed: 1.11.0]
  • requests-toolbelt [required: ==0.8.0, installed: 0.8.0]
    • requests [required: >=2.0.1,<3.0.0, installed: 2.18.4]
      • certifi [required: >=2017.4.17, installed: 2018.1.18]
      • chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
      • idna [required: >=2.5,<2.7, installed: 2.6]
      • urllib3 [required: >=1.21.1,<1.23, installed: 1.22]
  • scipy [required: ==1.0.0, installed: 1.0.0]
    • numpy [required: >=1.8.2, installed: 1.14.0]
  • setuptools [required: ==38.5.1, installed: 38.5.1]
  • six [required: ==1.11.0, installed: 1.11.0]
  • sortedcontainers [required: ==1.5.9, installed: 1.5.9]
  • sqlalchemy [required: ==1.2.2, installed: 1.2.2]
  • statsmodels [required: ==0.8.0, installed: 0.8.0]
    • pandas [required: Any, installed: 0.19.2]
      • numpy [required: >=1.7.0, installed: 1.14.0]
      • python-dateutil [required: >=2, installed: 2.7.3]
        • six [required: >=1.5, installed: 1.11.0]
      • pytz [required: >=2011k, installed: 2016.4]
    • patsy [required: Any, installed: 0.5.0]
      • numpy [required: >=1.4, installed: 1.14.0]
      • six [required: Any, installed: 1.11.0]
    • scipy [required: Any, installed: 1.0.0]
      • numpy [required: >=1.8.2, installed: 1.14.0]
  • tables [required: ==3.4.2, installed: 3.4.2]
    • numexpr [required: >=2.5.2, installed: 2.6.4]
      • numpy [required: >=1.7, installed: 1.14.0]
    • numpy [required: >=1.8.0, installed: 1.14.0]
    • six [required: >=1.9.0, installed: 1.11.0]
  • toolz [required: ==0.9.0, installed: 0.9.0]
  • web3 [required: ==4.2.1, installed: 4.2.1]
    • cytoolz [required: >=0.9.0,<1.0.0, installed: 0.9.0.1]
      • toolz [required: >=0.8.0, installed: 0.9.0]
    • eth-abi [required: >=1.1.0,<2, installed: 1.1.1]
      • eth-utils [required: >=1.0.1,<2.0.0, installed: 1.0.3]
        • cytoolz [required: >=0.8.2,<1.0.0, installed: 0.9.0.1]
          • toolz [required: >=0.8.0, installed: 0.9.0]
        • eth-hash [required: >=0.1.0,<1.0.0, installed: 0.1.4]
      • parsimonious [required: ==0.8.0, installed: 0.8.0]
        • six [required: >=1.9.0, installed: 1.11.0]
    • eth-account [required: >=0.2.1,<0.3.0, installed: 0.2.3]
      • attrdict [required: >=2.0.0,<3, installed: 2.0.0]
        • six [required: Any, installed: 1.11.0]
      • eth-keyfile [required: >=0.5.0,<0.6.0, installed: 0.5.1]
        • cytoolz [required: >=0.9.0,<1.0.0, installed: 0.9.0.1]
          • toolz [required: >=0.8.0, installed: 0.9.0]
        • eth-keys [required: >=0.1.0-beta.4,<1.0.0, installed: 0.2.0b3]
          • cytoolz [required: >=0.9.0,<1.0.0, installed: 0.9.0.1]
            • toolz [required: >=0.8.0, installed: 0.9.0]
          • eth-utils [required: >=1.0.0-beta.2,<2.0.0, installed: 1.0.3]
            • cytoolz [required: >=0.8.2,<1.0.0, installed: 0.9.0.1]
              • toolz [required: >=0.8.0, installed: 0.9.0]
            • eth-hash [required: >=0.1.0,<1.0.0, installed: 0.1.4]
        • eth-utils [required: >=1.0.0-beta.1,<2.0.0, installed: 1.0.3]
          • cytoolz [required: >=0.8.2,<1.0.0, installed: 0.9.0.1]
            • toolz [required: >=0.8.0, installed: 0.9.0]
          • eth-hash [required: >=0.1.0,<1.0.0, installed: 0.1.4]
        • pycryptodome [required: >=3.4.7,<4.0.0, installed: 3.6.4]
      • eth-keys [required: >=0.2.0b3,<0.3.0, installed: 0.2.0b3]
        • cytoolz [required: >=0.9.0,<1.0.0, installed: 0.9.0.1]
          • toolz [required: >=0.8.0, installed: 0.9.0]
        • eth-utils [required: >=1.0.0-beta.2,<2.0.0, installed: 1.0.3]
          • cytoolz [required: >=0.8.2,<1.0.0, installed: 0.9.0.1]
            • toolz [required: >=0.8.0, installed: 0.9.0]
          • eth-hash [required: >=0.1.0,<1.0.0, installed: 0.1.4]
      • eth-rlp [required: >=0.1.2,<1, installed: 0.1.2]
        • eth-utils [required: >=1.0.1,<2, installed: 1.0.3]
          • cytoolz [required: >=0.8.2,<1.0.0, installed: 0.9.0.1]
            • toolz [required: >=0.8.0, installed: 0.9.0]
          • eth-hash [required: >=0.1.0,<1.0.0, installed: 0.1.4]
        • hexbytes [required: >=0.1.0,<1, installed: 0.1.0]
          • eth-utils [required: >=1.0.1,<2.0.0, installed: 1.0.3]
            • cytoolz [required: >=0.8.2,<1.0.0, installed: 0.9.0.1]
              • toolz [required: >=0.8.0, installed: 0.9.0]
            • eth-hash [required: >=0.1.0,<1.0.0, installed: 0.1.4]
        • rlp [required: >=0.6.0,<2, installed: 1.0.2]
          • eth-utils [required: >=1.0.2,<2, installed: 1.0.3]
            • cytoolz [required: >=0.8.2,<1.0.0, installed: 0.9.0.1]
              • toolz [required: >=0.8.0, installed: 0.9.0]
            • eth-hash [required: >=0.1.0,<1.0.0, installed: 0.1.4]
      • eth-utils [required: >=1.0.2,<2, installed: 1.0.3]
        • cytoolz [required: >=0.8.2,<1.0.0, installed: 0.9.0.1]
          • toolz [required: >=0.8.0, installed: 0.9.0]
        • eth-hash [required: >=0.1.0,<1.0.0, installed: 0.1.4]
      • hexbytes [required: >=0.1.0,<1, installed: 0.1.0]
        • eth-utils [required: >=1.0.1,<2.0.0, installed: 1.0.3]
          • cytoolz [required: >=0.8.2,<1.0.0, installed: 0.9.0.1]
            • toolz [required: >=0.8.0, installed: 0.9.0]
          • eth-hash [required: >=0.1.0,<1.0.0, installed: 0.1.4]
    • eth-hash [required: Any, installed: 0.1.4]
    • eth-utils [required: >=1.0.1,<2.0.0, installed: 1.0.3]
      • cytoolz [required: >=0.8.2,<1.0.0, installed: 0.9.0.1]
        • toolz [required: >=0.8.0, installed: 0.9.0]
      • eth-hash [required: >=0.1.0,<1.0.0, installed: 0.1.4]
    • hexbytes [required: >=0.1.0,<1.0.0, installed: 0.1.0]
      • eth-utils [required: >=1.0.1,<2.0.0, installed: 1.0.3]
        • cytoolz [required: >=0.8.2,<1.0.0, installed: 0.9.0.1]
          • toolz [required: >=0.8.0, installed: 0.9.0]
        • eth-hash [required: >=0.1.0,<1.0.0, installed: 0.1.4]
    • lru-dict [required: >=1.1.6,<2.0.0, installed: 1.1.6]
    • requests [required: >=2.16.0,<3.0.0, installed: 2.18.4]
      • certifi [required: >=2017.4.17, installed: 2018.1.18]
      • chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
      • idna [required: >=2.5,<2.7, installed: 2.6]
      • urllib3 [required: >=1.21.1,<1.23, installed: 1.22]
    • websockets [required: >=4.0.1, installed: 6.0]
      matplotlib==2.2.2
  • cycler [required: >=0.10, installed: 0.10.0]
    • six [required: Any, installed: 1.11.0]
  • kiwisolver [required: >=1.0.1, installed: 1.0.1]
    • setuptools [required: Any, installed: 38.5.1]
  • numpy [required: >=1.7.1, installed: 1.14.0]
  • pyparsing [required: >=2.0.1,!=2.1.6,!=2.1.2,!=2.0.4, installed: 2.2.0]
  • python-dateutil [required: >=2.1, installed: 2.7.3]
    • six [required: >=1.5, installed: 1.11.0]
  • pytz [required: Any, installed: 2016.4]
  • six [required: >=1.10, installed: 1.11.0]