Problem ingesting poloniex data


#1

I already posted this on Discord, but maybe it will have more chances to be noticed here.

When I ingest data from poloniex, I randomly get errors looking like the one attached below. It seems to complain about a temporary folder not being emply. I can empty this folder manually and reingest. Then I may or may not get again a random error like this one. Even if I ingest the full data successfully, I still get errors in Catalyst when backtesting. At the price of excluding XRP-BTC, I have been able to backtest BTC pairs from Jan 2016 to end of Nov 2017. I can also backtest from Dec 3 to Dec 29. In other periods I get repeated error messages claiming the data hasn’t been ingested.

What can be done about that?

Samuel


Ingesting exchange bundle poloniex…
[================== ] Ingesting minute price data on poloniex: 51% 0d 00:03:05[2018-02-27 19:47
:13.468000] WARNING: exchange_bundle: skipping ctable for grc_btc from 2018-02-01 00:00:00+00:00 to 2018-02-14 23:59:00+
00:00: [Errno 2] No such file or directory: u’C:\Users\Sam\.catalyst\data\exchanges\poloniex\temp_bundles\poloni
ex-minute-grc_btc-2018-02\67\95\679521.bcolz\open\meta\sizes’
[========================= ] Ingesting minute price data on poloniex: 69% 0d 00:01:53
Traceback (most recent call last):
File “c:\users\sam\anaconda2\envs\catalyst\lib\runpy.py”, line 174, in _run_module_as_main
"main", fname, loader, pkg_name)
File “c:\users\sam\anaconda2\envs\catalyst\lib\runpy.py”, line 72, in run_code
exec code in run_globals
File "C:\Users\Sam\Anaconda2\envs\catalyst\Scripts\catalyst.exe_main
.py", line 9, in
File “c:\users\sam\anaconda2\envs\catalyst\lib\site-packages\click\core.py”, line 722, in call
return self.main(*args, **kwargs)
File “c:\users\sam\anaconda2\envs\catalyst\lib\site-packages\click\core.py”, line 697, in main
rv = self.invoke(ctx)
File “c:\users\sam\anaconda2\envs\catalyst\lib\site-packages\click\core.py”, line 1066, in invoke
return process_result(sub_ctx.command.invoke(sub_ctx))
File “c:\users\sam\anaconda2\envs\catalyst\lib\site-packages\click\core.py”, line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File “c:\users\sam\anaconda2\envs\catalyst\lib\site-packages\click\core.py”, line 535, in invoke
return callback(*args, **kwargs)
File “c:\users\sam\anaconda2\envs\catalyst\lib\site-packages\click\decorators.py”, line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File "c:\users\sam\anaconda2\envs\catalyst\lib\site-packages\catalyst_main
.py", line 594, in ingest_exchange
csv=csv
File “c:\users\sam\anaconda2\envs\catalyst\lib\site-packages\catalyst\exchange\exchange_bundle.py”, line 823, in inges
t
show_report=show_report
File “c:\users\sam\anaconda2\envs\catalyst\lib\site-packages\catalyst\exchange\exchange_bundle.py”, line 642, in inges
t_assets
cleanup=True
File “c:\users\sam\anaconda2\envs\catalyst\lib\site-packages\catalyst\exchange\exchange_bundle.py”, line 420, in inges
t_ctable
shutil.rmtree(reader._rootdir)
File “c:\users\sam\anaconda2\envs\catalyst\lib\shutil.py”, line 247, in rmtree
rmtree(fullname, ignore_errors, onerror)
File “c:\users\sam\anaconda2\envs\catalyst\lib\shutil.py”, line 247, in rmtree
rmtree(fullname, ignore_errors, onerror)
File “c:\users\sam\anaconda2\envs\catalyst\lib\shutil.py”, line 256, in rmtree
onerror(os.rmdir, path, sys.exc_info())
File “c:\users\sam\anaconda2\envs\catalyst\lib\shutil.py”, line 254, in rmtree
os.rmdir(path)
WindowsError: [Error 145] The directory is not empty: u’C:\Users\Sam\.catalyst\data\exchanges\poloniex\temp_bundl
es\poloniex-minute-pasc_btc-2018-02\11\18’


#2

Thanks for posting. I was not able to reproduce this error, please try to delete the folders temp_bundles and minute_bundle that created during running the ingest command, then run the command:
catalyst ingest-exchange -x poloniex -f minute -i xrp_btc.

In addition, can you please give me more information about your code? and what are your Python and Catalyst versions?


#3

Thanks for your help.

So I tried to follow your suggestion. I didn’t delete the minute_bundle folder, because it took me a lot of time to get a reasonably complete data bundle, I just renamed it. I deleted the temp_bundles folder and run the command. It completed ok. But I’m not trading xrp_btc only, I need all the *_btc pairs on poloniex. So after that I run:
catalyst ingest-exchange -x poloniex -f minute
and got again the same kind of error. By running it enough times, I do eventually have a full ingestion without error. Something I find a bit strange is that if I run once again the command after a successful ingestion, it still seems to download data, hinting that maybe I didn’t have all the data despite the successful ingestion. And random errors may appear again.

Concerning my system:

  • Windows 7
  • Python 2.7.13
  • Conda 4.3.25
  • Catalyst 0.5.2
    Now I realize that I created the catalyst virtual environment with Python, not with Conda.

Thanks again,

Samuel


#4

Just to give an update. I rerun catalyst ingest-exchange -x poloniex -f minute and it succeeded. However, I have problems backtesting exactly in the same periods as mentioned in the first email. Maybe these are really gaps in the Poloniex minute dataset?


#5

I see you are using an old version of enigma-catalyst, try upgrading to 0.5.3


#6

It did solve the problem, as detailed there:

Thanks for the help!