Gaps in Bitfinex minute data stream?


#1

While backtesting our algorithm, we detected some missing values in the bitfinex minute data stream. To be fair, the data is not exactly missing, but it seems like the values are not up-to-date for a few days/weeks.

Identified periods so far (bitfinex, btc_usd, minute):
04/01/2016 00:01am - 06/01/2016 00:00am - price is always 416.19
08/02/2016 06:03pm - 08/10/2016 03:53pm - price is always 604.1
01/19/2018 00:01am - 01/20/2018 00:00am - price is always 10,975

We eliminated the problem on our side, because the same issue occurs running the first sample algorithm from the beginner tutorial.
Are we missing something? Thanks for help!


#2

Hi, I also observed this. Can anyone from the Enigma team confirm that this is a problem with the catalyst database? Will this be fixed?

Thanks


#3

Hi,

Thanks @Thomas and @svedankae for reporting this.
We will check this. Could you please open a Github issue to continue our investigation there?

Thanks,
Lena


#4

@Thomas, I was able to reproduce and see your first two observations:

04/01/2016 00:01am - 06/01/2016 00:00am - price is always 416.19
08/02/2016 06:03pm - 08/10/2016 03:53pm - price is always 604.1

Regarding the last time interval, I wasn’t able to see what you’ve experienced.


#5

Hi Lena,

see the attached graph for the last time interval.
Maybe some others could check as well to see if it’s just a local or a general problem?

Cheers
Jan


#6

@Thomas, @svedankae You can see from the chart below from Bitfinex that the second interval is correct, meaning Bitfinex has a flat line for the following interval:

08/02/2016 06:03pm - 08/10/2016 03:53pm - price is always 604.1

So Catalyst data is consistent with that from the exchange.

I’m investigating the other intervals…


#7

@Thomas, @svedankae, @Jan
Data for the third interval is indeed there:

01/19/2018 00:01am - 01/20/2018 00:00am - price is always 10,975

You are looking at data from an old bundle. We did have an issue with bundling data around that time, but it was fixed more than a month ago, however because data ingestion is incremental to save bandwidth, posterior data was appended to your local copies without overriding the missing data. In order to fix this third interval (which applies to all minute data for all coins at Poloniex and Bitfinex) you have to clean up and re-ingest:

catalyst clean-exchange -x bitfinex -f minute catalyst clean-exchange -x poloniex -f minute

So out of the 3 intervals reported on the first post on this thread, I can confirm that only the first one presents a real mismatch between Catalyst and data on the exchange, and will take a bit longer to fix. We’re looking into it.


#8

Hi Victor -

thank you very much for looking into this and answering the question. The clean up and re-ingest tip is helpful! Can you let us know when the “real mismatch” is fixed?

Thanks,

Sven


#9

Hi Victor,

I followed your advide and can confirm that it solved the problem for the missing data on 01/19/2018.
Thanks!

Jan


#10

Thank you very much!
I just opened a Github issue just for the first interval.


#11

Hi Victor and everyone else who might be reading.

We found another gap in the data stream: In May 2015. The screenshot with the white background shows the graph created with catalyst data. I have checked on Bitfinex itself and the data is there.

Could you please investigate and if possible fix the catalyst data for that timeframe as well (in addition to the other timeframe identified in the prior posts) ? Many thanks.


#12

Hi @svedankae,

I have checked May 2015 and saw the phenomena you have described, thanks for catching and reporting this!
We will address this as well.
I will update the Github issue @Thomas has opened with this information.

Thanks,
Lena