We’re testing sentry v10 by streaming a “production-like” amount of events into it. After some time, we’re this in snuba-consumer:
/var/lib/clickhouse/data/default/sentry_local/tmp_insert_20200113-90_1180_1180_0/browser.null.bin, errno: 28, strerror: No space left on device.
As far as I can see, snuba-consumer doesnt have any “external” volumes. Is there a process responsible for cleaning up this folder, that might not be working as it should?
So, I made sure snuba-cleanup runs (it does so now). It keeps logging this: Dropped 0 partitions....
I’m still seeting intermittend restarts from the snubaconsumer container tho, last logged line before it crashes is:
snuba.clickhouse.http.ClickHouseError: [76] DB::ErrnoException: Cannot open file /var/lib/clickhouse/data/default/sentry_local/tmp_insert_20200113-90_164_164_0/browser.null.bin, errno: 28, strerror: No space left on device
Was it not before? Is there anything we should fix in the default config?
Regarding this issue, our Snuba experts say “seems like Clickhouse ran out of disk space”. Now whether that’s expected or not is beyond me but you may wanna try giving it some more space and see what happens.