Snubaconsumer eating disk space

We’re testing sentry v10 by streaming a “production-like” amount of events into it. After some time, we’re this in snuba-consumer:

/var/lib/clickhouse/data/default/sentry_local/tmp_insert_20200113-90_1180_1180_0/browser.null.bin, errno: 28, strerror: No space left on device.

As far as I can see, snuba-consumer doesnt have any “external” volumes. Is there a process responsible for cleaning up this folder, that might not be working as it should?

There’s this snuba-cleanup cron that is supposed to run every 5 minutes:

Maybe you can check its logs or try triggering it manually via docker-compose run --rm snuba-api cleanup

got it, thanks!

1 Like

So, I made sure snuba-cleanup runs (it does so now). It keeps logging this:
Dropped 0 partitions....

I’m still seeting intermittend restarts from the snubaconsumer container tho, last logged line before it crashes is:

snuba.clickhouse.http.ClickHouseError: [76] DB::ErrnoException: Cannot open file /var/lib/clickhouse/data/default/sentry_local/tmp_insert_20200113-90_164_164_0/browser.null.bin, errno: 28, strerror: No space left on device

Was it not before? Is there anything we should fix in the default config?

Regarding this issue, our Snuba experts say “seems like Clickhouse ran out of disk space”. Now whether that’s expected or not is beyond me but you may wanna try giving it some more space and see what happens.

I guess you might have the space out due to docker volume, who is created at docker’s default mount point.

try a df -h to check which mout point docker is using, and whether it is running out of space.

Or you may modify your docker-compose.yml to specify the volume in a different folder path with enough space such as:

    << : *restart_policy
    image: 'yandex/clickhouse-server:19.4'
        soft: 262144
        hard: 262144
      - '/big-disk/sentry-clickhouse:/var/lib/clickhouse'

thanks! I just go thrown that the error was at clickhouse, not snuba. Expanded the clickhouse volume now, will see how that runs.

just for info, we’re not using Sentry’s on-prem template/scripts directly, we’ve built a kubernetes-compatible setup based off of it.