Snubaconsumer eating disk space

#1

We’re testing sentry v10 by streaming a “production-like” amount of events into it. After some time, we’re this in snuba-consumer:

/var/lib/clickhouse/data/default/sentry_local/tmp_insert_20200113-90_1180_1180_0/browser.null.bin, errno: 28, strerror: No space left on device.

As far as I can see, snuba-consumer doesnt have any “external” volumes. Is there a process responsible for cleaning up this folder, that might not be working as it should?

#2

There’s this snuba-cleanup cron that is supposed to run every 5 minutes: https://github.com/getsentry/onpremise/blob/dffd59723c05eb7a8a13f441a6cbb8b4e85fc437/docker-compose.yml#L112-L119

Maybe you can check its logs or try triggering it manually via docker-compose run --rm snuba-api cleanup

#3

got it, thanks!

1 Like
#4

So, I made sure snuba-cleanup runs (it does so now). It keeps logging this:
Dropped 0 partitions....

I’m still seeting intermittend restarts from the snubaconsumer container tho, last logged line before it crashes is:

snuba.clickhouse.http.ClickHouseError: [76] DB::ErrnoException: Cannot open file /var/lib/clickhouse/data/default/sentry_local/tmp_insert_20200113-90_164_164_0/browser.null.bin, errno: 28, strerror: No space left on device
#5

Was it not before? Is there anything we should fix in the default config?

Regarding this issue, our Snuba experts say “seems like Clickhouse ran out of disk space”. Now whether that’s expected or not is beyond me but you may wanna try giving it some more space and see what happens.

#6

I guess you might have the space out due to docker volume, who is created at docker’s default mount point.

try a df -h to check which mout point docker is using, and whether it is running out of space.

Or you may modify your docker-compose.yml to specify the volume in a different folder path with enough space such as:

  clickhouse:
    << : *restart_policy
    image: 'yandex/clickhouse-server:19.4'
    ulimits:
      nofile:
        soft: 262144
        hard: 262144
    volumes:
      - '/big-disk/sentry-clickhouse:/var/lib/clickhouse'
#7

thanks! I just go thrown that the error was at clickhouse, not snuba. Expanded the clickhouse volume now, will see how that runs.

just for info, we’re not using Sentry’s on-prem template/scripts directly, we’ve built a kubernetes-compatible setup based off of it.