We’re testing sentry v10 by streaming a “production-like” amount of events into it. After some time, we’re this in snuba-consumer: /var/lib/clickhouse/data/default/sentry_local/tmp_insert_20200113-90_1180_1180_0/browser.null.bin, errno: 28, strerror: No space left on device. As far as I can see, s…

There’s this snuba-cleanup cron that is supposed to run every 5 minutes: https://github.com/getsentry/onpremise/blob/dffd59723c05eb7a8a13f441a6cbb8b4e85fc437/docker-compose.yml#L112-L119 Maybe you can check its logs or try triggering it manually via docker-compose run --rm snuba-api cleanup

So, I made sure snuba-cleanup runs (it does so now). It keeps logging this: Dropped 0 partitions.... I’m still seeting intermittend restarts from the snubaconsumer container tho, last logged line before it crashes is: snuba.clickhouse.http.ClickHouseError: [76] DB::ErrnoException: Cannot open fil…

[image] trondhindenes: So, I made sure snuba-cleanup runs (it does so now). Was it not before? Is there anything we should fix in the default config? Regarding this issue, our Snuba experts say “seems like Clickhouse ran out of disk space”. Now whether that’s expected or not is beyond me but…

I guess you might have the space out due to docker volume, who is created at docker’s default mount point. try a df -h to check which mout point docker is using, and whether it is running out of space. Or you may modify your docker-compose.yml to specify the volume in a different folder path with …

thanks! I just go thrown that the error was at clickhouse, not snuba. Expanded the clickhouse volume now, will see how that runs. just for info, we’re not using Sentry’s on-prem template/scripts directly, we’ve built a kubernetes-compatible setup based off of it.

Snubaconsumer eating disk space

On-Premise

trondhindenes January 16, 2020, 3:47pm 3

got it, thanks!

Topic		Replies	Views
Snuba cleanup transactions dropped 0 partitions	0	1519	November 23, 2021
Cant get email alert anymore On-Premise	6	1406	December 12, 2020
Sentry not logging events On-Premise	1	4141	July 13, 2020
Events have stopped appearing On-Premise	5	3125	January 3, 2021
ERROR: for clickhouse Cannot create container for service clickhouse On-Premise	1	1933	April 11, 2021

Snubaconsumer eating disk space

Related topics