I’m running an on-premise installation of Sentry. I have Postgres running on a separate server. The Postgres box has a 300GB disk for storing the data.
My Postgres box recently ran out of disk space. When it did, Sentry was unable to connect to the database. I cleared out a few GBs of space by getting rid of some old logs, and tried to execute “sentry cleanup”. The cleanup process ran for a few hours, and then failed, saying that the device is out of space. The space I’d cleared on the DB server was now filled up.
I’ve tried running cleanup several times now, but each time the disk usage on the DB box seems to increase, rather than remain constant (since Postgres does not release the space to OS).
Here’s the output of the cleanup operation:
Removing expired values for LostPasswordHash Removing old NodeStore values Removing GroupRuleStatus for days=30 project=* Removing GroupTagValue for days=30 project=* Removing TagValue for days=30 project=* Removing GroupEmailThread for days=30 project=* Removing expired values for EventMapping Cleaning up unused FileBlob references File Blobs: 100% |###############| Time: 0:00:00 Removing Event for days=30 project=*
This output is typically followed by an error after a few hours, saying that the device has run out of disk space, and any space I’d previously freed up on my DB box gets used up.
I don’t think it’s the DB size that’s increasing. This is the error I usually get:
django.db.utils.OperationalError: could not access status of transaction 0 DETAIL: Could not write to file "pg_subtrans/5A78" at offset 221184: No space left on device.
I now have only one more GB that I can free up on the DB server, and I don’t think I should let that get used up.
I’m unable to run VACUUM FULL due to the lack of disk space.
Now, I really should’ve set up the cleanup as a cron job, but I wasn’t expecting this much data.
Please advise on how I can get out of this mess I’ve got myself into.
Worst case scenario, I’m thinking about dropping the entire database and creating a new one. I’ve exported sentry data using “sentry export”. Once I set up a new DB, I’ll use “sentry import” to get all the necessary data in place. I wish to avoid this option if possible, since this will result in the loss of all data ingested so far.
Thanks in Advance.