Cleanup doesn't work

On-Premise
1 
# sentry cleanup --days 30
12:39:35 [WARNING] sentry.utils.geo: settings.GEOIP_PATH_MMDB not configured.
12:39:38 [INFO] sentry.plugins.github: apps-not-configured
Removing expired values for LostPasswordHash
Removing expired values for OrganizationMember
Removing expired values for ApiGrant
Removing expired values for ApiToken
Removing expired files associated with ExportedData
Removing old NodeStore values

…
5 hours later. the process is frozen

[root@tntp-sentry-201 ~]# ps auxfww | grep clean
root       453  0.0  0.0      0     0 ?        S<   May08   0:00  \_ [kvm-irqfd-clean]
root     18316  0.0  1.3 265280 104984 pts/1   S+   15:39   0:05  |   |       \_ /usr/local/bin/python /usr/local/bin/sentry cleanup --days 30
root     18327  0.0  0.3  62860 26888 pts/1    S+   15:39   0:00  |   |           \_ /usr/local/bin/python /usr/local/bin/sentry cleanup --days 30
root      2926  0.0  0.0 112704   968 pts/1    S+   20:00   0:00          \_ grep --color=auto clean
[root@tntp-sentry-201 ~]#
[root@tntp-sentry-201 ~]#
[root@tntp-sentry-201 ~]# ps -eo pid,pcpu,pmem,comm,status,wchan | grep 18316
18316  0.0  1.3 /usr/local/bin/      - poll_schedule_timeout
[root@tntp-sentry-201 ~]#
[root@tntp-sentry-201 ~]#
[root@tntp-sentry-201 ~]# ps -eo pid,pcpu,pmem,comm,status,wchan | grep 18327
18327  0.0  0.3 sentry               - pipe_wait
[root@tntp-sentry-201 ~]#
[root@tntp-sentry-201 ~]#
[root@tntp-sentry-201 ~]# cat /proc/18316/status | grep voluntary
voluntary_ctxt_switches:	890
nonvoluntary_ctxt_switches:	175
[root@tntp-sentry-201 ~]#
[root@tntp-sentry-201 ~]#
[root@tntp-sentry-201 ~]# cat /proc/18316/status | grep voluntary
voluntary_ctxt_switches:	891
nonvoluntary_ctxt_switches:	175
[root@tntp-sentry-201 ~]#
[root@tntp-sentry-201 ~]#
[root@tntp-sentry-201 ~]# cat /proc/18327/status | grep voluntary
voluntary_ctxt_switches:	1
nonvoluntary_ctxt_switches:	0
[root@tntp-sentry-201 ~]#

postgresql db size is 100GB

2

What is your question here exactly? Seems like it is taking a lot of time to remove the old NodeStore values? And that’s why this process is separated from the regular Celery workers?

3

Why didn’t “sentry cleanup --days 30” return a result in 5 hours?
Can you please tell me how to delete old data from storage?

4

As a result:

0 1 * * * root cd <path_to_compose> && { time docker-compose run --rm worker cleanup --days 30; } &> /var/log/cleanup.log
0 8 * * * root { time docker exec -i $(docker ps --format "table {{.Names}}"|grep postgres) vacuumdb -U postgres -d postgres -v -f --analyze; } &> /var/log/vacuumdb.log
1 Like
5

Is cleaning up with a cronjob still needed?
We have the worker cleanup --days 30 active on Sentry 9.0.0 but moving to 20.12.1 (and higher) I wonder if it is setup in a different way.
I think it is: in the onpremise README.md I see:

Sentry comes with a cleanup cron job that prunes events older than 90 days by default.
and the SENTRY_EVENT_RETENTION_DAYS to be set in .env.

And how about the vacuuming of Postgres?

6

Both of them are required, yes.

7

Thanks, Burak!

So it’s not that Docker is taking care of this and one still needs system cronjobs just as in Sentry sentry:8.12-onbuild or 9.0.0, right?

Is this documented somewhere?
I am unable to find it in the GitHub getsentry/onpremise repo, nor in https://develop.sentry.dev/self-hosted/

Hope you can point us into the right direction. :innocent:

1 Like
8

Ah, sorry for the confusion. We do take care of this for you: https://github.com/getsentry/onpremise/blob/659e03e9cf7f664d843815a3d57498ba7073c272/docker-compose.yml#L205

So if you use the onpremise repo, you are all covered.