I was running sentry 7.1.1 on premise on an AWS EC2 instance.
We appeared to run out of disk space, preventing sentry from running, so first tried a sentry clean, however discovered that on our version of sentry, it would run out of memory before removing any actual records (we’d would have had at least 3 years of data there).
So upgraded to 7.4.3 (via pip install -U sentry==7.4.3 then sentry upgrade), and ran sentry clean successfully. We didn’t really reclaim any disk space so then ran a vacuum full from psql.
This gave us 5GB back however now we don’t appear to be picking up any new tasks and in the worker logs, we now just see an endless stream of the following sort of log:
[2018-06-12 01:45:56,405: WARNING/MainProcess] Skipping revoked task: sentry.tasks.post_process.plugin_post_process_group[7ff25d11-c0bc-49c4-8e74-26d12cd2bd49] with the occasional: [2018-06-07 06:54:00,861: ERROR/PoolWorker-2] Data not available in preprocess_event (cache_key=e:86a3ac3271f440ae9b179f78a9edaf02). I believe I saw the event_id of the event I created manually come through as this type of error.
Is this expected behaviour? Can I check the task queue to see if events are actually being processed, or what would you suggest for troubleshooting here?