I have a Sentry setup built for high throughput (4 workers, 2 web nodes, Redis cluster and PostgresDB). Everything had been running fine for the last couple of months until recently when we started receiving more traffic. The worker logs are filled with these errors:
> @40000000587ce3af283622d4 15:15:49 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=‘e:d6af76ff95b24729b19706544a9c4783:3’)
@40000000587ce3b00283c6a4 15:15:50 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=‘e:1a6190c7916f4bbeac5ebb348c84a1d4:3’)
@40000000587ce3b00c47a37c 15:15:50 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=‘e:158fb3d1bae64843a7d4ec60d22d756b:3’)
@40000000587ce3b032c70fc4 15:15:50 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=‘e:12e0035d0f764376a79f207e7e5f0829:3’)
@40000000587ce3b109ca10bc 15:15:51 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=‘e:dcd660a305184a27a07770d123575de3:3’)
@40000000587ce3b11f968e34 15:15:51 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=‘e:7813a722e75743a48458b2a22da0c93f:3’)
@40000000587ce3b1378b98cc 15:15:51 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=‘e:5d1e6d3d8834481fa9e64d3e4816c449:3’)
@40000000587ce3b20a6ab5bc 15:15:52 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=‘e:5dcf971139134d108dd4b8c6cd16ca7d:3’)
@40000000587ce3b21c064b24 15:15:52 [ERROR] sentry.errors.events: preprocess.failed.empty (cache_key=‘e:fca3f0dd44784b01aacf674b7ab5a03a:3’)
Sentry is lagging behind by around an hour and losing some events as well. queues list shows: events.preprocess_event 114727. I thought the size of our cache instance could be the issue but even after bumping it’s size (10x capacity) the problem still persists.
Any ideas what could be causing this?
Any help much appreciated,
Christian