Hiya! We recently started running Sentry on-premise and are experiencing a large backlog of events.process_event and events.save_event. Setup Sentry 20.9.0bb3d590 Azure machine with 4 cpus and 16 GB of memory Traffic from 2 - 150 incoming events per minute All services running on machine except…

Thanks for the response @BYK ! I did add the workers already based on that thread (you can see the config in the initial post), I also added one for save_event. This doesn’t really seem to be making a dent in the queue. The Clickhouse max memory flag is also set to 0.3. Did you mean something else f…

[image] yo-gi: (you can see the config in the initial post) Ooops, sorry for zombie responding :smiley: [image] yo-gi: The Clickhouse max memory flag is also set to 0.3. This may actually be limiting you so you may want to try increasing it a bit. Also how do your worker logs look li…

re: Clickhouse, I’ll try that! All the worker logs look like this. I don’t see any errors or warnings that aren’t these worker5_1 | 2020-10-19T20:22:38.193458842Z InsecureRequestWarning) worker5_1 | 2020-10-19T20:23:12.644595224Z /usr/local/lib/python2.7/…

Those errors might be the reason for your issues. Are you using Sentry with SSL? Do you have any custom config? If so can you share that with us?

We are using Sentry with SSL through an nginx ingress (managed by Kubernetes) that sits in front of this. When I purge the queues, events do get processed for a few hours before they get backlogged again. How should I address these errors?

One other thing is that setting up a worker to ingest events.save_event tasks doesn’t work. This is the config, and I don’t see any events being ingested in the logs, whereas our workers that ingest events.process_event have processing logs. worker6: << : *sentry_defaults command: run wor…

I don’t have any pointers or clues right now. Were you able to solve this? If not, sharing full logs may help with finding a solution.

Large backlog of events.process_event and events.save_event

On-Premise

BYK October 19, 2020, 8:09pm 2

I think this thread can help you: How to clear backlog and monitor it

Topic		Replies	Views
Not processing events On-Premise	8	1834	September 28, 2020
Sentry: 429 errors On-Premise	7	3197	September 29, 2020
How to clear backlog and monitor it On-Premise	20	14962	January 14, 2021
Sentry consuming all CPU	3	5241	November 29, 2019
Sentry stopped logging after spike? On-Premise	3	5423	July 26, 2021

Large backlog of events.process_event and events.save_event

Related topics