Counters-0 queue is rising continuously

The counters-0 queue in the elasticache we are using is continuously increasing thus making the redis run out of memory. Ours is a high load environment almost getting 4k-6k events per minute.

Any suggestions around how we can stop the counters-0 queue from growing that fast would definitely help

You need more workers on this queue to process the load.

we have almost 80 worker pods (2cores and 2GB memory each) running, each allowed to create 4 subprocesses (I mean worker docker started with -c 4 option)

Can this unprecedented growth of counters-0 queue be due to the following:

I see that none of the events is getting aggregated into issues. I see a separate issue created on the UI for each event even if they should be bundled under the same issue. I triggered a few tag deletions 3 days back and since then we are not seeing any of the events aggregated.

Can this increase the counters queue to keep the count for each of these issues?

Here is a screenshot of what I am referring to:

Yeah, distinct groups would yield more traffic to counters queue.

You can run workers dedicated to a certain queue too that might help, doing sentry run worker -Q counters-0

Then this worker will only attach to this queue and not compete for other resources. You will likely get better utilization this way since the workload of counters-0 is very fast relative to other jobs being run, so running a small number of them dedicated to this should keep it chill. 5-6k per minute is a very small load, so this shouldn’t be any issue. You’ll probably be hurting your database first from write volume.

1 Like

Thanks for the suggestion. I would definitely like to try that.

However the problem of having separate issues for events is still there and it is not intentional from my side and this is causing the increase in the counters queue. This started happening since I deleted some tags from the projects in the first place. And I don’t know how to solve that now and in turn reduce the length of the counters queue.

@matt It would be really helpful if you could point me towards how to solve this.

I don’t really have any context of what’s going on, but grouping has nothing to do with tags. Grouping is based on properties on the event that’s submit. So there’s some other reason they’re not being grouped. And without having access to them, it’s hard to say why.

All I can likely say is they have differing grouphash values which is the table sentry_grouphash.