We went from 8.11.0 -> 8.14.1 recently and have noticed that our workers simply can’t keep up with the events that we are getting. Normally we impose an organization and project limit and that has worked well, but now we don’t see any rate limiting happening on the stats page.
We can usually only handle about 70k, but the events are getting TTLd in the buffer and processing takes about an hour, which I assume is in sync with the TTL.
We noticed that the rate limiting changed, but we can only find one place to change this in the UI. We have it set to 500, but still no rate limiting.
And yet we are still not seeing any rate limiting:
We’ve also moved the quotas storage to a persistent redis instance away from the pure memory redis buffer instance, but that doesn’t seem to affect anything.
We used to see red lines on the stats page from all the rate limited events, but it seems to be gobbling up everything it possibly can.
Is there something else to check with the rate limiting?
Hey @matt, @HakShak
We’re noticing the same issue here on Sentry 8.13.0, we haven’t tried using config.yml, but we see the settings in the organization rate limit being applied. All the requests get a 2xx HTTP code and try to get processed.
