No Rate Limiting after upgrade


#1

We went from 8.11.0 -> 8.14.1 recently and have noticed that our workers simply can’t keep up with the events that we are getting. Normally we impose an organization and project limit and that has worked well, but now we don’t see any rate limiting happening on the stats page.

We can usually only handle about 70k, but the events are getting TTLd in the buffer and processing takes about an hour, which I assume is in sync with the TTL.

We noticed that the rate limiting changed, but we can only find one place to change this in the UI. We have it set to 500, but still no rate limiting.

Any ideas?

Thanks


#2

So we changed the rate limiting in the conf just to be sure there wasn’t a bug with this UI:

Which is confirmed by the info mouse over:

The project settings rate limit confirms this:

And yet we are still not seeing any rate limiting:

We’ve also moved the quotas storage to a persistent redis instance away from the pure memory redis buffer instance, but that doesn’t seem to affect anything.

We used to see red lines on the stats page from all the rate limited events, but it seems to be gobbling up everything it possibly can.

Is there something else to check with the rate limiting?


#3

Hmmmm, I’m going to look into this today and see if we broke something here. Seems suspicious.


#4

@matt any signs of a bug there or is this just a bad environment? Thanks for taking a look.


#5

Any luck? We are getting some record numbers now :slight_smile:


#6

I have spend exactly 0 efforts looking into this so far. :frowning:


#7

Hey @matt, any spare time for insights here?


#8

Hey @matt, @HakShak
We’re noticing the same issue here on Sentry 8.13.0, we haven’t tried using config.yml, but we see the settings in the organization rate limit being applied. All the requests get a 2xx HTTP code and try to get processed.


#9

Added an issue here in hopes it gets on a list somewhere :smiley:

https://github.com/getsentry/sentry/issues/5227


#10

Just want to chime in and add that system.rate-limit has no effect in version 8.20