Digging in further, we have found the following:
We found this was happening due to the requests hitting timeouts. We started by raising the sentry-web timeout itself from the default 30 seconds to 300. This seemed to help tremendously once actually getting logged into the interface if you could get there. Unfortunately we were still seeing a lot of 504 errors.
This was due to the nginx proxy that sits in front of the sentry-web instances having its own timeout of 60 seconds. After raising this one as well, it most certainly helped the ability to login but not the speed and there are still 502 errors occurring. I believe this is due to the instability of the sentry-web instances and the resources they are demanding. Ultimately they are running away with memory usage until eventually the request dies out or k8s evicts the pod because it using too many resources.
Memory usage for sentry-web skyrocketing - note that this is only one instance at the moment but when multiple users are trying to access the gui this will multiply.
We then brought up another Sentry instance, with no data, to compare. This is the production login:
and this is the non-production login:
We enabled APM tracing to see what was happening and we see the query that seems to be taking a long while:
We then ran that query against the postgres (in RDS) database:
and you can see it is 25 million rows and just over 1GB of data.
Admittedly, we have a lot of events, I’m just trying to understand how we might make this a better experience for our users.