Underperformance detection approaches

Is there any simple or not so simple method of detecting the exactt place where on-premise sentry stack is underperforming ?

There are several indirect signs:

  • redis database growing
  • event stopped flowing

but are there any direct ones, which can tell “okay, this means sentry consumers lacking performance” or “okay, this means that snuba is dmanding some scaling” etc ? I’m aware only of “sentry queues list”, but this is showing only zeroes, which I treat like “everything is just perfect”. New 21.x sentry graphs include performance ones, but from my point of view these are merey observational.

I’d look at ingestion delay as a high-level metric: the time between an event is received and it shows up on the API.

Maybe we have some better metrics that @jasonious can point us to.