We run one instance of sentry on premise 20.8.0 for applications from different teams. We limit the number of events per minute, but sometimes one application puts so much error events, that there is the potential to loose the events of other applications.
Therefore i would like to push the stats about rejected, received and blacklisted events per project to our time series databases so i configure alerting for this stats.
The http api from sentry offeres this information but for a large number of projects it is inconvenient and slow because it results in a lot of http requests.
Is there any faster possibility to grep this stats? I did not find any of this stats in kafka or redis.
How do you monitor your sentry on premise instance?
You’ll particularly be interested in the following metrics described on the above page:
event.accepted
event.rejected
events.outcomes (logs more reasons for rejected events):
The documentation outcomes isn’t great at the moment, but the reported tag values are pretty self-explanatory: filtered, rate_limited, invalid and abuse.
Sorry, I should’ve read your question more carefully. We’re not tagging the organization or project on those metrics. In order to get stats per project or organization, you have a few options:
Consume the organization stats and project stats endpoints. You can see an example for this when you navigate to “Stats” in the main page.
Snuba uses the outcomes dataset, which is populated from the outcomes Kafka topic. The messages are JSON payloads containing organization, project, DSN key, outcome, and reason.
Considering you would like to process a large number of projects, my suggestion would be to write a Kafka consumer for the outcomes topic. For example, have a look at Snuba’s own consumer.