Alert when Background workers haven't checked in recently

It would be nice to receive some sort of notification (Email or Slack notification) when such happens because right now I can only know if such error happened if I’m a staff account AND if I logout and login (OR if I see that there are zero events in the last hours)

1 Like

I agree, unfortunately we generally send emails using celery (and do this check only when a page is loaded)

mmm, a webhook could be viable? (not sure if celery is used for everything)

For now I’m just calling a discover query every 30 minutes to see if there are events in the last hour, not a elegant solution but it’s not something “real time”.

There’s a sentry queues list command that you can use via docker-compose run:

docker-compose run --rm web sentry queues list

Also, there are a bunch of metrics we collect and you can set a metrics backend via the SENTRY_METRICS_BACKEND setting (pick one from https://github.com/getsentry/sentry/tree/master/src/sentry/metrics) and get this information and more I think.

See https://github.com/getsentry/sentry/blob/fbe9d3b0d6934282f9443f2d201b6df25391d885/src/sentry/utils/metrics.py for the metrics helpers (and search for them to find what metrics sentry collect)

3 Likes

Thanks BYK and Untitaker

1 Like