We are running on premise sentry 21.2.0 on a bare-metal Kubernetes Cluster since about February this year. Everything seemed to work fine but we are now facing an issue that new events don’t show up in the GUI. We were aware of an failure since yesterday since some test events didn’t show up. We then saw that two of the snuba consumer pods failed (CrashLoopBack) (I’m sorry that I don’t remember which of them failed exactly). After running snuba migrations the two pods came up and running again. But none of the new events are displayed in the gui (we are still receiving alters via mail or our chat system via webhook).
I’ve already tried to reset the consumer-groups and topics offset as stated in https://github.com/getsentry/onpremise/issues/478#issuecomment-666254392 - but still no luck. The newest events are 6 days old (about the date that the 2 consumer pods failed).
Is there anything we can do to bring our events back to be displayed in the GUI?
Any help is appreciated - thanks in advance!
How does your relay and kafka services look like?
Hi Burak,
thanks for your response.
Not really sure about our Kafka but we had a performance issue (Performance Sentry API - on Premise, Kubernetes - #2 by chadwhitacre) shortly before this issue, and I suppose this was Kafka related.
What I did now (since our Devs were already a little impatient) was to completely nuke the Kafka Volume (and Zookeeper of course) and, on this occasion, I upgraded to 21.3.1 (will do further upgrades when I have a little more time). So we lost the events from (I suppose) the last week, but that’s quite ok, since our Sentry is running as supposed again.
A lot of efforts has been made in the self hosted documentation (https://develop.sentry.dev/self-hosted/), I appreciate this very much, but what I’m still missing is some documentation about the Kafka Setup (which topics are used, which consumer group is responsible for which topic and so on…). I think this could be really helpful for debugging Kafka Issues… maybe there is someone really familiar (community or Sentry dev maybe?) with the Sentry Kafka setup and could complete the documentation?
(Marked as solution - bc: nuked Kafka/Zookeeper Volume)
I think once we resolve Support Kafka topic configuration migration/synchronization. · Issue #857 · getsentry/self-hosted · GitHub, it would provide some self-documentation around this. We also have some plans around explaining our ingestion pipeline better both for our own engineers and self-hosted operators but no commitments on the timeline yet.
Glad that you were able to overcome the issue and thanks a lot for the feedback!
This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.