Events not shown in UI

Hi Team,

I have my onpremise set up version: 21.2.0 in kubernetes EKS. The kafka is supported by AWS Managed Stream for Kafka, postgresql by RDS and Redis by elastic cache.

The setup was working fine receiving and processing traffic, but suddenly after 7 days the UI is not showing the events and the UI looks like its a new installation with no traffic forwarded, though the traffic is being sent to relay. I looked for errors and there are no errors in any of the components everything looks normal.

relay logs:
2021-04-07T20:15:21Z [relay_server::actors::store] TRACE: Sending event item of envelope to kafka
2021-04-07T20:15:21Z [relay_server::actors::events] TRACE: sending envelope to kafka
2021-04-07T20:15:21Z [relay_server::actors::store] TRACE: Sending event item of envelope to kafka
2021-04-07T20:15:21Z [relay_server::actors::outcome::processing] TRACE: handling outcome
2021-04-07T20:15:21Z [relay_server::actors::outcome::processing] TRACE: handling outcome
2021-04-07T20:15:21Z [relay_server::actors::outcome::processing] TRACE: handling outcome
2021-04-07T20:15:21Z [relay_server::actors::outcome::processing] TRACE: handling outcome

sentry-ingest-consumer logs:
20:18:16 [DEBUG] batching-kafka-consumer: Committed offsets: [TopicPartition{topic=ingest-
attachments,partition=0,offset=-1001,error=None}, TopicPartition{topic=ingest-
events,partition=0,offset=8154676,error=None}, TopicPartition{topic=ingest-
transactions,partition=0,offset=-1001,error=None}]
20:18:16 [DEBUG] batching-kafka-consumer: Kafka offset commit took 9ms
20:18:16 [DEBUG] batching-kafka-consumer: Resetting in-memory batch
20:18:16 [INFO] batching-kafka-consumer: Flushing 100 items (from {(‘ingest-events’, 0): [8154676,
8154775]}): forced:False size:True time:False
20:18:16 [DEBUG] batching-kafka-consumer: Flushing batch via worker

Question:
From the issue 502 I observed that the _consumer_offsets topic in AWS Managed Kafka runs with 50 partitions. Could this be the reason for the issue? Kindly guide.

I’d check all other consumers first and see if any of them are stuck.

I’m guessing you mean no new events show up. If you lost your old events too, I’d suspect a data loss on Clickhouse.