Sentry not logging issues

On-Premise
1

Hi,
Our on-premise sentry was running fine for some time until one-day server disk space was full. So one of our developer ran docker volume prune and that reclaimed 9 GB of space by removing kafka stuff (Don’t know what it was). Now the error which is shown in docker logs sentry_onpremise_worker_1 is

: sentry.eventstream.kafka.backend: Could not publish message (error: KafkaError{code=_MSG_TIMED_OUT,val=-192,str=“Local: Message timed out”}): <cimpl.Message object at 0x7fb4ae3ca5d0>

And kafka container says: sentry_onpremise_kafka_1 : [2020-05-26 12:14:07,334] ERROR [Controller id=1002 epoch=50] Controller 1002 epoch 50 failed to change state for partition __consumer_offsets-40 from OfflinePartition to OnlinePartition (state.change.logger)
kafka.common.StateChangeFailedException: Failed to elect leader for partition __consumer_offsets-40 under strategy OfflinePartitionLeaderElectionStrategy
at kafka.controller.PartitionStateMachine$$anonfun$doElectLeaderForPartitions$3.apply(PartitionStateMachine.scala:366)
at kafka.controller.PartitionStateMachine$$anonfun$doElectLeaderForPartitions$3.apply(PartitionStateMachine.scala:364)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at kafka.controller.PartitionStateMachine.doElectLeaderForPartitions(PartitionStateMachine.scala:364)
at kafka.controller.PartitionStateMachine.electLeaderForPartitions(PartitionStateMachine.scala:292)
at kafka.controller.PartitionStateMachine.doHandleStateChanges(PartitionStateMachine.scala:210)
at kafka.controller.PartitionStateMachine.handleStateChanges(PartitionStateMachine.scala:133)
at kafka.controller.PartitionStateMachine.triggerOnlinePartitionStateChange(PartitionStateMachine.scala:123)
at kafka.controller.PartitionStateMachine.triggerOnlinePartitionStateChange(PartitionStateMachine.scala:109)
at kafka.controller.PartitionStateMachine.startup(PartitionStateMachine.scala:66)
at kafka.controller.KafkaController.onControllerFailover(KafkaController.scala:260)
at kafka.controller.KafkaController.kafka$controller$KafkaController$$elect(KafkaController.scala:1221)
at kafka.controller.KafkaController$Startup$.process(KafkaController.scala:1134)
at kafka.controller.ControllerEventManager$ControllerEventThread$$anonfun$doWork$1.apply$mcV$sp(ControllerEventManager.scala:89)
at kafka.controller.ControllerEventManager$ControllerEventThread$$anonfun$doWork$1.apply(ControllerEventManager.scala:89)
at kafka.controller.ControllerEventManager$ControllerEventThread$$anonfun$doWork$1.apply(ControllerEventManager.scala:89)
at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:31)
at kafka.controller.ControllerEventManager$ControllerEventThread.doWork(ControllerEventManager.scala:88)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82)
^C
.

Can someone please help out on this.

2

You probably need to remove other Kafka volumes and the Zookeeper volume to reset them fully.

3

I tried removing and making new volume, it didn’t help altough here is another error in sentry_onpremise_post-process-forwarder_1: ERROR|rdkafka#consumer-2| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#192.168.0.10:9092 failed: Connection refused
%3|1590558013.314|ERROR|rdkafka#consumer-2| [thrd:kafka:9092/bootstrap]: 1/1 brokers are down.
Im unable to solve it anyhelp will be appreciated.

Thanks.

4

That tells us that your kafka service is not running. Check that and your zookeeper service first to ensure they are up and running.