Kafka Error after Upgrading

cwang · December 8, 2019, 8:50am

After grading from commit 5d064c to commit 89e80 gives this error.

worker_1 | %3|1575794821.759|FAIL|rdkafka#producer-2| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#192.168.64.10:9092 failed: Connection refused
worker_1 | %3|1575794821.759|FAIL|rdkafka#producer-2| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#192.168.64.10:9092 failed: Connection refused
worker_1 | %3|1575794821.759|ERROR|rdkafka#producer-2| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#192.168.64.10:9092 failed: Connection refused
worker_1 | %3|1575794821.759|ERROR|rdkafka#producer-2| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#192.168.64.10:9092 failed: Connection refused
worker_1 | %3|1575794827.761|FAIL|rdkafka#producer-2| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Failed to resolve ‘kafka:9092’: Name or service not known
worker_1 | %3|1575794827.761|ERROR|rdkafka#producer-2| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Failed to resolve ‘kafka:9092’: Name or service not known
worker_1 | %3|1575794827.762|FAIL|rdkafka#producer-2| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Failed to resolve ‘kafka:9092’: Name or service not known

Logs from kafka

kafka_1 | [2019-12-08 08:51:34,121] ERROR [KafkaServer id=3130] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
kafka_1 | org.apache.kafka.common.KafkaException: Found directory /var/lib/kafka/data/data, ‘data’ is not in the form of topic-partition or topic-partition.uniqueId-delete (if marked for deletion).
kafka_1 | Kafka’s log directories (and children) should only contain Kafka topic data.
kafka_1 | at kafka.log.Log$.exception$1(Log.scala:2265)
kafka_1 | at kafka.log.Log$.parseTopicPartitionName(Log.scala:2272)
kafka_1 | at kafka.log.LogManager.kafka$log$LogManager$$loadLog(LogManager.scala:260)
kafka_1 | at kafka.log.LogManager$$anonfun$loadLogs$2$$anonfun$11$$anonfun$apply$15$$anonfun$apply$2.apply$mcV$sp(LogManager.scala:345)
kafka_1 | at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:63)
kafka_1 | at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
kafka_1 | at java.util.concurrent.FutureTask.run(FutureTask.java:266)
kafka_1 | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
kafka_1 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
kafka_1 | at java.lang.Thread.run(Thread.java:748)

Update:

I changed

volumes:
- ‘sentry-kafka:/var/lib/kafka/data’

to

volumes:
- ‘sentry-kafka:/var/lib/kafka’

Everything is working fine now

BYK · December 9, 2019, 7:22pm

If you look at the diff here https://github.com/getsentry/onpremise/compare/5d064c..89e80#diff-4e5e90c6228fd48698d074241c2ba760R92 it shows the change you had to adjust, yes.

I’d rather you migrate your sentry-kafka volume or wipe it: docker volume rm sentry-kafka && docker volume create sentry-kafka. Note that you may lose some events if you wipe it so if you chose to migrate, you can do something like

docker run --rm -it -v sentry-kafka:/kafka alpine ash -c \
  "mv /kafka/data/* /kafka; rm -rf /kafka/data"

cwang · December 10, 2019, 4:08am

After wipe and recreate volume we got the following error

kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 kafka_1 | [2019-12-10 04:05:18,245] ERROR [Controller id=3150 epoch=5] Controller 3150 epoch 5 failed to change state for partition __consumer_offsets-1 from OfflinePartition to OnlinePartition (state.change.logger)
| kafka.common.StateChangeFailedException: Failed to elect leader for partition __consumer_offsets-1 under strategy OfflinePartitionLeaderElectionStrategy
| at kafka.controller.PartitionStateMachine$$anonfun$doElectLeaderForPartitions$3.apply(PartitionStateMachine.scala:366)
| at kafka.controller.PartitionStateMachine$$anonfun$doElectLeaderForPartitions$3.apply(PartitionStateMachine.scala:364)
| at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
| at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
| at kafka.controller.PartitionStateMachine.doElectLeaderForPartitions(PartitionStateMachine.scala:364)
| at kafka.controller.PartitionStateMachine.electLeaderForPartitions(PartitionStateMachine.scala:292)
| at kafka.controller.PartitionStateMachine.doHandleStateChanges(PartitionStateMachine.scala:210)
| at kafka.controller.PartitionStateMachine.handleStateChanges(PartitionStateMachine.scala:133)
| at kafka.controller.PartitionStateMachine.triggerOnlinePartitionStateChange(PartitionStateMachine.scala:123)
| at kafka.controller.PartitionStateMachine.triggerOnlinePartitionStateChange(PartitionStateMachine.scala:109)
| at kafka.controller.PartitionStateMachine.startup(PartitionStateMachine.scala:66)
| at kafka.controller.KafkaController.onControllerFailover(KafkaController.scala:260)
| at kafka.controller.KafkaController.kafka$controller$KafkaController$$elect(KafkaController.scala:1221)
| at kafka.controller.KafkaController$Startup$.process(KafkaController.scala:1134)
| at kafka.controller.ControllerEventManager$ControllerEventThread$$anonfun$doWork$1.apply$mcV$sp(ControllerEventManager.scala:89)
| at kafka.controller.ControllerEventManager$ControllerEventThread$$anonfun$doWork$1.apply(ControllerEventManager.scala:89)
| at kafka.controller.ControllerEventManager$ControllerEventThread$$anonfun$doWork$1.apply(ControllerEventManager.scala:89)
| at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:31)
| at kafka.controller.ControllerEventManager$ControllerEventThread.doWork(ControllerEventManager.scala:88)
| at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82)
| [2019-12-10 04:05:18,246] ERROR [Controller id=3150 epoch=5] Controller 3150 epoch 5 failed to change state for partition __consumer_offsets-5 from OfflinePartition to OnlinePartition (state.change.logger)
| kafka.common.StateChangeFailedException: Failed to elect leader for partition __consumer_offsets-5 under strategy OfflinePartitionLeaderElectionStrategy
| at kafka.controller.PartitionStateMachine$$anonfun$doElectLeaderForPartitions$3.apply(PartitionStateMachine.scala:366)
| at kafka.controller.PartitionStateMachine$$anonfun$doElectLeaderForPartitions$3.apply(PartitionStateMachine.scala:364)
| at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
| at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
| at kafka.controller.PartitionStateMachine.doElectLeaderForPartitions(PartitionStateMachine.scala:364)
| at kafka.controller.PartitionStateMachine.electLeaderForPartitions(PartitionStateMachine.scala:292)
| at kafka.controller.PartitionStateMachine.doHandleStateChanges(PartitionStateMachine.scala:210)
| at kafka.controller.PartitionStateMachine.handleStateChanges(PartitionStateMachine.scala:133)
| at kafka.controller.PartitionStateMachine.triggerOnlinePartitionStateChange(PartitionStateMachine.scala:123)
| at kafka.controller.PartitionStateMachine.triggerOnlinePartitionStateChange(PartitionStateMachine.scala:109)
| at kafka.controller.PartitionStateMachine.startup(PartitionStateMachine.scala:66)
| at kafka.controller.KafkaController.onControllerFailover(KafkaController.scala:260)
| at kafka.controller.KafkaController.kafka$controller$KafkaController$$elect(KafkaController.scala:1221)
| at kafka.controller.KafkaController$Startup$.process(KafkaController.scala:1134)
| at kafka.controller.ControllerEventManager$ControllerEventThread$$anonfun$doWork$1.apply$mcV$sp(ControllerEventManager.scala:89)
| at kafka.controller.ControllerEventManager$ControllerEventThread$$anonfun$doWork$1.apply(ControllerEventManager.scala:89)
| at kafka.controller.ControllerEventManager$ControllerEventThread$$anonfun$doWork$1.apply(ControllerEventManager.scala:89)
| at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:31)
| at kafka.controller.ControllerEventManager$ControllerEventThread.doWork(ControllerEventManager.scala:88)
| at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82)
| [2019-12-10 04:05:18,246] ERROR [Controller id=3150 epoch=5] Controller 3150 epoch 5 failed to change state for partition __consumer_offsets-26 from OfflinePartition to OnlinePartition (state.change.logger)
| kafka.common.StateChangeFailedException: Failed to elect leader for partition __consumer_offsets-26 under strategy OfflinePartitionLeaderElectionStrategy
| at kafka.controller.PartitionStateMachine$$anonfun$doElectLeaderForPartitions$3.apply(PartitionStateMachine.scala:366)
| at kafka.controller.PartitionStateMachine$$anonfun$doElectLeaderForPartitions$3.apply(PartitionStateMachine.scala:364)
| at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
| at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
| at kafka.controller.PartitionStateMachine.doElectLeaderForPartitions(PartitionStateMachine.scala:364)
| at kafka.controller.PartitionStateMachine.electLeaderForPartitions(PartitionStateMachine.scala:292)
| at kafka.controller.PartitionStateMachine.doHandleStateChanges(PartitionStateMachine.scala:210)
| at kafka.controller.PartitionStateMachine.handleStateChanges(PartitionStateMachine.scala:133)
| at kafka.controller.PartitionStateMachine.triggerOnlinePartitionStateChange(PartitionStateMachine.scala:123)
| at kafka.controller.PartitionStateMachine.triggerOnlinePartitionStateChange(PartitionStateMachine.scala:109)
| at kafka.controller.PartitionStateMachine.startup(PartitionStateMachine.scala:66)
| at kafka.controller.KafkaController.onControllerFailover(KafkaController.scala:260)
| at kafka.controller.KafkaController.kafka$controller$KafkaController$$elect(KafkaController.scala:1221)
| at kafka.controller.KafkaController$Startup$.process(KafkaController.scala:1134)
| at kafka.controller.ControllerEventManager$ControllerEventThread$$anonfun$doWork$1.apply$mcV$sp(ControllerEventManager.scala:89)
| at kafka.controller.ControllerEventManager$ControllerEventThread$$anonfun$doWork$1.apply(ControllerEventManager.scala:89)
| at kafka.controller.ControllerEventManager$ControllerEventThread$$anonfun$doWork$1.apply(ControllerEventManager.scala:89)
| at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:31)
| at kafka.controller.ControllerEventManager$ControllerEventThread.doWork(ControllerEventManager.scala:88)
| at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82)
| [2019-12-10 04:05:18,246] ERROR [Controller id=3150 epoch=5] Controller 3150 epoch 5 failed to change state for partition __consumer_offsets-29 from OfflinePartition to OnlinePartition (state.change.logger)
| kafka.common.StateChangeFailedException: Failed to elect leader for partition __consumer_offsets-29 under strategy OfflinePartitionLeaderElectionStrategy
| at kafka.controller.PartitionStateMachine$$anonfun$doElectLeaderForPartitions$3.apply(PartitionStateMachine.scala:366)
| at kafka.controller.PartitionStateMachine$$anonfun$doElectLeaderForPartitions$3.apply(PartitionStateMachine.scala:364)
| at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
| at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
| at kafka.controller.PartitionStateMachine.doElectLeaderForPartitions(PartitionStateMachine.scala:364)
| at kafka.controller.PartitionStateMachine.electLeaderForPartitions(PartitionStateMachine.scala:292)
| at kafka.controller.PartitionStateMachine.doHandleStateChanges(PartitionStateMachine.scala:210)
| at kafka.controller.PartitionStateMachine.handleStateChanges(PartitionStateMachine.scala:133)
| at kafka.controller.PartitionStateMachine.triggerOnlinePartitionStateChange(PartitionStateMachine.scala:123)
| at kafka.controller.PartitionStateMachine.triggerOnlinePartitionStateChange(PartitionStateMachine.scala:109)
| at kafka.controller.PartitionStateMachine.startup(PartitionStateMachine.scala:66)
| at kafka.controller.KafkaController.onControllerFailover(KafkaController.scala:260)
| at kafka.controller.KafkaController.kafka$controller$KafkaController$$elect(KafkaController.scala:1221)
| at kafka.controller.KafkaController$Startup$.process(KafkaController.scala:1134)
| at kafka.controller.ControllerEventManager$ControllerEventThread$$anonfun$doWork$1.apply$mcV$sp(ControllerEventManager.scala:89)
| at kafka.controller.ControllerEventManager$ControllerEventThread$$anonfun$doWork$1.apply(ControllerEventManager.scala:89)
| at kafka.controller.ControllerEventManager$ControllerEventThread$$anonfun$doWork$1.apply(ControllerEventManager.scala:89)
| at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:31)
| at kafka.controller.ControllerEventManager$ControllerEventThread.doWork(ControllerEventManager.scala:88)
| at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82)
| [2019-12-10 04:05:18,246] ERROR [Controller id=3150 epoch=5] Controller 3150 epoch 5 failed to change state for partition __consumer_offsets-34 from OfflinePartition to OnlinePartition (state.change.logger)
| kafka.common.StateChangeFailedException: Failed to elect leader for partition __consumer_offsets-34 under strategy OfflinePartitionLeaderElectionStrategy
| at kafka.controller.PartitionStateMachine$$anonfun$doElectLeaderForPartitions$3.apply(PartitionStateMachine.scala:366)
| at kafka.controller.PartitionStateMachine$$anonfun$doElectLeaderForPartitions$3.apply(PartitionStateMachine.scala:364)
| at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
| at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
| at kafka.controller.PartitionStateMachine.doElectLeaderForPartitions(PartitionStateMachine.scala:364)
| at kafka.controller.PartitionStateMachine.electLeaderForPartitions(PartitionStateMachine.scala:292)
| at kafka.controller.PartitionStateMachine.doHandleStateChanges(PartitionStateMachine.scala:210)
| at kafka.controller.PartitionStateMachine.handleStateChanges(PartitionStateMachine.scala:133)
| at kafka.controller.PartitionStateMachine.triggerOnlinePartitionStateChange(PartitionStateMachine.scala:123)
| at kafka.controller.PartitionStateMachine.triggerOnlinePartitionStateChange(PartitionStateMachine.scala:109)
| at kafka.controller.PartitionStateMachine.startup(PartitionStateMachine.scala:66)
| at kafka.controller.KafkaController.onControllerFailover(KafkaController.scala:260)
| at kafka.controller.KafkaController.kafka$controller$KafkaController$$elect(KafkaController.scala:1221)
| at kafka.controller.KafkaController$Startup$.process(KafkaController.scala:1134)
| at kafka.controller.ControllerEventManager$ControllerEventThread$$anonfun$doWork$1.apply$mcV$sp(ControllerEventManager.scala:89)
| at kafka.controller.ControllerEventManager$ControllerEventThread$$anonfun$doWork$1.apply(ControllerEventManager.scala:89)
| at kafka.controller.ControllerEventManager$ControllerEventThread$$anonfun$doWork$1.apply(ControllerEventManager.scala:89)
| at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:31)
| at kafka.controller.ControllerEventManager$ControllerEventThread.doWork(ControllerEventManager.scala:88)
| at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82)
| [2019-12-10 04:05:18,246] ERROR [Controller id=3150 epoch=5] Controller 3150 epoch 5 failed to change state for partition __consumer_offsets-10 from OfflinePartition to OnlinePartition (state.change.logger)
| kafka.common.StateChangeFailedException: Failed to elect leader for partition __consumer_offsets-10 under strategy OfflinePartitionLeaderElectionStrategy
| at kafka.controller.PartitionStateMachine$$anonfun$doElectLeaderForPartitions$3.apply(PartitionStateMachine.scala:366)
| at kafka.controller.PartitionStateMachine$$anonfun$doElectLeaderForPartitions$3.apply(PartitionStateMachine.scala:364)
| at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
| at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
| at kafka.controller.PartitionStateMachine.doElectLeaderForPartitions(PartitionStateMachine.scala:364)
| at kafka.controller.PartitionStateMachine.electLeaderForPartitions(PartitionStateMachine.scala:292)
| at kafka.controller.PartitionStateMachine.doHandleStateChanges(PartitionStateMachine.scala:210)
| at kafka.controller.PartitionStateMachine.handleStateChanges(PartitionStateMachine.scala:133)
| at kafka.controller.PartitionStateMachine.triggerOnlinePartitionStateChange(PartitionStateMachine.scala:123)
| at kafka.controller.PartitionStateMachine.triggerOnlinePartitionStateChange(PartitionStateMachine.scala:109)
| at kafka.controller.PartitionStateMachine.startup(PartitionStateMachine.scala:66)
| at kafka.controller.KafkaController.onControllerFailover(KafkaController.scala:260)
| at kafka.controller.KafkaController.kafka$controller$KafkaController$$elect(KafkaController.scala:1221)
| at kafka.controller.KafkaController$Startup$.process(KafkaController.scala:1134)
| at kafka.controller.ControllerEventManager$ControllerEventThread$$anonfun$doWork$1.apply$mcV$sp(ControllerEventManager.scala:89)
| at kafka.controller.ControllerEventManager$ControllerEventThread$$anonfun$doWork$1.apply(ControllerEventManager.scala:89)
| at kafka.controller.ControllerEventManager$ControllerEventThread$$anonfun$doWork$1.apply(ControllerEventManager.scala:89)
| at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:31)
| at kafka.controller.ControllerEventManager$ControllerEventThread.doWork(ControllerEventManager.scala:88)
| at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82)
| [2019-12-10 04:05:18,246] ERROR [Controller id=3150 epoch=5] Controller 3150 epoch 5 failed to change state for partition __consumer_offsets-32 from OfflinePartition to OnlinePartition (state.change.logger)
| kafka.common.StateChangeFailedException: Failed to elect leader for partition __consumer_offsets-32 under strategy OfflinePartitionLeaderElectionStrategy
| at kafka.controller.PartitionStateMachine$$anonfun$doElectLeaderForPartitions$3.apply(PartitionStateMachine.scala:366)
| at kafka.controller.PartitionStateMachine$$anonfun$doElectLeaderForPartitions$3.apply(PartitionStateMachine.scala:364)
| at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
| at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
| at kafka.controller.PartitionStateMachine.doElectLeaderForPartitions(PartitionStateMachine.scala:364)
| at kafka.controller.PartitionStateMachine.electLeaderForPartitions(PartitionStateMachine.scala:292)
| at kafka.controller.PartitionStateMachine.doHandleStateChanges(PartitionStateMachine.scala:210)
| at kafka.controller.PartitionStateMachine.handleStateChanges(PartitionStateMachine.scala:133)
| at kafka.controller.PartitionStateMachine.triggerOnlinePartitionStateChange(PartitionStateMachine.scala:123)
| at kafka.controller.PartitionStateMachine.triggerOnlinePartitionStateChange(PartitionStateMachine.scala:109)
| at kafka.controller.PartitionStateMachine.startup(PartitionStateMachine.scala:66)
| at kafka.controller.KafkaController.onControllerFailover(KafkaController.scala:260)
| at kafka.controller.KafkaController.kafka$controller$KafkaController$$elect(KafkaController.scala:1221)
| at kafka.controller.KafkaController$Startup$.process(KafkaController.scala:1134)
| at kafka.controller.ControllerEventManager$ControllerEventThread$$anonfun$doWork$1.apply$mcV$sp(ControllerEventManager.scala:89)
| at kafka.controller.ControllerEventManager$ControllerEventThread$$anonfun$doWork$1.apply(ControllerEventManager.scala:89)
| at kafka.controller.ControllerEventManager$ControllerEventThread$$anonfun$doWork$1.apply(ControllerEventManager.scala:89)
| at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:31)
| at kafka.controller.ControllerEventManager$ControllerEventThread.doWork(ControllerEventManager.scala:88)
| at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82)
| [2019-12-10 04:05:18,246] ERROR [Controller id=3150 epoch=5] Controller 3150 epoch 5 failed to change state for partition __consumer_offsets-40 from OfflinePartition to OnlinePartition (state.change.logger)
| kafka.common.StateChangeFailedException: Failed to elect leader for partition __consumer_offsets-40 under strategy OfflinePartitionLeaderElectionStrategy
| at kafka.controller.PartitionStateMachine$$anonfun$doElectLeaderForPartitions$3.apply(PartitionStateMachine.scala:366)
| at kafka.controller.PartitionStateMachine$$anonfun$doElectLeaderForPartitions$3.apply(PartitionStateMachine.scala:364)
| at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
| at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
| at kafka.controller.PartitionStateMachine.doElectLeaderForPartitions(PartitionStateMachine.scala:364)
| at kafka.controller.PartitionStateMachine.electLeaderForPartitions(PartitionStateMachine.scala:292)
| at kafka.controller.PartitionStateMachine.doHandleStateChanges(PartitionStateMachine.scala:210)
| at kafka.controller.PartitionStateMachine.handleStateChanges(PartitionStateMachine.scala:133)
| at kafka.controller.PartitionStateMachine.triggerOnlinePartitionStateChange(PartitionStateMachine.scala:123)
| at kafka.controller.PartitionStateMachine.triggerOnlinePartitionStateChange(PartitionStateMachine.scala:109)
| at kafka.controller.PartitionStateMachine.startup(PartitionStateMachine.scala:66)
| at kafka.controller.KafkaController.onControllerFailover(KafkaController.scala:260)
| at kafka.controller.KafkaController.kafka$controller$KafkaController$$elect(KafkaController.scala:1221)
| at kafka.controller.KafkaController$Startup$.process(KafkaController.scala:1134)
| at kafka.controller.ControllerEventManager$ControllerEventThread$$anonfun$doWork$1.apply$mcV$sp(ControllerEventManager.scala:89)
| at kafka.controller.ControllerEventManager$ControllerEventThread$$anonfun$doWork$1.apply(ControllerEventManager.scala:89)
| at kafka.controller.ControllerEventManager$ControllerEventThread$$anonfun$doWork$1.apply(ControllerEventManager.scala:89)
| at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:31)
| at kafka.controller.ControllerEventManager$ControllerEventThread.doWork(ControllerEventManager.scala:88)
| at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82)

I assume it’s as expected?

BYK · December 10, 2019, 9:06am

Thanks for reporting back and no that’s not very expected. I’m not an expert on Kafka but maybe we need to clear the zookeeper volume too to prevent this?

cwang · December 10, 2019, 11:19am

We removed zookeeper and redis volume then rebuild, but to no avail. A serious side effect was Sentry hangs if you try to delete any event, so we ended up rebuilding the whole stack (lost all data ).

BYK · December 11, 2019, 7:25pm

@cwang - sorry you had to go through this. This is why we have marked v10 as beta for now. I’ll explore ways of a cleaner migration from earlier v10 installations in the meantime.

cwang · December 12, 2019, 7:35am

No worries. Glad we can make some contribution. You guys are doing a really awesome job

akshaynarayan · May 27, 2020, 6:58am

Im having the same issue. ERROR [Controller id=1004 epoch=53] Controller 1004 epoch 53 failed to change state for partition __consumer_offsets-40 from OfflinePartition to OnlinePartition (state.change.logger)

Our sentry is not showing any issues on dashboard.

turbo124 · August 5, 2020, 6:52am

We are hitting this also, deleting volumes doesn’t seem to fix the issue either… any other ideas?

BYK · August 11, 2020, 10:09am

Which volumes have you tried deleting?

turbo124 · August 11, 2020, 10:40am

sentry-data
sentry-redis
sentry-zookeeper
sentry-kafka
sentry-clickhouse
sentry-symbolicator

BYK · August 12, 2020, 9:18am

Can you also try removing the following:

sentry-zookeeper-log
sentry-kafka-log

turbo124 · August 12, 2020, 9:43am

@BYK unfortunately still no go.

BYK · August 12, 2020, 10:24am

@turbo124 if you can share some logs it may help us understand if there’s something else going on?

turbo124 · August 12, 2020, 12:10pm

sure, when running ./install.sh i see this. Did you want any from a specific container also?

+ exec gosu snuba snuba bootstrap --force^M
%3|1597226008.239|FAIL|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#192.168.16.6:9092 failed: Connection refused (after 1ms in state CONNECT)^M
%3|1597226009.236|FAIL|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#192.168.16.6:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppress`Preformatted text`


cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str="Failed to get metadata: Local: Broker transport failure"}^M
%3|1597226010.238|FAIL|rdkafka#producer-2| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#192.168.16.6:9092 failed: Connection refused (after 0ms in state CONNECT)^M
%3|1597226011.238|FAIL|rdkafka#producer-2| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#192.168.16.6:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppresse



2020-08-12 09:53:36,456 Failed to create topic snuba-commit-log^M
Traceback (most recent call last):^M
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 92, in bootstrap^M
    future.result()^M
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 432, in result^M
    return self.__get_result()^M
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result^M
    raise self._exception^M
cimpl.KafkaException: KafkaError{code=TOPIC_ALREADY_EXISTS,val=36,str="Topic 'snuba-commit-log' already exists."}^M

amit1 · August 12, 2020, 4:17pm

I was getting the same issues @turbo124

hheexx · August 12, 2020, 8:50pm

We have similiar problem - maybe same.

Try adding KAFKA_OFFSETS_TOPIC_NUM_PARTITIONS: ‘2’ to kafka config in docker-compose

amit1 · August 13, 2020, 4:33am

@BYK, after removing the kafka and zookeeper volumes and changing the KAFKA_OFFSETS_TOPIC_NUM_PARTITIONS: ‘2’. I see these errors

snuba-replacer_1 | + ‘[’ r = - ‘]’
snuba-replacer_1 | + snuba replacer --help
snuba-replacer_1 | + set – snuba replacer --storage events --auto-offset-reset=latest --max-batch-size 3
snuba-replacer_1 | + set gosu snuba snuba replacer --storage events --auto-offset-reset=latest --max-batch-size 3
snuba-replacer_1 | + exec gosu snuba snuba replacer --storage events --auto-offset-reset=latest --max-batch-size 3
snuba-replacer_1 | Traceback (most recent call last):
snuba-replacer_1 | File “/usr/local/bin/snuba”, line 33, in
snuba-replacer_1 | sys.exit(load_entry_point(‘snuba’, ‘console_scripts’, ‘snuba’)())
snuba-replacer_1 | File “/usr/local/lib/python3.8/site-packages/click/core.py”, line 722, in call
snuba-replacer_1 | return self.main(*args, **kwargs)
snuba-replacer_1 | File “/usr/local/lib/python3.8/site-packages/click/core.py”, line 697, in main
snuba-replacer_1 | rv = self.invoke(ctx)
snuba-replacer_1 | File “/usr/local/lib/python3.8/site-packages/click/core.py”, line 1066, in invoke
snuba-replacer_1 | return _process_result(sub_ctx.command.invoke(sub_ctx))
snuba-replacer_1 | File “/usr/local/lib/python3.8/site-packages/click/core.py”, line 895, in invoke
snuba-replacer_1 | return ctx.invoke(self.callback, **ctx.params)
snuba-replacer_1 | File “/usr/local/lib/python3.8/site-packages/click/core.py”, line 535, in invoke
snuba-replacer_1 | return callback(*args, **kwargs)
snuba-replacer_1 | File “/usr/src/snuba/snuba/cli/replacer.py”, line 132, in replacer
snuba-replacer_1 | replacer.run()
snuba-replacer_1 | File “/usr/src/snuba/snuba/utils/streams/processing.py”, line 132, in run
snuba-replacer_1 | self._run_once()
snuba-replacer_1 | File “/usr/src/snuba/snuba/utils/streams/processing.py”, line 138, in _run_once
snuba-replacer_1 | msg = self.__consumer.poll(timeout=1.0)
snuba-replacer_1 | File “/usr/src/snuba/snuba/utils/streams/kafka.py”, line 412, in poll
snuba-replacer_1 | raise ConsumerError(str(error))
snuba-replacer_1 | snuba.utils.streams.consumer.ConsumerError: KafkaError{code=UNKNOWN_TOPIC_OR_PART,val=3,str=“Subscribed topic not available: event-replacements: Broker: Unknown topic or partition”}

amit1 · August 13, 2020, 5:27am

No longer getting the above issue , after
re-running ./install.sh. Workers still not processing tasks. The only error i see is from the snuba-replacer_1 to kafka

@BYK @hheexx, any idea

sentry_onpremise_snuba-replacer_1

%3|1597293627.097|FAIL|rdkafka#consumer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.24.0.9:9092 failed: Connection refused (after 168ms in state CONNECT)
%3|1597293627.930|FAIL|rdkafka#consumer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.24.0.9:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
2020-08-13 04:40:49,183 New partitions assigned: {Partition(topic=Topic(name=‘event-replacements’), index=0): 0}

turbo124 · August 13, 2020, 9:34pm

@hheexx tried adding the kafka offset, no luck unfortunately.

Topic		Replies	Views
Kafka issues when upgrade from 9.1.2 to 20.8.0 On-Premise	10	2407	December 8, 2020
Nothing can connect to Kafka On-Premise	18	11059	April 5, 2021
New install fails on Ubuntu 20.04 - kafta errors On-Premise	31	2970	March 22, 2021
After the sentry upgrade no more catch errors On-Premise	3	2962	September 20, 2021
Error with Sentry OnPremise 21.2.0 On-Premise	2	1085	June 1, 2021

Kafka Error after Upgrading

Related topics