Regarding Kafka error

[2021-05-24 09:57:54,426] ERROR [Controller id=1004 epoch=239] Controller 1004 epoch 239 failed to change state for partition __consumer_offsets-0 from OfflinePartition to OnlinePartition (state.change.logger)
kafka.common.StateChangeFailedException: Failed to elect leader for partition __consumer_offsets-0 under strategy OfflinePartitionLeaderElectionStrategy(false)
at kafka.controller.ZkPartitionStateMachine.$anonfun$doElectLeaderForPartitions$7(PartitionStateMachine.scala:427)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at kafka.controller.ZkPartitionStateMachine.doElectLeaderForPartitions(PartitionStateMachine.scala:424)
at kafka.controller.ZkPartitionStateMachine.electLeaderForPartitions(PartitionStateMachine.scala:335)
at kafka.controller.ZkPartitionStateMachine.doHandleStateChanges(PartitionStateMachine.scala:236)
at kafka.controller.ZkPartitionStateMachine.handleStateChanges(PartitionStateMachine.scala:157)
at kafka.controller.PartitionStateMachine.triggerOnlineStateChangeForPartitions(PartitionStateMachine.scala:73)
at kafka.controller.PartitionStateMachine.triggerOnlinePartitionStateChange(PartitionStateMachine.scala:58)
at kafka.controller.PartitionStateMachine.startup(PartitionStateMachine.scala:41)
at kafka.controller.KafkaController.onControllerFailover(KafkaController.scala:306)
at kafka.controller.KafkaController.elect(KafkaController.scala:1404)
at kafka.controller.KafkaController.processStartup(KafkaController.scala:1291)
at kafka.controller.KafkaController.process(KafkaController.scala:1924)
at kafka.controller.QueuedEvent.process(ControllerEventManager.scala:53)
at kafka.controller.ControllerEventManager$ControllerEventThread.process$1(ControllerEventManager.scala:136)
at kafka.controller.ControllerEventManager$ControllerEventThread.$anonfun$doWork$1(ControllerEventManager.scala:139)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:31)
at kafka.controller.ControllerEventManager$ControllerEventThread.doWork(ControllerEventManager.scala:139)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)

Sentry version:- 21.1.0

I am facing above error in self hosted sentry Or we can’t get the event in sentry.

Have you checked https://develop.sentry.dev/self-hosted/troubleshooting/#kafka

Remain facing same error.

xecuting consumer group command failed due to org.apache.kafka.common.KafkaException: Failed to find brokers to send ListGroups
java.util.concurrent.ExecutionException: org.apache.kafka.common.KafkaException: Failed to find brokers to send ListGroups
at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
at kafka.admin.ConsumerGroupCommand$ConsumerGroupService.listConsumerGroups(ConsumerGroupCommand.scala:205)
at kafka.admin.ConsumerGroupCommand$ConsumerGroupService.listGroups(ConsumerGroupCommand.scala:200)
at kafka.admin.ConsumerGroupCommand$.run(ConsumerGroupCommand.scala:74)
at kafka.admin.ConsumerGroupCommand$.main(ConsumerGroupCommand.scala:63)
at kafka.admin.ConsumerGroupCommand.main(ConsumerGroupCommand.scala)
Caused by: org.apache.kafka.common.KafkaException: Failed to find brokers to send ListGroups
at org.apache.kafka.clients.admin.KafkaAdminClient$24.handleFailure(KafkaAdminClient.java:3272)
at org.apache.kafka.clients.admin.KafkaAdminClient$Call.failWithTimeout(KafkaAdminClient.java:813)
at org.apache.kafka.clients.admin.KafkaAdminClient$Call.fail(KafkaAdminClient.java:784)
at org.apache.kafka.clients.admin.KafkaAdminClient$TimeoutProcessor.handleTimeouts(KafkaAdminClient.java:907)
at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.timeoutPendingCalls(KafkaAdminClient.java:975)
at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1296)
at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1260)
at java.base/java.lang.Thread.run(Thread.java:829)

You didn’t share which steps you tried. Sharing the Kafka stack traces is not very useful. You may want to share Zookeeper logs and more logs from Kafka.

Right now looks like your Kafka broker cannot start. This can be due to so many reasons. Do you have enough disk space?

I remove the docker volume of Kafka and Zookeeper. after that I run install.sh file.
My sentry version is 21.1.0.

Zookeeper log

===> ENV Variables …
ALLOW_UNSIGNED=false
COMPONENT=zookeeper
CONFLUENT_DEB_VERSION=1
CONFLUENT_PLATFORM_LABEL=
CONFLUENT_SUPPORT_METRICS_ENABLE=false
CONFLUENT_VERSION=5.4.0
CUB_CLASSPATH=/etc/confluent/docker/docker-utils.jar
HOME=/root
HOSTNAME=3e75c257f1b9
KAFKA_OPTS=-Dzookeeper.4lw.commands.whitelist=ruok
KAFKA_VERSION=
LANG=C.UTF-8
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/
PYTHON_PIP_VERSION=8.1.2
PYTHON_VERSION=2.7.9-1
SCALA_VERSION=2.12
SHLVL=1
ZOOKEEPER_CLIENT_PORT=2181
ZOOKEEPER_LOG4J_ROOT_LOGLEVEL=WARN
ZOOKEEPER_TOOLS_LOG4J_LOGLEVEL=WARN
ZULU_OPENJDK_VERSION=8=8.38.0.13
_=/usr/bin/env
===> User
uid=0(root) gid=0(root) groups=0(root)
===> Configuring …
===> Running preflight checks …
===> Check if /var/lib/zookeeper/data is writable …
===> Check if /var/lib/zookeeper/log is writable …
===> Launching …
===> Launching zookeeper …
[2021-05-25 03:58:59,241] WARN Either no config or no quorum defined in config, running in standalone mode (org.apache.zookeeper.server.quorum.QuorumPeerMain)
[2021-05-25 03:58:59,823] WARN o.e.j.s.ServletContextHandler@26aa12dd{/,null,UNAVAILABLE} contextPath ends with /* (org.eclipse.jetty.server.handler.ContextHandler)
[2021-05-25 03:58:59,823] WARN Empty contextPath (org.eclipse.jetty.server.handler.ContextHandler)

Our server configuration is 160 GB harddisk and 8 GB RAM.

After update the kafka version to 6.1.0,
Kafka working fine.
I haven’t find out the any issue. Why Kafka suddenly working.