Sentry has stopped registering events

I am seeing the following error:

%3|1612807090.484|FAIL|rdkafka#producer-3| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.21.0.5:9092 failed: Connection refused (after 0ms in state CONNECT)

I saw some threads about issues with Kafka but couldn’t find a solution that worked for anyone. Can someone please advice what to do in this case? I have confirmed the sentry is receiving events but for some reason it isn;t recording or showing them in the UI. I was also seeing that the environment value was not getting captured last week before sentry stopped recording events completely.

Can you share your Kafka and ZooKeeper logs?

Here are zookeeper logs:

**`===> ENV Variables ...`**

**`ALLOW_UNSIGNED=false`**

**`COMPONENT=zookeeper`**

**`CONFLUENT_DEB_VERSION=1`**

**`CONFLUENT_PLATFORM_LABEL=`**

**`CONFLUENT_SUPPORT_METRICS_ENABLE=false`**

**`CONFLUENT_VERSION=5.5.0`**

**`CUB_CLASSPATH=/etc/confluent/docker/docker-utils.jar`**

**`HOME=/root`**

**`HOSTNAME=9f3c3f3104d9`**

**`KAFKA_VERSION=`**

**`LANG=C.UTF-8`**

**`PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin`**

**`PWD=/`**

**`PYTHON_PIP_VERSION=8.1.2`**

**`PYTHON_VERSION=2.7.9-1`**

**`SCALA_VERSION=2.12`**

**`SHLVL=1`**

**`ZOOKEEPER_CLIENT_PORT=2181`**

**`ZOOKEEPER_LOG4J_ROOT_LOGLEVEL=WARN`**

**`ZOOKEEPER_TOOLS_LOG4J_LOGLEVEL=WARN`**

**`ZULU_OPENJDK_VERSION=8=8.38.0.13`**

**`_=/usr/bin/env`**

**`===> User`**

**`uid=0(root) gid=0(root) groups=0(root)`**

**`===> Configuring ...`**

**`===> Running preflight checks ... `**

**`===> Check if /var/lib/zookeeper/data is writable ...`**

**`===> Check if /var/lib/zookeeper/log is writable ...`**

**`===> Launching ... `**

**`===> Launching zookeeper ... `**

**`[2021-02-09 17:24:21,329] WARN Either no config or no quorum defined in config, running  in standalone mode (org.apache.zookeeper.server.quorum.QuorumPeerMain)`**

**`[2021-02-09 17:24:21,544] WARN o.e.j.s.ServletContextHandler@4d95d2a2{/,null,UNAVAILABLE} contextPath ends with /* (org.eclipse.jetty.server.handler.ContextHandler)`**

**`[2021-02-09 17:24:21,544] WARN Empty contextPath (org.eclipse.jetty.server.handler.ContextHandler)`**

**`[2021-02-09 17:24:21,730] WARN Exception causing close of session 0x0: ZooKeeperServer not running (org.apache.zookeeper.server.NIOServerCnxn)`**

**`===> ENV Variables ...`**

**`ALLOW_UNSIGNED=false`**

**`COMPONENT=zookeeper`**

**`CONFLUENT_DEB_VERSION=1`**

**`CONFLUENT_PLATFORM_LABEL=`**

**`CONFLUENT_SUPPORT_METRICS_ENABLE=false`**

**`CONFLUENT_VERSION=5.5.0`**

**`CUB_CLASSPATH=/etc/confluent/docker/docker-utils.jar`**

**`HOME=/root`**

**`HOSTNAME=9f3c3f3104d9`**

**`KAFKA_VERSION=`**

**`LANG=C.UTF-8`**

**`PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin`**

**`PWD=/`**

**`PYTHON_PIP_VERSION=8.1.2`**

**`PYTHON_VERSION=2.7.9-1`**

**`SCALA_VERSION=2.12`**

**`SHLVL=1`**

**`ZOOKEEPER_CLIENT_PORT=2181`**

**`ZOOKEEPER_LOG4J_ROOT_LOGLEVEL=WARN`**

**`ZOOKEEPER_TOOLS_LOG4J_LOGLEVEL=WARN`**

**`ZULU_OPENJDK_VERSION=8=8.38.0.13`**

**`_=/usr/bin/env`**

**`===> User`**

**`uid=0(root) gid=0(root) groups=0(root)`**

**`===> Configuring ...`**

**`===> Running preflight checks ... `**

**`===> Check if /var/lib/zookeeper/data is writable ...`**

**`===> Check if /var/lib/zookeeper/log is writable ...`**

**`===> Launching ... `**

**`===> Launching zookeeper ... `**

**`[2021-02-09 17:27:12,093] WARN Either no config or no quorum defined in config, running  in standalone mode (org.apache.zookeeper.server.quorum.QuorumPeerMain)`**

**`[2021-02-09 17:27:12,923] WARN o.e.j.s.ServletContextHandler@4d95d2a2{/,null,UNAVAILABLE} contextPath ends with /* (org.eclipse.jetty.server.handler.ContextHandler)`**

**`[2021-02-09 17:27:12,923] WARN Empty contextPath (org.eclipse.jetty.server.handler.ContextHandler)`**

Here are kafka logs:

**`[2021-02-09 17:24:25,947] INFO [kafka-log-cleaner-thread-0]: Starting (kafka.log.LogCleaner)`**

**`[2021-02-09 17:24:26,375] INFO Awaiting socket connections on 0.0.0.0:9092. (kafka.network.Acceptor)`**

**`[2021-02-09 17:24:26,420] INFO [SocketServer brokerId=1001] Created data-plane acceptor and processors for endpoint : EndPoint(0.0.0.0,9092,ListenerName(PLAINTEXT),PLAINTEXT) (kafka.network.SocketServer)`**

**`[2021-02-09 17:24:26,421] INFO [SocketServer brokerId=1001] Started 1 acceptor threads for data-plane (kafka.network.SocketServer)`**

**`[2021-02-09 17:24:26,524] INFO Creating /brokers/ids/1001 (is it secure? false) (kafka.zk.KafkaZkClient)`**

**`[2021-02-09 17:24:26,552] INFO Stat of the created znode at /brokers/ids/1001 is: 185,185,1612891466540,1612891466540,1,0,0,72063223398662145,180,0,185`**

**` (kafka.zk.KafkaZkClient)`**

**`[2021-02-09 17:24:26,553] INFO Registered broker 1001 at path /brokers/ids/1001 with addresses: ArrayBuffer(EndPoint(kafka,9092,ListenerName(PLAINTEXT),PLAINTEXT)), czxid (broker epoch): 185 (kafka.zk.KafkaZkClient)`**

**`[2021-02-09 17:24:26,910] INFO [/config/changes-event-process-thread]: Starting (kafka.common.ZkNodeChangeNotificationListener$ChangeEventProcessThread)`**

**`[2021-02-09 17:24:27,026] INFO [SocketServer brokerId=1001] Started data-plane processors for 1 acceptors (kafka.network.SocketServer)`**

**`[2021-02-09 17:25:06,884] INFO [/config/changes-event-process-thread]: Shutting down (kafka.common.ZkNodeChangeNotificationListener$ChangeEventProcessThread)`**

**`[2021-02-09 17:25:06,884] INFO [/config/changes-event-process-thread]: Stopped (kafka.common.ZkNodeChangeNotificationListener$ChangeEventProcessThread)`**

**`[2021-02-09 17:25:06,884] INFO [/config/changes-event-process-thread]: Shutdown completed (kafka.common.ZkNodeChangeNotificationListener$ChangeEventProcessThread)`**

**`[2021-02-09 17:25:06,885] INFO [SocketServer brokerId=1001] Stopping socket server request processors (kafka.network.SocketServer)`**

**`[2021-02-09 17:25:06,894] INFO [SocketServer brokerId=1001] Stopped socket server request processors (kafka.network.SocketServer)`**

**`[2021-02-09 17:25:07,467] INFO Shutting down the log cleaner. (kafka.log.LogCleaner)`**

**`[2021-02-09 17:25:07,468] INFO [kafka-log-cleaner-thread-0]: Shutting down (kafka.log.LogCleaner)`**

**`[2021-02-09 17:25:07,468] INFO [kafka-log-cleaner-thread-0]: Stopped (kafka.log.LogCleaner)`**

**`[2021-02-09 17:25:07,469] INFO [kafka-log-cleaner-thread-0]: Shutdown completed (kafka.log.LogCleaner)`**

**`[2021-02-09 17:25:10,597] INFO [SocketServer brokerId=1001] Shutting down socket server (kafka.network.SocketServer)`**

**`[2021-02-09 17:25:10,619] INFO [SocketServer brokerId=1001] Shutdown completed (kafka.network.SocketServer)`**

**`===> ENV Variables ...`**

**`ALLOW_UNSIGNED=false`**

**`COMPONENT=kafka`**

**`CONFLUENT_DEB_VERSION=1`**

**`CONFLUENT_PLATFORM_LABEL=`**

**`CONFLUENT_SUPPORT_METRICS_ENABLE=false`**

**`CONFLUENT_VERSION=5.5.0`**

**`CUB_CLASSPATH=/etc/confluent/docker/docker-utils.jar`**

**`HOME=/root`**

**`HOSTNAME=9bbf71c18ec6`**

**`KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092`**

**`KAFKA_LOG4J_LOGGERS=kafka.cluster=WARN,kafka.controller=WARN,kafka.coordinator=WARN,kafka.log=WARN,kafka.server=WARN,kafka.zookeeper=WARN,state.change.logger=WARN`**

**`KAFKA_LOG4J_ROOT_LOGLEVEL=WARN`**

**`KAFKA_LOG_RETENTION_HOURS=24`**

**`KAFKA_MAX_REQUEST_SIZE=50000000`**

**`KAFKA_MESSAGE_MAX_BYTES=50000000`**

**`KAFKA_OFFSETS_TOPIC_NUM_PARTITIONS=1`**

**`KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1`**

**`KAFKA_TOOLS_LOG4J_LOGLEVEL=WARN`**

**`KAFKA_VERSION=`**

**`KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181`**

**`LANG=C.UTF-8`**

**`PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin`**

**`PWD=/`**

**`PYTHON_PIP_VERSION=8.1.2`**

**`PYTHON_VERSION=2.7.9-1`**

**`SCALA_VERSION=2.12`**

**`SHLVL=1`**

**`ZULU_OPENJDK_VERSION=8=8.38.0.13`**

**`_=/usr/bin/env`**

**`===> User`**

**`uid=0(root) gid=0(root) groups=0(root)`**

**`===> Configuring ...`**

**`===> Running preflight checks ... `**

**`===> Check if /var/lib/kafka/data is writable ...`**

**`===> Check if Zookeeper is healthy ...`**

**`[main] INFO org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.5.7-f0fdd52973d373ffd9c86b81d99842dc2c7f660e, built on 02/10/2020 11:30 GMT`**

**`[main] INFO org.apache.zookeeper.ZooKeeper - Client environment:host.name=9bbf71c18ec6`**

**`[main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.8.0_212`**

**`[main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Azul Systems, Inc.`**

**`[main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/zulu-8-amd64/jre`**

**`[main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/etc/confluent/docker/docker-utils.jar`**

**`[main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib`**

**`[main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp`**

**`[main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA>`**

**`[main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux`**

**`[main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64`**

**`[main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.version=3.10.0-1062.4.1.el7.x86_64`**

**`[main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.name=root`**

**`[main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.home=/root`**

**`[main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/`**

**`[main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.free=113MB`**

**`[main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.max=1735MB`**

**`[main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.total=117MB`**

**`[main] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=zookeeper:2181 sessionTimeout=40000 watcher=io.confluent.admin.utils.ZookeeperConnectionWatcher@cc34f4d`**

**`[main] INFO org.apache.zookeeper.common.X509Util - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation`**

**`[main] INFO org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 4194304 Bytes`**

**`[main] INFO org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=`**

**`[main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/172.25.0.6:2181. Will not attempt to authenticate using SASL (unknown error)`**

**`[main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /172.25.0.11:41502, server: zookeeper/172.25.0.6:2181`**

**`[main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server zookeeper/172.25.0.6:2181, sessionid = 0x1000521524a0000, negotiated timeout = 40000`**

**`[main] INFO org.apache.zookeeper.ZooKeeper - Session: 0x1000521524a0000 closed`**

**`[main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x1000521524a0000`**

**`===> Launching ... `**

**`===> Launching kafka ... `**

**`[2021-02-09 17:27:20,750] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)`**

**`[2021-02-09 17:27:22,050] WARN The package io.confluent.support.metrics.collectors.FullCollector for collecting the full set of support metrics could not be loaded, so we are reverting to anonymous, basic metric collection. If you are a Confluent customer, please refer to the Confluent Platform documentation, section Proactive Support, on how to activate full metrics collection. (io.confluent.support.metrics.KafkaSupportConfig)`**

**`[2021-02-09 17:27:22,050] WARN The support metrics collection feature ("Metrics") of Proactive Support is disabled. (io.confluent.support.metrics.SupportedServerStartable)`**

**`[2021-02-09 17:27:23,706] INFO Starting the log cleaner (kafka.log.LogCleaner)`**

**`[2021-02-09 17:27:23,848] INFO [kafka-log-cleaner-thread-0]: Starting (kafka.log.LogCleaner)`**

**`[2021-02-09 17:27:24,541] INFO Awaiting socket connections on 0.0.0.0:9092. (kafka.network.Acceptor)`**

**`[2021-02-09 17:27:24,592] INFO [SocketServer brokerId=1001] Created data-plane acceptor and processors for endpoint : EndPoint(0.0.0.0,9092,ListenerName(PLAINTEXT),PLAINTEXT) (kafka.network.SocketServer)`**

**`[2021-02-09 17:27:24,593] INFO [SocketServer brokerId=1001] Started 1 acceptor threads for data-plane (kafka.network.SocketServer)`**

**`[2021-02-09 17:27:24,750] INFO Creating /brokers/ids/1001 (is it secure? false) (kafka.zk.KafkaZkClient)`**

**`[2021-02-09 17:27:24,801] INFO Stat of the created znode at /brokers/ids/1001 is: 206,206,1612891644786,1612891644786,1,0,0,72063234710568961,180,0,206`**

**` (kafka.zk.KafkaZkClient)`**

**`[2021-02-09 17:27:24,801] INFO Registered broker 1001 at path /brokers/ids/1001 with addresses: ArrayBuffer(EndPoint(kafka,9092,ListenerName(PLAINTEXT),PLAINTEXT)), czxid (broker epoch): 206 (kafka.zk.KafkaZkClient)`**

**`[2021-02-09 17:27:25,530] INFO [/config/changes-event-process-thread]: Starting (kafka.common.ZkNodeChangeNotificationListener$ChangeEventProcessThread)`**

**`[2021-02-09 17:27:25,868] INFO [SocketServer brokerId=1001] Started data-plane processors for 1 acceptors (kafka.network.SocketServer)`**

Here are the logs from sentry_onpremise_subscription-consumer-events_1:

%3|1612891639.992|FAIL|rdkafka#producer-2| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.25.0.11:9092 failed: Connection refused (after 6ms in state CONNECT)

%3|1612891640.986|FAIL|rdkafka#producer-2| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.25.0.11:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)

%3|1612891640.997|FAIL|rdkafka#consumer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.25.0.11:9092 failed: Connection refused (after 7ms in state CONNECT)

%3|1612891641.991|FAIL|rdkafka#consumer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.25.0.11:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)

Kafka and ZK seems healthy so I’m assuming they got restarted at some point and changed IP addresses while subscription-consumer-events is still trying to use the old IP. Can you try restarting that service:

docker-compose restart subscription-consumer-events