Sentry has stopped registering events

On-Premise
1

I am seeing the following error:

%3|1612807090.484|FAIL|rdkafka#producer-3| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.21.0.5:9092 failed: Connection refused (after 0ms in state CONNECT)

I saw some threads about issues with Kafka but couldn’t find a solution that worked for anyone. Can someone please advice what to do in this case? I have confirmed the sentry is receiving events but for some reason it isn;t recording or showing them in the UI. I was also seeing that the environment value was not getting captured last week before sentry stopped recording events completely.

2

Can you share your Kafka and ZooKeeper logs?

3

Here are zookeeper logs:

**`===> ENV Variables ...`**

**`ALLOW_UNSIGNED=false`**

**`COMPONENT=zookeeper`**

**`CONFLUENT_DEB_VERSION=1`**

**`CONFLUENT_PLATFORM_LABEL=`**

**`CONFLUENT_SUPPORT_METRICS_ENABLE=false`**

**`CONFLUENT_VERSION=5.5.0`**

**`CUB_CLASSPATH=/etc/confluent/docker/docker-utils.jar`**

**`HOME=/root`**

**`HOSTNAME=9f3c3f3104d9`**

**`KAFKA_VERSION=`**

**`LANG=C.UTF-8`**

**`PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin`**

**`PWD=/`**

**`PYTHON_PIP_VERSION=8.1.2`**

**`PYTHON_VERSION=2.7.9-1`**

**`SCALA_VERSION=2.12`**

**`SHLVL=1`**

**`ZOOKEEPER_CLIENT_PORT=2181`**

**`ZOOKEEPER_LOG4J_ROOT_LOGLEVEL=WARN`**

**`ZOOKEEPER_TOOLS_LOG4J_LOGLEVEL=WARN`**

**`ZULU_OPENJDK_VERSION=8=8.38.0.13`**

**`_=/usr/bin/env`**

**`===> User`**

**`uid=0(root) gid=0(root) groups=0(root)`**

**`===> Configuring ...`**

**`===> Running preflight checks ... `**

**`===> Check if /var/lib/zookeeper/data is writable ...`**

**`===> Check if /var/lib/zookeeper/log is writable ...`**

**`===> Launching ... `**

**`===> Launching zookeeper ... `**

**`[2021-02-09 17:24:21,329] WARN Either no config or no quorum defined in config, running  in standalone mode (org.apache.zookeeper.server.quorum.QuorumPeerMain)`**

**`[2021-02-09 17:24:21,544] WARN o.e.j.s.ServletContextHandler@4d95d2a2{/,null,UNAVAILABLE} contextPath ends with /* (org.eclipse.jetty.server.handler.ContextHandler)`**

**`[2021-02-09 17:24:21,544] WARN Empty contextPath (org.eclipse.jetty.server.handler.ContextHandler)`**

**`[2021-02-09 17:24:21,730] WARN Exception causing close of session 0x0: ZooKeeperServer not running (org.apache.zookeeper.server.NIOServerCnxn)`**

**`===> ENV Variables ...`**

**`ALLOW_UNSIGNED=false`**

**`COMPONENT=zookeeper`**

**`CONFLUENT_DEB_VERSION=1`**

**`CONFLUENT_PLATFORM_LABEL=`**

**`CONFLUENT_SUPPORT_METRICS_ENABLE=false`**

**`CONFLUENT_VERSION=5.5.0`**

**`CUB_CLASSPATH=/etc/confluent/docker/docker-utils.jar`**

**`HOME=/root`**

**`HOSTNAME=9f3c3f3104d9`**

**`KAFKA_VERSION=`**

**`LANG=C.UTF-8`**

**`PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin`**

**`PWD=/`**

**`PYTHON_PIP_VERSION=8.1.2`**

**`PYTHON_VERSION=2.7.9-1`**

**`SCALA_VERSION=2.12`**

**`SHLVL=1`**

**`ZOOKEEPER_CLIENT_PORT=2181`**

**`ZOOKEEPER_LOG4J_ROOT_LOGLEVEL=WARN`**

**`ZOOKEEPER_TOOLS_LOG4J_LOGLEVEL=WARN`**

**`ZULU_OPENJDK_VERSION=8=8.38.0.13`**

**`_=/usr/bin/env`**

**`===> User`**

**`uid=0(root) gid=0(root) groups=0(root)`**

**`===> Configuring ...`**

**`===> Running preflight checks ... `**

**`===> Check if /var/lib/zookeeper/data is writable ...`**

**`===> Check if /var/lib/zookeeper/log is writable ...`**

**`===> Launching ... `**

**`===> Launching zookeeper ... `**

**`[2021-02-09 17:27:12,093] WARN Either no config or no quorum defined in config, running  in standalone mode (org.apache.zookeeper.server.quorum.QuorumPeerMain)`**

**`[2021-02-09 17:27:12,923] WARN o.e.j.s.ServletContextHandler@4d95d2a2{/,null,UNAVAILABLE} contextPath ends with /* (org.eclipse.jetty.server.handler.ContextHandler)`**

**`[2021-02-09 17:27:12,923] WARN Empty contextPath (org.eclipse.jetty.server.handler.ContextHandler)`**

Here are kafka logs:

**`[2021-02-09 17:24:25,947] INFO [kafka-log-cleaner-thread-0]: Starting (kafka.log.LogCleaner)`**

**`[2021-02-09 17:24:26,375] INFO Awaiting socket connections on 0.0.0.0:9092. (kafka.network.Acceptor)`**

**`[2021-02-09 17:24:26,420] INFO [SocketServer brokerId=1001] Created data-plane acceptor and processors for endpoint : EndPoint(0.0.0.0,9092,ListenerName(PLAINTEXT),PLAINTEXT) (kafka.network.SocketServer)`**

**`[2021-02-09 17:24:26,421] INFO [SocketServer brokerId=1001] Started 1 acceptor threads for data-plane (kafka.network.SocketServer)`**

**`[2021-02-09 17:24:26,524] INFO Creating /brokers/ids/1001 (is it secure? false) (kafka.zk.KafkaZkClient)`**

**`[2021-02-09 17:24:26,552] INFO Stat of the created znode at /brokers/ids/1001 is: 185,185,1612891466540,1612891466540,1,0,0,72063223398662145,180,0,185`**

**` (kafka.zk.KafkaZkClient)`**

**`[2021-02-09 17:24:26,553] INFO Registered broker 1001 at path /brokers/ids/1001 with addresses: ArrayBuffer(EndPoint(kafka,9092,ListenerName(PLAINTEXT),PLAINTEXT)), czxid (broker epoch): 185 (kafka.zk.KafkaZkClient)`**

**`[2021-02-09 17:24:26,910] INFO [/config/changes-event-process-thread]: Starting (kafka.common.ZkNodeChangeNotificationListener$ChangeEventProcessThread)`**

**`[2021-02-09 17:24:27,026] INFO [SocketServer brokerId=1001] Started data-plane processors for 1 acceptors (kafka.network.SocketServer)`**

**`[2021-02-09 17:25:06,884] INFO [/config/changes-event-process-thread]: Shutting down (kafka.common.ZkNodeChangeNotificationListener$ChangeEventProcessThread)`**

**`[2021-02-09 17:25:06,884] INFO [/config/changes-event-process-thread]: Stopped (kafka.common.ZkNodeChangeNotificationListener$ChangeEventProcessThread)`**

**`[2021-02-09 17:25:06,884] INFO [/config/changes-event-process-thread]: Shutdown completed (kafka.common.ZkNodeChangeNotificationListener$ChangeEventProcessThread)`**

**`[2021-02-09 17:25:06,885] INFO [SocketServer brokerId=1001] Stopping socket server request processors (kafka.network.SocketServer)`**

**`[2021-02-09 17:25:06,894] INFO [SocketServer brokerId=1001] Stopped socket server request processors (kafka.network.SocketServer)`**

**`[2021-02-09 17:25:07,467] INFO Shutting down the log cleaner. (kafka.log.LogCleaner)`**

**`[2021-02-09 17:25:07,468] INFO [kafka-log-cleaner-thread-0]: Shutting down (kafka.log.LogCleaner)`**

**`[2021-02-09 17:25:07,468] INFO [kafka-log-cleaner-thread-0]: Stopped (kafka.log.LogCleaner)`**

**`[2021-02-09 17:25:07,469] INFO [kafka-log-cleaner-thread-0]: Shutdown completed (kafka.log.LogCleaner)`**

**`[2021-02-09 17:25:10,597] INFO [SocketServer brokerId=1001] Shutting down socket server (kafka.network.SocketServer)`**

**`[2021-02-09 17:25:10,619] INFO [SocketServer brokerId=1001] Shutdown completed (kafka.network.SocketServer)`**

**`===> ENV Variables ...`**

**`ALLOW_UNSIGNED=false`**

**`COMPONENT=kafka`**

**`CONFLUENT_DEB_VERSION=1`**

**`CONFLUENT_PLATFORM_LABEL=`**

**`CONFLUENT_SUPPORT_METRICS_ENABLE=false`**

**`CONFLUENT_VERSION=5.5.0`**

**`CUB_CLASSPATH=/etc/confluent/docker/docker-utils.jar`**

**`HOME=/root`**

**`HOSTNAME=9bbf71c18ec6`**

**`KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092`**

**`KAFKA_LOG4J_LOGGERS=kafka.cluster=WARN,kafka.controller=WARN,kafka.coordinator=WARN,kafka.log=WARN,kafka.server=WARN,kafka.zookeeper=WARN,state.change.logger=WARN`**

**`KAFKA_LOG4J_ROOT_LOGLEVEL=WARN`**

**`KAFKA_LOG_RETENTION_HOURS=24`**

**`KAFKA_MAX_REQUEST_SIZE=50000000`**

**`KAFKA_MESSAGE_MAX_BYTES=50000000`**

**`KAFKA_OFFSETS_TOPIC_NUM_PARTITIONS=1`**

**`KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1`**

**`KAFKA_TOOLS_LOG4J_LOGLEVEL=WARN`**

**`KAFKA_VERSION=`**

**`KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181`**

**`LANG=C.UTF-8`**

**`PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin`**

**`PWD=/`**

**`PYTHON_PIP_VERSION=8.1.2`**

**`PYTHON_VERSION=2.7.9-1`**

**`SCALA_VERSION=2.12`**

**`SHLVL=1`**

**`ZULU_OPENJDK_VERSION=8=8.38.0.13`**

**`_=/usr/bin/env`**

**`===> User`**

**`uid=0(root) gid=0(root) groups=0(root)`**

**`===> Configuring ...`**

**`===> Running preflight checks ... `**

**`===> Check if /var/lib/kafka/data is writable ...`**

**`===> Check if Zookeeper is healthy ...`**

**`[main] INFO org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.5.7-f0fdd52973d373ffd9c86b81d99842dc2c7f660e, built on 02/10/2020 11:30 GMT`**

**`[main] INFO org.apache.zookeeper.ZooKeeper - Client environment:host.name=9bbf71c18ec6`**

**`[main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.8.0_212`**

**`[main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Azul Systems, Inc.`**

**`[main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/zulu-8-amd64/jre`**

**`[main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/etc/confluent/docker/docker-utils.jar`**

**`[main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib`**

**`[main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp`**

**`[main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA>`**

**`[main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux`**

**`[main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64`**

**`[main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.version=3.10.0-1062.4.1.el7.x86_64`**

**`[main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.name=root`**

**`[main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.home=/root`**

**`[main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/`**

**`[main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.free=113MB`**

**`[main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.max=1735MB`**

**`[main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.total=117MB`**

**`[main] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=zookeeper:2181 sessionTimeout=40000 watcher=io.confluent.admin.utils.ZookeeperConnectionWatcher@cc34f4d`**

**`[main] INFO org.apache.zookeeper.common.X509Util - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation`**

**`[main] INFO org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 4194304 Bytes`**

**`[main] INFO org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=`**

**`[main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/172.25.0.6:2181. Will not attempt to authenticate using SASL (unknown error)`**

**`[main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /172.25.0.11:41502, server: zookeeper/172.25.0.6:2181`**

**`[main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server zookeeper/172.25.0.6:2181, sessionid = 0x1000521524a0000, negotiated timeout = 40000`**

**`[main] INFO org.apache.zookeeper.ZooKeeper - Session: 0x1000521524a0000 closed`**

**`[main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x1000521524a0000`**

**`===> Launching ... `**

**`===> Launching kafka ... `**

**`[2021-02-09 17:27:20,750] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)`**

**`[2021-02-09 17:27:22,050] WARN The package io.confluent.support.metrics.collectors.FullCollector for collecting the full set of support metrics could not be loaded, so we are reverting to anonymous, basic metric collection. If you are a Confluent customer, please refer to the Confluent Platform documentation, section Proactive Support, on how to activate full metrics collection. (io.confluent.support.metrics.KafkaSupportConfig)`**

**`[2021-02-09 17:27:22,050] WARN The support metrics collection feature ("Metrics") of Proactive Support is disabled. (io.confluent.support.metrics.SupportedServerStartable)`**

**`[2021-02-09 17:27:23,706] INFO Starting the log cleaner (kafka.log.LogCleaner)`**

**`[2021-02-09 17:27:23,848] INFO [kafka-log-cleaner-thread-0]: Starting (kafka.log.LogCleaner)`**

**`[2021-02-09 17:27:24,541] INFO Awaiting socket connections on 0.0.0.0:9092. (kafka.network.Acceptor)`**

**`[2021-02-09 17:27:24,592] INFO [SocketServer brokerId=1001] Created data-plane acceptor and processors for endpoint : EndPoint(0.0.0.0,9092,ListenerName(PLAINTEXT),PLAINTEXT) (kafka.network.SocketServer)`**

**`[2021-02-09 17:27:24,593] INFO [SocketServer brokerId=1001] Started 1 acceptor threads for data-plane (kafka.network.SocketServer)`**

**`[2021-02-09 17:27:24,750] INFO Creating /brokers/ids/1001 (is it secure? false) (kafka.zk.KafkaZkClient)`**

**`[2021-02-09 17:27:24,801] INFO Stat of the created znode at /brokers/ids/1001 is: 206,206,1612891644786,1612891644786,1,0,0,72063234710568961,180,0,206`**

**` (kafka.zk.KafkaZkClient)`**

**`[2021-02-09 17:27:24,801] INFO Registered broker 1001 at path /brokers/ids/1001 with addresses: ArrayBuffer(EndPoint(kafka,9092,ListenerName(PLAINTEXT),PLAINTEXT)), czxid (broker epoch): 206 (kafka.zk.KafkaZkClient)`**

**`[2021-02-09 17:27:25,530] INFO [/config/changes-event-process-thread]: Starting (kafka.common.ZkNodeChangeNotificationListener$ChangeEventProcessThread)`**

**`[2021-02-09 17:27:25,868] INFO [SocketServer brokerId=1001] Started data-plane processors for 1 acceptors (kafka.network.SocketServer)`**
4

Here are the logs from sentry_onpremise_subscription-consumer-events_1:

%3|1612891639.992|FAIL|rdkafka#producer-2| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.25.0.11:9092 failed: Connection refused (after 6ms in state CONNECT)

%3|1612891640.986|FAIL|rdkafka#producer-2| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.25.0.11:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)

%3|1612891640.997|FAIL|rdkafka#consumer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.25.0.11:9092 failed: Connection refused (after 7ms in state CONNECT)

%3|1612891641.991|FAIL|rdkafka#consumer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.25.0.11:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)

5

Kafka and ZK seems healthy so I’m assuming they got restarted at some point and changed IP addresses while subscription-consumer-events is still trying to use the old IP. Can you try restarting that service:

docker-compose restart subscription-consumer-events
6

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.