After the sentry upgrade no more catch errors

On-Premise
1

I suspect it is a relay problem, This is the error log

error logs

Part of the relay log

relay_1 | 2021-04-19T11:10:32Z [rdkafka::client] ERROR: librdkafka: FAIL [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.21.0.10:9092 failed: Connection refused (after 11ms in state CONNECT)
relay_1 | 2021-04-19T11:10:32Z [rdkafka::client] ERROR: librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): kafka:9092/bootstrap: Connect to ipv4#172.21.0.10:9092 failed: Connection refused (after 11ms in state CONNECT)
relay_1 | 2021-04-19T11:10:32Z [rdkafka::client] ERROR: librdkafka: Global error: AllBrokersDown (Local: All broker connections are down): 1/1 brokers are down
relay_1 | 2021-04-19T11:10:33Z [rdkafka::client] ERROR: librdkafka: FAIL [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.21.0.10:9092 failed: Connection refused (after 17ms in state CONNECT)
relay_1 | 2021-04-19T11:10:33Z [rdkafka::client] ERROR: librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): kafka:9092/bootstrap: Connect to ipv4#172.21.0.10:9092 failed: Connection refused (after 17ms in state CONNECT)
relay_1 | 2021-04-19T11:10:33Z [rdkafka::client] ERROR: librdkafka: Global error: AllBrokersDown (Local: All broker connections are down): 1/1 brokers are down
relay_1 | 2021-04-19T11:10:33Z [relay_server::actors::upstream] ERROR: authentication encountered error: upstream request returned error 503 Service Unavailable
relay_1 | caused by: no error details
relay_1 | 2021-04-19T11:10:33Z [relay_server::actors::upstream] ERROR: authentication encountered error: upstream request returned error 503 Service Unavailable
relay_1 | caused by: no error details
relay_1 | 2021-04-19T11:10:33Z [rdkafka::client] ERROR: librdkafka: FAIL [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.21.0.10:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
relay_1 | 2021-04-19T11:10:33Z [rdkafka::client] ERROR: librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): kafka:9092/bootstrap: Connect to ipv4#172.21.0.10:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
relay_1 | 2021-04-19T11:10:33Z [rdkafka::client] ERROR: librdkafka: FAIL [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.21.0.10:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
relay_1 | 2021-04-19T11:10:33Z [rdkafka::client] ERROR: librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): kafka:9092/bootstrap: Connect to ipv4#172.21.0.10:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
relay_1 | 2021-04-19T11:10:34Z [relay_server::actors::upstream] ERROR: authentication encountered error: upstream request returned error 503 Service Unavailable
relay_1 | caused by: no error details
relay_1 | 2021-04-19T11:10:35Z [relay_server::actors::upstream] ERROR: authentication encountered error: upstream request returned error 503 Service Unavailable
relay_1 | caused by: no error details
relay_1 | 2021-04-19T11:10:37Z [relay_server::actors::upstream] ERROR: authentication encountered error: upstream request returned error 503 Service Unavailable
relay_1 | caused by: no error details
relay_1 | 2021-04-19T11:10:41Z [relay_server::actors::upstream] ERROR: authentication encountered error: upstream request returned error 503 Service Unavailable
relay_1 | caused by: no error details
kafka_1 | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/172.21.0.10:2181. Will not attempt to authenticate using SASL (unknown error)
relay_1 | 2021-04-19T11:10:46Z [relay_server::actors::upstream] ERROR: authentication encountered error: upstream request returned error 503 Service Unavailable
relay_1 | caused by: no error details
relay_1 | 2021-04-19T11:10:53Z [relay_server::actors::upstream] ERROR: authentication encountered error: upstream request returned error 503 Service Unavailable
relay_1 | caused by: no error details
relay_1 | 2021-04-19T11:11:05Z [relay_server::actors::upstream] ERROR: authentication encountered error: upstream request returned error 503 Service Unavailable
relay_1 | caused by: no error details
relay_1 | 2021-04-19T11:11:22Z [relay_server::actors::upstream] ERROR: authentication encountered error: upstream request returned error 503 Service Unavailable
relay_1 | caused by: no error details
relay_1 | 2021-04-19T11:11:48Z [relay_server::actors::upstream] ERROR: authentication encountered error: upstream request returned error 503 Service Unavailable
relay_1 | caused by: no error details
kafka_1 | [2021-04-19 11:19:50,273] ERROR Error while creating ephemeral at /brokers/ids/1001, node already exists and owner ‘72058566602915841’ does not match current session ‘72058588960129025’ (kafka.zk.KafkaZkClient$CheckedEphemeral)
relay_1 | 2021-04-19T11:12:26Z [relay_server::actors::upstream] ERROR: authentication encountered error: upstream request returned error 503 Service Unavailable
kafka_1 | [2021-04-19 11:19:50,296] ERROR [KafkaServer id=1001] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
relay_1 | caused by: no error details
relay_1 | 2021-04-19T11:12:33Z [relay_server::utils::shutdown] ERROR: Dropped unfinished future during shutdown: process_event
kafka_1 | org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists
relay_1 | 2021-04-19T11:12:33Z [relay_server::utils::shutdown] ERROR: Dropped unfinished future during shutdown: process_event
relay_1 | 2021-04-19T11:12:33Z [relay_server::utils::shutdown] ERROR: Dropped unfinished future during shutdown: process_event
relay_1 | 2021-04-19T11:12:33Z [relay_server::utils::shutdown] ERROR: Dropped unfinished future during shutdown: process_event

2

Yeah, seems like relay cannot connect to Kafka. You may want to try restarting relay (and only relay after all services are up) to see if it helps.

3

Met the same error, but after register relay in Sentry registering-relay-with-sentry , and restart relay after some time, this error doesn’t appear in relay log.

4

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.