After the sentry upgrade no more catch errors

I suspect it is a relay problem, This is the error log

error logs

Part of the relay log

relay_1 | 2021-04-19T11:10:32Z [rdkafka::client] ERROR: librdkafka: FAIL [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.21.0.10:9092 failed: Connection refused (after 11ms in state CONNECT)
relay_1 | 2021-04-19T11:10:32Z [rdkafka::client] ERROR: librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): kafka:9092/bootstrap: Connect to ipv4#172.21.0.10:9092 failed: Connection refused (after 11ms in state CONNECT)
relay_1 | 2021-04-19T11:10:32Z [rdkafka::client] ERROR: librdkafka: Global error: AllBrokersDown (Local: All broker connections are down): 1/1 brokers are down
relay_1 | 2021-04-19T11:10:33Z [rdkafka::client] ERROR: librdkafka: FAIL [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.21.0.10:9092 failed: Connection refused (after 17ms in state CONNECT)
relay_1 | 2021-04-19T11:10:33Z [rdkafka::client] ERROR: librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): kafka:9092/bootstrap: Connect to ipv4#172.21.0.10:9092 failed: Connection refused (after 17ms in state CONNECT)
relay_1 | 2021-04-19T11:10:33Z [rdkafka::client] ERROR: librdkafka: Global error: AllBrokersDown (Local: All broker connections are down): 1/1 brokers are down
relay_1 | 2021-04-19T11:10:33Z [relay_server::actors::upstream] ERROR: authentication encountered error: upstream request returned error 503 Service Unavailable
relay_1 | caused by: no error details
relay_1 | 2021-04-19T11:10:33Z [relay_server::actors::upstream] ERROR: authentication encountered error: upstream request returned error 503 Service Unavailable
relay_1 | caused by: no error details
relay_1 | 2021-04-19T11:10:33Z [rdkafka::client] ERROR: librdkafka: FAIL [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.21.0.10:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
relay_1 | 2021-04-19T11:10:33Z [rdkafka::client] ERROR: librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): kafka:9092/bootstrap: Connect to ipv4#172.21.0.10:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
relay_1 | 2021-04-19T11:10:33Z [rdkafka::client] ERROR: librdkafka: FAIL [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.21.0.10:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
relay_1 | 2021-04-19T11:10:33Z [rdkafka::client] ERROR: librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): kafka:9092/bootstrap: Connect to ipv4#172.21.0.10:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
relay_1 | 2021-04-19T11:10:34Z [relay_server::actors::upstream] ERROR: authentication encountered error: upstream request returned error 503 Service Unavailable
relay_1 | caused by: no error details
relay_1 | 2021-04-19T11:10:35Z [relay_server::actors::upstream] ERROR: authentication encountered error: upstream request returned error 503 Service Unavailable
relay_1 | caused by: no error details
relay_1 | 2021-04-19T11:10:37Z [relay_server::actors::upstream] ERROR: authentication encountered error: upstream request returned error 503 Service Unavailable
relay_1 | caused by: no error details
relay_1 | 2021-04-19T11:10:41Z [relay_server::actors::upstream] ERROR: authentication encountered error: upstream request returned error 503 Service Unavailable
relay_1 | caused by: no error details
kafka_1 | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/172.21.0.10:2181. Will not attempt to authenticate using SASL (unknown error)
relay_1 | 2021-04-19T11:10:46Z [relay_server::actors::upstream] ERROR: authentication encountered error: upstream request returned error 503 Service Unavailable
relay_1 | caused by: no error details
relay_1 | 2021-04-19T11:10:53Z [relay_server::actors::upstream] ERROR: authentication encountered error: upstream request returned error 503 Service Unavailable
relay_1 | caused by: no error details
relay_1 | 2021-04-19T11:11:05Z [relay_server::actors::upstream] ERROR: authentication encountered error: upstream request returned error 503 Service Unavailable
relay_1 | caused by: no error details
relay_1 | 2021-04-19T11:11:22Z [relay_server::actors::upstream] ERROR: authentication encountered error: upstream request returned error 503 Service Unavailable
relay_1 | caused by: no error details
relay_1 | 2021-04-19T11:11:48Z [relay_server::actors::upstream] ERROR: authentication encountered error: upstream request returned error 503 Service Unavailable
relay_1 | caused by: no error details
kafka_1 | [2021-04-19 11:19:50,273] ERROR Error while creating ephemeral at /brokers/ids/1001, node already exists and owner ‘72058566602915841’ does not match current session ‘72058588960129025’ (kafka.zk.KafkaZkClient$CheckedEphemeral)
relay_1 | 2021-04-19T11:12:26Z [relay_server::actors::upstream] ERROR: authentication encountered error: upstream request returned error 503 Service Unavailable
kafka_1 | [2021-04-19 11:19:50,296] ERROR [KafkaServer id=1001] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
relay_1 | caused by: no error details
relay_1 | 2021-04-19T11:12:33Z [relay_server::utils::shutdown] ERROR: Dropped unfinished future during shutdown: process_event
kafka_1 | org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists
relay_1 | 2021-04-19T11:12:33Z [relay_server::utils::shutdown] ERROR: Dropped unfinished future during shutdown: process_event
relay_1 | 2021-04-19T11:12:33Z [relay_server::utils::shutdown] ERROR: Dropped unfinished future during shutdown: process_event
relay_1 | 2021-04-19T11:12:33Z [relay_server::utils::shutdown] ERROR: Dropped unfinished future during shutdown: process_event

Yeah, seems like relay cannot connect to Kafka. You may want to try restarting relay (and only relay after all services are up) to see if it helps.

Met the same error, but after register relay in Sentry registering-relay-with-sentry , and restart relay after some time, this error doesn’t appear in relay log.