Hi,
My Sentry is reporting There is 1 issue with your system configuration from the UI and doesn’t contain any link to the error for details.
When I look into the logs, I found a couple of error messages:
clickhouse_1 | Logging errors to /var/log/clickhouse-server/clickhouse-server.err.log kafka_1 | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/172.22.0.9:2181. Will not attempt to authenticate using SASL (unknown error) ingest-consumer_1 | 04:27:49 [INFO] batching-kafka-consumer: New partitions assigned: [TopicPartition{topic=ingest-attachments,partition=0,offset=-1001,error=None}, TopicPartition{topic=ingest-events,partition=0,offset=-1001,error=None}, TopicPartition{topic=ingest-transactions,partition=0,offset=-1001,error=None}] relay_1 | 2020-11-02T04:27:18Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream relay_1 | 2020-11-02T04:27:18Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream relay_1 | 2020-11-02T04:27:19Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream relay_1 | 2020-11-02T04:27:20Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream relay_1 | 2020-11-02T04:27:22Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream relay_1 | 2020-11-02T04:27:26Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream relay_1 | 2020-11-02T04:27:31Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream relay_1 | 2020-11-02T04:27:39Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream relay_1 | caused by: Failed to connect to host: No route to host (os error 113) relay_1 | caused by: No route to host (os error 113) relay_1 | caused by: No route to host (os error 113) postgres_1 | ERROR: duplicate key value violates unique constraint "sentry_environmentprojec_project_id_environment_i_91da82f2_uniq" postgres_1 | ERROR: duplicate key value violates unique constraint "sentry_groupsubscription_group_id_user_id_62180a04_uniq"
The logs in relay service look suspecting:
2020-11-02T04:27:18Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
caused by: Failed to connect to host: Failed resolving hostname: no record found for name: web.[MY_DOMAIN]. type: AAAA class: IN
caused by: Failed resolving hostname: no record found for name: web.[MY_DOMAIN]. type: AAAA class: IN
caused by: Failed resolving hostname: no record found for name: web.[MY_DOMAIN]. type: AAAA class: IN
2020-11-02T04:27:18Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
caused by: Failed to connect to host: Failed resolving hostname: no record found for name: web.[MY_DOMAIN]. type: AAAA class: IN
caused by: Failed resolving hostname: no record found for name: web.[MY_DOMAIN]. type: AAAA class: IN
caused by: Failed resolving hostname: no record found for name: web.[MY_DOMAIN]. type: AAAA class: IN
2020-11-02T04:27:19Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
caused by: Failed to connect to host: Failed resolving hostname: no record found for name: web.[MY_DOMAIN]. type: AAAA class: IN
caused by: Failed resolving hostname: no record found for name: web.[MY_DOMAIN]. type: AAAA class: IN
caused by: Failed resolving hostname: no record found for name: web.[MY_DOMAIN]. type: AAAA class: IN
2020-11-02T04:27:20Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
caused by: Failed to connect to host: Failed resolving hostname: no record found for name: web.[MY_DOMAIN]. type: AAAA class: IN
caused by: Failed resolving hostname: no record found for name: web.[MY_DOMAIN]. type: AAAA class: IN
caused by: Failed resolving hostname: no record found for name: web.[MY_DOMAIN]. type: AAAA class: IN
2020-11-02T04:27:22Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
caused by: Failed to connect to host: Failed resolving hostname: no record found for name: web.[MY_DOMAIN]. type: AAAA class: IN
caused by: Failed resolving hostname: no record found for name: web.[MY_DOMAIN]. type: AAAA class: IN
caused by: Failed resolving hostname: no record found for name: web.[MY_DOMAIN]. type: AAAA class: IN
2020-11-02T04:27:26Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
caused by: Failed to connect to host: Failed resolving hostname: no record found for name: web.[MY_DOMAIN]. type: AAAA class: IN
caused by: Failed resolving hostname: no record found for name: web.[MY_DOMAIN]. type: AAAA class: IN
caused by: Failed resolving hostname: no record found for name: web.[MY_DOMAIN]. type: AAAA class: IN
2020-11-02T04:27:31Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
caused by: Failed to connect to host: Failed resolving hostname: no record found for name: web.[MY_DOMAIN]. type: AAAA class: IN
caused by: Failed resolving hostname: no record found for name: web.[MY_DOMAIN]. type: AAAA class: IN
caused by: Failed resolving hostname: no record found for name: web.[MY_DOMAIN]. type: AAAA class: IN
2020-11-02T04:27:39Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
caused by: Failed to connect to host: No route to host (os error 113)
caused by: No route to host (os error 113)
caused by: No route to host (os error 113)
I’ve searched for errors in relay service but found that nobody has resolved this.