New errors stuck in relay_server

On-Premise
1

Hello there,

I previously ran an older version of Sentry and decided not to hassle with a “proper” update but to just install the new version and let go of all those old events. The two applications that report errors to Sentry (one Angular, one Rails) worked just fine with the “old” setup.

So I ran install.sh from revision 4040e682670fec64fc4a6e862dc72e7be489392a and the server runs fine in general: I navigated to the admin backend, created the projects, configured users … Worked great! But everytime I trigger any of the “please cause an error”-targets on either the Angular-client or the Rails-server, no incident shows up in the UI. BUT the client and the server both make “proper” requests that both end with a 200 Statuscode and some kind of ID in the JSON payload.

I ran through the logs and recognized some issues related to the relay server. These are all the entries from the relevant docker output:

May 03 15:32:10 kleopatra docker-compose[798278]: relay_1                    | 2020-05-03T15:31:56Z [rdkafka::client] ERROR: librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): kafka:9092/bootstrap: Connect to ipv4#172.26.0.10:9092 failed: Connection refused (after 0ms in state CONNECT)
May 03 15:32:10 kleopatra docker-compose[798278]: relay_1                    | 2020-05-03T15:31:56Z [rdkafka::client] ERROR: librdkafka: Global error: AllBrokersDown (Local: All broker connections are down): 1/1 brokers are down
May 03 15:32:10 kleopatra docker-compose[798278]: relay_1                    | 2020-05-03T15:31:56Z [rdkafka::client] ERROR: librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): kafka:9092/bootstrap: Connect to ipv4#172.26.0.10:9092 failed: Connection refused (after 0ms in state CONNECT)
May 03 15:32:10 kleopatra docker-compose[798278]: relay_1                    | 2020-05-03T15:31:56Z [rdkafka::client] ERROR: librdkafka: Global error: AllBrokersDown (Local: All broker connections are down): 1/1 brokers are down
May 03 15:32:10 kleopatra docker-compose[798278]: relay_1                    | 2020-05-03T15:31:56Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
May 03 15:32:10 kleopatra docker-compose[798278]: relay_1                    |   caused by: Failed to connect to host: Failed resolving hostname: no record found for name: web type: AAAA class: IN
May 03 15:32:10 kleopatra docker-compose[798278]: relay_1                    |   caused by: Failed resolving hostname: no record found for name: web type: AAAA class: IN
May 03 15:32:10 kleopatra docker-compose[798278]: relay_1                    |   caused by: Failed resolving hostname: no record found for name: web type: AAAA class: IN
May 03 15:32:10 kleopatra docker-compose[798278]: relay_1                    | 2020-05-03T15:31:56Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
May 03 15:32:10 kleopatra docker-compose[798278]: relay_1                    |   caused by: Failed to connect to host: Failed resolving hostname: no record found for name: web type: AAAA class: IN
May 03 15:32:10 kleopatra docker-compose[798278]: relay_1                    |   caused by: Failed resolving hostname: no record found for name: web type: AAAA class: IN
May 03 15:32:10 kleopatra docker-compose[798278]: relay_1                    |   caused by: Failed resolving hostname: no record found for name: web type: AAAA class: IN
May 03 15:32:10 kleopatra docker-compose[798278]: relay_1                    | 2020-05-03T15:31:57Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
May 03 15:32:10 kleopatra docker-compose[798278]: relay_1                    |   caused by: Failed to connect to host: Failed resolving hostname: no record found for name: web type: AAAA class: IN
May 03 15:32:10 kleopatra docker-compose[798278]: relay_1                    |   caused by: Failed resolving hostname: no record found for name: web type: AAAA class: IN
May 03 15:32:10 kleopatra docker-compose[798278]: relay_1                    |   caused by: Failed resolving hostname: no record found for name: web type: AAAA class: IN
May 03 15:32:10 kleopatra docker-compose[798278]: relay_1                    | 2020-05-03T15:31:58Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
May 03 15:32:10 kleopatra docker-compose[798278]: relay_1                    |   caused by: Failed to connect to host: Failed resolving hostname: no record found for name: web type: AAAA class: IN
May 03 15:32:10 kleopatra docker-compose[798278]: relay_1                    |   caused by: Failed resolving hostname: no record found for name: web type: AAAA class: IN
May 03 15:32:10 kleopatra docker-compose[798278]: relay_1                    |   caused by: Failed resolving hostname: no record found for name: web type: AAAA class: IN
May 03 15:32:10 kleopatra docker-compose[798278]: relay_1                    | 2020-05-03T15:32:02Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
May 03 15:32:10 kleopatra docker-compose[798278]: relay_1                    |   caused by: Timeout while waiting for response
May 03 15:32:10 kleopatra docker-compose[798278]: relay_1                    | 2020-05-03T15:32:05Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
May 03 15:32:10 kleopatra docker-compose[798278]: relay_1                    |   caused by: Failed to connect to host: Connection refused (os error 111)
May 03 15:32:10 kleopatra docker-compose[798278]: relay_1                    |   caused by: Connection refused (os error 111)
May 03 15:32:10 kleopatra docker-compose[798278]: relay_1                    |   caused by: Connection refused (os error 111)
May 03 15:32:22 kleopatra docker-compose[798278]: relay_1                    | 2020-05-03T15:32:22Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
May 03 15:32:22 kleopatra docker-compose[798278]: relay_1                    |   caused by: Server disconnected
May 03 15:32:22 kleopatra docker-compose[798278]: relay_1                    |   caused by: Server disconnected
May 03 15:37:53 kleopatra docker-compose[798278]: relay_1                    | 2020-05-03T15:37:52Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
May 03 15:37:53 kleopatra docker-compose[798278]: relay_1                    |   caused by: Server disconnected
May 03 15:37:53 kleopatra docker-compose[798278]: relay_1                    |   caused by: Server disconnected
May 03 15:38:46 kleopatra docker-compose[798278]: relay_1                    | 2020-05-03T15:38:46Z [relay_server::actors::project_upstream] ERROR: error fetching project states: attempted to send request while not yet authenticated
May 03 15:38:53 kleopatra docker-compose[798278]: relay_1                    | 2020-05-03T15:38:53Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
May 03 15:38:53 kleopatra docker-compose[798278]: relay_1                    |   caused by: Server disconnected
May 03 15:38:53 kleopatra docker-compose[798278]: relay_1                    |   caused by: Server disconnected

For me this seems to be some issue with the relay_1 container not having proper network access? The kafka related errors seem be caused by the relay beeing up before kafka.

The full installation and runtime logs are available at:

It would be great if someone with more docker or sentry experience could shed some light on my problem.

Best regards

Marcus

2

Heya! Can you remove the relay/credentials.json file and run ./install.sh again?

3

Thanks for the idea, but sadly this doesn’t change anything about the situation. From what I can tell the log still spits out the same error messages, I uploaded the new run to http://playground.marcusriemer.de/sentry/20200505-run-sentry.log

4

@MarcusRiemer it now looks like Sentry web instance terminating the connection early or something as it responds with a 200. Also can you make sure your sentry.conf.py has a matching key in SENTRY_RELAY_WHITELIST_PK with the public_key field in your relay/credentials.json file?

5

Is the SENTRY_RELAY_WHITELIST_PK option still used? I can’t find any code in the repo that is using this variable. I have added it to my config but it did not help. However I have a feeling that snuba is the real problem. I will investigate in this direction.

6

You shouldn’t need this if your sentry and relay instances are on the same network.

7

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.