Relay authentication

Hello,

I’m use self-hosted Sentry 9.1.2, and now try to update it to 21.3.0. I created separate instances for redis, clickhouse, postgres, kafka. On main server I will start only sentry web, snubs and relay.

For this I will use docker-compose.yml from your repository, just removes some services. And will provide variables with my existing services. All works just fine, across relay.

I got these errors after start:

relay_1                                     | 2021-04-02T12:26:37Z [relay_server::actors::upstream] WARN: Network outage, scheduling another check in 0ns
relay_1                                     | 2021-04-02T12:26:37Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1                                     |   caused by: could not send request using reqwest
relay_1                                     |   caused by: error sending request for url (http://web:9000/api/0/relays/register/challenge/): operation timed out

My resolve.conf from relay:

$ docker-compose exec relay bash
root@c61c2db57f11:/work# cat /etc/resolv.conf
nameserver 127.0.0.11
options ndots:0

I can send requests from relay container after installation curl:

root@c61c2db57f11:/work# curl -I http://web:9000/
HTTP/1.1 302 Found
Content-Type: text/html; charset=utf-8
Location: /auth/login/
X-Robots-Tag: noindex, nofollow
Vary: Accept-Language, Cookie
Content-Language: en
Content-Length: 0
X-Frame-Options: deny
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Set-Cookie:  sentrysid=gAJ9cQBYBQAAAF9uZXh0cQFYAQAAAC9xAnMu:1lSIe9:sYmc8R-RAfPzKt7TCa4CKzUuf4c; expires=Fri, 16-Apr-2021 12:11:29 GMT; HttpOnly; Max-Age=1209600; Path=/

root@c61c2db57f11:/work# curl -I http://web:9000/api/0/relays/register/challenge/
HTTP/1.1 405 Method Not Allowed
Content-Type: application/json
Allow: POST, OPTIONS
Access-Control-Allow-Methods: POST, OPTIONS
Access-Control-Allow-Headers: X-Sentry-Auth, X-Requested-With, Origin, Accept, Content-Type, Authentication, Authorization, Content-Encoding
Access-Control-Expose-Headers: X-Sentry-Error, Retry-After
Access-Control-Allow-Origin: *
Vary: Accept-Language, Cookie
Content-Language: en
Content-Length: 41
X-Frame-Options: deny
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block

But relay container logged what it cannot send request. I use relay settings from your repository, and created credentials.json from install.sh file.

What happend? :roll_eyes:

Docker version 20.10.5, build 55c4c88
docker-compose version 1.28.5, build c4eb3a1f

I was try downgrade docker too, but relay do not work:

Docker version 19.03.14, build 5eb3275d40

I got ip address for web container and try it to use in relay, but got the same error:

relay_1                                     | 2021-04-02T13:50:56Z [relay_server::actors::upstream] ERROR:     authentication encountered error: could not send request to upstream
relay_1                                     |   caused by: could not send request using reqwest
relay_1                                     |   caused by: error sending request for url (http://172.17.0.21:9000/api/0/relays/register/challenge/): operation timed out

It’s not dns issue. And connection sucessful:

root@1d457d6999fc:/work# curl -I http://172.17.0.21:9000/api/0/relays/register/challenge/
HTTP/1.1 405 Method Not Allowed
Content-Type: application/json
Allow: POST, OPTIONS
Access-Control-Allow-Methods: POST, OPTIONS
Access-Control-Allow-Headers: X-Sentry-Auth, X-Requested-With, Origin, Accept, Content-Type, Authentication, Authorization, Content-Encoding
Access-Control-Expose-Headers: X-Sentry-Error, Retry-After
Access-Control-Allow-Origin: *
Vary: Accept-Language, Cookie
Content-Language: en
Content-Length: 41
X-Frame-Options: deny
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block

root@1d457d6999fc:/work# curl -I http://172.17.0.21:9000/
HTTP/1.1 302 Found
Content-Type: text/html; charset=utf-8
Location: /auth/login/
X-Robots-Tag: noindex, nofollow
Vary: Accept-Language, Cookie
Content-Language: en
Content-Length: 0
X-Frame-Options: deny
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Set-Cookie:  sentrysid=gAJ9cQBYBQAAAF9uZXh0cQFYAQAAAC9xAnMu:1lSKCP:NZ3QFfPwzQZuF0dAo9abN-pKxgE; expires=Fri, 16-Apr-2021 13:50:57 GMT; HttpOnly; Max-Age=1209600; Path=/

I was try to configure relay as external service. Run it from different docker-compose file with configuration, when upstream has my server name. I got the same errors.

Starting relay_relay_1 ... done
Attaching to relay_relay_1
relay_1  | 2021-04-03T15:12:19Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1  |   caused by: could not send request using reqwest
relay_1  |   caused by: error sending request for url (http://server-name:9000/api/0/relays/register/challenge/): operation timed out
relay_1  | 2021-04-03T15:12:24Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1  |   caused by: could not send request using reqwest
relay_1  |   caused by: error sending request for url (http://server-name:9000/api/0/relays/register/challenge/): operation timed out
relay_1  | 2021-04-03T15:12:30Z [relay_server::actors::upstream] WARN: Network outage, scheduling another check in 0ns
relay_1  | 2021-04-03T15:12:30Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1  |   caused by: could not send request using reqwest
relay_1  |   caused by: error sending request for url (http://server-name:9000/api/0/relays/register/challenge/): operation timed out
relay_1  | 2021-04-03T15:12:37Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1  |   caused by: could not send request using reqwest
relay_1  |   caused by: error sending request for url (http://server-name:9000/api/0/relays/register/challenge/): operation timed out
relay_1  | 2021-04-03T15:12:44Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1  |   caused by: could not send request using reqwest
relay_1  |   caused by: error sending request for url (http://server-name:9000/api/0/relays/register/challenge/): operation timed out

But server-name worked. I can curl it.

I changed settings for nginx for passing all request to web, and got new error:

web_1 | 19:25:38 [WARNING] django.security.csrf: Forbidden (CSRF token missing or incorrect.): /api/1/envelope/ (status_code=403 request=<WSGIRequest: POST '/api/1/envelope/?sentry_key=00dd44821b6a4be286862b9ee54be4e7&sentry_version=7'>)

Maybe these affect relay work too?

Yes, got it. Problem with CSRF on request from relay to web. After this web just forbid request from relay, But why? And how I can fix it?

I configure releay for run in proxy mode, logs:

relay_1                                     | 2021-04-04T06:39:47Z [relay::setup] INFO: launching relay from config folder /work/.relay
relay_1                                     | 2021-04-04T06:39:47Z [relay::setup] INFO:   relay mode: proxy
relay_1                                     | 2021-04-04T06:39:47Z [relay::setup] INFO:   relay id: bc87bba9-c183-4ae4-828d-84baea3e1459
relay_1                                     | 2021-04-04T06:39:47Z [relay::setup] INFO:   public key: YqGom0miMCqlQDdmYki7TzPfuv18Jq3Ok8NIwhGCb1s
relay_1                                     | 2021-04-04T06:39:47Z [relay::setup] INFO:   log level: TRACE
relay_1                                     | 2021-04-04T06:39:47Z [relay_server::actors::events] INFO: starting 2 event processing workers
relay_1                                     | 2021-04-04T06:39:47Z [relay_server::actors::outcome::processing] INFO: OutcomeProducer started.
relay_1                                     | 2021-04-04T06:39:47Z [relay_server::service] INFO: spawning http server
relay_1                                     | 2021-04-04T06:39:47Z [relay_server::service] INFO:   listening on: http://0.0.0.0:3000/
relay_1                                     | 2021-04-04T06:39:47Z [actix_net::server::server] INFO: Starting 2 workers
relay_1                                     | 2021-04-04T06:39:47Z [actix_net::server::server] INFO: Starting server on 0.0.0.0:3000
relay_1                                     | 2021-04-04T06:39:47Z [relay_server::actors::controller] INFO: relay server starting
relay_1                                     | 2021-04-04T06:39:47Z [relay_server::actors::connector] INFO: metered connector started
relay_1                                     | 2021-04-04T06:39:47Z [relay_server::actors::events] INFO: event manager started
relay_1                                     | 2021-04-04T06:39:47Z [relay_server::actors::project_local] INFO: project local cache started
relay_1                                     | 2021-04-04T06:39:47Z [relay_server::actors::project_upstream] INFO: project upstream cache started
relay_1                                     | 2021-04-04T06:39:47Z [relay_server::actors::project_cache] INFO: project cache started
relay_1                                     | 2021-04-04T06:39:47Z [relay_server::actors::relays] INFO: key cache started
relay_1                                     | 2021-04-04T06:39:47Z [relay_server::actors::upstream] INFO: upstream relay started
relay_1                                     | 2021-04-04T06:40:04Z [relay_server::actors::project] DEBUG: project b56bd2b668d24ffca311c4cef88fefe3 initialized without state
relay_1                                     | 2021-04-04T06:40:04Z [relay_server::actors::project] DEBUG: project b56bd2b668d24ffca311c4cef88fefe3 state requested
relay_1                                     | 2021-04-04T06:40:04Z [relay_server::actors::events] TRACE: queued event
relay_1                                     | 2021-04-04T06:40:04Z [relay_server::actors::project] DEBUG: project b56bd2b668d24ffca311c4cef88fefe3 state request amended
relay_1                                     | 2021-04-04T06:40:04Z [relay_server::actors::project] DEBUG: project state b56bd2b668d24ffca311c4cef88fefe3 updated
relay_1                                     | 2021-04-04T06:40:04Z [relay_server::actors::events] TRACE: processing json event
relay_1                                     | 2021-04-04T06:40:04Z [relay_server::actors::events] TRACE: sending event to sentry endpoint
relay_1                                     | 2021-04-04T06:40:10Z [relay_server::actors::events] DEBUG: dropped event: could not send event to upstream
relay_1                                     |   caused by: upstream request returned error 403 Forbidden
relay_1                                     |   caused by: no error details

web logs:

web_1                                       | WSGI app 0 (mountpoint='') ready in 17 seconds on interpreter 0x563780876510 pid: 24 (default app)
web_1                                       | 06:38:43 [INFO] sentry.plugins.github: apps-not-configured
web_1                                       | WSGI app 0 (mountpoint='') ready in 17 seconds on interpreter 0x563780876510 pid: 23 (default app)
web_1                                       | WSGI app 0 (mountpoint='') ready in 17 seconds on interpreter 0x563780876510 pid: 25 (default app)
web_1                                       | 06:40:05 [WARNING] django.security.csrf: Forbidden (CSRF cookie not set.): /api/126/envelope/ (status_code=403 request=<WSGIRequest: POST '/api/126/envelope/'>)

relay config:

relay:
  mode: proxy
  upstream: "http://web:9000/"
  host: 0.0.0.0
  port: 3000
logging:
  level: TRACE
http:
  timeout: 60
  connection_timeout: 15

Why web asked to set CSRF cookie, when releay used http protocol?

Got it!

http:
  timeout: 60
  connection_timeout: 15

These settings on relay resolves my issues with auth relay on web. After this I need only adding block processing to relay.

One question: why your documentation Relay Options do not include information about processing block?

The relay documentation is primarily targeted at users who want to run an external relay in front of a functional sentry installation. The onpremise repo documents what is necessary to put relay into an alternate mode where it will not forward via HTTP but rather write into Kafka as an internal component of the sentry installation.

I’d say taking anywhere close to 15 seconds for connecting from relay to web is not normal, and neither is waiting almost a minute for a response. I know nothing about your network but there is something seriously wrong there. Not to say that this isn’t an adequate workaround.

1 Like

Ok, thank you!