CSRF Issue when using SSL via nginx

On-Premise
1

Hi everyone,

I have an CSRF issue as I can’t get Sentry to work with a Nginx Reverse SSL Proxy. Followed the setup as described here: https://docs.sentry.io/server/installation/

Then installed Nginx with a config based on this one: https://docs.sentry.io/server/nginx/

However, whenever I post something via the web UI, I get 403s with the following error via the developer tools:

{"detail":"CSRF Failed: CSRF token missing or incorrect."}

Here is my setup:

Nginx config in /etc/nginx/sites-available/sentry:

# set REMOTE_ADDR from any internal proxies
# see http://nginx.org/en/docs/http/ngx_http_realip_module.html
set_real_ip_from 127.0.0.1;
set_real_ip_from 10.0.0.0/8;
real_ip_header X-Forwarded-For;
real_ip_recursive on;

# SSL configuration -- change these certs to match yours
ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;

server {
  listen   80;
  server_name example.com;

  location / {
    if ($request_method = GET) {
      rewrite  ^ https://$host$request_uri? permanent;
    }
    return 405;
  }
}

server {
  listen   443 ssl;
  server_name example.com

  proxy_set_header   Host                 $http_host;
  proxy_set_header   X-Forwarded-Proto    $scheme;
  proxy_set_header   X-Forwarded-For      $remote_addr;
  proxy_redirect     off;

  # keepalive + raven.js is a disaster
  keepalive_timeout 0;

  # use very aggressive timeouts
  proxy_read_timeout 5s;
  proxy_send_timeout 5s;
  send_timeout 5s;
  resolver_timeout 5s;
  client_body_timeout 5s;

  # buffer larger messages
  client_max_body_size 5m;
  client_body_buffer_size 100k;

  location / {
    proxy_pass        http://localhost:9000;

    add_header Strict-Transport-Security "max-age=31536000";
  }
}

Then I added the following line to the default config.yml:
system.url-prefix: 'https://example.com'

And enabled these lines in the sentry.conf.py:
###########
# SSL/TLS #
###########

# If you're using a reverse SSL proxy, you should enable the X-Forwarded-Proto
# header and enable the settings below

SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')
# USE_X_FORWARDED_HOST = True # I added this line, but didn't work either
SESSION_COOKIE_SECURE = True
CSRF_COOKIE_SECURE = True
SOCIAL_AUTH_REDIRECT_IS_HTTPS = True

# End of SSL/TLS settings

What is wrong with this setup?

2

BTW Of course I replaced example.com with my real domain everywhere

3

Found this warning in the web log:
[WARNING] django.security.csrf: Forbidden (Referer checking failed - https://sentry.service.aws.aumio.de/auth/login/sentry/ does not match any trusted origins.): /auth/login/sentry/ (status_code=403 request=<WSGIRequest: POST u'/auth/login/sentry/'>)

Which is also the error message I see when e.g. creating new projects in the Web UI.

4

This sounds similar to https://github.com/getsentry/sentry/issues/2266

I’m guessing you haven’t changed anything else in your config? If you did, can you share as much as you can? I’m also assuming your url-prefix is sentry.service.aws.aumio.de?

5

Well in the config.yml I added the line:
system.url-prefix: 'https://sentry.service.aws.aumio.de'

So I included the scheme as defined in the docs: https://docs.sentry.io/server/config

Besides the three config files mentioned above I haven’t changed anything, correct.

So except for the nginx config via sites-available instead of nginx.conf, it is exactly the setup that is described in the docs…

6

BTW what is the correct way to reload the config files after a change?

In the docs it says “you’d need to rebuild your images ( docker-compose build ) if you ever want to change your configuration settings”.

What I did so far is use
docker-compose build --no-cache web
to rebuild the sentry image and then
docker-compose up -d --force-recreate web
to create a new web container.

Is that’s how it’s supposed to be done?

7

Yeah, this is correct. I was just being a lazy typist, sorry :slight_smile:

As of the latest master of the on-premise repo, this is no longer necessary. Simply restarting the service should load the most recent configuration as it is mounted as a volume now, instead of built into the image. The documented method will do that anyway so it is still correct, just slower.

I think our Nginx docs are a bit outdated. I have a feeling that applying this proxy_header fix would resolve your issue: NGINX Reverse Proxy | NGINX Documentation

8

Thanks! That makes reloading a lot faster :slight_smile:

Except the headers set in the nginx.conf above, what other request headers would you set?

9

Ah, sorry I forgot you shared your Nginx config. Well then I don’t really know why this is getting rejected. Maybe we can use some more logging to see what these values are.

Sorry for all this hassle but can you just log the full HTTP headers that is being sent to Sentry? Not sure if this is easy on the Nginx side but with debug level logging, we should see this from Sentry side: https://docs.sentry.io/server/config/#logging

10

Btw. my suspicion is a mismatch between the Host header and the Referrer header that is passed to Sentry.

11

@BYK are you sure this is true? I e.g. just changed the from email in the config.yml and restarted the web service. But on the web UI the from email is still the old one. Then I shutdown the docker-compose services and rebuild the sentry-onpremise-local image. Started up docker-compose again, but still the from email is the old one and not matching the one in the config.yml. What’s wrong there?

12

Ok, my bad. Did work!

1 Like
13

I just got it to work. But I dont know the exact issue. What I did (besides a few other things) is move the nginx config from sites-available / sites-enabled directly into the nginx.conf. So I guess it was related to the nginx conf. Not sure why it made a difference though or what exact line caused the problem …

14

Maybe it was file permissions?

15

Maybe, but don’t think so since nginx still routed toward the correct upstream server. My guess is that maybe there was some inconsistency / duplicate setting in the nginx.conf and the sentry.conf.

Thanks for your help @BYK!

1 Like