CSRF Validation Failed

Hi,

I m using Sentry-on Prem version 20.12.1
After configuring SSL, login to Sentry gives CSRF Validation Failure.

The nginx.conf file is configured as below :

user nginx;
worker_processes 1;

error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;

events {
worker_connections 1024;
}

http {
default_type application/octet-stream;

log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';

access_log /var/log/nginx/access.log main;

sendfile on;
tcp_nopush on;
tcp_nodelay on;
reset_timedout_connection on;

keepalive_timeout 75s;

gzip off;
server_tokens off;

server_names_hash_bucket_size 64;
types_hash_max_size 2048;
types_hash_bucket_size 64;
client_max_body_size 100m;

proxy_http_version 1.1;
proxy_redirect off;
proxy_buffering off;
proxy_next_upstream error timeout invalid_header http_502 http_503 non_idempotent;
proxy_next_upstream_tries 2;

# Remove the Connection header if the client sends it,
# it could be "close" to close a keepalive connection
proxy_set_header Connection '';
# proxy_set_header Host $host;
# proxy_set_header X-Real-IP $remote_addr;
# proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Request-Id $request_id;
# proxy_read_timeout 30s;
proxy_send_timeout 5s;

upstream relay {
	server relay:3000;
}

upstream sentry {
	server web:9000;
}

server {
listen [::]:443;
listen 80;
server_name orchestration-senty-test.dryice.ai;

	ssl_certificate /ssl/bundle.crt;
		ssl_certificate_key /ssl/orchestration-senty-test.dryice.ai.key;
	

	 ssl on;
		 ssl_session_cache  builtin:1000  shared:SSL:10m;
		 ssl_protocols  TLSv1 TLSv1.1 TLSv1.2;
		 ssl_ciphers HIGH:!aNULL:!eNULL:!EXPORT:!CAMELLIA:!DES:!MD5:!PSK:!RC4;
             ssl_prefer_server_ciphers on;

	access_log /var/log/nginx/data-access.log combined;


	location /api/store/ {
		proxy_pass http://relay;
	}
	location ~ ^/api/[1-9]\d*/ {
		proxy_pass http://relay;
	}
	location / {
		proxy_pass http://sentry;
		proxy_set_header X-Real-IP  $remote_addr;
   			proxy_set_header X-Forwarded-For $remote_addr;
		proxy_set_header X-Forwarded-Proto https;
   			#proxy_set_header X-Forwarded-Proto $scheme;
		proxy_redirect http://sentry https://orchestration-senty-test.dryice.ai:9000;
   			proxy_http_version 1.1;
   			proxy_set_header Upgrade $http_upgrade;
   			#proxy_set_header Connection $connection_upgrade;
   			proxy_read_timeout 20d;
   			proxy_buffering off;
	}
	proxy_set_header Host $host;
}

}

Sentry.conf.py file :

##############

Web Server

##############

SENTRY_WEB_HOST = “0.0.0.0”
SENTRY_WEB_PORT = 9000
SENTRY_WEB_OPTIONS = {
“http”: “%s:%s” % (SENTRY_WEB_HOST, SENTRY_WEB_PORT),
“protocol”: “uwsgi”,
# This is needed in order to prevent https://git.io/fj7Lw
“uwsgi-socket”: None,
“so-keepalive”: True,
# Keep this between 15s-75s as that’s what Relay supports
“http-keepalive”: 15,
“http-chunked-input”: True,
# the number of web workers
“workers”: 3,
“threads”: 4,
“memory-report”: False,
# Some stuff so uwsgi will cycle workers sensibly
“max-requests”: 100000,
“max-requests-delta”: 500,
“max-worker-lifetime”: 86400,
# Duplicate options from sentry default just so we don’t get
# bit by sentry changing a default value that we depend on.
“thunder-lock”: True,
“log-x-forwarded-for”: False,
“buffer-size”: 32768,
“limit-post”: 209715200,
“disable-logging”: True,
“reload-on-rss”: 600,
“ignore-sigpipe”: True,
“ignore-write-errors”: True,
“disable-write-exception”: True,
}

###########

SSL/TLS

###########

If you’re using a reverse SSL proxy, you should enable the X-Forwarded-Proto

header and enable the settings below

SECURE_PROXY_SSL_HEADER = (‘HTTP_X_FORWARDED_PROTO’, ‘https’)
SESSION_COOKIE_SECURE = True
CSRF_COOKIE_SECURE = True
SOCIAL_AUTH_REDIRECT_IS_HTTPS = True

End of SSL/TLS settings

Below is the error log i found in sentry_web container :

08:55:42 [WARNING] django.security.csrf: Forbidden (Referer checking failed - https://orchestration-senty-test.dryice.ai:9000/auth/login/sentry/ does not match any trusted origins.): /auth/login/sentry/ (status_code=403 request=<WSGIRequest: POST ‘/auth/login/sentry/’>)

Please suggest as i couldn’t find any topics on forum that could help me solve this issue.

Seems like system.url-prefix is not set to me: https://develop.sentry.dev/config/#general

Hi,

that is done too via UI - admin - settings - set system url

I explicitely put the system.url-prefix in the config.yml but still CSRF validation fails.

system.url-prefix: "https://orchestration-senty-test.dryice.ai:9000"

Is there a reason why you have modified the nginx config so much? My next suspect would be proxy_set_header Host $host; not working somehow (incorrect context or placement?)

Was setting up DNS along with SSL for the sentry

I have now decluttered the nginx.conf file when i found duplicate proxy_header entries. Presently, my nginx.conf looks as below- almost similar to the one i get when downloaded from sentry/getsentry. Please look below and see if any discrepencies exists in the configuration

user nginx;
worker_processes 1;

error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;


events {
	worker_connections 1024;
}


http {
	default_type application/octet-stream;

	log_format main '$remote_addr - $remote_user [$time_local] "$request" '
	'$status $body_bytes_sent "$http_referer" '
	'"$http_user_agent" "$http_x_forwarded_for"';

	access_log /var/log/nginx/access.log main;

	sendfile on;
	tcp_nopush on;
	tcp_nodelay on;
	reset_timedout_connection on;

	keepalive_timeout 75s;

	gzip off;
	server_tokens on;

	server_names_hash_bucket_size 64;
	types_hash_max_size 2048;
	types_hash_bucket_size 64;
	client_max_body_size 100m;

	proxy_http_version 1.1;
	proxy_redirect http://sentry https://orchestration-senty-test.dryice.ai:9000;
	proxy_buffering off;
	proxy_next_upstream error timeout invalid_header http_502 http_503 non_idempotent;
	proxy_next_upstream_tries 2;

	# Remove the Connection header if the client sends it,
	# it could be "close" to close a keepalive connection
	proxy_set_header Connection '';
	proxy_set_header Host $host;
	proxy_set_header X-Real-IP $remote_addr;
	proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
	proxy_set_header X-Forwarded-Proto $scheme;
	proxy_set_header X-Request-Id $request_id;
	proxy_read_timeout 30s;
	proxy_send_timeout 5s;

	upstream relay {
		server relay:3000;
	}

	upstream sentry {
		server web:9000;
	}
	
	
server {
		listen [::]:443;
		listen 80;
		server_name orchestration-senty-test.dryice.ai;

		ssl_certificate /ssl/bundle.crt;
    		ssl_certificate_key /ssl/orchestration-senty-test.dryice.ai.key;
		

		 ssl on;
    		 ssl_session_cache  builtin:1000  shared:SSL:10m;
    		 ssl_protocols  TLSv1 TLSv1.1 TLSv1.2;
    		 ssl_ciphers HIGH:!aNULL:!eNULL:!EXPORT:!CAMELLIA:!DES:!MD5:!PSK:!RC4;
                 ssl_prefer_server_ciphers on;

		access_log /var/log/nginx/data-access.log combined;


		location /api/store/ {
			proxy_pass http://relay;
		}
		location ~ ^/api/[1-9]\d*/ {
			proxy_pass http://relay;
		}
		location / {
			proxy_pass http://sentry;
			
			
		}
		
	}
	
}

Even after this i see CSRF Validation failed.

Logs from the sentry web container

07:56:10 [WARNING] django.security.csrf: Forbidden (Referer checking failed - https://orchestration-senty-test.dryice.ai:9000/auth/login/sentry/ does not match any trusted origins.): /auth/login/sentry/ (status_code=403 request=<WSGIRequest: POST '/auth/login/sentry/'>)
07:57:09 [WARNING] django.security.csrf: Forbidden (Referer checking failed - https://169.38.110.197:9000/auth/login/sentry/ does not match any trusted origins.): /auth/login/sentry/ (status_code=403 request=<WSGIRequest: POST '/auth/login/sentry/'>)

Can you share some of your Nginx logs? Maybe it is about the referrer header manipulation somewhere?

I dont see any logs in nginx related to referrer header .

PFB logs :

2021/06/02 08:23:01 [warn] 1#1: the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /etc/nginx/nginx.conf:72
nginx: [warn] the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /etc/nginx/nginx.conf:72
2021/06/02 08:32:51 [warn] 1#1: the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /etc/nginx/nginx.conf:72
nginx: [warn] the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /etc/nginx/nginx.conf:72
2021/06/03 07:00:56 [warn] 1#1: the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /etc/nginx/nginx.conf:72
nginx: [warn] the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /etc/nginx/nginx.conf:72
2021/06/03 07:20:15 [warn] 1#1: the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /etc/nginx/nginx.conf:72
nginx: [warn] the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /etc/nginx/nginx.conf:72
2021/06/03 07:25:37 [warn] 1#1: the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /etc/nginx/nginx.conf:72
nginx: [warn] the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /etc/nginx/nginx.conf:72
2021/06/03 07:42:32 [warn] 1#1: the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /etc/nginx/nginx.conf:72
nginx: [warn] the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /etc/nginx/nginx.conf:72
2021/06/03 07:49:18 [warn] 1#1: the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /etc/nginx/nginx.conf:72
nginx: [warn] the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /etc/nginx/nginx.conf:72
2021/06/03 07:51:23 [alert] 7#7: *1021 1024 worker_connections are not enough while connecting to upstream, client: 172.18.0.1, server: orchestration-senty-test.dryice.ai, request: "GET /organizations/sentry/projects/ HTTP/1.1", upstream: "https://169.38.110.197:9000/organizations/sentry/projects/", host: "orchestration-senty-test.dryice.ai"
2021/06/03 07:55:13 [warn] 1#1: the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /etc/nginx/nginx.conf:72
nginx: [warn] the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /etc/nginx/nginx.conf:72
2021/06/03 08:05:40 [warn] 1#1: the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /etc/nginx/nginx.conf:95
nginx: [warn] the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /etc/nginx/nginx.conf:95

This is your access log format. If you can get those, we may extract some more information.

2021/06/07 09:40:54 [error] 7#7: *25 connect() failed (111: Connection refused) while connecting to upstream, client: 223.177.170.63, server: orchestration-senty-test.dryice.ai, request: “POST /api/1/envelope/?sentry_key=65321f5f59f84fd081690d8e63221752&sentry_version=7 HTTP/1.1”, upstream: “http://172.18.0.20:3000/api/1/envelope/?sentry_key=65321f5f59f84fd081690d8e63221752&sentry_version=7”, host: “orchestration-senty-test.dryice.ai:9000”, referrer: “https://orchestration-senty-test.dryice.ai:9000/
2021/06/07 09:41:54 [error] 7#7: *36 connect() failed (111: Connection refused) while connecting to upstream, client: 223.177.170.63, server: orchestration-senty-test.dryice.ai, request: “POST /api/1/envelope/?sentry_key=65321f5f59f84fd081690d8e63221752&sentry_version=7 HTTP/1.1”, upstream: “http://172.18.0.20:3000/api/1/envelope/?sentry_key=65321f5f59f84fd081690d8e63221752&sentry_version=7”, host: “orchestration-senty-test.dryice.ai:9000”, referrer: “https://orchestration-senty-test.dryice.ai:9000/
2021/06/07 09:42:07 [error] 7#7: *35 connect() failed (111: Connection refused) while connecting to upstream, client: 223.177.170.63, server: orchestration-senty-test.dryice.ai, request: “POST /api/1/envelope/?sentry_key=65321f5f59f84fd081690d8e63221752&sentry_version=7 HTTP/1.1”, upstream: “http://172.18.0.20:3000/api/1/envelope/?sentry_key=65321f5f59f84fd081690d8e63221752&sentry_version=7”, host: “orchestration-senty-test.dryice.ai:9000”, referrer: “https://orchestration-senty-test.dryice.ai:9000/

I think this would make us understand the issue

Seems like nginx cannot connect to Relay with a connection refused error. Is your Relay server up and running correctly?

You may also see other potential reasons from here: c - What can be the reasons of connection refused errors? - Stack Overflow

after restarting the containers, the only logs i see getting generated is in Sentry web container.

17:38:19 [WARNING] django.security.csrf: Forbidden (Referer checking failed - https://orchestration-senty-test.dryice.ai:9000/auth/login/sentry/ does not match any trusted origins.): /auth/login/sentry/ (status_code=403 request=<WSGIRequest: POST '/auth/login/sentry/'>)
17:39:41 [WARNING] django.security.csrf: Forbidden (Referer checking failed - https://orchestration-senty-test.dryice.ai:9000/auth/login/sentry/ does not match any trusted origins.): /auth/login/sentry/ (status_code=403 request=<WSGIRequest: POST '/auth/login/sentry/'>)
17:39:45 [WARNING] django.security.csrf: Forbidden (Referer checking failed - https://orchestration-senty-test.dryice.ai:9000/auth/login/sentry/ does not match any trusted origins.): /auth/login/sentry/ (status_code=403 request=<WSGIRequest: POST '/auth/login/sentry/'>)
17:39:49 [WARNING] django.security.csrf: Forbidden (Referer checking failed - https://orchestration-senty-test.dryice.ai:9000/auth/login/sentry/ does not match any trusted origins.): /auth/login/sentry/ (status_code=403 request=<WSGIRequest: POST '/auth/login/sentry/'>)
17:43:15 [WARNING] django.security.csrf: Forbidden (Referer checking failed - https://orchestration-senty-test.dryice.ai:9000/auth/login/sentry/ does not match any trusted origins.): /auth/login/sentry/ (status_code=403 request=<WSGIRequest: POST '/auth/login/sentry/'>)
17:44:11 [WARNING] django.security.csrf: Forbidden (Referer checking failed - https://orchestration-senty-test.dryice.ai:9000/auth/login/sentry/ does not match any trusted origins.): /auth/login/sentry/ (status_code=403 request=<WSGIRequest: POST '/auth/login/sentry/'>)

below are the headers from the webpage being sent

Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
Cache-Control: max-age=0
Connection: keep-alive
Content-Length: 141
Content-Type: application/x-www-form-urlencoded
Cookie: sentrysid=gAJ9cQAoWAoAAAB0ZXN0Y29va2llcQFYBgAAAHdvcmtlZHECWAUAAABfbmV4dHEDWB8AAAAvb3JnYW5pemF0aW9ucy9zZW50cnkvcHJvamVjdHMvcQR1Lg:1lqJNM:wCKYyYGw85GNDLOqpb-IZk_NPac; sc=mlcnDPlPce7CF8wNelt61AUg5oqPxV5GLZUo3B4WCcmGDYbmiTFAifRSdknHBJUY
Host: orchestration-senty-test.dryice.ai:9000
Origin: https://orchestration-senty-test.dryice.ai:9000
Referer: https://orchestration-senty-test.dryice.ai:9000/auth/login/sentry/
sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="90", "Google Chrome";v="90"
sec-ch-ua-mobile: ?0
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: same-origin
Sec-Fetch-User: ?1
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36

Response :

Request URL: https://orchestration-senty-test.dryice.ai:9000/auth/login/sentry/
Request Method: POST
Status Code: 403 Forbidden
Remote Address: 169.38.110.197:9000
Referrer Policy: strict-origin-when-cross-origin

I’m not sure if your proxy_set_header Host $host; directive works for the Sentry backend.

I also don’t understand why a browser would send the Origin header when it is on the same page. Either there is some httphttps redirect swapping happening somewhere or that host header forwarding does not work.

This is now set as it was originally provided in nginx.conf file. Did not make any such changes.

Ah, just found out what this is!

This part should be proxy_redirect off; instead.

Did not resolve the issue

Did you use off or oof as the value?

i used off. I mistyped it here.

Then I’m sorry to let you know that I’m out of ideas here.