Getting 403 on all POST requests when behind a reverse proxy

rfer · February 18, 2020, 12:42pm

I’ve installed directly from getsentry/onpremise on an EC2 instance on AWS, which is behind a WAF.

Everything works fine when I access and browse Sentry through https://my-sentry.company.com, except POST requests, which are always returning a 403.

I’m pretty sure this has to do with my setup, as tunneling to the EC2 instance and configuring Sentry’s Root URL to localhost and accessing Sentry through localhost works perfectly fine.

Here are my configs:

config.yml

system.url-prefix: 'https://my-sentry.company.com'
system.internal-url-prefix: 'http://web:9000'

sentry.conf.py

##############
# Web Server #
##############

SENTRY_WEB_HOST = "0.0.0.0"
SENTRY_WEB_PORT = 9000
SENTRY_WEB_OPTIONS = {
    "http": "%s:%s" % (SENTRY_WEB_HOST, SENTRY_WEB_PORT),
    "protocol": "uwsgi",
    "uwsgi-socket": None,
    "http-keepalive": True,
    "http-chunked-input": True,
    "memory-report": False,
    # 'workers': 3,  # the number of web workers
}

###########
# SSL/TLS #
###########

# If you're using a reverse SSL proxy, you should enable the X-Forwarded-Proto
# header and enable the settings below

#SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')
#USE_X_FORWARDED_HOST = True
#SESSION_COOKIE_SECURE = True
#CSRF_COOKIE_SECURE = True
#SOCIAL_AUTH_REDIRECT_IS_HTTPS = True

Apache proxy settings:

  <VirtualHost *:80>
  ServerName my-sentry.company.com

  RewriteEngine on
  RewriteCond %{REQUEST_METHOD} ^(TRACE|TRACK)
  RewriteRule .* - [F]

  ProxyPreserveHost On
  RequestHeader set X-Forwarded-Proto "https"
  RequestHeader set X-Request-ID "%{UNIQUE_UUID}e"

  ProxyPass "/" "http://my-sentry:9000/" disableReuse=On
  ProxyPassReverse "/" "http://my-sentry:9000/"

  CustomLog /var/log/httpd/access_eng_sentry.log combined
  ErrorLog /var/log/httpd/error_eng_sentry.log

  {{ macros.require_access_by_ip_whitelisting_on_proxy(client_tla) }}

  # Timeout in seconds
  TimeOut 300

  # Timeout in seconds
  TimeOut 300
</VirtualHost>

Any ideas on what I’m missing?

BYK · February 18, 2020, 1:18pm

Two things that look interesting:

Your Apache config seems only for port 80 but you say you configured it for HTTPS which is over port 443. There’s some mismatch or missing information there.
Since Apache is acting as a reverse-proxy, you should be enabling some (all?) settings under the # SSL/TLS # section. At least pay do what the note on reverse proxies says:
```
# If you're using a reverse SSL proxy, you should enable the X-Forwarded-Proto
# header and enable the settings below
```

rfer · February 18, 2020, 1:35pm

Actually, the way we have it setup is with an ELB in front of the WAF. Https is configured on the ELB, and from the ELB to the WAF it’s http. So:

--- https ---> ELB --- http ---> WAF --- http ---> Sentry Server

I’ve configured the SSL properties on sentry.conf.py as follows, but the behavior is still the same:

SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')
USE_X_FORWARDED_HOST = True
SESSION_COOKIE_SECURE = True
CSRF_COOKIE_SECURE = True
SOCIAL_AUTH_REDIRECT_IS_HTTPS = True

BYK · February 18, 2020, 2:06pm

Do you also set those headers in ELB?

Also, can you share the logs around those 403 errors? I’m fairly sure they are due to CSRF failures but want to be sure before making more assumptions.

rfer · February 18, 2020, 2:48pm

My bad, only the X-Forwarded-Proto is set (on both the ELB, it’s by default.

But I’ve tried the same set of configs, with the USE_X_FORWARDED_HOST set to False, and the result is the same.

This is what I see on the Sentry-Web container logs:

14:46:00 [INFO] sentry.superuser: superuser.request (user_id=1 url=u'https://my-sentry.company.com/api/0/organizations/company/' method=u'PUT' ip_address=u'<IP_ADDRESS>')
10.10.177.243 - - [18/Feb/2020:14:46:00 +0000] "PUT /api/0/organizations/company/ HTTP/1.1" 403 1106 "https://my-sentry.company.com/settings/company/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36"
14:46:00 [INFO] sentry.superuser: superuser.request (user_id=1 url=u'https://my-sentry.company.com/api/1/store/?sentry_key=6caea05ee5e349b183be05895240180d&sentry_version=7' method=u'POST' ip_address=u'<IP_ADDRESS>')
10.10.177.243 - - [18/Feb/2020:14:46:00 +0000] "POST /api/1/store/?sentry_key=6caea05ee5e349b183be05895240180d&sentry_version=7 HTTP/1.1" 200 1171 "https://my-sentry.company.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36"
14:46:00 [INFO] sentry.superuser: superuser.request (user_id=1 url=u'https://my-sentry.company.com/api/1/events/6f48d37931454c6ca9ee96a447f36419/attachments/?sentry_key=6caea05ee5e349b183be05895240180d&sentry_version=7&sentry_client=rrweb' method=u'POST' ip_address=u'<IP_ADDRESS>')
10.10.177.243 - - [18/Feb/2020:14:46:00 +0000] "POST /api/1/events/6f48d37931454c6ca9ee96a447f36419/attachments/?sentry_key=6caea05ee5e349b183be05895240180d&sentry_version=7&sentry_client=rrweb HTTP/1.1" 403 1279 "https://my-sentry.company.com/settings/company/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36"

Are there any other logs that you recommend enabling or looking at?

BYK · February 18, 2020, 5:10pm

Eh, these are quite succinct. I think you may benefit from enabling debug-level logging by following: https://docs.sentry.io/server/config/#logging

rfer · February 18, 2020, 6:01pm

Thanks, I’ll update the logging level.

In the meantime, I’ve done the following experiments:

When I use Sentry through https://my-sentry.company.com, the 403’s response is a:

{"detail":"CSRF Failed: CSRF token missing or incorrect."}

and indeed, the x-csrftoken header on the request is null.

However, when I open a tunnel to the instance and use sentry from http://localhost:9000, the x-csrftoken is not null.

EDIT:

This doesn’t happen for all requests, but it does happen for the following:

https://my-sentry.company.com/api/0/organizations/company/

Full request:

:authority: my-sentry.company.com
:method: PUT
:path: /api/0/organizations/company/
:scheme: https
accept: application/json; charset=utf-8
accept-encoding: gzip, deflate, br
accept-language: en-US,en;q=0.9,es;q=0.8,pt;q=0.7,de;q=0.6,de-DE;q=0.5
content-length: 15
content-type: application/json
cookie: _ga=GA1.2.1463822824.1544037105; source=facebook; utm_campaign_first=Website; utm_medium_first=social; utm_content_first=80859770; __insp_uid=2282350042; km_ai=eBElyy5NH3iSS%2FhI%2F3IGID6p%2Bxk%3D; km_lv=x; __insp_ss=1550862901576; km_ni=abcceo%40gmail.com; loadingTipActions=zoom-out; utm_campaign_last=march newsletter 2019; __cfduid=d81477e3adcbdcdf9b4f450adf694236b1573029302; experimentation_subject_id=ImU3MzczNjU3LTY3MTktNDhmNS1iZDdkLTY0ZjMwMDhiYTQxNiI%3D--8b8efd80ec32787a354889b3e93f4117c4bd755d; __hssrc=1; current_source=facebook; utm_medium_last=social; utm_content_last=116326368; ei_client_id=5e43b83e1acc3f0012fe1d7e; kvcd=1581496382836; __insp_wid=229749723; __insp_nv=false; __insp_targlpu=aHR0cHM6Ly9mZWVkemFpLmNvbS9ibG9nL3RoZS1yaXNreS1idXNpbmVzcy1vZi1idWlsZGluZy15b3VyLW93bi1tYWNoaW5lLWxlYXJuaW5nLXBsYXRmb3JtLz91dG1fY29udGVudD0xMTYzMjYzNjgmdXRtX21lZGl1bT1zb2NpYWwmdXRtX3NvdXJjZT1mYWNlYm9vayZoc3NfY2hhbm5lbD1mYnAtMTg1MzY4MzM4MTYzMjMz; __insp_targlpt=VGhlIFJpc2t5IEJ1c2luZXNzIG9mIEJ1aWxkaW5nIFlvdXIgT3duIE1hY2hpbmUgTGVhcm5pbmcgUGxhdGZvcm0gLSBGZWVkemFp; __insp_pad=1; __insp_sid=2420705137; __insp_slim=1581496401646; lt-session-data="0.ce27149f1705439a341"; lt-pageview-id="0.30d397551705439a343"; sc=mxr2ngDrHUNZhqVsr9bq90gz5dgCEzzOtdasYNj5WbrfvHXQlHh3iaxHuHNXdYqD; sudo="MeJ8Og482V1W:1j47Cm:2djrY5Mw_es2W4-5Cp2J0Yw-WGE"; sentrysid=".eJxNjj9PwzAUxNO6tKICIfERmDJFcf64zggSC1JhMs0WOfFLYyXUOLYpIFXio2NQhm7v_e50dz_zk56F7LriznaVMzBWUpSzIAgwu-SNlR-gxn258qAFEN9csoVxQpVXnmzhib7sM5q84l259sCCsY1SvYRy6d-jGnsQ7PYsvOZNDwfB7gwc7PgVOSsHE_3p0eMbl8O9vx4mDzJGTVNuziI6bjoWCpylJCtaTIEUNW6BA20EB1LTguR8k8QYeCpyhirjTnoeMmRVr9H_cLNVuKC7zsbPR71gSIpBX7A1zmkS5_GGFAw5KfRyakfw-a5Xk05S3-tc9AuA_lz1:1j47Cr:Kxovf20JkN6Jm5CLXqYdKUSyi-Y"; su="sMo198Wht0Nw:1j47Cr:qdEo67QxNM_QQHbtlewB2xBYpps"
origin: https://my-sentry.company.com
referer: https://my-sentry.company.com/settings/company/
sec-fetch-mode: cors
sec-fetch-site: same-origin
user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36
x-csrftoken: null
x-requested-with: XMLHttpRequest

rfer · February 18, 2020, 10:11pm

Quick update, I’m now sure that this has to due with the “sc” cookie used by Sentry to get the CSRF token being marked HttpOnly when I access through https.

Something along the way must be forcing all cookies (including this one to be set as HttpOnly)…

matt · February 18, 2020, 10:52pm

The flag httpOnly doesn’t mean the protocol http vs https. It just means the cookie isn’t accessible via JavaScript. The secure flag means only sent over https.

rfer · February 19, 2020, 8:58am

I know. But Sentry’s frontend does read the csrf token from that cookie.

If the cookie is marked as HttpOnly, then it won’t be able to read it and thus will send a null csrf-token on the POST requests, which will then be invalidated.

rfer · February 19, 2020, 10:23am

For anyone’s reference, the issue was indeed related with the “sc” cookie being marked as HttpOnly on Apache.

Fixing this solved the problem.

Thanks for the support.

simon_aumio · March 18, 2020, 12:44pm

@rfer how did you solve it?

Topic		Replies	Views
Sentry behind reverse proxy On-Premise	10	8723	October 24, 2021
302 redirect while accessing sentry from a reverse proxy on suburl	5	3180	June 22, 2020
CSRF Issue when using SSL via nginx On-Premise	14	10159	March 19, 2020
UserFeedback Form sends data to http:// Link on a https:// page On-Premise	1	1377	August 8, 2018
UserFeedback Form sends data to http:// Link on a https:// page On-Premise	7	2096	August 20, 2018

Getting 403 on all POST requests when behind a reverse proxy

Related Topics