Getting 403 on all POST requests when behind a reverse proxy

On-Premise
1

I’ve installed directly from getsentry/onpremise on an EC2 instance on AWS, which is behind a WAF.

Everything works fine when I access and browse Sentry through https://my-sentry.company.com, except POST requests, which are always returning a 403.

I’m pretty sure this has to do with my setup, as tunneling to the EC2 instance and configuring Sentry’s Root URL to localhost and accessing Sentry through localhost works perfectly fine.

Here are my configs:

config.yml

system.url-prefix: 'https://my-sentry.company.com'
system.internal-url-prefix: 'http://web:9000'

sentry.conf.py

##############
# Web Server #
##############

SENTRY_WEB_HOST = "0.0.0.0"
SENTRY_WEB_PORT = 9000
SENTRY_WEB_OPTIONS = {
    "http": "%s:%s" % (SENTRY_WEB_HOST, SENTRY_WEB_PORT),
    "protocol": "uwsgi",
    "uwsgi-socket": None,
    "http-keepalive": True,
    "http-chunked-input": True,
    "memory-report": False,
    # 'workers': 3,  # the number of web workers
}

###########
# SSL/TLS #
###########

# If you're using a reverse SSL proxy, you should enable the X-Forwarded-Proto
# header and enable the settings below

#SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')
#USE_X_FORWARDED_HOST = True
#SESSION_COOKIE_SECURE = True
#CSRF_COOKIE_SECURE = True
#SOCIAL_AUTH_REDIRECT_IS_HTTPS = True

Apache proxy settings:

  <VirtualHost *:80>
  ServerName my-sentry.company.com

  RewriteEngine on
  RewriteCond %{REQUEST_METHOD} ^(TRACE|TRACK)
  RewriteRule .* - [F]

  ProxyPreserveHost On
  RequestHeader set X-Forwarded-Proto "https"
  RequestHeader set X-Request-ID "%{UNIQUE_UUID}e"

  ProxyPass "/" "http://my-sentry:9000/" disableReuse=On
  ProxyPassReverse "/" "http://my-sentry:9000/"

  CustomLog /var/log/httpd/access_eng_sentry.log combined
  ErrorLog /var/log/httpd/error_eng_sentry.log

  {{ macros.require_access_by_ip_whitelisting_on_proxy(client_tla) }}

  # Timeout in seconds
  TimeOut 300

  # Timeout in seconds
  TimeOut 300
</VirtualHost>

Any ideas on what I’m missing?

2

Two things that look interesting:

  1. Your Apache config seems only for port 80 but you say you configured it for HTTPS which is over port 443. There’s some mismatch or missing information there.
  2. Since Apache is acting as a reverse-proxy, you should be enabling some (all?) settings under the # SSL/TLS # section. At least pay do what the note on reverse proxies says:
    # If you're using a reverse SSL proxy, you should enable the X-Forwarded-Proto
# header and enable the settings below
3

Actually, the way we have it setup is with an ELB in front of the WAF. Https is configured on the ELB, and from the ELB to the WAF it’s http. So:

--- https ---> ELB --- http ---> WAF --- http ---> Sentry Server

I’ve configured the SSL properties on sentry.conf.py as follows, but the behavior is still the same:

SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')
USE_X_FORWARDED_HOST = True
SESSION_COOKIE_SECURE = True
CSRF_COOKIE_SECURE = True
SOCIAL_AUTH_REDIRECT_IS_HTTPS = True
4

Do you also set those headers in ELB?

Also, can you share the logs around those 403 errors? I’m fairly sure they are due to CSRF failures but want to be sure before making more assumptions.

5

My bad, only the X-Forwarded-Proto is set (on both the ELB, it’s by default.

But I’ve tried the same set of configs, with the USE_X_FORWARDED_HOST set to False, and the result is the same.

This is what I see on the Sentry-Web container logs:

14:46:00 [INFO] sentry.superuser: superuser.request (user_id=1 url=u'https://my-sentry.company.com/api/0/organizations/company/' method=u'PUT' ip_address=u'<IP_ADDRESS>')
10.10.177.243 - - [18/Feb/2020:14:46:00 +0000] "PUT /api/0/organizations/company/ HTTP/1.1" 403 1106 "https://my-sentry.company.com/settings/company/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36"
14:46:00 [INFO] sentry.superuser: superuser.request (user_id=1 url=u'https://my-sentry.company.com/api/1/store/?sentry_key=6caea05ee5e349b183be05895240180d&sentry_version=7' method=u'POST' ip_address=u'<IP_ADDRESS>')
10.10.177.243 - - [18/Feb/2020:14:46:00 +0000] "POST /api/1/store/?sentry_key=6caea05ee5e349b183be05895240180d&sentry_version=7 HTTP/1.1" 200 1171 "https://my-sentry.company.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36"
14:46:00 [INFO] sentry.superuser: superuser.request (user_id=1 url=u'https://my-sentry.company.com/api/1/events/6f48d37931454c6ca9ee96a447f36419/attachments/?sentry_key=6caea05ee5e349b183be05895240180d&sentry_version=7&sentry_client=rrweb' method=u'POST' ip_address=u'<IP_ADDRESS>')
10.10.177.243 - - [18/Feb/2020:14:46:00 +0000] "POST /api/1/events/6f48d37931454c6ca9ee96a447f36419/attachments/?sentry_key=6caea05ee5e349b183be05895240180d&sentry_version=7&sentry_client=rrweb HTTP/1.1" 403 1279 "https://my-sentry.company.com/settings/company/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36"

Are there any other logs that you recommend enabling or looking at?

6

Eh, these are quite succinct. I think you may benefit from enabling debug-level logging by following: https://docs.sentry.io/server/config/#logging

7

Thanks, I’ll update the logging level.

In the meantime, I’ve done the following experiments:

When I use Sentry through https://my-sentry.company.com, the 403’s response is a:

{"detail":"CSRF Failed: CSRF token missing or incorrect."}

and indeed, the x-csrftoken header on the request is null.

However, when I open a tunnel to the instance and use sentry from http://localhost:9000, the x-csrftoken is not null.

EDIT:

This doesn’t happen for all requests, but it does happen for the following:

https://my-sentry.company.com/api/0/organizations/company/

Full request:

:authority: my-sentry.company.com
:method: PUT
:path: /api/0/organizations/company/
:scheme: https
accept: application/json; charset=utf-8
accept-encoding: gzip, deflate, br
accept-language: en-US,en;q=0.9,es;q=0.8,pt;q=0.7,de;q=0.6,de-DE;q=0.5
content-length: 15
content-type: application/json
cookie: _ga=GA1.2.1463822824.1544037105; source=facebook; utm_campaign_first=Website; utm_medium_first=social; utm_content_first=80859770; __insp_uid=2282350042; km_ai=eBElyy5NH3iSS%2FhI%2F3IGID6p%2Bxk%3D; km_lv=x; __insp_ss=1550862901576; km_ni=abcceo%40gmail.com; loadingTipActions=zoom-out; utm_campaign_last=march newsletter 2019; __cfduid=d81477e3adcbdcdf9b4f450adf694236b1573029302; experimentation_subject_id=ImU3MzczNjU3LTY3MTktNDhmNS1iZDdkLTY0ZjMwMDhiYTQxNiI%3D--8b8efd80ec32787a354889b3e93f4117c4bd755d; __hssrc=1; current_source=facebook; utm_medium_last=social; utm_content_last=116326368; ei_client_id=5e43b83e1acc3f0012fe1d7e; kvcd=1581496382836; __insp_wid=229749723; __insp_nv=false; __insp_targlpu=aHR0cHM6Ly9mZWVkemFpLmNvbS9ibG9nL3RoZS1yaXNreS1idXNpbmVzcy1vZi1idWlsZGluZy15b3VyLW93bi1tYWNoaW5lLWxlYXJuaW5nLXBsYXRmb3JtLz91dG1fY29udGVudD0xMTYzMjYzNjgmdXRtX21lZGl1bT1zb2NpYWwmdXRtX3NvdXJjZT1mYWNlYm9vayZoc3NfY2hhbm5lbD1mYnAtMTg1MzY4MzM4MTYzMjMz; __insp_targlpt=VGhlIFJpc2t5IEJ1c2luZXNzIG9mIEJ1aWxkaW5nIFlvdXIgT3duIE1hY2hpbmUgTGVhcm5pbmcgUGxhdGZvcm0gLSBGZWVkemFp; __insp_pad=1; __insp_sid=2420705137; __insp_slim=1581496401646; lt-session-data="0.ce27149f1705439a341"; lt-pageview-id="0.30d397551705439a343"; sc=mxr2ngDrHUNZhqVsr9bq90gz5dgCEzzOtdasYNj5WbrfvHXQlHh3iaxHuHNXdYqD; sudo="MeJ8Og482V1W:1j47Cm:2djrY5Mw_es2W4-5Cp2J0Yw-WGE"; sentrysid=".eJxNjj9PwzAUxNO6tKICIfERmDJFcf64zggSC1JhMs0WOfFLYyXUOLYpIFXio2NQhm7v_e50dz_zk56F7LriznaVMzBWUpSzIAgwu-SNlR-gxn258qAFEN9csoVxQpVXnmzhib7sM5q84l259sCCsY1SvYRy6d-jGnsQ7PYsvOZNDwfB7gwc7PgVOSsHE_3p0eMbl8O9vx4mDzJGTVNuziI6bjoWCpylJCtaTIEUNW6BA20EB1LTguR8k8QYeCpyhirjTnoeMmRVr9H_cLNVuKC7zsbPR71gSIpBX7A1zmkS5_GGFAw5KfRyakfw-a5Xk05S3-tc9AuA_lz1:1j47Cr:Kxovf20JkN6Jm5CLXqYdKUSyi-Y"; su="sMo198Wht0Nw:1j47Cr:qdEo67QxNM_QQHbtlewB2xBYpps"
origin: https://my-sentry.company.com
referer: https://my-sentry.company.com/settings/company/
sec-fetch-mode: cors
sec-fetch-site: same-origin
user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36
x-csrftoken: null
x-requested-with: XMLHttpRequest
8

Quick update, I’m now sure that this has to due with the “sc” cookie used by Sentry to get the CSRF token being marked HttpOnly when I access through https.

Something along the way must be forcing all cookies (including this one to be set as HttpOnly)…

9

The flag httpOnly doesn’t mean the protocol http vs https. It just means the cookie isn’t accessible via JavaScript. The secure flag means only sent over https.

10

I know. But Sentry’s frontend does read the csrf token from that cookie.

If the cookie is marked as HttpOnly, then it won’t be able to read it and thus will send a null csrf-token on the POST requests, which will then be invalidated.

1 Like
11

For anyone’s reference, the issue was indeed related with the “sc” cookie being marked as HttpOnly on Apache.

Fixing this solved the problem.

Thanks for the support.

1 Like
12

@rfer how did you solve it?

1 Like