Internal error in sentry_sdk (Tunnel connection failed: 503 Service Unavailable)

On-Premise
1

Hello,
After upgrade sentry on-promise to v20.11.1, sentry working all right, but I see these errors into container sentry_onpremise_web_1:

sentry_onpremise_web_1 
nginx_1                                     | 172.25.0.1 - - [20/Nov/2020:09:24:15 +0000] "POST /api/0/organizations/sentry/members/50/teams/sentry/ HTTP/1.0" 201 734 "https://sentry-01.*****/settings/sentry/teams/sentry/members/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:83.0) Gecko/20100101 Firefox/83.0" "10.60.4.28, 10.20.0.11"
web_1                                       | Traceback (most recent call last):
web_1                                       |   File "/usr/local/lib/python2.7/site-packages/django/core/handlers/exception.py", line 41, in inner
web_1                                       |     response = get_response(request)
web_1                                       |   File "/usr/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 249, in _legacy_get_response
web_1                                       |     response = self._get_response(request)
web_1                                       |   File "/usr/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 187, in _get_response
web_1                                       |     response = self.process_exception_by_middleware(e, request)
web_1                                       |   File "/usr/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 185, in _get_response
web_1                                       |     response = wrapped_callback(request, *callback_args, **callback_kwargs)
web_1                                       |   File "/usr/local/lib/python2.7/site-packages/sentry_sdk/integrations/django/views.py", line 63, in callback
web_1                                       |     return old_callback(*args, **kwargs)
web_1                                       |   File "/usr/local/lib/python2.7/site-packages/django/views/generic/base.py", line 68, in view
web_1                                       |     return self.dispatch(request, *args, **kwargs)
web_1                                       |   File "/usr/local/lib/python2.7/site-packages/django/views/generic/base.py", line 88, in dispatch
web_1                                       |     return handler(request, *args, **kwargs)
web_1                                       |   File "/usr/local/lib/python2.7/site-packages/sentry/web/frontend/base.py", line 553, in get
web_1                                       |     photo_file = photo.getfile()
web_1                                       |   File "/usr/local/lib/python2.7/site-packages/sentry/models/file.py", line 338, in getfile
web_1                                       |     impl = self._get_chunked_blob(mode, prefetch)
web_1                                       |   File "/usr/local/lib/python2.7/site-packages/sentry/models/file.py", line 330, in _get_chunked_blob
web_1                                       |     delete=delete,
web_1                                       |   File "/usr/local/lib/python2.7/site-packages/sentry/models/file.py", line 472, in __init__
web_1                                       |     self.open()
web_1                                       |   File "/usr/local/lib/python2.7/site-packages/sentry/models/file.py", line 509, in open
web_1                                       |     self.seek(0)
web_1                                       |   File "/usr/local/lib/python2.7/site-packages/sentry/models/file.py", line 565, in seek
web_1                                       |     self._nextidx()
web_1                                       |   File "/usr/local/lib/python2.7/site-packages/sentry/models/file.py", line 495, in _nextidx
web_1                                       |     self._curfile = self._curidx.blob.getfile()
web_1                                       |   File "/usr/local/lib/python2.7/site-packages/sentry/models/file.py", line 299, in getfile
web_1                                       |     return storage.open(self.path)
web_1                                       |   File "/usr/local/lib/python2.7/site-packages/django/core/files/storage.py", line 38, in open
web_1                                       |     return self._open(name, mode)
web_1                                       |   File "/usr/local/lib/python2.7/site-packages/django/core/files/storage.py", line 300, in _open
web_1                                       |     return File(open(self.path(name), mode))
web_1                                       | IOError: [Errno 2] No such file or directory: u'/data/files/18044/31320/ae9cd20df6294d8ea724a34f5fbdfb50'
web_1                                       | 09:24:15 [ERROR] django.request: Internal Server Error: /avatar/d701fcb92eb04223abd8680eb5aa83b4/ (status_code=500 request=<WSGIRequest: GET u'/avatar/d701fcb92eb04223abd8680eb5aa83b4/?s=120'>)
web_1                                       | 09:24:15 [INFO] sentry.superuser: superuser.request (user_id=42 url=u'http://sentry-01.****/api/0/organizations/sentry/members/?query=' method=u'GET' ip_address=u'172.25.0.1')
web_1                                       | Traceback (most recent call last):
web_1                                       |   File "/usr/local/lib/python2.7/site-packages/sentry_sdk/transport.py", line 314, in send_event_wrapper
web_1                                       |     self._send_event(event)
web_1                                       |   File "/usr/local/lib/python2.7/site-packages/sentry_sdk/transport.py", line 234, in _send_event
web_1                                       |     headers={"Content-Type": "application/json", "Content-Encoding": "gzip"},
web_1                                       |   File "/usr/local/lib/python2.7/site-packages/sentry_sdk/transport.py", line 180, in _send_request
web_1                                       |     headers=headers,
web_1                                       |   File "/usr/local/lib/python2.7/site-packages/urllib3/request.py", line 72, in request
web_1                                       |     **urlopen_kw)
web_1                                       |   File "/usr/local/lib/python2.7/site-packages/urllib3/request.py", line 150, in request_encode_body
web_1                                       |     return self.urlopen(method, url, **extra_kw)
web_1                                       |   File "/usr/local/lib/python2.7/site-packages/urllib3/poolmanager.py", line 449, in urlopen
web_1                                       |     return super(ProxyManager, self).urlopen(method, url, redirect=redirect, **kw)
web_1                                       |   File "/usr/local/lib/python2.7/site-packages/urllib3/poolmanager.py", line 324, in urlopen
web_1                                       |     response = conn.urlopen(method, u.request_uri, **kw)
web_1                                       |   File "/usr/local/lib/python2.7/site-packages/urllib3/connectionpool.py", line 667, in urlopen
web_1                                       |     **response_kw)
web_1                                       |   File "/usr/local/lib/python2.7/site-packages/urllib3/connectionpool.py", line 667, in urlopen
web_1                                       |     **response_kw)
web_1                                       |   File "/usr/local/lib/python2.7/site-packages/urllib3/connectionpool.py", line 667, in urlopen
web_1                                       |     **response_kw)
web_1                                       |   File "/usr/local/lib/python2.7/site-packages/urllib3/connectionpool.py", line 638, in urlopen
web_1                                       |     _stacktrace=sys.exc_info()[2])
web_1                                       |   File "/usr/local/lib/python2.7/site-packages/urllib3/util/retry.py", line 399, in increment
web_1                                       |     raise MaxRetryError(_pool, url, error or ResponseError(cause))
web_1                                       | MaxRetryError: HTTPSConnectionPool(host='sentry-01.****', port=443): Max retries exceeded with url: /api/1/store/ (Caused by ProxyError('Cannot connect to proxy.', error('Tunnel connection failed: 503 Service Unavailable',)))
web_1                                       | 09:24:15 [ERROR] sentry_sdk.errors: Internal error in sentry_sdk

And concerned me about the error:
web_1 | MaxRetryError: HTTPSConnectionPool(host='sentry-01.****', port=443): Max retries exceeded with url: /api/1/store/ (Caused by ProxyError('Cannot connect to proxy.', error('Tunnel connection failed: 503 Service Unavailable',)))

My sentry architecture on the host:
nginx-proxy(:443) -> sentry_onpremise_nginx_1(:9000) -> sentry_onpremise_web_1

Any suggestions or comments?
Thanks

2

So it looks like there’s a missing file (possibly an avatar) in the filestore and Sentry raises an exception for this. And while it is trying to report that exception to itself, it tries to contact itself via sentry-01.**** using HTTPS and fails due to some proxying issue.

@untitaker are we just using the system.url-prefix in Sentry internal reporting now? Could there be an alternative to that?

3

You can choose to override your entire internal systems reporting with
a custom DSN:

SENTRY_SDK_CONFIG["relay_dsn"] = "..."

You can additionally set “dsn” to send the event to two dsns.

4

thanks for your quick responses)
It is fixed my problem:

error

web_1 | MaxRetryError: HTTPSConnectionPool(host='sentry-01.****', port=443): Max retries exceeded with url: /api/1/store/ (Caused by ProxyError('Cannot connect to proxy.', error('Tunnel connection failed: 503 Service Unavailable',)))

fix:
SENTRY_SDK_CONFIG[“relay_dsn”] = “…”

I detect another one error. I trying integration senty with jira-server and see this error in log container:

sentry_onpremise_web_1 
web_1                                       | 14:34:25 [INFO] sentry.superuser: superuser.request (user_id=42 url=u'http://sentry-01.*****/api/0/sentry-apps/?status=published' method=u'GET' ip_address=u'172.18.0.1')
web_1                                       | 14:34:25 [INFO] sentry.superuser: superuser.request (user_id=42 url=u'http://sentry-01.*****/api/0/organizations/sentry/plugins/configs/' method=u'GET' ip_address=u'172.18.0.1')
web_1                                       | 14:34:25 [INFO] sentry.superuser: superuser.request (user_id=42 url=u'http://sentry-01.*****/api/0/organizations/sentry/sentry-app-installations/' method=u'GET' ip_address=u'172.18.0.1')
web_1                                       | 14:34:30 [INFO] sentry.superuser: superuser.logged-out (user_id=43 ip_address=u'172.18.0.1')
web_1                                       | 14:34:32 [INFO] sentry.superuser: superuser.request (user_id=42 url=u'http://sentry-01.*****/api/0/organizations/sentry/integrations/?provider_key=jira_server&includeConfig=0' method=u'GET' ip_address=u'172.18.0.1')
web_1                                       | 14:34:32 [INFO] sentry.superuser: superuser.request (user_id=42 url=u'http://sentry-01.*****/api/0/organizations/sentry/config/integrations/?provider_key=jira_server' method=u'GET' ip_address=u'172.18.0.1')
web_1                                       | 14:34:34 [INFO] sentry.superuser: superuser.request (user_id=42 url=u'http://sentry-01.*****/organizations/sentry/integrations/jira_server/setup/' method=u'GET' ip_address=u'172.18.0.1')
web_1                                       | 14:35:13 [INFO] sentry.superuser: superuser.request (user_id=42 url=u'http://sentry-01.*****/organizations/sentry/integrations/jira_server/setup/' method=u'POST' ip_address=u'172.18.0.1')
web_1                                       | 14:35:13 [INFO] sentry.integrations.client: integration.http_response (status_string=u'connection_error' integration=u'jira_server_setup' error=u"HTTPSConnectionPool(host='jira.*****', port=443): Max retries exceeded with url: /plugins/servlet/oauth/request-token (Caused by ProxyError('Cannot connect to proxy.', error('Tunnel connection failed: 503 Service Unavailable',)))")
web_1                                       | 14:35:13 [INFO] sentry.integrations.jira_server: identity.jira-server.request-token (url=u'https://jira.*****' error=ApiHostError(u'Unable to reach host: jira.*****',))
web_1                                       | 14:35:13 [ERROR] sentry.integration.jira_server: pipeline error (error=u'Could not fetch a request token from Jira. Unable to reach host: jira.*****' organization_id=1L provider=u'jira_server')

error from web:

error_web
error_web809×227 15.7 KB

5

It stems from she same issue: your Sentry instance cannot talk to your Jira instance due to some proxy issue. I think you need to set some proxy variables for your Sentry instance so it can talk to the outside world when it needs to.