Internal error in sentry_sdk

pa1op · July 6, 2021, 6:43am

Hello!

I’m running on-premise Sentry version 21.5.1 and getting repetitive errors in my sentry-web and sentry-worker logs. There isn’t really much else than “Connection reset by peer” so I have no idea where they are trying to connect.

At the moment I can receive events from my apps in sentry and basic functionality seems to work, though I have been unable to set up performance monitoring. This isn’t really blocking me, but spam like this in logs disturbs me and I’d like to fix it somehow, though I don’t know where to start from. Do you have any ideas how I could try to solve this?

Error as it is in logs:

6:29:00 [ERROR] sentry_sdk.errors: Internal error in sentry_sdk
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/sentry/../urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python3.6/site-packages/sentry/../urllib3/connectionpool.py", line 343, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.6/site-packages/sentry/../urllib3/connectionpool.py", line 839, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.6/site-packages/sentry/../urllib3/connection.py", line 344, in connect
    ssl_context=context)
  File "/usr/local/lib/python3.6/site-packages/sentry/../urllib3/util/ssl_.py", line 347, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/local/lib/python3.6/ssl.py", line 407, in wrap_socket
    _context=self, _session=session)
  File "/usr/local/lib/python3.6/ssl.py", line 817, in __init__
    self.do_handshake()
  File "/usr/local/lib/python3.6/ssl.py", line 1077, in do_handshake
    self._sslobj.do_handshake()
  File "/usr/local/lib/python3.6/ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/sentry/../sentry_sdk/transport.py", line 340, in send_envelope_wrapper
    self._send_envelope(envelope)
  File "/usr/local/lib/python3.6/site-packages/sentry/../sentry_sdk/transport.py", line 267, in _send_envelope
    endpoint_type="envelope",
  File "/usr/local/lib/python3.6/site-packages/sentry/utils/sdk.py", line 222, in patched_send_request
    return _send_request(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/sentry/../sentry_sdk/transport.py", line 180, in _send_request
    headers=headers,
  File "/usr/local/lib/python3.6/site-packages/sentry/../urllib3/request.py", line 72, in request
    **urlopen_kw)
  File "/usr/local/lib/python3.6/site-packages/sentry/../urllib3/request.py", line 150, in request_encode_body
    return self.urlopen(method, url, **extra_kw)
  File "/usr/local/lib/python3.6/site-packages/sentry/../urllib3/poolmanager.py", line 324, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/usr/local/lib/python3.6/site-packages/sentry/../urllib3/connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/local/lib/python3.6/site-packages/sentry/../urllib3/util/retry.py", line 368, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python3.6/site-packages/sentry/../urllib3/packages/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.6/site-packages/sentry/../urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python3.6/site-packages/sentry/../urllib3/connectionpool.py", line 343, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.6/site-packages/sentry/../urllib3/connectionpool.py", line 839, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.6/site-packages/sentry/../urllib3/connection.py", line 344, in connect
    ssl_context=context)
  File "/usr/local/lib/python3.6/site-packages/sentry/../urllib3/util/ssl_.py", line 347, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/local/lib/python3.6/ssl.py", line 407, in wrap_socket
    _context=self, _session=session)
  File "/usr/local/lib/python3.6/ssl.py", line 817, in __init__
    self.do_handshake()
  File "/usr/local/lib/python3.6/ssl.py", line 1077, in do_handshake
    self._sslobj.do_handshake()
  File "/usr/local/lib/python3.6/ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

BYK · July 6, 2021, 5:23pm

Are you using Sentry with HTTPS?

pa1op · July 7, 2021, 6:59am

I have an nginx Ingress controller in my Kubernetes cluster that forwards all traffic to sentry-nginx as http. I have also tried enabling/disabling Sentry SSL/TLS settings in sentry.conf.py but it didn’t help.

Does this help? I can also provide more information if you need it.

pa1op · July 7, 2021, 7:12am

And just to be clear, these are the settings I’ve been enabling/disabling

    SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')
    SESSION_COOKIE_SECURE = True
    CSRF_COOKIE_SECURE = True
    SOCIAL_AUTH_REDIRECT_IS_HTTPS = True

BYK · July 7, 2021, 8:06am

Have you modified your system.url-prefix and system.internal-url-prefix settings?

It looks like sentry_sdk is trying to connect to Sentry itself and tries it over HTTPS and it fails as the port is not open.

pa1op · July 7, 2021, 9:34am

At the moment system.url-prefix is not set and system.internal-url-prefix: 'http://sentry-web:9000'

BYK · July 7, 2021, 12:14pm

@untitaker do you know why this might be happening then? I’m out of ideas.

untitaker · July 7, 2021, 12:16pm

If the SDK tries to connect via HTTPS then your DSN contains https://. If that’s not correct you need to replace it with http. I’m not entirely clear on whether you’re now running with HTTPS or not.

pa1op · July 7, 2021, 1:49pm

When I go to project settings, Sentry offers me a DSN with HTTPS protocol. I can then use that DSN with my app to forward events to Sentry and they are visible on my dashboards.

Problem in this case is that my logs are cluttered with Connection reset by peer errors which I want to fix.

Trying to change DSN to use HTTP will result in 403 errors in my sentry-nginx logs.

pa1op · July 7, 2021, 3:33pm

So to be clear again, Sentry-SDK for my app works fine. This problem is related to an on-premise installation of Sentry and in particular the sentry-web deployment.

untitaker · July 8, 2021, 9:44am

You got any error logs from relay, nginx or issues in the internal project of Sentry? I believe you’re sometimes sending too large requests, perhaps.

aamarques · August 18, 2021, 5:29pm

I have a behavior quite like this after update to 21.8.0.
system.url-prefix not set and system.internal-url-prefix: 'http://web:9000'
An nothing appears on dashboards

pa1op · September 14, 2021, 6:38am

There are no errors in relay or nginx logs and no errors in internal project. I am able to receive errors from other projects though. They are visible in sentry-nginx logs as:

<ip removed> - - [14/Sep/2021:06:22:59 +0000] "POST /api/36/store/ HTTP/1.1" 200 41 "-" "sentry.python/1.1.0"

Settings for SENTRY_WEB_OPTIONS have been configured like this:

    SENTRY_WEB_OPTIONS = {
        "http": "%s:%s" % (SENTRY_WEB_HOST, SENTRY_WEB_PORT),
        "protocol": "uwsgi",
        # This is needed to prevent https://git.io/fj7Lw
        "uwsgi-socket": None,

        # These ase for proper HTTP/1.1 support from uWSGI
        # Without these it doesn't do keep-alives causing
        # issues with Relay's direct requests.
        "http-keepalive": True,
        "http-chunked-input": True,
        # the number of web workers
        'workers': 3,
        # Turn off memory reporting
        "memory-report": False,
        # Some stuff so uwsgi will cycle workers sensibly
        'max-requests': 100000,
        'max-requests-delta': 500,
        'max-worker-lifetime': 86400,
        # Duplicate options from sentry default just so we don't get
        # bit by sentry changing a default value that we depend on.
        'thunder-lock': True,
        'log-x-forwarded-for': False,
        'buffer-size': 32768,
        'limit-post': 209715200,
        'disable-logging': True,
        'reload-on-rss': 600,
        'ignore-sigpipe': True,
        'ignore-write-errors': True,
        'disable-write-exception': True,
    }

It also seems that I get an identical sentry_sdk.errors: Internal error in sentry_sdk error from one of the workers as well

system · December 13, 2021, 6:38am

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Internal error in sentry_sdk (Tunnel connection failed: 503 Service Unavailable) On-Premise	5	2766	December 1, 2020
SSL errors with Python client On-Premise	6	4806	June 21, 2021
Sentry On-Premise: Internal Error on Projects and Issues On-Premise	5	2984	November 3, 2021
SSL Error when reload page On-Premise	6	2352	September 21, 2021
SnubaError and 500 Internal Error On-Premise	12	5400	August 7, 2020

Internal error in sentry_sdk

Related Topics