SnubaError and 500 Internal Error

I am running On-Premise docker [Version 10.0.0.dev0], used the install script and then the docker-compose, however when opening Issues, Events, Releases, Dashboards, and Stats, multiple requests return a 500 Internal Server Error.

Some examples:

GET https://sentry.mydomain.com/api/0/organizations/sentry/stats/?since=1573646398&until=1574251198&resolution=1h&stat=received

GET https://sentry.mydomain.com/api/0/organizations/sentry/tags/

GET https://sentry.mydomain.com/api/0/organizations/sentry/events/?statsPeriod=14d

Example Response:
{"detail":"Internal Error","errorId":"96216ebdaabe4ebaaf9698c8f63960c8"}

StackTrace

12:01:44 [INFO] sentry.superuser: superuser.request (user_id=1 url=u'https://sentry.mydomain.com/api/0/organizations/sentry/events-stats/?interval=30m&query=&statsPeriod=28d' method=u'GET' ip_address=u'148.69.164.170')
192.168.160.19 - - [20/Nov/2019:12:01:44 +0000] "GET /api/0/organizations/sentry/releases/ HTTP/1.1" 500 723 "https://sentry.mydomain.com/organizations/sentry/events/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36"
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/sentry/api/base.py", line 89, in handle_exception
    response = super(Endpoint, self).handle_exception(exc)
  File "/usr/local/lib/python2.7/site-packages/rest_framework/views.py", line 434, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/usr/local/lib/python2.7/site-packages/sentry/api/base.py", line 196, in dispatch
    response = handler(request, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/api/endpoints/organization_events_stats.py", line 47, in get
    limit=10000,
  File "/usr/local/lib/python2.7/site-packages/sentry/utils/snuba.py", line 525, in transform_aliases_and_query
    result = dataset_query(**kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/utils/snuba.py", line 1038, in dataset_query
    **kwargs
  File "/usr/local/lib/python2.7/site-packages/sentry/utils/snuba.py", line 755, in raw_query
    return bulk_raw_query([snuba_params], referrer=referrer)[0]
  File "/usr/local/lib/python2.7/site-packages/sentry/utils/snuba.py", line 784, in bulk_raw_query
    query_results = [snuba_query(query_param_list[0])]
  File "/usr/local/lib/python2.7/site-packages/sentry/utils/snuba.py", line 777, in snuba_query
    raise SnubaError(err)
SnubaError: HTTPConnectionPool(host='localhost', port=1218): Max retries exceeded with url: /query (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f5d315cf0d0>: Failed to establish a new connection: [Errno 111] Connection refused',))

Configuration

I am running a nginx reverse proxy with a Let’s Encrypt Companion in docker and then added the necessary env vars for discovery on the Web Pod.

config.yml

###################
# System Settings #
###################

system.secret-key: '************'

system.url-prefix: 'https://sentry.mydomain.com'

################
# File storage #
################

filestore.backend: 'filesystem'
filestore.options:
  location: '/var/lib/sentry/files'

system.internal-url-prefix: 'http://web:9000'
symbolicator.enabled: true
symbolicator.options:
  url: "http://symbolicator:3021"

sentry.config.py

##############
# Web Server #
##############

SENTRY_WEB_HOST = "0.0.0.0"
SENTRY_WEB_PORT = 9000
SENTRY_WEB_OPTIONS = {
    "http": "%s:%s" % (SENTRY_WEB_HOST, SENTRY_WEB_PORT),
    "protocol": "uwsgi",
    # This is needed to prevent https://git.io/fj7Lw
    "uwsgi-socket": None,
    "http-keepalive": True,
    "memory-report": False,
    # 'workers': 3,  # the number of web workers
}

###########
# SSL/TLS #
###########

# If you're using a reverse SSL proxy, you should enable the X-Forwarded-Proto
# header and enable the settings below

SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')
SESSION_COOKIE_SECURE = True
CSRF_COOKIE_SECURE = True
SOCIAL_AUTH_REDIRECT_IS_HTTPS = True

Everything else on these config files are default.

Is there something I am missing?

Looks like it is trying to connect to snuba over localhost which should only happen for development settings. This is set inside the docker-compose.yml file: https://github.com/getsentry/onpremise/blob/master/docker-compose.yml#L22

Could you be overriding that value somehow?

1 Like

Yes, that was the problem!
Thank you

1 Like