SnubaError and 500 Internal Error

On-Premise
#1

I am running On-Premise docker [Version 10.0.0.dev0], used the install script and then the docker-compose, however when opening Issues, Events, Releases, Dashboards, and Stats, multiple requests return a 500 Internal Server Error.

Some examples:

GET https://sentry.mydomain.com/api/0/organizations/sentry/stats/?since=1573646398&until=1574251198&resolution=1h&stat=received

GET https://sentry.mydomain.com/api/0/organizations/sentry/tags/

GET https://sentry.mydomain.com/api/0/organizations/sentry/events/?statsPeriod=14d

Example Response:
{"detail":"Internal Error","errorId":"96216ebdaabe4ebaaf9698c8f63960c8"}

StackTrace

12:01:44 [INFO] sentry.superuser: superuser.request (user_id=1 url=u'https://sentry.mydomain.com/api/0/organizations/sentry/events-stats/?interval=30m&query=&statsPeriod=28d' method=u'GET' ip_address=u'148.69.164.170')
192.168.160.19 - - [20/Nov/2019:12:01:44 +0000] "GET /api/0/organizations/sentry/releases/ HTTP/1.1" 500 723 "https://sentry.mydomain.com/organizations/sentry/events/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36"
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/sentry/api/base.py", line 89, in handle_exception
    response = super(Endpoint, self).handle_exception(exc)
  File "/usr/local/lib/python2.7/site-packages/rest_framework/views.py", line 434, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/usr/local/lib/python2.7/site-packages/sentry/api/base.py", line 196, in dispatch
    response = handler(request, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/api/endpoints/organization_events_stats.py", line 47, in get
    limit=10000,
  File "/usr/local/lib/python2.7/site-packages/sentry/utils/snuba.py", line 525, in transform_aliases_and_query
    result = dataset_query(**kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/utils/snuba.py", line 1038, in dataset_query
    **kwargs
  File "/usr/local/lib/python2.7/site-packages/sentry/utils/snuba.py", line 755, in raw_query
    return bulk_raw_query([snuba_params], referrer=referrer)[0]
  File "/usr/local/lib/python2.7/site-packages/sentry/utils/snuba.py", line 784, in bulk_raw_query
    query_results = [snuba_query(query_param_list[0])]
  File "/usr/local/lib/python2.7/site-packages/sentry/utils/snuba.py", line 777, in snuba_query
    raise SnubaError(err)
SnubaError: HTTPConnectionPool(host='localhost', port=1218): Max retries exceeded with url: /query (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f5d315cf0d0>: Failed to establish a new connection: [Errno 111] Connection refused',))

Configuration

I am running a nginx reverse proxy with a Let’s Encrypt Companion in docker and then added the necessary env vars for discovery on the Web Pod.

config.yml

###################
# System Settings #
###################

system.secret-key: '************'

system.url-prefix: 'https://sentry.mydomain.com'

################
# File storage #
################

filestore.backend: 'filesystem'
filestore.options:
  location: '/var/lib/sentry/files'

system.internal-url-prefix: 'http://web:9000'
symbolicator.enabled: true
symbolicator.options:
  url: "http://symbolicator:3021"

sentry.config.py

##############
# Web Server #
##############

SENTRY_WEB_HOST = "0.0.0.0"
SENTRY_WEB_PORT = 9000
SENTRY_WEB_OPTIONS = {
    "http": "%s:%s" % (SENTRY_WEB_HOST, SENTRY_WEB_PORT),
    "protocol": "uwsgi",
    # This is needed to prevent https://git.io/fj7Lw
    "uwsgi-socket": None,
    "http-keepalive": True,
    "memory-report": False,
    # 'workers': 3,  # the number of web workers
}

###########
# SSL/TLS #
###########

# If you're using a reverse SSL proxy, you should enable the X-Forwarded-Proto
# header and enable the settings below

SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')
SESSION_COOKIE_SECURE = True
CSRF_COOKIE_SECURE = True
SOCIAL_AUTH_REDIRECT_IS_HTTPS = True

Everything else on these config files are default.

Is there something I am missing?

#2

Looks like it is trying to connect to snuba over localhost which should only happen for development settings. This is set inside the docker-compose.yml file: https://github.com/getsentry/onpremise/blob/master/docker-compose.yml#L22

Could you be overriding that value somehow?

2 Likes
#3

Yes, that was the problem!
Thank you

1 Like
#4

I have a similar problem, do I need to change the environment variable?

environment:
SNUBA: ‘http://snuba-api:1218’

#5

I had same issue, problem was that I use nginx-proxy which needs environment variables on sentry “web” service, adding them removed inherited SNUBA env.

Adding SNUBA: 'http://snuba-api:1218' directly to web server fixed the issue for me

2 Likes
#6

I have that problem too, and I can’t solve it. How can I override snuba configuration in the docker-compose.yml#L22?

#7

I have see the answers, but I still don’t know how to solve it. Please help me!

#8

When I changed the setting to SENTRY_SNUBA = ‘http://snuba-api:1218’,it still has a problem like the following:

SnubaError: HTTPConnectionPool(host='snuba-api', port=1218): Max retries exceeded with url: /query (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fbe08721050>: Failed to establish a new connection: [Errno -2] Name or service not known',))

Can you please let me know how should I fix it?

#9

@heya0817 can you provide more details about your setup and environment?

#10

Hi, BYK, Thank you so much for replying this. We are self-hosting the sentry v10, but we don’t use the docker to install. We use ansible playbook to install the services. We have installed postgres and redis by their respective roles and connect them with sentry role. But I am not very clear if I also need to created separate roles for snuba, clickhouse, kafaka etc and then connect them with sentry.

As for the setting, I copied the example file (https://github.com/getsentry/onpremise/blob/cfaa3683060ee58ca6fe42ab854557fe4194db83/sentry/sentry.conf.example.py) and override with by adding this line: " http://snuba-api:1218/".

#11

@heya0817 - sorry, I know nothing about the Ansible playbook or your particular environment to help you. The error means your Sentry instance cannot reach Snuba using that provided URL. For snuba-api domain to work, you need to set it in your hosts file if you are not using Docker Compose’s internal DNS system (or set up your own DNS resolver that resolves that address to your snuba api machine).

#12

@BYK, I am building sentry with separate containers postgres, redis and sentry(v10.0.0). even i don’t have any attribute related to snuba but getting this error “sentry.utils.snuba.snubaError : HTTPConnectionPool(host=“localhost”, port-1218) max tries exceeded”. How can i solve this error.

#13

@vishnu123sai - Running Snuba, Clickhouse, Kafka, and Zookeeper is not optional. You need to run all these services for Sentry to function properly. That’s why you are getting this error.