How to troubleshoot: There is 1 issue with your system configuration

HungPhann · November 2, 2020, 5:04am

Hi,

My Sentry is reporting There is 1 issue with your system configuration from the UI and doesn’t contain any link to the error for details.

When I look into the logs, I found a couple of error messages:

clickhouse_1                   | Logging errors to /var/log/clickhouse-server/clickhouse-server.err.log
kafka_1                        | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/172.22.0.9:2181. Will not attempt to authenticate using SASL (unknown error)
ingest-consumer_1              | 04:27:49 [INFO] batching-kafka-consumer: New partitions assigned: [TopicPartition{topic=ingest-attachments,partition=0,offset=-1001,error=None}, TopicPartition{topic=ingest-events,partition=0,offset=-1001,error=None}, TopicPartition{topic=ingest-transactions,partition=0,offset=-1001,error=None}]
relay_1                        | 2020-11-02T04:27:18Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1                        | 2020-11-02T04:27:18Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1                        | 2020-11-02T04:27:19Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1                        | 2020-11-02T04:27:20Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1                        | 2020-11-02T04:27:22Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1                        | 2020-11-02T04:27:26Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1                        | 2020-11-02T04:27:31Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1                        | 2020-11-02T04:27:39Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1                        |   caused by: Failed to connect to host: No route to host (os error 113)
relay_1                        |   caused by: No route to host (os error 113)
relay_1                        |   caused by: No route to host (os error 113)
postgres_1                     | ERROR:  duplicate key value violates unique constraint "sentry_environmentprojec_project_id_environment_i_91da82f2_uniq"
postgres_1                     | ERROR:  duplicate key value violates unique constraint "sentry_groupsubscription_group_id_user_id_62180a04_uniq"

The logs in relay service look suspecting:

2020-11-02T04:27:18Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
caused by: Failed to connect to host: Failed resolving hostname: no record found for name: web.[MY_DOMAIN]. type: AAAA class: IN
caused by: Failed resolving hostname: no record found for name: web.[MY_DOMAIN]. type: AAAA class: IN
caused by: Failed resolving hostname: no record found for name: web.[MY_DOMAIN]. type: AAAA class: IN
2020-11-02T04:27:18Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
caused by: Failed to connect to host: Failed resolving hostname: no record found for name: web.[MY_DOMAIN]. type: AAAA class: IN
caused by: Failed resolving hostname: no record found for name: web.[MY_DOMAIN]. type: AAAA class: IN
caused by: Failed resolving hostname: no record found for name: web.[MY_DOMAIN]. type: AAAA class: IN
2020-11-02T04:27:19Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
caused by: Failed to connect to host: Failed resolving hostname: no record found for name: web.[MY_DOMAIN]. type: AAAA class: IN
caused by: Failed resolving hostname: no record found for name: web.[MY_DOMAIN]. type: AAAA class: IN
caused by: Failed resolving hostname: no record found for name: web.[MY_DOMAIN]. type: AAAA class: IN
2020-11-02T04:27:20Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
caused by: Failed to connect to host: Failed resolving hostname: no record found for name: web.[MY_DOMAIN]. type: AAAA class: IN
caused by: Failed resolving hostname: no record found for name: web.[MY_DOMAIN]. type: AAAA class: IN
caused by: Failed resolving hostname: no record found for name: web.[MY_DOMAIN]. type: AAAA class: IN
2020-11-02T04:27:22Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
caused by: Failed to connect to host: Failed resolving hostname: no record found for name: web.[MY_DOMAIN]. type: AAAA class: IN
caused by: Failed resolving hostname: no record found for name: web.[MY_DOMAIN]. type: AAAA class: IN
caused by: Failed resolving hostname: no record found for name: web.[MY_DOMAIN]. type: AAAA class: IN
2020-11-02T04:27:26Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
caused by: Failed to connect to host: Failed resolving hostname: no record found for name: web.[MY_DOMAIN]. type: AAAA class: IN
caused by: Failed resolving hostname: no record found for name: web.[MY_DOMAIN]. type: AAAA class: IN
caused by: Failed resolving hostname: no record found for name: web.[MY_DOMAIN]. type: AAAA class: IN
2020-11-02T04:27:31Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
caused by: Failed to connect to host: Failed resolving hostname: no record found for name: web.[MY_DOMAIN]. type: AAAA class: IN
caused by: Failed resolving hostname: no record found for name: web.[MY_DOMAIN]. type: AAAA class: IN
caused by: Failed resolving hostname: no record found for name: web.[MY_DOMAIN]. type: AAAA class: IN
2020-11-02T04:27:39Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
caused by: Failed to connect to host: No route to host (os error 113)
caused by: No route to host (os error 113)
caused by: No route to host (os error 113)

I’ve searched for errors in relay service but found that nobody has resolved this.

BYK · November 9, 2020, 8:26pm

As mentioned in other places, it seems like you have a DNS or network configuration issue on your system.

HungPhann · November 10, 2020, 2:37am

Thanks for your reply.

I’m running Sentry on-premises with docker-compose. So is it possible to have a DNS or network configuration issue with this setup? And how can I resolve this?

Thanks.

BYK · November 23, 2020, 9:37am

Sorry for the late reply,

These issues can still happen with your host system. A certain DNS or firewall setting can cause issues like this. Were you able to solve it in the meantime or still stuck?

HungPhann · November 23, 2020, 10:56am

Thanks for your reply.

Unfortunately, I’m still stuck with this.
Is there anything I should do to troubleshoot the DNS or firewall mis-config? In addition, should Sentry get any trouble with those errors in the relay service?

BYK · November 23, 2020, 11:25am

Ah, actually it may be a configuration issue causing the web service to not run properly, causing this issue.

Can you share the logs for the web service?

HungPhann · November 23, 2020, 11:46am

This is the logs for the web service:

10:50:13 [WARNING] sentry.utils.geo: settings.GEOIP_PATH_MMDB not configured.
/usr/local/lib/python2.7/site-packages/OpenSSL/crypto.py:12: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in a future release.
  from cryptography import x509
10:50:16 [INFO] sentry.plugins.github: apps-not-configured
*** Starting uWSGI 2.0.19.1 (64bit) on [Mon Nov 23 10:50:17 2020] ***
compiled with version: 8.3.0 on 15 October 2020 22:29:34
os: Linux-3.10.0-1062.1.2.el7.x86_64 #1 SMP Mon Sep 30 14:19:46 UTC 2019
nodename: fbdb35456c5f
machine: x86_64
clock source: unix
detected number of CPU cores: 56
current working directory: /
detected binary path: /usr/local/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
your memory page size is 4096 bytes
detected max file descriptor number: 1048576
lock engine: pthread robust mutexes
thunder lock: enabled
uWSGI http bound on 0.0.0.0:9000 fd 6
uwsgi socket 0 bound to TCP address 127.0.0.1:43401 (port auto-assigned) fd 3
Python version: 2.7.16 (default, Oct 17 2019, 07:39:30)  [GCC 8.3.0]
Set PythonHome to /usr/local
Python main interpreter initialized at 0x55c4185a2fd0
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
setting request body buffering size to 65536 bytes
mapped 1924224 bytes (1879 KB) for 12 cores
*** Operational MODE: preforking+threaded ***
spawned uWSGI master process (pid: 18)
spawned uWSGI worker 1 (pid: 23, cores: 4)
spawned uWSGI worker 2 (pid: 24, cores: 4)
spawned uWSGI worker 3 (pid: 25, cores: 4)
spawned uWSGI http 1 (pid: 26)
10:50:18 [WARNING] sentry.utils.geo: settings.GEOIP_PATH_MMDB not configured.
10:50:18 [WARNING] sentry.utils.geo: settings.GEOIP_PATH_MMDB not configured.
10:50:18 [WARNING] sentry.utils.geo: settings.GEOIP_PATH_MMDB not configured.
/usr/local/lib/python2.7/site-packages/OpenSSL/crypto.py:12: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in a future release.
  from cryptography import x509
/usr/local/lib/python2.7/site-packages/OpenSSL/crypto.py:12: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in a future release.
  from cryptography import x509
/usr/local/lib/python2.7/site-packages/OpenSSL/crypto.py:12: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in a future release.
  from cryptography import x509
10:50:21 [INFO] sentry.plugins.github: apps-not-configured
10:50:21 [INFO] sentry.plugins.github: apps-not-configured
WSGI app 0 (mountpoint='') ready in 4 seconds on interpreter 0x55c4185a2fd0 pid: 25 (default app)
WSGI app 0 (mountpoint='') ready in 4 seconds on interpreter 0x55c4185a2fd0 pid: 23 (default app)
10:50:21 [INFO] sentry.plugins.github: apps-not-configured
WSGI app 0 (mountpoint='') ready in 4 seconds on interpreter 0x55c4185a2fd0 pid: 24 (default app)
10:51:05 [WARNING] django.request: Not Found: /_static/1606128617/sentry/js/ads.js (status_code=404 request=<WSGIRequest: GET u'/_static/1606128617/sentry/js/ads.js'>)
10:51:07 [WARNING] django.request: Not Found: /_static/1606128617/sentry/js/ads.js (status_code=404 request=<WSGIRequest: GET u'/_static/1606128617/sentry/js/ads.js'>)
/usr/local/lib/python2.7/site-packages/sentry/models/project.py:180: DeprecationWarning: Project.callsign is deprecated. Use Group.get_short_id() instead.
  "Project.callsign is deprecated. Use Group.get_short_id() instead.", DeprecationWarning
10:51:08 [WARNING] django.request: Not Found: /_static/1606128617/sentry/images/favicon-dark.png (status_code=404 request=<WSGIRequest: GET u'/_static/1606128617/sentry/images/favicon-dark.png'>)
10:52:07 [WARNING] django.request: Not Found: /_static/1606128617/sentry/js/ads.js (status_code=404 request=<WSGIRequest: GET u'/_static/1606128617/sentry/js/ads.js'>)
/usr/local/lib/python2.7/site-packages/sentry/models/project.py:180: DeprecationWarning: Project.callsign is deprecated. Use Group.get_short_id() instead.
  "Project.callsign is deprecated. Use Group.get_short_id() instead.", DeprecationWarning
10:52:12 [INFO] sentry.superuser: superuser.logged-out (user_id=1 ip_address=u'10.128.136.12')
10:52:13 [WARNING] django.request: Not Found: /_static/1606128618/sentry/js/ads.js (status_code=404 request=<WSGIRequest: GET u'/_static/1606128618/sentry/js/ads.js'>)
10:52:14 [WARNING] django.request: Not Found: /_static/1606128618/sentry/images/logos/default-organization-logo.png (status_code=404 request=<WSGIRequest: GET u'/_static/1606128618/sentry/images/logos/default-organization-logo.png'>)
10:52:14 [WARNING] django.request: Not Found: /_static/1606128618/sentry/js/ads.js (status_code=404 request=<WSGIRequest: GET u'/_static/1606128618/sentry/js/ads.js'>)
10:52:15 [WARNING] django.request: Not Found: /_static/1606128618/sentry/images/favicon-dark.png (status_code=404 request=<WSGIRequest: GET u'/_static/1606128618/sentry/images/favicon-dark.png'>)
10:52:17 [INFO] sentry.superuser: superuser.logged-in (user_id=1 ip_address=u'10.128.136.12')
10:52:17 [INFO] sentry.auth: user.auth.success (username=u'jack.phan@my-org.com' organization_id=1L ip_address=u'10.128.136.12')
10:52:17 [INFO] sentry.superuser: superuser.request (user_id=1 url=u'http://10.128.155.67/auth/login/' method=u'GET' ip_address=u'10.128.136.12')
10:52:18 [INFO] sentry.superuser: superuser.request (user_id=1 url=u'http://10.128.155.67/organizations/my-org-sre/issues/' method=u'GET' ip_address=u'10.128.136.12')
10:52:18 [WARNING] django.request: Not Found: /_static/1606128618/sentry/js/ads.js (status_code=404 request=<WSGIRequest: GET u'/_static/1606128618/sentry/js/ads.js'>)
10:52:18 [INFO] sentry.superuser: superuser.request (user_id=1 url=u'http://10.128.155.67/api/0/organizations/?member=1' method=u'GET' ip_address=u'10.128.136.12')
10:52:18 [INFO] sentry.superuser: superuser.request (user_id=1 url=u'http://10.128.155.67/api/0/internal/health/' method=u'GET' ip_address=u'10.128.136.12')
10:52:18 [INFO] sentry.superuser: superuser.request (user_id=1 url=u'http://10.128.155.67/api/0/organizations/my-org-sre/?detailed=0' method=u'GET' ip_address=u'10.128.136.12')
10:52:18 [INFO] sentry.superuser: superuser.request (user_id=1 url=u'http://10.128.155.67/api/0/assistant/?v2' method=u'GET' ip_address=u'10.128.136.12')
10:52:18 [INFO] sentry.superuser: superuser.request (user_id=1 url=u'http://10.128.155.67/api/0/organizations/my-org-sre/projects/?all_projects=1' method=u'GET' ip_address=u'10.128.136.12')
10:52:18 [INFO] sentry.superuser: superuser.request (user_id=1 url=u'http://10.128.155.67/api/0/organizations/my-org-sre/teams/' method=u'GET' ip_address=u'10.128.136.12')
10:52:18 [INFO] sentry.superuser: superuser.request (user_id=1 url=u'http://10.128.155.67/api/0/organizations/my-org-sre/projects/?per_page=50' method=u'GET' ip_address=u'10.128.136.12')
10:52:18 [INFO] sentry.superuser: superuser.request (user_id=1 url=u'http://10.128.155.67/api/0/organizations/my-org-sre/broadcasts/' method=u'GET' ip_address=u'10.128.136.12')
10:52:18 [INFO] sentry.superuser: superuser.request (user_id=1 url=u'http://10.128.155.67/api/0/organizations/my-org-sre/users/' method=u'GET' ip_address=u'10.128.136.12')
10:52:18 [INFO] sentry.superuser: superuser.request (user_id=1 url=u'http://10.128.155.67/api/0/organizations/my-org-sre/tags/?statsPeriod=14d&use_cache=1' method=u'GET' ip_address=u'10.128.136.12')
/usr/local/lib/python2.7/site-packages/sentry/models/project.py:180: DeprecationWarning: Project.callsign is deprecated. Use Group.get_short_id() instead.
  "Project.callsign is deprecated. Use Group.get_short_id() instead.", DeprecationWarning
10:52:18 [INFO] sentry.superuser: superuser.request (user_id=1 url=u'http://10.128.155.67/api/0/organizations/my-org-sre/searches/?use_org_level=1' method=u'GET' ip_address=u'10.128.136.12')
10:52:19 [INFO] sentry.superuser: superuser.request (user_id=1 url=u'http://10.128.155.67/api/0/organizations/my-org-sre/issues/?limit=25&query=is%3Aunresolved&shortIdLookup=1&statsPeriod=14d' method=u'GET' ip_address=u'10.128.136.12')
10:52:19 [INFO] sentry.superuser: superuser.request (user_id=1 url=u'http://10.128.155.67/api/0/organizations/my-org-sre/recent-searches/?query=&type=0&limit=3' method=u'GET' ip_address=u'10.128.136.12')
10:52:19 [INFO] sentry.superuser: superuser.request (user_id=1 url=u'http://10.128.155.67/api/0/organizations/my-org-sre/processingissues/' method=u'GET' ip_address=u'10.128.136.12')

BYK · November 23, 2020, 1:06pm

Where does this IP address belong to: 10.128.136.12.

Do you have any customizations to the default setup we provide in the repo?

HungPhann · November 24, 2020, 2:29am

10.128.136.12 is the address of our jump host server.

Comparing with the default setup, I just configured the SMTP server for sending email and added SENTRY_RELAY_WHITELIST_PK.

BYK · November 24, 2020, 10:59am

You shouldn’t be needing this in recent versions. Anyway, just looking at these, I unfortunately cannot make any useful suggestions. All I can say is for some reason there’s an issue in your docker-compose internal DNS resolver and Relay cannot resolve the web service’s internal IP address.

HungPhann · November 25, 2020, 2:52am

Thanks for spending your time.
Have a nice day

chadwhitacre · December 4, 2020, 9:46pm