How to properly debug sentry and find cause for 502 nginx error

Hi :wave:

I’m new to sentry and Nginx, I’m having a lot of trouble finding out what is happening in my instance, and why I’m getting 502 from Nginx.

Where I can find the root of the problem? I try using tail -100 /var/log/nginx/error.log but I don’t see where the problem is…

output from tail /var/log/nginx/error.log:

2021/04/30 17:08:55 [error] 68825#68825: *7 upstream timed out (110: Connection timed out) while connecting to upstream, client: 172.28.132.95, server: mydomain.com, request: "GET / HTTP/1.1", upstream: "https://myip:9000/", host: "mydomain.com"
2021/04/30 17:09:08 [alert] 68825#68825: *17 open socket #9 left in connection 3
2021/04/30 17:09:08 [alert] 68825#68825: *16 open socket #8 left in connection 4
2021/04/30 17:09:08 [alert] 68825#68825: *15 open socket #13 left in connection 5
2021/04/30 17:09:08 [alert] 68825#68825: aborting

The only thing weird that I found was that upstream use my instance IP address and the host is being solved as mydomain.com

my Nginx /etc/nginx/sites-available/default:

    server {
      server_name mydomain.com;
      location / {
        proxy_pass         http://localhost:9000;
        proxy_redirect     off;
        proxy_set_header   Host              $host;
        proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Proto $scheme;
      }
    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/mydomain/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/mydomain/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}
    server {
    if ($host = mydom) {
        return 301 https://$host$request_uri;
    } # managed by Certbot
      listen 80;
      server_name mydomain.com;
    return 404; # managed by Certbot
}

docker-compose.yml (nginx):

      nginx:
    <<: *restart_policy
    ports:
      - '127.0.0.1:$SENTRY_BIND:80/tcp'
    image: "nginx:1.16"
    volumes:
      - type: bind
        read_only: true
        source: ./nginx
        target: /etc/nginx
    depends_on:
      - web
      - relay

system.internal-url-prefix:

system.internal-url-prefix: 'https://mydomain:9000'

sentry.conf.py :

##############
# Web Server #
##############

SENTRY_WEB_HOST = "0.0.0.0"
SENTRY_WEB_PORT = 9000
SENTRY_WEB_OPTIONS = {
    "http": "%s:%s" % (SENTRY_WEB_HOST, SENTRY_WEB_PORT),
    "protocol": "uwsgi",
    # This is needed in order to prevent https://git.io/fj7Lw
    "uwsgi-socket": None,
    "so-keepalive": True,
    # Keep this between 15s-75s as that's what Relay supports
    "http-keepalive": 15,
    "http-chunked-input": True,
    # the number of web workers
    "workers": 3,
    "threads": 4,
    "memory-report": False,
    # Some stuff so uwsgi will cycle workers sensibly
    "max-requests": 100000,
    "max-requests-delta": 500,
    "max-worker-lifetime": 86400,
    # Duplicate options from sentry default just so we don't get
    # bit by sentry changing a default value that we depend on.
    "thunder-lock": True,
    "log-x-forwarded-for": False,
    "buffer-size": 32768,
    "limit-post": 209715200,
    "disable-logging": True,
    "reload-on-rss": 600,
    "ignore-sigpipe": True,
    "ignore-write-errors": True,
    "disable-write-exception": True,
}

###########
# SSL/TLS #
###########

# If you're using a reverse SSL proxy, you should enable the X-Forwarded-Proto
# header and enable the settings below

SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')
SESSION_COOKIE_SECURE = True
CSRF_COOKIE_SECURE = True
SOCIAL_AUTH_REDIRECT_IS_HTTPS = True

Any ideas what I can be missing?

Thanks in advance and kind regards

A 502 indicates unreachable backend so I’d look at web and relay logs:

docker-compose logs web relay

Thank you very much for your answer @BYK!
The problem seems to be with relay_1

relay_1                                     | 2021-05-01T13:56:17Z [rdkafka::client] ERROR: librdkafka: FAIL [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.20.0.11:9092 failed: Connection refused (after 0ms in state CONNECT, 30 identical error(s) suppressed)

relay_1                                     | 2021-05-01T13:56:17Z [rdkafka::client] ERROR: librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): kafka:9092/bootstrap: Connect to ipv4#172.20.0.11:9092 failed: Connection refused (after 0ms in state CONNECT, 30 identical error(s) suppressed)

relay_1                                     | 2021-05-01T13:57:19Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1                                     |   caused by: error sending request for url (http://web:9000/api/0/relays/register/challenge/): operation timed out

Thanks!

1 Like

Heya, no worries! That said these errors indicate an issue with Kafka, not Relay