How to properly debug sentry and find cause for 502 nginx error

Hi :wave:

I’m new to sentry and Nginx, I’m having a lot of trouble finding out what is happening in my instance, and why I’m getting 502 from Nginx.

Where I can find the root of the problem? I try using tail -100 /var/log/nginx/error.log but I don’t see where the problem is…

output from tail /var/log/nginx/error.log:

2021/04/30 17:08:55 [error] 68825#68825: *7 upstream timed out (110: Connection timed out) while connecting to upstream, client: 172.28.132.95, server: mydomain.com, request: "GET / HTTP/1.1", upstream: "https://myip:9000/", host: "mydomain.com"
2021/04/30 17:09:08 [alert] 68825#68825: *17 open socket #9 left in connection 3
2021/04/30 17:09:08 [alert] 68825#68825: *16 open socket #8 left in connection 4
2021/04/30 17:09:08 [alert] 68825#68825: *15 open socket #13 left in connection 5
2021/04/30 17:09:08 [alert] 68825#68825: aborting

The only thing weird that I found was that upstream use my instance IP address and the host is being solved as mydomain.com

my Nginx /etc/nginx/sites-available/default:

    server {
      server_name mydomain.com;
      location / {
        proxy_pass         http://localhost:9000;
        proxy_redirect     off;
        proxy_set_header   Host              $host;
        proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Proto $scheme;
      }
    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/mydomain/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/mydomain/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}
    server {
    if ($host = mydom) {
        return 301 https://$host$request_uri;
    } # managed by Certbot
      listen 80;
      server_name mydomain.com;
    return 404; # managed by Certbot
}

docker-compose.yml (nginx):

      nginx:
    <<: *restart_policy
    ports:
      - '127.0.0.1:$SENTRY_BIND:80/tcp'
    image: "nginx:1.16"
    volumes:
      - type: bind
        read_only: true
        source: ./nginx
        target: /etc/nginx
    depends_on:
      - web
      - relay

system.internal-url-prefix:

system.internal-url-prefix: 'https://mydomain:9000'

sentry.conf.py :

##############
# Web Server #
##############

SENTRY_WEB_HOST = "0.0.0.0"
SENTRY_WEB_PORT = 9000
SENTRY_WEB_OPTIONS = {
    "http": "%s:%s" % (SENTRY_WEB_HOST, SENTRY_WEB_PORT),
    "protocol": "uwsgi",
    # This is needed in order to prevent https://git.io/fj7Lw
    "uwsgi-socket": None,
    "so-keepalive": True,
    # Keep this between 15s-75s as that's what Relay supports
    "http-keepalive": 15,
    "http-chunked-input": True,
    # the number of web workers
    "workers": 3,
    "threads": 4,
    "memory-report": False,
    # Some stuff so uwsgi will cycle workers sensibly
    "max-requests": 100000,
    "max-requests-delta": 500,
    "max-worker-lifetime": 86400,
    # Duplicate options from sentry default just so we don't get
    # bit by sentry changing a default value that we depend on.
    "thunder-lock": True,
    "log-x-forwarded-for": False,
    "buffer-size": 32768,
    "limit-post": 209715200,
    "disable-logging": True,
    "reload-on-rss": 600,
    "ignore-sigpipe": True,
    "ignore-write-errors": True,
    "disable-write-exception": True,
}

###########
# SSL/TLS #
###########

# If you're using a reverse SSL proxy, you should enable the X-Forwarded-Proto
# header and enable the settings below

SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')
SESSION_COOKIE_SECURE = True
CSRF_COOKIE_SECURE = True
SOCIAL_AUTH_REDIRECT_IS_HTTPS = True

Any ideas what I can be missing?

Thanks in advance and kind regards

A 502 indicates unreachable backend so I’d look at web and relay logs:

docker-compose logs web relay

Thank you very much for your answer @BYK!
The problem seems to be with relay_1

relay_1                                     | 2021-05-01T13:56:17Z [rdkafka::client] ERROR: librdkafka: FAIL [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.20.0.11:9092 failed: Connection refused (after 0ms in state CONNECT, 30 identical error(s) suppressed)

relay_1                                     | 2021-05-01T13:56:17Z [rdkafka::client] ERROR: librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): kafka:9092/bootstrap: Connect to ipv4#172.20.0.11:9092 failed: Connection refused (after 0ms in state CONNECT, 30 identical error(s) suppressed)

relay_1                                     | 2021-05-01T13:57:19Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1                                     |   caused by: error sending request for url (http://web:9000/api/0/relays/register/challenge/): operation timed out

Thanks!

1 Like

Heya, no worries! That said these errors indicate an issue with Kafka, not Relay

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.