Relay errors in fresh new on premise install

On-Premise
1

I did a complete fresh new sentry installation, and got everything works, except… events!

Looking at the logs, when i generate an event i can see it on the web log:

nginx_1 | 138.59.32.18 - - [14/May/2020:13:38:04 +0000] "POST /api/2/store/ HTTP/1.1" 200 41 "-" "sentry.python/0.14.4"

But also, at the same time, the really show some error:

relay_1                    | 2020-05-14T13:38:04Z [relay_server::actors::project_upstream] ERROR: error fetching project states: attempted to send request while not yet authenticated
relay_1                    | 2020-05-14T13:38:06Z [relay_server::actors::project_upstream] ERROR: error fetching project states: attempted to send request while not yet authenticated
relay_1                    | 2020-05-14T13:38:07Z [relay_server::actors::project_upstream] ERROR: error fetching project states: attempted to send request while not yet authenticated
relay_1                    | 2020-05-14T13:38:08Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1                    |   caused by: IO error: Connection reset by peer (os error 104)
relay_1                    |   caused by: IO error: Connection reset by peer (os error 104)
relay_1                    |   caused by: IO error: Connection reset by peer (os error 104)
relay_1                    |   caused by: Connection reset by peer (os error 104)
relay_1                    | 2020-05-14T13:38:10Z [relay_server::actors::project_upstream] ERROR: error fetching project states: attempted to send request while not yet authenticated

I guess the problem that event is not showing up is related to this… how can i fix it?

Full relay logs are here:

2020-05-14T13:52:46Z [rdkafka::client] ERROR: librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): kafka:9092/bootstrap: Connect to ipv4#172.23.0.10:9092 failed: Connection refused (after 2ms in state CONNECT)
2020-05-14T13:52:46Z [rdkafka::client] ERROR: librdkafka: Global error: AllBrokersDown (Local: All broker connections are down): 1/1 brokers are down
2020-05-14T13:52:46Z [rdkafka::client] ERROR: librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): kafka:9092/bootstrap: Connect to ipv4#172.23.0.10:9092 failed: Connection refused (after 10ms in state CONNECT)
2020-05-14T13:52:46Z [rdkafka::client] ERROR: librdkafka: Global error: AllBrokersDown (Local: All broker connections are down): 1/1 brokers are down
2020-05-14T13:52:46Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
caused by: Failed to connect to host: Failed resolving hostname: no record found for name: web type: AAAA class: IN
caused by: Failed resolving hostname: no record found for name: web type: AAAA class: IN
caused by: Failed resolving hostname: no record found for name: web type: AAAA class: IN
2020-05-14T13:52:46Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
caused by: Failed to connect to host: Failed resolving hostname: no record found for name: web type: AAAA class: IN
caused by: Failed resolving hostname: no record found for name: web type: AAAA class: IN
caused by: Failed resolving hostname: no record found for name: web type: AAAA class: IN
2020-05-14T13:52:48Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
caused by: Timeout while waiting for response
2020-05-14T13:52:50Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
caused by: Failed to connect to host: Connection refused (os error 111)
caused by: Connection refused (os error 111)
caused by: Connection refused (os error 111)
2020-05-14T13:52:52Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
caused by: Failed to connect to host: Connection refused (os error 111)
caused by: Connection refused (os error 111)
caused by: Connection refused (os error 111)
2020-05-14T13:52:56Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
caused by: Failed to connect to host: Connection refused (os error 111)
caused by: Connection refused (os error 111)
caused by: Connection refused (os error 111)
2020-05-14T13:53:01Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
caused by: IO error: Connection reset by peer (os error 104)
caused by: IO error: Connection reset by peer (os error 104)
caused by: IO error: Connection reset by peer (os error 104)
caused by: Connection reset by peer (os error 104)
2020-05-14T13:53:09Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
caused by: IO error: Connection reset by peer (os error 104)
caused by: IO error: Connection reset by peer (os error 104)
caused by: IO error: Connection reset by peer (os error 104)
caused by: Connection reset by peer (os error 104)
2020-05-14T13:53:20Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
caused by: IO error: Connection reset by peer (os error 104)
caused by: IO error: Connection reset by peer (os error 104)
caused by: IO error: Connection reset by peer (os error 104)
caused by: Connection reset by peer (os error 104)
2020-05-14T13:53:37Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
caused by: IO error: Connection reset by peer (os error 104)
caused by: IO error: Connection reset by peer (os error 104)
caused by: IO error: Connection reset by peer (os error 104)
caused by: Connection reset by peer (os error 104)
2020-05-14T13:54:03Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
caused by: IO error: Connection reset by peer (os error 104)
caused by: IO error: Connection reset by peer (os error 104)
caused by: IO error: Connection reset by peer (os error 104)
caused by: Connection reset by peer (os error 104)
2020-05-14T13:54:13Z [relay_server::actors::project_upstream] ERROR: error fetching project states: attempted to send request while not yet authenticated
2020-05-14T13:54:14Z [relay_server::actors::project_upstream] ERROR: error fetching project states: attempted to send request while not yet authenticated
2020-05-14T13:54:16Z [relay_server::actors::project_upstream] ERROR: error fetching project states: attempted to send request while not yet authenticated
2020-05-14T13:54:18Z [relay_server::actors::project_upstream] ERROR: error fetching project states: attempted to send request while not yet authenticated
2020-05-14T13:54:22Z [relay_server::actors::project_upstream] ERROR: error fetching project states: attempted to send request while not yet authenticated
2020-05-14T13:54:27Z [relay_server::actors::project_upstream] ERROR: error fetching project states: attempted to send request while not yet authenticated
2020-05-14T13:54:35Z [relay_server::actors::project_upstream] ERROR: error fetching project states: attempted to send request while not yet authenticated
2020-05-14T13:54:41Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
caused by: IO error: Connection reset by peer (os error 104)
caused by: IO error: Connection reset by peer (os error 104)
caused by: IO error: Connection reset by peer (os error 104)
caused by: Connection reset by peer (os error 104)
2020-05-14T13:54:46Z [relay_server::actors::events] ERROR: error processing event: failed to resolve project information
caused by: failed to fetch project state from upstream
2020-05-14T13:54:46Z [relay_server::actors::events] ERROR: error processing event: failed to resolve project information
caused by: failed to fetch project state from upstream
2020-05-14T13:54:58Z [relay_server::actors::project_upstream] ERROR: error fetching project states: attempted to send request while not yet authenticated
2020-05-14T13:54:59Z [relay_server::actors::project_upstream] ERROR: error fetching project states: attempted to send request while not yet authenticated
2020-05-14T13:55:01Z [relay_server::actors::project_upstream] ERROR: error fetching project states: attempted to send request while not yet authenticated
2020-05-14T13:55:03Z [relay_server::actors::project_upstream] ERROR: error fetching project states: attempted to send request while not yet authenticated
2020-05-14T13:55:07Z [relay_server::actors::project_upstream] ERROR: error fetching project states: attempted to send request while not yet authenticated
2020-05-14T13:55:12Z [relay_server::actors::project_upstream] ERROR: error fetching project states: attempted to send request while not yet authenticated
2020-05-14T13:55:20Z [relay_server::actors::project_upstream] ERROR: error fetching project states: attempted to send request while not yet authenticated
2020-05-14T13:55:31Z [relay_server::actors::events] ERROR: error processing event: failed to resolve project information
caused by: failed to fetch project state from upstream
2020-05-14T13:55:31Z [relay_server::actors::events] ERROR: error processing event: failed to resolve project information
caused by: failed to fetch project state from upstream
2020-05-14T13:55:39Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
caused by: IO error: Connection reset by peer (os error 104)
caused by: IO error: Connection reset by peer (os error 104)
caused by: IO error: Connection reset by peer (os error 104)
caused by: Connection reset by peer (os error 104)
2020-05-14T13:56:39Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
caused by: IO error: Connection reset by peer (os error 104)
caused by: IO error: Connection reset by peer (os error 104)
caused by: IO error: Connection reset by peer (os error 104)
caused by: Connection reset by peer (os error 104)
2020-05-14T13:57:39Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
caused by: IO error: Connection reset by peer (os error 104)
caused by: IO error: Connection reset by peer (os error 104)
caused by: IO error: Connection reset by peer (os error 104)
caused by: Connection reset by peer (os error 104)
2020-05-14T13:58:39Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
caused by: IO error: Connection reset by peer (os error 104)
caused by: IO error: Connection reset by peer (os error 104)
caused by: IO error: Connection reset by peer (os error 104)
caused by: Connection reset by peer (os error 104)

About my setup:
CentOS Linux release 8.1.1911 (Core)
Accessing sentry by external IP and port 9000, without any other service than the one that docker put up
No https

2

Looks like Relay cannot talk to Kafka or Sentry and this may be due to IPv6. Can you try adding the following under services -> relay in your docker-compose.yml file and see if it helps:

sysctls:
    - net.ipv6.conf.all.disable_ipv6=1
3

I am experiencing the exact same issue on Ubuntu 18.04.

Adding

sysctls:
    - net.ipv6.conf.all.disable_ipv6=1

under services -> relay in my docker-compose.yml eliminates these DNS-related errors:

caused by: Failed to connect to host: Failed resolving hostname: no record found for name: web type: AAAA class: IN
caused by: Failed resolving hostname: no record found for name: web type: AAAA class: IN
caused by: Failed resolving hostname: no record found for name: web type: AAAA class: IN

However, the main error occurs:

 [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
caused by: Failed to connect to host: Connection refused (os error 111)
caused by: Connection refused (os error 111)
caused by: Connection refused (os error 111)
4

Can you all try this branch and see if it fixes the issues: https://github.com/getsentry/onpremise/pull/499

You’d need to copy my changes in the sentry.conf.example.py into your existing sentry.conf.py file.

5

Thanks for the response!

Unfortunately, I get the same error:

$ docker-compose logs relay
Attaching to sentry_onpremise_relay_1
relay_1                    | 2020-05-20T13:16:24Z [rdkafka::client] ERROR: librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): kafka:9092/bootstrap: Connect to ipv4#172.30.0.10:9092 failed: Connection refused (after 2ms in state CONNECT)
relay_1                    | 2020-05-20T13:16:24Z [rdkafka::client] ERROR: librdkafka: Global error: AllBrokersDown (Local: All broker connections are down): 1/1 brokers are down
relay_1                    | 2020-05-20T13:16:24Z [rdkafka::client] ERROR: librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): kafka:9092/bootstrap: Connect to ipv4#172.30.0.10:9092 failed: Connection refused (after 0ms in state CONNECT)
relay_1                    | 2020-05-20T13:16:24Z [rdkafka::client] ERROR: librdkafka: Global error: AllBrokersDown (Local: All broker connections are down): 1/1 brokers are down
relay_1                    | 2020-05-20T13:16:24Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1                    |   caused by: Failed to connect to host: Failed resolving hostname: no record found for name: web.REDACTED.com. type: AAAA class: IN
relay_1                    |   caused by: Failed resolving hostname: no record found for name: web.REDACTED.com. type: AAAA class: IN
relay_1                    |   caused by: Failed resolving hostname: no record found for name: web.REDACTED.com. type: AAAA class: IN
relay_1                    | 2020-05-20T13:16:24Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1                    |   caused by: Failed to connect to host: Failed resolving hostname: no record found for name: web.REDACTED.com. type: AAAA class: IN
relay_1                    |   caused by: Failed resolving hostname: no record found for name: web.REDACTED.com. type: AAAA class: IN
relay_1                    |   caused by: Failed resolving hostname: no record found for name: web.REDACTED.com. type: AAAA class: IN
relay_1                    | 2020-05-20T13:16:25Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1                    |   caused by: Failed to connect to host: Failed resolving hostname: no record found for name: web.REDACTED.com. type: AAAA class: IN
relay_1                    |   caused by: Failed resolving hostname: no record found for name: web.REDACTED.com. type: AAAA class: IN
relay_1                    |   caused by: Failed resolving hostname: no record found for name: web.REDACTED.com. type: AAAA class: IN
relay_1                    | 2020-05-20T13:16:26Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1                    |   caused by: Failed to connect to host: Failed resolving hostname: no record found for name: web.REDACTED.com. type: AAAA class: IN
relay_1                    |   caused by: Failed resolving hostname: no record found for name: web.REDACTED.com. type: AAAA class: IN
relay_1                    |   caused by: Failed resolving hostname: no record found for name: web.REDACTED.com. type: AAAA class: IN
relay_1                    | 2020-05-20T13:16:29Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1                    |   caused by: Failed to connect to host: Failed resolving hostname: no record found for name: web.REDACTED.com. type: AAAA class: IN
relay_1                    |   caused by: Failed resolving hostname: no record found for name: web.REDACTED.com. type: AAAA class: IN
relay_1                    |   caused by: Failed resolving hostname: no record found for name: web.REDACTED.com. type: AAAA class: IN
relay_1                    | 2020-05-20T13:16:33Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1                    |   caused by: Timeout while waiting for response
relay_1                    | 2020-05-20T13:16:38Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1                    |   caused by: Failed to connect to host: Connection refused (os error 111)
relay_1                    |   caused by: Connection refused (os error 111)
relay_1                    |   caused by: Connection refused (os error 111)
relay_1                    | 2020-05-20T13:16:46Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1                    |   caused by: IO error: Connection reset by peer (os error 104)
relay_1                    |   caused by: IO error: Connection reset by peer (os error 104)
relay_1                    |   caused by: IO error: Connection reset by peer (os error 104)
relay_1                    |   caused by: Connection reset by peer (os error 104)
relay_1                    | 2020-05-20T13:16:57Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1                    |   caused by: IO error: Connection reset by peer (os error 104)
relay_1                    |   caused by: IO error: Connection reset by peer (os error 104)
relay_1                    |   caused by: IO error: Connection reset by peer (os error 104)
relay_1                    |   caused by: Connection reset by peer (os error 104)
relay_1                    | 2020-05-20T13:17:14Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1                    |   caused by: IO error: Connection reset by peer (os error 104)
relay_1                    |   caused by: IO error: Connection reset by peer (os error 104)
relay_1                    |   caused by: IO error: Connection reset by peer (os error 104)
relay_1                    |   caused by: Connection reset by peer (os error 104)
relay_1                    | 2020-05-20T13:17:23Z [relay_server::actors::project_upstream] ERROR: error fetching project states: attempted to send request while not yet authenticated
relay_1                    | 2020-05-20T13:17:24Z [relay_server::actors::project_upstream] ERROR: error fetching project states: attempted to send request while not yet authenticated
relay_1                    | 2020-05-20T13:17:26Z [relay_server::actors::project_upstream] ERROR: error fetching project states: attempted to send request while not yet authenticated
relay_1                    | 2020-05-20T13:17:28Z [relay_server::actors::project_upstream] ERROR: error fetching project states: attempted to send request while not yet authenticated
relay_1                    | 2020-05-20T13:17:32Z [relay_server::actors::project_upstream] ERROR: error fetching project states: attempted to send request while not yet authenticated
relay_1                    | 2020-05-20T13:17:37Z [relay_server::actors::project_upstream] ERROR: error fetching project states: attempted to send request while not yet authenticated
relay_1                    | 2020-05-20T13:17:40Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1                    |   caused by: IO error: Connection reset by peer (os error 104)
relay_1                    |   caused by: IO error: Connection reset by peer (os error 104)
relay_1                    |   caused by: IO error: Connection reset by peer (os error 104)
relay_1                    |   caused by: Connection reset by peer (os error 104)
relay_1                    | 2020-05-20T13:17:45Z [relay_server::actors::project_upstream] ERROR: error fetching project states: attempted to send request while not yet authenticated
relay_1                    | 2020-05-20T13:17:56Z [relay_server::actors::events] ERROR: error processing event: failed to resolve project information
relay_1                    |   caused by: failed to fetch project state from upstream
relay_1                    | 2020-05-20T13:17:56Z [relay_server::actors::events] ERROR: error processing event: failed to resolve project information
relay_1                    |   caused by: failed to fetch project state from upstream
relay_1                    | 2020-05-20T13:17:56Z [relay_server::actors::events] ERROR: error processing event: failed to resolve project information
relay_1                    |   caused by: failed to fetch project state from upstream
relay_1                    | 2020-05-20T13:17:56Z [relay_server::actors::events] ERROR: error processing event: failed to resolve project information
relay_1                    |   caused by: failed to fetch project state from upstream
relay_1                    | 2020-05-20T13:17:56Z [relay_server::actors::events] ERROR: error processing event: failed to resolve project information
relay_1                    |   caused by: failed to fetch project state from upstream
relay_1                    | 2020-05-20T13:17:56Z [relay_server::actors::events] ERROR: error processing event: failed to resolve project information
relay_1                    |   caused by: failed to fetch project state from upstream
relay_1                    | 2020-05-20T13:17:56Z [relay_server::actors::events] ERROR: error processing event: failed to resolve project information
relay_1                    |   caused by: failed to fetch project state from upstream
relay_1                    | 2020-05-20T13:17:56Z [relay_server::actors::events] ERROR: error processing event: failed to resolve project information
relay_1                    |   caused by: failed to fetch project state from upstream
relay_1                    | 2020-05-20T13:17:56Z [relay_server::actors::events] ERROR: error processing event: failed to resolve project information
relay_1                    |   caused by: failed to fetch project state from upstream
relay_1                    | 2020-05-20T13:17:56Z [relay_server::actors::events] ERROR: error processing event: failed to resolve project information
relay_1                    |   caused by: failed to fetch project state from upstream
relay_1                    | 2020-05-20T13:17:56Z [relay_server::actors::events] ERROR: error processing event: failed to resolve project information
relay_1                    |   caused by: failed to fetch project state from upstream
relay_1                    | 2020-05-20T13:17:56Z [relay_server::actors::events] ERROR: error processing event: failed to resolve project information
relay_1                    |   caused by: failed to fetch project state from upstream
relay_1                    | 2020-05-20T13:17:56Z [relay_server::actors::events] ERROR: error processing event: failed to resolve project information
relay_1                    |   caused by: failed to fetch project state from upstream

I tried this with a completely fresh clone (including removing previous volumes and images) on the branch byk/ref/uwsgi, both with and without IPv6 disabled

6

This indicates that your Docker networking configuration has a problem. web here should resolve to the web service in docker-compose, it should not look up in the public internet but on the internal, sentry_onpremise-specific network.

Do you have any modifications to the network config?

7

I do not believe I have modifications to the network config.

Here is my docker config and container/network inspect output:

$ git pull
Already up to date.

$ git status
On branch byk/ref/uwsgi
Your branch is up to date with 'origin/byk/ref/uwsgi'.

nothing to commit, working tree clean

$ git diff docker-compose.yml

$ cat /etc/docker/daemon.json
{
  "log-driver": "json-file"
}

$ docker container inspect sentry_onpremise_relay_1
[
    {
        "Id": "de7c5b11ba08e0064b25ba71a05121ab5079be6c973be2380b326993303586ad",
        "Created": "2020-05-20T14:05:52.661832345Z",
        "Path": "/bin/bash",
        "Args": [
            "/docker-entrypoint.sh",
            "run"
        ],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 105550,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2020-05-20T14:05:56.056447612Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },
        "Image": "sha256:09a4541f6b10120dbfe1623995d02f5af459ba869297e3a148c0a2d7aa9b2435",
        "ResolvConfPath": "/var/lib/docker/containers/de7c5b11ba08e0064b25ba71a05121ab5079be6c973be2380b326993303586ad/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/de7c5b11ba08e0064b25ba71a05121ab5079be6c973be2380b326993303586ad/hostname",
        "HostsPath": "/var/lib/docker/containers/de7c5b11ba08e0064b25ba71a05121ab5079be6c973be2380b326993303586ad/hosts",
        "LogPath": "/var/lib/docker/containers/de7c5b11ba08e0064b25ba71a05121ab5079be6c973be2380b326993303586ad/de7c5b11ba08e0064b25ba71a05121ab5079be6c973be2380b326993303586ad-json.log",
        "Name": "/sentry_onpremise_relay_1",
        "RestartCount": 0,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "docker-default",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "sentry_onpremise_default",
            "PortBindings": {},
            "RestartPolicy": {
                "Name": "unless-stopped",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": [],
            "CapAdd": null,
            "CapDrop": null,
            "Capabilities": null,
            "Dns": null,
            "DnsOptions": null,
            "DnsSearch": null,
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "shareable",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": null,
            "DeviceCgroupRules": null,
            "DeviceRequests": null,
            "KernelMemory": 0,
            "KernelMemoryTCP": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": false,
            "PidsLimit": null,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "Mounts": [
                {
                    "Type": "bind",
                    "Source": "/srv/op9/relay",
                    "Target": "/work/.relay",
                    "ReadOnly": true
                }
            ],
            "MaskedPaths": [
                "/proc/asound",
                "/proc/acpi",
                "/proc/kcore",
                "/proc/keys",
                "/proc/latency_stats",
                "/proc/timer_list",
                "/proc/timer_stats",
                "/proc/sched_debug",
                "/proc/scsi",
                "/sys/firmware"
            ],
            "ReadonlyPaths": [
                "/proc/bus",
                "/proc/fs",
                "/proc/irq",
                "/proc/sys",
                "/proc/sysrq-trigger"
            ]
        },
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/a890af60c3908d876268f296b62f1d7f803eee0f9e3fe6e5031d0ad42e1e9829-init/diff:/var/lib/docker/overlay2/83271fc487bbfcf7413a9cb949153dc6c4eabc865e5f7ff4d9d3d151845831f4/diff:/var/lib/docker/overlay2/c1089fe508709e9134bef7bd24572a8b73c1be9b804a84eb69cbf3da650500d3/diff:/var/lib/docker/overlay2/d11c49e61b272e2b2a8a4b862e12d6e9931302e17a48ca747da2236054d6f07b/diff:/var/lib/docker/overlay2/5725caf7684e2a6f66962881cea9d867555d2bf0ba8584152b3148b524263267/diff:/var/lib/docker/overlay2/697585f69d7b10a745695a92fdfce1d91b0f91756a60507c4162322e6c1d7267/diff:/var/lib/docker/overlay2/189c376721d3d1b443e8a368d9303982e2554f4556f43fe04d2c50c115b24b00/diff:/var/lib/docker/overlay2/3a929ca4029ad189ffa135ad17038605ac6ed2061bc0b0ea06d3051e3a325446/diff",
                "MergedDir": "/var/lib/docker/overlay2/a890af60c3908d876268f296b62f1d7f803eee0f9e3fe6e5031d0ad42e1e9829/merged",
                "UpperDir": "/var/lib/docker/overlay2/a890af60c3908d876268f296b62f1d7f803eee0f9e3fe6e5031d0ad42e1e9829/diff",
                "WorkDir": "/var/lib/docker/overlay2/a890af60c3908d876268f296b62f1d7f803eee0f9e3fe6e5031d0ad42e1e9829/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/srv/op9/relay",
                "Destination": "/work/.relay",
                "Mode": "",
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Type": "volume",
                "Name": "c6fdbbea0a7b91dd3171750430b56a42d9b1f14d20f87847d21e898ea35e1e78",
                "Source": "/var/lib/docker/volumes/c6fdbbea0a7b91dd3171750430b56a42d9b1f14d20f87847d21e898ea35e1e78/_data",
                "Destination": "/etc/relay",
                "Driver": "local",
                "Mode": "",
                "RW": true,
                "Propagation": ""
            },
            {
                "Type": "volume",
                "Name": "b43a5d00761e10d41d825e09ff748fb5372c289707de604d558ce5f58bb5987d",
                "Source": "/var/lib/docker/volumes/b43a5d00761e10d41d825e09ff748fb5372c289707de604d558ce5f58bb5987d/_data",
                "Destination": "/work",
                "Driver": "local",
                "Mode": "",
                "RW": true,
                "Propagation": ""
            }
        ],
        "Config": {
            "Hostname": "de7c5b11ba08",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "3000/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "RELAY_UID=10001",
                "RELAY_GID=10001"
            ],
            "Cmd": [
                "run"
            ],
            "Image": "getsentry/relay:latest",
            "Volumes": {
                "/etc/relay": {},
                "/work": {}
            },
            "WorkingDir": "/work",
            "Entrypoint": [
                "/bin/bash",
                "/docker-entrypoint.sh"
            ],
            "OnBuild": null,
            "Labels": {
                "com.docker.compose.config-hash": "18d50831cc59fc36392fe1ddda48ef91c21531dcaf5dcf29ce28780bf8a6a251",
                "com.docker.compose.container-number": "1",
                "com.docker.compose.oneoff": "False",
                "com.docker.compose.project": "sentry_onpremise",
                "com.docker.compose.project.config_files": "docker-compose.yml,docker-compose.override.yml",
                "com.docker.compose.project.working_dir": "/srv/op9",
                "com.docker.compose.service": "relay",
                "com.docker.compose.version": "1.25.5"
            }
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "0e7900c05d7523533f8ef37de1c85c0a09fd329c47ccb1bed905bfb7ee1fb3e4",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {
                "3000/tcp": null
            },
            "SandboxKey": "/var/run/docker/netns/0e7900c05d75",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "sentry_onpremise_default": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": [
                        "relay",
                        "de7c5b11ba08"
                    ],
                    "NetworkID": "1ed243b181718d686a1757dc58cd1b850b6147b0fe928b5c8114548fd98e9b56",
                    "EndpointID": "458b767d1c2f76f07db1cd3e2cf8686fad9a45f531ef33ddeddb4ce3f03fb35f",
                    "Gateway": "192.168.48.1",
                    "IPAddress": "192.168.48.11",
                    "IPPrefixLen": 20,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:c0:a8:30:0b",
                    "DriverOpts": null
                }
            }
        }
    }
]

$ docker network inspect 1ed243b181718d686a1757dc58cd1b850b6147b0fe928b5c8114548fd98e9b56
[
    {
        "Name": "sentry_onpremise_default",
        "Id": "1ed243b181718d686a1757dc58cd1b850b6147b0fe928b5c8114548fd98e9b56",
        "Created": "2020-05-20T10:05:00.625345407-04:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "192.168.48.0/20",
                    "Gateway": "192.168.48.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": true,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "14bef6b2e6e3e4d84c1e973d37e3f217ace765246d0206447a0575e3e75d1a68": {
                "Name": "sentry_onpremise_postgres_1",
                "EndpointID": "869aa2e8f498f38f3d1f4ff3f4fa5c4f7a3654e0694bd9293aaa6ed4f33fe8c8",
                "MacAddress": "02:42:c0:a8:30:08",
                "IPv4Address": "192.168.48.8/20",
                "IPv6Address": ""
            },
            "2d16e7f97f12cb8b04e766aa73b4e36d275965cea4c1944929e66fd335778131": {
                "Name": "sentry_onpremise_post-process-forwarder_1",
                "EndpointID": "b935130ed94b81bda019fa40ca29608c7eb2a46c07a8df32ce42239967d032d4",
                "MacAddress": "02:42:c0:a8:30:14",
                "IPv4Address": "192.168.48.20/20",
                "IPv6Address": ""
            },
            "3d9eaec3dd9ae7af231a148304b00c33b2d127517fdca1ec7710c515840d824a": {
                "Name": "sentry_onpremise_snuba-cleanup_1",
                "EndpointID": "323a4017778c41d07142f47d9d54485fa397089f1c2636d914573ffdfe45d7e3",
                "MacAddress": "02:42:c0:a8:30:10",
                "IPv4Address": "192.168.48.16/20",
                "IPv6Address": ""
            },
            "4771ad16d226828ccdf580886178f63a293cc7b517fd0a6bc38f87dbd2161e93": {
                "Name": "sentry_onpremise_kafka_1",
                "EndpointID": "fe009d04b8f0fd6d52b2cf4d37fafd7258e192b6c42c093966130d12bd762dbe",
                "MacAddress": "02:42:c0:a8:30:0a",
                "IPv4Address": "192.168.48.10/20",
                "IPv6Address": ""
            },
            "4ec07d30309a75e879c216c6dcf530facbec615433c9f289e9db00463f79fa09": {
                "Name": "sentry_onpremise_worker_1",
                "EndpointID": "6bd11c8554d79762531ff663c0ff63e7fb5d68b0d558a9f87ac4253fda2fff22",
                "MacAddress": "02:42:c0:a8:30:11",
                "IPv4Address": "192.168.48.17/20",
                "IPv6Address": ""
            },
            "702d611f8ed049d364ed2a0684c960f1b2bfe864eb93c50340da7481286319ed": {
                "Name": "sentry_onpremise_web_1",
                "EndpointID": "4a311cde0d0c4c63874da8bef2755091a7fe5c305ceb6e2c544a14632a8e3857",
                "MacAddress": "02:42:c0:a8:30:12",
                "IPv4Address": "192.168.48.18/20",
                "IPv6Address": ""
            },
            "7622b7dc72d868294ddcea3c3fac1c294e3dbd234615fdf06662ec6a192f8d52": {
                "Name": "sentry_onpremise_symbolicator-cleanup_1",
                "EndpointID": "a9fe56620b75c7cdf66cc4b23fb788373d43738106c1ff5f44f29149b1015c5d",
                "MacAddress": "02:42:c0:a8:30:09",
                "IPv4Address": "192.168.48.9/20",
                "IPv6Address": ""
            },
            "775924bf80c30467e63516df9d083e84178f6f13f812b5dc72537a67bfb45425": {
                "Name": "sentry_onpremise_snuba-api_1",
                "EndpointID": "40534f7f5737a121d5f917905f4090e96d9b4d9878a2315ae730fdcf872cff20",
                "MacAddress": "02:42:c0:a8:30:0c",
                "IPv4Address": "192.168.48.12/20",
                "IPv6Address": ""
            },
            "7cefebb776676447e7a76fc9f5c7364e24ae646b2567449dfa700413cb007f67": {
                "Name": "sentry_onpremise_symbolicator_1",
                "EndpointID": "5931194f794a7b46c3e605d464d2f7600205522bf3131f98d8904ea5c3761a35",
                "MacAddress": "02:42:c0:a8:30:05",
                "IPv4Address": "192.168.48.5/20",
                "IPv6Address": ""
            },
            "8b6a138747b31006cd26082f5d33267e34d6c7217c30afc240e0faddfbc15367": {
                "Name": "sentry_onpremise_smtp_1",
                "EndpointID": "2a8ab5f8d33eb36cdb09bb6686d1061f85c47c0c67fdc51531ae1f7af2982b67",
                "MacAddress": "02:42:c0:a8:30:07",
                "IPv4Address": "192.168.48.7/20",
                "IPv6Address": ""
            },
            "95e3d0183458cd0b3fadba54908673787f29d0763e26d23662559a5b630a9bf1": {
                "Name": "sentry_onpremise_snuba-outcomes-consumer_1",
                "EndpointID": "4bb3114020b6ea7588cd087ed216f92e5cf12a0efa0a2aff48626ded9015bdd8",
                "MacAddress": "02:42:c0:a8:30:0d",
                "IPv4Address": "192.168.48.13/20",
                "IPv6Address": ""
            },
            "960f65dba5585dea07f7d97607acbb861be24eb424f996cf69ca3eafd4a9484d": {
                "Name": "sentry_onpremise_memcached_1",
                "EndpointID": "ee59aca4e947ecd2da9e5cf6a63a9c6b4303f13c85ca4782d152a43e0c5164df",
                "MacAddress": "02:42:c0:a8:30:02",
                "IPv4Address": "192.168.48.2/20",
                "IPv6Address": ""
            },
            "9e654f09cfa28407f82aecc29eaf2894f8dd720805223844db6d57616012363e": {
                "Name": "sentry_onpremise_nginx_1",
                "EndpointID": "f1661772038743e5e2cf3a96c15532e00e0000b9fb18a3a549e8feed8d17f9bf",
                "MacAddress": "02:42:c0:a8:30:17",
                "IPv4Address": "192.168.48.23/20",
                "IPv6Address": ""
            },
            "a309fdd8df23e7c3bb79808aebe40b36ea0655d3755e26588a66c917bb45bec2": {
                "Name": "sentry_onpremise_cron_1",
                "EndpointID": "37c6cde47d832c1d85e6d3da3cd64f2b775ac5eefbd17adef879186ffa8b4fdc",
                "MacAddress": "02:42:c0:a8:30:15",
                "IPv4Address": "192.168.48.21/20",
                "IPv6Address": ""
            },
            "a675e867b7805a6afaea6300284ac8871b873e145a92ca0e9ad9b928cb30aea7": {
                "Name": "sentry_onpremise_ingest-consumer_1",
                "EndpointID": "0a5fdb083cba42ec047a45218d029ebdb53d4d3128b60c3c291b0a2be877a421",
                "MacAddress": "02:42:c0:a8:30:16",
                "IPv4Address": "192.168.48.22/20",
                "IPv6Address": ""
            },
            "b26ce70a974c388a9643ccd301379f9a4acf8ae53b099711c22ec0260d8fabba": {
                "Name": "sentry_onpremise_redis_1",
                "EndpointID": "14dc461bff6eb0089b18c37a70984d2403fe5f58620d13592e29c378ce03735e",
                "MacAddress": "02:42:c0:a8:30:04",
                "IPv4Address": "192.168.48.4/20",
                "IPv6Address": ""
            },
            "c6f2d2f321004b19f1b3aa033116a06c7b361d50cc5abaa516b53495bb41def8": {
                "Name": "sentry_onpremise_zookeeper_1",
                "EndpointID": "6c55f6d3f1448296fe211a309b96244b4456c27f729fec6209ec7f1fd113683e",
                "MacAddress": "02:42:c0:a8:30:03",
                "IPv4Address": "192.168.48.3/20",
                "IPv6Address": ""
            },
            "d1082b16c28e145e1ec2abe6e4429e71c78ab8b45e969502f64b8130f01e11fa": {
                "Name": "sentry_onpremise_sentry-cleanup_1",
                "EndpointID": "779eced8b5a5f6c30ff8be9463c06aa0a82c2ee4d99ecf9cb1fd0eb738ecf7ad",
                "MacAddress": "02:42:c0:a8:30:13",
                "IPv4Address": "192.168.48.19/20",
                "IPv6Address": ""
            },
            "d1569b9e13f6df861c07486e35614e4ba4c2399fc2bcee338a8086359ba21b2b": {
                "Name": "sentry_onpremise_snuba-consumer_1",
                "EndpointID": "770c99cc6b2d7cb3127471ec1e0c79159a9b53eb86ad1adff806c25ece94beca",
                "MacAddress": "02:42:c0:a8:30:0f",
                "IPv4Address": "192.168.48.15/20",
                "IPv6Address": ""
            },
            "de7c5b11ba08e0064b25ba71a05121ab5079be6c973be2380b326993303586ad": {
                "Name": "sentry_onpremise_relay_1",
                "EndpointID": "458b767d1c2f76f07db1cd3e2cf8686fad9a45f531ef33ddeddb4ce3f03fb35f",
                "MacAddress": "02:42:c0:a8:30:0b",
                "IPv4Address": "192.168.48.11/20",
                "IPv6Address": ""
            },
            "e3d7f76ee8adcc5065fcc65ed7a5b22eb383cd78731724e41c9f6bc73bbb6fab": {
                "Name": "sentry_onpremise_snuba-replacer_1",
                "EndpointID": "a8d38f7f4ee09308718400d6cde88f49cd4c5984fef2caf00ba1edb27bb3214b",
                "MacAddress": "02:42:c0:a8:30:0e",
                "IPv4Address": "192.168.48.14/20",
                "IPv6Address": ""
            },
            "f05dce0f4e3cdfe10b83b2b736f4dbe5d04369b7df73a2c9438b712d5fcc5b65": {
                "Name": "sentry_onpremise_clickhouse_1",
                "EndpointID": "13d60a3ef80506ecda66a47d0604d25736f8ccee48c1c3ece2453974a60e3afa",
                "MacAddress": "02:42:c0:a8:30:06",
                "IPv4Address": "192.168.48.6/20",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {
            "com.docker.compose.network": "default",
            "com.docker.compose.project": "sentry_onpremise",
            "com.docker.compose.version": "1.25.5"
        }
    }
]
8

This seems similar to the issue reported here: https://github.com/docker/for-linux/issues/179

Is it possible that you have some DNS resolvers listed in your /etc/resolv.conf that is interfering with Docker’s embedded DNS?

9

Still having the same problem.

Tried to reser my resolv.conf to use only the google dns, but still failing with the last version.
I can resolve the “web” hostname inside the relay container without problem

Processing triggers for libc-bin (2.24-11+deb9u4) ...
root@10256c8e4653:/work# ping web
PING web (172.29.0.23) 56(84) bytes of data.
64 bytes from sentry_onpremise_web_1.sentry_onpremise_default (172.29.0.23): icmp_seq=1 ttl=64 time=0.050 ms

Relay logs

relay_1                    | 2020-06-17T12:06:59Z [rdkafka::client] ERROR: librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): kafka:9092/bootstrap: Connect to ipv4#172.29.0.10:9092 failed: Connection refused (after 7ms in state CONNECT)
relay_1                    | 2020-06-17T12:06:59Z [rdkafka::client] ERROR: librdkafka: Global error: AllBrokersDown (Local: All broker connections are down): 1/1 brokers are down
relay_1                    | 2020-06-17T12:06:59Z [rdkafka::client] ERROR: librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): kafka:9092/bootstrap: Connect to ipv4#172.29.0.10:9092 failed: Connection refused (after 2ms in state CONNECT)
relay_1                    | 2020-06-17T12:06:59Z [rdkafka::client] ERROR: librdkafka: Global error: AllBrokersDown (Local: All broker connections are down): 1/1 brokers are down
relay_1                    | 2020-06-17T12:06:59Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1                    |   caused by: Failed to connect to host: Failed resolving hostname: no record found for name: web type: AAAA class: IN
relay_1                    |   caused by: Failed resolving hostname: no record found for name: web type: AAAA class: IN
relay_1                    |   caused by: Failed resolving hostname: no record found for name: web type: AAAA class: IN
relay_1                    | 2020-06-17T12:06:59Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1                    |   caused by: Failed to connect to host: Failed resolving hostname: no record found for name: web type: AAAA class: IN
relay_1                    |   caused by: Failed resolving hostname: no record found for name: web type: AAAA class: IN
relay_1                    |   caused by: Failed resolving hostname: no record found for name: web type: AAAA class: IN
relay_1                    | 2020-06-17T12:07:01Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1                    |   caused by: Timeout while waiting for response
relay_1                    | 2020-06-17T12:07:03Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1                    |   caused by: Failed to connect to host: Connection refused (os error 111)
relay_1                    |   caused by: Connection refused (os error 111)
relay_1                    |   caused by: Connection refused (os error 111)
relay_1                    | 2020-06-17T12:07:05Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1                    |   caused by: Failed to connect to host: Connection refused (os error 111)
relay_1                    |   caused by: Connection refused (os error 111)
relay_1                    |   caused by: Connection refused (os error 111)
relay_1                    | 2020-06-17T12:07:11Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1                    |   caused by: IO error: Connection reset by peer (os error 104)
relay_1                    |   caused by: IO error: Connection reset by peer (os error 104)
relay_1                    |   caused by: IO error: Connection reset by peer (os error 104)
relay_1                    |   caused by: Connection reset by peer (os error 104)
relay_1                    | 2020-06-17T12:07:16Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1                    |   caused by: IO error: Connection reset by peer (os error 104)
relay_1                    |   caused by: IO error: Connection reset by peer (os error 104)
relay_1                    |   caused by: IO error: Connection reset by peer (os error 104)
relay_1                    |   caused by: Connection reset by peer (os error 104)
relay_1                    | 2020-06-17T12:07:24Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1                    |   caused by: IO error: Connection reset by peer (os error 104)
relay_1                    |   caused by: IO error: Connection reset by peer (os error 104)
relay_1                    |   caused by: IO error: Connection reset by peer (os error 104)
relay_1                    |   caused by: Connection reset by peer (os error 104)
relay_1                    | 2020-06-17T12:07:35Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1                    |   caused by: Server disconnected
relay_1                    |   caused by: Server disconnected
relay_1                    | 2020-06-17T12:07:52Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1                    |   caused by: IO error: Connection reset by peer (os error 104)
relay_1                    |   caused by: IO error: Connection reset by peer (os error 104)
relay_1                    |   caused by: IO error: Connection reset by peer (os error 104)
relay_1                    |   caused by: Connection reset by peer (os error 104)
relay_1                    | 2020-06-17T12:08:18Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1                    |   caused by: IO error: Connection reset by peer (os error 104)
relay_1                    |   caused by: IO error: Connection reset by peer (os error 104)
relay_1                    |   caused by: IO error: Connection reset by peer (os error 104)
relay_1                    |   caused by: Connection reset by peer (os error 104)
relay_1                    | 2020-06-17T12:08:57Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1                    |   caused by: IO error: Connection reset by peer (os error 104)
relay_1                    |   caused by: IO error: Connection reset by peer (os error 104)
relay_1                    |   caused by: IO error: Connection reset by peer (os error 104)
relay_1                    |   caused by: Connection reset by peer (os error 104)
relay_1                    | 2020-06-17T12:09:54Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1                    |   caused by: IO error: Connection reset by peer (os error 104)
relay_1                    |   caused by: IO error: Connection reset by peer (os error 104)
relay_1                    |   caused by: IO error: Connection reset by peer (os error 104)
relay_1                    |   caused by: Connection reset by peer (os error 104)
relay_1                    | 2020-06-17T12:10:54Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1                    |   caused by: IO error: Connection reset by peer (os error 104)
relay_1                    |   caused by: IO error: Connection reset by peer (os error 104)
relay_1                    |   caused by: IO error: Connection reset by peer (os error 104)
relay_1                    |   caused by: Connection reset by peer (os error 104)
relay_1                    | 2020-06-17T12:11:54Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1                    |   caused by: IO error: Connection reset by peer (os error 104)
relay_1                    |   caused by: IO error: Connection reset by peer (os error 104)
relay_1                    |   caused by: IO error: Connection reset by peer (os error 104)
relay_1                    |   caused by: Connection reset by peer (os error 104)
relay_1                    | 2020-06-17T12:12:54Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1                    |   caused by: IO error: Connection reset by peer (os error 104)
relay_1                    |   caused by: IO error: Connection reset by peer (os error 104)
relay_1                    |   caused by: IO error: Connection reset by peer (os error 104)
relay_1                    |   caused by: Connection reset by peer (os error 104)
relay_1                    | 2020-06-17T12:13:54Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1                    |   caused by: IO error: Connection reset by peer (os error 104)
relay_1                    |   caused by: IO error: Connection reset by peer (os error 104)
relay_1                    |   caused by: IO error: Connection reset by peer (os error 104)
relay_1                    |   caused by: Connection reset by peer (os error 104)
relay_1                    | 2020-06-17T12:14:54Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1                    |   caused by: IO error: Connection reset by peer (os error 104)
relay_1                    |   caused by: IO error: Connection reset by peer (os error 104)
relay_1                    |   caused by: IO error: Connection reset by peer (os error 104)
relay_1                    |   caused by: Connection reset by peer (os error 104)
relay_1                    | 2020-06-17T12:15:54Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1                    |   caused by: IO error: Connection reset by peer (os error 104)
relay_1                    |   caused by: IO error: Connection reset by peer (os error 104)
relay_1                    |   caused by: IO error: Connection reset by peer (os error 104)
relay_1                    |   caused by: Connection reset by peer (os error 104)
relay_1                    | 2020-06-17T12:16:54Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1                    |   caused by: IO error: Connection reset by peer (os error 104)
relay_1                    |   caused by: IO error: Connection reset by peer (os error 104)
relay_1                    |   caused by: IO error: Connection reset by peer (os error 104)
relay_1                    |   caused by: Connection reset by peer (os error 104)

Sentry 20.7.0.dev02c38a38

I can give access to my environment if some sentry engineer wants to debug it…

10

We are aware of this issue and are working on a fix. Please keep an eye on the GitHub issue: https://github.com/getsentry/onpremise/issues/486

1 Like
11

For any reason this is not taking effect for me. I’ve added the lines to my docker-compose file but relay is still trying to resolve the IPv6 for the service name.

root@basecamp /srv/sentry # docker-compose config
networks:
  default: {}
  sentry-proxy:
    external: true
    name: nginx-proxy_sentry
services:
  relay:
    depends_on:
      kafka:
        condition: service_started
      redis:
        condition: service_started
    image: getsentry/relay:20.12.1
    restart: unless-stopped
    sysctls:
      net.ipv6.conf.all.disable_ipv6: '1'
    volumes:
    - read_only: true
      source: /srv/sentry/geoip
      target: /geoip
      type: bind
    - read_only: true
      source: /srv/sentry/relay
      target: /work/.relay
      type: bind

The error is still the same:

2020-12-29T23:17:54Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
  caused by: could not send request using actix-web client
  caused by: Failed to connect to host: Failed resolving hostname: no record found for name: web type: AAAA class: IN
  caused by: Failed resolving hostname: no record found for name: web type: AAAA class: IN
  caused by: Failed resolving hostname: no record found for name: web type: AAAA class: IN
2020-12-29T23:18:08Z [relay_server::actors::upstream] WARN: Network outage, scheduling another check in 60s

Looking up the DNS (IPv4) from the container itself using dig works:

root@07af2dcb0fac:/work# dig web
; <<>> DiG 9.11.5-P4-5.1+deb10u2-Debian <<>> web
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 19165
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;web.                           IN      A

;; ANSWER SECTION:
web.                    600     IN      A       172.27.0.21

;; Query time: 0 msec
;; SERVER: 127.0.0.11#53(127.0.0.11)
;; WHEN: Tue Dec 29 23:22:02 UTC 2020
;; MSG SIZE  rcvd: 40

Github Issue: https://github.com/getsentry/onpremise/issues/608
Same Topic here in the forum: How to troubleshoot: There is 1 issue with your system configuration

12

Have you tried the suggestions on the issues as people reported following them solves the issue.

The forum topic and the config issue warning is unrelated.

13

Yes, I’ve added the sysctls option to disable IPv6 for relay service, but it still tries to resolve the AAAA DNS record for the web service/container.

root@basecamp /srv/sentry # docker-compose exec relay bash

root@32e9f80659a7:/work# sysctl   
bash: sysctl: command not found

root@32e9f80659a7:/work# test -f /proc/net/if_inet6 && echo "Running kernel is IPv6 ready"
Running kernel is IPv6 ready

root@32e9f80659a7:/work# cat /etc/issue
Debian GNU/Linux 10 \n \l

root@32e9f80659a7:/work# cat /proc/sys/net/ipv6/conf/all/disable_ipv6
1
14

You cannot disable IPv6 support in individual containers like that. You should probably add a sysctls section under the relay service definition in the docker-compose file and add net.ipv6.conf.all.disable_ipv6: 1 there.

@jauer @chadwhitacre @matt - maybe we should revisit https://github.com/getsentry/onpremise/pull/496?

15

That’s actually what I’ve done, as you can see in the snippets I’ve posted before.
But it seems like the relay service/application is still trying to resolve IPv6 DNS.

16

The reason I’m against this is because it prevents external IPv6 networking too, which is becoming increasingly more common.

17

I’m still experiencing this error, here’s the log of docker-compose logs relay

relay_1 | 2021-01-04T21:17:45Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1 | caused by: could not send request using actix-web client
relay_1 | caused by: Failed to connect to host: Failed resolving hostname: no record found for name: web type: AAAA class: IN
relay_1 | caused by: Failed resolving hostname: no record found for name: web type: AAAA class: IN
relay_1 | caused by: Failed resolving hostname: no record found for name: web type: AAAA class: IN

I have the mentioned sysctls config in docker-compose.yml to disable IPv6, but asking the container if it has taken effect (after running install.sh / up again), it says no:

$ docker-compose exec relay bash
root@d3d34e5d34f2:/work$ cat /sys/module/ipv6/parameters/disable
0

IPv6 is not enabled in the network created by docker-compose, but has all containers with IPv4 addresses.

$ docker network inspect sentry_onpremise_default
[
    {
        "Name": "sentry_onpremise_default",
        "Id": "176eb9a9033eff01d1973ff9aac12e6f268b29f81643642d0b394dfbee83d2bc",
        "Created": "2021-01-05T09:57:32.23549224+01:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
...

The relay container can reach the web container via ping or curl (after installing them).

$ docker-compose exec relay bash
root@d3d34e5d34f2:/work$ apt-get update && apt-get install curl
# ...
root@d3d34e5d34f2:/work$ curl -I http://web:9000/
HTTP/1.1 302 Found
Content-Type: text/html; charset=utf-8
Location: /auth/login/
# ...

So I’m stumped why the relay tries to resolve web with an AAAA record.

Trying to setup IPv6 with sentry’s docker-compose file does not work, as apparently IPv6 is not supported in docker-compose v3.x.

For debugging purposes I added an extra_hosts to the relay section mapping web to the IP (that obviously does not work for the right IP as it changes after down/up). That actually leads to the following relay log which sounds like the resolving then works:

relay_1 | 2021-01-04T19:33:02Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1 | caused by: could not send request using actix-web client
relay_1 | caused by: Timeout while waiting for response

One hypothesis I have is that the relay tries to resolve web before it’s available and gets stuck looking for AAAA IPs?

18

I have the exact same problem as @stefanw, and have attempted the same things, and I’m not really sure where to go from here. I’m running docker compose 1.26.2 with docker 20.10.2 on Ubuntu Focal, and can’t seem to get past this point.

I’ve flailed around a bit trying to disable ipv6 in different ways, but have made no progress. I’d really appreciate some help, as this is blocking me from upgrading from a much older version.

20

This seems pretty testable by bringing up the web service, confirming it’s up, then bringing up relay. @JanMalte @stefanw @oldpatricka Can one of you give this a try?

21

There is just one warning in the logs, before the DNS IPv6 error is reported.

root@basecamp /srv/sentry # docker-compose rm -s -v relay
Going to remove sentry_onpremise_relay_1
Are you sure? [yN] y
Removing sentry_onpremise_relay_1 ... done

root@basecamp /srv/sentry # docker-compose up -d relay 
dosentry_onpremise_redis_1 is up-to-date
sentry_onpremise_zookeeper_1 is up-to-date
sentry_onpremise_kafka_1 is up-to-date
Creating sentry_onpremise_relay_1 ... done

root@basecamp /srv/sentry # docker-compose logs -f relay
Attaching to sentry_onpremise_relay_1
relay_1 | 2021-01-07T12:06:47Z [actix::actors::resolver] WARN: Can not create system dns resolver: io error
relay_1 | 2021-01-07T12:06:47Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1 |   caused by: could not send request using actix-web client
relay_1 |   caused by: Failed to connect to host: Failed resolving hostname: no record found for name: web type: AAAA class: IN
relay_1 |   caused by: Failed resolving hostname: no record found for name: web type: AAAA class: IN
relay_1 |   caused by: Failed resolving hostname: no record found for name: web type: AAAA class: IN