Relay errors in fresh new on premise install

On-Premise
26

Today I stumbled across this issue while migrating our on-prem installation onto another system and upgrading from 9.0 to 20.12.1.

I got the exact same error as shown in JanMalte’s post.

The key here is this line:

relay_1 | 2021-01-07T12:06:47Z [actix::actors::resolver] WARN: Can not create system dns resolver: io error

After having spent most of the day debugging this issue, to me it looks like this:

relay version 20.12.1 uses actix version 0.7.9 which in turn uses trust-dns-resolver(1) 0.10.0. This version doesn’t recognize newer resolver options in resolv.conf - in my case it was the trust-ad option that led to an exception InvalidOption(17). That exception is unfortunately masked by actix and only leads to the generic Can not create system dns resolver: io error message. I compiled and ran an example resolver(2) provided by trust-dns to debug this issue to confirm that removing the problematic option makes the error go away.

To workaround this issue, I have added this section to my docker-compose.yml:

  relay:
    entrypoint:
      - sh
      - -c
      - |
        # NOTE: Using temp file since `sed -i` fails with `Device or resource busy`.
        tmp_resolv_conf="$$(mktemp)"
        sed '/^options /s/\<trust-ad\>//g' /etc/resolv.conf >"$$tmp_resolv_conf"
        cp "$$tmp_resolv_conf" /etc/resolv.conf
        rm -f "$$tmp_resolv_conf"
        exec bash /docker-entrypoint.sh run

This simply removes the trust-ad option from resolv.conf and this way makes the resolver errors go away and event ingestion working again.

A real fix would involve updating trust-dns-resolver to a newer version. I tested the latest version, 0.20.0, using the CLI client(3) provided with this version and confirmed it parses resolv.conf correctly even with trust-ad option enabled.

But 0.20.0 is only provided with actix starting from 0.11.0-beta.1 (the latest at the time of writing). I don’t know if there’s an earlier version of trust-dns-resolver that already has a fix and is shipped with earlier versions of actix…

Links:

  1. Github: /actix/actix/blob/0.7/Cargo.toml#L67
  2. Github: /bluejekyll/trust-dns/blob/r0.10.0/crates/resolver/examples/global_resolver.rs
  3. Github: /bluejekyll/trust-dns/tree/v0.20.0#using-the-trust-dns-resolver-cli

(Sorry for the obfuscated links, but I was greeted with a friendly “new users are not allowed to post more than 2 links” message…)

3 Likes