Today I stumbled across this issue while migrating our on-prem installation onto another system and upgrading from 9.0 to 20.12.1.
I got the exact same error as shown in JanMalte’s post.
The key here is this line:
relay_1 | 2021-01-07T12:06:47Z [actix::actors::resolver] WARN: Can not create system dns resolver: io error
After having spent most of the day debugging this issue, to me it looks like this:
relay version 20.12.1 uses actix version 0.7.9 which in turn uses trust-dns-resolver(1) 0.10.0. This version doesn’t recognize newer resolver options in
resolv.conf - in my case it was the
trust-ad option that led to an exception
InvalidOption(17). That exception is unfortunately masked by actix and only leads to the generic
Can not create system dns resolver: io error message. I compiled and ran an example resolver(2) provided by trust-dns to debug this issue to confirm that removing the problematic option makes the error go away.
To workaround this issue, I have added this section to my
# NOTE: Using temp file since `sed -i` fails with `Device or resource busy`.
sed '/^options /s/\<trust-ad\>//g' /etc/resolv.conf >"$$tmp_resolv_conf"
cp "$$tmp_resolv_conf" /etc/resolv.conf
rm -f "$$tmp_resolv_conf"
exec bash /docker-entrypoint.sh run
This simply removes the
trust-ad option from
resolv.conf and this way makes the resolver errors go away and event ingestion working again.
A real fix would involve updating
trust-dns-resolver to a newer version. I tested the latest version, 0.20.0, using the CLI client(3) provided with this version and confirmed it parses resolv.conf correctly even with
trust-ad option enabled.
But 0.20.0 is only provided with actix starting from 0.11.0-beta.1 (the latest at the time of writing). I don’t know if there’s an earlier version of
trust-dns-resolver that already has a fix and is shipped with earlier versions of actix…
- Github: /actix/actix/blob/0.7/Cargo.toml#L67
- Github: /bluejekyll/trust-dns/blob/r0.10.0/crates/resolver/examples/global_resolver.rs
- Github: /bluejekyll/trust-dns/tree/v0.20.0#using-the-trust-dns-resolver-cli
(Sorry for the obfuscated links, but I was greeted with a friendly “new users are not allowed to post more than 2 links” message…)