Relay messages are not sent to AWS MSK's Kafka Topic

Hello experts,

I am installing a self-hosted sentry server(Sentry 20.9.0.dev05d24254) in my AWS EC2 instance. However, I am running few componenets of sentry in AWS managed services.

With one including the kafka(aws managed MSK version:2.2.1) & other services running in my ec2 instances as a docker containers. I see the relay services are not able to produce messages into topic.

Below are the error statments

relay in TRACE mode:

2020-09-08T20:15:26Z [relay_server::actors::upstream] DEBUG: sending register challenge response
2020-09-08T20:15:26Z [relay_server::actors::upstream] INFO: relay successfully registered with upstream
^[[A2020-09-08T20:16:37Z [relay_server::actors::project] DEBUG: project 13 initialized without state
2020-09-08T20:16:37Z [relay_server::actors::project] DEBUG: project 13 state requested
2020-09-08T20:16:37Z [relay_server::actors::events] TRACE: queued event
2020-09-08T20:16:37Z [relay_server::actors::project] DEBUG: project 13 state request amended
2020-09-08T20:16:37Z [relay_server::actors::project_upstream] DEBUG: updating project states for 1/1 projects (attempt 1)
2020-09-08T20:16:37Z [relay_server::actors::project_upstream] DEBUG: sending request of size 1
2020-09-08T20:16:38Z [relay_server::actors::project] DEBUG: project 13 state updated
2020-09-08T20:16:38Z [relay_server::actors::events] TRACE: processing json event
2020-09-08T20:16:38Z [relay_server::actors::events] TRACE: sending envelope to kafka
2020-09-08T20:16:38Z [relay_server::actors::store] TRACE: Sending event item of envelope to kafka
2020-09-08T20:21:38Z [relay_server::utils::kafka] ERROR: failed to produce message to Kafka (delivery callback): Message production error: MessageTimedOut (Local: Message timed out)
2020-09-08T20:26:59Z [relay_server::actors::project] DEBUG: project 13 removed from cache

Tried to connect from the ec2 instance (not from the relay container) & got the below error :

ubuntu@ip-XXXXX:~/kafka/kafka_2.12-2.2.1$ bin/kafka-console-consumer.sh --bootstrap-server=XXXXXXX.kafka.eu-central-1.amazonaws.com:9092 --topic events --from-beginning
[2020-09-08 20:36:50,743] WARN Couldn’t resolve server XXXXXXX.kafka.eu-central-1.amazonaws.com:9092 from bootstrap.servers as DNS resolution failed for XXXXXXX.kafka.eu-central-1.amazonaws.com (org.apache.kafka.clients.ClientUtils)
[2020-09-08 20:36:50,750] ERROR Unknown error when running consumer: (kafka.tools.ConsoleConsumer$)
org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
at org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:811)
at org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:659)
at kafka.tools.ConsoleConsumer$.run(ConsoleConsumer.scala:67)
at kafka.tools.ConsoleConsumer$.main(ConsoleConsumer.scala:54)
at kafka.tools.ConsoleConsumer.main(ConsoleConsumer.scala)
Caused by: org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls given in bootstrap.servers
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:88)
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:47)
at org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:721)
… 4 more

With other settings double checked, i could query the zoo-keeper (managed by aws) to list down the topics. It is successful. I also tried disabling the ip6 for the relay service using sysctls flag. It is not helping thoug.

Looks like an issue with hostnames and advertised host names to me. Maybe they are not on the same network or you are not configuring hostname settings across the projects properly?

Thanks for the response. It looks like the problem is with the AWS’s kafka version. The recommended older version 2.2.1 seems to have this issue. Now, upgraded to recent version(2.4.1.1) & everything is back to normal !

Good day. :beers:

1 Like