Sentry does not receive events (kafka error)

On-Premise
1

I have installed Sentry On-Premise, but events do not reach Sentry.

Versions:

New Ubuntu Server 20.04.
Docker 19.03.15 ~ 3-0 ~ ubuntu-focal (not the last, so that there are no AAAA-dns errors in the relay)
SENTRY_IMAGE = getsentry / sentry: latest (Sentry 21.3.0 96d5172)
SNUBA_IMAGE = getsentry / snuba: nightly
RELAY_IMAGE = getsentry / relay: nightly
SYMBOLICATOR_IMAGE = getsentry / symbolicator: nightly
(in variations. tried and only latest or only nightly)

Error logs

clickhouse_1                                | 2021.03.23 12:25:18.082883 [ 1 ] {} <Information> Application: It looks like the process has no CAP_NET_ADMIN capability, 'taskstats' performance statistics will be disabled. It could happen due to incorrect ClickHouse package installation. You could resolve the problem manually with 'sudo setcap cap_net_admin=+ep /usr/bin/clickhouse'. Note that it will not work on 'nosuid' mounted filesystems. It also doesn't work if you run clickhouse-server inside network namespace as it happens in some containers.
clickhouse_1                                | 2021.03.23 12:25:18.082909 [ 1 ] {} <Information> Application: It looks like the process has no CAP_SYS_NICE capability, the setting 'os_thread_nice' will have no effect. It could happen due to incorrect ClickHouse package installation. You could resolve the problem manually with 'sudo setcap cap_sys_nice=+ep /usr/bin/clickhouse'. Note that it will not work on 'nosuid' mounted filesystems.
clickhouse_1                                | 2021.03.23 12:25:18.084148 [ 1 ] {} <Error> Application: Listen [::]:8123 failed: Poco::Exception. Code: 1000, e.code() = 0, e.displayText() = DNS error: EAI: -9 (version 20.3.9.70 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
clickhouse_1                                | 2021.03.23 12:25:18.084464 [ 1 ] {} <Error> Application: Listen [::]:9000 failed: Poco::Exception. Code: 1000, e.code() = 0, e.displayText() = DNS error: EAI: -9 (version 20.3.9.70 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
clickhouse_1                                | 2021.03.23 12:25:18.084661 [ 1 ] {} <Error> Application: Listen [::]:9009 failed: Poco::Exception. Code: 1000, e.code() = 0, e.displayText() = DNS error: EAI: -9 (version 20.3.9.70 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
clickhouse_1                                | 2021.03.23 12:25:18.084889 [ 1 ] {} <Error> Application: Listen [::]:9004 failed: Poco::Exception. Code: 1000, e.code() = 0, e.displayText() = DNS error: EAI: -9 (version 20.3.9.70 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>

geoipupdate_1                               | error loading configuration file /sentry/GeoIP.conf: error opening file: open /sentry/GeoIP.conf: no such file or directory

ingest-consumer_1                           | 12:56:47 [WARNING] batching-kafka-consumer: Topic 'ingest-events' or its partitions are not ready, retrying...
ingest-consumer_1                           | 12:56:51 [INFO] batching-kafka-consumer: New partitions assigned: [TopicPartition{topic=ingest-attachments,partition=0,offset=-1001,error=None}, TopicPartition{topic=ingest-events,partition=0,offset=-1001,error=None}, TopicPartition{topic=ingest-transactions,partition=0,offset=-1001,error=None}]


relay_1                                     | 2021-03-23T12:56:37Z [rdkafka::client] ERROR: librdkafka: FAIL [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.19.0.11:9092 failed: Connection refused (after 1ms in state CONNECT)
relay_1                                     | 2021-03-23T12:56:37Z [rdkafka::client] ERROR: librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): kafka:9092/bootstrap: Connect to ipv4#172.19.0.11:9092 failed: Connection refused (after 1ms in state CONNECT)
relay_1                                     | 2021-03-23T12:56:37Z [rdkafka::client] ERROR: librdkafka: Global error: AllBrokersDown (Local: All broker connections are down): 1/1 brokers are down

snuba-outcomes-consumer_1                   | %3|1616504198.457|FAIL|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.19.0.11:9092 failed: Connection refused (after 25ms in state CONNECT)
snuba-outcomes-consumer_1                   | %3|1616504198.458|FAIL|rdkafka#consumer-2| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.19.0.11:9092 failed: Connection refused (after 24ms in state CONNECT)
snuba-outcomes-consumer_1                   | %3|1616504199.433|FAIL|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.19.0.11:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
snuba-outcomes-consumer_1                   | %3|1616504199.433|FAIL|rdkafka#consumer-2| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.19.0.11:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
snuba-outcomes-consumer_1                   | 2021-03-23 12:56:51,262 New partitions assigned: {Partition(topic=Topic(name='outcomes'), index=0): 114}
snuba-outcomes-consumer_1                   | 2021-03-23 12:57:15,172 Completed processing <Batch: 5 messages, open for 1.25 seconds>.
snuba-consumer_1                            | %3|1616504198.817|FAIL|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.19.0.11:9092 failed: Connection refused (after 13ms in state CONNECT)
snuba-consumer_1                            | %3|1616504198.825|FAIL|rdkafka#consumer-2| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.19.0.11:9092 failed: Connection refused (after 7ms in state CONNECT)
snuba-consumer_1                            | %3|1616504199.804|FAIL|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.19.0.11:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
snuba-consumer_1                            | %3|1616504199.804|FAIL|rdkafka#consumer-2| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.19.0.11:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
snuba-consumer_1                            | 2021-03-23 12:56:51,297 New partitions assigned: {Partition(topic=Topic(name='events'), index=0): 0}
snuba-replacer_1                            | %3|1616504199.114|FAIL|rdkafka#consumer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.19.0.11:9092 failed: Connection refused (after 1ms in state CONNECT)
snuba-replacer_1                            | %3|1616504200.099|FAIL|rdkafka#consumer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.19.0.11:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
snuba-replacer_1                            | 2021-03-23 12:56:51,312 New partitions assigned: {Partition(topic=Topic(name='event-replacements'), index=0): 0}
snuba-sessions-consumer_1                   | %3|1616504199.975|FAIL|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.19.0.11:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
snuba-sessions-consumer_1                   | 2021-03-23 12:56:51,266 New partitions assigned: {Partition(topic=Topic(name='ingest-sessions'), index=0): 13}
snuba-sessions-consumer_1                   | 2021-03-23 12:58:19,952 Completed processing <Batch: 2 messages, open for 1.02 seconds>.
snuba-subscription-consumer-events_1        | %3|1616504199.289|FAIL|rdkafka#consumer-2| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.19.0.11:9092 failed: Connection refused (after 10ms in state CONNECT)
snuba-subscription-consumer-events_1        | %3|1616504200.276|FAIL|rdkafka#consumer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.19.0.11:9092 failed: Connection refused (after 0ms in state CONNECT)
snuba-subscription-consumer-events_1        | %3|1616504200.277|FAIL|rdkafka#consumer-2| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.19.0.11:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
snuba-transactions-consumer_1               | %3|1616504198.420|FAIL|rdkafka#consumer-2| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.19.0.11:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
snuba-transactions-consumer_1               | 2021-03-23 12:56:51,611 New partitions assigned: {Partition(topic=Topic(name='events'), index=0): 0}
subscription-consumer-events_1              | 12:56:47 [INFO] sentry.plugins.github: apps-not-configured
subscription-consumer-events_1              | 12:56:51 [INFO] sentry.snuba.query_subscription_consumer: query-subscription-consumer.on_assign (offsets='{0: None}' partitions='[TopicPartition{topic=events-subscription-results,partition=0,offset=-1001,error=None}]')
subscription-consumer-transactions_1        | 12:56:46 [INFO] sentry.plugins.github: apps-not-configured
subscription-consumer-transactions_1        | 12:56:47 [WARNING] batching-kafka-consumer: Topic 'transactions-subscription-results' or its partitions are not ready, retrying...

But, kafka name resolved and available in containers.

root@fff8531ec759:/work# nmap -p 9092 kafka
Starting Nmap 7.70 ( https://nmap.org ) at 2021-03-23 13:10 UTC
Nmap scan report for kafka (172.19.0.11)
Host is up (0.00015s latency).
rDNS record for 172.19.0.11: sentry_onpremise_kafka_1.sentry_onpremise_default

PORT     STATE SERVICE
9092/tcp open  XmlIpcRegSvc
MAC Address: 02:42:AC:13:00:0B (Unknown)

Nmap done: 1 IP address (1 host up) scanned in 0.57 seconds

test.sh create event locally

Please help me: how can I fix this and receive events?

2

https://stackoverflow.com/a/2333446/90297 lists 3 possible reasons for connection refused errors. Have you checked your firewall settings?

3

Thanks for the answer.
Yes, that’s okay. Checked also by telnet:

#: telnet kafka 9092
Trying 172.19.0.11...
Connected to kafka.
Escape character is '^]'.
4

Well then, I cannot really speculate more with this much information and without access to your system (which I am not asking because it is not feasible for us to do this one-by-one debugging for everyone :slight_smile: )

5

I debugged this
Really, with each start such errors appear in the logs. But after start, everything works. In my case, the problem turned out to be in the verification method: I used python sdk, in tcpdump on the proxy I see that the packets were not received. I tried to check in Django sdk and it turned out that everything works.
:woozy_face:

1 Like
6

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.