In the Sentry UI I am getting the message “There is 1 issue with your system configuration.”, but no link on how to debug. Any advice on where to start here?
Error logs:
clickhouse_1 | 2021.01.04 17:23:48.797175 [ 1 ] {} <Error> Application: Listen [::]:8123 failed: Poco::Exception. Code: 1000, e.code() = 0, e.displayText() = DNS Error: EAI: -9 (version 20.3.9.70 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
clickhouse_1 | 2021.01.04 17:23:48.797418 [ 1 ] {} <Error> Application: Listen [::]:9000 failed: Poco::Exception. Code: 1000, e.code() = 0, e.displayText() = DNS Error: EAI: -9 (version 20.3.9.70 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
clickhouse_1 | 2021.01.04 17:23:48.797595 [ 1 ] {} <Error> Application: Listen [::]:9009 failed: Poco::Exception. Code: 1000, e.code() = 0, e.displayText() = DNS Error: EAI: -9 (version 20.3.9.70 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
clickhouse_1 | 2021.01.04 17:23:48.797785 [ 1 ] {} <Error> Application: Listen [::]:9004 failed: Poco::Exception. Code: 1000, e.code() = 0, e.displayText() = DNS Error: EAI: -9 (version 20.3.9.70 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
ingest-consumer_1 | %3|1609781046.425|FAIL|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.18.0.11:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
ingest-consumer_1 | ConnectionRefusedError: [Errno 111] Connection refused
geoipupdate_1 | error loading configuration file /sentry/GeoIP.conf: error opening file: open /sentry/GeoIP.conf: no such file or directory
ingest-consumer_1 | urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f216056ca20>: Failed to establish a new connection: [Errno 111] Connection refused
ingest-consumer_1 | raise MaxRetryError(_pool, url, Error or ResponseError(cause))
ingest-consumer_1 | urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=9000): Max retries exceeded with url: /api/1/store/ (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f216056ca20>: Failed to establish a new connection: [Errno 111] Connection refused',))
ingest-consumer_1 | 17:24:09 [ERROR] sentry_sdk.ERRORs: Internal ERROR in sentry_sdk
ingest-consumer_1 | raise Exception(msg.error())
ingest-consumer_1 | Exception: KafkaError{code=COORDINATOR_LOAD_IN_PROGRESS,val=14,str="JoinGroup failed: Broker: Coordinator load in progress"}
ingest-consumer_1 | 17:24:17 [INFO] batching-kafka-consumer: New partitions assigned: [TopicPartition{topic=ingest-attachments,partition=0,offset=-1001,error=None}, TopicPartition{topic=ingest-events,partition=0,offset=-1001,error=None}, TopicPartition{topic=ingest-transactions,partition=0,offset=-1001,error=None}]
kafka_1 | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/172.18.0.8:2181. Will not attempt to authenticate using SASL (unknown error)
kafka_1 | [2021-01-04 17:24:09,122] ERROR [KafkaApi-1001] Number of alive brokers '0' does not meet the required replication factor '1' for the offsets topic (configured via 'offsets.topic.replication.factor'). This ERROR can be ignored if the cluster is starting up and not all brokers are up yet. (kafka.server.KafkaApis)
post-process-forwarder_1 | %3|1609781046.133|FAIL|rdkafka#consumer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.18.0.11:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
post-process-forwarder_1 | %3|1609781046.133|FAIL|rdkafka#consumer-2| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.18.0.11:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
nginx_1 | 172.18.0.1 - - [04/Jan/2021:17:30:03 +0000] "GET /_static/1609781046/sentry/dist/ErrorRobot.js HTTP/1.1" 200 3318 "http://localhost:9000/organizations/sentry/issues/?project=1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36" "-"
relay_1 | 2021-01-04T17:23:51Z [rdkafka::client] ERROR: librdkafka: FAIL [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.18.0.11:9092 failed: Connection refused (after 2ms in state CONNECT)
relay_1 | 2021-01-04T17:23:51Z [rdkafka::client] ERROR: librdkafka: Global ERROR: BrokerTransportFailure (Local: Broker transport failure): kafka:9092/bootstrap: Connect to ipv4#172.18.0.11:9092 failed: Connection refused (after 2ms in state CONNECT)
relay_1 | 2021-01-04T17:23:51Z [rdkafka::client] ERROR: librdkafka: Global ERROR: AllBrokersDown (Local: All broker connections are down): 1/1 brokers are down
relay_1 | 2021-01-04T17:23:51Z [rdkafka::client] ERROR: librdkafka: FAIL [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.18.0.11:9092 failed: Connection refused (after 1ms in state CONNECT)
relay_1 | 2021-01-04T17:23:51Z [rdkafka::client] ERROR: librdkafka: Global ERROR: BrokerTransportFailure (Local: Broker transport failure): kafka:9092/bootstrap: Connect to ipv4#172.18.0.11:9092 failed: Connection refused (after 1ms in state CONNECT)
relay_1 | 2021-01-04T17:23:51Z [rdkafka::client] ERROR: librdkafka: Global ERROR: AllBrokersDown (Local: All broker connections are down): 1/1 brokers are down
relay_1 | 2021-01-04T17:23:51Z [relay_server::actors::upstream] ERROR: authentication encountered ERROR: could not send request to upstream
relay_1 | 2021-01-04T17:23:51Z [relay_server::actors::upstream] ERROR: authentication encountered ERROR: could not send request to upstream
relay_1 | 2021-01-04T17:23:52Z [rdkafka::client] ERROR: librdkafka: FAIL [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.18.0.11:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical ERROR(s) suppressed)
relay_1 | 2021-01-04T17:23:52Z [rdkafka::client] ERROR: librdkafka: Global ERROR: BrokerTransportFailure (Local: Broker transport failure): kafka:9092/bootstrap: Connect to ipv4#172.18.0.11:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical ERROR(s) suppressed)
relay_1 | 2021-01-04T17:23:52Z [rdkafka::client] ERROR: librdkafka: FAIL [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.18.0.11:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical ERROR(s) suppressed)
relay_1 | 2021-01-04T17:23:52Z [rdkafka::client] ERROR: librdkafka: Global ERROR: BrokerTransportFailure (Local: Broker transport failure): kafka:9092/bootstrap: Connect to ipv4#172.18.0.11:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical ERROR(s) suppressed)
relay_1 | 2021-01-04T17:23:52Z [relay_server::actors::upstream] ERROR: authentication encountered ERROR: could not send request to upstream
relay_1 | 2021-01-04T17:23:53Z [relay_server::actors::upstream] ERROR: authentication encountered ERROR: could not send request to upstream
relay_1 | caused by: Failed to connect to host: Connection refused (os error 111)
relay_1 | caused by: Connection refused (os error 111)
relay_1 | caused by: Connection refused (os error 111)
relay_1 | 2021-01-04T17:23:55Z [relay_server::actors::upstream] ERROR: authentication encountered ERROR: could not send request to upstream
relay_1 | caused by: Failed to connect to host: Connection refused (os error 111)
relay_1 | caused by: Connection refused (os error 111)
relay_1 | caused by: Connection refused (os error 111)
relay_1 | 2021-01-04T17:23:59Z [relay_server::actors::upstream] ERROR: authentication encountered ERROR: could not send request to upstream
relay_1 | caused by: Failed to connect to host: Connection refused (os error 111)
relay_1 | caused by: Connection refused (os error 111)
relay_1 | caused by: Connection refused (os error 111)
relay_1 | 2021-01-04T17:24:04Z [relay_server::actors::upstream] ERROR: authentication encountered ERROR: could not send request to upstream
relay_1 | caused by: Failed to connect to host: Connection refused (os error 111)
relay_1 | caused by: Connection refused (os error 111)
relay_1 | caused by: Connection refused (os error 111)
snuba-consumer_1 | %3|1609781043.505|FAIL|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.18.0.11:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
snuba-consumer_1 | %3|1609781043.506|FAIL|rdkafka#consumer-2| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.18.0.11:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
snuba-outcomes-consumer_1 | %3|1609781043.860|FAIL|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.18.0.11:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
snuba-outcomes-consumer_1 | %3|1609781043.901|FAIL|rdkafka#consumer-2| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.18.0.11:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
snuba-subscription-consumer-transactions_1 | %3|1609781043.468|FAIL|rdkafka#consumer-2| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.18.0.11:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
snuba-subscription-consumer-transactions_1 | %3|1609781044.454|FAIL|rdkafka#consumer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.18.0.11:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
snuba-subscription-consumer-events_1 | %3|1609781044.208|FAIL|rdkafka#consumer-2| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.18.0.11:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
snuba-subscription-consumer-events_1 | %3|1609781045.207|FAIL|rdkafka#consumer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.18.0.11:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
snuba-replacer_1 | %3|1609781043.713|FAIL|rdkafka#consumer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.18.0.11:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
snuba-transactions-consumer_1 | %3|1609781043.062|FAIL|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.18.0.11:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
snuba-transactions-consumer_1 | %3|1609781043.064|FAIL|rdkafka#consumer-2| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.18.0.11:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
snuba-sessions-consumer_1 | %3|1609781043.772|FAIL|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.18.0.11:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
snuba-sessions-consumer_1 | %3|1609781043.802|FAIL|rdkafka#consumer-2| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.18.0.11:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
subscription-consumer-transactions_1 | %3|1609781046.172|FAIL|rdkafka#producer-2| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.18.0.11:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
subscription-consumer-transactions_1 | %3|1609781047.172|FAIL|rdkafka#consumer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.18.0.11:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
subscription-consumer-transactions_1 | 17:24:13 [INFO] sentry.snuba.query_subscription_consumer: query-subscription-consumer.on_assign (offsets='{0: None}' partitions='[TopicPartition{topic=transactions-subscription-results,partition=0,offset=-1001,error=None}]')
subscription-consumer-events_1 | %3|1609781046.554|FAIL|rdkafka#producer-2| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.18.0.11:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
subscription-consumer-events_1 | %3|1609781047.553|FAIL|rdkafka#consumer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.18.0.11:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
subscription-consumer-events_1 | 17:24:13 [INFO] sentry.snuba.query_subscription_consumer: query-subscription-consumer.on_assign (offsets='{0: None}' partitions='[TopicPartition{topic=events-subscription-results,partition=0,offset=-1001,error=None}]')```