@BYK , my instance was running fine , till i had to start it back up
Startup errors, after doing a ./install.sh.
snuba-consumer_1 | 2021-03-30 21:27:45,601 Caught ConsumerError('KafkaError{code=COORDINATOR_LOAD_IN_PROGRESS,val=14,str="JoinGroup failed: Broker: Coordinator load in progress"}'), shutting down...
snuba-consumer_1 | Traceback (most recent call last):
snuba-consumer_1 | File "/usr/local/bin/snuba", line 33, in <module>
snuba-consumer_1 | sys.exit(load_entry_point('snuba', 'console_scripts', 'snuba')())
snuba-consumer_1 | File "/usr/local/lib/python3.8/site-packages/click/core.py", line 829, in __call__
snuba-consumer_1 | return self.main(*args, **kwargs)
snuba-consumer_1 | File "/usr/local/lib/python3.8/site-packages/click/core.py", line 782, in main
snuba-consumer_1 | rv = self.invoke(ctx)
snuba-consumer_1 | File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
snuba-consumer_1 | return _process_result(sub_ctx.command.invoke(sub_ctx))
snuba-consumer_1 | File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
snuba-consumer_1 | return ctx.invoke(self.callback, **ctx.params)
snuba-consumer_1 | File "/usr/local/lib/python3.8/site-packages/click/core.py", line 610, in invoke
snuba-consumer_1 | return callback(*args, **kwargs)
snuba-consumer_1 | File "/usr/src/snuba/snuba/cli/consumer.py", line 161, in consumer
snuba-consumer_1 | consumer.run()
snuba-consumer_1 | File "/usr/src/snuba/snuba/utils/streams/processing/processor.py", line 112, in run
snuba-consumer_1 | self._run_once()
snuba-consumer_1 | File "/usr/src/snuba/snuba/utils/streams/processing/processor.py", line 142, in _run_once
snuba-consumer_1 | self.__message = self.__consumer.poll(timeout=1.0)
snuba-consumer_1 | File "/usr/src/snuba/snuba/utils/streams/backends/kafka.py", line 773, in poll
snuba-consumer_1 | return super().poll(timeout)
snuba-consumer_1 | File "/usr/src/snuba/snuba/utils/streams/backends/kafka.py", line 405, in poll
snuba-consumer_1 | raise ConsumerError(str(error))
snuba-consumer_1 | snuba.utils.streams.backends.abstract.ConsumerError: KafkaError{code=COORDINATOR_LOAD_IN_PROGRESS,val=14,str="JoinGroup failed: Broker: Coordinator load in progress"}
snuba-consumer_1 | 2021-03-30 21:28:00,478 New partitions assigned: {Partition(topic=Topic(name='events'), index=0): 14335800}
relay_1 | 2021-03-30T21:27:31Z [rdkafka::client] ERROR: librdkafka: FAIL [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.28.0.9:9092 failed: Connection refused (after 23ms in state CONNECT)
relay_1 | 2021-03-30T21:27:31Z [rdkafka::client] ERROR: librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): kafka:9092/bootstrap: Connect to ipv4#172.28.0.9:9092 failed: Connection refused (after 23ms in state CONNECT)
relay_1 | 2021-03-30T21:27:31Z [rdkafka::client] ERROR: librdkafka: Global error: AllBrokersDown (Local: All broker connections are down): 1/1 brokers are down
relay_1 | 2021-03-30T21:27:31Z [rdkafka::client] ERROR: librdkafka: FAIL [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.28.0.9:9092 failed: Connection refused (after 0ms in state CONNECT)
relay_1 | 2021-03-30T21:27:31Z [rdkafka::client] ERROR: librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): kafka:9092/bootstrap: Connect to ipv4#172.28.0.9:9092 failed: Connection refused (after 0ms in state CONNECT)
relay_1 | 2021-03-30T21:27:31Z [rdkafka::client] ERROR: librdkafka: Global error: AllBrokersDown (Local: All broker connections are down): 1/1 brokers are down
relay_1 | 2021-03-30T21:27:31Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1 | caused by: error sending request for url (http://web:9000/api/0/relays/register/challenge/): error trying to connect: dns error: no record found for name: web.apple.com. type: AAAA class: IN
relay_1 | 2021-03-30T21:27:31Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1 | caused by: error sending request for url (http://web:9000/api/0/relays/register/challenge/): error trying to connect: dns error: no record found for name: web.apple.com. type: AAAA class: IN
relay_1 | 2021-03-30T21:27:32Z [rdkafka::client] ERROR: librdkafka: FAIL [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.28.0.9:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
relay_1 | 2021-03-30T21:27:32Z [rdkafka::client] ERROR: librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): kafka:9092/bootstrap: Connect to ipv4#172.28.0.9:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
relay_1 | 2021-03-30T21:27:32Z [rdkafka::client] ERROR: librdkafka: FAIL [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.28.0.9:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
relay_1 | 2021-03-30T21:27:32Z [rdkafka::client] ERROR: librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): kafka:9092/bootstrap: Connect to ipv4#172.28.0.9:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
relay_1 | 2021-03-30T21:27:32Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1 | caused by: error sending request for url (http://web:9000/api/0/relays/register/challenge/): error trying to connect: dns error: no record found for name: web.apple.com. type: AAAA class: IN
relay_1 | 2021-03-30T21:27:34Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1 | caused by: error sending request for url (http://web:9000/api/0/relays/register/challenge/): error trying to connect: dns error: no record found for name: web.apple.com. type: AAAA class: IN
relay_1 | 2021-03-30T21:27:36Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1 | caused by: error sending request for url (http://web:9000/api/0/relays/register/challenge/): error trying to connect: dns error: no record found for name: web.apple.com. type: AAAA class: IN
relay_1 | 2021-03-30T21:27:40Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1 | caused by: error sending request for url (http://web:9000/api/0/relays/register/challenge/): error trying to connect: dns error: no record found for name: web.apple.com. type: AAAA class: IN
relay_1 | 2021-03-30T21:27:45Z [relay_server::actors::upstream] WARN: Network outage, scheduling another check in 0ns
relay_1 | 2021-03-30T21:27:45Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1 | caused by: error sending request for url (http://web:9000/api/0/relays/register/challenge/): error trying to connect: dns error: no record found for name: web.apple.com. type: AAAA class: IN
relay_1 | 2021-03-30T21:27:45Z [relay_server::actors::upstream] WARN: Network outage, scheduling another check in 1s
relay_1 | 2021-03-30T21:27:46Z [relay_server::actors::upstream] WARN: Network outage, scheduling another check in 1.5s
relay_1 | 2021-03-30T21:27:47Z [relay_server::actors::upstream] WARN: Network outage, scheduling another check in 2.25s
relay_1 | 2021-03-30T21:27:50Z [relay_server::actors::upstream] WARN: Network outage, scheduling another check in 3.375s
relay_1 | 2021-03-30T21:27:53Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1 | caused by: error sending request for url (http://web:9000/api/0/relays/register/challenge/): error trying to connect: dns error: no record found for name: web.apple.com. type: AAAA class: IN
relay_1 | 2021-03-30T21:27:53Z [relay_server::actors::upstream] WARN: Network outage, scheduling another check in 5.0625s
relay_1 | 2021-03-30T21:27:58Z [relay_server::actors::upstream] WARN: Network outage, scheduling another check in 7.59375s
memcached_1 | Signal handled: Terminated.
snuba-outcomes-consumer_1 | 2021-03-30 21:27:01,324 New partitions assigned: {Partition(topic=Topic(name='outcomes'), index=0): 14619363}
snuba-outcomes-consumer_1 | 2021-03-30 21:27:13,546 Partitions revoked: [Partition(topic=Topic(name='outcomes'), index=0)]
snuba-outcomes-consumer_1 | %3|1617139655.040|FAIL|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.28.0.9:9092 failed: Connection refused (after 1ms in state CONNECT)
snuba-outcomes-consumer_1 | %3|1617139655.040|FAIL|rdkafka#consumer-2| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.28.0.9:9092 failed: Connection refused (after 0ms in state CONNECT)
snuba-outcomes-consumer_1 | %3|1617139656.041|FAIL|rdkafka#consumer-2| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.28.0.9:9092 failed: Connection refused (after 1ms in state CONNECT, 1 identical error(s) suppressed)
snuba-outcomes-consumer_1 | %3|1617139656.041|FAIL|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.28.0.9:9092 failed: Connection refused (after 1ms in state CONNECT, 1 identical error(s) suppressed)
snuba-outcomes-consumer_1 | 2021-03-30 21:27:45,445 Caught ConsumerError('KafkaError{code=COORDINATOR_LOAD_IN_PROGRESS,val=14,str="JoinGroup failed: Broker: Coordinator load in progress"}'), shutting down...
snuba-outcomes-consumer_1 | Traceback (most recent call last):
snuba-outcomes-consumer_1 | File "/usr/local/bin/snuba", line 33, in <module>
snuba-outcomes-consumer_1 | sys.exit(load_entry_point('snuba', 'console_scripts', 'snuba')())
snuba-outcomes-consumer_1 | File "/usr/local/lib/python3.8/site-packages/click/core.py", line 829, in __call__
snuba-outcomes-consumer_1 | return self.main(*args, **kwargs)
snuba-outcomes-consumer_1 | File "/usr/local/lib/python3.8/site-packages/click/core.py", line 782, in main
snuba-outcomes-consumer_1 | rv = self.invoke(ctx)
snuba-outcomes-consumer_1 | File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
snuba-outcomes-consumer_1 | return _process_result(sub_ctx.command.invoke(sub_ctx))
snuba-outcomes-consumer_1 | File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
snuba-outcomes-consumer_1 | return ctx.invoke(self.callback, **ctx.params)
snuba-outcomes-consumer_1 | File "/usr/local/lib/python3.8/site-packages/click/core.py", line 610, in invoke
snuba-outcomes-consumer_1 | return callback(*args, **kwargs)
snuba-outcomes-consumer_1 | File "/usr/src/snuba/snuba/cli/consumer.py", line 161, in consumer
snuba-outcomes-consumer_1 | consumer.run()
snuba-outcomes-consumer_1 | File "/usr/src/snuba/snuba/utils/streams/processing/processor.py", line 112, in run
snuba-outcomes-consumer_1 | self._run_once()
snuba-outcomes-consumer_1 | File "/usr/src/snuba/snuba/utils/streams/processing/processor.py", line 142, in _run_once
snuba-outcomes-consumer_1 | self.__message = self.__consumer.poll(timeout=1.0)
snuba-outcomes-consumer_1 | File "/usr/src/snuba/snuba/utils/streams/backends/kafka.py", line 405, in poll
snuba-outcomes-consumer_1 | raise ConsumerError(str(error))
snuba-outcomes-consumer_1 | snuba.utils.streams.backends.abstract.ConsumerError: KafkaError{code=COORDINATOR_LOAD_IN_PROGRESS,val=14,str="JoinGroup failed: Broker: Coordinator load in progress"}
snuba-outcomes-consumer_1 | 2021-03-30 21:27:57,402 New partitions assigned: {Partition(topic=Topic(name='outcomes'), index=0): 14619363}
snuba-outcomes-consumer_1 | 2021-03-30 21:28:00,425 Partitions revoked: [Partition(topic=Topic(name='outcomes'), index=0)]
snuba-outcomes-consumer_1 | 2021-03-30 21:28:00,478 New partitions assigned: {Partition(topic=Topic(name='outcomes'), index=0): 14619363}
Authentication failure 
relay_1 | 2021-03-30T21:27:32Z [rdkafka::client] ERROR: librdkafka: FAIL [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.28.0.9:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
relay_1 | 2021-03-30T21:27:32Z [rdkafka::client] ERROR: librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): kafka:9092/bootstrap: Connect to ipv4#172.28.0.9:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
relay_1 | 2021-03-30T21:27:32Z [rdkafka::client] ERROR: librdkafka: FAIL [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.28.0.9:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
relay_1 | 2021-03-30T21:27:32Z [rdkafka::client] ERROR: librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): kafka:9092/bootstrap: Connect to ipv4#172.28.0.9:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
relay_1 | 2021-03-30T21:27:32Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1 | caused by: error sending request for url (http://web:9000/api/0/relays/register/challenge/): error trying to connect: dns error: no record found for name: web.apple.com. type: AAAA class: IN
relay_1 | 2021-03-30T21:27:34Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1 | caused by: error sending request for url (http://web:9000/api/0/relays/register/challenge/): error trying to connect: dns error: no record found for name: web.apple.com. type: AAAA class: IN
relay_1 | 2021-03-30T21:27:36Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1 | caused by: error sending request for url (http://web:9000/api/0/relays/register/challenge/): error trying to connect: dns error: no record found for name: web.apple.com. type: AAAA class: IN
relay_1 | 2021-03-30T21:27:40Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1 | caused by: error sending request for url (http://web:9000/api/0/relays/register/challenge/): error trying to connect: dns error: no record found for name: web.apple.com. type: AAAA class: IN
relay_1 | 2021-03-30T21:27:45Z [relay_server::actors::upstream] WARN: Network outage, scheduling another check in 0ns
relay_1 | 2021-03-30T21:27:45Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1 | caused by: error sending request for url (http://web:9000/api/0/relays/register/challenge/): error trying to connect: dns error: no record found for name: web.xx.com. type: AAAA class: IN
relay_1 | 2021-03-30T21:28:04Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1 | caused by: error sending request for url (http://web:9000/api/0/relays/register/challenge/): error trying to connect: dns error: no record found for name: web.apple.com. type: AAAA class: IN
relay_1 | 2021-03-30T21:28:59Z [relay_server::endpoints::common] ERROR: error handling request: failed to queue envelope
relay_1 | caused by: Too many events (event_buffer_size reached)
nginx_1 | 17.70.207.70 - - [30/Mar/2021:21:28:59 +0000] "POST /api/19/store/ HTTP/1.1" 503 94 "-" "sentry.python/0.19.5" "-"
relay_1 | 2021-03-30T21:28:59Z [relay_server::endpoints::common] ERROR: error handling request: failed to queue envelope
relay_1 | caused by: Too many events (event_buffer_size reached)
nginx_1 | 17.70.207.70 - - [30/Mar/2021:21:28:59 +0000] "POST /api/19/store/ HTTP/1.1" 503 94 "-" "sentry.python/0.19.5" "-"
relay_1 | 2021-03-30T21:28:59Z [relay_server::endpoints::common] ERROR: error handling request: failed to queue envelope
relay_1 | caused by: Too many events (event_buffer_size reached)
nginx_1 | 17.70.207.70 - - [30/Mar/2021:21:28:59 +0000] "POST /api/19/store/ HTTP/1.1" 503 94 "-" "sentry.python/0.19.5" "-"
clickhouse_1 | 2021.03.30 21:28:59.380090 [ 110 ] {86efe316-741a-44c1-9706-c91d0cec96a7} <Information> executeQuery: Read 6 rows, 353.00 B in 0.004 sec., 1352 rows/sec., 77.69 KiB/sec.
clickhouse_1 | 2021.03.30 21:28:59.380286 [ 110 ] {} <Information> HTTPHandler: Done processing query
snuba-outcomes-consumer_1 | 2021-03-30 21:28:59,382 Completed processing <Batch: 6 messages, open for 0.84 seconds>.
relay_1 | 2021-03-30T21:28:59Z [relay_server::endpoints::common] ERROR: error handling request: failed to queue envelope
relay_1 | caused by: Too many events (event_buffer_size reached)