New install fails on Ubuntu 20.04 - kafta errors

On-Premise
1

This is section from sentry_install_log-2020-12-13_11-42-42.txt

reating sentry_onpremise_clickhouse_1 ... done
Creating sentry_onpremise_kafka_1      ... done
+ '[' b = - ']'
+ snuba bootstrap --help
+ set -- snuba bootstrap --no-migrate --force
+ set gosu snuba snuba bootstrap --no-migrate --force
+ exec gosu snuba snuba bootstrap --no-migrate --force
%3|1607859859.801|FAIL|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.22.0.5:9092 failed: Connection refused (after 3ms in state CONNECT)
%3|1607859860.798|FAIL|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.22.0.5:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical er
ror(s) suppressed)
2020-12-13 11:44:20,798 Connection to Kafka failed (attempt 0)
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 55, in bootstrap
    client.list_topics(timeout=1)
cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str="Failed to get metadata: Local: Broker transport failure"}
%3|1607859861.800|FAIL|rdkafka#producer-2| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.22.0.5:9092 failed: Connection refused (after 0ms in state CONNECT)
%3|1607859862.800|FAIL|rdkafka#producer-2| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.22.0.5:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical er
ror(s) suppressed)
2020-12-13 11:44:22,806 Connection to Kafka failed (attempt 1)
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 55, in bootstrap
    client.list_topics(timeout=1)
cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str="Failed to get metadata: Local: Broker transport failure"}
%3|1607859863.808|FAIL|rdkafka#producer-3| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.22.0.5:9092 failed: Connection refused (after 0ms in state CONNECT)
%3|1607859864.808|FAIL|rdkafka#producer-3| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.22.0.5:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical er
ror(s) suppressed)
2020-12-13 11:44:24,809 Connection to Kafka failed (attempt 2)
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 55, in bootstrap
    client.list_topics(timeout=1)
cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str="Failed to get metadata: Local: Broker transport failure"}
2020-12-13 11:44:26,091 Failed to create topic outcomes
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 91, in bootstrap
    future.result()
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 439, in result
    return self.__get_result()
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
    raise self._exception
cimpl.KafkaException: KafkaError{code=TOPIC_ALREADY_EXISTS,val=36,str="Topic 'outcomes' already exists."}
2020-12-13 11:44:26,093 Failed to create topic events
2

Tried it on a different Ubuntu 20.10 server…exactly the same error seen in install logs.

Looks likely to be related to this issue…but the fix given there to rm kafta lock has no effect

3 
...
Docker images built.
Removing network onpremise_default
Network onpremise_default not found.
Removing network sentry_onpremise_default
Network sentry_onpremise_default not found.
Creating network "sentry_onpremise_default" with the default driver
Creating volume "sentry_onpremise_sentry-secrets" with default driver
Creating volume "sentry_onpremise_sentry-smtp" with default driver
Creating volume "sentry_onpremise_sentry-zookeeper-log" with default driver
Creating volume "sentry_onpremise_sentry-kafka-log" with default driver
Creating volume "sentry_onpremise_sentry-smtp-log" with default driver
Creating volume "sentry_onpremise_sentry-clickhouse-log" with default driver
Creating sentry_onpremise_zookeeper_run ... 
Creating sentry_onpremise_zookeeper_run ... done
Bootstrapping and migrating Snuba...
Creating sentry_onpremise_redis_1 ... 
Creating sentry_onpremise_zookeeper_1 ... 
Creating sentry_onpremise_clickhouse_1 ... 
Creating sentry_onpremise_clickhouse_1 ... done
Creating sentry_onpremise_redis_1      ... done
Creating sentry_onpremise_zookeeper_1  ... done
Creating sentry_onpremise_kafka_1      ... 
Creating sentry_onpremise_kafka_1      ... done
Creating sentry_onpremise_snuba-api_run ... 
Creating sentry_onpremise_snuba-api_run ... done
+ '[' b = - ']'
+ snuba bootstrap --help
+ set -- snuba bootstrap --no-migrate --force
+ set gosu snuba snuba bootstrap --no-migrate --force
+ exec gosu snuba snuba bootstrap --no-migrate --force
%3|1607862186.614|FAIL|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.18.0.5:9092 failed: Connection refused (after 2ms in state CONNECT)
%3|1607862187.612|FAIL|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.18.0.5:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
2020-12-13 12:23:07,612 Connection to Kafka failed (attempt 0)
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 55, in bootstrap
    client.list_topics(timeout=1)
cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str="Failed to get metadata: Local: Broker transport failure"}
%3|1607862188.614|FAIL|rdkafka#producer-2| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.18.0.5:9092 failed: Connection refused (after 0ms in state CONNECT)
%3|1607862189.613|FAIL|rdkafka#producer-2| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.18.0.5:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
2020-12-13 12:23:09,614 Connection to Kafka failed (attempt 1)
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 55, in bootstrap
    client.list_topics(timeout=1)
cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str="Failed to get metadata: Local: Broker transport failure"}
%3|1607862190.616|FAIL|rdkafka#producer-3| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.18.0.5:9092 failed: Connection refused (after 0ms in state CONNECT)
2020-12-13 12:23:11,617 Connection to Kafka failed (attempt 2)
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 55, in bootstrap
    client.list_topics(timeout=1)
cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str="Failed to get metadata: Local: Broker transport failure"}
2020-12-13 12:23:13,415 Topic outcomes created
2020-12-13 12:23:13,416 Topic events created
2020-12-13 12:23:13,416 Topic errors-replacements created
2020-12-13 12:23:13,417 Topic cdc created
2020-12-13 12:23:13,417 Topic event-replacements created
2020-12-13 12:23:13,417 Topic snuba-commit-log created
2020-12-13 12:23:13,417 Topic ingest-sessions created
Creating sentry_onpremise_snuba-api_run ... 
Creating sentry_onpremise_snuba-api_run ... done
+ '[' m = - ']'
+ snuba migrations --help
+ set -- snuba migrations migrate --force
+ set gosu snuba snuba migrations migrate --force
+ exec gosu snuba snuba migrations migrate --force
Finished running migrations

Creating additional Kafka topics...
Creating sentry_onpremise_kafka_run ... 
Creating sentry_onpremise_kafka_run ... done
Created topic ingest-attachments.

Creating additional Kafka topics...
Creating sentry_onpremise_kafka_run ... 
Creating sentry_onpremise_kafka_run ... done
Created topic ingest-transactions.

Creating additional Kafka topics...
Creating sentry_onpremise_kafka_run ... 
Creating sentry_onpremise_kafka_run ... done
Created topic ingest-events.


Setting up database...
Creating sentry_onpremise_symbolicator_1 ... 
Creating sentry_onpremise_postgres_1     ... 
Creating sentry_onpremise_memcached_1    ... 
Creating sentry_onpremise_smtp_1         ... 
Creating sentry_onpremise_snuba-outcomes-consumer_1 ... 
Creating sentry_onpremise_snuba-sessions-consumer_1 ... 
Creating sentry_onpremise_snuba-transactions-consumer_1 ... 
Creating sentry_onpremise_snuba-replacer_1              ... 
Creating sentry_onpremise_snuba-consumer_1              ... 
Creating sentry_onpremise_snuba-api_1                   ... 
Creating sentry_onpremise_symbolicator_1                ... done
Creating sentry_onpremise_postgres_1                    ... done
Creating sentry_onpremise_smtp_1                        ... done
Creating sentry_onpremise_snuba-transactions-consumer_1 ... done
Creating sentry_onpremise_memcached_1                   ... done
Creating sentry_onpremise_snuba-outcomes-consumer_1     ... done
Creating sentry_onpremise_snuba-sessions-consumer_1     ... done
Creating sentry_onpremise_snuba-api_1                   ... done
Creating sentry_onpremise_snuba-consumer_1              ... done
Creating sentry_onpremise_snuba-replacer_1              ... done
Creating sentry_onpremise_web_run                       ... 
Creating sentry_onpremise_web_run                       ... done
/usr/local/lib/python3.6/site-packages/sentry/runner/initializer.py:189: DeprecatedSettingWarning: The EMAIL_USE_SSL setting is deprecated. Please use SENTRY_OPTIONS['mail.use-ssl'] instead.
  warnings.warn(DeprecatedSettingWarning(options_mapper[k], "SENTRY_OPTIONS['%s']" % k))
12:23:47 [WARNING] sentry.utils.geo: settings.GEOIP_PATH_MMDB not configured.
12:23:56 [INFO] sentry.plugins.github: apps-not-configured
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, jira_ac, nodestore, sentry, sessions, sites, social_auth
Running migrations:
  Applying sentry.0001_initial... OK
  Applying contenttypes.0001_initial... OK
4

seems like you have the same or similar issues to me, just for reference here’s my thread: Nothing can connect to Kafka

5

I think this looks like a Ubuntu Docker issue with DNS / container connectivity.

Just trying on a third server…

âžś  onpremise git:(master) ./install.sh 
Checking minimum requirements...

Creating volumes for persistent storage...
Created sentry-data.
Created sentry-postgres.
Created sentry-redis.
Created sentry-zookeeper.
Created sentry-kafka.
Created sentry-clickhouse.
Created sentry-symbolicator.

Creating sentry/sentry.conf.py...
Creating sentry/config.yml...
Creating sentry/requirements.txt...
Creating symbolicator/config.yml...
Creating relay/config.yml...

Generating secret key...
Secret key written to sentry/config.yml

Fetching and updating Docker images...



Some service image(s) must be built from source by running:
    docker-compose build subscription-consumer-events sentry-cleanup symbolicator-cleanup subscription-consumer-transactions snuba-cleanup post-process-forwarder worker cron ingest-consumer web
nightly: Pulling from getsentry/sentry
8559a31e96f4: Pulling fs layer
62e60f3ef11e: Pulling fs layer
6f66b7415d8a: Pulling fs layer
7814b5d1b3ca: Pulling fs layer
0ac0b60bec89: Pulling fs layer
3fd81d69d2f9: Pulling fs layer
3fec722b988a: Pulling fs layer
b9f6871d5dc3: Pulling fs layer
c7c72c8e24b1: Pulling fs layer
0edfe5806820: Pulling fs layer
d33840f142bb: Pulling fs layer
3fec722b988a: Waiting
b9f6871d5dc3: Waiting
7814b5d1b3ca: Waiting
0ac0b60bec89: Waiting
c7c72c8e24b1: Waiting
0edfe5806820: Waiting
3fd81d69d2f9: Waiting
d33840f142bb: Waiting
4ffca871adb1: Pulling fs layer
7889caf1e848: Pulling fs layer
108f9e4d8d76: Pulling fs layer
ff92af2dbf0b: Pulling fs layer

...

...


Creating volume "sentry_onpremise_sentry-zookeeper-log" with default driver
Creating volume "sentry_onpremise_sentry-kafka-log" with default driver
Creating volume "sentry_onpremise_sentry-smtp-log" with default driver
Creating volume "sentry_onpremise_sentry-clickhouse-log" with default driver
Creating sentry_onpremise_zookeeper_run ... 
Creating sentry_onpremise_zookeeper_run ... done
Bootstrapping and migrating Snuba...
Creating sentry_onpremise_zookeeper_1 ... 
Creating sentry_onpremise_redis_1     ... 
Creating sentry_onpremise_clickhouse_1 ... 
Creating sentry_onpremise_zookeeper_1  ... done
Creating sentry_onpremise_kafka_1      ... 
Creating sentry_onpremise_redis_1      ... done
Creating sentry_onpremise_clickhouse_1 ... done
Creating sentry_onpremise_kafka_1      ... done
Creating sentry_onpremise_snuba-api_run ... 
Creating sentry_onpremise_snuba-api_run ... done
+ '[' b = - ']'
+ snuba bootstrap --help
+ set -- snuba bootstrap --no-migrate --force
+ set gosu snuba snuba bootstrap --no-migrate --force
+ exec gosu snuba snuba bootstrap --no-migrate --force
%3|1607965663.606|FAIL|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.18.0.5:9092 failed: Connection refused (after 0ms in state CONNECT)
%3|1607965664.605|FAIL|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.18.0.5:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
2020-12-14 17:07:44,605 Connection to Kafka failed (attempt 0)
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 55, in bootstrap
    client.list_topics(timeout=1)
cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str="Failed to get metadata: Local: Broker transport failure"}
%3|1607965665.609|FAIL|rdkafka#producer-2| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.18.0.5:9092 failed: Connection refused (after 0ms in state CONNECT)
%3|1607965666.609|FAIL|rdkafka#producer-2| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.18.0.5:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
2020-12-14 17:07:46,612 Connection to Kafka failed (attempt 1)
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 55, in bootstrap
    client.list_topics(timeout=1)
cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str="Failed to get metadata: Local: Broker transport failure"}
2020-12-14 17:07:48,065 Topic events created
2020-12-14 17:07:48,065 Topic errors-replacements created
2020-12-14 17:07:48,065 Topic outcomes created
6

How can I upload a test file here ??

docker-compose logs … gives…
[[35mingest-consumer_1 |^[[0m %3|1607966058.386|FAIL|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.18.0.11:9092 failed: Connection refused (after 0ms in state CONNECT)
^[[35mingest-consumer_1 |^[[0m %3|1607966059.383|FAIL|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.18.0.11:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
^[[35mingest-consumer_1 |^[[0m %3|1607966089.387|FAIL|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.18.0.11:9092 failed: Connection refused (after 1ms in state CONNECT, 30 identical error(s) suppressed)

7

Screenshot from 2020-12-14 17-29-30
Screenshot from 2020-12-14 17-29-301593Ă—702 92.7 KB

Not sure if the configuration error is relevant ? But this is from a clean install.

8

dave@staging:~/onpremise$ docker run --rm --network=my-pre-existing-network centos:7 ping -c 4 172.22.0.10
PING 172.22.0.10 (172.22.0.10) 56(84) bytes of data.
64 bytes from 172.22.0.10: icmp_seq=1 ttl=64 time=0.252 ms
64 bytes from 172.22.0.10: icmp_seq=2 ttl=64 time=0.082 ms
64 bytes from 172.22.0.10: icmp_seq=3 ttl=64 time=0.060 ms
64 bytes from 172.22.0.10: icmp_seq=4 ttl=64 time=0.079 ms

9

I added a docker network to the docker-compose to run this test…

dave@staging:~/onpremise$ docker run --rm --network=my-pre-existing-network centos:7 ping -c 4 172.22.0.10

10

dave@staging:~/onpremise$ docker run --rm --network=my-pre-existing-network centos:7 ping -c 4 kafka
PING kafka (172.22.0.10) 56(84) bytes of data.
64 bytes from sentry_onpremise_kafka_1.my-pre-existing-network (172.22.0.10): icmp_seq=1 ttl=64 time=0.303 ms
64 bytes from sentry_onpremise_kafka_1.my-pre-existing-network (172.22.0.10): icmp_seq=2 ttl=64 time=0.075 ms
64 bytes from sentry_onpremise_kafka_1.my-pre-existing-network (172.22.0.10): icmp_seq=3 ttl=64 time=0.109 ms
64 bytes from sentry_onpremise_kafka_1.my-pre-existing-network (172.22.0.10): icmp_seq=4 ttl=64 time=0.121 ms

11

5a8fa1a463a getsentry/snuba:nightly “./docker_entrypoint…” 6 minutes ago Up 6 minutes 1218/tcp sentry_onpremise_snuba-subscription-consumer-events_1
c0ca30c0a50d confluentinc/cp-kafka:5.5.0 “/etc/confluent/dock…” 6 minutes ago Up 6 minutes 9092/tcp sentry_onpremise_kafka_1
24e20ff1d49d memcached:1.5-alpine “docker-entrypoint.s…” 6 minutes ago Up 6 minutes 11211/tcp sentry_onpremise_memcached_1
be1f087ec193 tianon/exim4 “docker-entrypoint.s…” 6 minutes ago Up 6 minutes 25/tcp sentry_onpremise_smtp_1
5569513f3a24 getsentry/symbolicator:nightly “/bin/bash /docker-e…” 6 minutes ago Up 6 minutes 3021/tcp sentry_onpremise_symbolicator_1
16865da28fc4 postgres:9.6 “docker-entrypoint.s…” 6 minutes ago Up 6 minutes 5432/tcp sentry_onpremise_postgres_1
07943aa2a2eb yandex/clickhouse-server:20.3.9.70 “/entrypoint.sh” 6 minutes ago Up 6 minutes 8123/tcp, 9000/tcp, 9009/tcp sentry_onpremise_clickhouse_1
6df8e7c57a71 symbolicator-cleanup-onpremise-local “/entrypoint.sh '55 …” 6 minutes ago Up 6 minutes 3021/tcp sentry_onpremise_symbolicator-cleanup_1
8f48517e3dbc confluentinc/cp-zookeeper:5.5.0 “/etc/confluent/dock…” 6 minutes ago Up 6 minutes 2181/tcp, 2888/tcp, 3888/tcp sentry_onpremise_zookeeper_1
831f4ad5c3c3 redis:5.0-alpine “docker-entrypoint.s…” 6 minutes ago Up 6 minutes 6379/tcp sentry_onpremise_redis_1
dave@staging:~/onpremise$ docker exec -it c0ca30c0a50d /bin/bash
root@c0ca30c0a50d:/#
root@c0ca30c0a50d:/#
root@c0ca30c0a50d:/#
root@c0ca30c0a50d:/# ls
bin boot dev etc home lib lib64 media mnt opt proc root run sbin srv sys tmp usr var
root@c0ca30c0a50d:/# ps -ef
UID PID PPID C STIME TTY TIME CMD
root 1 0 7 17:45 ? 00:00:37 java -Xmx1G -Xms1G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGC
root 206 0 0 17:53 pts/0 00:00:00 /bin/bash
root 214 206 0 17:53 pts/0 00:00:00 ps -ef
root@c0ca30c0a50d:/#

So I can login into kafka container…but have no idea what to run to test this ?

can ping all the names like zookeeper / web etc

12

Have tried to put as much debug info into this ticket…hopefully someone can see enough to figure out what is wrong - but essentially onpremise fails on Ubuntu 20.04 to start receiving events.

13

can confirm, im also on ubuntu 20.04

14

I have seen that you have tried on different servers. What docker version are you using?
Which onPrem-version did you try? “nightly: Pulling from getsentry/sentry” - Only this? Have you tried 20.11.1?
Do you clone the official repo or do you have your own?

15

Thank you for helping.

git clone the official repository.

Docker version 20.10.0, build 7287ab3

How do I get 20.11.1 installed - happy to try this.

Pretty sure you will see the same errors if you try to run onpremise on Ubuntu 20.04 or 20.10

Let me know if there is anything else needed, can provide a temporary login to a Ubuntu cloud server - if helpful.

Kind regards Dave

16

No problem. The forum is there for us to help eachother, right :slight_smile: ?
Instead of master, you clone the branch named 20.11.1.
On my Ubuntu 20.04 Im running version 19.03.11 of Docker and version 1.25.5 of Docker-compose.
Maybe you could try with these versions? And if possible, wipe all images, volumes etc to start of from scratch.

Tell me how it goes

1 Like
17

hey not the same person here but i have the same issue:
i tried the release:20.11.1 branch on my ubuntu 20.04 with the same issue, at that point i had already updated the docker version though. and yea did wipe everything.

18

Are you able to downgrade the docker version and try?
I might be naive but running sentry from scratch with no special changes should work. That’s why I think it could be good to try a different docker version.

19

im running like 20 other containers and dont know how that will affect things, i did try sentry on my previous docker version which was some variant of 19 though i dont remember which exact version. I re-created it at least 5 times on the old version with the same error.

20

Yeah I wouldn’t recommend downgrade when running critical containers.
Anyways I don’t know the solutions just what I would try if I had these issues. Also i have seen that for example Home Assistant had some issues with the 20.* Versions.

Edit: try updating docker to 20.10.1 looks like there has been some fixes to it vs 20.10.0