New Sentry 10 docker and git versions unable to get past error

On-Premise
21

Google Cloud Build just uses the same servers from Google Cloud server. It’s just a specialized version for CI stuff. I’m sorry I cannot get you a quicker answer on this one :frowning:

One thing you may try in the meantime is getting your own base Snuba image and add more logging into it to see why it simply fails. I suspect some environment variable fails somewhere but can’t be entirely sure. This is how Snuba is built: https://github.com/getsentry/snuba/blob/master/cloudbuild.yaml (essentially just docker build -t getsentry/snuba:latest --build-arg SNUBA_VERSION_SHA=<commit_sha> in the repo).

22

Let the mystery get even more mysterious. I was messing around and doing multiple back to back clean up everything docker, get git again, and then re-installs on the same server and 2 times now out of 7-8 installs the outcome tonight was different. I was trying to understand snuba a bit more and was breaking everything so I kept restarting over and over. Never saw this until tonight, here is the ending message of the install:

Added to organization: sentry
Creating missing DSNs
Correcting Group.num_comments counter
Starting crashes_zookeeper_1  ... done
Starting crashes_clickhouse_1 ... done
Starting crashes_kafka_1      ... done
+ '[' bootstrap = bash ']'e_1 ... done
+ '[' b = - ']'
+ '[' bootstrap = api ']'
+ snuba bootstrap --help
+ set -- snuba bootstrap --force
+ exec gosu snuba snuba bootstrap --force
Topic outcomes created
Failed to create topic cdc: KafkaError{code=INVALID_REQUEST,val=42,str="Create topics request from client `rdkafka` contains multiple entries for the following topics: cdc,events"}
Failed to create topic events: KafkaError{code=INVALID_REQUEST,val=42,str="Create topics request from client `rdkafka` contains multiple entries for the following topics: cdc,events"}
Failed to create topic event-replacements: KafkaError{code=TOPIC_ALREADY_EXISTS,val=36,str="Topic 'event-replacements' already exists."}
Topic snuba-commit-log created

To add also the first time I got the difference above, the site mainly worked with the occasional 404 regarding searching events. The second time around though it has not spewed out a single error yet. I’d say I have a working site at the moment though scared that in a day it’ll start on fire but we’ll see as I haven’t sent any events to it yet but at least the site runs. The bug is definitely still repeatable if you spin up an AWS or Linode still so I’d give that a try and hope you solve the mystery. Thank you once again for the earlier help and good luck!

23

Thanks a lot for all the collaboration @SuperServer! I think I nailed it down: some of the services snuba depends on takes a bit longer to get up but docker-compose only waits for container readiness so snuba simply fails at that point. I’ve added an until - do loop in the install script with a check for this table which should resolve the issue: https://github.com/getsentry/onpremise/pull/220#issuecomment-551997213

Would you dare to try again with that fix? :smiley:

1 Like
24

I took that dare and it worked and you can see it in action too:

Added to organization: sentry
Creating missing DSNs
Correcting Group.num_comments counter
Boostrapping Snuba...
crashes_zookeeper_1 is up-to-date
crashes_clickhouse_1 is up-to-date
crashes_redis_1 is up-to-date
crashes_kafka_1 is up-to-date
Starting crashes_zookeeper_1 ... done
Starting crashes_clickhouse_1 ... done
Starting crashes_redis_1      ... done
Starting crashes_kafka_1      ... done
+ '[' bootstrap = bash ']'
+ '[' b = - ']'
+ '[' bootstrap = api ']'
+ snuba bootstrap --help
+ set -- snuba bootstrap --force
+ exec gosu snuba snuba bootstrap --force
Starting crashes_redis_1      ... done
Starting crashes_zookeeper_1  ... done
Starting crashes_clickhouse_1 ... done
Starting crashes_kafka_1      ... done
+ '[' bootstrap = bash ']'
+ '[' b = - ']'
+ '[' bootstrap = api ']'
+ snuba bootstrap --help
+ set -- snuba bootstrap --force
+ exec gosu snuba snuba bootstrap --force
Failed to create topic outcomes: KafkaError{code=TOPIC_ALREADY_EXISTS,val=36,str="Topic 'outcomes' already exists."}
Failed to create topic events: KafkaError{code=INVALID_REQUEST,val=42,str="Create topics request from client `rdkafka` contains multiple entries for the following topics: cdc,events"}
Failed to create topic event-replacements: KafkaError{code=TOPIC_ALREADY_EXISTS,val=36,str="Topic 'event-replacements' already exists."}
Failed to create topic snuba-commit-log: KafkaError{code=TOPIC_ALREADY_EXISTS,val=36,str="Topic 'snuba-commit-log' already exists."}
Failed to create topic cdc: KafkaError{code=INVALID_REQUEST,val=42,str="Create topics request from client `rdkafka` contains multiple entries for the following topics: cdc,events"}

Cleaning up...

----------------
You're all done! Run the following command to get Sentry running:

Site seems to be in good working order. Now I just need to figure out how to get email and clickup connected but that’s another adventure for another day hehe. Thank you once again!

1 Like