New Sentry 10 docker and git versions unable to get past error

Trying on both the latest getsentry/sentry docker image (updated 15 hours ago) and even git master locally build I always get the same error regardless of what I attempt to do.

Starting local_smtp_1 … done
Starting local_postgres_1 … done
16:38:05 [WARNING] sentry.utils.geo: settings.GEOIP_PATH_MMDB not configured.
16:38:10 [INFO] sentry.plugins.github: apps-not-configured
Traceback (most recent call last):
File “/usr/local/bin/sentry”, line 8, in
sys.exit(main())
File “/usr/local/lib/python2.7/site-packages/sentry/runner/init.py”, line 164, in main
cli(prog_name=get_prog(), obj={}, max_content_width=100)
File “/usr/local/lib/python2.7/site-packages/click/core.py”, line 722, in call
return self.main(*args, **kwargs)
File “/usr/local/lib/python2.7/site-packages/click/core.py”, line 697, in main
rv = self.invoke(ctx)
File “/usr/local/lib/python2.7/site-packages/click/core.py”, line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File “/usr/local/lib/python2.7/site-packages/click/core.py”, line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File “/usr/local/lib/python2.7/site-packages/click/core.py”, line 535, in invoke
return callback(*args, **kwargs)
File “/usr/local/lib/python2.7/site-packages/click/decorators.py”, line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File “/usr/local/lib/python2.7/site-packages/sentry/runner/decorators.py”, line 30, in inner
return ctx.invoke(f, *args, **kwargs)
File “/usr/local/lib/python2.7/site-packages/click/core.py”, line 535, in invoke
return callback(*args, **kwargs)
File “/usr/local/lib/python2.7/site-packages/click/decorators.py”, line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File “/usr/local/lib/python2.7/site-packages/sentry/runner/commands/upgrade.py”, line 168, in upgrade
_upgrade(not noinput, traceback, verbosity, not no_repair)
File “/usr/local/lib/python2.7/site-packages/sentry/runner/commands/upgrade.py”, line 121, in _upgrade
_migrate_from_south(verbosity)
File “/usr/local/lib/python2.7/site-packages/sentry/runner/commands/upgrade.py”, line 93, in _migrate_from_south
if not _has_south_history(connection):
File “/usr/local/lib/python2.7/site-packages/sentry/runner/commands/upgrade.py”, line 78, in _has_south_history
cursor = connection.cursor()
File “/usr/local/lib/python2.7/site-packages/django/db/backends/base/base.py”, line 164, in cursor
cursor = self.make_cursor(self._cursor())
File “/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py”, line 44, in inner
return func(self, *args, **kwargs)
File “/usr/local/lib/python2.7/site-packages/sentry/db/postgres/base.py”, line 97, in _cursor
cursor = super(DatabaseWrapper, self)._cursor()
File “/usr/local/lib/python2.7/site-packages/django/db/backends/base/base.py”, line 135, in _cursor
self.ensure_connection()
File “/usr/local/lib/python2.7/site-packages/django/db/backends/base/base.py”, line 130, in ensure_connection
self.connect()
File “/usr/local/lib/python2.7/site-packages/django/db/utils.py”, line 98, in exit
six.reraise(dj_exc_type, dj_exc_value, traceback)
File “/usr/local/lib/python2.7/site-packages/django/db/backends/base/base.py”, line 130, in ensure_connection
self.connect()
File “/usr/local/lib/python2.7/site-packages/django/db/backends/base/base.py”, line 119, in connect
self.connection = self.get_new_connection(conn_params)
File “/usr/local/lib/python2.7/site-packages/django/db/backends/postgresql_psycopg2/base.py”, line 176, in get_new_connection
connection = Database.connect(**conn_params)
File “/usr/local/lib/python2.7/site-packages/psycopg2/init.py”, line 125, in connect
dsn = _ext.make_dsn(dsn, **kwargs)
File “/usr/local/lib/python2.7/site-packages/psycopg2/extensions.py”, line 174, in make_dsn
parse_dsn(dsn)
django.db.utils.ProgrammingError: invalid dsn: invalid connection option “autocommit”

I saw another forum post regarding this issue a year ago by removing autocommit from the sentry file and then also the commit https://github.com/getsentry/onpremise/pull/195

However I do not see this autocommit setting anywhere in the configuration files so I’m not sure where’s its running into this issue. Just that the web upgrade command never works without error on new installs of sentry 10. Ideas?

Can you share your complete setup and your config files please?

I’ve tried multiple different ways but here’s the latest/simplest ways I can get there.

  1. Installed Ubuntu 18.04 and do basic update. Install docker and docker compose per their instructions for 18.04
  2. Got this: https://github.com/getsentry/onpremise
  3. No alteration to the config.yml, .env, or sentry.conf.py
  4. Alter Dockerfile to use FROM ${SENTRY_IMAGE:-getsentry/sentry } (verified the digests match up so I am getting this image from https://hub.docker.com/r/getsentry/sentry)
  5. Alter Makefile so that Repo is getsentry/sentry and the image is getsentry/sentry.
  6. Run ./install.sh
  7. Get the message above once it gets to the upgrade piece.

I got past the initial error by spinning up the container, altering the container’s /etc/sentry/sentry.conf.py and making a new docker image. Though I’m debating if this is really the big issue. Because then I did a nice fresh install based off the altered image and the error went past but now the worker process is on fire basically. The site itself works great except when it comes to viewing issues. The worker itself must be spewing out errors which I’m having a really difficult time trying to figure out how to see those errors. Can’t fix what I can’t see. But as you can see in the picture, it’s making a few.

sentry_errors

Oh, if you are using the on-premise repo, you need to be using the v10 branch as Sentry 10 requires a bit more infrastructure. Give it a shot with that repo and with the included ./install.sh script and LMK how that goes!

Thank you! That helped it get past the initial setup issues. But even with a fresh everything most of the site doesn’t work at all. This is once again with no customization of any sort. Should v10 be mostly working or is there still a lot of issues with it?

Projects, Events, Issues, Releases don’t work at all, instead they give a popup of " It looks like we’re having issues.", “Oops! Something went wrong”, or Internal Error.

Stats and Discover pages work without issue, stats keeps saying an error is appearing like 20-30 errors a minute.

Steps taken:

  1. Ubuntu 18.04 with updates, docker and docker compose.
  2. git clone https://github.com/getsentry/onpremise.git
  3. git checkout v10
  4. Verified switch that we’re on the v10 stuff now.
  5. ./install.sh
  6. docker-compose up -d
  7. Went to site and we have fire!

Edit: Added sentry containers image.

That’s quite weird as this is the test we are running per built image to make sure v10 works off of the latest master. Moreover, I just did an install about an hour ago and things were working pretty well.

Can you share the logs with us: docker-compose logs web to see what is going wrong?

I think I can solve it in 1 line that’s repeated a lot in the logs:

DB::Exception: Table default.sentry_local doesn’t exist… Stack trace:

My recommendation would be to do the following for a full, clean install:

docker-compose down
docker-compose rm
docker volume rm sentry-data sentry-postgres sentry-redis sentry-zookeeper sentry-kafka sentry-clickhouse sentry-symbolicator
docker rmi getsentry/snuba:latest getsentry/sentry:latest us.gcr.io/sentryio/symbolicator:latest sentry-onpremise-local

This should make sure all the data is gone and all the images are pulled fresh.

Same issue sadly. Attached some screenshots of the process so its definitely a nice clean install and it says its doing everything it needs to. Sorry for all the pictures, easier than copy and pasting out of putty without it splitting everything up. DB::Exception: Table default.sentry_local doesn’t exist is just everywhere in the log and that’s it.

Additionally, perhaps the issue is related to: https://github.com/ClickHouse/ClickHouse/issues/2250

Connecting to the DB does show that sentry_local doesn’t exist and I’m guessing it did at one point during setup? I could remake the table however I’m not sure what was setup in here.

Edit: Though when I look at the snuba-api logs I see this:

Error running query: SELECT project_id, (sentry:release AS tags[sentry:release]), (count() AS times_seen), (min(timestamp) AS first_seen), (max(timestamp) AS last_seen) FROM sentry_local PREWHERE tags[sentry:release] IN (‘10.0.0.dev0’) WHERE project_id IN (1) AND timestamp >= toDateTime(‘1970-01-01T00:00:00’) AND timestamp < toDateTime(‘2019-11-06T03:35:16’) AND deleted = 0 GROUP BY (project_id, tags[sentry:release]) LIMIT 0, 1000

It really wants that sentry_local DB entry.

Okay, my best guess is the following step failing for you:

Probably because it takes too long for Kafka, Zookeeper etc. to come up. Can you run that command (docker-compose run --rm snuba-api bootstrap --force) manually until you get a success message?

Ideally you’d run this a short while after running docker-compose up -d when all the services are up.

What’s a success message look like? These say done almost instantly along with the rest of it. When setting it all up too this part all the services starting went really quick. If AMD EPYC 7501 32-Core Processors on nvme drives can’t do this fast enough I’m quite concern about the requirements of Sentry. Ran it twice but I don’t see like a “message”:

root@localhost:/sentry/testing# docker-compose run --rm snuba-api bootstrap --force
Starting testing_zookeeper_1  ... done
Starting testing_clickhouse_1 ... done
Starting testing_redis_1      ... done
Starting testing_kafka_1      ... done
+ '[' bootstrap = bash ']'
+ '[' b = - ']'
+ '[' bootstrap = api ']'
+ snuba bootstrap --help
+ set -- snuba bootstrap --force
+ exec gosu snuba snuba bootstrap --force
root@localhost:/sentry/testing# docker-compose run --rm snuba-api bootstrap --force
Starting testing_clickhouse_1 ... done
Starting testing_zookeeper_1  ... done
Starting testing_redis_1      ... done
Starting testing_kafka_1      ... done
+ '[' bootstrap = bash ']'
+ '[' b = - ']'
+ '[' bootstrap = api ']'
+ snuba bootstrap --help
+ set -- snuba bootstrap --force
+ exec gosu snuba snuba bootstrap --force

Okay not seeing a message looks like the actual problem. Also it is a bit weird that this is happening inside the web_1 container.

Can you share the full install logs in text form so I can investigate more? Also the output of docker images (feel free to omit redacting non-sentry parts).

Made a new server and everything again just to be clean about it all. I put the output of docker images and containers at the very end. I’m assuming this is the logs you’re looking for, if not let me know which other ones to pull.

I put the entire output here: https://pastebin.com/WNDbRA1X

Sorry but I cannot figure out what the issue is. For some reason Snuba bootstrap silently fails which causes the issue. Do you have any custom networking, or sysctl settings on your system?

Nothing at all like that since they are just fresh installs of Ubuntu on servers that are directly connected to the Internet. No firewalls or any sort of customized settings. The only pieces you’re missing from the start of that log is logging into the server and doing apt-get update along with getting docker per docker’s instructions.

Decided to try some server providers, this both being instances in AWS and Linode. The same results using basic images they provide so I’m quite out of ideas what to try. I could try older versions of Debian and Ubuntu perhaps to see one of those work since Ubuntu 18.04 and Debian 9 both do not.

Okay, I’ll try to do this myself and come back but it may fall to next week. Is that okay? I’ve been testing these out on Google Cloud Build and on my WSL/Ubuntu 18.04 without any issues so I’m quite confused about what’s wrong here.

Really need it as soon as possible but when ever you get time it’s very much appreciated. I’m not familiar with Google Cloud builds though so I’m guessing this isn’t an option for us? I could spin up a Google Cloud server but I’m sure a plain server is different than what google cloud build does. Thank you for your time!

1 Like