Sentry upgrade from 9.1.2 to 10 fails

On-Premise
1

Hi,

I have tried several approaches mainly the documented ones, in order to upgrade from 9.1.2 to 10. It is said to be supported out of the box onpremise.

I have runned ./install.sh resulting in:

Traceback (most recent call last):
  File "/usr/local/bin/sentry", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/__init__.py", line 166, in main
    cli(prog_name=get_prog(), obj={}, max_content_width=100)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/decorators.py", line 30, in inner
    return ctx.invoke(f, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/commands/upgrade.py", line 174, in upgrade
    _upgrade(not noinput, traceback, verbosity, not no_repair, with_nodestore)
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/commands/upgrade.py", line 121, in _upgrade
    _migrate_from_south(verbosity)
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/commands/upgrade.py", line 93, in _migrate_from_south
    if not _has_south_history(connection):
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/commands/upgrade.py", line 78, in _has_south_history
    cursor = connection.cursor()
  File "/usr/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 254, in cursor
    return self._cursor()
  File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py", line 44, in inner
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/base.py", line 97, in _cursor
    return super(DatabaseWrapper, self)._cursor()
  File "/usr/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 229, in _cursor
    self.ensure_connection()
  File "/usr/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 213, in ensure_connection
    self.connect()
  File "/usr/local/lib/python2.7/site-packages/django/db/utils.py", line 94, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/usr/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 213, in ensure_connection
    self.connect()
  File "/usr/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 189, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/usr/local/lib/python2.7/site-packages/django/db/backends/postgresql/base.py", line 176, in get_new_connection
    connection = Database.connect(**conn_params)
  File "/usr/local/lib/python2.7/site-packages/psycopg2/__init__.py", line 127, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not translate host name "postgres" to address: Name or service not known

root@sentry:/home/cosmic/sentry/onpremise#

By runnig simply ./install.sh i get pg errors

`syncing data to disk ... ok`

Success. You can now start the database server using:

`pg_ctl -D /var/lib/postgresql/9.6/data -l logfile start`


WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.

*failure*
Consult the last few lines of "pg_upgrade_server.log" for
the probable cause of the failure.

There seems to be a postmaster servicing the old cluster.
Please shutdown that postmaster and try again.
Failure, exiting
Cleaning up...
root@sentry:/home/cosmic/sentry/onpremise#

I have tried manually docker-compose

docker-compose build --pull --force-rm web
docker-compose build --force-rm

Resulting in above errors:

root@sentry:/home/cosmic/sentry/onpremise# docker -v
Docker version 19.03.11, build dd360c7
root@sentry:/home/cosmic/sentry/onpremise# docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

docker volume ls

local               sentry-clickhouse
local               sentry-data
local               sentry-kafka
local               sentry-postgres
local               sentry-postgres-new
local               sentry-redis
local               sentry-symbolicator
local               sentry-zookeeper
local               sentry_onpremise_sentry-clickhouse-log
local               sentry_onpremise_sentry-kafka-log
local               sentry_onpremise_sentry-secrets
local               sentry_onpremise_sentry-smtp
local               sentry_onpremise_sentry-smtp-log
local               sentry_onpremise_sentry-zookeeper-log
2

Here is the full ./install.sh log:

./install.sh 
Checking minimum requirements...
Removing network onpremise_default
Network onpremise_default not found.
Removing sentry_onpremise_kafka_1      ... 
Removing sentry_onpremise_clickhouse_1 ... 
Removing sentry_onpremise_zookeeper_1  ... 
Removing sentry_onpremise_redis_1      ... 
Removing sentry_onpremise_redis_1      ... done
Removing sentry_onpremise_clickhouse_1 ... done
Removing sentry_onpremise_zookeeper_1  ... done
Removing sentry_onpremise_kafka_1      ... done
Removing network sentry_onpremise_default

Creating volumes for persistent storage...
Created sentry-data.
Created sentry-postgres.
Created sentry-redis.
Created sentry-zookeeper.
Created sentry-kafka.
Created sentry-clickhouse.
Created sentry-symbolicator.

sentry/sentry.conf.py already exists, skipped creation.
sentry/config.yml already exists, skipped creation.
sentry/requirements.txt already exists, skipped creation.

Fetching and updating Docker images...

Some service image(s) must be built from source by running:
    docker-compose build ingest-consumer web snuba-cleanup cron post-process-forwarder worker sentry-cleanup symbolicator-cleanup
latest: Pulling from getsentry/sentry
Digest: sha256:57323e24d454aea64ca10ecf222b6c92022f0f1637caecafba3e08ffd83756be
Status: Image is up to date for getsentry/sentry:latest
docker.io/getsentry/sentry:latest

Building and tagging Docker images...

Building web
Step 1/5 : ARG SENTRY_VERSION=latest
Step 2/5 : ARG SENTRY_IMAGE
Step 3/5 : FROM ${SENTRY_IMAGE:-getsentry/sentry:$SENTRY_VERSION}
 ---> 3edde8d4b2ac
Step 4/5 : COPY . /usr/src/sentry
 ---> Using cache
 ---> f0c27ea60b52
Step 5/5 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then pip install -r /usr/src/sentry/requirements.txt; fi
 ---> Using cache
 ---> 8d26f1b660de

Successfully built 8d26f1b660de
Successfully tagged sentry-onpremise-local:latest
zookeeper uses an image, skipping
memcached uses an image, skipping
postgres uses an image, skipping
redis uses an image, skipping
kafka uses an image, skipping
relay uses an image, skipping
smtp uses an image, skipping
clickhouse uses an image, skipping
snuba-api uses an image, skipping
snuba-outcomes-consumer uses an image, skipping
snuba-replacer uses an image, skipping
snuba-consumer uses an image, skipping
snuba-sessions-consumer uses an image, skipping
symbolicator uses an image, skipping
nginx uses an image, skipping
Building symbolicator-cleanup   ... 
Building snuba-cleanup          ... 
Building ingest-consumer        ... 
Building sentry-cleanup         ... 
Building web                    ... 
Building cron                   ... 
Building worker                 ... 
Building post-process-forwarder ... 
Building cron
Building ingest-consumer
Building snuba-cleanup
Building worker
Building post-process-forwarder
Step 1/5 : ARG SENTRY_VERSION=latest
Step 2/5 : ARG SENTRY_IMAGE
Step 3/5 : FROM ${SENTRY_IMAGE:-getsentry/sentry:$SENTRY_VERSION}
 ---> 3edde8d4b2ac
Step 4/5 : COPY . /usr/src/sentry
 ---> Using cache
 ---> f0c27ea60b52
Step 5/5 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then pip install -r /usr/src/sentry/requirements.txt; fi
 ---> Using cache
 ---> 8d26f1b660de

Successfully built 8d26f1b660de
Step 1/5 : ARG SENTRY_VERSION=latest
Step 2/5 : ARG SENTRY_IMAGE
Step 3/5 : FROM ${SENTRY_IMAGE:-getsentry/sentry:$SENTRY_VERSION}
Successfully tagged sentry-onpremise-local:latest
Building cron                   ... done
 ---> 3edde8d4b2ac
Building web
Step 4/5 : COPY . /usr/src/sentry
 ---> Using cache
 ---> f0c27ea60b52
Step 5/5 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then pip install -r /usr/src/sentry/requirements.txt; fi
 ---> Using cache
 ---> 8d26f1b660de

Successfully built 8d26f1b660de
Successfully tagged sentry-onpremise-local:latest
Building post-process-forwarder ... done
Building sentry-cleanup
Step 1/5 : ARG SENTRY_VERSION=latest
Step 2/5 : ARG SENTRY_IMAGE
Step 3/5 : FROM ${SENTRY_IMAGE:-getsentry/sentry:$SENTRY_VERSION}
 ---> 3edde8d4b2ac
Step 4/5 : COPY . /usr/src/sentry
 ---> Using cache
 ---> f0c27ea60b52
Step 5/5 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then pip install -r /usr/src/sentry/requirements.txt; fi
 ---> Using cache
 ---> 8d26f1b660de

Successfully built 8d26f1b660de
Step 1/5 : ARG SENTRY_VERSION=latest
Step 2/5 : ARG SENTRY_IMAGE
Step 3/5 : FROM ${SENTRY_IMAGE:-getsentry/sentry:$SENTRY_VERSION}
Successfully tagged sentry-onpremise-local:latest
Building worker                 ... done
Building symbolicator-cleanup
 ---> 3edde8d4b2ac
Step 4/5 : COPY . /usr/src/sentry
 ---> Using cache
 ---> f0c27ea60b52
Step 5/5 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then pip install -r /usr/src/sentry/requirements.txt; fi
 ---> Using cache
 ---> 8d26f1b660de

Successfully built 8d26f1b660de
Step 1/5 : ARG BASE_IMAGE
Step 2/5 : FROM ${BASE_IMAGE}
 ---> 1370ed20ef93
Successfully tagged sentry-onpremise-local:latest
Building ingest-consumer        ... done
Step 3/5 : RUN apt-get update && apt-get install -y --no-install-recommends cron &&     rm -r /var/lib/apt/lists/*
 ---> Using cache
 ---> a8a3f6ce591b
Step 4/5 : COPY entrypoint.sh /entrypoint.sh
 ---> Using cache
 ---> 6d21933addf7
Step 5/5 : ENTRYPOINT ["/entrypoint.sh"]
 ---> Using cache
 ---> bcf336b7637b

Successfully built bcf336b7637b
Step 1/5 : ARG SENTRY_VERSION=latest
Step 2/5 : ARG SENTRY_IMAGE
Step 3/5 : FROM ${SENTRY_IMAGE:-getsentry/sentry:$SENTRY_VERSION}
Step 1/5 : ARG BASE_IMAGE
Step 2/5 : FROM ${BASE_IMAGE}
Step 1/5 : ARG BASE_IMAGE
Step 2/5 : FROM ${BASE_IMAGE}
Successfully tagged snuba-cleanup-onpremise-local:latest
 ---> 3edde8d4b2ac
Step 4/5 : COPY . /usr/src/sentry
Building snuba-cleanup          ... done
 ---> Using cache
 ---> f0c27ea60b52
Step 5/5 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then pip install -r /usr/src/sentry/requirements.txt; fi
 ---> Using cache
 ---> 8d26f1b660de

Successfully built 8d26f1b660de
 ---> 8d26f1b660de
Step 3/5 : RUN apt-get update && apt-get install -y --no-install-recommends cron &&     rm -r /var/lib/apt/lists/*
 ---> d6cf0be44380
Step 3/5 : RUN apt-get update && apt-get install -y --no-install-recommends cron &&     rm -r /var/lib/apt/lists/*
 ---> Using cache
 ---> 011b3c7d84d5
Step 4/5 : COPY entrypoint.sh /entrypoint.sh
 ---> Using cache
 ---> 11a211879d05
Step 5/5 : ENTRYPOINT ["/entrypoint.sh"]
Successfully tagged sentry-onpremise-local:latest
Building web                    ... done
 ---> Using cache
 ---> c0c735cedaf7
Step 4/5 : COPY entrypoint.sh /entrypoint.sh
 ---> Using cache
 ---> 4793ef51f86c
Step 5/5 : ENTRYPOINT ["/entrypoint.sh"]
 ---> Using cache
 ---> 68076a15d109

Successfully built 68076a15d109
 ---> Using cache
 ---> 16633d7826a4

Successfully built 16633d7826a4
Successfully tagged sentry-cleanup-onpremise-local:latest
Successfully tagged symbolicator-cleanup-onpremise-local:latest
Building sentry-cleanup         ... done
Building symbolicator-cleanup   ... done

Docker images built.
Creating network "sentry_onpremise_default" with the default driver
Bootstrapping and migrating Snuba...
Creating sentry_onpremise_zookeeper_1 ... 
Creating sentry_onpremise_clickhouse_1 ... 
Creating sentry_onpremise_redis_1      ... 
Creating sentry_onpremise_zookeeper_1  ... done
Creating sentry_onpremise_kafka_1      ... 
Creating sentry_onpremise_redis_1      ... done
Creating sentry_onpremise_clickhouse_1 ... done
Creating sentry_onpremise_kafka_1      ... done
+ '[' b = - ']'
+ snuba bootstrap --help
+ set -- snuba bootstrap --force
+ set gosu snuba snuba bootstrap --force
+ exec gosu snuba snuba bootstrap --force
2020-06-11 19:32:43,717 Connection to Kafka failed (attempt 0)
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 56, in bootstrap
    client.list_topics(timeout=1)
cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str="Failed to get metadata: Local: Broker transport failure"}
2020-06-11 19:32:44,948 Failed to create topic events
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 92, in bootstrap
    future.result()
  File "/usr/local/lib/python3.7/concurrent/futures/_base.py", line 435, in result
    return self.__get_result()
  File "/usr/local/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
cimpl.KafkaException: KafkaError{code=TOPIC_ALREADY_EXISTS,val=36,str="Topic 'events' already exists."}
2020-06-11 19:32:44,949 Failed to create topic errors-replacements
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 92, in bootstrap
    future.result()
  File "/usr/local/lib/python3.7/concurrent/futures/_base.py", line 428, in result
    return self.__get_result()
  File "/usr/local/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
cimpl.KafkaException: KafkaError{code=TOPIC_ALREADY_EXISTS,val=36,str="Topic 'errors-replacements' already exists."}
2020-06-11 19:32:44,950 Failed to create topic outcomes
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 92, in bootstrap
    future.result()
  File "/usr/local/lib/python3.7/concurrent/futures/_base.py", line 428, in result
    return self.__get_result()
  File "/usr/local/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
cimpl.KafkaException: KafkaError{code=TOPIC_ALREADY_EXISTS,val=36,str="Topic 'outcomes' already exists."}
2020-06-11 19:32:44,950 Failed to create topic cdc
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 92, in bootstrap
    future.result()
  File "/usr/local/lib/python3.7/concurrent/futures/_base.py", line 428, in result
    return self.__get_result()
  File "/usr/local/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
cimpl.KafkaException: KafkaError{code=TOPIC_ALREADY_EXISTS,val=36,str="Topic 'cdc' already exists."}
2020-06-11 19:32:44,950 Failed to create topic event-replacements
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 92, in bootstrap
    future.result()
  File "/usr/local/lib/python3.7/concurrent/futures/_base.py", line 428, in result
    return self.__get_result()
  File "/usr/local/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
cimpl.KafkaException: KafkaError{code=TOPIC_ALREADY_EXISTS,val=36,str="Topic 'event-replacements' already exists."}
2020-06-11 19:32:44,950 Failed to create topic snuba-commit-log
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 92, in bootstrap
    future.result()
  File "/usr/local/lib/python3.7/concurrent/futures/_base.py", line 428, in result
    return self.__get_result()
  File "/usr/local/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
cimpl.KafkaException: KafkaError{code=TOPIC_ALREADY_EXISTS,val=36,str="Topic 'snuba-commit-log' already exists."}
2020-06-11 19:32:44,950 Failed to create topic ingest-sessions
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 92, in bootstrap
    future.result()
  File "/usr/local/lib/python3.7/concurrent/futures/_base.py", line 428, in result
    return self.__get_result()
  File "/usr/local/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
cimpl.KafkaException: KafkaError{code=TOPIC_ALREADY_EXISTS,val=36,str="Topic 'ingest-sessions' already exists."}
2020-06-11 19:32:45,034 Creating tables for storage events
2020-06-11 19:32:45,037 Migrating storage events
2020-06-11 19:32:45,076 Creating tables for storage errors
2020-06-11 19:32:45,079 Migrating storage errors
2020-06-11 19:32:45,106 Creating tables for storage groupedmessages
2020-06-11 19:32:45,107 Migrating storage groupedmessages
2020-06-11 19:32:45,113 Creating tables for storage groupassignees
2020-06-11 19:32:45,114 Migrating storage groupassignees
2020-06-11 19:32:45,118 Creating tables for storage outcomes_raw
2020-06-11 19:32:45,119 Migrating storage outcomes_raw
2020-06-11 19:32:45,140 Creating tables for storage outcomes_hourly
2020-06-11 19:32:45,143 Migrating storage outcomes_hourly
2020-06-11 19:32:45,145 Creating tables for storage sessions_raw
2020-06-11 19:32:45,146 Migrating storage sessions_raw
2020-06-11 19:32:45,150 Creating tables for storage sessions_hourly
2020-06-11 19:32:45,157 Migrating storage sessions_hourly
2020-06-11 19:32:45,162 Creating tables for storage transactions
2020-06-11 19:32:45,172 Migrating storage transactions

sentry-postgres-new
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/9.6/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default timezone ... Etc/UTC
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

Success. You can now start the database server using:

    pg_ctl -D /var/lib/postgresql/9.6/data -l logfile start


WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.

*failure*
Consult the last few lines of "pg_upgrade_server.log" for
the probable cause of the failure.

There seems to be a postmaster servicing the old cluster.
Please shutdown that postmaster and try again.
Failure, exiting
Cleaning up...
3

Later on I also tried to restore version 9.1.2.

root@sentry:/home/cosmic/sentry/old# docker ps
CONTAINER ID        IMAGE                  COMMAND                  CREATED             STATUS                         PORTS                      NAMES
e3030d11c47a        nginx:latest           "nginx -g 'daemon of…"   2 minutes ago       Restarting (1) 6 seconds ago                              old_nginx_1
578466f2bee5        old_web                "/entrypoint.sh run …"   2 minutes ago       Up 2 minutes                   127.0.0.1:9000->9000/tcp   old_web_1
392a8daa143a        old_worker             "/entrypoint.sh run …"   2 minutes ago       Up 2 minutes                   9000/tcp                   old_worker_1
a5cd9211098f        old_cron               "/entrypoint.sh run …"   2 minutes ago       Up 2 minutes                   9000/tcp                   old_cron_1
d35461333b38        postgres:9.5           "docker-entrypoint.s…"   2 minutes ago       Up 2 minutes                   5432/tcp                   old_postgres_1
aa79f07080cd        redis:3.2-alpine       "docker-entrypoint.s…"   2 minutes ago       Up 2 minutes                   6379/tcp                   old_redis_1
007a79dfa224        tianon/exim4           "docker-entrypoint.s…"   2 minutes ago       Up 2 minutes                   25/tcp                     old_smtp_1
3d44a8a71371        memcached:1.5-alpine   "docker-entrypoint.s…"   2 minutes ago       Up 2 minutes                   11211/tcp                  old_memcached_1
root@sentry:/home/cosmic/sentry/old#

https://sentry.uhlhost.net site is back online. I just had to update the nginx upstream to old_web_1 from onpremise_web_1 .

For now at least our instance its up, would be great to find a solution to the above failes upgrades.