Unable to upgrade, stuck on version 9.1.2

Unable to upgrade from

to 10 / 20 .

root@sentry:/home/cosmic/sentry/onpremise# docker-compose run --rm web upgrade && docker-compose up -d
Starting sentry_onpremise_memcached_1 … done
Starting sentry_onpremise_zookeeper_1 … done
Starting sentry_onpremise_postgres_1 … done
Starting sentry_onpremise_clickhouse_1 … done
Starting sentry_onpremise_redis_1 … done
Starting sentry_onpremise_smtp_1 … done
Starting sentry_onpremise_symbolicator_1 … done
Starting sentry_onpremise_kafka_1 … done
Starting sentry_onpremise_snuba-sessions-consumer_1 … done
Starting sentry_onpremise_snuba-consumer_1 … done
Starting sentry_onpremise_snuba-replacer_1 … done
Starting sentry_onpremise_snuba-api_1 … done
Starting sentry_onpremise_snuba-outcomes-consumer_1 … done
22:01:38 [WARNING] sentry.utils.geo: settings.GEOIP_PATH_MMDB not configured.
/usr/local/lib/python2.7/site-packages/cryptography/init.py:39: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in a future release.
CryptographyDeprecationWarning,

22:02:06 [INFO] sentry.plugins.github: apps-not-configured
Traceback (most recent call last):
  File "/usr/local/bin/sentry", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/__init__.py", line 166, in main
    cli(prog_name=get_prog(), obj={}, max_content_width=100)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/decorators.py", line 30, in inner
    return ctx.invoke(f, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/commands/upgrade.py", line 174, in upgrade
    _upgrade(not noinput, traceback, verbosity, not no_repair, with_nodestore)
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/commands/upgrade.py", line 121, in _upgrade
    _migrate_from_south(verbosity)
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/commands/upgrade.py", line 93, in _migrate_from_south
    if not _has_south_history(connection):
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/commands/upgrade.py", line 78, in _has_south_history
    cursor = connection.cursor()
  File "/usr/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 254, in cursor
    return self._cursor()
  File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py", line 44, in inner
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/base.py", line 97, in _cursor
    return super(DatabaseWrapper, self)._cursor()
  File "/usr/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 229, in _cursor
    self.ensure_connection()
  File "/usr/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 213, in ensure_connection
    self.connect()
  File "/usr/local/lib/python2.7/site-packages/django/db/utils.py", line 94, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/usr/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 213, in ensure_connection
    self.connect()
  File "/usr/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 189, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/usr/local/lib/python2.7/site-packages/django/db/backends/postgresql/base.py", line 176, in get_new_connection
    connection = Database.connect(**conn_params)
  File "/usr/local/lib/python2.7/site-packages/psycopg2/__init__.py", line 127, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
	Is the server running on host "postgres" (172.20.0.17) and accepting
	TCP/IP connections on port 5432?

root@sentry:/home/cosmic/sentry/onpremise# 

I have tried all what was written online, here on forums, nothing helped. And a few statements about beying on version 9.1.2 to be able to use the script to automatically upgrade are bening, since this procedure clearly fails. If I would be the only one with this issue, there wont be so many posts about the upgrade from 9 to 10.

I am confused as to what is an upgrade procedure, I am thinking of testing with clean machine deployent fresh install and migrate db? Is there a better procedure to fix this?

I have runned also:

root@sentry:/home/cosmic/sentry/onpremise# docker-compose build --pull && docker-compose run --rm web upgrade && docker-compose up -d

zookeeper uses an image, skipping

kafka uses an image, skipping

postgres uses an image, skipping

symbolicator uses an image, skipping

smtp uses an image, skipping

memcached uses an image, skipping

redis uses an image, skipping

relay uses an image, skipping

clickhouse uses an image, skipping

snuba-api uses an image, skipping

snuba-replacer uses an image, skipping

snuba-outcomes-consumer uses an image, skipping

snuba-consumer uses an image, skipping

snuba-sessions-consumer uses an image, skipping

nginx uses an image, skipping

Building snuba-cleanup

Step 1/5 : ARG BASE_IMAGE

Step 2/5 : FROM ${BASE_IMAGE}

latest: Pulling from getsentry/snuba

Digest: sha256:e1dc2c55ab940b4e9a5d733164885b305e994d112028460a711f0470b35e0668

Status: Image is up to date for getsentry/snuba:latest

---> 4e51e4e88582

Step 3/5 : RUN apt-get update && apt-get install -y --no-install-recommends cron && rm -r /var/lib/apt/lists/*

---> Using cache

---> 9df8a5845c23

Step 4/5 : COPY entrypoint.sh /entrypoint.sh

---> Using cache

---> 1831a97bd459

Step 5/5 : ENTRYPOINT ["/entrypoint.sh"]

---> Using cache

---> 19da727a0295

Successfully built 19da727a0295

Successfully tagged snuba-cleanup-onpremise-local:latest

Building worker

Step 1/5 : ARG SENTRY_VERSION=latest

Step 2/5 : ARG SENTRY_IMAGE

Step 3/5 : FROM ${SENTRY_IMAGE:-getsentry/sentry:$SENTRY_VERSION}

latest: Pulling from getsentry/sentry

Digest: sha256:27d0771025f721702a57f45631cf4ba35d2502b6d8d957fe03b788af890172d6

Status: Image is up to date for getsentry/sentry:latest

---> 01780055e2bf

Step 4/5 : COPY . /usr/src/sentry

---> Using cache

---> 4eca4bab4894

Step 5/5 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then pip install -r /usr/src/sentry/requirements.txt; fi

---> Using cache

---> 78a78f51f233

Successfully built 78a78f51f233

Successfully tagged sentry-onpremise-local:latest

Building sentry-cleanup

Step 1/5 : ARG BASE_IMAGE

Step 2/5 : FROM ${BASE_IMAGE}

ERROR: Service 'sentry-cleanup' failed to build: pull access denied for sentry-onpremise-local, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

That gave also errors, so at every step are issues, I got it working last on 9.1.2 after a restart I cannot seem to get it working, telling me nginx is in use, I also opened this github issue to help track such issues and maybe find a fix so others can avoid it.

This latest log saved from running install.sh attempts from version 9.1.2. according to the documentation.

root@sentry:/home/cosmic/sentry/onpremise# cat sentry_install_log-2020-06-11_19-08-07.txt 
Checking minimum requirements...
Removing network onpremise_default
Network onpremise_default not found.
Stopping sentry_onpremise_snuba-sessions-consumer_1 ... 
Stopping sentry_onpremise_snuba-consumer_1          ... 
Stopping sentry_onpremise_snuba-outcomes-consumer_1 ... 
Stopping sentry_onpremise_snuba-api_1               ... 
Stopping sentry_onpremise_snuba-replacer_1          ... 
Stopping sentry_onpremise_smtp_1                    ... 
Stopping sentry_onpremise_postgres_1                ... 
Stopping sentry_onpremise_symbolicator_1            ... 
Stopping sentry_onpremise_memcached_1               ... 
Stopping sentry_onpremise_kafka_1                   ... 
Stopping sentry_onpremise_clickhouse_1              ... 
Stopping sentry_onpremise_zookeeper_1               ... 
Stopping sentry_onpremise_redis_1                   ... 
Stopping sentry_onpremise_postgres_1                ... done
Stopping sentry_onpremise_memcached_1               ... done
Stopping sentry_onpremise_smtp_1                    ... done
Stopping sentry_onpremise_symbolicator_1            ... done
Stopping sentry_onpremise_snuba-api_1               ... done
Stopping sentry_onpremise_snuba-sessions-consumer_1 ... done
Stopping sentry_onpremise_snuba-replacer_1          ... done
Stopping sentry_onpremise_snuba-outcomes-consumer_1 ... done
Stopping sentry_onpremise_snuba-consumer_1          ... done
Stopping sentry_onpremise_redis_1                   ... done
Stopping sentry_onpremise_clickhouse_1              ... done
Stopping sentry_onpremise_kafka_1                   ... done
Stopping sentry_onpremise_zookeeper_1               ... done
Removing sentry_onpremise_snuba-sessions-consumer_1 ... 
Removing sentry_onpremise_snuba-consumer_1          ... 
Removing sentry_onpremise_snuba-outcomes-consumer_1 ... 
Removing sentry_onpremise_snuba-api_1               ... 
Removing sentry_onpremise_snuba-replacer_1          ... 
Removing sentry_onpremise_smtp_1                    ... 
Removing sentry_onpremise_postgres_1                ... 
Removing sentry_onpremise_symbolicator_1            ... 
Removing sentry_onpremise_memcached_1               ... 
Removing sentry_onpremise_kafka_1                   ... 
Removing sentry_onpremise_clickhouse_1              ... 
Removing sentry_onpremise_zookeeper_1               ... 
Removing sentry_onpremise_redis_1                   ... 
Removing sentry_onpremise_snuba-sessions-consumer_1 ... done
Removing sentry_onpremise_snuba-api_1               ... done
Removing sentry_onpremise_snuba-replacer_1          ... done
Removing sentry_onpremise_snuba-consumer_1          ... done
Removing sentry_onpremise_snuba-outcomes-consumer_1 ... done
Removing sentry_onpremise_memcached_1               ... done
Removing sentry_onpremise_smtp_1                    ... done
Removing sentry_onpremise_symbolicator_1            ... done
Removing sentry_onpremise_postgres_1                ... done
Removing sentry_onpremise_redis_1                   ... done
Removing sentry_onpremise_clickhouse_1              ... done
Removing sentry_onpremise_zookeeper_1               ... done
Removing sentry_onpremise_kafka_1                   ... done
Removing network sentry_onpremise_default

Creating volumes for persistent storage...
Created sentry-data.
Created sentry-postgres.
Created sentry-redis.
Created sentry-zookeeper.
Created sentry-kafka.
Created sentry-clickhouse.
Created sentry-symbolicator.

sentry/sentry.conf.py already exists, skipped creation.
sentry/config.yml already exists, skipped creation.
sentry/requirements.txt already exists, skipped creation.

Fetching and updating Docker images...

Some service image(s) must be built from source by running:
docker-compose build symbolicator-cleanup snuba-cleanup sentry-cleanup worker ingest-consumer web cron post-process-forwarder
latest: Pulling from getsentry/sentry
Digest: sha256:57323e24d454aea64ca10ecf222b6c92022f0f1637caecafba3e08ffd83756be
Status: Image is up to date for getsentry/sentry:latest
docker.io/getsentry/sentry:latest

Building and tagging Docker images...

Building web
Step 1/5 : ARG SENTRY_VERSION=latest
Step 2/5 : ARG SENTRY_IMAGE
Step 3/5 : FROM ${SENTRY_IMAGE:-getsentry/sentry:$SENTRY_VERSION}
 ---> 3edde8d4b2ac
Step 4/5 : COPY . /usr/src/sentry
 ---> Using cache
 ---> f0c27ea60b52
Step 5/5 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then pip install -r /usr/src/sentry/requirements.txt; fi
 ---> Using cache
 ---> 8d26f1b660de

Successfully built 8d26f1b660de
Successfully tagged sentry-onpremise-local:latest
memcached uses an image, skipping
postgres uses an image, skipping
redis uses an image, skipping
symbolicator uses an image, skipping
smtp uses an image, skipping
zookeeper uses an image, skipping
kafka uses an image, skipping
relay uses an image, skipping
clickhouse uses an image, skipping
snuba-api uses an image, skipping
snuba-outcomes-consumer uses an image, skipping
snuba-consumer uses an image, skipping
snuba-replacer uses an image, skipping
snuba-sessions-consumer uses an image, skipping
nginx uses an image, skipping
Building symbolicator-cleanup   ... 
Building ingest-consumer        ... 
Building post-process-forwarder ... 
Building cron                   ... 
Building web                    ... 
Building worker                 ... 
Building snuba-cleanup          ... 
Building sentry-cleanup         ... 
Building symbolicator-cleanup
Building cron
Building ingest-consumer
Building worker
Building sentry-cleanup
Step 1/5 : ARG BASE_IMAGE
Step 2/5 : FROM ${BASE_IMAGE}
 ---> d6cf0be44380
Step 3/5 : RUN apt-get update && apt-get install -y --no-install-recommends cron &&     rm -r /var/lib/apt/lists/*
 ---> Using cache
 ---> 011b3c7d84d5
Step 4/5 : COPY entrypoint.sh /entrypoint.sh
 ---> Using cache
 ---> 11a211879d05
Step 5/5 : ENTRYPOINT ["/entrypoint.sh"]
 ---> Using cache
 ---> 16633d7826a4

Successfully built 16633d7826a4
Step 1/5 : ARG BASE_IMAGE
Step 2/5 : FROM ${BASE_IMAGE}
Step 1/5 : ARG SENTRY_VERSION=latest
Step 2/5 : ARG SENTRY_IMAGE
Step 3/5 : FROM ${SENTRY_IMAGE:-getsentry/sentry:$SENTRY_VERSION}
Step 1/5 : ARG SENTRY_VERSION=latestSuccessfully tagged symbolicator-cleanup-onpremise-local:latest

 ---> 8d26f1b660de
Building symbolicator-cleanup   ... done
Building snuba-cleanup
 ---> 3edde8d4b2ac
Step 4/5 : COPY . /usr/src/sentry
 ---> Using cache
 ---> f0c27ea60b52
Step 5/5 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then pip install -r /usr/src/sentry/requirements.txt; fi
 ---> Using cache
 ---> 8d26f1b660de

Successfully built 8d26f1b660de
Step 2/5 : ARG SENTRY_IMAGE
Step 3/5 : FROM ${SENTRY_IMAGE:-getsentry/sentry:$SENTRY_VERSION}
Step 3/5 : RUN apt-get update && apt-get install -y --no-install-recommends cron &&     rm -r /var/lib/apt/lists/*
Step 1/5 : ARG SENTRY_VERSION=latest
Step 2/5 : ARG SENTRY_IMAGE
Step 3/5 : FROM ${SENTRY_IMAGE:-getsentry/sentry:$SENTRY_VERSION}
 ---> 3edde8d4b2ac
Step 4/5 : COPY . /usr/src/sentry
 ---> Using cache
 ---> f0c27ea60b52
Step 5/5 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then pip install -r /usr/src/sentry/requirements.txt; fi
 ---> Using cache
 ---> 8d26f1b660de

Successfully built 8d26f1b660de
Successfully tagged sentry-onpremise-local:latest
Building ingest-consumer        ... done
Building post-process-forwarder
 ---> 3edde8d4b2ac
Step 4/5 : COPY . /usr/src/sentry
 ---> Using cache
 ---> f0c27ea60b52
Step 5/5 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then pip install -r /usr/src/sentry/requirements.txt; fi
 ---> Using cache
 ---> 8d26f1b660de
 ---> Using cache
 ---> c0c735cedaf7
Step 4/5 : COPY entrypoint.sh /entrypoint.sh

Successfully built 8d26f1b660de
Step 1/5 : ARG BASE_IMAGE
Step 2/5 : FROM ${BASE_IMAGE}
Step 1/5 : ARG SENTRY_VERSION=latest
Step 2/5 : ARG SENTRY_IMAGE
Step 3/5 : FROM ${SENTRY_IMAGE:-getsentry/sentry:$SENTRY_VERSION}
Successfully tagged sentry-onpremise-local:latest
Building worker                 ... done
Building web
 ---> Using cache
 ---> 4793ef51f86c
Step 5/5 : ENTRYPOINT ["/entrypoint.sh"]
 ---> Using cache
 ---> 68076a15d109

Successfully built 68076a15d109
 ---> 1370ed20ef93
Step 3/5 : RUN apt-get update && apt-get install -y --no-install-recommends cron &&     rm -r /var/lib/apt/lists/*
Step 1/5 : ARG SENTRY_VERSION=latest
Step 2/5 : ARG SENTRY_IMAGE
Step 3/5 : FROM ${SENTRY_IMAGE:-getsentry/sentry:$SENTRY_VERSION}
Successfully tagged sentry-cleanup-onpremise-local:latest
 ---> 3edde8d4b2ac
Building sentry-cleanup         ... done
Successfully tagged sentry-onpremise-local:latest
Building cron                   ... done
Step 4/5 : COPY . /usr/src/sentry
 ---> Using cache
 ---> f0c27ea60b52
Step 5/5 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then pip install -r /usr/src/sentry/requirements.txt; fi
 ---> Using cache
 ---> 8d26f1b660de

Successfully built 8d26f1b660de
 ---> 3edde8d4b2ac
Step 4/5 : COPY . /usr/src/sentry
Successfully tagged sentry-onpremise-local:latest
 ---> Using cache
 ---> a8a3f6ce591b
Step 4/5 : COPY entrypoint.sh /entrypoint.sh
 ---> Using cache
 ---> 6d21933addf7
Step 5/5 : ENTRYPOINT ["/entrypoint.sh"]
 ---> Using cache
 ---> bcf336b7637b

Successfully built bcf336b7637b
Building web                    ... done
 ---> Using cache
Successfully tagged snuba-cleanup-onpremise-local:latest
Building snuba-cleanup          ... done
 ---> f0c27ea60b52
Step 5/5 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then pip install -r /usr/src/sentry/requirements.txt; fi
 ---> Using cache
 ---> 8d26f1b660de

Successfully built 8d26f1b660de
Successfully tagged sentry-onpremise-local:latest
Building post-process-forwarder ... done

Docker images built.
Creating network "sentry_onpremise_default" with the default driver
Bootstrapping and migrating Snuba...
Creating sentry_onpremise_redis_1 ... 
Creating sentry_onpremise_clickhouse_1 ... 
Creating sentry_onpremise_zookeeper_1  ... 
Creating sentry_onpremise_clickhouse_1 ... done
Creating sentry_onpremise_redis_1      ... done
Creating sentry_onpremise_zookeeper_1  ... done
Creating sentry_onpremise_kafka_1      ... 
Creating sentry_onpremise_kafka_1      ... done
+ '[' b = - ']'
+ snuba bootstrap --help
+ set -- snuba bootstrap --force
+ set gosu snuba snuba bootstrap --force
+ exec gosu snuba snuba bootstrap --force
2020-06-11 19:09:08,856 Connection to Kafka failed (attempt 0)
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 56, in bootstrap
client.list_topics(timeout=1)
cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str="Failed to get metadata: Local: Broker transport failure"}
2020-06-11 19:09:10,860 Connection to Kafka failed (attempt 1)
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 56, in bootstrap
client.list_topics(timeout=1)
cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str="Failed to get metadata: Local: Broker transport failure"}
2020-06-11 19:09:11,985 Failed to create topic events
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 92, in bootstrap
future.result()
  File "/usr/local/lib/python3.7/concurrent/futures/_base.py", line 435, in result
return self.__get_result()
  File "/usr/local/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
raise self._exception
cimpl.KafkaException: KafkaError{code=TOPIC_ALREADY_EXISTS,val=36,str="Topic 'events' already exists."}
2020-06-11 19:09:11,986 Failed to create topic outcomes
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 92, in bootstrap
future.result()
  File "/usr/local/lib/python3.7/concurrent/futures/_base.py", line 428, in result
return self.__get_result()
  File "/usr/local/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
raise self._exception
cimpl.KafkaException: KafkaError{code=TOPIC_ALREADY_EXISTS,val=36,str="Topic 'outcomes' already exists."}
2020-06-11 19:09:11,986 Failed to create topic cdc
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 92, in bootstrap
future.result()
  File "/usr/local/lib/python3.7/concurrent/futures/_base.py", line 428, in result
return self.__get_result()
  File "/usr/local/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
raise self._exception
cimpl.KafkaException: KafkaError{code=TOPIC_ALREADY_EXISTS,val=36,str="Topic 'cdc' already exists."}
2020-06-11 19:09:11,986 Failed to create topic errors-replacements
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 92, in bootstrap
future.result()
  File "/usr/local/lib/python3.7/concurrent/futures/_base.py", line 428, in result
return self.__get_result()
  File "/usr/local/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
raise self._exception
cimpl.KafkaException: KafkaError{code=TOPIC_ALREADY_EXISTS,val=36,str="Topic 'errors-replacements' already exists."}
2020-06-11 19:09:11,986 Failed to create topic ingest-sessions
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 92, in bootstrap
future.result()
  File "/usr/local/lib/python3.7/concurrent/futures/_base.py", line 428, in result
return self.__get_result()
  File "/usr/local/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
raise self._exception
cimpl.KafkaException: KafkaError{code=TOPIC_ALREADY_EXISTS,val=36,str="Topic 'ingest-sessions' already exists."}
2020-06-11 19:09:11,987 Failed to create topic event-replacements
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 92, in bootstrap
future.result()
  File "/usr/local/lib/python3.7/concurrent/futures/_base.py", line 428, in result
return self.__get_result()
  File "/usr/local/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
raise self._exception
cimpl.KafkaException: KafkaError{code=TOPIC_ALREADY_EXISTS,val=36,str="Topic 'event-replacements' already exists."}
2020-06-11 19:09:11,987 Failed to create topic snuba-commit-log
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 92, in bootstrap
future.result()
  File "/usr/local/lib/python3.7/concurrent/futures/_base.py", line 428, in result
return self.__get_result()
  File "/usr/local/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
raise self._exception
cimpl.KafkaException: KafkaError{code=TOPIC_ALREADY_EXISTS,val=36,str="Topic 'snuba-commit-log' already exists."}
2020-06-11 19:09:12,028 Creating tables for storage events
2020-06-11 19:09:12,032 Migrating storage events
2020-06-11 19:09:12,066 Creating tables for storage errors
2020-06-11 19:09:12,069 Migrating storage errors
2020-06-11 19:09:12,093 Creating tables for storage groupedmessages
2020-06-11 19:09:12,095 Migrating storage groupedmessages
2020-06-11 19:09:12,100 Creating tables for storage groupassignees
2020-06-11 19:09:12,109 Migrating storage groupassignees
2020-06-11 19:09:12,114 Creating tables for storage outcomes_raw
2020-06-11 19:09:12,117 Migrating storage outcomes_raw
2020-06-11 19:09:12,124 Creating tables for storage outcomes_hourly
2020-06-11 19:09:12,127 Migrating storage outcomes_hourly
2020-06-11 19:09:12,130 Creating tables for storage sessions_raw
2020-06-11 19:09:12,139 Migrating storage sessions_raw
2020-06-11 19:09:12,145 Creating tables for storage sessions_hourly
2020-06-11 19:09:12,190 Migrating storage sessions_hourly
2020-06-11 19:09:12,197 Creating tables for storage transactions
2020-06-11 19:09:12,199 Migrating storage transactions

sentry-postgres-new
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/9.6/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default timezone ... Etc/UTC
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

Success. You can now start the database server using:

pg_ctl -D /var/lib/postgresql/9.6/data -l logfile start


WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.

*failure*
Consult the last few lines of "pg_upgrade_server.log" for
the probable cause of the failure.

There seems to be a postmaster servicing the old cluster.
Please shutdown that postmaster and try again.
Failure, exiting
Cleaning up...
root@sentry:/home/cosmic/sentry/onpremise#

There seems to be a postmaster servicing the old cluster.
Please shutdown that postmaster and try again.

Seems like you tried to upgrade Sentry without first shutting down Postgres properly. I’d try the following:

docker-compose down --remove-orphans
docker-compose run postgres bash -c "rm -f /var/lib/postgresql/data/postmaster.pid"
./install.sh

And see what happens.

Edit: fixed docker-compose stop, changed to docker-compose down as suggested below.

sudo docker-compose stop --remove-orphans
Stop running containers without removing them.

They can be started again with `docker-compose start`.

Usage: stop [options] [SERVICE...]

Options:
  -t, --timeout TIMEOUT      Specify a shutdown timeout in seconds.
                             (default: 10)
cosmic@sentry:~/sentry/onpremise$ 

This is what i get running the mentioned command.

Sorry, it should have been docker-compose down --remove-orphans not stop.

No problems.

Here is the output.

**cosmic@sentry** : **~/sentry/onpremise** $ sudo docker-compose down --remove-orphans
Removing sentry_onpremise_kafka_1 ... done
Removing sentry_onpremise_redis_1 ... done
Removing sentry_onpremise_clickhouse_1 ... done
Removing sentry_onpremise_zookeeper_1 ... done
Removing network sentry_onpremise_default
**cosmic@sentry** : **~/sentry/onpremise** $ docker-compose run postgres bash -c "rm -f /var/lib/postgresql/data/postmaster.pid
> ./install.sh
> ^C
**cosmic@sentry** : **~/sentry/onpremise** $ sudo docker-compose run postgres bash -c "rm -f /var/lib/postgresql/data/postmaster.pid" ./install.sh
Creating network "sentry_onpremise_default" with the default driver
**cosmic@sentry** : **~/sentry/onpremise** $

What else I need to do now? Rerun install?

Here is what is currently running under docker.

cosmic@sentry:~/sentry/onpremise$ sudo docker ps
CONTAINER ID        IMAGE                  COMMAND                  CREATED             STATUS              PORTS                                                                                                      NAMES
e3030d11c47a        nginx:latest           "nginx -g 'daemon of…"   2 months ago        Up 45 hours         0.0.0.0:25->25/tcp, 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp, 0.0.0.0:465->465/tcp, 0.0.0.0:587->587/tcp   old_nginx_1
578466f2bee5        old_web                "/entrypoint.sh run …"   2 months ago        Up 45 hours         127.0.0.1:9000->9000/tcp                                                                                   old_web_1
392a8daa143a        old_worker             "/entrypoint.sh run …"   2 months ago        Up 45 hours         9000/tcp                                                                                                   old_worker_1
a5cd9211098f        old_cron               "/entrypoint.sh run …"   2 months ago        Up 45 hours         9000/tcp                                                                                                   old_cron_1
d35461333b38        postgres:9.5           "docker-entrypoint.s…"   2 months ago        Up 4 minutes        5432/tcp                                                                                                   old_postgres_1
aa79f07080cd        redis:3.2-alpine       "docker-entrypoint.s…"   2 months ago        Up 45 hours         6379/tcp                                                                                                   old_redis_1
007a79dfa224        f077f7830685           "docker-entrypoint.s…"   2 months ago        Up 45 hours         25/tcp                                                                                                     old_smtp_1
3d44a8a71371        memcached:1.5-alpine   "docker-entrypoint.s…"   2 months ago        Up 45 hours         11211/tcp                                                                                                  old_memcached_1

Yeah the docker-compose run postgres bash -c "rm -f /var/lib/postgresql/data/postmaster.pid" command was missing the closing quote and you moved the final ./install.sh command I put there into the docker-compose run call which was not correct.

I also should have said docker-compose run --rm postgres bash -c "rm -f /var/lib/postgresql/data/postmaster.pid" (note the --rm there).

Anyway, I’d just run the install script again and see if goes through. That said I don’t understand how you ended up with the old_xxx instances there. I think you need to docker stop <container_id> them one by one before moving forward as docker-compose down seems to be missing them.

I have stopped all containers manually, runned sudo ./install.sh

And then I got this:

Upgrade Complete
----------------
Optimizer statistics are not transferred by pg_upgrade so,
once you start the new server, consider running:
    ./analyze_new_cluster.sh

Running this script will delete the old cluster's data files:
    ./delete_old_cluster.sh
Error response from daemon: remove sentry-postgres: volume is in use - [d35461333b3886ddb79ff16ee4d7e6ecbcefbc095a71a5a9545e8da4f25167c0]
Cleaning up...
cosmic@sentry:~/sentry/onpremise$ sudo docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
cosmic@sentry:~/sentry/onpremise$

Note this was linked to Github Issue :warning:

Hopefully this last steps do not destroy all my data without leaving me options to recover.

I have personally tried this and added the missing " it seems this allowed me to continue to setup to the above point.

Also I have checked most of the forum related issues about upgrade procedures and solutions including; Upgrade Guide for v9 to v10 or v20.6.0 it does not seem to be any related to the issue I meet now.

As long as your volumes stay, you should be able to go back to 9.1.2.

I think what is left is running docker ps -a and removing the stopped postgres container to release the volume (via docker rm <container_id>).

@BYK

I got to this poin:

Audit Log Entrys: 100% |#########################################| Time: 0:00:00
**OK**
Applying sentry.0091_alertruleactivity... **OK**
Applying sentry.0092_remove_trigger_threshold_type_nullable... **OK**
Applying sentry.0093_make_identity_user_id_textfield... **OK**
Applying sentry.0094_cleanup_unreferenced_event_files... **OK**
Applying sentry.0095_ruleactivity... **OK**
Applying sentry.0096_sentry_app_component_skip_load_on_open... **OK**
Applying sentry.0097_add_sentry_app_id_to_sentry_alertruletriggeraction... **OK**
Applying sentry.0098_add-performance-onboarding... **OK**
Applying sites.0002_alter_domain_unique... **OK**
Creating missing DSNs
Correcting Group.num_comments counter 

It seems to take a while, for now looks like stuck yet I cannot say will leave it surely will timeout in case of errors.

1 Like

@BYK
After all it seems it looks all is good, yet my website is not resolving nor getting any errors displayed.

https://sentry.uhlhost.net

18:21:19 [WARNING] sentry.eventstream.kafka.backend: Could not publish message (error: KafkaError{code=_MSG_TIMED_OUT,val=-192,str="Local: Message timed out"}): <cimpl.Message object at 0x7f27ca978dd0>
18:21:19 [WARNING] sentry.eventstream.kafka.backend: Could not publish message (error: KafkaError{code=_MSG_TIMED_OUT,val=-192,str="Local: Message timed out"}): <cimpl.Message object at 0x7f27ca978dd0>

Generating Relay credentials...
Relay credentials written to relay/credentials.json
Relay public key written to sentry/sentry.conf.py

Cleaning up...

----------------
You're all done! Run the following command to get Sentry running:

  docker-compose up -d

cosmic@sentry:~/sentry/onpremise$ sudo docker-compose up -d
Starting sentry_onpremise_redis_1                ... done
Starting sentry_onpremise_memcached_1            ... done
Starting sentry_onpremise_clickhouse_1           ... done
Starting sentry_onpremise_symbolicator_1         ... done
Starting sentry_onpremise_smtp_1                 ... done
Starting sentry_onpremise_postgres_1                ... done
Starting sentry_onpremise_zookeeper_1            ... done
Creating sentry_onpremise_symbolicator-cleanup_1 ... done
Starting sentry_onpremise_kafka_1                ... done
Starting sentry_onpremise_snuba-consumer_1          ... done
Starting sentry_onpremise_snuba-api_1               ... done
Starting sentry_onpremise_snuba-outcomes-consumer_1 ... done
Starting sentry_onpremise_snuba-replacer_1          ... done
Starting sentry_onpremise_snuba-sessions-consumer_1 ... done
Creating sentry_onpremise_snuba-cleanup_1           ... done
Creating sentry_onpremise_relay_1                   ... done
Creating sentry_onpremise_sentry-cleanup_1          ... done
Creating sentry_onpremise_post-process-forwarder_1  ... done
Creating sentry_onpremise_web_1                     ... done
Creating sentry_onpremise_cron_1                    ... done
Creating sentry_onpremise_ingest-consumer_1         ... done
Creating sentry_onpremise_worker_1                  ... done
Creating sentry_onpremise_nginx_1                   ... done
cosmic@sentry:~/sentry/onpremise$

That depends on how your site is configured. You may want to share some logs from your installation.

Here is the full log of latest install.sh

Install logs won’t be useful at this point. I mean logs from services using docker-compose logs. Also, seems like your instance is available over at http://sentry.uhlhost.net:9000/auth/login/uhlhosting/

1 Like

@BYK

https://sentry.uhlhost.net should be resolving while the one resolving now does not work over https
https://gist.github.com/uhlhosting/600e5adbccd969107a027aa1784d41b9

These are all running docker ps:

cosmic@sentry:~/sentry/onpremise/sentry$ sudo docker ps
CONTAINER ID        IMAGE                                  COMMAND                  CREATED             STATUS              PORTS                          NAMES
034252ebd65e        nginx:1.16                             "nginx -g 'daemon of…"   2 hours ago         Up 2 hours          0.0.0.0:9000->80/tcp           sentry_onpremise_nginx_1
60d95ea6ea68        sentry-onpremise-local                 "/bin/sh -c 'exec /d…"   2 hours ago         Up 2 hours          9000/tcp                       sentry_onpremise_worker_1
92f9444785d4        sentry-onpremise-local                 "/bin/sh -c 'exec /d…"   2 hours ago         Up 2 hours          9000/tcp                       sentry_onpremise_ingest-consumer_1
9e409be59e06        sentry-onpremise-local                 "/bin/sh -c 'exec /d…"   2 hours ago         Up 2 hours          9000/tcp                       sentry_onpremise_web_1
2e801596fe47        sentry-onpremise-local                 "/bin/sh -c 'exec /d…"   2 hours ago         Up 2 hours          9000/tcp                       sentry_onpremise_cron_1
e7fc2325d146        sentry-onpremise-local                 "/bin/sh -c 'exec /d…"   2 hours ago         Up 2 hours          9000/tcp                       sentry_onpremise_post-process-forwarder_1
fa141fb3ba49        sentry-cleanup-onpremise-local         "/entrypoint.sh '0 0…"   2 hours ago         Up 2 hours          9000/tcp                       sentry_onpremise_sentry-cleanup_1
666f2e26ceaf        getsentry/relay:latest                 "/bin/bash /docker-e…"   2 hours ago         Up 2 hours          3000/tcp                       sentry_onpremise_relay_1
f800be35df81        snuba-cleanup-onpremise-local          "/entrypoint.sh '*/5…"   2 hours ago         Up 2 hours          1218/tcp                       sentry_onpremise_snuba-cleanup_1
520f697eb67b        symbolicator-cleanup-onpremise-local   "/entrypoint.sh '55 …"   2 hours ago         Up 2 hours          3021/tcp                       sentry_onpremise_symbolicator-cleanup_1
243a4061acd1        getsentry/snuba:latest                 "./docker_entrypoint…"   2 hours ago         Up 2 hours          1218/tcp                       sentry_onpremise_snuba-sessions-consumer_1
6d56386e8b2e        getsentry/snuba:latest                 "./docker_entrypoint…"   2 hours ago         Up 2 hours          1218/tcp                       sentry_onpremise_snuba-outcomes-consumer_1
f38031910ffd        getsentry/snuba:latest                 "./docker_entrypoint…"   2 hours ago         Up 2 hours          1218/tcp                       sentry_onpremise_snuba-consumer_1
bf4b93fbf8ff        getsentry/snuba:latest                 "./docker_entrypoint…"   2 hours ago         Up 2 hours          1218/tcp                       sentry_onpremise_snuba-replacer_1
37e03169f194        getsentry/snuba:latest                 "./docker_entrypoint…"   2 hours ago         Up 2 hours          1218/tcp                       sentry_onpremise_snuba-api_1
0f4942481208        tianon/exim4                           "docker-entrypoint.s…"   2 hours ago         Up 2 hours          25/tcp                         sentry_onpremise_smtp_1
b77be9d95032        memcached:1.5-alpine                   "docker-entrypoint.s…"   2 hours ago         Up 2 hours          11211/tcp                      sentry_onpremise_memcached_1
c4b821549722        getsentry/symbolicator:latest          "/bin/bash /docker-e…"   2 hours ago         Up 2 hours          3021/tcp                       sentry_onpremise_symbolicator_1
84d14c79bce5        postgres:9.6                           "docker-entrypoint.s…"   2 hours ago         Up 2 hours          5432/tcp                       sentry_onpremise_postgres_1
03c56a1fe2b5        confluentinc/cp-kafka:5.5.0            "/etc/confluent/dock…"   2 hours ago         Up 2 hours          9092/tcp                       sentry_onpremise_kafka_1
40823919be8d        yandex/clickhouse-server:19.17         "/entrypoint.sh"         2 hours ago         Up 2 hours          8123/tcp, 9000/tcp, 9009/tcp   sentry_onpremise_clickhouse_1
66f6faf54e23        confluentinc/cp-zookeeper:5.5.0        "/etc/confluent/dock…"   2 hours ago         Up 2 hours          2181/tcp, 2888/tcp, 3888/tcp   sentry_onpremise_zookeeper_1
afbcc84e5238        redis:5.0-alpine                       "docker-entrypoint.s…"   2 hours ago         Up 2 hours          6379/tcp                       sentry_onpremise_redis_1

Yes, you need to terminate SSL traffic at your load balancer and redirect it to localhost:9000 for https. You also need to change a few settings in your config and set your system.url-prefix accordingly.