Cleanly applying Sentry database upgrades in a Fargate environment

1

Hi,

For several years we were running our self-hosted Sentry environment on an EC2 host. We’ve been using (hosted) AWS RDS for the Sentry Postgres database.

For neatness, more recently we switched this environment from EC2 to Fargate, while continuing to use the same Postgres database. This mostly works, but we find that we’re unable add new users, so we realized that we must run the Sentry database migrations to migrate from Sentry 9.11 to the latest-and-greatest.

Unfortunately, in Fargate we don’t have the means to ssh to the Sentry software. So instead on a bastion host which can talk to the hosted Postgres, we’re running a docker-compose stack. I’ll try to fill-in the details in the thread below, but the “sentry upgrade” command fails when trying to create a postgres Table which already exists:

FATAL ERROR - The following SQL query failed: CREATE TABLE “sentry_identityprovider” (“id” bigserial NOT NULL PRIMARY KEY, “type” varchar(64) NOT NULL, “instance” varchar(64) NOT NULL)
The error was: ProgrammingError(‘relation “sentry_identityprovider” already exists\n’,)
SQL: CREATE TABLE “sentry_identityprovider” (“id” bigserial NOT NULL PRIMARY KEY, “type” varchar(64) NOT NULL, “instance” varchar(64) NOT NULL)
Error in migration:

2

Here’s the docker-compose stack:

$ cat docker-compose.yml | sed “s/SENTRY_SECRET_KEY./SENTRY_SECRET_KEY: REDACTED1/;s/SENTRY_DB_PASSWORD./SENTRY_DB_PASSWORD: REDACTED2/;s/image:.sentry./image: REDACTED_SENTRY_IMAGE_NAME/”
version: ‘2’

volumes:
pgdb:

services:
redis:
image: redis

sentry:
image: REDACTED_SENTRY_IMAGE_NAME
links:
- redis
ports:
- 9000:9000
environment:
SENTRY_SECRET_KEY: REDACTED1
SENTRY_POSTGRES_HOST: MANUALLYREDACTED
SENTRY_DB_USER: sentry
SENTRY_DB_PASSWORD: REDACTED2
SENTRY_REDIS_HOST: redis

cron:
image: REDACTED_SENTRY_IMAGE_NAME
links:
- redis
command: “sentry run cron”
environment:
SENTRY_SECRET_KEY: REDACTED1
SENTRY_POSTGRES_HOST: MANUALLYREDACTED
SENTRY_DB_USER: sentry
SENTRY_DB_PASSWORD: REDACTED2
SENTRY_REDIS_HOST: redis

worker:
image: REDACTED_SENTRY_IMAGE_NAME
links:
- redis
command: “sentry run worker”
environment:
SENTRY_SECRET_KEY: REDACTED1
SENTRY_POSTGRES_HOST: MANUALLYREDACTED
SENTRY_DB_USER: sentry
SENTRY_DB_PASSWORD: REDACTED2
SENTRY_REDIS_HOST: redis

I can see that the table creation error corresponds to this bit of Sentry code

db.create_table(‘sentry_identityprovider’,…

but there are hundreds of tables and I suspect that doing something like adding a try-except block will just yield dozens of more errors to debug. I’m hoping that there’s a better way …

3

Heya! I’m guessing you’re trying to upgrade from 9.1.1 to 9.1.2, is that correct?

4

Yes, correct. Thanks for the catching my typo above.

5

Any thoughts on this @BYK et al?

6

Which version of Postgres are you using? I found https://github.com/getsentry/sentry/issues/14338#issuecomment-526195909 that may be similar/related to what you are experiencing.

7

It’s hosted RDS Postgres, version 9.6.8

8

So, if this is happening during an upgrade from 9.1.1 to 9.1.2, looks like south is trying to run a migration that has already been run. Some more logs around which migration it fails may help us devise a solution.

If this happened when trying to upgrade from 9.1.1 to 10.0.0.dev then you need to upgrade to 9.1.2 first for things to work properly as the migration system has changed in v10.