Upgrade 9.1.2 -> current via export&import?

Hello,

We are happy Sentry users and use the cloud offering for our production services, and have a small on-premise sentry instance for our development environments.

I’m tasked with upgrading our on-prem install from version 9.1.2 (docker-compose based, with the oidc auth plugin to authenticate against gitlab) to version 21.4 (or “current” in general).

I was wondering:

Considering the export and import procedures described at https://develop.sentry.dev/self-hosted/backup/#quick-backup can I perform an export on the old sentry and then an import on a new empty sentry installation?

Any reason why you are not considering an in-place upgrade which is an option you can take? (after getting some backups of course)

Hello @BYK and thank you for your reply.

upgrade in place is definitely an option, but some initial tests did not go very well, probably because we also use the oidc plugin that seems to need to be updated on its own.

Also, regarding the upgrade in place, it’s not clear to me if I need to perform 9.1.2 → 10.x or if I can go straight to 21.4

We test the latest self-hosted builds with sentry-auth-oidc installed. It seems to work (at least it installs and Sentry keeps working :sweat_smile:)

Fair point, we should clarify this. Right now it is safe and recommended to go directly to 21.4

Hello @BYK and thank you for your reply.

I will try to perform the jump as you indicate (9.1.2 → 21.4) and use sentry-auth-oidc.

Thank you again!

@esantoro, my pleasure!

Please don’t forget to get backups (not via import/export, full backups) and let us know how it goes :slight_smile:

Hi @BYK
After some more work I managed to get the update going, although I still have some problems that I’m working on.

In my case (I’m writing this in case other people are in a similar situation) I had to do the following to get database migration going:

  1. set myself as admin user (as I’m taking over this sentry installation from another colleague)
$ docker exec -it sentry-postgres /bin/sh
$ psql -U postgres 
postgres=# update auth_user set is_superuser=True where email = '<myemail>' ;                                                                                     
UPDATE 1                                                                                                                                                                           
postgres=# update auth_user set is_staff=True where email = '<myemail>' ;                                                                                         
UPDATE 1                                                                                                                                                                           
postgres=# update auth_user set is_active=True where email = '<myemail>' ;  
  1. disable email sending from the testing instance (remove SMTP-related environment variables)
  2. change my user’s password (as I initially signed up using old gitlab auth):
postgres=# select username from auth_user where email='<myemail>';
             username             
----------------------------------
 2402e6ea0FFFFFFFFFFFFFFFFF
(1 row)

postgres=#

then

docker exec -it  sentry_web_1 /bin/bash
root@9d36d3f6d0e1:/usr/local/bin# sentry django changepassword 2402e6ea0FFFFFFFFFFFFFFFFF
14:57:21 [WARNING] sentry.utils.geo: settings.GEOIP_PATH_MMDB not configured.
14:57:28 [INFO] sentry.plugins.github: apps-not-configured
Changing password for user '2402e6ea0FFFFFFFFFFFFFFFFF'
Password: 
Password (again): 
Password changed successfully for user '2402e6ea0FFFFFFFFFFFFFFFFF'
  1. start the old sentry and disable gitlab authentication

  2. remove old events

docker exec -it  sentry_web_1 /bin/bash
root@9d36d3f6d0e1:/usr/local/bin# sentry cleanup --days 0

this is going to take a while, and then I also run a full vacuumdb just to be sure.

  1. shut down sentry except for postgres, and delete releases (table sentry_releases) from the database migrating those have create index problems. This sadly implies deleting data from a bunch of other tables due to foreign keys. It’s mostly okay I guess because I mostly care about keeping users, projects and DSNs.
    So:
delete from sentry_release_project ;  
delete from sentry_releaseprojectenvironment ;   
delete from sentry_deploy ;   

delete from sentry_grouphash ;  
delete from sentry_activity ;   
delete from sentry_groupemailthread ;  

delete from sentry_grouprulestatus ;  
delete from sentry_groupseen ;  
delete from sentry_groupsubscription ;  
delete from sentry_groupasignee ;  
delete from sentry_groupbookmark ;   

delete from sentry_groupedmessage ;  

delete from sentry_releaseheadcommit ;  
delete from sentry_releasecommit ;  
delete from sentry_releasefile;

delete from sentry_release;   
  1. Now I can start the actual migration. I copy the postgres folders along with the sentry-data folder over to the new installation and create docker volumes pointing to those directories:
docker volume create \
       -d local \
       -o type=none \
       -o o=bind \
       -o "device=/storage/sentry21/sentry/data/sentry" \
       sentry-data


docker volume create \
       -d local \
       -o type=none \
       -o o=bind \
       -o "device=/storage/sentry21/sentry/data/postgres" \
       sentry-postgres
  1. patch the docker-compose.yml file so that nginx can load tls certificates and binds to port 443.
  2. run the install.sh
  3. add sentry-auth-oidc==5.0.0 to sentry’s requirements.txt and then configure the sentry-auth-oidc plugin (OIDC_CLIENT_ID, OIDC_CLIENT_SECRET, OIDC_SCOPE and OIDC_DOMAIN)
  4. boot the sentry containers
  5. login with my email and password set in step 3
  6. configure login with gitlab again.

This kinda works, projects are still available, users and DSNs too.

However, this minor inconvenience came up while testing with an user: when he logged in via oidc (backed by gitlab) a new user with the same Full Name and email is created, and of course a different username (username as in username property of the django auth system, reflected in the auth_user table in postgres).

I tried digging up the source code for the sentry-oidc-auth plugin as well as some of the Sentry code but it’s a bit too much of a rabbit hole for me.

@BYK is there anything that you think I could do to have existing users in sentry be matched to users in gitlab? That’s effectively the last thing I need to get our development sentry updated (we use sentry cloud for production).

Thank you in advance!

esantoro

1 Like

Sorry, I honestly have no idea about how this plugin works. That said our enterprise team is working on better SSO and SAML support which may help in the near future. My guess is there’s something in the database that links social accounts with normal user accounts and for some reason this is not established for you.

Hello @BYK and thank you anyway for your help and support so far.

I have some theories that I’ll work on so far:

  • I have create a separate client in gitlab for SSO authentication, I might have to try with the already existing one. I’m not sure if gitlab issues different subs for different OIDC clients.
  • sub format chaged in version 11.1 of gitlab, there are now sub and sub_legacy. The user i tested with has been in the company since before gitlab 11.1 and might have the old sub_legacy (maybe? I guess?).

I’ll try with a more recent user, with the main OIDC client in gitlab and with a mix of the two, I guess.

1 Like

Just a quick update: it seems to be working correctly.

What I didn’t know before starting this sentry installation takeover is that Sentry has both the concepts of users (as in users of the underlying django framework) and members (users of the actual sentry code).

After testing with some more users the following behavior emerged:

  • some gitlab users, authenticated via oidc, are mapped back onto the already existing django user (and thus to the correct member)
  • some gitlab users, authenticated via oidc, are mapped onto a new django user, but then again mapped to the right member

So after some initial tests with more users the upgrade seems to be successful (projects still exists, users can see their project and the old DSNs are still accepted by the system).

1 Like

@esantoro thanks a lot for sharing the final status and sorry for the late response. Glad you got it working!