Sentry Upgrade from 20.7.2 to 21.6.3

[chhetripradeep]

Hi Sentry Team,

We are currently running sentry 20.7.2 and we would like to upgrade it to latest sentry version. We noticed we first need to upgrade to 21.6.3 but we have a question around clickhouse schema change.

We noticed the primary key on errors_local table changed in between these versions.

In 20.7.2

ORDER BY (org_id, project_id, toStartOfDay(timestamp), primary_hash_hex, event_hash)

In 21.6.3

ORDER BY (project_id, toStartOfDay(timestamp), primary_hash, cityHash64(event_id))

CH doesn’t allow changing the primary key of a table. We looked at the snuba migration tool’s migration for this table snuba/0011_rebuild_errors.py at 92e15e47b180897ef974609a86bf215d47a6321d · getsentry/snuba · GitHub

It seems like we just drop the existing table and create fresh new table with updated schema.

It will create data loss unless we are replaying events from kafka to CH.

I am curious how do you suggest doing it without any data loss.

Regards,

[BYK]

The migration should transition all the data to the new table before dropping the old one. @fpacifici and @lynnagara to confirm.

[turing_machine]

I’d definitely want to get confirmation from the team before running that update, so if there’s any way to get explicit confirmation, I’d really appreciate it.

[chhetripradeep]

Hi @turing_machine

Yesterday I realised that in 20.7.2, errors_local table isn’t being used at all and is completely empty. That must be the reason why in the migration (one I linked in the question), we aren’t copying the data. Since I am not sure which version of sentry you are currently running, you can login to clickhouse via clickhouse-client and check the number of rows to validate.

Let us know!

[chadwhitacre]

@turing_machine

if there’s any way to get explicit confirmation

Best way would be to run the migration in a sandbox and inspect the results directly.

[lynnagara]

Hi @chhetripradeep Just to confirm what you said earlier was right, the original version of the errors_local table was not used. It was just a test and was not actually being used anywhere when we deleted and rebuilt it in the 0011_rebuild_errors migration with the new primary key, etc. The errors_local table is the replacement for sentry_local. Migration 0014_backfill_errors is the one that copies the data over to the new table. After that should should be using the new table everywhere.