Error importing events during upgrade from 9.1.2 to 20.11 (and up)

Currently (still) at sentry 9.0.0 and trying to upgrade to 20.x. We started last October (No events / issue details after upgrade from 9.1.2 to 20.9/latest - #16 by sraisl) but decided to wait with the final migration/upgrade till we cleaned out our issues/events in 9.0.0 (had over 16 Mio events we didn’t want to import in the upgraded instance) and furthermore we were waiting for an upgrade to our Kubernetes Cluster.
I tried the upgrade yesterday again (with a “cleaned” database, reduced events to about 2 mio) but during the upgrade I run into an error. Let me just describe the steps I tried to upgrade:

  • created a dump of the sentry 9.0.0 postgres database

  • restored the database in a new (clean) database

  • upgraded the database to 9.1.2 (went through without any errors)

  • fired up the Kubernetes Deployment with a Sentry 20.11 Release

  • after all went to running state
  • executed ‘snuba bootstrap --force’ followed by ‘snuba migrations migrate’ in the snuba api pod

  • executed ‘sentry upgrade --noinput’ in the sentry web pod
    (these steps worked for my last try last October)
    during importing the events I get this error and the upgrade stops:

      Traceback (most recent call last):
    File "/usr/local/bin/sentry", line 8, in <module>
      sys.exit(main())
    File "/usr/local/lib/python3.6/site-packages/sentry/runner/__init__.py", line 166, in main
      cli(prog_name=get_prog(), obj={}, max_content_width=100)
    File "/usr/local/lib/python3.6/site-packages/click/core.py", line 829, in __call__
      return self.main(*args, **kwargs)
    File "/usr/local/lib/python3.6/site-packages/click/core.py", line 782, in main
      rv = self.invoke(ctx)
    File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
      return ctx.invoke(self.callback, **ctx.params)
    File "/usr/local/lib/python3.6/site-packages/click/core.py", line 610, in invoke
      return callback(*args, **kwargs)
    File "/usr/local/lib/python3.6/site-packages/click/decorators.py", line 21, in new_func
      return f(get_current_context(), *args, **kwargs)
    File "/usr/local/lib/python3.6/site-packages/sentry/runner/decorators.py", line 30, in inner
      return ctx.invoke(f, *args, **kwargs)
    File "/usr/local/lib/python3.6/site-packages/click/core.py", line 610, in invoke
      return callback(*args, **kwargs)
    File "/usr/local/lib/python3.6/site-packages/click/decorators.py", line 21, in new_func
      return f(get_current_context(), *args, **kwargs)
    File "/usr/local/lib/python3.6/site-packages/sentry/runner/commands/upgrade.py", line 174, in upgrade
      _upgrade(not noinput, traceback, verbosity, not no_repair, with_nodestore)
    File "/usr/local/lib/python3.6/site-packages/sentry/runner/commands/upgrade.py", line 130, in _upgrade
      ignore_ghost_migrations=True,
    File "/usr/local/lib/python3.6/site-packages/django/core/management/__init__.py", line 131, in call_command
      return command.execute(*args, **defaults)
    File "/usr/local/lib/python3.6/site-packages/django/core/management/base.py", line 330, in execute
      output = self.handle(*args, **options)
    File "/usr/local/lib/python3.6/site-packages/django/core/management/commands/migrate.py", line 204, in handle
      fake_initial=fake_initial,
    File "/usr/local/lib/python3.6/site-packages/django/db/migrations/executor.py", line 115, in migrate
      state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
    File "/usr/local/lib/python3.6/site-packages/django/db/migrations/executor.py", line 145, in _migrate_all_forwards
      state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
    File "/usr/local/lib/python3.6/site-packages/sentry/new_migrations/monkey/executor.py", line 26, in apply_migration
      state, migration, fake=fake, fake_initial=fake_initial
    File "/usr/local/lib/python3.6/site-packages/django/db/migrations/executor.py", line 244, in apply_migration
      state = migration.apply(state, schema_editor)
    File "/usr/local/lib/python3.6/site-packages/django/db/migrations/migration.py", line 129, in apply
      operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
    File "/usr/local/lib/python3.6/site-packages/django/db/migrations/operations/special.py", line 193, in database_forwards
      self.code(from_state.apps, schema_editor)
    File "/usr/local/lib/python3.6/site-packages/sentry/migrations/0024_auto_20191230_2052.py", line 83, in backfill_eventstream
      group = event.group
    File "/usr/local/lib/python3.6/site-packages/sentry/eventstore/models.py", line 281, in group
      self._group_cache = Group.objects.get(id=self.group_id)
    File "/usr/local/lib/python3.6/site-packages/django/db/models/manager.py", line 85, in manager_method
      return getattr(self.get_queryset(), name)(*args, **kwargs)
    File "/usr/local/lib/python3.6/site-packages/django/db/models/query.py", line 380, in get
      self.model._meta.object_name
    

sentry.models.group.DoesNotExist: Group matching query does not exist.

Upgrading to 20.11 works only if I skip importing the events, then everything seems to work fine, but clearly all issues and events don’t show up.

I get the same error with later releases, event with a nightly build I ran into the same error.

Any help is appreciated…

There is a fix introduced in 20.12.1 to prevent this from happening: fix(snuba-migration): Add try-except for invalid groups (#22221) · getsentry/sentry@3948552 · GitHub

I recommend using the latest version: 21.1.0

Sorry, should have mentioned that this was the last version I tried with:

sentry, version 21.1.0 (486d790a8f52)

tried again with sentry, version 21.2.0.dev0 (cdc07c1af5ee) - still the same error…

Can you share the new error and the stacktrace as they cannot be the same error (the message may be the same but the stacktrace should be different)

sure, here you are:

sentry, version 21.2.0.dev0 (cdc07c1af5ee):

Traceback (most recent call last):
  File "/usr/local/bin/sentry", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/site-packages/sentry/runner/__init__.py", line 164, in main
    cli(prog_name=get_prog(), obj={}, max_content_width=100)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/sentry/runner/decorators.py", line 28, in inner
    return ctx.invoke(f, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/sentry/runner/commands/upgrade.py", line 172, in upgrade
    _upgrade(not noinput, traceback, verbosity, not no_repair, with_nodestore)
  File "/usr/local/lib/python3.6/site-packages/sentry/runner/commands/upgrade.py", line 128, in _upgrade
    ignore_ghost_migrations=True,
  File "/usr/local/lib/python3.6/site-packages/django/core/management/__init__.py", line 131, in call_command
    return command.execute(*args, **defaults)
  File "/usr/local/lib/python3.6/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.6/site-packages/django/core/management/commands/migrate.py", line 204, in handle
    fake_initial=fake_initial,
  File "/usr/local/lib/python3.6/site-packages/django/db/migrations/executor.py", line 115, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python3.6/site-packages/django/db/migrations/executor.py", line 145, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python3.6/site-packages/sentry/new_migrations/monkey/executor.py", line 24, in apply_migration
    state, migration, fake=fake, fake_initial=fake_initial
  File "/usr/local/lib/python3.6/site-packages/django/db/migrations/executor.py", line 244, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/usr/local/lib/python3.6/site-packages/django/db/migrations/migration.py", line 129, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/usr/local/lib/python3.6/site-packages/django/db/migrations/operations/special.py", line 193, in database_forwards
    self.code(from_state.apps, schema_editor)
  File "/usr/local/lib/python3.6/site-packages/sentry/migrations/0024_auto_20191230_2052.py", line 81, in backfill_eventstream
    group = event.group
  File "/usr/local/lib/python3.6/site-packages/sentry/eventstore/models.py", line 279, in group
    self._group_cache = Group.objects.get(id=self.group_id)
  File "/usr/local/lib/python3.6/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/django/db/models/query.py", line 380, in get
    self.model._meta.object_name
sentry.models.group.DoesNotExist: Group matching query does not exist.

Ah, thank you very much! The exception handling is failing because the Exception type seems to be coming from a different place. Will submit a fix and let you know here.

1 Like

Can you try with SENTRY_IMAGE=us.gcr.io/sentryio/sentry:6b7b2f4d7fb3438065cc62dea462bf8f52f782fe./install.sh and see if it fixes?

This is the PR btw: https://github.com/getsentry/sentry/pull/23484

Waiting for confirmation from @sraisl before merging.

Seems to work as far as I didn’t receive the error any more at my last try - but the migration is still running (35 hours till now), can’t say If this will be successful. Maybe I’d give it another try after doing a further cleanup of the existing events.

1 Like

Merging the PR then, thanks a lot for confirming!

Also, curious about the end result still :slight_smile:

Thanks a lot so far, I marked your answer as solution, it fixed this issue. But sorry to say that there is another one, I opened a new topic: Snuba consumer crashes during migration 9.1.2 -> 21.x

1 Like

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.