Upgrading Sentry helm chart with an external PostgreSQL database fails to migrate to latest schema definition

On-Premise
1

Hey folks. We recently upgraded our Sentry helm chart version from 4.8.1 to 8.1.2, and are running into issues with sentry-web, sentry-worker, sentry-cron attempting to access nonexistent tables (e.g. sentry_scheduledjob, sentry_groupsnooze, sentry_monitor, sentry_projectkey, among many others).

I suspect that this is as a result of a migration that would have been performed had we not configured for Sentry to reference an external Postgres database (or potentially due to having upgraded at once from v4.8.1 to v8.1.2 instead of in steps).

How do we ensure that our external Postgres database schema definition is updated accordingly when updating our Sentry version? Does some process exist to perform the proper migration?

A few examples of errors - sentry-web:

Jan 22 20:04:02 sentry-web-68c48f864b-t4xvz sentry-web psycopg2.errors.UndefinedTable: relation "sentry_projectkey" does not exist
Jan 22 20:04:02 sentry-web-68c48f864b-t4xvz sentry-web LINE 1: ...te_limit_window", "sentry_projectkey"."data" FROM "sentry_pr...

and sentry-worker:

Jan 22 20:04:04 sentry-worker-7cb899649b-86fxr sentry-worker django.db.utils.ProgrammingError: UndefinedTable('relation "sentry_groupsnooze" does not exist\nLINE 1: SELECT "sentry_groupsnooze"."group_id" FROM "sentry_groupsno...\n                                                    ^\n',)
Jan 22 20:04:04 sentry-worker-7cb899649b-86fxr sentry-worker SQL: SELECT "sentry_groupsnooze"."group_id" FROM "sentry_groupsnooze" WHERE "sentry_groupsnooze"."until" <= %s
Jan 22 20:04:04 sentry-worker-7cb899649b-86fxr sentry-worker ERROR 03:04:04 [ERROR] celery.app.trace: Task sentry.tasks.clear_expired_snoozes[dc65bbea-184b-4254-ac0a-fd526fd0ce6a] raised unexpected: ProgrammingError('UndefinedTable(\'relation "sentry_groupsnooze" does not exist\\nLINE 1: SELECT "sentry_groupsnooze"."group_id" FROM "sentry_groupsno...\\n                                                    ^\\n\',)\nSQL: SELECT "sentry_groupsnooze"."group_id" FROM "sentry_groupsnooze" WHERE "sentry_groupsnooze"."until" <= %s',) (data={'hostname': 'sentry-worker-7cb899649b-86fxr', 'id': 'dc65bbea-184b-4254-ac0a-fd526fd0ce6a', 'name': 'sentry.tasks.clear_expired_snoozes', 'exc': 'ProgrammingError(\'UndefinedTable(\\\'relation "sentry_groupsnooze" does not exist\\\\nLINE 1: SELECT "sentry_groupsnooze"."group_id" FROM "sentry_groupsno...\\\\n                                                    ^\\\\n\\\',)\\nSQL: SELECT "sentry_groupsnooze"."group_id" FROM "sentry_groupsnooze" WHERE "sentry_groupsnooze"."until" <= %s\',)', 'traceback': 'Traceback (most recent call last):\n  File "/usr/local/lib/python3.6/site-packages/sentry/db/postgres/decorators.py", line 95, in inner\n    return func(self, sql, *args, **kwargs)\n  File "/usr/local/lib/python3.6/site-packages/sentry/db/postgres/base.py", line 78, in execute\n    return self.cursor.execute(sql, clean_bad_params(params))\npsycopg2.errors.UndefinedTable: relation "sentry_groupsnooze" does not exist\nLINE 1: SELECT "sentry_groupsnooze"."group_id" FROM "sentry_groupsno...\n                                                    ^\n\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File "/usr/local/lib/python3.6/site-packages/django/db/backends/utils.py", line 64, in execute\n    return self.cursor.execute(sql, params)\n  File "/usr/local/lib/python3.6/site-packages/sentry/db/postgres/decorators.py", line 81, in inner\n    raise_the_exception(self.db, e)\n  File "/usr/local/lib/python3.6/site-packages/sentry/db/postgres/decorators.py", line 79, in inner\n    return func(self, *args, **kwargs)\n  File "/usr/local/lib/python3.6/site-packages/sentry/db/postgres/decorators.py", line 22, in inner\n    return func(self, *args, **kwargs)\n  File "/usr/local/lib/python3.6/site-packages/sentry/db/postgres/decorators.py", line 99, in inner\n    six.reraise(exc_info[0], exc_info[0](msg), exc_info[2])\n  File "/usr/local/lib/python3.6/site-packages/six.py", line 692, in reraise\n    raise value.with_traceback(tb)\n  File "/usr/local/lib/python3.6/site-packages/sentry/db/postgres/decorators.py", line 95, in inner\n    return func(self, sql, *args, **kwargs)\n  File "/usr/local/lib/python3.6/site-packages/sentry/db/postgres/base.py", line 78, in execute\n    return self.cursor.execute(sql, clean_bad_params(params))\npsycopg2.errors.UndefinedTable: UndefinedTable(\'relation "sentry_groupsnooze" does not exist\\nLINE 1: SELECT "sentry_groupsnooze"."group_id" FROM "sentry_groupsno...\\n                                                    ^\\n\',)\nSQL: SELECT "sentry_groupsnooze"."group_id" FROM "sentry_groupsnooze" WHERE "sentry_groupsnooze"."until" <= %s\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n  File "/usr/local/lib/python3.6/site-packages/celery/app/trace.py", line 412, in trace_task\n    R = retval = fun(*args, **kwargs)\n  File "/usr/local/lib/python3.6/site-packages/celery/app/trace.py", line 704, in __protected_call__\n    return self.run(*args, **kwargs)\n  File "/usr/local/lib/python3.6/site-packages/sentry_sdk/integrations/celery.py", line 197, in _inner\n    reraise(*exc_info)\n  File "/usr/local/lib/python3.6/site-packages/sentry_sdk/_compat.py", line 54, in reraise\n    raise value\n  File "/usr/local/lib/python3.6/site-packages/sentry_sdk/integrations/celery.py", line 192, in _inner\n    return f(*args, **kwargs)\n  File "/usr/local/lib/python3.6/site-packages/sentry/tasks/base.py", line 48, in _wrapped\n    result = func(*args, **kwargs)\n  File "/usr/local/lib/python3.6/site-packages/sentry/tasks/clear_expired_snoozes.py", line 13, in clear_expired_snoozes\n    GroupSnooze.objects.filter(until__lte=timezone.now()).values_list("group", flat=True)\n  File "/usr/local/lib/python3.6/site-packages/django/db/models/query.py", line 250, in __iter__\n    self._fetch_all()\n  File "/usr/local/lib/python3.6/site-packages/django/db/models/query.py", line 1121, in _fetch_all\n    self._result_cache = list(self._iterable_class(self))\n  File "/usr/local/lib/python3.6/site-packages/django/db/models/query.py", line 152, in __iter__\n    for row in compiler.results_iter(chunked_fetch=self.chunked_fetch):\n  File "/usr/local/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 841, in results_iter\n    results = self.execute_sql(MULTI, chunked_fetch=chunked_fetch)\n  File "/usr/local/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 899, in execute_sql\n    raise original_exception\n  File "/usr/local/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 889, in execute_sql\n    cursor.execute(sql, params)\n  File "/usr/local/lib/python3.6/site-packages/sentry_sdk/integrations/django/__init__.py", line 489, in execute\n    return real_execute(self, sql, params)\n  File "/usr/local/lib/python3.6/site-packages/django/db/backends/utils.py", line 64, in execute\n    return self.cursor.execute(sql, params)\n  File "/usr/local/lib/python3.6/site-packages/django/db/utils.py", line 94, in __exit__\n    six.reraise(dj_exc_type, dj_exc_value, traceback)\n  File "/usr/local/lib/python3.6/site-packages/django/utils/six.py", line 685, in reraise\n    raise value.with_traceback(tb)\n  File "/usr/local/lib/python3.6/site-packages/django/db/backends/utils.py", line 64, in execute\n    return self.cursor.execute(sql, params)\n  File "/usr/local/lib/python3.6/site-packages/sentry/db/postgres/decorators.py", line 81, in inner\n    raise_the_exception(self.db, e)\n  File "/usr/local/lib/python3.6/site-packages/sentry/db/postgres/decorators.py", line 79, in inner\n    return func(self, *args, **kwargs)\n  File "/usr/local/lib/python3.6/site-packages/sentry/db/postgres/decorators.py", line 22, in inner\n    return func(self, *args, **kwargs)\n  File "/usr/local/lib/python3.6/site-packages/sentry/db/postgres/decorators.py", line 99, in inner\n    six.reraise(exc_info[0], exc_info[0](msg), exc_info[2])\n  File "/usr/local/lib/python3.6/site-packages/six.py", line 692, in reraise\n    raise value.with_traceback(tb)\n  File "/usr/local/lib/python3.6/site-packages/sentry/db/postgres/decorators.py", line 95, in inner\n    return func(self, sql, *args, **kwargs)\n  File "/usr/local/lib/python3.6/site-packages/sentry/db/postgres/base.py", line 78, in execute\n    return self.cursor.execute(sql, clean_bad_params(params))\ndjango.db.utils.ProgrammingError: UndefinedTable(\'relation "sentry_groupsnooze" does not exist\\nLINE 1: SELECT "sentry_groupsnooze"."group_id" FROM "sentry_groupsno...\\n                                                    ^\\n\',)\nSQL: SELECT "sentry_groupsnooze"."group_id" FROM "sentry_groupsnooze" WHERE "sentry_groupsnooze"."until" <= %s\n', 'args': '()', 'kwargs': '{}', 'description': 'raised unexpected', 'internal': False})

For context, our infra is provisioned via Terraform. Our config is as follows (values correspond directly with the helm chart values.yaml):

resource "helm_release" "sentry" {
  name       = "sentry"
  repository = "https://sentry-kubernetes.github.io/charts"
  chart      = "sentry"
  version    = "8.1.2"

  set {
    name  = "service.type"
    value = "ClusterIP"
  }

  set {
    name  = "postgresql.enabled"
    value = false
  }

  set {
    name  = "externalPostgresql.database"
    value = var.sentry_config.db_name
  }

  set {
    name  = "externalPostgresql.host"
    value = var.sentry_config.db_host
  }

  set {
    name  = "externalPostgresql.username"
    value = var.sentry_config.db_user
  }

  set_sensitive {
    name  = "externalPostgresql.password"
    value = var.sentry_config.db_password
  }

  set {
    name  = "user.email"
    value = var.sentry_config.email
  }

  set_sensitive {
    name  = "user.password"
    value = var.sentry_config.password
  }

  set {
    name  = "mail.backend"
    value = "smtp"
  }

  set {
    name  = "mail.host"
    value = var.mailgun_config.smtp_host
  }

  set {
    name  = "mail.port"
    value = 587
  }

  set {
    name  = "mail.username"
    value = var.mailgun_config.smtp_user
  }

  set_sensitive {
    name  = "mail.password"
    value = var.mailgun_config.smtp_password
  }

  set {
    name  = "mail.useTls"
    value = true
  }

  set {
    name  = "config.configYml"
    value = <<-EOT
      slack.client-id: "${var.sentry_config.slack_client_id}"
      slack.client-secret: "${var.sentry_config.slack_client_secret}"
      slack.verification-token: "${var.sentry_config.slack_verification_token}"
      github-app.id: ${var.sentry_config.github_app_id}
      github-app.name: "${var.sentry_config.github_app_name}"
      github-app.webhook-secret: "${var.sentry_config.github_app_webhook_secret}"
      github-app.private-key: "${replace(var.sentry_config.github_app_private_key, "/\n/", "\\n")}"
      github-app.client-id: "${var.sentry_config.github_app_client_id}"
      github-app.client-secret: "${var.sentry_config.github_app_client_secret}"
    EOT
  }

  set {
    name  = "config.sentryConfPy"
    value = "SENTRY_FEATURES['projects:custom-inbound-filters'] = True"
  }

}
2

Sentry has an upgrade command which runs the needed DB migrations. That said if you are upgrading from a version earlier than 9.1.2 you first need to get to 9.1.2, run upgrade and then upgrade to the final version you want and run upgrade one more time.

3

Thanks - given that I’m using the Sentry Helm chart, where would I be running the upgrade command? Could you point me to documentation with more details on running upgrade?

4

It’s worth noting that, because of the migration issue, sentry-web and sentry-worker fail to start, so it’s impossible to kubectl exec into the pods in order to run sentry migrate. Do you have suggestions for potential alternatives? All it seems I need to do is migrate our Sentry schema, but I’m having quite a bit of difficulty doing so.

5

You should be able to run a new container, directly with the upgrade command instead of trying to start it and then doing an exec inside. This is how we do it over on the official onpremise repo.

6

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.