Upgrading to the latest version from 9.1.2

sunjha · September 16, 2020, 10:55am

So after spending a day and fixing all the issue I faced while upgrading to the latest version(Major once being from the dependencies added in docker-compose.yml and the GEOIP_PATH_MMDB issue) I was able to run the containers
The only issue now i am facing is that these new container are not connected to existing sentry file and because of that I am getting these exceptions:

10:41:53 [WARNING] django.security.csrf: Forbidden (CSRF cookie not set.): /api/13/store/ (status_code=403 request=<WSGIRequest: POST u'/api/13/store/?sentry_version=7&sentry_client=raven-js%2F3.27.0&sentry_key=bd476e681adb4665a92346ddc602e1f1'>)
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
  File "/usr/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 249, in _legacy_get_response
    response = self._get_response(request)
  File "/usr/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 178, in _get_response
    response = middleware_method(request, callback, callback_args, callback_kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry_sdk/integrations/django/middleware.py", line 90, in sentry_wrapped_method
    return old_method(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/django/middleware/csrf.py", line 294, in process_view
    return self._reject(request, REASON_NO_CSRF_COOKIE)
  File "/usr/local/lib/python2.7/site-packages/django/middleware/csrf.py", line 163, in _reject
    return _get_failure_view()(request, reason=reason)
  File "/usr/local/lib/python2.7/site-packages/django/views/generic/base.py", line 68, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/django/utils/decorators.py", line 67, in _wrapper
    return bound_func(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/django/utils/decorators.py", line 63, in bound_func
    return func.__get__(self, type(self))(*args2, **kwargs2)
  File "/usr/local/lib/python2.7/site-packages/sentry/web/frontend/csrf_failure.py", line 16, in dispatch
    return render_to_response("sentry/403-csrf-failure.html", context, request, status=403)
  File "/usr/local/lib/python2.7/site-packages/sentry/web/helpers.py", line 97, in render_to_response
    response = HttpResponse(render_to_string(template, context, request))
  File "/usr/local/lib/python2.7/site-packages/sentry/web/helpers.py", line 93, in render_to_string
    return loader.render_to_string(template, context=context, request=request)
  File "/usr/local/lib/python2.7/site-packages/django/template/loader.py", line 68, in render_to_string
    return template.render(context, request)
  File "/usr/local/lib/python2.7/site-packages/django/template/backends/django.py", line 66, in render
    return self.template.render(context)
  File "/usr/local/lib/python2.7/site-packages/django/template/base.py", line 207, in render
    return self._render(context)
  File "/usr/local/lib/python2.7/site-packages/django/template/base.py", line 199, in _render
    return self.nodelist.render(context)
  File "/usr/local/lib/python2.7/site-packages/django/template/base.py", line 990, in render
    bit = node.render_annotated(context)
  File "/usr/local/lib/python2.7/site-packages/django/template/base.py", line 957, in render_annotated
    return self.render(context)
  File "/usr/local/lib/python2.7/site-packages/django/template/loader_tags.py", line 177, in render
    return compiled_parent._render(context)
  File "/usr/local/lib/python2.7/site-packages/django/template/base.py", line 199, in _render
    return self.nodelist.render(context)
  File "/usr/local/lib/python2.7/site-packages/django/template/base.py", line 990, in render
    bit = node.render_annotated(context)
  File "/usr/local/lib/python2.7/site-packages/django/template/base.py", line 957, in render_annotated
    return self.render(context)
  File "/usr/local/lib/python2.7/site-packages/django/template/loader_tags.py", line 177, in render
    return compiled_parent._render(context)
  File "/usr/local/lib/python2.7/site-packages/django/template/base.py", line 199, in _render
    return self.nodelist.render(context)
  File "/usr/local/lib/python2.7/site-packages/django/template/base.py", line 990, in render
    bit = node.render_annotated(context)
  File "/usr/local/lib/python2.7/site-packages/django/template/base.py", line 957, in render_annotated
    return self.render(context)
  File "/usr/local/lib/python2.7/site-packages/django/template/library.py", line 203, in render
    output = self.func(*resolved_args, **resolved_kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/templatetags/sentry_react.py", line 13, in get_react_config
    context = get_client_config(context.get("request", None))
  File "/usr/local/lib/python2.7/site-packages/sentry/web/client_config.py", line 144, in get_client_config
    public_dsn = _get_public_dsn()
  File "/usr/local/lib/python2.7/site-packages/sentry/web/client_config.py", line 90, in _get_public_dsn
    key = _get_project_key(project_id)
  File "/usr/local/lib/python2.7/site-packages/sentry/web/client_config.py", line 76, in _get_project_key
    )[0]
  File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 289, in __getitem__
    return list(qs)[0]
  File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 250, in __iter__
    self._fetch_all()
  File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 1121, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 53, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch)
  File "/usr/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 887, in execute_sql
    cursor = self.connection.cursor()
  File "/usr/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 254, in cursor
    return self._cursor()
  File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py", line 44, in inner
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/base.py", line 99, in _cursor
    return super(DatabaseWrapper, self)._cursor()
  File "/usr/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 229, in _cursor
    self.ensure_connection()
  File "/usr/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 213, in ensure_connection
    self.connect()
  File "/usr/local/lib/python2.7/site-packages/django/db/utils.py", line 94, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/usr/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 213, in ensure_connection
    self.connect()
  File "/usr/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 189, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/usr/local/lib/python2.7/site-packages/django/db/backends/postgresql/base.py", line 176, in get_new_connection
    connection = Database.connect(**conn_params)
  File "/usr/local/lib/python2.7/site-packages/psycopg2/__init__.py", line 126, in connect
    dsn = _ext.make_dsn(dsn, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/psycopg2/extensions.py", line 175, in make_dsn
    parse_dsn(dsn)
ProgrammingError: invalid dsn: invalid connection option "autocommit"

10:41:53 [ERROR] django.request: Internal Server Error: /api/13/store/ (status_code=500 request=<WSGIRequest: POST u'/api/13/store/?sentry_version=7&sentry_client=raven-js%2F3.27.0&sentry_key=bd476e681adb4665a92346ddc602e1f1'>)
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/sentry_sdk/integrations/excepthook.py", line 59, in sentry_sdk_excepthook
    hub.capture_event(event, hint=hint)
  File "/usr/local/lib/python2.7/site-packages/sentry_sdk/hub.py", line 323, in capture_event
    rv = client.capture_event(event, hint, scope)
  File "/usr/local/lib/python2.7/site-packages/sentry_sdk/client.py", line 327, in capture_event
    event_opt = self._prepare_event(event, hint, scope)
  File "/usr/local/lib/python2.7/site-packages/sentry_sdk/client.py", line 158, in _prepare_event
    event_ = scope.apply_to_event(event, hint)
  File "/usr/local/lib/python2.7/site-packages/sentry_sdk/scope.py", line 58, in wrapper
    return fn(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry_sdk/scope.py", line 353, in apply_to_event
    return _drop(event, event_processor, "event processor")
  File "/usr/local/lib/python2.7/site-packages/sentry_sdk/scope.py", line 308, in _drop
    logger.info("%s (%s) dropped event (%s)", ty, cause, event)
  File "/usr/local/lib/python2.7/logging/__init__.py", line 1174, in info
    self._log(INFO, msg, args, **kwargs)
  File "/usr/local/lib/python2.7/logging/__init__.py", line 1293, in _log
    self.handle(record)
  File "/usr/local/lib/python2.7/logging/__init__.py", line 1303, in handle
    self.callHandlers(record)
  File "/usr/local/lib/python2.7/site-packages/sentry_sdk/integrations/logging.py", line 83, in sentry_patched_callhandlers
    return old_callhandlers(self, record)
  File "/usr/local/lib/python2.7/logging/__init__.py", line 1343, in callHandlers
    hdlr.handle(record)
  File "/usr/local/lib/python2.7/logging/__init__.py", line 766, in handle
    self.emit(record)
  File "/usr/local/lib/python2.7/site-packages/sentry/logging/handlers.py", line 97, in emit
    logger.log(**kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/../structlog/stdlib.py", line 102, in log
    return self._proxy_to_logger(_LEVEL_TO_NAME[level], event, *args, **kw)
  File "/usr/local/lib/python2.7/site-packages/sentry/../structlog/stdlib.py", line 119, in _proxy_to_logger
    **event_kw)
  File "/usr/local/lib/python2.7/site-packages/sentry/../structlog/_base.py", line 176, in _proxy_to_logger
    args, kw = self._process_event(method_name, event, event_kw)
  File "/usr/local/lib/python2.7/site-packages/sentry/../structlog/_base.py", line 136, in _process_event
    event_dict = proc(self._logger, method_name, event_dict)
  File "/usr/local/lib/python2.7/site-packages/sentry/../structlog/stdlib.py", line 275, in __call__
    event_dict["event"] = event_dict["event"] % args
  File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 226, in __repr__
    data = list(self[:REPR_OUTPUT_SIZE + 1])
  File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 250, in __iter__
    self._fetch_all()
  File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 1121, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 53, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch)
  File "/usr/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 887, in execute_sql
    cursor = self.connection.cursor()
  File "/usr/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 254, in cursor
    return self._cursor()
  File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py", line 44, in inner
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/base.py", line 99, in _cursor
    return super(DatabaseWrapper, self)._cursor()
  File "/usr/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 229, in _cursor
    self.ensure_connection()
  File "/usr/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 213, in ensure_connection
    self.connect()
  File "/usr/local/lib/python2.7/site-packages/django/db/utils.py", line 94, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/usr/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 213, in ensure_connection
    self.connect()
  File "/usr/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 189, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/usr/local/lib/python2.7/site-packages/django/db/backends/postgresql/base.py", line 176, in get_new_connection
    connection = Database.connect(**conn_params)
  File "/usr/local/lib/python2.7/site-packages/psycopg2/__init__.py", line 126, in connect
    dsn = _ext.make_dsn(dsn, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/psycopg2/extensions.py", line 175, in make_dsn
    parse_dsn(dsn)
ProgrammingError: invalid dsn: invalid connection option "autocommit"

10:41:53 [ERROR] sentry_sdk.errors: Internal error in sentry_sdk
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/sentry/wsgi.py", line 36, in __call__
    response = super(FileWrapperWSGIHandler, self).__call__(environ, start_response)
  File "/usr/local/lib/python2.7/site-packages/sentry_sdk/integrations/django/__init__.py", line 120, in sentry_patched_wsgi_handler
    return SentryWsgiMiddleware(bound_old_app)(environ, start_response)
  File "/usr/local/lib/python2.7/site-packages/sentry_sdk/integrations/wsgi.py", line 129, in __call__
    reraise(*_capture_exception(hub))
  File "/usr/local/lib/python2.7/site-packages/sentry_sdk/integrations/wsgi.py", line 125, in __call__
    _sentry_start_response, start_response, transaction
  File "/usr/local/lib/python2.7/site-packages/django/core/handlers/wsgi.py", line 157, in __call__
    response = self.get_response(request)
  File "/usr/local/lib/python2.7/site-packages/sentry_sdk/integrations/django/__init__.py", line 350, in sentry_patched_get_response
    return old_get_response(self, request)
  File "/usr/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 124, in get_response
    response = self._middleware_chain(request)
  File "/usr/local/lib/python2.7/site-packages/django/core/handlers/exception.py", line 43, in inner
    response = response_for_exception(request, exc)
  File "/usr/local/lib/python2.7/site-packages/django/core/handlers/exception.py", line 93, in response_for_exception
    response = handle_uncaught_exception(request, get_resolver(get_urlconf()), sys.exc_info())
  File "/usr/local/lib/python2.7/site-packages/django/core/handlers/exception.py", line 143, in handle_uncaught_exception
    return callback(request, **param_dict)
  File "/usr/local/lib/python2.7/site-packages/django/views/generic/base.py", line 68, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/web/frontend/error_500.py", line 38, in dispatch
    return render_to_response("sentry/500.html", status=500, context=context, request=request)
  File "/usr/local/lib/python2.7/site-packages/sentry/web/helpers.py", line 97, in render_to_response
    response = HttpResponse(render_to_string(template, context, request))
  File "/usr/local/lib/python2.7/site-packages/sentry/web/helpers.py", line 93, in render_to_string
    return loader.render_to_string(template, context=context, request=request)
  File "/usr/local/lib/python2.7/site-packages/django/template/loader.py", line 68, in render_to_string
    return template.render(context, request)
  File "/usr/local/lib/python2.7/site-packages/django/template/backends/django.py", line 66, in render
    return self.template.render(context)
  File "/usr/local/lib/python2.7/site-packages/django/template/base.py", line 207, in render
    return self._render(context)
  File "/usr/local/lib/python2.7/site-packages/django/template/base.py", line 199, in _render
    return self.nodelist.render(context)
  File "/usr/local/lib/python2.7/site-packages/django/template/base.py", line 990, in render
    bit = node.render_annotated(context)
  File "/usr/local/lib/python2.7/site-packages/django/template/base.py", line 957, in render_annotated
    return self.render(context)
  File "/usr/local/lib/python2.7/site-packages/django/template/loader_tags.py", line 177, in render
    return compiled_parent._render(context)
  File "/usr/local/lib/python2.7/site-packages/django/template/base.py", line 199, in _render
    return self.nodelist.render(context)
  File "/usr/local/lib/python2.7/site-packages/django/template/base.py", line 990, in render
    bit = node.render_annotated(context)
  File "/usr/local/lib/python2.7/site-packages/django/template/base.py", line 957, in render_annotated
    return self.render(context)
  File "/usr/local/lib/python2.7/site-packages/django/template/loader_tags.py", line 177, in render
    return compiled_parent._render(context)
  File "/usr/local/lib/python2.7/site-packages/django/template/base.py", line 199, in _render
    return self.nodelist.render(context)
  File "/usr/local/lib/python2.7/site-packages/django/template/base.py", line 990, in render
    bit = node.render_annotated(context)
  File "/usr/local/lib/python2.7/site-packages/django/template/base.py", line 957, in render_annotated
    return self.render(context)
  File "/usr/local/lib/python2.7/site-packages/django/template/library.py", line 203, in render
    output = self.func(*resolved_args, **resolved_kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/templatetags/sentry_react.py", line 13, in get_react_config
    context = get_client_config(context.get("request", None))
  File "/usr/local/lib/python2.7/site-packages/sentry/web/client_config.py", line 144, in get_client_config
    public_dsn = _get_public_dsn()
  File "/usr/local/lib/python2.7/site-packages/sentry/web/client_config.py", line 90, in _get_public_dsn
    key = _get_project_key(project_id)
  File "/usr/local/lib/python2.7/site-packages/sentry/web/client_config.py", line 76, in _get_project_key
    )[0]
  File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 289, in __getitem__
    return list(qs)[0]
  File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 250, in __iter__
    self._fetch_all()
  File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 1121, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 53, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch)
  File "/usr/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 887, in execute_sql
    cursor = self.connection.cursor()
  File "/usr/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 254, in cursor
    return self._cursor()
  File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py", line 44, in inner
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/base.py", line 99, in _cursor
    return super(DatabaseWrapper, self)._cursor()
  File "/usr/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 229, in _cursor
    self.ensure_connection()
  File "/usr/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 213, in ensure_connection
    self.connect()
  File "/usr/local/lib/python2.7/site-packages/django/db/utils.py", line 94, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/usr/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 213, in ensure_connection
    self.connect()
  File "/usr/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 189, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/usr/local/lib/python2.7/site-packages/django/db/backends/postgresql/base.py", line 176, in get_new_connection
    connection = Database.connect(**conn_params)
  File "/usr/local/lib/python2.7/site-packages/psycopg2/__init__.py", line 126, in connect
    dsn = _ext.make_dsn(dsn, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/psycopg2/extensions.py", line 175, in make_dsn
    parse_dsn(dsn)
django.db.utils.ProgrammingError: invalid dsn: invalid connection option "autocommit"

Somewhere the connection is not retained with the existing setup
My docker-compose.yml file

version: '3.4'
x-restart-policy: &restart_policy
  restart: unless-stopped
x-sentry-defaults: &sentry_defaults
  << : *restart_policy
  build:
    context: ./sentry
    args:
      - SENTRY_IMAGE
  image: sentry-onpremise-local
  environment:
    SENTRY_CONF: '/etc/sentry'
  env_file: .env
  environment:
    SENTRY_REDIS_HOST: **.***.172.61
    SENTRY_REDIS_POST: 6379
    SENTRY_POSTGRES_HOST: **.***.172.61
    SENTRY_POSTGRES_PORT: 5432
    SENTRY_DB_PASSWORD: secret
    SENTRY_DB_NAME: sentry
    SENTRY_DB_USER: sentry
    SENTRY_USE_SSL: 1
    SENTRY_SECRET_KEY: 9@g*_5u308vxtf@0=@u5=g^n1a=9q6uu&l8de%s@qw#o59^_1&
  volumes:
    - /home/docker/volumes/sentry:/var/lib/sentry/files
    - './sentry:/etc/sentry'
services:
  
  web:
    << : *sentry_defaults
    ports:
      - '9000:9000'
  cron:
    << : *sentry_defaults
    command: run cron
  worker:
    << : *sentry_defaults
    command: run worker
  sentry-cleanup:
    << : *sentry_defaults
    image: sentry-cleanup-onpremise-local
    build:
      context: ./cron
      args:
        BASE_IMAGE: 'sentry-onpremise-local'
    command: '"0 0 * * * gosu sentry sentry cleanup --days $SENTRY_EVENT_RETENTION_DAYS"'
volumes:
  sentry-data:
    external: true
  sentry-postgres:
    external: true

Any help here?

BYK · September 16, 2020, 9:29pm

Your compose file looks woefully out of date, please use the one from https://github.com/getsentry/onpremise/blob/21d1b6d42eee14d0de93ba77fe981f9f2a82447e/docker-compose.yml (or the one for the version you installed using the tags on the repo) and then add your customizations on top using the docker-compose.override.yml file if possible.

sunjha · September 16, 2020, 9:49pm

I have used the same file, just removed all the extra things which I won’t need.
Like kafka, snuba, nginx, redis, postgres.
I guess I would just need sentry web, cron and worked running on my machine. Also my redis and postgres are hosted on another server.

Could you please suggest if anything required got removed from the compose file?

BYK · September 16, 2020, 11:11pm

There is no way you can run any Sentry 10+ version w/o Kafka, Snuba, and Clickhouse and using Relay & Nginx are required for versions 20.7+. There is a reason why we have things in a certain way in this repo, I strongly encourage you to not remove services assuming you don’t need them.

sunjha · September 17, 2020, 4:08am

@BYK Sure. Will add those back and try again

sunjha · September 22, 2020, 8:28pm

@BYK So I didn’t changed anything this time and at first the setup worked and the UI was up and running but with empty DB.
I them added environment variables both in .env and docker-compose file as mentioned before but I am still seeing the same error. Sentry is not able to connect to the database. Any specific places where I should change in case I have to connect sentry to Database on another server?

I tried exporting the json file as mentioned here but its throwing this error

Traceback (most recent call last):
  File "/usr/local/bin/sentry", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/__init__.py", line 166, in main
    cli(prog_name=get_prog(), obj={}, max_content_width=100)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/decorators.py", line 30, in inner
    return ctx.invoke(f, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/commands/backup.py", line 15, in import_
    for obj in serializers.deserialize("json", src, stream=True, use_natural_keys=True):
  File "/usr/local/lib/python2.7/site-packages/django/core/serializers/json.py", line 88, in Deserializer
    six.reraise(DeserializationError, DeserializationError(e), sys.exc_info()[2])
  File "/usr/local/lib/python2.7/site-packages/django/core/serializers/json.py", line 82, in Deserializer
    for obj in PythonDeserializer(objects, **options):
  File "/usr/local/lib/python2.7/site-packages/django/core/serializers/python.py", line 129, in Deserializer
    field = Model._meta.get_field(field_name)
  File "/usr/local/lib/python2.7/site-packages/django/db/models/options.py", line 619, in get_field
    raise FieldDoesNotExist("%s has no field named '%s'" % (self.object_name, field_name))
django.core.serializers.base.DeserializationError: ContentType has no field named 'name'

BYK · September 28, 2020, 7:37pm

Any references to the host postgres in your config file need to be changed. See this section in our example config: https://github.com/getsentry/onpremise/blob/0a4b2ad033be4f2a99f22b317a01ec112bd2fb36/sentry/sentry.conf.example.py#L36-L45

sunjha · October 1, 2020, 9:49pm

@BYK, The last comment worked. But something still seems to be broken as I am getting this in logs

21:44:12 [WARNING] django.security.csrf: Forbidden (CSRF cookie not set.): /api/21/store/ (status_code=403 request=<WSGIRequest: POST u'/api/21/store/'>)

Any more changes to be done?

BYK · October 2, 2020, 4:36pm

You seem to be missing the nginx instance to route this traffic to relay

sunjha · October 2, 2020, 9:57pm

@BYK Seems like something went wrong with nginx but its working now. But I am now facing some kind of data related issue. On the UI I can see my old projects but opening any of them is giving this error

ProgrammingError: UndefinedColumn('column sentry_organizationaccessrequest.requester_id does not exist\nLINE 1: ..., "sentry_organizationaccessrequest"."member_id", "sentry_or...\n                                                             ^\n',)
SQL: SELECT "sentry_organizationaccessrequest"."id", "sentry_organizationaccessrequest"."team_id", "sentry_organizationaccessrequest"."member_id", "sentry_organizationaccessrequest"."requester_id", "sentry_team"."id", "sentry_team"."organization_id", "sentry_team"."slug", "sentry_team"."name", "sentry_team"."status", "sentry_team"."date_added", "sentry_organizationmember"."id", "sentry_organizationmember"."organization_id", "sentry_organizationmember"."user_id", "sentry_organizationmember"."email", "sentry_organizationmember"."role", "sentry_organizationmember"."flags", "sentry_organizationmember"."token", "sentry_organizationmember"."date_added", "sentry_organizationmember"."token_expires_at", "sentry_organizationmember"."has_global_access", "sentry_organizationmember"."inviter_id", "sentry_organizationmember"."invite_status", "sentry_organizationmember"."type", "auth_user"."password", "auth_user"."last_login", "auth_user"."id", "auth_user"."username", "auth_user"."first_name", "auth_user"."email", "auth_user"."is_staff", "auth_user"."is_active", "auth_user"."is_superuser", "auth_user"."is_managed", "auth_user"."is_sentry_app", "auth_user"."is_password_expired", "auth_user"."last_password_change", "auth_user"."flags", "auth_user"."session_nonce", "auth_user"."date_joined", "auth_user"."last_active" FROM "sentry_organizationaccessrequest" INNER JOIN "sentry_team" ON ("sentry_organizationaccessrequest"."team_id" = "sentry_team"."id") INNER JOIN "sentry_organizationmember" ON ("sentry_organizationaccessrequest"."member_id" = "sentry_organizationmember"."id") INNER JOIN "auth_user" ON ("sentry_organizationmember"."user_id" = "auth_user"."id") WHERE ("sentry_team"."organization_id" = %s AND "auth_user"."is_active" = %s)ProgrammingError: UndefinedColumn('column sentry_organizationaccessrequest.requester_id does not exist\nLINE 1: ..., "sentry_organizationaccessrequest"."member_id", "sentry_or...\n                                                             ^\n',)
SQL: SELECT "sentry_organizationaccessrequest"."id", "sentry_organizationaccessrequest"."team_id", "sentry_organizationaccessrequest"."member_id", "sentry_organizationaccessrequest"."requester_id", "sentry_team"."id", "sentry_team"."organization_id", "sentry_team"."slug", "sentry_team"."name", "sentry_team"."status", "sentry_team"."date_added", "sentry_organizationmember"."id", "sentry_organizationmember"."organization_id", "sentry_organizationmember"."user_id", "sentry_organizationmember"."email", "sentry_organizationmember"."role", "sentry_organizationmember"."flags", "sentry_organizationmember"."token", "sentry_organizationmember"."date_added", "sentry_organizationmember"."token_expires_at", "sentry_organizationmember"."has_global_access", "sentry_organizationmember"."inviter_id", "sentry_organizationmember"."invite_status", "sentry_organizationmember"."type", "auth_user"."password", "auth_user"."last_login", "auth_user"."id", "auth_user"."username", "auth_user"."first_name", "auth_user"."email", "auth_user"."is_staff", "auth_user"."is_active", "auth_user"."is_superuser", "auth_user"."is_managed", "auth_user"."is_sentry_app", "auth_user"."is_password_expired", "auth_user"."last_password_change", "auth_user"."flags", "auth_user"."session_nonce", "auth_user"."date_joined", "auth_user"."last_active" FROM "sentry_organizationaccessrequest" INNER JOIN "sentry_team" ON ("sentry_organizationaccessrequest"."team_id" = "sentry_team"."id") INNER JOIN "sentry_organizationmember" ON ("sentry_organizationaccessrequest"."member_id" = "sentry_organizationmember"."id") INNER JOIN "auth_user" ON ("sentry_organizationmember"."user_id" = "auth_user"."id") WHERE ("sentry_team"."organization_id" = %s AND "auth_user"."is_active" = %s)

And thats not the single table which is giving me missing column error, there are few more.

I tried running sentry upgrade from inside the docker container but it that as well is getting timed out while doing some operation with kafka

21:42:33 [WARNING] sentry.eventstream.kafka.backend: Could not publish message (error: KafkaError{code=_MSG_TIMED_OUT,val=-192,str="Local: Message timed out"}): <cimpl.Message object at 0x7f6b3c7b0750>
21:42:33 [WARNING] sentry.eventstream.kafka.backend: Could not publish message (error: KafkaError{code=_MSG_TIMED_OUT,val=-192,str="Local: Message timed out"}): <cimpl.Message object at 0x7f6b3c7b0750>
21:42:33 [WARNING] sentry.eventstream.kafka.backend: Could not publish message (error: KafkaError{code=_MSG_TIMED_OUT,val=-192,str="Local: Message timed out"}): <cimpl.Message object at 0x7f6b3c7b0750>
21:42:33 [WARNING] sentry.eventstream.kafka.backend: Could not publish message (error: KafkaError{code=_MSG_TIMED_OUT,val=-192,str="Local: Message timed out"}): <cimpl.Message object at 0x7f6b3c7b0750>
21:42:33 [WARNING] sentry.eventstream.kafka.backend: Could not publish message (error: KafkaError{code=_MSG_TIMED_OUT,val=-192,str="Local: Message timed out"}): <cimpl.Message object at 0x7f6b3c7b0750>
21:42:33 [WARNING] sentry.eventstream.kafka.backend: Could not publish message (error: KafkaError{code=_MSG_TIMED_OUT,val=-192,str="Local: Message timed out"}): <cimpl.Message object at 0x7f6b3c7b0750>
21:42:33 [WARNING] sentry.eventstream.kafka.backend: Could not publish message (error: KafkaError{code=_MSG_TIMED_OUT,val=-192,str="Local: Message timed out"}): <cimpl.Message object at 0x7f6b3c7b0750>
21:42:33 [WARNING] sentry.eventstream.kafka.backend: Could not publish message (error: KafkaError{code=_MSG_TIMED_OUT,val=-192,str="Local: Message timed out"}): <cimpl.Message object at 0x7f6b3c7b0750>
21:42:33 [WARNING] sentry.eventstream.kafka.backend: Could not publish message (error: KafkaError{code=_MSG_TIMED_OUT,val=-192,str="Local: Message timed out"}): <cimpl.Message object at 0x7f6b3c7b0750>
%5|1601675011.855|REQTMOUT|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/1001: Timed out ProduceRequest in flight (after 1628ms, timeout #0)
%5|1601675011.856|REQTMOUT|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/1001: Timed out ProduceRequest in flight (after 1621ms, timeout #1)
%5|1601675011.856|REQTMOUT|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/1001: Timed out ProduceRequest in flight (after 1615ms, timeout #2)
%5|1601675011.856|REQTMOUT|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/1001: Timed out ProduceRequest in flight (after 1604ms, timeout #3)
%5|1601675011.856|REQTMOUT|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/1001: Timed out ProduceRequest in flight (after 1595ms, timeout #4)
%4|1601675011.856|REQTMOUT|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/1001: Timed out 71 in-flight, 0 retry-queued, 0 out-queue, 0 partially-sent requests
%3|1601675011.856|FAIL|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/1001: 71 request(s) timed out: disconnect (after 56034ms in state UP)21:42:33 [WARNING] sentry.eventstream.kafka.backend: Could not publish message (error: KafkaError{code=_MSG_TIMED_OUT,val=-192,str="Local: Message timed out"}): <cimpl.Message object at 0x7f6b3c7b0750>
21:42:33 [WARNING] sentry.eventstream.kafka.backend: Could not publish message (error: KafkaError{code=_MSG_TIMED_OUT,val=-192,str="Local: Message timed out"}): <cimpl.Message object at 0x7f6b3c7b0750>
21:42:33 [WARNING] sentry.eventstream.kafka.backend: Could not publish message (error: KafkaError{code=_MSG_TIMED_OUT,val=-192,str="Local: Message timed out"}): <cimpl.Message object at 0x7f6b3c7b0750>
21:42:33 [WARNING] sentry.eventstream.kafka.backend: Could not publish message (error: KafkaError{code=_MSG_TIMED_OUT,val=-192,str="Local: Message timed out"}): <cimpl.Message object at 0x7f6b3c7b0750>
21:42:33 [WARNING] sentry.eventstream.kafka.backend: Could not publish message (error: KafkaError{code=_MSG_TIMED_OUT,val=-192,str="Local: Message timed out"}): <cimpl.Message object at 0x7f6b3c7b0750>
21:42:33 [WARNING] sentry.eventstream.kafka.backend: Could not publish message (error: KafkaError{code=_MSG_TIMED_OUT,val=-192,str="Local: Message timed out"}): <cimpl.Message object at 0x7f6b3c7b0750>
21:42:33 [WARNING] sentry.eventstream.kafka.backend: Could not publish message (error: KafkaError{code=_MSG_TIMED_OUT,val=-192,str="Local: Message timed out"}): <cimpl.Message object at 0x7f6b3c7b0750>
21:42:33 [WARNING] sentry.eventstream.kafka.backend: Could not publish message (error: KafkaError{code=_MSG_TIMED_OUT,val=-192,str="Local: Message timed out"}): <cimpl.Message object at 0x7f6b3c7b0750>
21:42:33 [WARNING] sentry.eventstream.kafka.backend: Could not publish message (error: KafkaError{code=_MSG_TIMED_OUT,val=-192,str="Local: Message timed out"}): <cimpl.Message object at 0x7f6b3c7b0750>
%5|1601675011.855|REQTMOUT|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/1001: Timed out ProduceRequest in flight (after 1628ms, timeout #0)
%5|1601675011.856|REQTMOUT|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/1001: Timed out ProduceRequest in flight (after 1621ms, timeout #1)
%5|1601675011.856|REQTMOUT|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/1001: Timed out ProduceRequest in flight (after 1615ms, timeout #2)
%5|1601675011.856|REQTMOUT|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/1001: Timed out ProduceRequest in flight (after 1604ms, timeout #3)
%5|1601675011.856|REQTMOUT|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/1001: Timed out ProduceRequest in flight (after 1595ms, timeout #4)
%4|1601675011.856|REQTMOUT|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/1001: Timed out 71 in-flight, 0 retry-queued, 0 out-queue, 0 partially-sent requests
%3|1601675011.856|FAIL|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/1001: 71 request(s) timed out: disconnect (after 56034ms in state UP)

Please help

BYK · October 5, 2020, 9:05pm

Your whole instance seems to be in an unhealthy state. It is hard to determine what exactly is going wrong but seems like Kafka is not working properly, which is required to finish the migrations.

And you should not be running migrations manually inside the container. The install.sh script handles these steps properly and you should be following that.

sunjha · October 7, 2020, 8:55pm

@BYK So I kind of faced too many issues migrating over to the latest version from 9.1.2 so I rolled everything back. I though faced issues again with the database schema which got changed. So I took a dump from old one, created a new postgres instance and pushed it on it(only the insert queries)
The system works now but I can’t see any users on the UI. I can the data in auth_user table. Is there any was I can get that data back so that users can login?

BYK · October 12, 2020, 10:13am

If you backed up the sentry-postgres volume, you should have no issues getting back your user and other config data.

system · December 9, 2020, 8:04pm

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
CSRF Verification Failed since Sentry upgrade	6	5536	July 30, 2020
Password authentication failed for user "postgres" On-Premise	2	4058	February 15, 2017
CSRF Verification Failed On-Premise	6	14556	August 10, 2021
Migration Error from Sentry8.9.0(Python) to Sentry9.0(docker-compose) On-Premise	0	1014	December 27, 2018
Issue with CSRF token after sentry upgrade On-Premise	4	7936	October 8, 2020

Upgrading to the latest version from 9.1.2

Related Topics