Upgrade Sentry 20.9 to 21.3

On-Premise
1

I am getting this on the install.sh . Any recommendations.
@BYK

Running migrations:
  Applying sentry.0101_backfill_file_type_on_event_attachment...Traceback (most recent call last):
  File "/usr/local/bin/sentry", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/site-packages/sentry/runner/__init__.py", line 164, in main
    cli(prog_name=get_prog(), obj={}, max_content_width=100)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/sentry/runner/decorators.py", line 28, in inner
    return ctx.invoke(f, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/sentry/runner/commands/upgrade.py", line 171, in upgrade
    _upgrade(not noinput, traceback, verbosity, not no_repair, with_nodestore)
  File "/usr/local/lib/python3.6/site-packages/sentry/runner/commands/upgrade.py", line 127, in _upgrade
    ignore_ghost_migrations=True,
  File "/usr/local/lib/python3.6/site-packages/django/core/management/__init__.py", line 131, in call_command
    return command.execute(*args, **defaults)
  File "/usr/local/lib/python3.6/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.6/site-packages/django/core/management/commands/migrate.py", line 204, in handle
    fake_initial=fake_initial,
  File "/usr/local/lib/python3.6/site-packages/django/db/migrations/executor.py", line 115, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python3.6/site-packages/django/db/migrations/executor.py", line 145, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python3.6/site-packages/sentry/new_migrations/monkey/executor.py", line 23, in apply_migration
    return super().apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python3.6/site-packages/django/db/migrations/executor.py", line 244, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/usr/local/lib/python3.6/site-packages/django/db/migrations/migration.py", line 129, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/usr/local/lib/python3.6/site-packages/django/db/migrations/operations/special.py", line 193, in database_forwards
    self.code(from_state.apps, schema_editor)
  File "/usr/local/lib/python3.6/site-packages/sentry/migrations/0101_backfill_file_type_on_event_attachment.py", line 21, in backfill_file_type
    file = File.objects.get(event_attachment.file_id)
  File "/usr/local/lib/python3.6/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/django/db/models/query.py", line 371, in get
    clone = self.filter(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/django/db/models/query.py", line 787, in filter
    return self._filter_or_exclude(False, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/django/db/models/query.py", line 805, in _filter_or_exclude
    clone.query.add_q(Q(*args, **kwargs))
  File "/usr/local/lib/python3.6/site-packages/django/db/models/sql/query.py", line 1250, in add_q
    clause, _ = self._add_q(q_object, self.used_aliases)
  File "/usr/local/lib/python3.6/site-packages/django/db/models/sql/query.py", line 1276, in _add_q
    allow_joins=allow_joins, split_subq=split_subq,
  File "/usr/local/lib/python3.6/site-packages/django/db/models/sql/query.py", line 1151, in build_filter
    arg, value = filter_expr
TypeError: 'int' object is not iterable
An error occurred, caught SIGERR on line 332
3

Can you share more details with full install logs?

4

Here is the full log

:arrow_forward: Defining variables and helpers …

▶ Parsing command line ...

▶ Setting up error handling ...

▶ Checking minimum requirements ...

▶ Creating volumes for persistent storage ...
Created sentry-data.
Created sentry-postgres.
Created sentry-redis.
Created sentry-zookeeper.
Created sentry-kafka.
Created sentry-clickhouse.
Created sentry-symbolicator.

▶ Ensuring files from examples ...
sentry/sentry.conf.py already exists, skipped creation.
sentry/config.yml already exists, skipped creation.
sentry/requirements.txt already exists, skipped creation.
Creating symbolicator/config.yml...
relay/config.yml already exists, skipped creation.

▶ Generating secret key ...
Secret key written to sentry/config.yml

▶ Replacing TSDB ...

▶ Fetching and updating Docker images ...
Some service image(s) must be built from source by running:
    docker-compose build sentry-cleanup snuba-cleanup symbolicator-cleanup
nightly: Pulling from getsentry/sentry
Digest: sha256:498f22cc0f09144615f0969748e85d3226cd50c972456c56fa0c82f63f3da3bd
Status: Image is up to date for getsentry/sentry:nightly
docker.io/getsentry/sentry:nightly

▶ Building and tagging Docker images ...

smtp uses an image, skipping
memcached uses an image, skipping
redis uses an image, skipping
postgres uses an image, skipping
zookeeper uses an image, skipping
kafka uses an image, skipping
clickhouse uses an image, skipping
geoipupdate uses an image, skipping
snuba-api uses an image, skipping
snuba-consumer uses an image, skipping
snuba-outcomes-consumer uses an image, skipping
snuba-sessions-consumer uses an image, skipping
snuba-transactions-consumer uses an image, skipping
snuba-replacer uses an image, skipping
snuba-subscription-consumer-events uses an image, skipping
snuba-subscription-consumer-transactions uses an image, skipping
symbolicator uses an image, skipping
web uses an image, skipping
cron uses an image, skipping
worker1 uses an image, skipping
worker2 uses an image, skipping
worker3 uses an image, skipping
worker4 uses an image, skipping
worker5 uses an image, skipping
worker6 uses an image, skipping
worker uses an image, skipping
ingest-consumer uses an image, skipping
post-process-forwarder uses an image, skipping
subscription-consumer-events uses an image, skipping
subscription-consumer-transactions uses an image, skipping
relay uses an image, skipping
nginx uses an image, skipping
Building snuba-cleanup
Step 1/5 : ARG BASE_IMAGE
Step 2/5 : FROM ${BASE_IMAGE}
 ---> 4065de1365eb
Step 3/5 : RUN apt-get update && apt-get install -y --no-install-recommends cron &&     rm -r /var/lib/apt/lists/*
 ---> Running in 676b77afed5b
Get:1 http://deb.debian.org/debian buster InRelease [122 kB]
Get:2 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]
Get:3 http://deb.debian.org/debian buster-updates InRelease [51.9 kB]
Get:4 http://security.debian.org/debian-security buster/updates/main amd64 Packages [268 kB]
Get:5 http://deb.debian.org/debian buster/main amd64 Packages [7907 kB]
Get:6 http://deb.debian.org/debian buster-updates/main amd64 Packages [9504 B]
Fetched 8423 kB in 2s (4499 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
  lsb-base sensible-utils
Suggested packages:
  anacron logrotate checksecurity
Recommended packages:
  default-mta | mail-transport-agent
The following NEW packages will be installed:
  cron lsb-base sensible-utils
0 upgraded, 3 newly installed, 0 to remove and 2 not upgraded.
Need to get 143 kB of archives.
After this operation, 383 kB of additional disk space will be used.
Get:1 http://deb.debian.org/debian buster/main amd64 sensible-utils all 0.0.12 [15.8 kB]
Get:2 http://deb.debian.org/debian buster/main amd64 lsb-base all 10.2019051400 [28.4 kB]
Get:3 http://deb.debian.org/debian buster/main amd64 cron amd64 3.0pl1-134+deb10u1 [99.0 kB]
e[91mdebconf: delaying package configuration, since apt-utils is not installed
e[0mFetched 143 kB in 0s (754 kB/s)
Selecting previously unselected package sensible-utils.
(Reading database ... 
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 6840 files and directories currently installed.)
Preparing to unpack .../sensible-utils_0.0.12_all.deb ...
Unpacking sensible-utils (0.0.12) ...
Selecting previously unselected package lsb-base.
Preparing to unpack .../lsb-base_10.2019051400_all.deb ...
Unpacking lsb-base (10.2019051400) ...
Selecting previously unselected package cron.
Preparing to unpack .../cron_3.0pl1-134+deb10u1_amd64.deb ...
Unpacking cron (3.0pl1-134+deb10u1) ...
Setting up lsb-base (10.2019051400) ...
Setting up sensible-utils (0.0.12) ...
Setting up cron (3.0pl1-134+deb10u1) ...
Adding group `crontab' (GID 101) ...
Done.
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of start.
Removing intermediate container 676b77afed5b
 ---> cafb3959edfe
Step 4/5 : COPY entrypoint.sh /entrypoint.sh
 ---> f501003ad52f
Step 5/5 : ENTRYPOINT ["/entrypoint.sh"]
 ---> Running in e01bbe9ea998
Removing intermediate container e01bbe9ea998
 ---> 595dfc6df1a1

Successfully built 595dfc6df1a1
Successfully tagged snuba-cleanup-onpremise-local:latest
Building symbolicator-cleanup
Step 1/5 : ARG BASE_IMAGE
Step 2/5 : FROM ${BASE_IMAGE}
 ---> 559a77d1d880
Step 3/5 : RUN apt-get update && apt-get install -y --no-install-recommends cron &&     rm -r /var/lib/apt/lists/*
 ---> Running in af579b70e671
Get:1 http://security.debian.org/debian-security stretch/updates InRelease [53.0 kB]
Ign:2 http://deb.debian.org/debian stretch InRelease
Get:3 http://deb.debian.org/debian stretch-updates InRelease [93.6 kB]
Get:4 http://security.debian.org/debian-security stretch/updates/main amd64 Packages [656 kB]
Get:5 http://deb.debian.org/debian stretch Release [118 kB]
Get:6 http://deb.debian.org/debian stretch Release.gpg [2410 B]
Get:7 http://deb.debian.org/debian stretch-updates/main amd64 Packages [2596 B]
Get:8 http://deb.debian.org/debian stretch/main amd64 Packages [7080 kB]
Fetched 8006 kB in 1s (4294 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
Suggested packages:
  anacron logrotate checksecurity
Recommended packages:
  exim4 | postfix | mail-transport-agent
The following NEW packages will be installed:
  cron
0 upgraded, 1 newly installed, 0 to remove and 2 not upgraded.
Need to get 95.4 kB of archives.
After this operation, 257 kB of additional disk space will be used.
Get:1 http://deb.debian.org/debian stretch/main amd64 cron amd64 3.0pl1-128+deb9u1 [95.4 kB]
e[91mdebconf: delaying package configuration, since apt-utils is not installed
e[0mFetched 95.4 kB in 0s (813 kB/s)
Selecting previously unselected package cron.
(Reading database ... 
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 6661 files and directories currently installed.)
Preparing to unpack .../cron_3.0pl1-128+deb9u1_amd64.deb ...
Unpacking cron (3.0pl1-128+deb9u1) ...
Setting up cron (3.0pl1-128+deb9u1) ...
Adding group `crontab' (GID 101) ...
Done.
update-rc.d: warning: start and stop actions are no longer supported; falling back to defaults
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of start.
Removing intermediate container af579b70e671
 ---> 15bc2f09f45f
Step 4/5 : COPY entrypoint.sh /entrypoint.sh
 ---> fd0b8aaafcf9
Step 5/5 : ENTRYPOINT ["/entrypoint.sh"]
 ---> Running in aa1f9ae6517d
Removing intermediate container aa1f9ae6517d
 ---> a7e3826b7d89

Successfully built a7e3826b7d89
Successfully tagged symbolicator-cleanup-onpremise-local:latest
Building sentry-cleanup
Step 1/5 : ARG BASE_IMAGE
Step 2/5 : FROM ${BASE_IMAGE}
 ---> 66a9b52b96c2
Step 3/5 : RUN apt-get update && apt-get install -y --no-install-recommends cron &&     rm -r /var/lib/apt/lists/*
 ---> Running in 6ba1b51f26fe
Get:1 http://deb.debian.org/debian buster InRelease [122 kB]
Get:2 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]
Get:3 http://deb.debian.org/debian buster-updates InRelease [51.9 kB]
Get:4 http://security.debian.org/debian-security buster/updates/main amd64 Packages [268 kB]
Get:5 http://deb.debian.org/debian buster/main amd64 Packages [7907 kB]
Get:6 http://deb.debian.org/debian buster-updates/main amd64 Packages [9504 B]
Fetched 8423 kB in 2s (4159 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
  lsb-base sensible-utils
Suggested packages:
  anacron logrotate checksecurity
Recommended packages:
  default-mta | mail-transport-agent
The following NEW packages will be installed:
  cron lsb-base sensible-utils
0 upgraded, 3 newly installed, 0 to remove and 2 not upgraded.
Need to get 143 kB of archives.
After this operation, 383 kB of additional disk space will be used.
Get:1 http://deb.debian.org/debian buster/main amd64 sensible-utils all 0.0.12 [15.8 kB]
Get:2 http://deb.debian.org/debian buster/main amd64 lsb-base all 10.2019051400 [28.4 kB]
Get:3 http://deb.debian.org/debian buster/main amd64 cron amd64 3.0pl1-134+deb10u1 [99.0 kB]
e[91mdebconf: delaying package configuration, since apt-utils is not installed
e[0mFetched 143 kB in 0s (1232 kB/s)
Selecting previously unselected package sensible-utils.
(Reading database ... 
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 11935 files and directories currently installed.)
Preparing to unpack .../sensible-utils_0.0.12_all.deb ...
Unpacking sensible-utils (0.0.12) ...
Selecting previously unselected package lsb-base.
Preparing to unpack .../lsb-base_10.2019051400_all.deb ...
Unpacking lsb-base (10.2019051400) ...
Selecting previously unselected package cron.
Preparing to unpack .../cron_3.0pl1-134+deb10u1_amd64.deb ...
Unpacking cron (3.0pl1-134+deb10u1) ...
Setting up lsb-base (10.2019051400) ...
Setting up sensible-utils (0.0.12) ...
Setting up cron (3.0pl1-134+deb10u1) ...
Adding group `crontab' (GID 101) ...
Done.
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of start.
Removing intermediate container 6ba1b51f26fe
 ---> dc47801622d4
Step 4/5 : COPY entrypoint.sh /entrypoint.sh
 ---> 46dcc4c200b6
Step 5/5 : ENTRYPOINT ["/entrypoint.sh"]
 ---> Running in 0ba1d39bfbf2
Removing intermediate container 0ba1d39bfbf2
 ---> 6da26bdac5a1

Successfully built 6da26bdac5a1
Successfully tagged sentry-cleanup-onpremise-local:latest

Docker images built.

▶ Turning things off ...
Removing network onpremise_default
Network onpremise_default not found.
Removing network sentry_onpremise_default
Network sentry_onpremise_default not found.

▶ Setting up Zookeeper ...
Creating network "sentry_onpremise_default" with the default driver

▶ Bootstrapping and migrating Snuba ...
Creating sentry_onpremise_redis_1 ... 
Creating sentry_onpremise_clickhouse_1 ... 
Creating sentry_onpremise_zookeeper_1  ... 
Creating sentry_onpremise_redis_1      ... done
Creating sentry_onpremise_zookeeper_1  ... done
Creating sentry_onpremise_kafka_1      ... 
Creating sentry_onpremise_clickhouse_1 ... done
Creating sentry_onpremise_kafka_1      ... done
+ '[' b = - ']'
+ snuba bootstrap --help
+ set -- snuba bootstrap --no-migrate --force
+ set gosu snuba snuba bootstrap --no-migrate --force
+ exec gosu snuba snuba bootstrap --no-migrate --force
%3|1614624408.607|FAIL|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.19.0.5:9092 failed: Connection refused (after 24ms in state CONNECT)
%3|1614624409.583|FAIL|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.19.0.5:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
2021-03-01 18:46:49,583 Connection to Kafka failed (attempt 0)
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 55, in bootstrap
    client.list_topics(timeout=1)
cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str="Failed to get metadata: Local: Broker transport failure"}
%3|1614624410.585|FAIL|rdkafka#producer-2| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.19.0.5:9092 failed: Connection refused (after 0ms in state CONNECT)
%3|1614624411.585|FAIL|rdkafka#producer-2| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.19.0.5:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
2021-03-01 18:46:51,586 Connection to Kafka failed (attempt 1)
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 55, in bootstrap
    client.list_topics(timeout=1)
cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str="Failed to get metadata: Local: Broker transport failure"}
%3|1614624412.606|FAIL|rdkafka#producer-3| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.19.0.5:9092 failed: Connection refused (after 18ms in state CONNECT)
%3|1614624413.592|FAIL|rdkafka#producer-3| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.19.0.5:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
2021-03-01 18:46:53,592 Connection to Kafka failed (attempt 2)
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 55, in bootstrap
    client.list_topics(timeout=1)
cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str="Failed to get metadata: Local: Broker transport failure"}
%3|1614624414.599|FAIL|rdkafka#producer-4| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.19.0.5:9092 failed: Connection refused (after 0ms in state CONNECT)
%3|1614624415.603|FAIL|rdkafka#producer-4| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.19.0.5:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
2021-03-01 18:46:55,600 Connection to Kafka failed (attempt 3)
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 55, in bootstrap
    client.list_topics(timeout=1)
cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str="Failed to get metadata: Local: Broker transport failure"}
%3|1614624416.622|FAIL|rdkafka#producer-5| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.19.0.5:9092 failed: Connection refused (after 0ms in state CONNECT)
%3|1614624417.621|FAIL|rdkafka#producer-5| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.19.0.5:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
2021-03-01 18:46:57,624 Connection to Kafka failed (attempt 4)
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 55, in bootstrap
    client.list_topics(timeout=1)
cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str="Failed to get metadata: Local: Broker transport failure"}
%3|1614624418.625|FAIL|rdkafka#producer-6| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.19.0.5:9092 failed: Connection refused (after 0ms in state CONNECT)
2021-03-01 18:46:59,625 Connection to Kafka failed (attempt 5)
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 55, in bootstrap
    client.list_topics(timeout=1)
cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str="Failed to get metadata: Local: Broker transport failure"}
2021-03-01 18:47:01,044 Failed to create topic cdc
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 91, in bootstrap
    future.result()
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 439, in result
    return self.__get_result()
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
    raise self._exception
cimpl.KafkaException: KafkaError{code=TOPIC_ALREADY_EXISTS,val=36,str="Topic 'cdc' already exists."}
2021-03-01 18:47:01,045 Failed to create topic events
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 91, in bootstrap
    future.result()
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
    raise self._exception
cimpl.KafkaException: KafkaError{code=TOPIC_ALREADY_EXISTS,val=36,str="Topic 'events' already exists."}
2021-03-01 18:47:01,045 Failed to create topic event-replacements
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 91, in bootstrap
    future.result()
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
    raise self._exception
cimpl.KafkaException: KafkaError{code=TOPIC_ALREADY_EXISTS,val=36,str="Topic 'event-replacements' already exists."}
2021-03-01 18:47:01,045 Failed to create topic snuba-commit-log
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 91, in bootstrap
    future.result()
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
    raise self._exception
cimpl.KafkaException: KafkaError{code=TOPIC_ALREADY_EXISTS,val=36,str="Topic 'snuba-commit-log' already exists."}
2021-03-01 18:47:01,046 Failed to create topic outcomes
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 91, in bootstrap
    future.result()
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
    raise self._exception
cimpl.KafkaException: KafkaError{code=TOPIC_ALREADY_EXISTS,val=36,str="Topic 'outcomes' already exists."}
2021-03-01 18:47:01,046 Failed to create topic ingest-sessions
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 91, in bootstrap
    future.result()
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
    raise self._exception
cimpl.KafkaException: KafkaError{code=TOPIC_ALREADY_EXISTS,val=36,str="Topic 'ingest-sessions' already exists."}
Starting sentry_onpremise_redis_1 ... 
Starting sentry_onpremise_zookeeper_1 ... 
Starting sentry_onpremise_clickhouse_1 ... 
Starting sentry_onpremise_clickhouse_1 ... done
Starting sentry_onpremise_zookeeper_1  ... done
Starting sentry_onpremise_kafka_1      ... 
Starting sentry_onpremise_redis_1      ... done
Starting sentry_onpremise_kafka_1      ... done
+ '[' m = - ']'
+ snuba migrations --help
+ set -- snuba migrations migrate --force
+ set gosu snuba snuba migrations migrate --force
+ exec gosu snuba snuba migrations migrate --force
Finished running migrations

▶ Creating additional Kafka topics ...

▶ Ensuring proper PostgreSQL version ...

▶ Setting up database ...
Starting sentry_onpremise_clickhouse_1 ... 
Creating sentry_onpremise_symbolicator_1 ... 
Starting sentry_onpremise_clickhouse_1   ... done
Starting sentry_onpremise_zookeeper_1    ... 
Starting sentry_onpremise_zookeeper_1    ... done
Creating sentry_onpremise_memcached_1    ... 
Starting sentry_onpremise_redis_1        ... 
Starting sentry_onpremise_redis_1        ... done
Starting sentry_onpremise_kafka_1        ... 
Starting sentry_onpremise_kafka_1        ... done
Creating sentry_onpremise_postgres_1     ... 
Creating sentry_onpremise_snuba-subscription-consumer-transactions_1 ... 
Creating sentry_onpremise_smtp_1                                     ... 
Creating sentry_onpremise_snuba-api_1                                ... 
Creating sentry_onpremise_snuba-outcomes-consumer_1                  ... 
Creating sentry_onpremise_snuba-subscription-consumer-events_1       ... 
Creating sentry_onpremise_snuba-sessions-consumer_1                  ... 
Creating sentry_onpremise_snuba-transactions-consumer_1              ... 
Creating sentry_onpremise_snuba-consumer_1                           ... 
Creating sentry_onpremise_snuba-replacer_1                           ... 
Creating sentry_onpremise_symbolicator_1                             ... done
Creating sentry_onpremise_memcached_1                                ... done
Creating sentry_onpremise_snuba-replacer_1                           ... done
Creating sentry_onpremise_smtp_1                                     ... done
Creating sentry_onpremise_postgres_1                                 ... done
Creating sentry_onpremise_snuba-consumer_1                           ... done
Creating sentry_onpremise_snuba-transactions-consumer_1              ... done
Creating sentry_onpremise_snuba-subscription-consumer-transactions_1 ... done
Creating sentry_onpremise_snuba-api_1                                ... done
Creating sentry_onpremise_snuba-outcomes-consumer_1                  ... done
Creating sentry_onpremise_snuba-sessions-consumer_1                  ... done
Creating sentry_onpremise_snuba-subscription-consumer-events_1       ... done
Installing additional dependencies...

18:48:12 [WARNING] sentry.utils.geo: Error opening GeoIP database: /geoip/GeoLite2-City.mmdb
18:48:12 [WARNING] sentry.utils.geo: Error opening GeoIP database in Rust: /geoip/GeoLite2-City.mmdb
18:48:17 [INFO] sentry.plugins.github: apps-not-configured
* Unknown config option found: 'postprocess.use-cache-key'
* Unknown config option found: 'slack.legacy-app'
e[36;1mOperations to perform:e[0m
e[1m  Apply all migrations: e[0madmin, auth, contenttypes, jira_ac, nodestore, sentry, sessions, sites, social_auth
e[36;1mRunning migrations:e[0m
  Applying nodestore.0002_nodestore_no_dictfield...e[32;1m OKe[0m
  Applying sentry.0100_file_type_on_event_attachment...e[32;1m OKe[0m
  Applying sentry.0101_backfill_file_type_on_event_attachment...Traceback (most recent call last):
  File "/usr/local/bin/sentry", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/site-packages/sentry/runner/__init__.py", line 164, in main
    cli(prog_name=get_prog(), obj={}, max_content_width=100)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/sentry/runner/decorators.py", line 28, in inner
    return ctx.invoke(f, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/sentry/runner/commands/upgrade.py", line 171, in upgrade
    _upgrade(not noinput, traceback, verbosity, not no_repair, with_nodestore)
  File "/usr/local/lib/python3.6/site-packages/sentry/runner/commands/upgrade.py", line 127, in _upgrade
    ignore_ghost_migrations=True,
  File "/usr/local/lib/python3.6/site-packages/django/core/management/__init__.py", line 131, in call_command
    return command.execute(*args, **defaults)
  File "/usr/local/lib/python3.6/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.6/site-packages/django/core/management/commands/migrate.py", line 204, in handle
    fake_initial=fake_initial,
  File "/usr/local/lib/python3.6/site-packages/django/db/migrations/executor.py", line 115, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python3.6/site-packages/django/db/migrations/executor.py", line 145, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python3.6/site-packages/sentry/new_migrations/monkey/executor.py", line 23, in apply_migration
    return super().apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python3.6/site-packages/django/db/migrations/executor.py", line 244, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/usr/local/lib/python3.6/site-packages/django/db/migrations/migration.py", line 129, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/usr/local/lib/python3.6/site-packages/django/db/migrations/operations/special.py", line 193, in database_forwards
    self.code(from_state.apps, schema_editor)
  File "/usr/local/lib/python3.6/site-packages/sentry/migrations/0101_backfill_file_type_on_event_attachment.py", line 21, in backfill_file_type
    file = File.objects.get(event_attachment.file_id)
  File "/usr/local/lib/python3.6/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/django/db/models/query.py", line 371, in get
    clone = self.filter(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/django/db/models/query.py", line 787, in filter
    return self._filter_or_exclude(False, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/django/db/models/query.py", line 805, in _filter_or_exclude
    clone.query.add_q(Q(*args, **kwargs))
  File "/usr/local/lib/python3.6/site-packages/django/db/models/sql/query.py", line 1250, in add_q
    clause, _ = self._add_q(q_object, self.used_aliases)
  File "/usr/local/lib/python3.6/site-packages/django/db/models/sql/query.py", line 1276, in _add_q
    allow_joins=allow_joins, split_subq=split_subq,
  File "/usr/local/lib/python3.6/site-packages/django/db/models/sql/query.py", line 1151, in build_filter
    arg, value = filter_expr
TypeError: 'int' object is not iterable
An error occurred, caught SIGERR on line 332
Cleaning up...
5

@BYK , i assumed the Kafka topics already exist , so may be thats expected ,
Apply migrations fail.

The other thing , may be if it needs to be documented , is removing the kafka and zookeeper volumes and rerunning the install. Not sure if it will help

6

@amit1 - Kafka errors are fine, I actually have a task to make that step less noisy as those errors are fine (and expected).

Seems like that failing migration has a bug in it so thanks for reporting! I’ve submitted fix(migrations): Fix backfill_file_type_on_event_attachement by BYK · Pull Request #24236 · getsentry/sentry · GitHub to fix the issue so you can follow the progress from there.

7

@BYK , i see the changes have been merged to master. Should i give it another shot to upgrade my instance?

8

Yup if you are okay using a nightly build. 21.3.0 will be released on March 15th if you want to wait till then. You should be able to upgrade to that from the current nightlies too.

9

Oh. Really need my instance upgraded and running @BYK. I will try the nightly build with the latest merge

1 Like
10

Hi @BYK ,
Upgrade worked this time.

here is a upgrade run INFO

  warnings.warn(DeprecatedSettingWarning(options_mapper[k], "SENTRY_OPTIONS['%s']" % k))
00:10:39 [WARNING] sentry.utils.geo: Error opening GeoIP database: /geoip/GeoLite2-City.mmdb
00:10:39 [WARNING] sentry.utils.geo: Error opening GeoIP database in Rust: /geoip/GeoLite2-City.mmdb
00:10:43 [INFO] sentry.plugins.github: apps-not-configured
* Unknown config option found: 'slack.legacy-app'
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, jira_ac, nodestore, sentry, sessions, sites, social_auth
Running migrations:
  Applying sentry.0101_backfill_file_type_on_event_attachment... OK
  Applying sentry.0102_collect_relay_analytics... OK
  Applying sentry.0103_project_has_alert_filters... OK
  Applying sentry.0104_collect_relay_public_key_usage... OK
  Applying sentry.0105_remove_nullability_of_event_attachment_type... OK
  Applying sentry.0106_service_hook_project_id_nullable... OK
  Applying sentry.0107_remove_spaces_from_slugs... OK
  Applying sentry.0108_update_fileblob_action... OK
  Applying sentry.0109_sentry_app_creator... OK
  Applying sentry.0110_sentry_app_creator_backill... OK
  Applying sentry.0111_snuba_query_event_type... OK
  Applying sentry.0112_groupinboxmodel... OK
  Applying sentry.0113_add_repositoryprojectpathconfig... OK
  Applying sentry.0114_add_unhandled_savedsearch... OK
  Applying sentry.0115_add_checksum_to_debug_file... OK
  Applying sentry.0116_backfill_debug_file_checksum... OK
  Applying sentry.0117_dummy-activityupdate... OK
  Applying sentry.0118_backfill_snuba_query_event_types... OK
  Applying sentry.0119_fix_set_none... OK
  Applying sentry.0120_commit_author_charfield... OK
GroupInbox: 100% |#                                             | ETA:  --:--:--
 OK
  Applying sentry.0122_add_release_status... OK
  Applying sentry.0123_groupinbox_addprojandorg... OK
  Applying sentry.0124_add_release_status_model... OK
  Applying sentry.0125_add_platformexternalissue_project_id... OK
  Applying sentry.0126_make_platformexternalissue_group_id_flexfk... OK
  Applying sentry.0127_backfill_platformexternalissue_project_id... OK
  Applying sentry.0128_change_dashboards... OK
  Applying sentry.0129_remove_dashboard_keys... OK
  Applying sentry.0130_remove_old_widget_models... OK
  Applying sentry.0131_drop_widget_tables... OK
  Applying sentry.0132_groupownermodel... OK
  Applying sentry.0133_dashboard_delete_object_status... OK
  Applying sentry.0134_dashboard_drop_object_status_column... OK
  Applying sentry.0135_removinguniquegroupownerconstraint... OK
Organizations: 100% |############################################| Time: 0:00:00
 OK
  Applying sentry.0137_dashboard_widget_interval... OK
  Applying sentry.0138_widget_query_remove_interval... OK
  Applying sentry.0139_remove_widgetquery_interval... OK
  Applying sentry.0140_subscription_checker... OK
  Applying sentry.0141_remove_widget_constraints... OK
  Applying sentry.0142_add_dashboard_tombstone... OK
  Applying sentry.0143_add_alerts_integrationfeature... OK
  Applying sentry.0144_add_publish_request_inprogress_status... OK
  Applying sentry.0145_rename_alert_rule_feature... OK
Organizations: 100% |############################################| Time: 0:00:00
 OK
  Applying sentry.0147_add_groupinbox_date_added_index... OK
  Applying sentry.0148_group_id_bigint... OK
  Applying sentry.0149_bigint... OK
  Applying sentry.0150_remove_userreport_eventattachment_constraints... OK
  Applying sentry.0151_add_world_map_dashboard_widget_type... OK
  Applying sentry.0152_remove_slack_workspace_orgintegrations... OK
  Applying sentry.0153_add_big_number_dashboard_widget_type... OK
  Applying sentry.0154_groupedmessage_inbox_sort... OK
  Applying sentry.0155_add_dashboard_query_orderby... OK
  Applying sentry.0156_add_mark_reviewed_activity... OK
  Applying sentry.0157_make_repositoryprojectpathconfig_organization_integration_nullable... OK
  Applying sentry.0158_create_externalteam_table... OK
  Applying sentry.0159_create_externaluser_table... OK
  Applying sentry.0160_create_projectcodeowners_table... OK
  Applying sentry.0161_add_saved_search_sort... OK
Saved Searchs: 100% |############################################| Time: 0:00:00
 OK
  Applying sentry.0163_add_organizationmember_and_external_name... OK
  Applying sentry.0164_add_protect_on_delete_codeowners... OK
  Applying sentry.0165_metric_alerts_fix_group_ids... OK
  Applying sentry.0166_create_notificationsetting_table... OK
  Applying sentry.0167_rm_organization_integration_from_projectcodeowners... OK
  Applying sentry.0168_demo_orgs_users... OK
  Applying sentry.0169_delete_organization_integration_from_projectcodeowners... OK
# This file is just Python, with a touch of Django which means
# you can inherit and tweak settings to your hearts content.

from sentry.conf.server import *  # NOQA


# Generously adapted from pynetlinux: https://git.io/JJmga
def get_internal_network():
    import ctypes
    import fcntl
    import math
    import socket
    import struct

    iface = b"eth0"
    sockfd = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    ifreq = struct.pack(b"16sH14s", iface, socket.AF_INET, b"\x00" * 14)

    try:
        ip = struct.unpack(
            b"!I", struct.unpack(b"16sH2x4s8x", fcntl.ioctl(sockfd, 0x8915, ifreq))[2]
        )[0]
        netmask = socket.ntohl(
            struct.unpack(b"16sH2xI8x", fcntl.ioctl(sockfd, 0x891B, ifreq))[2]
        )
    except IOError:
        return ()
    base = socket.inet_ntoa(struct.pack(b"!I", ip & netmask))
    netmask_bits = 32 - int(round(math.log(ctypes.c_uint32(~netmask).value + 1, 2), 1))
    return "{0:s}/{1:d}".format(base, netmask_bits)


INTERNAL_SYSTEM_IPS = (get_internal_network(),)


DATABASES = {
    "default": {
        "ENGINE": "sentry.db.postgres",
        "NAME": "postgres",
        "USER": "postgres",
        "PASSWORD": "",
        "HOST": "postgres",
        "PORT": "",
    }
}

# You should not change this setting after your database has been created
# unless you have altered all schemas first
SENTRY_USE_BIG_INTS = True

# If you're expecting any kind of real traffic on Sentry, we highly recommend
# configuring the CACHES and Redis settings

###########
# General #
###########

# Instruct Sentry that this install intends to be run by a single organization
# and thus various UI optimizations should be enabled.
SENTRY_SINGLE_ORGANIZATION = True

SENTRY_OPTIONS["system.event-retention-days"] = int(
    env("SENTRY_EVENT_RETENTION_DAYS", "30")
)
SENTRY_OPTIONS["system.url-prefix"] = "https://retail-sentry.apple.com"
/GITHUB
# header and enable the settings below
  Applying sentry.0170_actor_introduction... OK
# This file is just Python, with a touch of Django which means
# you can inherit and tweak settings to your hearts content.

from sentry.conf.server import *  # NOQA


# Generously adapted from pynetlinux: https://git.io/JJmga
def get_internal_network():
    import ctypes
    import fcntl
    import math
    import socket
    import struct

    iface = b"eth0"
    sockfd = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    ifreq = struct.pack(b"16sH14s", iface, socket.AF_INET, b"\x00" * 14)

    try:
        ip = struct.unpack(
            b"!I", struct.unpack(b"16sH2x4s8x", fcntl.ioctl(sockfd, 0x8915, ifreq))[2]
        )[0]
        netmask = socket.ntohl(
            struct.unpack(b"16sH2xI8x", fcntl.ioctl(sockfd, 0x891B, ifreq))[2]
        )
    except IOError:
        return ()
    base = socket.inet_ntoa(struct.pack(b"!I", ip & netmask))
    netmask_bits = 32 - int(round(math.log(ctypes.c_uint32(~netmask).value + 1, 2), 1))
    return "{0:s}/{1:d}".format(base, netmask_bits)


INTERNAL_SYSTEM_IPS = (get_internal_network(),)


DATABASES = {
    "default": {
        "ENGINE": "sentry.db.postgres",
        "NAME": "postgres",
        "USER": "postgres",
        "PASSWORD": "",
        "HOST": "postgres",
        "PORT": "",
    }
}

# You should not change this setting after your database has been created
# unless you have altered all schemas first
SENTRY_USE_BIG_INTS = True

# If you're expecting any kind of real traffic on Sentry, we highly recommend
# configuring the CACHES and Redis settings

###########
# General #
###########

# Instruct Sentry that this install intends to be run by a single organization
# and thus various UI optimizations should be enabled.
SENTRY_SINGLE_ORGANIZATION = True

SENTRY_OPTIONS["system.event-retention-days"] = int(
    env("SENTRY_EVENT_RETENTION_DAYS", "30")
)
SENTRY_OPTIONS["system.url-prefix"] = "https://retail-sentry.apple.com"
:$
# SSL/TLS #
Users: 100% |####################################################| Time: 0:00:00
Teams: 100% |####################################################| Time: 0:00:00
 OK
  Applying sentry.0172_remove_demo_flag... OK
Creating missing DSNs
Correcting Group.num_comments counter

▶ Migrating file storage ...

▶ Generating Relay credentials ...
Relay credentials written to relay/credentials.json

▶ Setting up GeoIP integration ...
Setting up IP address geolocation ...
Installing (empty) IP address geolocation database ... done.
IP address geolocation is not configured for updates.
See https://develop.sentry.dev/self-hosted/geolocation/ for instructions.
Error setting up IP address geolocation.


-----------------------------------------------------------------

You're all done! Run the following command to get Sentry running:

  docker-compose up -d

-----------------------------------------------------------------

i see some errors after the upgrade on relay . Does this impact ingesting data?

relay_1                                     | 2021-03-10T00:41:09Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1                                     |   caused by: could not send request using reqwest
relay_1                                     |   caused by: error sending request for url (http://web:9000/api/0/relays/register/challenge/): error trying to connect: operation timed out
redis_1                                     | 1:M 10 Mar 2021 00:41:18.060 * 10000 changes in 60 seconds. Saving...
redis_1                                     | 1:M 10 Mar 2021 00:41:18.142 * Background saving started by pid 12
relay_1                                     | 2021-03-10T00:41:18Z [relay_server::actors::upstream] WARN: Network outage, scheduling another check in 11.390625s
nginx_1                                     | 17.26.239.70 - - [10/Mar/2021:00:41:21 +0000] "POST /api/19/store/ HTTP/1.1" 200 41 "-" "sentry.python/0.19.5" "-"
relay_1                                     | 2021-03-10T00:41:24Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1                                     |   caused by: could not send request using reqwest
relay_1                                     |   caused by: error sending request for url (http://web:9000/api/0/relays/register/challenge/): error trying to connect: operation timed out
11

Also , after the application came back up , i donot see anything on the UI on new events. No errors in the log, just this message about flussing items.

Post looks good, and seems like data is saved. but just these statements. can you provide some input

nginx_1                                     | 17.179.181.193 - - [10/Mar/2021:02:02:23 +0000] "POST /api/6/store/ HTTP/1.1" 200 41 "-" "sentry-ruby/3.1.1" "-"
nginx_1                                     | 17.179.181.193 - - [10/Mar/2021:02:02:23 +0000] "POST /api/6/store/ HTTP/1.1" 200 41 "-" "sentry-ruby/3.1.1" "-"
nginx_1                                     | 17.179.181.193 - - [10/Mar/2021:02:02:23 +0000] "POST /api/6/store/ HTTP/1.1" 200 41 "-" "sentry-ruby/3.1.1" "-"
ingest-consumer_1                           | 02:02:24 [INFO] batching-kafka-consumer: Flushing 3 items (from {('ingest-events', 0): [5452564, 5452566]}): forced:False size:False time:True
ingest-consumer_1                           | 02:02:25 [INFO] batching-kafka-consumer: Worker flush took 47ms
12

If you still see the relay errors, that would be the reason. Otherwise things look normal, I’d need more logs to provide useful input.

13

@BYK , Sentry just needed to catchup with all the backlog looks like.

Batch processing started and i see data on the UI . Nothing was changed.
How can I change the log timestamps, Logs seem to register data not based on the host time, may be GMT . Any way to change that.

Also i previously had 6 workers , based on your recommendations. Is this breakup still recommended on the new build?

worker1:
<< : *sentry_defaults
command: run worker -Q events.process_event

worker2:
<< : *sentry_defaults
command: run worker -Q events.reprocessing.process_event
worker3:
<< : *sentry_defaults
command: run worker -Q events.reprocess_events
worker4:
<< : *sentry_defaults
command: run worker -Q events.save_event
worker5:
<< : *sentry_defaults
command: run worker -Q subscriptions
worker6:
<< : *sentry_defaults
command: run worker -Q integrations
worker:
<< : *sentry_defaults
command: run worker

1 Like
14

Yay :partying_face:

Sorry, no idea about this :frowning: Maybe @untitaker knows?

Yup, just keep it as long as you can (based on your load is about the same).

15

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.