Upgrading from 9.12 -> 10.0.0 -> v20.6.0

Hello guys,

Im using Sentry Python based installation on Ubuntu.
OS - Ubuntu 18.04.1.
Sentry version - version 9.1.2
PostgreSQL - 10.5

At the moment we want to upgrade from 9.12 -> 10.0.0 and then to v20.6.0.
Is there any official upgrade guidelines? If not, are those steps looks more or less ok?:

from 9.12 to 10.0.0

  1. backup existing DB
  2. pip install sentry==10.0.0
  3. sentry upgrade

then from 10.0.0 to 20.6.0

  1. backup existing DB
  2. pip install sentry==20.6.0
  3. sentry upgrade

Any suggestions or comments?

Kristaps

The very first suggestion is to switch using to our official self-hosted repository as recommended in the documentation: https://develop.sentry.dev/self-hosted/

Second one is to jump directly to 20.9.0 or 20.10.0 that is releasing tomorrow.

thanks for very quick answer. two additional questions:

1)so i can “jump” directly from 9.12 to 20.9.0 or 20.10.0?

2)i use on-prem. local installation (from python packages) without docker or any containers, but on sentry official onpremise repo (https://github.com/getsentry/onpremise) i found there is requirements for docker and docker-compose. How i can avoid this?

Requirements

  • Docker 19.03.6+
  • Compose 1.24.1+

Kristaps

Yes. We normally recommend going through each major version but for now it is actually better to use a more recent version as we have some fixes in the migrations from 9.1.2.

You cannot, that’s the point. The Python package is only a very small piece of the puzzle. Sentry now requires using services like Snuba, Kafka, Clickhouse, and more so we doubled down on the Docker and Docker Compose based setup. You can read more about this on our blog post:

Thanks for detailed answer.

in this case for me best solution looks like :

0)make copy of 9.12 sentry config.yml and sentry.conf.py
1)make dump of 9.12 sentry postgres DB
2)on another server install last containerized sentry version from scratch (https://github.com/getsentry/onpremise) using config.yml / sentry.conf.py from 9.12 sentry
3)restore 9.12 sentry postgres dump into newly deployed postgres container

so it will be fresh installed containerized sentry (lets say version 20.10.0) but with config.yml /sentry.conf.py and posgres dump from sentry 9.12.

from sentry point of view its should work?

Kristaps

That sounds like a good plan to me. You can probably just dump the DB upfront and then proceed with the install.

FYI you’d need to get the new configuration updates manually for new versions. Also, don’t forget to use this branch for 9.1.2 (instead of master): https://github.com/getsentry/onpremise/tree/9.1.2

So you mean when im setting up containerized version from scratch i need to setup 9.1.2 instead of last available version ?

1.install 9.1.2 containerized from scratch
2.import postgres from existing 9.1.2 python-pip installation to newly deployed 9.1.2 containerized
3.upgrade newly deployed 9.1.2 containerized to 20.10.0?

correct?

Kristaps

@lvdombrkr yup, sounds about right to me. I think you can go directly with the upgrade path you suggested earlier but this felt safer to me.

Hey @BYK , first of all thanks for your quick support, so what i did:

-migrated from 9.12 not-containerized to containerized one
1)installed Sentry 9.1.2 containerized from scratch (https://github.com/getsentry/onpremise)
2)imported old dump from not-containerized setup to new one

so far all was fine, dump imported without any issues, dashboard was reachable with all data inside etc.

next step was to upgrade from 9.12 containerized to 20.6.0
What i did:
1)git checkout releases/20.9.0 (instead of 9.12)
2)./install.sh
3) after 5-10 minutes get this error:

Faking migration for contenttypes.0001_initial
Faking migration for contenttypes.0002_remove_content_type_name
Faking migration for auth.0001_initial
Faking migration for auth.0002_alter_permission_name_max_length
Faking migration for auth.0003_alter_user_email_max_length
Faking migration for auth.0004_alter_user_username_opts
Faking migration for auth.0005_alter_user_last_login_null
Faking migration for auth.0006_require_contenttypes_0002
Faking migration for sites.0001_initial
Faking migration for admin.0001_initial
Faking migration for sessions.0001_initial
Operations to perform:
Apply all migrations: admin, auth, contenttypes, jira_ac, nodestore, sentry, sessions, sites, social_auth
Running migrations:
Applying admin.0002_logentry_remove_auto_add… OK
Applying auth.0007_alter_validators_add_error_messages… OK
Applying auth.0008_alter_user_username_max_length… OK
Applying sentry.0002_912_to_recent…Traceback (most recent call last):
File “/usr/local/bin/sentry”, line 8, in
sys.exit(main())
File “/usr/local/lib/python2.7/site-packages/sentry/runner/init.py”, line 166, in main
cli(prog_name=get_prog(), obj={}, max_content_width=100)
File “/usr/local/lib/python2.7/site-packages/click/core.py”, line 722, in call
return self.main(*args, **kwargs)
File “/usr/local/lib/python2.7/site-packages/click/core.py”, line 697, in main
rv = self.invoke(ctx)
File “/usr/local/lib/python2.7/site-packages/click/core.py”, line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File “/usr/local/lib/python2.7/site-packages/click/core.py”, line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File “/usr/local/lib/python2.7/site-packages/click/core.py”, line 535, in invoke
return callback(*args, **kwargs)
File “/usr/local/lib/python2.7/site-packages/click/decorators.py”, line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File “/usr/local/lib/python2.7/site-packages/sentry/runner/decorators.py”, line 30, in inner
return ctx.invoke(f, *args, **kwargs)
File “/usr/local/lib/python2.7/site-packages/click/core.py”, line 535, in invoke
return callback(*args, **kwargs)
File “/usr/local/lib/python2.7/site-packages/click/decorators.py”, line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File “/usr/local/lib/python2.7/site-packages/sentry/runner/commands/upgrade.py”, line 174, in upgrade
_upgrade(not noinput, traceback, verbosity, not no_repair, with_nodestore)
File “/usr/local/lib/python2.7/site-packages/sentry/runner/commands/upgrade.py”, line 130, in _upgrade
ignore_ghost_migrations=True,
File “/usr/local/lib/python2.7/site-packages/django/core/management/init.py”, line 131, in call_command
return command.execute(*args, **defaults)
File “/usr/local/lib/python2.7/site-packages/django/core/management/base.py”, line 330, in execute
output = self.handle(*args, **options)
File “/usr/local/lib/python2.7/site-packages/django/core/management/commands/migrate.py”, line 204, in handle
fake_initial=fake_initial,
File “/usr/local/lib/python2.7/site-packages/django/db/migrations/executor.py”, line 115, in migrate
state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
File “/usr/local/lib/python2.7/site-packages/django/db/migrations/executor.py”, line 145, in _migrate_all_forwards
state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
File “/usr/local/lib/python2.7/site-packages/sentry/new_migrations/monkey/executor.py”, line 26, in apply_migration
state, migration, fake=fake, fake_initial=fake_initial
File “/usr/local/lib/python2.7/site-packages/django/db/migrations/executor.py”, line 244, in apply_migration
state = migration.apply(state, schema_editor)
File “/usr/local/lib/python2.7/site-packages/django/db/migrations/migration.py”, line 129, in apply
operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
File “/usr/local/lib/python2.7/site-packages/django/db/migrations/operations/fields.py”, line 221, in database_forwards
schema_editor.alter_field(from_model, from_field, to_field)
File “/usr/local/lib/python2.7/site-packages/django/db/backends/base/schema.py”, line 531, in alter_field
old_db_params, new_db_params, strict)
File “/usr/local/lib/python2.7/site-packages/django/db/backends/postgresql/schema.py”, line 112, in _alter_field
new_db_params, strict,
File “/usr/local/lib/python2.7/site-packages/django/db/backends/base/schema.py”, line 706, in _alter_field
params,
File “/usr/local/lib/python2.7/site-packages/django/db/backends/base/schema.py”, line 136, in execute
cursor.execute(sql, params)
File “/usr/local/lib/python2.7/site-packages/sentry_sdk/integrations/django/init.py”, line 489, in execute
return real_execute(self, sql, params)
File “/usr/local/lib/python2.7/site-packages/django/db/backends/utils.py”, line 64, in execute
return self.cursor.execute(sql, params)
File “/usr/local/lib/python2.7/site-packages/django/db/utils.py”, line 94, in exit
six.reraise(dj_exc_type, dj_exc_value, traceback)
File “/usr/local/lib/python2.7/site-packages/django/db/backends/utils.py”, line 64, in execute
return self.cursor.execute(sql, params)
File “/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py”, line 81, in inner
raise_the_exception(self.db, e)
File “/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py”, line 79, in inner
return func(self, *args, **kwargs)
File “/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py”, line 22, in inner
return func(self, *args, **kwargs)
File “/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py”, line 99, in inner
six.reraise(exc_info[0], exc_info0, exc_info[2])
File “/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py”, line 95, in inner
return func(self, sql, *args, **kwargs)
File “/usr/local/lib/python2.7/site-packages/sentry/db/postgres/base.py”, line 78, in execute
return self.cursor.execute(sql, clean_bad_params(params))
django.db.utils.IntegrityError: NotNullViolation(‘column “project_id” contains null values\n’,)
SQL: ALTER TABLE “sentry_groupedmessage” ALTER COLUMN “project_id” SET NOT NULL
An error occurred, caught SIGERR on line 280
Cleaning up…
[root@jumphost onpremise]#

any ideas how to avoid this error and what went wrong?

Kristaps

Looks like your database has some invalid rows. I’d try to remove all rows with a NULL project_id column from the sentry_groupedmessage table manually and try the migration again.

Also, I’d try upgrading to 20.10.1 instead of 20.6.0 as it has some fixes around upgrading from 9.1.2.

so i :
1)i deleted manually NULLs
DELETE FROM sentry_groupedmessage WHERE project_id IS NULL;

2)switched 20.10.1
git checkout releases/20.10.1

3)re-run install.sh
./install.sh

now im passing stage where i was failing previously but now migration is stucked without any errors etc.(with no outputs or errors) its just stucked during last 30-40 minutes.
maybe its still in process im not sure at the moment. its stucked on last line:

Creating sentry_onpremise_snuba-sessions-consumer_1 …
Creating sentry_onpremise_postgres_1 … done
Creating sentry_onpremise_symbolicator_1 … done
Creating sentry_onpremise_smtp_1 … done
Creating sentry_onpremise_snuba-sessions-consumer_1 … done
Creating sentry_onpremise_snuba-consumer_1 … done
Creating sentry_onpremise_snuba-outcomes-consumer_1 … done
Creating sentry_onpremise_snuba-replacer_1 … done
Creating sentry_onpremise_memcached_1 … done
Creating sentry_onpremise_snuba-api_1 … done
Creating sentry_onpremise_snuba-transactions-consumer_1 … done
Creating sentry_onpremise_web_run …
Creating sentry_onpremise_web_run … done
/usr/local/lib/python2.7/site-packages/sentry/runner/initializer.py:197: DeprecatedSettingWarning: The GITHUB_EXTENDED_PERMISSIONS setting is deprecated. Please use SENTRY_OPTIONS[‘github-login.extended-permissions’] instead.
warnings.warn(DeprecatedSettingWarning(options_mapper[k], “SENTRY_OPTIONS[’%s’]” % k))
16:14:16 [WARNING] sentry.utils.geo: settings.GEOIP_PATH_MMDB not configured.
/usr/local/lib/python2.7/site-packages/OpenSSL/crypto.py:12: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in a future release.
from cryptography import x509
16:14:39 [INFO] sentry.plugins.github: apps-not-configured
Operations to perform:
Apply all migrations: admin, auth, contenttypes, jira_ac, nodestore, sentry, sessions, sites, social_auth
Running migrations:
Applying sentry.0002_912_to_recent… OK
Applying sentry.0003_auto_20191022_0122… OK
Applying sentry.0004_bitfieldtestmodel_blankjsonfieldtestmodel_callabledefaultmodel_jsonfieldtestmodel_jsonfieldwithdefau… OK
Applying sentry.0005_fix_content_types… OK
Applying sentry.0006_sentryapp_date_published… OK
Applying sentry.0007_auto_20191029_0131… OK
Applying sentry.0008_auto_20191030_0016… OK
Applying sentry.0009_auto_20191101_1608… OK
Applying sentry.0010_auto_20191104_1641… OK
Applying sentry.0011_remove_pagerdutyservice_service_id_from_state… OK
Applying sentry.0012_remove_pagerdutyservice_service_id… OK
Applying sentry.0013_auto_20191111_1829… OK
Applying sentry.0014_delete_sentryappwebhookerror… OK
Applying sentry.0015_delete_sentryappwebhookerror_db… OK
Applying sentry.0016_delete_alert_rule_deprecated_fields… OK
Applying sentry.0017_incident_aggregation… OK
Applying sentry.0018_discoversavedquery_version… OK
Applying sentry.0019_auto_20191114_2040… OK
Applying sentry.0020_auto_20191125_1420… OK
Applying sentry.0021_auto_20191203_1803… OK
Applying sentry.0021_auto_20191202_1716… OK
Applying sentry.0022_merge… OK
Applying sentry.0023_hide_environment_none_20191126… OK
Applying sentry.0024_auto_20191230_2052…Events to process: 1208918

Yeah, 1.2M events is quite a lot to process :smiley:

We may add some indication that this is working but it would really clobber (and swell) the log outputs. What do you think?

@BYK Hello again ))

After half night processing we are done ))

After docker-compose up all containers is up and running but we have 2 issues for the momens:

-two key auth. is not working so im not able login in my profile :
1)im entering my user and password
2)im rediricted to 2auth page
3)when im entering Authenticator code on dashboard i see “Invalid confirmation code. Try again.” on logs i see:

snuba-consumer_1 | 2020-10-20 07:34:23,763 Completed processing <Batch: 471 messages, open for 0.98 seconds>.
snuba-transactions-consumer_1 | 2020-10-20 07:34:24,527 Completed processing <Batch: 241 messages, open for 1.00 seconds>.
snuba-consumer_1 | 2020-10-20 07:34:24,769 Completed processing <Batch: 467 messages, open for 1.00 seconds>.
snuba-transactions-consumer_1 | 2020-10-20 07:34:25,530 Completed processing <Batch: 264 messages, open for 1.00 seconds>.
snuba-consumer_1 | 2020-10-20 07:34:25,766 Completed processing <Batch: 353 messages, open for 0.99 seconds>.
nginx_1 | 10.0.2.2 - - [20/Oct/2020:07:34:26 +0000] “POST /auth/2fa/ HTTP/1.1” 200 5180 “http://127.0.0.1:9000/auth/2fa/” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36” “-”
web_1 | 07:34:26 [WARNING] django.request: Not Found: /_static/1603178751/sentry/js/ads.js (status_code=404 request=<WSGIRequest: GET u’/_static/1603178751/sentry/js/ads.js’>)
nginx_1 | 10.0.2.2 - - [20/Oct/2020:07:34:26 +0000] “GET /_static/1603178751/sentry/js/ads.js HTTP/1.1” 404 0 “http://127.0.0.1:9000/auth/2fa/” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36” “-”
snuba-transactions-consumer_1 | 2020-10-20 07:34:26,537 Completed processing <Batch: 326 messages, open for 1.00 seconds>.
snuba-consumer_1 | 2020-10-20 07:34:26,786 Completed processing <Batch: 391 messages, open for 1.02 seconds>.
snuba-transactions-consumer_1 | 2020-10-20 07:34:27,537 Completed processing <Batch: 422 messages, open for 1.00 seconds>.
snuba-consumer_1 | 2020-10-20 07:34:27,808 Completed processing <Batch: 424 messages, open for 1.02 seconds>.
snuba-transactions-consumer_1 | 2020-10-20 07:34:28,549 Completed processing <Batch: 154 messages, open for 1.01 seconds>.

I see error message in logs “status_code=404 request=<WSGIRequest: GET u’/_static/1603178751/sentry/js/ads.js’” but i not have any clues what it means ))

second issue:
-password restoring not working.
1)i press “Lost your password?”
2)entering my email
3) get message :
We have sent an email to the address registered with this account containing further instructions to reset your password.
on logs:

snuba-transactions-consumer_1 | 2020-10-20 07:31:59,926 Completed processing <Batch: 429 messages, open for 1.01 seconds>.
snuba-consumer_1 | 2020-10-20 07:32:00,177 Completed processing <Batch: 379 messages, open for 1.00 seconds>.
web_1 | 07:32:00 [INFO] sentry.mail: mail.queued (message_to=(u’xxxx.xxx@xxxx’,) user_id=124 message_id=u’20201020073159.25.41314@localhost’ message_type=u’user.password_recovery’)
web_1 | 07:32:00 [INFO] sentry.accounts: recover.sent (user_id=124 user_recovered=u’xxxx.xxx@xxxx’ ip_address=u’10.0.2.2’ passwordhash_id=48L user_agent=u’Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36’)
nginx_1 | 10.0.2.2 - - [20/Oct/2020:07:32:00 +0000] “POST /account/recover/ HTTP/1.1” 200 3715 “http://127.0.0.1:9000/account/recover/” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36” “-”
web_1 | 07:32:00 [WARNING] django.request: Not Found: /_static/1603178751/sentry/js/ads.js (status_code=404 request=<WSGIRequest: GET u’/_static/1603178751/sentry/js/ads.js’>)
nginx_1 | 10.0.2.2 - - [20/Oct/2020:07:32:00 +0000] “GET /_static/1603178751/sentry/js/ads.js HTTP/1.1” 404 0 “http://127.0.0.1:9000/account/recover/” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36” “-”
smtp_1 | 270 LOG: MAIN
smtp_1 | 270 <= root@localhost H=sentry_onpremise_worker_1.sentry_onpremise_default (69862f9903ef) [172.23.0.21] P=esmtp S=6262 id=20201020073159.25.41314@localhost
smtp_1 | 270 LOG: smtp_connection MAIN
smtp_1 | 270 SMTP connection from sentry_onpremise_worker_1.sentry_onpremise_default (69862f9903ef) [172.23.0.21] closed by QUIT
worker_1 | 07:32:00 [INFO] sentry.mail: mail.sent (size=5995 message_id=u’20201020073159.25.41314@localhost’)
smtp_1 | 271 Exim version 4.92 uid=101 gid=101 pid=271 D=80001
smtp_1 | Berkeley DB: Berkeley DB 5.3.28: (September 9, 2013)
smtp_1 | Support for: crypteq iconv() IPv6 GnuTLS move_frozen_messages DANE DKIM DNSSEC Event OCSP PRDR SOCKS TCP_Fast_Open
smtp_1 | Lookups (built-in): lsearch wildlsearch nwildlsearch iplsearch cdb dbm dbmjz dbmnz dnsdb dsearch nis nis0 passwd
smtp_1 | Authenticators: cram_md5 plaintext
smtp_1 | Routers: accept dnslookup ipliteral manualroute queryprogram redirect
smtp_1 | Transports: appendfile/maildir/mailstore autoreply lmtp pipe smtp
smtp_1 | Fixed never_users: 0
smtp_1 | Configure owner: 0:0
smtp_1 | Size of off_t: 8
smtp_1 | 271 delivering 1kUm7k-00004M-Nn
smtp_1 | 271 R: dnslookup for xxxx.xxx@xxxx
snuba-transactions-consumer_1 | 2020-10-20 07:32:00,938 Completed processing <Batch: 336 messages, open for 1.01 seconds>.
smtp_1 | 272 T: remote_smtp for xxxx.xxx@xxxx
smtp_1 | 272 Connecting to aspmx.l.google.com [2a00:1450:4010:c01::1a]:25 … failed: Network is unreachable
smtp_1 | 272 LOG: MAIN
smtp_1 | 272 H=aspmx.l.google.com [2a00:1450:4010:c01::1a] Network is unreachable
snuba-consumer_1 | 2020-10-20 07:32:01,178 Completed processing <Batch: 378 messages, open for 1.00 seconds>.
snuba-transactions-consumer_1 | 2020-10-20 07:32:01,944 Completed processing <Batch: 258 messages, open for 1.00 seconds>.

log from smtp docker container:

[root@jumphost volumes]# docker logs sentry_onpremise_smtp_1

  • sed -ri ’
    s/^#?(dc_local_interfaces)=./\1=’’‘0.0.0.0 ; ::0’’’/;
    s/^#?(dc_other_hostnames)=.
    /\1=’’’’’’/;
    s/^#?(dc_relay_nets)=./\1=’’‘0.0.0.0/0’’’/;
    s/^#?(dc_eximconfig_configtype)=.
    /\1=’’‘internet’’’/;
    ’ /etc/exim4/update-exim4.conf.conf
  • update-exim4.conf -v
    using non-split configuration scheme from /etc/exim4/exim4.conf.template
    269 LOG: MAIN
    269 exim 4.92 daemon started: pid=269, no queue runs, listening for SMTP on port 25 (IPv6 and IPv4)
    270 LOG: MAIN
    270 <= root@localhost H=sentry_onpremise_worker_1.sentry_onpremise_default (69862f9903ef) [172.23.0.21] P=esmtp S=6262 id=20201020073159.25.41314@localhost
    270 LOG: smtp_connection MAIN
    270 SMTP connection from sentry_onpremise_worker_1.sentry_onpremise_default (69862f9903ef) [172.23.0.21] closed by QUIT
    271 Exim version 4.92 uid=101 gid=101 pid=271 D=80001
    Berkeley DB: Berkeley DB 5.3.28: (September 9, 2013)
    Support for: crypteq iconv() IPv6 GnuTLS move_frozen_messages DANE DKIM DNSSEC Event OCSP PRDR SOCKS TCP_Fast_Open
    Lookups (built-in): lsearch wildlsearch nwildlsearch iplsearch cdb dbm dbmjz dbmnz dnsdb dsearch nis nis0 passwd
    Authenticators: cram_md5 plaintext
    Routers: accept dnslookup ipliteral manualroute queryprogram redirect
    Transports: appendfile/maildir/mailstore autoreply lmtp pipe smtp
    Fixed never_users: 0
    Configure owner: 0:0
    Size of off_t: 8
    271 delivering 1kUm7k-00004M-Nn
    271 R: dnslookup for xxxxxs@xxxxx
    272 T: remote_smtp forxxxxx@xxxx
    272 Connecting to aspmx.l.google.com [2a00:1450:4010:c01::1a]:25 … failed: Network is unreachable
    272 LOG: MAIN
    272 H=aspmx.l.google.com [2a00:1450:4010:c01::1a] Network is unreachable
    272 Connecting to aspmx.l.google.com [64.233.165.26]:25 … failed: Connection timed out (timeout=5m)
    272 LOG: MAIN
    272 H=aspmx.l.google.com [64.233.165.26] Connection timed out
    272 Connecting to alt2.aspmx.l.google.com [2607:f8b0:400e:c04::1a]:25 … failed: Network is unreachable
    272 LOG: MAIN
    272 H=alt2.aspmx.l.google.com [2607:f8b0:400e:c04::1a] Network is unreachable
    272 Connecting to alt1.aspmx.l.google.com [2404:6800:4008:c00::1b]:25 … failed: Network is unreachable
    272 LOG: MAIN
    272 H=alt1.aspmx.l.google.com [2404:6800:4008:c00::1b] Network is unreachable
    272 Connecting to alt2.aspmx.l.google.com [74.125.28.26]:25 … failed: Connection timed out (timeout=5m)
    272 LOG: MAIN
    272 H=alt2.aspmx.l.google.com [74.125.28.26] Connection timed out
    272 Connecting to alt4.aspmx.l.google.com [2607:f8b0:4023:1000::1b]:25 … failed: Network is unreachable
    272 LOG: MAIN
    272 H=alt4.aspmx.l.google.com [2607:f8b0:4023:1000::1b] Network is unreachable
    271 LOG: MAIN
    271 == xxxxx@xxxxxx R=dnslookup T=remote_smtp defer (101): Network is unreachable
    621 LOG: MAIN
    621 <= root@localhost H=sentry_onpremise_worker_1.sentry_onpremise_default (69862f9903ef) [172.23.0.21] P=esmtp S=6262 id=20201020074748.25.66673@localhost
    621 LOG: smtp_connection MAIN
    621 SMTP connection from sentry_onpremise_worker_1.sentry_onpremise_default (69862f9903ef) [172.23.0.21] closed by QUIT
    622 Exim version 4.92 uid=101 gid=101 pid=622 D=80001
    Berkeley DB: Berkeley DB 5.3.28: (September 9, 2013)
    Support for: crypteq iconv() IPv6 GnuTLS move_frozen_messages DANE DKIM DNSSEC Event OCSP PRDR SOCKS TCP_Fast_Open
    Lookups (built-in): lsearch wildlsearch nwildlsearch iplsearch cdb dbm dbmjz dbmnz dnsdb dsearch nis nis0 passwd
    Authenticators: cram_md5 plaintext
    Routers: accept dnslookup ipliteral manualroute queryprogram redirect
    Transports: appendfile/maildir/mailstore autoreply lmtp pipe smtp
    Fixed never_users: 0
    Configure owner: 0:0
    Size of off_t: 8
    622 delivering 1kUmN2-0000A1-Pi
    622 R: dnslookup for xxxxxx@xxxx
    623 T: remote_smtp for xxxxx@xxxxxx
    623 Connecting to aspmx.l.google.com [2a00:1450:4010:c01::1b]:25 … failed: Network is unreachable
    623 LOG: MAIN
    623 H=aspmx.l.google.com [2a00:1450:4010:c01::1b] Network is unreachable
    623 Connecting to aspmx.l.google.com [64.233.162.27]:25 … failed: Connection timed out (timeout=5m)
    623 LOG: MAIN
    623 H=aspmx.l.google.com [64.233.162.27] Connection timed out
    623 Connecting to alt2.aspmx.l.google.com [2607:f8b0:400e:c04::1b]:25 … failed: Network is unreachable
    623 LOG: MAIN
    623 H=alt2.aspmx.l.google.com [2607:f8b0:400e:c04::1b] Network is unreachable
    623 Connecting to alt1.aspmx.l.google.com [2404:6800:4008:c00::1a]:25 … failed: Network is unreachable
    623 LOG: MAIN
    623 H=alt1.aspmx.l.google.com [2404:6800:4008:c00::1a] Network is unreachable
    623 Connecting to alt1.aspmx.l.google.com [108.177.97.26]:25 … failed: Connection timed out (timeout=5m)
    623 LOG: MAIN
    623 H=alt1.aspmx.l.google.com [108.177.97.26] Connection timed out
    623 Connecting to alt3.aspmx.l.google.com [2607:f8b0:4023:c03::1b]:25 … failed: Network is unreachable
    623 LOG: MAIN
    623 H=alt3.aspmx.l.google.com [2607:f8b0:4023:c03::1b] Network is unreachable
    622 LOG: MAIN
    622 == xxxx@xxxxx.com R=dnslookup T=remote_smtp defer (101): Network is unreachable
    [root@jumphost volumes]#

From last log i see that " 272 Connecting to aspmx.l.google.com [2a00:1450:4010:c01::1a]:25 … failed: Network is unreachable" but if i exec smtp container i can reach this server, at least i can ping it:

root@eb3580f0ee21:/# ping aspmx.l.google.com
PING aspmx.l.google.com (209.85.233.27) 56(84) bytes of data.
64 bytes from lr-in-f27.1e100.net (209.85.233.27): icmp_seq=1 ttl=105 time=22.5 ms
64 bytes from lr-in-f27.1e100.net (209.85.233.27): icmp_seq=2 ttl=105 time=22.7 ms
64 bytes from lr-in-f27.1e100.net (209.85.233.27): icmp_seq=3 ttl=105 time=22.3 ms
64 bytes from lr-in-f27.1e100.net (209.85.233.27): icmp_seq=4 ttl=105 time=22.7 ms
^C
aspmx.l.google.com ping statistics —

any clues about both issues?

Kristaps

im continue to investigate both issues and this is logs from relay container, i think it can be of the be rootcouse:

[root@jumphost onpremise]# docker logs sentry_onpremise_relay_1
2020-10-20T07:25:36Z [rdkafka::client] ERROR: librdkafka: FAIL [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.23.0.10:9092 failed: Connection refused (after 5ms in state CONNECT)
2020-10-20T07:25:36Z [rdkafka::client] ERROR: librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): kafka:9092/bootstrap: Connect to ipv4#172.23.0.10:9092 failed: Connection refused (after 5ms in state CONNECT)
2020-10-20T07:25:36Z [rdkafka::client] ERROR: librdkafka: Global error: AllBrokersDown (Local: All broker connections are down): 1/1 brokers are down
2020-10-20T07:25:36Z [rdkafka::client] ERROR: librdkafka: FAIL [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.23.0.10:9092 failed: Connection refused (after 22ms in state CONNECT)
2020-10-20T07:25:36Z [rdkafka::client] ERROR: librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): kafka:9092/bootstrap: Connect to ipv4#172.23.0.10:9092 failed: Connection refused (after 22ms in state CONNECT)
2020-10-20T07:25:36Z [rdkafka::client] ERROR: librdkafka: Global error: AllBrokersDown (Local: All broker connections are down): 1/1 brokers are down
2020-10-20T07:25:37Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
caused by: Failed to connect to host: Failed resolving hostname: no record found for name: web type: AAAA class: IN
caused by: Failed resolving hostname: no record found for name: web type: AAAA class: IN
caused by: Failed resolving hostname: no record found for name: web type: AAAA class: IN
2020-10-20T07:25:37Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
caused by: Failed to connect to host: Failed resolving hostname: no record found for name: web type: AAAA class: IN
caused by: Failed resolving hostname: no record found for name: web type: AAAA class: IN
caused by: Failed resolving hostname: no record found for name: web type: AAAA class: IN
2020-10-20T07:25:37Z [rdkafka::client] ERROR: librdkafka: FAIL [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.23.0.10:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
2020-10-20T07:25:37Z [rdkafka::client] ERROR: librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): kafka:9092/bootstrap: Connect to ipv4#172.23.0.10:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
2020-10-20T07:25:37Z [rdkafka::client] ERROR: librdkafka: FAIL [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.23.0.10:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
2020-10-20T07:25:37Z [rdkafka::client] ERROR: librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): kafka:9092/bootstrap: Connect to ipv4#172.23.0.10:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
2020-10-20T07:25:39Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
caused by: Failed to connect to host: Connection refused (os error 111)
caused by: Connection refused (os error 111)
caused by: Connection refused (os error 111)
2020-10-20T07:25:40Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
caused by: Failed to connect to host: Connection refused (os error 111)
caused by: Connection refused (os error 111)
caused by: Connection refused (os error 111)
2020-10-20T07:25:42Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
caused by: Failed to connect to host: Connection refused (os error 111)
caused by: Connection refused (os error 111)
caused by: Connection refused (os error 111)
2020-10-20T07:25:46Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
caused by: Failed to connect to host: Connection refused (os error 111)
caused by: Connection refused (os error 111)
caused by: Connection refused (os error 111)
2020-10-20T07:25:56Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
caused by: Timeout while waiting for response
[root@jumphost onpremise]#

log from ingest-consumer

[root@jumphost onpremise]# docker logs sentry_onpremise_ingest-consumer_1
/usr/local/lib/python2.7/site-packages/sentry/runner/initializer.py:197: DeprecatedSettingWarning: The GITHUB_EXTENDED_PERMISSIONS setting is deprecated. Please use SENTRY_OPTIONS[‘github-login.extended-permissions’] instead.
warnings.warn(DeprecatedSettingWarning(options_mapper[k], “SENTRY_OPTIONS[’%s’]” % k))
07:25:42 [WARNING] sentry.utils.geo: settings.GEOIP_PATH_MMDB not configured.
/usr/local/lib/python2.7/site-packages/OpenSSL/crypto.py:12: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in a future release.
from cryptography import x509
07:25:49 [INFO] sentry.plugins.github: apps-not-configured
%3|1603178750.324|FAIL|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.23.0.10:9092 failed: Connection refused (after 0ms in state CONNECT)
%3|1603178751.323|FAIL|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.23.0.10:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
07:25:52 [WARNING] batching-kafka-consumer: Topic ‘ingest-transactions’ or its partitions are not ready, retrying…
07:25:52 [WARNING] batching-kafka-consumer: Topic ‘ingest-transactions’ or its partitions are not ready, retrying…
Traceback (most recent call last):
File “/usr/local/bin/sentry”, line 8, in
sys.exit(main())
File “/usr/local/lib/python2.7/site-packages/sentry/runner/init.py”, line 166, in main
cli(prog_name=get_prog(), obj={}, max_content_width=100)
File “/usr/local/lib/python2.7/site-packages/click/core.py”, line 722, in call
return self.main(*args, **kwargs)
File “/usr/local/lib/python2.7/site-packages/click/core.py”, line 697, in main
rv = self.invoke(ctx)
File “/usr/local/lib/python2.7/site-packages/click/core.py”, line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File “/usr/local/lib/python2.7/site-packages/click/core.py”, line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File “/usr/local/lib/python2.7/site-packages/click/core.py”, line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File “/usr/local/lib/python2.7/site-packages/click/core.py”, line 535, in invoke
return callback(*args, **kwargs)
File “/usr/local/lib/python2.7/site-packages/click/decorators.py”, line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File “/usr/local/lib/python2.7/site-packages/sentry/runner/decorators.py”, line 68, in inner
return ctx.invoke(f, *args, **kwargs)
File “/usr/local/lib/python2.7/site-packages/click/core.py”, line 535, in invoke
return callback(*args, **kwargs)
File “/usr/local/lib/python2.7/site-packages/click/decorators.py”, line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File “/usr/local/lib/python2.7/site-packages/sentry/runner/decorators.py”, line 30, in inner
return ctx.invoke(f, *args, **kwargs)
File “/usr/local/lib/python2.7/site-packages/click/core.py”, line 535, in invoke
return callback(*args, **kwargs)
File “/usr/local/lib/python2.7/site-packages/sentry/runner/commands/run.py”, line 440, in ingest_consumer
get_ingest_consumer(consumer_types=consumer_types, **options).run()
File “/usr/local/lib/python2.7/site-packages/sentry/utils/batching_kafka_consumer.py”, line 261, in run
self._run_once()
File “/usr/local/lib/python2.7/site-packages/sentry/utils/batching_kafka_consumer.py”, line 279, in _run_once
raise Exception(msg.error())
Exception: KafkaError{code=COORDINATOR_LOAD_IN_PROGRESS,val=14,str=“JoinGroup failed: Broker: Coordinator load in progress”}
/usr/local/lib/python2.7/site-packages/sentry/runner/initializer.py:197: DeprecatedSettingWarning: The GITHUB_EXTENDED_PERMISSIONS setting is deprecated. Please use SENTRY_OPTIONS[‘github-login.extended-permissions’] instead.
warnings.warn(DeprecatedSettingWarning(options_mapper[k], “SENTRY_OPTIONS[’%s’]” % k))
07:25:56 [WARNING] sentry.utils.geo: settings.GEOIP_PATH_MMDB not configured.
/usr/local/lib/python2.7/site-packages/OpenSSL/crypto.py:12: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in a future release.
from cryptography import x509
07:26:02 [INFO] sentry.plugins.github: apps-not-configured
07:26:05 [INFO] batching-kafka-consumer: New partitions assigned: [TopicPartition{topic=ingest-attachments,partition=0,offset=-1001,error=None}, TopicPartition{topic=ingest-events,partition=0,offset=-1001,error=None}, TopicPartition{topic=ingest-transactions,partition=0,offset=-1001,error=None}]
[root@jumphost onpremise]#

[root@jumphost onpremise]# docker logs sentry_onpremise_post-process-forwarder_1
/usr/local/lib/python2.7/site-packages/sentry/runner/initializer.py:197: DeprecatedSettingWarning: The GITHUB_EXTENDED_PERMISSIONS setting is deprecated. Please use SENTRY_OPTIONS[‘github-login.extended-permissions’] instead.
warnings.warn(DeprecatedSettingWarning(options_mapper[k], “SENTRY_OPTIONS[’%s’]” % k))
07:25:42 [WARNING] sentry.utils.geo: settings.GEOIP_PATH_MMDB not configured.
/usr/local/lib/python2.7/site-packages/OpenSSL/crypto.py:12: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in a future release.
from cryptography import x509
07:25:49 [INFO] sentry.plugins.github: apps-not-configured
%3|1603178750.004|FAIL|rdkafka#consumer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.23.0.10:9092 failed: Connection refused (after 0ms in state CONNECT)
%3|1603178750.004|FAIL|rdkafka#consumer-2| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.23.0.10:9092 failed: Connection refused (after 0ms in state CONNECT)
%3|1603178751.003|FAIL|rdkafka#consumer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.23.0.10:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
%3|1603178751.004|FAIL|rdkafka#consumer-2| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.23.0.10:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
Traceback (most recent call last):
File “/usr/local/bin/sentry”, line 8, in
sys.exit(main())
File “/usr/local/lib/python2.7/site-packages/sentry/runner/init.py”, line 166, in main
cli(prog_name=get_prog(), obj={}, max_content_width=100)
File “/usr/local/lib/python2.7/site-packages/click/core.py”, line 722, in call
return self.main(*args, **kwargs)
File “/usr/local/lib/python2.7/site-packages/click/core.py”, line 697, in main
rv = self.invoke(ctx)
File “/usr/local/lib/python2.7/site-packages/click/core.py”, line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File “/usr/local/lib/python2.7/site-packages/click/core.py”, line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File “/usr/local/lib/python2.7/site-packages/click/core.py”, line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File “/usr/local/lib/python2.7/site-packages/click/core.py”, line 535, in invoke
return callback(*args, **kwargs)
File “/usr/local/lib/python2.7/site-packages/click/decorators.py”, line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File “/usr/local/lib/python2.7/site-packages/sentry/runner/decorators.py”, line 68, in inner
return ctx.invoke(f, *args, **kwargs)
File “/usr/local/lib/python2.7/site-packages/click/core.py”, line 535, in invoke
return callback(*args, **kwargs)
File “/usr/local/lib/python2.7/site-packages/click/decorators.py”, line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File “/usr/local/lib/python2.7/site-packages/sentry/runner/decorators.py”, line 30, in inner
return ctx.invoke(f, *args, **kwargs)
File “/usr/local/lib/python2.7/site-packages/click/core.py”, line 535, in invoke
return callback(*args, **kwargs)
File “/usr/local/lib/python2.7/site-packages/sentry/runner/commands/run.py”, line 294, in post_process_forwarder
initial_offset_reset=options[“initial_offset_reset”],
File “/usr/local/lib/python2.7/site-packages/sentry/utils/services.py”, line 105, in
context[key] = (lambda f: lambda *a, **k: getattr(self, f)(*a, **k))(key)
File “/usr/local/lib/python2.7/site-packages/sentry/eventstream/kafka/backend.py”, line 195, in run_post_process_forwarder
raise Exception(error)
Exception: KafkaError{code=COORDINATOR_LOAD_IN_PROGRESS,val=14,str=“JoinGroup failed: Broker: Coordinator load in progress”}
/usr/local/lib/python2.7/site-packages/sentry/runner/initializer.py:197: DeprecatedSettingWarning: The GITHUB_EXTENDED_PERMISSIONS setting is deprecated. Please use SENTRY_OPTIONS[‘github-login.extended-permissions’] instead.
warnings.warn(DeprecatedSettingWarning(options_mapper[k], “SENTRY_OPTIONS[’%s’]” % k))
07:25:56 [WARNING] sentry.utils.geo: settings.GEOIP_PATH_MMDB not configured.
/usr/local/lib/python2.7/site-packages/OpenSSL/crypto.py:12: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in a future release.
from cryptography import x509
07:26:02 [INFO] sentry.plugins.github: apps-not-configured
[root@jumphost onpremise]#

[root@jumphost onpremise]# docker logs sentry_onpremise_web_1
/usr/local/lib/python2.7/site-packages/sentry/runner/initializer.py:197: DeprecatedSettingWarning: The GITHUB_EXTENDED_PERMISSIONS setting is deprecated. Please use SENTRY_OPTIONS[‘github-login.extended-permissions’] instead.
warnings.warn(DeprecatedSettingWarning(options_mapper[k], “SENTRY_OPTIONS[’%s’]” % k))
07:25:42 [WARNING] sentry.utils.geo: settings.GEOIP_PATH_MMDB not configured.
/usr/local/lib/python2.7/site-packages/OpenSSL/crypto.py:12: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in a future release.
from cryptography import x509
07:25:49 [INFO] sentry.plugins.github: apps-not-configured
*** Starting uWSGI 2.0.19.1 (64bit) on [Tue Oct 20 07:25:50 2020] ***
compiled with version: 8.3.0 on 15 October 2020 22:29:34
os: Linux-3.10.0-1127.19.1.el7.x86_64 #1 SMP Tue Aug 25 17:23:54 UTC 2020
nodename: 8feddb42f0aa
machine: x86_64
clock source: unix
detected number of CPU cores: 8
current working directory: /
detected binary path: /usr/local/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
your memory page size is 4096 bytes
detected max file descriptor number: 1048576
lock engine: pthread robust mutexes
thunder lock: enabled
uWSGI http bound on 0.0.0.0:9000 fd 6
uwsgi socket 0 bound to TCP address 127.0.0.1:45407 (port auto-assigned) fd 3
Python version: 2.7.16 (default, Oct 17 2019, 07:39:30) [GCC 8.3.0]
Set PythonHome to /usr/local
Python main interpreter initialized at 0x55f2e4923fd0
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
setting request body buffering size to 65536 bytes
mapped 1924224 bytes (1879 KB) for 12 cores
*** Operational MODE: preforking+threaded ***
spawned uWSGI master process (pid: 18)
spawned uWSGI worker 1 (pid: 23, cores: 4)
spawned uWSGI worker 2 (pid: 24, cores: 4)
spawned uWSGI worker 3 (pid: 25, cores: 4)
spawned uWSGI http 1 (pid: 26)
/usr/local/lib/python2.7/site-packages/sentry/runner/initializer.py:197: DeprecatedSettingWarning: The GITHUB_EXTENDED_PERMISSIONS setting is deprecated. Please use SENTRY_OPTIONS[‘github-login.extended-permissions’] instead.
warnings.warn(DeprecatedSettingWarning(options_mapper[k], “SENTRY_OPTIONS[’%s’]” % k))
/usr/local/lib/python2.7/site-packages/sentry/runner/initializer.py:197: DeprecatedSettingWarning: The GITHUB_EXTENDED_PERMISSIONS setting is deprecated. Please use SENTRY_OPTIONS[‘github-login.extended-permissions’] instead.
warnings.warn(DeprecatedSettingWarning(options_mapper[k], “SENTRY_OPTIONS[’%s’]” % k))
/usr/local/lib/python2.7/site-packages/sentry/runner/initializer.py:197: DeprecatedSettingWarning: The GITHUB_EXTENDED_PERMISSIONS setting is deprecated. Please use SENTRY_OPTIONS[‘github-login.extended-permissions’] instead.
warnings.warn(DeprecatedSettingWarning(options_mapper[k], “SENTRY_OPTIONS[’%s’]” % k))
07:25:51 [WARNING] sentry.utils.geo: settings.GEOIP_PATH_MMDB not configured.
07:25:51 [WARNING] sentry.utils.geo: settings.GEOIP_PATH_MMDB not configured.
07:25:51 [WARNING] sentry.utils.geo: settings.GEOIP_PATH_MMDB not configured.
/usr/local/lib/python2.7/site-packages/OpenSSL/crypto.py:12: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in a future release.
from cryptography import x509
/usr/local/lib/python2.7/site-packages/OpenSSL/crypto.py:12: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in a future release.
from cryptography import x509
/usr/local/lib/python2.7/site-packages/OpenSSL/crypto.py:12: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in a future release.
from cryptography import x509
07:25:57 [INFO] sentry.plugins.github: apps-not-configured
07:25:57 [INFO] sentry.plugins.github: apps-not-configured
07:25:57 [INFO] sentry.plugins.github: apps-not-configured
WSGI app 0 (mountpoint=’’) ready in 7 seconds on interpreter 0x55f2e4923fd0 pid: 23 (default app)
WSGI app 0 (mountpoint=’’) ready in 8 seconds on interpreter 0x55f2e4923fd0 pid: 24 (default app)
WSGI app 0 (mountpoint=’’) ready in 8 seconds on interpreter 0x55f2e4923fd0 pid: 25 (default app)
07:29:08 [WARNING] django.request: Not Found: /_static/1603178751/sentry/js/ads.js (status_code=404 request=<WSGIRequest: GET u’/_static/1603178751/sentry/js/ads.js’>)
07:29:08 [WARNING] django.request: Not Found: /_static/1603178751/sentry/js/ads.js (status_code=404 request=<WSGIRequest: GET u’/_static/1603178751/sentry/js/ads.js’>)
07:29:57 [WARNING] django.request: Not Found: /_static/1603178751/sentry/images/logos/default-organization-logo.png (status_code=404 request=<WSGIRequest: GET u’/_static/1603178751/sentry/images/logos/default-organization-logo.png’>)
07:30:54 [WARNING] django.request: Not Found: /_static/1603178751/sentry/js/ads.js (status_code=404 request=<WSGIRequest: GET u’/_static/1603178751/sentry/js/ads.js’>)
07:31:13 [WARNING] django.request: Not Found: /_static/1603178751/sentry/js/ads.js (status_code=404 request=<WSGIRequest: GET u’/_static/1603178751/sentry/js/ads.js’>)
07:31:57 [WARNING] django.request: Not Found: /_static/1603178751/sentry/js/ads.js (status_code=404 request=<WSGIRequest: GET u’/_static/1603178751/sentry/js/ads.js’>)
07:32:00 [INFO] sentry.mail: mail.queued (message_to=(u’kristaps.dombrovskis@printful.com’,) user_id=124 message_id=u’20201020073159.25.41314@localhost’ message_type=u’user.password_recovery’)
07:32:00 [INFO] sentry.accounts: recover.sent (user_id=124 user_recovered=u’kristaps.dombrovskis@printful.com’ ip_address=u’10.0.2.2’ passwordhash_id=48L user_agent=u’Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36’)
07:32:00 [WARNING] django.request: Not Found: /_static/1603178751/sentry/js/ads.js (status_code=404 request=<WSGIRequest: GET u’/_static/1603178751/sentry/js/ads.js’>)
07:33:16 [WARNING] django.request: Not Found: /_static/1603178751/sentry/js/ads.js (status_code=404 request=<WSGIRequest: GET u’/_static/1603178751/sentry/js/ads.js’>)
07:33:45 [WARNING] django.request: Not Found: /_static/1603178751/sentry/js/ads.js (status_code=404 request=<WSGIRequest: GET u’/_static/1603178751/sentry/js/ads.js’>)
07:34:13 [WARNING] django.request: Not Found: /_static/1603178751/sentry/js/ads.js (status_code=404 request=<WSGIRequest: GET u’/_static/1603178751/sentry/js/ads.js’>)
07:34:26 [WARNING] django.request: Not Found: /_static/1603178751/sentry/js/ads.js (status_code=404 request=<WSGIRequest: GET u’/_static/1603178751/sentry/js/ads.js’>)
07:43:24 [WARNING] django.request: Not Found: /_static/1603178751/sentry/js/ads.js (status_code=404 request=<WSGIRequest: GET u’/_static/1603178751/sentry/js/ads.js’>)
07:43:25 [WARNING] django.request: Not Found: /_static/1603178751/sentry/images/logos/default-organization-logo.png (status_code=404 request=<WSGIRequest: GET u’/_static/1603178751/sentry/images/logos/default-organization-logo.png’>)
07:43:25 [WARNING] django.request: Not Found: /_static/1603178751/sentry/js/ads.js (status_code=404 request=<WSGIRequest: GET u’/_static/1603178751/sentry/js/ads.js’>)
07:44:23 [WARNING] django.request: Not Found: /_static/1603178751/sentry/js/ads.js (status_code=404 request=<WSGIRequest: GET u’/_static/1603178751/sentry/js/ads.js’>)
07:44:23 [WARNING] django.request: Not Found: /_static/1603178751/sentry/js/ads.js (status_code=404 request=<WSGIRequest: GET u’/_static/1603178751/sentry/js/ads.js’>)
07:44:38 [WARNING] django.request: Not Found: /_static/1603178751/sentry/js/ads.js (status_code=404 request=<WSGIRequest: GET u’/_static/1603178751/sentry/js/ads.js’>)
07:44:38 [WARNING] django.request: Not Found: /_static/1603178751/sentry/js/ads.js (status_code=404 request=<WSGIRequest: GET u’/_static/1603178751/sentry/js/ads.js’>)
07:44:48 [WARNING] django.request: Not Found: /_static/1603178751/sentry/js/ads.js (status_code=404 request=<WSGIRequest: GET u’/_static/1603178751/sentry/js/ads.js’>)
07:47:46 [WARNING] django.request: Not Found: /_static/1603178751/sentry/js/ads.js (status_code=404 request=<WSGIRequest: GET u’/_static/1603178751/sentry/js/ads.js’>)
07:47:48 [INFO] sentry.mail: mail.queued (message_to=(u’kristaps.dombrovskis@printful.com’,) user_id=124 message_id=u’20201020074748.25.66673@localhost’ message_type=u’user.password_recovery’)
07:47:48 [INFO] sentry.accounts: recover.sent (user_id=124 user_recovered=u’kristaps.dombrovskis@printful.com’ ip_address=u’10.0.2.2’ passwordhash_id=48L user_agent=u’Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36’)
07:47:48 [WARNING] django.request: Not Found: /_static/1603178751/sentry/js/ads.js (status_code=404 request=<WSGIRequest: GET u’/_static/1603178751/sentry/js/ads.js’>)
08:28:53 [WARNING] django.request: Not Found: /_static/1603178751/sentry/js/ads.js (status_code=404 request=<WSGIRequest: GET u’/_static/1603178751/sentry/js/ads.js’>)
08:28:53 [WARNING] django.request: Not Found: /_static/1603178751/sentry/images/logos/default-organization-logo.png (status_code=404 request=<WSGIRequest: GET u’/_static/1603178751/sentry/images/logos/default-organization-logo.png’>)
08:28:54 [WARNING] django.request: Not Found: /_static/1603178751/sentry/js/ads.js (status_code=404 request=<WSGIRequest: GET u’/_static/1603178751/sentry/js/ads.js’>)
08:29:31 [WARNING] django.request: Not Found: /_static/1603178751/sentry/js/ads.js (status_code=404 request=<WSGIRequest: GET u’/_static/1603178751/sentry/js/ads.js’>)
[uwsgi-http key: 127.0.0.1 client_addr: 172.23.0.25 client_port: 56970] hr_write(): Broken pipe [plugins/http/http.c line 565]
08:29:37 [WARNING] django.request: Not Found: /_static/1603178751/sentry/js/ads.js (status_code=404 request=<WSGIRequest: GET u’/_static/1603178751/sentry/js/ads.js’>)
08:29:59 [WARNING] django.request: Not Found: /_static/1603178751/sentry/js/ads.js (status_code=404 request=<WSGIRequest: GET u’/_static/1603178751/sentry/js/ads.js’>)
[root@jumphost onpremise]#

[root@jumphost volumes]# docker logs sentry_onpremise_sentry-cleanup_1
SHELL=/bin/bash
BASH_ENV=/container.env
0 0 * * * gosu sentry sentry cleanup --days 90 > /proc/1/fd/1 2>/proc/1/fd/2
[root@jumphost volumes]# docker logs sentry_onpremise_snuba-sessions-consumer_1

  • ‘[’ c = - ‘]’
  • snuba consumer --help
  • set – snuba consumer --storage sessions_raw --auto-offset-reset=latest --max-batch-time-ms 750
  • set gosu snuba snuba consumer --storage sessions_raw --auto-offset-reset=latest --max-batch-time-ms 750
  • exec gosu snuba snuba consumer --storage sessions_raw --auto-offset-reset=latest --max-batch-time-ms 750
    %3|1603178744.922|FAIL|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.23.0.10:9092 failed: Connection refused (after 2ms in state CONNECT)
    %3|1603178744.924|FAIL|rdkafka#consumer-2| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.23.0.10:9092 failed: Connection refused (after 1ms in state CONNECT)
    %3|1603178745.922|FAIL|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.23.0.10:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
    %3|1603178745.930|FAIL|rdkafka#consumer-2| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.23.0.10:9092 failed: Connection refused (after 4ms in state CONNECT, 1 identical error(s) suppressed)
    2020-10-20 07:25:56,638 New partitions assigned: {Partition(topic=Topic(name=‘ingest-sessions’), index=0): 0}
    [root@jumphost volumes]#

[root@jumphost onpremise]# docker logs sentry_onpremise_kafka_1
===> ENV Variables …
ALLOW_UNSIGNED=false
COMPONENT=kafka
CONFLUENT_DEB_VERSION=1
CONFLUENT_PLATFORM_LABEL=
CONFLUENT_SUPPORT_METRICS_ENABLE=false
CONFLUENT_VERSION=5.5.0
CUB_CLASSPATH=/etc/confluent/docker/docker-utils.jar
HOME=/root
HOSTNAME=6eb1a4ea1869
KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092
KAFKA_LOG4J_LOGGERS=kafka.cluster=WARN,kafka.controller=WARN,kafka.coordinator=WARN,kafka.log=WARN,kafka.server=WARN,kafka.zookeeper=WARN,state.change.logger=WARN
KAFKA_LOG4J_ROOT_LOGLEVEL=WARN
KAFKA_LOG_RETENTION_HOURS=24
KAFKA_MAX_REQUEST_SIZE=50000000
KAFKA_MESSAGE_MAX_BYTES=50000000
KAFKA_OFFSETS_TOPIC_NUM_PARTITIONS=1
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1
KAFKA_TOOLS_LOG4J_LOGLEVEL=WARN
KAFKA_VERSION=
KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
LANG=C.UTF-8
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/
PYTHON_PIP_VERSION=8.1.2
PYTHON_VERSION=2.7.9-1
SCALA_VERSION=2.12
SHLVL=1
ZULU_OPENJDK_VERSION=8=8.38.0.13
_=/usr/bin/env
===> User
uid=0(root) gid=0(root) groups=0(root)
===> Configuring …
===> Running preflight checks …
===> Check if /var/lib/kafka/data is writable …
===> Check if Zookeeper is healthy …
[main] INFO org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.5.7-f0fdd52973d373ffd9c86b81d99842dc2c7f660e, built on 02/10/2020 11:30 GMT
[main] INFO org.apache.zookeeper.ZooKeeper - Client environment:host.name=6eb1a4ea1869
[main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.8.0_212
[main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Azul Systems, Inc.
[main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/zulu-8-amd64/jre
[main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/etc/confluent/docker/docker-utils.jar
[main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
[main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp
[main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=
[main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux
[main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64
[main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.version=3.10.0-1127.19.1.el7.x86_64
[main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.name=root
[main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.home=/root
[main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/
[main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.free=144MB
[main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.max=2172MB
[main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.total=148MB
[main] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=zookeeper:2181 sessionTimeout=40000 watcher=io.confluent.admin.utils.ZookeeperConnectionWatcher@cc34f4d
[main] INFO org.apache.zookeeper.common.X509Util - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation
[main] INFO org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 4194304 Bytes
[main] INFO org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=
[main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/172.23.0.2:2181. Will not attempt to authenticate using SASL (unknown error)
[main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /172.23.0.10:40402, server: zookeeper/172.23.0.2:2181
[main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server zookeeper/172.23.0.2:2181, sessionid = 0x10002d7c6e80000, negotiated timeout = 40000
[main] INFO org.apache.zookeeper.ZooKeeper - Session: 0x10002d7c6e80000 closed
[main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x10002d7c6e80000
===> Launching …
===> Launching kafka …
[2020-10-20 07:25:48,644] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2020-10-20 07:25:49,489] WARN The package io.confluent.support.metrics.collectors.FullCollector for collecting the full set of support metrics could not be loaded, so we are reverting to anonymous, basic metric collection. If you are a Confluent customer, please refer to the Confluent Platform documentation, section Proactive Support, on how to activate full metrics collection. (io.confluent.support.metrics.KafkaSupportConfig)
[2020-10-20 07:25:49,489] WARN The support metrics collection feature (“Metrics”) of Proactive Support is disabled. (io.confluent.support.metrics.SupportedServerStartable)
[2020-10-20 07:25:50,817] INFO Starting the log cleaner (kafka.log.LogCleaner)
[2020-10-20 07:25:50,903] INFO [kafka-log-cleaner-thread-0]: Starting (kafka.log.LogCleaner)
[2020-10-20 07:25:51,464] INFO Awaiting socket connections on 0.0.0.0:9092. (kafka.network.Acceptor)
[2020-10-20 07:25:51,529] INFO [SocketServer brokerId=1001] Created data-plane acceptor and processors for endpoint : EndPoint(0.0.0.0,9092,ListenerName(PLAINTEXT),PLAINTEXT) (kafka.network.SocketServer)
[2020-10-20 07:25:51,534] INFO [SocketServer brokerId=1001] Started 1 acceptor threads for data-plane (kafka.network.SocketServer)
[2020-10-20 07:25:51,683] INFO Creating /brokers/ids/1001 (is it secure? false) (kafka.zk.KafkaZkClient)
[2020-10-20 07:25:51,748] INFO Stat of the created znode at /brokers/ids/1001 is: 413,413,1603178751724,1603178751724,1,0,0,72060719816245249,180,0,413
(kafka.zk.KafkaZkClient)
[2020-10-20 07:25:51,749] INFO Registered broker 1001 at path /brokers/ids/1001 with addresses: ArrayBuffer(EndPoint(kafka,9092,ListenerName(PLAINTEXT),PLAINTEXT)), czxid (broker epoch): 413 (kafka.zk.KafkaZkClient)
[2020-10-20 07:25:52,184] INFO [/config/changes-event-process-thread]: Starting (kafka.common.ZkNodeChangeNotificationListener$ChangeEventProcessThread)
[2020-10-20 07:25:52,398] INFO [SocketServer brokerId=1001] Started data-plane processors for 1 acceptors (kafka.network.SocketServer)
[root@jumphost onpremise]#

i copy/pasted all containers logs where i found some errors rest of containers looks more ore less ok, maybe it will give you some clues…