[21.3.1 to 21.4.0] Connection refused :9000

Hi,

I am currently trying to upgrade to version 21.4.0 on my sentry selfhosted, I am experiencing a problem when upgrading… UI keeps telling me that the connection is refused

Docker-compose logs shows error in the clickhouse container

Processing configuration file '/etc/clickhouse-server/config.xml'.
Merging configuration file '/etc/clickhouse-server/config.d/docker_related_config.xml'.
Merging configuration file '/etc/clickhouse-server/config.d/sentry.xml'.
Include not found: clickhouse_remote_servers
Include not found: clickhouse_compression
Logging information to /var/log/clickhouse-server/clickhouse-server.log
Logging errors to /var/log/clickhouse-server/clickhouse-server.err.log
Logging information to console
2021.09.14 10:22:36.732410 [ 1 ] {} <Information> : Starting ClickHouse 20.3.9.70 with revision 54433
2021.09.14 10:22:36.735185 [ 1 ] {} <Information> Application: starting up
Include not found: networks
2021.09.14 10:22:36.742885 [ 1 ] {} <Information> Application: Uncompressed cache size was lowered to 7.76 GiB because the system has low amount of memory
2021.09.14 10:22:36.743112 [ 1 ] {} <Information> Application: Loading metadata from /var/lib/clickhouse/
2021.09.14 10:22:36.745242 [ 1 ] {} <Information> DatabaseOrdinary (default): Total 0 tables and 0 dictionaries.
2021.09.14 10:22:36.745281 [ 1 ] {} <Information> DatabaseOrdinary (default): Starting up tables.
2021.09.14 10:22:36.745437 [ 1 ] {} <Information> BackgroundSchedulePool: Create BackgroundSchedulePool with 16 threads
2021.09.14 10:22:36.754220 [ 1 ] {} <Information> Application: It looks like the process has no CAP_NET_ADMIN capability, 'taskstats' performance statistics will be disabled. It could happen due to incorrect ClickHouse package installation. You could resolve the problem manually with 'sudo setcap cap_net_admin=+ep /usr/bin/clickhouse'. Note that it will not work on 'nosuid' mounted filesystems. It also doesn't work if you run clickhouse-server inside network namespace as it happens in some containers.
2021.09.14 10:22:36.754296 [ 1 ] {} <Information> Application: It looks like the process has no CAP_SYS_NICE capability, the setting 'os_thread_nice' will have no effect. It could happen due to incorrect ClickHouse package installation. You could resolve the problem manually with 'sudo setcap cap_sys_nice=+ep /usr/bin/clickhouse'. Note that it will not work on 'nosuid' mounted filesystems.
2021.09.14 10:22:36.755025 [ 1 ] {} <Information> Application: Listening for http://0.0.0.0:8123
2021.09.14 10:22:36.755117 [ 1 ] {} <Information> Application: Listening for connections with native protocol (tcp): 0.0.0.0:9000
2021.09.14 10:22:36.755166 [ 1 ] {} <Information> Application: Listening for replica communication (interserver): http://0.0.0.0:9009
2021.09.14 10:22:37.031846 [ 1 ] {} <Information> Application: Listening for MySQL compatibility protocol: 0.0.0.0:9004
2021.09.14 10:22:37.032294 [ 1 ] {} <Error> Application: Listen [0.0.0.0]:8123 failed: Poco::Exception. Code: 1000, e.code() = 98, e.displayText() = Net Exception: Address already in use: 0.0.0.0:8123 (version 20.3.9.70 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2021.09.14 10:22:37.032595 [ 1 ] {} <Error> Application: Listen [0.0.0.0]:9000 failed: Poco::Exception. Code: 1000, e.code() = 98, e.displayText() = Net Exception: Address already in use: 0.0.0.0:9000 (version 20.3.9.70 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2021.09.14 10:22:37.032872 [ 1 ] {} <Error> Application: Listen [0.0.0.0]:9009 failed: Poco::Exception. Code: 1000, e.code() = 98, e.displayText() = Net Exception: Address already in use: 0.0.0.0:9009 (version 20.3.9.70 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2021.09.14 10:22:37.033140 [ 1 ] {} <Error> Application: Listen [0.0.0.0]:9004 failed: Poco::Exception. Code: 1000, e.code() = 98, e.displayText() = Net Exception: Address already in use: 0.0.0.0:9004 (version 20.3.9.70 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2021.09.14 10:22:37.033848 [ 1 ] {} <Information> Application: Available RAM: 15.53 GiB; physical cores: 4; logical cores: 4.
2021.09.14 10:22:37.033878 [ 1 ] {} <Information> Application: Ready for connections.
Include not found: clickhouse_remote_servers
Include not found: clickhouse_compression
2021.09.14 10:22:44.248805 [ 51 ] {} <Information> BackgroundProcessingPool: Create BackgroundProcessingPool with 16 threads
2021.09.14 10:23:12.097990 [ 71 ] {9f8aa0dc-86e0-4a3c-895d-6a54400d61e8} <Information> executeQuery: Read 1 rows, 1.00 B in 0.001 sec., 852 rows/sec., 852.60 B/sec.
2021.09.14 10:23:12.098209 [ 71 ] {} <Information> TCPHandler: Processed in 0.002 sec.
2021.09.14 10:23:12.141068 [ 71 ] {643108b6-c426-40a0-be75-85b83b10a3a0} <Error> executeQuery: Code: 60, e.displayText() = DB::Exception: Table default.migrations_local doesn't exist. (version 20.3.9.70 (official build)) (from 172.18.0.5:60560) (in query: SELECT group, migration_id, status FROM migrations_local FINAL WHERE group IN ('system', 'events', 'transactions', 'discover', 'outcomes', 'sessions')), Stack trace (when copying this message, always include the lines below):

0. Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x105351b0 in /usr/bin/clickhouse
1. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x8f4172d in /usr/bin/clickhouse
2. DB::Context::getTableImpl(DB::StorageID const&, std::__1::optional<DB::Exception>*) const @ 0xcfe2a24 in /usr/bin/clickhouse
3. DB::Context::getTable(DB::StorageID const&) const @ 0xcfe2bbb in /usr/bin/clickhouse
4. DB::Context::getTable(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) const @ 0xcfe2c7d in /usr/bin/clickhouse
5. DB::JoinedTables::getLeftTableStorage() @ 0xd454892 in /usr/bin/clickhouse
6. DB::InterpreterSelectQuery::InterpreterSelectQuery(std::__1::shared_ptr<DB::IAST> const&, DB::Context const&, std::__1::shared_ptr<DB::IBlockInputStream> const&, std::__1::optional<DB::Pipe>, std::__1::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) @ 0xd13b6d1 in /usr/bin/clickhouse
7. DB::InterpreterSelectQuery::InterpreterSelectQuery(std::__1::shared_ptr<DB::IAST> const&, DB::Context const&, DB::SelectQueryOptions const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) @ 0xd13c619 in /usr/bin/clickhouse
8. DB::InterpreterSelectWithUnionQuery::InterpreterSelectWithUnionQuery(std::__1::shared_ptr<DB::IAST> const&, DB::Context const&, DB::SelectQueryOptions const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) @ 0xd341686 in /usr/bin/clickhouse
9. DB::InterpreterFactory::get(std::__1::shared_ptr<DB::IAST>&, DB::Context&, DB::QueryProcessingStage::Enum) @ 0xd0909b4 in /usr/bin/clickhouse
10. ? @ 0xd550655 in /usr/bin/clickhouse
11. DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::Context&, bool, DB::QueryProcessingStage::Enum, bool, bool) @ 0xd553441 in /usr/bin/clickhouse
12. DB::TCPHandler::runImpl() @ 0x9024489 in /usr/bin/clickhouse
13. DB::TCPHandler::run() @ 0x9025470 in /usr/bin/clickhouse
14. Poco::Net::TCPServerConnection::start() @ 0xe3ac69b in /usr/bin/clickhouse
15. Poco::Net::TCPServerDispatcher::run() @ 0xe3acb1d in /usr/bin/clickhouse
16. Poco::PooledThread::run() @ 0x105c3317 in /usr/bin/clickhouse
17. Poco::ThreadImpl::runnableEntry(void*) @ 0x105bf11c in /usr/bin/clickhouse
18. ? @ 0x105c0abd in /usr/bin/clickhouse
19. start_thread @ 0x76db in /lib/x86_64-linux-gnu/libpthread-2.27.so
20. __clone @ 0x12188f in /lib/x86_64-linux-gnu/libc-2.27.so

2021.09.14 10:23:12.141457 [ 71 ] {} <Information> TCPHandler: Processed in 0.043 sec.
2021.09.14 10:23:12.142026 [ 71 ] {} <Information> TCPHandler: Done processing connection.
2021.09.14 10:23:12.151205 [ 71 ] {} <Information> TCPHandler: Processed in 0.006 sec.
2021.09.14 10:23:12.153359 [ 71 ] {} <Information> TCPHandler: Processed in 0.001 sec.
2021.09.14 10:23:12.176161 [ 71 ] {} <Information> TCPHandler: Processed in 0.022 sec.
2021.09.14 10:23:12.181596 [ 71 ] {} <Information> TCPHandler: Processed in 0.002 sec.
2021.09.14 10:23:12.184225 [ 71 ] {} <Information> TCPHandler: Processed in 0.002 sec.
2021.09.14 10:23:12.193924 [ 71 ] {} <Information> TCPHandler: Processed in 0.009 sec.
2021.09.14 10:23:12.196349 [ 71 ] {5c3815eb-b87c-465d-b893-65632dfcb1ab} <Information> executeQuery: Read 1 rows, 51.00 B in 0.001 sec., 694 rows/sec., 34.58 KiB/sec.
2021.09.14 10:23:12.196545 [ 71 ] {} <Information> TCPHandler: Processed in 0.002 sec.
2021.09.14 10:23:12.198590 [ 71 ] {} <Information> TCPHandler: Processed in 0.002 sec.
2021.09.14 10:23:12.201208 [ 71 ] {} <Information> TCPHandler: Processed in 0.001 sec.
2021.09.14 10:23:12.203564 [ 71 ] {} <Information> TCPHandler: Processed in 0.002 sec.
2021.09.14 10:23:12.212505 [ 71 ] {} <Information> TCPHandler: Processed in 0.008 sec.
2021.09.14 10:23:12.220372 [ 71 ] {} <Information> TCPHandler: Processed in 0.007 sec.
2021.09.14 10:23:12.229642 [ 71 ] {} <Information> TCPHandler: Processed in 0.009 sec.
2021.09.14 10:23:12.237764 [ 71 ] {} <Information> TCPHandler: Processed in 0.007 sec.
2021.09.14 10:23:12.245815 [ 71 ] {} <Information> TCPHandler: Processed in 0.007 sec.

clickhouse/config.xml

<yandex>
    <listen_host>0.0.0.0</listen_host>
</yandex>

I can also find a strange behavious in the install logs

  Applying sentry.0157_make_repositoryprojectpathconfig_organization_integration_nullable... OK
  Applying sentry.0158_create_externalteam_table... OK
  Applying sentry.0159_create_externaluser_table... OK
  Applying sentry.0160_create_projectcodeowners_table... OK
  Applying sentry.0161_add_saved_search_sort... OK
  Applying sentry.0162_backfill_saved_search_sort... OK
  Applying sentry.0163_add_organizationmember_and_external_name... OK
  Applying sentry.0164_add_protect_on_delete_codeowners... OK
  Applying sentry.0165_metric_alerts_fix_group_ids... OK
  Applying sentry.0166_create_notificationsetting_table... OK
  Applying sentry.0167_rm_organization_integration_from_projectcodeowners... OK
  Applying sentry.0168_demo_orgs_users... OK
  Applying sentry.0169_delete_organization_integration_from_projectcodeowners... OK
  Applying sentry.0170_actor_introduction... OK
  Applying sentry.0171_backfill_actors... OK
  Applying sentry.0172_rule_owner_fields... OK
  Applying sentry.0173_remove_demo_flag... OK
  Applying sentry.0174_my_issues_saved_search... OK
  Applying sentry.0175_make_targets_nullable... OK
  Applying sentry.0176_remove_targets... OK
  Applying sentry.0177_drop_targets... OK
  Applying sentry.0178_add_new_target_column... OK
  Applying sentry.0179_update_legacy_discover_saved_query_timestamps... OK
  Applying sentry.0180_add_saved_search_sorts... OK
  Applying sentry.0181_copy_useroptions_to_notificationsettings... OK
  Applying sentry.0182_update_user_misery_on_saved_queries... OK
  Applying sentry.0183_make_codemapping_unique_on_projectcodeowners... OK
  Applying sentry.0184_copy_useroptions_to_notificationsettings_2... OK
  Applying sentry.0185_rm_copied_useroptions... OK
  Applying sentry.0186_add_externalactor... OK
  Applying sessions.0001_initial... OK
  Applying sites.0001_initial... OK
  Applying sites.0002_alter_domain_unique... OK
  Applying social_auth.0001_initial... OK
16:28:23 [WARNING] sentry: Cannot initiate onboarding for organization (1) due to missing owners
Created internal Sentry project (slug=internal, id=1)

Would you like to create a user account now? [Y/n]: Aborted!
An error occurred, caught SIGERR on line 12
Cleaning up...

Session terminated, killing shell... ...killed.
Terminated

Do you have an idea ? :confused:

Looks like it cannot bind to the ports. Maybe you have a stray container instance running in the background?

Everything seems to start normaly, the install setup is initialized

sentry_install_log_xx.txt
▶ Parsing command line ...

▶ Setting up error handling ...

▶ Checking minimum requirements ...

▶ Creating volumes for persistent storage ...
Created sentry-clickhouse.
Created sentry-data.
Created sentry-kafka.
Created sentry-symbolicator.
Created sentry-zookeeper.

▶ Ensuring files from examples ...
../sentry/sentry.conf.py already exists, skipped creation.
../sentry/config.yml already exists, skipped creation.
Creating ../symbolicator/config.yml...
../sentry/requirements.txt already exists, skipped creation.

▶ Generating secret key ...
Secret key written to ../sentry/config.yml

▶ Replacing TSDB ...

▶ Fetching and updating Docker images ...
Some service image(s) must be built from source by running:
    docker-compose build sentry-cleanup ingest-consumer worker post-process-forwarder subscription-consumer-events web subscription-consumer-transactions snuba-cleanup snuba-transactions-cleanup symbolicator-cleanup cron
21.4.0: Pulling from getsentry/sentry
f7ec5a41d630: Already exists
6f318b7b4c32: Already exists
16da2ad15a23: Pulling fs layer
14bd3e3eee91: Pulling fs layer
e1449a280109: Pulling fs layer
8d161b0565c9: Pulling fs layer
f38f866b6e9b: Pulling fs layer
cf1ff04b89f5: Pulling fs layer
f9e06240fa5d: Pulling fs layer
2b2c9b2db809: Pulling fs layer
572d87935add: Pulling fs layer
791f85f01834: Pulling fs layer
c511d6ea211c: Pulling fs layer
c02326d64e65: Pulling fs layer
a195168eef29: Pulling fs layer
f9e06240fa5d: Waiting
2b2c9b2db809: Waiting
572d87935add: Waiting
791f85f01834: Waiting
a195168eef29: Waiting
c511d6ea211c: Waiting
c02326d64e65: Waiting
8d161b0565c9: Waiting
f38f866b6e9b: Waiting
cf1ff04b89f5: Waiting
14bd3e3eee91: Verifying Checksum
14bd3e3eee91: Download complete
e1449a280109: Verifying Checksum
e1449a280109: Download complete
16da2ad15a23: Download complete
8d161b0565c9: Verifying Checksum
8d161b0565c9: Download complete
f38f866b6e9b: Verifying Checksum
f38f866b6e9b: Download complete
cf1ff04b89f5: Verifying Checksum
cf1ff04b89f5: Download complete
16da2ad15a23: Pull complete
14bd3e3eee91: Pull complete
2b2c9b2db809: Verifying Checksum
2b2c9b2db809: Download complete
572d87935add: Verifying Checksum
572d87935add: Download complete
e1449a280109: Pull complete
8d161b0565c9: Pull complete
791f85f01834: Verifying Checksum
791f85f01834: Download complete
f38f866b6e9b: Pull complete
c511d6ea211c: Verifying Checksum
c511d6ea211c: Download complete
cf1ff04b89f5: Pull complete
c02326d64e65: Verifying Checksum
c02326d64e65: Download complete
a195168eef29: Verifying Checksum
a195168eef29: Download complete
f9e06240fa5d: Verifying Checksum
f9e06240fa5d: Download complete
f9e06240fa5d: Pull complete
2b2c9b2db809: Pull complete
572d87935add: Pull complete
791f85f01834: Pull complete
c511d6ea211c: Pull complete
c02326d64e65: Pull complete
a195168eef29: Pull complete
Digest: sha256:f795f813ee1a52bbf9cd3bd8b9dceb4053a4dd300dc47ba8296b1e3ed215ab8f
Status: Downloaded newer image for getsentry/sentry:21.4.0
docker.io/getsentry/sentry:21.4.0

▶ Building and tagging Docker images ...

memcached uses an image, skipping
zookeeper uses an image, skipping
kafka uses an image, skipping
clickhouse uses an image, skipping
snuba-api uses an image, skipping
snuba-consumer uses an image, skipping
snuba-outcomes-consumer uses an image, skipping
snuba-sessions-consumer uses an image, skipping
snuba-transactions-consumer uses an image, skipping
snuba-replacer uses an image, skipping
snuba-subscription-consumer-events uses an image, skipping
snuba-subscription-consumer-transactions uses an image, skipping
symbolicator uses an image, skipping
relay uses an image, skipping
nginx uses an image, skipping
Building snuba-cleanup
Step 1/5 : ARG BASE_IMAGE
Step 2/5 : FROM ${BASE_IMAGE}
 ---> 3bde1fc93fb0
Step 3/5 : RUN apt-get update && apt-get install -y --no-install-recommends cron &&     rm -r /var/lib/apt/lists/*
 ---> Running in 77bd6dd91930
Get:1 http://deb.debian.org/debian buster InRelease [122 kB]
Get:2 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]
Get:3 http://deb.debian.org/debian buster-updates InRelease [51.9 kB]
Get:4 http://deb.debian.org/debian buster/main amd64 Packages [7907 kB]
Get:5 http://deb.debian.org/debian buster-updates/main amd64 Packages [15.2 kB]
Get:6 http://security.debian.org/debian-security buster/updates/main amd64 Packages [303 kB]
Fetched 8465 kB in 2s (5371 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
  lsb-base sensible-utils
Suggested packages:
  anacron logrotate checksecurity
Recommended packages:
  default-mta | mail-transport-agent
The following NEW packages will be installed:
  cron lsb-base sensible-utils
0 upgraded, 3 newly installed, 0 to remove and 12 not upgraded.
Need to get 143 kB of archives.
After this operation, 383 kB of additional disk space will be used.
Get:1 http://deb.debian.org/debian buster/main amd64 sensible-utils all 0.0.12 [15.8 kB]
Get:2 http://deb.debian.org/debian buster/main amd64 lsb-base all 10.2019051400 [28.4 kB]
Get:3 http://deb.debian.org/debian buster/main amd64 cron amd64 3.0pl1-134+deb10u1 [99.0 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 143 kB in 0s (0 B/s)
Selecting previously unselected package sensible-utils.
(Reading database ... 6840 files and directories currently installed.)
Preparing to unpack .../sensible-utils_0.0.12_all.deb ...
Unpacking sensible-utils (0.0.12) ...
Selecting previously unselected package lsb-base.
Preparing to unpack .../lsb-base_10.2019051400_all.deb ...
Unpacking lsb-base (10.2019051400) ...
Selecting previously unselected package cron.
Preparing to unpack .../cron_3.0pl1-134+deb10u1_amd64.deb ...
Unpacking cron (3.0pl1-134+deb10u1) ...
Setting up lsb-base (10.2019051400) ...
Setting up sensible-utils (0.0.12) ...
Setting up cron (3.0pl1-134+deb10u1) ...
Adding group `crontab' (GID 101) ...
Done.
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of start.
Removing intermediate container 77bd6dd91930
 ---> abfc1857afed
Step 4/5 : COPY entrypoint.sh /entrypoint.sh
 ---> 0a07cc7cc22d
Step 5/5 : ENTRYPOINT ["/entrypoint.sh"]
 ---> Running in 75d2fc4160f7
Removing intermediate container 75d2fc4160f7
 ---> a701d23d2ee0

Successfully built a701d23d2ee0
Successfully tagged snuba-cleanup-onpremise-local:latest
Building snuba-transactions-cleanup
Step 1/5 : ARG BASE_IMAGE
Step 2/5 : FROM ${BASE_IMAGE}
 ---> 3bde1fc93fb0
Step 3/5 : RUN apt-get update && apt-get install -y --no-install-recommends cron &&     rm -r /var/lib/apt/lists/*
 ---> Using cache
 ---> abfc1857afed
Step 4/5 : COPY entrypoint.sh /entrypoint.sh
 ---> Using cache
 ---> 0a07cc7cc22d
Step 5/5 : ENTRYPOINT ["/entrypoint.sh"]
 ---> Using cache
 ---> a701d23d2ee0

Successfully built a701d23d2ee0
Successfully tagged snuba-cleanup-onpremise-local:latest
Building symbolicator-cleanup
Step 1/5 : ARG BASE_IMAGE
Step 2/5 : FROM ${BASE_IMAGE}
 ---> 77d674e404a3
Step 3/5 : RUN apt-get update && apt-get install -y --no-install-recommends cron &&     rm -r /var/lib/apt/lists/*
 ---> Running in 0bcabc17b08c
Ign:1 http://deb.debian.org/debian stretch InRelease
Get:2 http://deb.debian.org/debian stretch-updates InRelease [93.6 kB]
Get:3 http://security.debian.org/debian-security stretch/updates InRelease [53.0 kB]
Get:4 http://deb.debian.org/debian stretch Release [118 kB]
Get:5 http://deb.debian.org/debian stretch Release.gpg [3177 B]
Get:6 http://security.debian.org/debian-security stretch/updates/main amd64 Packages [711 kB]
Get:7 http://deb.debian.org/debian stretch/main amd64 Packages [7080 kB]
Fetched 8058 kB in 1s (5262 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
Suggested packages:
  anacron logrotate checksecurity
Recommended packages:
  exim4 | postfix | mail-transport-agent
The following NEW packages will be installed:
  cron
0 upgraded, 1 newly installed, 0 to remove and 9 not upgraded.
Need to get 95.4 kB of archives.
After this operation, 257 kB of additional disk space will be used.
Get:1 http://deb.debian.org/debian stretch/main amd64 cron amd64 3.0pl1-128+deb9u1 [95.4 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 95.4 kB in 0s (0 B/s)
Selecting previously unselected package cron.
(Reading database ... 6661 files and directories currently installed.)
Preparing to unpack .../cron_3.0pl1-128+deb9u1_amd64.deb ...
Unpacking cron (3.0pl1-128+deb9u1) ...
Setting up cron (3.0pl1-128+deb9u1) ...
Adding group `crontab' (GID 101) ...
Done.
update-rc.d: warning: start and stop actions are no longer supported; falling back to defaults
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of start.
Removing intermediate container 0bcabc17b08c
 ---> 74437ed68e51
Step 4/5 : COPY entrypoint.sh /entrypoint.sh
 ---> 2625f325e9b4
Step 5/5 : ENTRYPOINT ["/entrypoint.sh"]
 ---> Running in 3f630de8d56f
Removing intermediate container 3f630de8d56f
 ---> d42c20975caf

Successfully built d42c20975caf
Successfully tagged symbolicator-cleanup-onpremise-local:latest
Building web
Step 1/6 : ARG SENTRY_IMAGE
Step 2/6 : ARG SENTRY_PYTHON2
Step 3/6 : FROM ${SENTRY_IMAGE}${SENTRY_PYTHON2:+-py2}
 ---> b679e17d6454
Step 4/6 : COPY . /usr/src/sentry
 ---> 84ea3b11203d
Step 5/6 : COPY ./sentry_auth_gitlab /usr/src/sentry/sentry_auth_gitlab
 ---> 4942a134e3af
Step 6/6 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then pip install -r /usr/src/sentry/requirements.txt; fi
 ---> Running in 11b6b80ab5bf
Processing /usr/src/sentry/sentry_auth_gitlab
Building wheels for collected packages: sentry-auth-gitlab
  Building wheel for sentry-auth-gitlab (setup.py): started
  Building wheel for sentry-auth-gitlab (setup.py): finished with status 'done'
  Created wheel for sentry-auth-gitlab: filename=sentry_auth_gitlab-1.0-py3-none-any.whl size=4051 sha256=30ae15b3c252b0e777bc70d2a5a174726dfd77b2d20ac287f5aa1e50d7dbd32f
  Stored in directory: /tmp/pip-ephem-wheel-cache-x7phyc4g/wheels/36/fc/ae/eaa95ba7813d3fc0b7c460f55d26da9b02200659bc719979bc
Successfully built sentry-auth-gitlab
Installing collected packages: sentry-auth-gitlab
Successfully installed sentry-auth-gitlab-1.0
Removing intermediate container 11b6b80ab5bf
 ---> a7910dfa5d43

Successfully built a7910dfa5d43
Successfully tagged sentry-onpremise-local:latest
Building cron
Step 1/6 : ARG SENTRY_IMAGE
Step 2/6 : ARG SENTRY_PYTHON2
Step 3/6 : FROM ${SENTRY_IMAGE}${SENTRY_PYTHON2:+-py2}
 ---> b679e17d6454
Step 4/6 : COPY . /usr/src/sentry
 ---> Using cache
 ---> 84ea3b11203d
Step 5/6 : COPY ./sentry_auth_gitlab /usr/src/sentry/sentry_auth_gitlab
 ---> Using cache
 ---> 4942a134e3af
Step 6/6 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then pip install -r /usr/src/sentry/requirements.txt; fi
 ---> Using cache
 ---> a7910dfa5d43

Successfully built a7910dfa5d43
Successfully tagged sentry-onpremise-local:latest
Building worker
Step 1/6 : ARG SENTRY_IMAGE
Step 2/6 : ARG SENTRY_PYTHON2
Step 3/6 : FROM ${SENTRY_IMAGE}${SENTRY_PYTHON2:+-py2}
 ---> b679e17d6454
Step 4/6 : COPY . /usr/src/sentry
 ---> Using cache
 ---> 84ea3b11203d
Step 5/6 : COPY ./sentry_auth_gitlab /usr/src/sentry/sentry_auth_gitlab
 ---> Using cache
 ---> 4942a134e3af
Step 6/6 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then pip install -r /usr/src/sentry/requirements.txt; fi
 ---> Using cache
 ---> a7910dfa5d43

Successfully built a7910dfa5d43
Successfully tagged sentry-onpremise-local:latest
Building ingest-consumer
Step 1/6 : ARG SENTRY_IMAGE
Step 2/6 : ARG SENTRY_PYTHON2
Step 3/6 : FROM ${SENTRY_IMAGE}${SENTRY_PYTHON2:+-py2}
 ---> b679e17d6454
Step 4/6 : COPY . /usr/src/sentry
 ---> Using cache
 ---> 84ea3b11203d
Step 5/6 : COPY ./sentry_auth_gitlab /usr/src/sentry/sentry_auth_gitlab
 ---> Using cache
 ---> 4942a134e3af
Step 6/6 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then pip install -r /usr/src/sentry/requirements.txt; fi
 ---> Using cache
 ---> a7910dfa5d43

Successfully built a7910dfa5d43
Successfully tagged sentry-onpremise-local:latest
Building post-process-forwarder
Step 1/6 : ARG SENTRY_IMAGE
Step 2/6 : ARG SENTRY_PYTHON2
Step 3/6 : FROM ${SENTRY_IMAGE}${SENTRY_PYTHON2:+-py2}
 ---> b679e17d6454
Step 4/6 : COPY . /usr/src/sentry
 ---> Using cache
 ---> 84ea3b11203d
Step 5/6 : COPY ./sentry_auth_gitlab /usr/src/sentry/sentry_auth_gitlab
 ---> Using cache
 ---> 4942a134e3af
Step 6/6 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then pip install -r /usr/src/sentry/requirements.txt; fi
 ---> Using cache
 ---> a7910dfa5d43

Successfully built a7910dfa5d43
Successfully tagged sentry-onpremise-local:latest
Building subscription-consumer-events
Step 1/6 : ARG SENTRY_IMAGE
Step 2/6 : ARG SENTRY_PYTHON2
Step 3/6 : FROM ${SENTRY_IMAGE}${SENTRY_PYTHON2:+-py2}
 ---> b679e17d6454
Step 4/6 : COPY . /usr/src/sentry
 ---> Using cache
 ---> 84ea3b11203d
Step 5/6 : COPY ./sentry_auth_gitlab /usr/src/sentry/sentry_auth_gitlab
 ---> Using cache
 ---> 4942a134e3af
Step 6/6 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then pip install -r /usr/src/sentry/requirements.txt; fi
 ---> Using cache
 ---> a7910dfa5d43

Successfully built a7910dfa5d43
Successfully tagged sentry-onpremise-local:latest
Building subscription-consumer-transactions
Step 1/6 : ARG SENTRY_IMAGE
Step 2/6 : ARG SENTRY_PYTHON2
Step 3/6 : FROM ${SENTRY_IMAGE}${SENTRY_PYTHON2:+-py2}
 ---> b679e17d6454
Step 4/6 : COPY . /usr/src/sentry
 ---> Using cache
 ---> 84ea3b11203d
Step 5/6 : COPY ./sentry_auth_gitlab /usr/src/sentry/sentry_auth_gitlab
 ---> Using cache
 ---> 4942a134e3af
Step 6/6 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then pip install -r /usr/src/sentry/requirements.txt; fi
 ---> Using cache
 ---> a7910dfa5d43

Successfully built a7910dfa5d43
Successfully tagged sentry-onpremise-local:latest
Building sentry-cleanup
Step 1/5 : ARG BASE_IMAGE
Step 2/5 : FROM ${BASE_IMAGE}
 ---> a7910dfa5d43
Step 3/5 : RUN apt-get update && apt-get install -y --no-install-recommends cron &&     rm -r /var/lib/apt/lists/*
 ---> Running in 42b49115c2e1
Get:1 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]
Get:2 http://deb.debian.org/debian buster InRelease [122 kB]
Get:3 http://deb.debian.org/debian buster-updates InRelease [51.9 kB]
Get:4 http://security.debian.org/debian-security buster/updates/main amd64 Packages [303 kB]
Get:5 http://deb.debian.org/debian buster/main amd64 Packages [7907 kB]
Get:6 http://deb.debian.org/debian buster-updates/main amd64 Packages [15.2 kB]
Fetched 8465 kB in 2s (5343 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
  lsb-base sensible-utils
Suggested packages:
  anacron logrotate checksecurity
Recommended packages:
  default-mta | mail-transport-agent
The following NEW packages will be installed:
  cron lsb-base sensible-utils
0 upgraded, 3 newly installed, 0 to remove and 23 not upgraded.
Need to get 143 kB of archives.
After this operation, 383 kB of additional disk space will be used.
Get:1 http://deb.debian.org/debian buster/main amd64 sensible-utils all 0.0.12 [15.8 kB]
Get:2 http://deb.debian.org/debian buster/main amd64 lsb-base all 10.2019051400 [28.4 kB]
Get:3 http://deb.debian.org/debian buster/main amd64 cron amd64 3.0pl1-134+deb10u1 [99.0 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 143 kB in 0s (5868 kB/s)
Selecting previously unselected package sensible-utils.
(Reading database ... 11935 files and directories currently installed.)
Preparing to unpack .../sensible-utils_0.0.12_all.deb ...
Unpacking sensible-utils (0.0.12) ...
Selecting previously unselected package lsb-base.
Preparing to unpack .../lsb-base_10.2019051400_all.deb ...
Unpacking lsb-base (10.2019051400) ...
Selecting previously unselected package cron.
Preparing to unpack .../cron_3.0pl1-134+deb10u1_amd64.deb ...
Unpacking cron (3.0pl1-134+deb10u1) ...
Setting up lsb-base (10.2019051400) ...
Setting up sensible-utils (0.0.12) ...
Setting up cron (3.0pl1-134+deb10u1) ...
Adding group `crontab' (GID 101) ...
Done.
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of start.
Removing intermediate container 42b49115c2e1
 ---> aa536a45a649
Step 4/5 : COPY entrypoint.sh /entrypoint.sh
 ---> 43972aba4176
Step 5/5 : ENTRYPOINT ["/entrypoint.sh"]
 ---> Running in d63a002b0b5d
Removing intermediate container d63a002b0b5d
 ---> 7b287c4a5e16

Successfully built 7b287c4a5e16
Successfully tagged sentry-cleanup-onpremise-local:latest

Docker images built.

▶ Turning things off ...
Removing network onpremise_default
Network onpremise_default not found.
Removing network sentry_onpremise_default
Network sentry_onpremise_default not found.

▶ Setting up Zookeeper ...
Creating network "sentry_onpremise_default" with the default driver
Creating volume "sentry_onpremise_sentry-secrets" with default driver
Creating volume "sentry_onpremise_sentry-zookeeper-log" with default driver
Creating volume "sentry_onpremise_sentry-kafka-log" with default driver
Creating volume "sentry_onpremise_sentry-clickhouse-log" with default driver

▶ Bootstrapping and migrating Snuba ...
Creating sentry_onpremise_clickhouse_1 ...
Creating sentry_onpremise_zookeeper_1  ...
Creating sentry_onpremise_zookeeper_1  ... done
Creating sentry_onpremise_kafka_1      ...
Creating sentry_onpremise_clickhouse_1 ... done
Creating sentry_onpremise_kafka_1      ... done
2021-09-14 14:59:44,850 Attempting to connect to Kafka (attempt 0)...
2021-09-14 14:59:46,854 Attempting to connect to Kafka (attempt 1)...
2021-09-14 14:59:46,965 Connected to Kafka on attempt 1
2021-09-14 14:59:47,200 Creating Kafka topics...
Starting sentry_onpremise_zookeeper_1 ...
Starting sentry_onpremise_zookeeper_1 ... done
Starting sentry_onpremise_clickhouse_1 ...
Starting sentry_onpremise_kafka_1      ...
Starting sentry_onpremise_clickhouse_1 ... done
Starting sentry_onpremise_kafka_1      ... done
Finished running migrations

▶ Creating additional Kafka topics ...

▶ Setting up / migrating database ...
Starting sentry_onpremise_clickhouse_1 ...
Creating sentry_onpremise_symbolicator_1 ...
Starting sentry_onpremise_clickhouse_1   ... done
Starting sentry_onpremise_zookeeper_1    ...
Starting sentry_onpremise_zookeeper_1    ... done
Creating sentry_onpremise_memcached_1    ...
Starting sentry_onpremise_kafka_1        ...
Starting sentry_onpremise_kafka_1        ... done
Creating sentry_onpremise_snuba-replacer_1 ...
Creating sentry_onpremise_snuba-transactions-consumer_1 ...
Creating sentry_onpremise_snuba-subscription-consumer-events_1 ...
Creating sentry_onpremise_snuba-subscription-consumer-transactions_1 ...
Creating sentry_onpremise_snuba-consumer_1                           ...
Creating sentry_onpremise_snuba-api_1                                ...
Creating sentry_onpremise_snuba-outcomes-consumer_1                  ...
Creating sentry_onpremise_snuba-sessions-consumer_1                  ...
Creating sentry_onpremise_memcached_1                                ... done
Creating sentry_onpremise_snuba-subscription-consumer-transactions_1 ... done
Creating sentry_onpremise_snuba-replacer_1                           ... done
Creating sentry_onpremise_snuba-sessions-consumer_1                  ... done
Creating sentry_onpremise_snuba-api_1                                ... done
Creating sentry_onpremise_snuba-subscription-consumer-events_1       ... done
Creating sentry_onpremise_snuba-transactions-consumer_1              ... done
Creating sentry_onpremise_snuba-outcomes-consumer_1                  ... done
Creating sentry_onpremise_snuba-consumer_1                           ... done
Creating sentry_onpremise_symbolicator_1                             ... done
15:00:49 [WARNING] sentry.utils.geo: settings.GEOIP_PATH_MMDB not configured.
15:00:53 [INFO] sentry.plugins.github: apps-not-configured
Aborted!
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, jira_ac, nodestore, sentry, sessions, sites, social_auth
Running migrations:
  No migrations to apply.

Would you like to create a user account now? [Y/n]: An error occurred, caught SIGERR on line 12
Cleaning up...

Clickhouse container:

root@13404e72f890:/# ps aux | cat
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
clickho+     1  1.8  0.8 1690068 143856 ?      Ssl  15:10   0:00 /usr/bin/clickhouse-server --config-file=/etc/clickhouse-server/config.xml
root        99  0.8  0.0  20180  3872 pts/0    Ss   15:10   0:00 /bin/bash
root       445  0.0  0.0  36076  3436 pts/0    R+   15:11   0:00 ps aux
root       446  0.0  0.0   6316   780 pts/0    S+   15:11   0:00 cat

And then all containers just disappear and installation process start again and again


Could you be running out of memory somehow? Those permission errors also look weird.

Its strange because there is no issue with the previous version.
The instance size is 16 GiB of RAM & 4 vCPUs (AWS EC2 t3a.xlarge)

I mean, I cannot understand what’s going on your system better than you :slightly_smiling_face: Those containers disappearing/dying is definitely not normal and that seems like the culprit.