Hi,
I am currently trying to upgrade to version 21.4.0 on my sentry selfhosted, I am experiencing a problem when upgrading… UI keeps telling me that the connection is refused
Docker-compose logs shows error in the clickhouse container
Processing configuration file '/etc/clickhouse-server/config.xml'.
Merging configuration file '/etc/clickhouse-server/config.d/docker_related_config.xml'.
Merging configuration file '/etc/clickhouse-server/config.d/sentry.xml'.
Include not found: clickhouse_remote_servers
Include not found: clickhouse_compression
Logging information to /var/log/clickhouse-server/clickhouse-server.log
Logging errors to /var/log/clickhouse-server/clickhouse-server.err.log
Logging information to console
2021.09.14 10:22:36.732410 [ 1 ] {} <Information> : Starting ClickHouse 20.3.9.70 with revision 54433
2021.09.14 10:22:36.735185 [ 1 ] {} <Information> Application: starting up
Include not found: networks
2021.09.14 10:22:36.742885 [ 1 ] {} <Information> Application: Uncompressed cache size was lowered to 7.76 GiB because the system has low amount of memory
2021.09.14 10:22:36.743112 [ 1 ] {} <Information> Application: Loading metadata from /var/lib/clickhouse/
2021.09.14 10:22:36.745242 [ 1 ] {} <Information> DatabaseOrdinary (default): Total 0 tables and 0 dictionaries.
2021.09.14 10:22:36.745281 [ 1 ] {} <Information> DatabaseOrdinary (default): Starting up tables.
2021.09.14 10:22:36.745437 [ 1 ] {} <Information> BackgroundSchedulePool: Create BackgroundSchedulePool with 16 threads
2021.09.14 10:22:36.754220 [ 1 ] {} <Information> Application: It looks like the process has no CAP_NET_ADMIN capability, 'taskstats' performance statistics will be disabled. It could happen due to incorrect ClickHouse package installation. You could resolve the problem manually with 'sudo setcap cap_net_admin=+ep /usr/bin/clickhouse'. Note that it will not work on 'nosuid' mounted filesystems. It also doesn't work if you run clickhouse-server inside network namespace as it happens in some containers.
2021.09.14 10:22:36.754296 [ 1 ] {} <Information> Application: It looks like the process has no CAP_SYS_NICE capability, the setting 'os_thread_nice' will have no effect. It could happen due to incorrect ClickHouse package installation. You could resolve the problem manually with 'sudo setcap cap_sys_nice=+ep /usr/bin/clickhouse'. Note that it will not work on 'nosuid' mounted filesystems.
2021.09.14 10:22:36.755025 [ 1 ] {} <Information> Application: Listening for http://0.0.0.0:8123
2021.09.14 10:22:36.755117 [ 1 ] {} <Information> Application: Listening for connections with native protocol (tcp): 0.0.0.0:9000
2021.09.14 10:22:36.755166 [ 1 ] {} <Information> Application: Listening for replica communication (interserver): http://0.0.0.0:9009
2021.09.14 10:22:37.031846 [ 1 ] {} <Information> Application: Listening for MySQL compatibility protocol: 0.0.0.0:9004
2021.09.14 10:22:37.032294 [ 1 ] {} <Error> Application: Listen [0.0.0.0]:8123 failed: Poco::Exception. Code: 1000, e.code() = 98, e.displayText() = Net Exception: Address already in use: 0.0.0.0:8123 (version 20.3.9.70 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2021.09.14 10:22:37.032595 [ 1 ] {} <Error> Application: Listen [0.0.0.0]:9000 failed: Poco::Exception. Code: 1000, e.code() = 98, e.displayText() = Net Exception: Address already in use: 0.0.0.0:9000 (version 20.3.9.70 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2021.09.14 10:22:37.032872 [ 1 ] {} <Error> Application: Listen [0.0.0.0]:9009 failed: Poco::Exception. Code: 1000, e.code() = 98, e.displayText() = Net Exception: Address already in use: 0.0.0.0:9009 (version 20.3.9.70 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2021.09.14 10:22:37.033140 [ 1 ] {} <Error> Application: Listen [0.0.0.0]:9004 failed: Poco::Exception. Code: 1000, e.code() = 98, e.displayText() = Net Exception: Address already in use: 0.0.0.0:9004 (version 20.3.9.70 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2021.09.14 10:22:37.033848 [ 1 ] {} <Information> Application: Available RAM: 15.53 GiB; physical cores: 4; logical cores: 4.
2021.09.14 10:22:37.033878 [ 1 ] {} <Information> Application: Ready for connections.
Include not found: clickhouse_remote_servers
Include not found: clickhouse_compression
2021.09.14 10:22:44.248805 [ 51 ] {} <Information> BackgroundProcessingPool: Create BackgroundProcessingPool with 16 threads
2021.09.14 10:23:12.097990 [ 71 ] {9f8aa0dc-86e0-4a3c-895d-6a54400d61e8} <Information> executeQuery: Read 1 rows, 1.00 B in 0.001 sec., 852 rows/sec., 852.60 B/sec.
2021.09.14 10:23:12.098209 [ 71 ] {} <Information> TCPHandler: Processed in 0.002 sec.
2021.09.14 10:23:12.141068 [ 71 ] {643108b6-c426-40a0-be75-85b83b10a3a0} <Error> executeQuery: Code: 60, e.displayText() = DB::Exception: Table default.migrations_local doesn't exist. (version 20.3.9.70 (official build)) (from 172.18.0.5:60560) (in query: SELECT group, migration_id, status FROM migrations_local FINAL WHERE group IN ('system', 'events', 'transactions', 'discover', 'outcomes', 'sessions')), Stack trace (when copying this message, always include the lines below):
0. Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x105351b0 in /usr/bin/clickhouse
1. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x8f4172d in /usr/bin/clickhouse
2. DB::Context::getTableImpl(DB::StorageID const&, std::__1::optional<DB::Exception>*) const @ 0xcfe2a24 in /usr/bin/clickhouse
3. DB::Context::getTable(DB::StorageID const&) const @ 0xcfe2bbb in /usr/bin/clickhouse
4. DB::Context::getTable(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) const @ 0xcfe2c7d in /usr/bin/clickhouse
5. DB::JoinedTables::getLeftTableStorage() @ 0xd454892 in /usr/bin/clickhouse
6. DB::InterpreterSelectQuery::InterpreterSelectQuery(std::__1::shared_ptr<DB::IAST> const&, DB::Context const&, std::__1::shared_ptr<DB::IBlockInputStream> const&, std::__1::optional<DB::Pipe>, std::__1::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) @ 0xd13b6d1 in /usr/bin/clickhouse
7. DB::InterpreterSelectQuery::InterpreterSelectQuery(std::__1::shared_ptr<DB::IAST> const&, DB::Context const&, DB::SelectQueryOptions const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) @ 0xd13c619 in /usr/bin/clickhouse
8. DB::InterpreterSelectWithUnionQuery::InterpreterSelectWithUnionQuery(std::__1::shared_ptr<DB::IAST> const&, DB::Context const&, DB::SelectQueryOptions const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) @ 0xd341686 in /usr/bin/clickhouse
9. DB::InterpreterFactory::get(std::__1::shared_ptr<DB::IAST>&, DB::Context&, DB::QueryProcessingStage::Enum) @ 0xd0909b4 in /usr/bin/clickhouse
10. ? @ 0xd550655 in /usr/bin/clickhouse
11. DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::Context&, bool, DB::QueryProcessingStage::Enum, bool, bool) @ 0xd553441 in /usr/bin/clickhouse
12. DB::TCPHandler::runImpl() @ 0x9024489 in /usr/bin/clickhouse
13. DB::TCPHandler::run() @ 0x9025470 in /usr/bin/clickhouse
14. Poco::Net::TCPServerConnection::start() @ 0xe3ac69b in /usr/bin/clickhouse
15. Poco::Net::TCPServerDispatcher::run() @ 0xe3acb1d in /usr/bin/clickhouse
16. Poco::PooledThread::run() @ 0x105c3317 in /usr/bin/clickhouse
17. Poco::ThreadImpl::runnableEntry(void*) @ 0x105bf11c in /usr/bin/clickhouse
18. ? @ 0x105c0abd in /usr/bin/clickhouse
19. start_thread @ 0x76db in /lib/x86_64-linux-gnu/libpthread-2.27.so
20. __clone @ 0x12188f in /lib/x86_64-linux-gnu/libc-2.27.so
2021.09.14 10:23:12.141457 [ 71 ] {} <Information> TCPHandler: Processed in 0.043 sec.
2021.09.14 10:23:12.142026 [ 71 ] {} <Information> TCPHandler: Done processing connection.
2021.09.14 10:23:12.151205 [ 71 ] {} <Information> TCPHandler: Processed in 0.006 sec.
2021.09.14 10:23:12.153359 [ 71 ] {} <Information> TCPHandler: Processed in 0.001 sec.
2021.09.14 10:23:12.176161 [ 71 ] {} <Information> TCPHandler: Processed in 0.022 sec.
2021.09.14 10:23:12.181596 [ 71 ] {} <Information> TCPHandler: Processed in 0.002 sec.
2021.09.14 10:23:12.184225 [ 71 ] {} <Information> TCPHandler: Processed in 0.002 sec.
2021.09.14 10:23:12.193924 [ 71 ] {} <Information> TCPHandler: Processed in 0.009 sec.
2021.09.14 10:23:12.196349 [ 71 ] {5c3815eb-b87c-465d-b893-65632dfcb1ab} <Information> executeQuery: Read 1 rows, 51.00 B in 0.001 sec., 694 rows/sec., 34.58 KiB/sec.
2021.09.14 10:23:12.196545 [ 71 ] {} <Information> TCPHandler: Processed in 0.002 sec.
2021.09.14 10:23:12.198590 [ 71 ] {} <Information> TCPHandler: Processed in 0.002 sec.
2021.09.14 10:23:12.201208 [ 71 ] {} <Information> TCPHandler: Processed in 0.001 sec.
2021.09.14 10:23:12.203564 [ 71 ] {} <Information> TCPHandler: Processed in 0.002 sec.
2021.09.14 10:23:12.212505 [ 71 ] {} <Information> TCPHandler: Processed in 0.008 sec.
2021.09.14 10:23:12.220372 [ 71 ] {} <Information> TCPHandler: Processed in 0.007 sec.
2021.09.14 10:23:12.229642 [ 71 ] {} <Information> TCPHandler: Processed in 0.009 sec.
2021.09.14 10:23:12.237764 [ 71 ] {} <Information> TCPHandler: Processed in 0.007 sec.
2021.09.14 10:23:12.245815 [ 71 ] {} <Information> TCPHandler: Processed in 0.007 sec.
clickhouse/config.xml
<yandex>
<listen_host>0.0.0.0</listen_host>
</yandex>
I can also find a strange behavious in the install logs
Applying sentry.0157_make_repositoryprojectpathconfig_organization_integration_nullable... OK
Applying sentry.0158_create_externalteam_table... OK
Applying sentry.0159_create_externaluser_table... OK
Applying sentry.0160_create_projectcodeowners_table... OK
Applying sentry.0161_add_saved_search_sort... OK
Applying sentry.0162_backfill_saved_search_sort... OK
Applying sentry.0163_add_organizationmember_and_external_name... OK
Applying sentry.0164_add_protect_on_delete_codeowners... OK
Applying sentry.0165_metric_alerts_fix_group_ids... OK
Applying sentry.0166_create_notificationsetting_table... OK
Applying sentry.0167_rm_organization_integration_from_projectcodeowners... OK
Applying sentry.0168_demo_orgs_users... OK
Applying sentry.0169_delete_organization_integration_from_projectcodeowners... OK
Applying sentry.0170_actor_introduction... OK
Applying sentry.0171_backfill_actors... OK
Applying sentry.0172_rule_owner_fields... OK
Applying sentry.0173_remove_demo_flag... OK
Applying sentry.0174_my_issues_saved_search... OK
Applying sentry.0175_make_targets_nullable... OK
Applying sentry.0176_remove_targets... OK
Applying sentry.0177_drop_targets... OK
Applying sentry.0178_add_new_target_column... OK
Applying sentry.0179_update_legacy_discover_saved_query_timestamps... OK
Applying sentry.0180_add_saved_search_sorts... OK
Applying sentry.0181_copy_useroptions_to_notificationsettings... OK
Applying sentry.0182_update_user_misery_on_saved_queries... OK
Applying sentry.0183_make_codemapping_unique_on_projectcodeowners... OK
Applying sentry.0184_copy_useroptions_to_notificationsettings_2... OK
Applying sentry.0185_rm_copied_useroptions... OK
Applying sentry.0186_add_externalactor... OK
Applying sessions.0001_initial... OK
Applying sites.0001_initial... OK
Applying sites.0002_alter_domain_unique... OK
Applying social_auth.0001_initial... OK
16:28:23 [WARNING] sentry: Cannot initiate onboarding for organization (1) due to missing owners
Created internal Sentry project (slug=internal, id=1)
Would you like to create a user account now? [Y/n]: Aborted!
An error occurred, caught SIGERR on line 12
Cleaning up...
Session terminated, killing shell... ...killed.
Terminated
Do you have an idea ? 
Those containers disappearing/dying is definitely not normal and that seems like the culprit.