Not my area of expertise but maybe @fpacifici knows something about this. That said since this happens at the TCP level, I’d suspect it is a virtual networking issue on your machine or Docker itself.
I just reduce the SENTRY_EVENT_RETENTION_DAYS to 30 and nothing else has been changed as I know.
On Sunday we stop to receive events from sentry but the containers were up and healthy. Today I decide to update to 21.8.0 and notice that. We continue without receive any events but now we have the clickhouse error (this not happens in 21.7.0) and also this error (from web container) that
I can’t figure out where fix the SSL
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='sentry.myorg.org', port=443): Max retries exceeded with url: /api/1/store/ (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),))
13:30:38 [ERROR] sentry_sdk.errors: Internal error in sentry_sdk
If it helps, here is the log of clickhouse in the after the update (only few lines)
Processing configuration file '/etc/clickhouse-server/config.xml'.
Merging configuration file '/etc/clickhouse-server/config.d/docker_related_config.xml'.
Merging configuration file '/etc/clickhouse-server/config.d/sentry.xml'.
Include not found: clickhouse_remote_servers
Include not found: clickhouse_compression
Logging information to /var/log/clickhouse-server/clickhouse-server.log
Logging errors to /var/log/clickhouse-server/clickhouse-server.err.log
Logging information to console
2021.08.17 09:40:38.475472 [ 1 ] {} <Information> : Starting ClickHouse 20.3.9.70 with revision 54433
2021.08.17 09:40:38.478272 [ 1 ] {} <Information> Application: starting up
Include not found: networks
2021.08.17 09:40:38.488430 [ 1 ] {} <Information> Application: Loading metadata from /var/lib/clickhouse/
2021.08.17 09:40:38.490182 [ 1 ] {} <Information> DatabaseOrdinary (system): Total 2 tables and 0 dictionaries.
2021.08.17 09:40:38.492784 [ 48 ] {} <Information> BackgroundProcessingPool: Create BackgroundProcessingPool with 16 threads
2021.08.17 09:40:38.582827 [ 1 ] {} <Information> DatabaseOrdinary (system): Starting up tables.
2021.08.17 09:40:38.592394 [ 1 ] {} <Information> DatabaseOrdinary (default): Total 13 tables and 0 dictionaries.
2021.08.17 09:40:38.731644 [ 1 ] {} <Information> DatabaseOrdinary (default): Starting up tables.
2021.08.17 09:40:38.733177 [ 1 ] {} <Information> BackgroundSchedulePool: Create BackgroundSchedulePool with 16 threads
2021.08.17 09:40:38.733568 [ 1 ] {} <Information> Application: It looks like the process has no CAP_NET_ADMIN capability, 'taskstats' performance statistics will be disabled. It could happen due to incorrect ClickHouse package installation. You could resolve the problem manually with 'sudo setcap cap_net_admin=+ep /usr/bin/clickhouse'. Note that it will not work on 'nosuid' mounted filesystems. It also doesn't work if you run clickhouse-server inside network namespace as it happens in some containers.
2021.08.17 09:40:38.733674 [ 1 ] {} <Information> Application: It looks like the process has no CAP_SYS_NICE capability, the setting 'os_thread_nice' will have no effect. It could happen due to incorrect ClickHouse package installation. You could resolve the problem manually with 'sudo setcap cap_sys_nice=+ep /usr/bin/clickhouse'. Note that it will not work on 'nosuid' mounted filesystems.
2021.08.17 09:40:38.735414 [ 1 ] {} <Error> Application: Listen [::]:8123 failed: Poco::Exception. Code: 1000, e.code() = 0, e.displayText() = DNS error: EAI: -9 (version 20.3.9.70 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2021.08.17 09:40:38.735876 [ 1 ] {} <Error> Application: Listen [::]:9000 failed: Poco::Exception. Code: 1000, e.code() = 0, e.displayText() = DNS error: EAI: -9 (version 20.3.9.70 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2021.08.17 09:40:38.748781 [ 1 ] {} <Error> Application: Listen [::]:9009 failed: Poco::Exception. Code: 1000, e.code() = 0, e.displayText() = DNS error: EAI: -9 (version 20.3.9.70 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2021.08.17 09:40:38.749269 [ 1 ] {} <Error> Application: Listen [::]:9004 failed: Poco::Exception. Code: 1000, e.code() = 0, e.displayText() = DNS error: EAI: -9 (version 20.3.9.70 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2021.08.17 09:40:38.749431 [ 1 ] {} <Information> Application: Listening for http://0.0.0.0:8123
2021.08.17 09:40:38.749528 [ 1 ] {} <Information> Application: Listening for connections with native protocol (tcp): 0.0.0.0:9000
2021.08.17 09:40:38.749624 [ 1 ] {} <Information> Application: Listening for replica communication (interserver): http://0.0.0.0:9009
2021.08.17 09:40:38.868683 [ 1 ] {} <Information> Application: Listening for MySQL compatibility protocol: 0.0.0.0:9004
2021.08.17 09:40:38.872125 [ 1 ] {} <Information> Application: Available RAM: 51.10 GiB; physical cores: 4; logical cores: 8.
2021.08.17 09:40:38.872180 [ 1 ] {} <Information> Application: Ready for connections.
Include not found: clickhouse_remote_servers
Include not found: clickhouse_compression
2021.08.17 09:40:41.107248 [ 89 ] {} <Error> ServerErrorHandler: Code: 101, e.displayText() = DB::NetException: Unexpected packet from client, Stack trace (when copying this message, always include the lines below):
0. Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x105351b0 in /usr/bin/clickhouse
1. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x8f4172d in /usr/bin/clickhouse
2. ? @ 0x90263ad in /usr/bin/clickhouse
3. DB::TCPHandler::runImpl() @ 0x90239e0 in /usr/bin/clickhouse
4. DB::TCPHandler::run() @ 0x9025470 in /usr/bin/clickhouse
5. Poco::Net::TCPServerConnection::start() @ 0xe3ac69b in /usr/bin/clickhouse
6. Poco::Net::TCPServerDispatcher::run() @ 0xe3acb1d in /usr/bin/clickhouse
7. Poco::PooledThread::run() @ 0x105c3317 in /usr/bin/clickhouse
8. Poco::ThreadImpl::runnableEntry(void*) @ 0x105bf11c in /usr/bin/clickhouse
9. ? @ 0x105c0abd in /usr/bin/clickhouse
10. start_thread @ 0x76db in /lib/x86_64-linux-gnu/libpthread-2.27.so
11. __clone @ 0x12188f in /lib/x86_64-linux-gnu/libc-2.27.so
(version 20.3.9.70 (official build))
@BYK help me to understand one thing. I have updated from 2.1.7 to 2.1.8. In 2.1.7 I have used the ‘default’ installation (configuring only the SMTP and host) and I did nothing related to SSL/TLS and everything works fine. So why do I need to have this configuration now? What am I misunderstanding?
I was just answering your specific question regarding that SSL error. Pretty sure it was there all the time but since your server didn’t have any errors, you never encountered it. You don’t need to do this, especially not with the new version. It is just a new feature that would help you as you seem to be using invalid/self-signed SSL certificates.
