Clickhouse config bind failed

clickhouse_1                                | Processing configuration file '/etc/clickhouse-server/config.xml'.
clickhouse_1                                | Merging configuration file '/etc/clickhouse-server/config.d/docker_related_config.xml'.
clickhouse_1                                | Merging configuration file '/etc/clickhouse-server/config.d/sentry.xml'.
clickhouse_1                                | Poco::Exception. Code: 1000, e.code() = 0, e.displayText() = Exception: Failed to merge config with '/etc/clickhouse-server/config.d/sentry.xml': Access to file denied: /etc/clickhouse-server/config.d/sentry.xml, Stack trace (when copying this message, always include the lines below):
clickhouse_1                                | 
clickhouse_1                                | 0. Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x105351b0 in /usr/bin/clickhouse
clickhouse_1                                | 1. ? @ 0xdefbd83 in /usr/bin/clickhouse
clickhouse_1                                | 2. DB::ConfigProcessor::loadConfig(bool) @ 0xdef9e97 in /usr/bin/clickhouse
clickhouse_1                                | 3. BaseDaemon::reloadConfiguration() @ 0x9157010 in /usr/bin/clickhouse
clickhouse_1                                | 4. BaseDaemon::initialize(Poco::Util::Application&) @ 0x91597d2 in /usr/bin/clickhouse
clickhouse_1                                | 5. DB::Server::initialize(Poco::Util::Application&) @ 0x8f96458 in /usr/bin/clickhouse
clickhouse_1                                | 6. Poco::Util::Application::run() @ 0x10457659 in /usr/bin/clickhouse
clickhouse_1                                | 7. DB::Server::run() @ 0x8f96045 in /usr/bin/clickhouse
clickhouse_1                                | 8. mainEntryClickHouseServer(int, char**) @ 0x8f8ce23 in /usr/bin/clickhouse
clickhouse_1                                | 9. main @ 0x8ee8799 in /usr/bin/clickhouse
clickhouse_1                                | 10. __libc_start_main @ 0x21b97 in /lib/x86_64-linux-gnu/libc-2.27.so
clickhouse_1                                | 11. _start @ 0x8ee802e in /usr/bin/clickhouse
clickhouse_1                                |  (version 20.3.9.70 (official build))
clickhouse_1                                | Processing configuration file '/etc/clickhouse-server/config.xml'.
clickhouse_1                                | Merging configuration file '/etc/clickhouse-server/config.d/docker_related_config.xml'.
clickhouse_1                                | Merging configuration file '/etc/clickhouse-server/config.d/sentry.xml'.
clickhouse_1                                | Poco::Exception. Code: 1000, e.code() = 0, e.displayText() = Exception: Failed to merge config with '/etc/clickhouse-server/config.d/sentry.xml': Access to file denied: /etc/clickhouse-server/config.d/sentry.xml, Stack trace (when copying this message, always include the lines below):
clickhouse_1                                | 
clickhouse_1                                | 0. Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x105351b0 in /usr/bin/clickhouse
clickhouse_1                                | 1. ? @ 0xdefbd83 in /usr/bin/clickhouse
clickhouse_1                                | 2. DB::ConfigProcessor::loadConfig(bool) @ 0xdef9e97 in /usr/bin/clickhouse
clickhouse_1                                | 3. BaseDaemon::reloadConfiguration() @ 0x9157010 in /usr/bin/clickhouse
clickhouse_1                                | 4. BaseDaemon::initialize(Poco::Util::Application&) @ 0x91597d2 in /usr/bin/clickhouse
clickhouse_1                                | 5. DB::Server::initialize(Poco::Util::Application&) @ 0x8f96458 in /usr/bin/clickhouse
clickhouse_1                                | 6. Poco::Util::Application::run() @ 0x10457659 in /usr/bin/clickhouse
clickhouse_1                                | 7. DB::Server::run() @ 0x8f96045 in /usr/bin/clickhouse
clickhouse_1                                | 8. mainEntryClickHouseServer(int, char**) @ 0x8f8ce23 in /usr/bin/clickhouse
clickhouse_1                                | 9. main @ 0x8ee8799 in /usr/bin/clickhouse
clickhouse_1                                | 10. __libc_start_main @ 0x21b97 in /lib/x86_64-linux-gnu/libc-2.27.so
clickhouse_1                                | 11. _start @ 0x8ee802e in /usr/bin/clickhouse
clickhouse_1                                |  (version 20.3.9.70 (official build))
clickhouse_1                                | Processing configuration file '/etc/clickhouse-server/config.xml'.
clickhouse_1                                | Merging configuration file '/etc/clickhouse-server/config.d/docker_related_config.xml'.
clickhouse_1                                | Merging configuration file '/etc/clickhouse-server/config.d/sentry.xml'.
clickhouse_1                                | Poco::Exception. Code: 1000, e.code() = 0, e.displayText() = Exception: Failed to merge config with '/etc/clickhouse-server/config.d/sentry.xml': Access to file denied: /etc/clickhouse-server/config.d/sentry.xml, Stack trace (when copying this message, always include the lines below):
clickhouse_1                                | 
  clickhouse:
    << : *restart_policy
    image: 'yandex/clickhouse-server:20.3.9.70'
    ulimits:
      nofile:
        soft: 262144
        hard: 262144
    volumes:
      - 'sentry-clickhouse:/var/lib/clickhouse'
      - 'sentry-clickhouse-log:/var/log/clickhouse-server'
      - type: bind
        read_only: true
        source: ./clickhouse/config.xml
        target: /etc/clickhouse-server/config.d/sentry.xml
    environment:
      # This limits Clickhouse's memory to 30% of the host memory
      # If you have high volume and your search return incomplete results
      # You might want to change this to a higher value (and ensure your host has enough memory)
      MAX_MEMORY_USAGE_RATIO: 0.3

install_sentry.log

Docker images built.
Removing network onpremise_default
Network onpremise_default not found.
Removing network sentry_onpremise_default
Network sentry_onpremise_default not found.
Creating network "sentry_onpremise_default" with the default driver
Creating volume "sentry_onpremise_sentry-secrets" with default driver
Creating volume "sentry_onpremise_sentry-smtp" with default driver
Creating volume "sentry_onpremise_sentry-zookeeper-log" with default driver
Creating volume "sentry_onpremise_sentry-kafka-log" with default driver
Creating volume "sentry_onpremise_sentry-smtp-log" with default driver
Creating volume "sentry_onpremise_sentry-clickhouse-log" with default driver
Creating sentry_onpremise_zookeeper_run ... ^M
Creating sentry_onpremise_zookeeper_run ... done^M
Bootstrapping and migrating Snuba...
Creating sentry_onpremise_clickhouse_1 ... ^M
Creating sentry_onpremise_zookeeper_1  ... ^M
Creating sentry_onpremise_redis_1      ... ^M
Creating sentry_onpremise_zookeeper_1  ... done^M
Creating sentry_onpremise_kafka_1      ... ^M
Creating sentry_onpremise_kafka_1      ... done^M
Creating sentry_onpremise_redis_1      ... done^M
Creating sentry_onpremise_clickhouse_1 ... done^M
Creating sentry_onpremise_snuba-api_run ... ^M
Creating sentry_onpremise_snuba-api_run ... done^M
+ '[' b = - ']'^M
+ snuba bootstrap --help^M
+ set -- snuba bootstrap --no-migrate --force^M
+ set gosu snuba snuba bootstrap --no-migrate --force^M
+ exec gosu snuba snuba bootstrap --no-migrate --force^M
+ '[' b = - ']'^M
+ snuba bootstrap --help^M
+ set -- snuba bootstrap --no-migrate --force^M
+ set gosu snuba snuba bootstrap --no-migrate --force^M
+ exec gosu snuba snuba bootstrap --no-migrate --force^M
%3|1612726849.446|FAIL|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.22.0.5:9092 failed: Connection refused (after 1ms in state CONNECT)^M
%3|1612726850.446|FAIL|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.22.0.5:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)^M
2021-02-07 19:40:50,446 Connection to Kafka failed (attempt 0)^M
Traceback (most recent call last):^M
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 55, in bootstrap^M
    client.list_topics(timeout=1)^M
cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str="Failed to get metadata: Local: Broker transport failure"}^M
2021-02-07 19:40:52,046 Topic outcomes created^M
2021-02-07 19:40:52,047 Topic events created^M
2021-02-07 19:40:52,047 Topic event-replacements created^M
2021-02-07 19:40:52,047 Topic snuba-commit-log created^M
2021-02-07 19:40:52,047 Topic cdc created^M
2021-02-07 19:40:52,047 Topic errors-replacements created^M
2021-02-07 19:40:52,047 Topic ingest-sessions created^M
Creating sentry_onpremise_snuba-api_run ... ^M
Creating sentry_onpremise_snuba-api_run ... done^M
+ '[' m = - ']'^M
+ snuba migrations --help^M
+ set -- snuba migrations migrate --force^M
+ set gosu snuba snuba migrations migrate --force^M
+ exec gosu snuba snuba migrations migrate --force^M
Failed to connect to clickhouse:9000^M
Traceback (most recent call last):^M
  File "/usr/local/lib/python3.8/site-packages/clickhouse_driver/connection.py", line 256, in connect^M
    return self._init_connection(host, port)^M
  File "/usr/local/lib/python3.8/site-packages/clickhouse_driver/connection.py", line 226, in _init_connection^M
    self.socket = self._create_socket(host, port)^M
  File "/usr/local/lib/python3.8/site-packages/clickhouse_driver/connection.py", line 202, in _create_socket^M
    for res in socket.getaddrinfo(host, port, 0, socket.SOCK_STREAM):^M
  File "/usr/local/lib/python3.8/socket.py", line 918, in getaddrinfo^M
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):^M
socket.gaierror: [Errno -3] Temporary failure in name resolution^M
Failed to connect to clickhouse:9000^M
Traceback (most recent call last):^M
  File "/usr/local/lib/python3.8/site-packages/clickhouse_driver/connection.py", line 256, in connect^M
    return self._init_connection(host, port)^M
  File "/usr/local/lib/python3.8/site-packages/clickhouse_driver/connection.py", line 226, in _init_connection^M
    self.socket = self._create_socket(host, port)^M
  File "/usr/local/lib/python3.8/site-packages/clickhouse_driver/connection.py", line 202, in _create_socket^M
    for res in socket.getaddrinfo(host, port, 0, socket.SOCK_STREAM):^M
  File "/usr/local/lib/python3.8/socket.py", line 918, in getaddrinfo^M
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):^M
socket.gaierror: [Errno -3] Temporary failure in name resolution^M
Connection to Clickhouse cluster clickhouse:9000 failed (attempt 0)^M
Traceback (most recent call last):^M
  File "/usr/src/snuba/snuba/clickhouse/native.py", line 67, in execute^M
    result = conn.execute(*args, **kwargs)^M
  File "/usr/local/lib/python3.8/site-packages/clickhouse_driver/client.py", line 199, in execute^M
    self.connection.force_connect()^M
  File "/usr/local/lib/python3.8/site-packages/clickhouse_driver/connection.py", line 180, in force_connect^M
    self.connect()^M
  File "/usr/local/lib/python3.8/site-packages/clickhouse_driver/connection.py", line 277, in connect^M
    raise err^M
clickhouse_driver.errors.NetworkError: Code: 210. Temporary failure in name resolution (None:None)^M

I am using 21.1.0 changed sentry/config.example.yml.

ran ./install.sh

Can someone tell me what is going on ? The bind mount looks correct and the config.xml file is in the clickhouse directory.

Running this on Ubuntu 18.04. I clean up the volumes and tried again. It is still the same. There is some issue with the clickhouse container.

can someone please tell me the environment where 21.1.0 was tested ? I have only changed the sentry/config.example.yml with my parameters. Its kind of a disappointment that it is not working straight out of the box.

Access to file denied: /etc/clickhouse-server/config.d/sentry.xml

There is your error. Just make sure this file is readable by anyone and it should be fine?

@BYK That worked. Thank you. May be you should put a note in the documentation some where about this file access issue. Many like me just overlook.

Yeah this makes sense. Thanks for the feedback!

PS: If you wanna take credit, you can add your words here: https://github.com/getsentry/develop/edit/master/src/docs/self-hosted/index.mdx

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.