Secure Clickhouse connection with on premise [snuba]

On-Premise
1

Hi. I’m trying to install on premise sentry with remote DBaaS based on Yandex Cloud. Sentry supports SASL_SSL on Kafka cluster.

The problem is YC using their own CA certificates with clickhouse, without them it would not work. So is there any way to start Snuba and sentry (maybe) to work with clickhouse on ssl/ca certs

Here is YC documentation which is not working inside Snuba container:

mkdir -p ~/.clickhouse-client /usr/local/share/ca-certificates/Yandex && \
wget "https://crls.yandex.net/allCAs.pem" -O /usr/local/share/ca-certificates/Yandex/YandexInternalRootCA.crt && \
wget "https://storage.yandexcloud.net/mdb/clickhouse-client.conf.example" -O ~/.clickhouse-client/config.xml

clickhouse-client --host sas-fsdfdsdb.yandex.net \
                  --secure \
                  --user <username> \
                  --database <databasename> \
                  --port 9440 \
                  --ask-password

Or here is python version:

def request():
    url = 'https://{host}:8443/?database={db}&query={query}'.format(
        host='sas-wfwqv2yebnxbtfrx.db.yandex.net',
        db='<databasename>',
        query='SELECT now()')
    auth = {
        'X-ClickHouse-User': '<username>',
        'X-ClickHouse-Key': '<password>',
    }

    res = requests.get(
        url,
        headers=auth,
        verify='/usr/local/share/ca-certificates/Yandex/YandexInternalRootCA.crt')
    res.raise_for_status()
    return res.text

print(request())

YC docs Подключение к базе данных в кластере ClickHouse | Yandex.Cloud - Документация

2

We recently added support for custom CAs for Sentry on the onpremise repo: https://develop.sentry.dev/self-hosted/custom-ca-roots/

You can check out the PR that added it and port its logic to Snuba images: feat: Support custom CA roots by Spice-King · Pull Request #1015 · getsentry/onpremise · GitHub

3

Unfortunately, this is not exactly what I mean. The problem is inside, I guess, mechanism how Snuba connecting to CH. Connection should be secure via --secure flag, here is example with ch-client:

SECURE flag

➜  ./clickhouse client --host  sas-wfwqv2yebnxbtfrx.db.yandex.net --user admin --secure --port 9440 --ask-password
ClickHouse client version 21.10.1.7827 (official build).
Connected to ClickHouse server version 21.3.15 revision 54447.

WITHOUT secure flag

./clickhouse client --host  sas-wfwqv2yebnxbtfrx.db.yandex.net --user admin --port 9440 --ask-password 
Connecting to sas-wfwqv2yebnxbtfrx.db.yandex.net:9440 as user admin.
Code: 102. DB::NetException: Unexpected packet from server sas-wfwqv2yebnxbtfrx.db.yandex.net:9440 (expected Hello or Exception, got Unknown packet). (UNEXPECTED_PACKET_FROM_SERVER)

So here is the problem as I understand. And I receive this message - Unexpected packet inside Snuba container:

root@sentry:/usr/local/share/ca-certificates# env | grep CLICK
CLICKHOUSE_PASSWORD=
CLICKHOUSE_PORT=9440
CLICKHOUSE_HOST=sas-.db.yandex.net
CLICKHOUSE_USER=admin
CLICKHOUSE_HTTP_PORT=9440

root@sentry:/usr/local# snuba migrations migrate

2021-08-19 16:17:33,507 Connection to Clickhouse cluster sas-.db.yandex.net:9440 failed (attempt 0)
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/clickhouse/native.py", line 81, in execute
    result: Sequence[Any] = conn.execute(
  File "/usr/local/lib/python3.8/dist-packages/clickhouse_driver/client.py", line 205, in execute
    self.connection.force_connect()
  File "/usr/local/lib/python3.8/dist-packages/clickhouse_driver/connection.py", line 180, in force_connect
    self.connect()
  File "/usr/local/lib/python3.8/dist-packages/clickhouse_driver/connection.py", line 260, in connect
    return self._init_connection(host, port)
  File "/usr/local/lib/python3.8/dist-packages/clickhouse_driver/connection.py", line 238, in _init_connection
    self.receive_hello()
  File "/usr/local/lib/python3.8/dist-packages/clickhouse_driver/connection.py", line 376, in receive_hello
    raise errors.UnexpectedPacketFromServerError(message)
clickhouse_driver.errors.UnexpectedPacketFromServerError: Code: 102. Unexpected packet from server None:None (expected Hello or Exception, got Unknown packet)

I think File "/usr/src/snuba/snuba/clickhouse/native.py", line 104, in execute uses not secure connection without secure flag.

I found this PR Add params for secure ClickHouse connections. by konstantin-popov · Pull Request #2018 · getsentry/snuba · GitHub</ which was reverted, maybe you can tell me what the problem why it reverted? Revert "Add params for secure ClickHouse connections." by evanh · Pull Request #2033 · getsentry/snuba · Or maybe you can tell me some instructions how to connect to CH with secure flag?

4

@BYK hi! maybe any news?

5

I guess you got your answer here: Revert "Add params for secure ClickHouse connections." by evanh · Pull Request #2033 · getsentry/snuba · :slightly_smiling_face:

I recommend following that issue/PR as you are already directly connected to the most relevant people there.

6

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.