CentOS 8 | Clickhouse server failed to come up in 5 tries

[shaqaruden]

I am trying to setup Sentry on a CentOS 8 server but I am running into the issue in the title of my post. Here is the output to the console.

Checking minimum requirements...

Creating volumes for persistent storage...
Created sentry-data.
Created sentry-postgres.
Created sentry-redis.
Created sentry-zookeeper.
Created sentry-kafka.
Created sentry-clickhouse.
Created sentry-symbolicator.

sentry/sentry.conf.py already exists, skipped creation.
sentry/config.yml already exists, skipped creation.
sentry/requirements.txt already exists, skipped creation.
symbolicator/config.yml already exists, skipped creation.
relay/config.yml already exists, skipped creation.

Fetching and updating Docker images...

latest: Pulling from getsentry/sentry
Digest: sha256:0f03140424dfbb3704fa3c9ea05941c2cce719cbbbb9b37ca2fbb8b01659f0e7
Status: Image is up to date for getsentry/sentry:latest
docker.io/getsentry/sentry:latest

Building and tagging Docker images...

Building web
Step 1/4 : ARG SENTRY_IMAGE
Step 2/4 : FROM ${SENTRY_IMAGE}
 ---> 26ad39335d08
Step 3/4 : COPY . /usr/src/sentry
 ---> Using cache
 ---> d0490600bf2a
Step 4/4 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then pip install -r /usr/src/sentry/requirements.txt; fi
 ---> Using cache
 ---> 43e3a23999e4

Successfully built 43e3a23999e4
Successfully tagged sentry-onpremise-local:latest
smtp uses an image, skipping
memcached uses an image, skipping
redis uses an image, skipping
postgres uses an image, skipping
zookeeper uses an image, skipping
kafka uses an image, skipping
clickhouse uses an image, skipping
snuba-api uses an image, skipping
snuba-consumer uses an image, skipping
snuba-outcomes-consumer uses an image, skipping
snuba-sessions-consumer uses an image, skipping
snuba-transactions-consumer uses an image, skipping
snuba-replacer uses an image, skipping
symbolicator uses an image, skipping
relay uses an image, skipping
nginx uses an image, skipping
Building snuba-cleanup          ... 
Building symbolicator-cleanup   ... 
Building web                    ... 
Building cron                   ... 
Building worker                 ... 
Building ingest-consumer        ... 
Building post-process-forwarder ... 
Building sentry-cleanup         ... 
Building worker
Building snuba-cleanup
Building web
Building symbolicator-cleanup
Building sentry-cleanup
Step 1/5 : ARG BASE_IMAGE
Step 2/5 : FROM ${BASE_IMAGE}
 ---> 7fa4e6e0d0c2
Step 3/5 : RUN apt-get update && apt-get install -y --no-install-recommends cron &&     rm -r /var/lib/apt/lists/*
 ---> Using cache
 ---> 329537df5a94
Step 4/5 : COPY entrypoint.sh /entrypoint.sh
 ---> Using cache
 ---> b09569639c68
Step 5/5 : ENTRYPOINT ["/entrypoint.sh"]
 ---> Using cache
 ---> 6b204afd1ef2

Successfully built 6b204afd1ef2
Successfully tagged snuba-cleanup-onpremise-local:latest
Building ingest-consumer
Building snuba-cleanup          ... done
Step 1/4 : ARG SENTRY_IMAGE
Step 2/4 : FROM ${SENTRY_IMAGE}
 ---> 26ad39335d08
Step 3/4 : COPY . /usr/src/sentry
 ---> Using cache
 ---> d0490600bf2a
Step 4/4 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then pip install -r /usr/src/sentry/requirements.txt; fi
Step 1/5 : ARG BASE_IMAGE ---> Using cache

 ---> 43e3a23999e4
Step 2/5 : FROM ${BASE_IMAGE}

Successfully built 43e3a23999e4
 ---> 43e3a23999e4
Successfully tagged sentry-onpremise-local:latest
Step 3/5 : RUN apt-get update && apt-get install -y --no-install-recommends cron &&     rm -r /var/lib/apt/lists/*
 ---> Using cache
 ---> 753fdebb9018
Step 4/5 : COPY entrypoint.sh /entrypoint.sh
 ---> Using cache
 ---> 53394dab3c2b
Building worker                 ... done
Building post-process-forwarder
Step 5/5 : ENTRYPOINT ["/entrypoint.sh"]
 ---> Using cache
 ---> e9ad79918f86

Successfully built e9ad79918f86
Successfully tagged sentry-cleanup-onpremise-local:latest
Building sentry-cleanup         ... done
Building cron
Step 1/4 : ARG SENTRY_IMAGE
Step 2/4 : FROM ${SENTRY_IMAGE}
 ---> 26ad39335d08
Step 3/4 : COPY . /usr/src/sentry
 ---> Using cache
 ---> d0490600bf2a
Step 4/4 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then pip install -r /usr/src/sentry/requirements.txt; fi
 ---> Using cache
 ---> 43e3a23999e4

Successfully built 43e3a23999e4
Successfully tagged sentry-onpremise-local:latest
Building web                    ... done
Step 1/5 : ARG BASE_IMAGE
Step 2/5 : FROM ${BASE_IMAGE}
 ---> a2ec8e59d81f
Step 3/5 : RUN apt-get update && apt-get install -y --no-install-recommends cron &&     rm -r /var/lib/apt/lists/*
 ---> Using cache
 ---> fe1b4e51fe1f
Step 4/5 : COPY entrypoint.sh /entrypoint.sh
 ---> Using cache
 ---> 487f71db3015
Step 5/5 : ENTRYPOINT ["/entrypoint.sh"]
 ---> Using cache
 ---> 6e570244c011

Successfully built 6e570244c011
Successfully tagged symbolicator-cleanup-onpremise-local:latest
Building symbolicator-cleanup   ... done
Step 1/4 : ARG SENTRY_IMAGE
Step 2/4 : FROM ${SENTRY_IMAGE}
 ---> 26ad39335d08
Step 3/4 : COPY . /usr/src/sentry
 ---> Using cache
 ---> d0490600bf2a
Step 4/4 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then pip install -r /usr/src/sentry/requirements.txt; fi
 ---> Using cache
 ---> 43e3a23999e4

Successfully built 43e3a23999e4
Successfully tagged sentry-onpremise-local:latest
Step 1/4 : ARG SENTRY_IMAGEBuilding ingest-consumer        ... done

Step 2/4 : FROM ${SENTRY_IMAGE}
 ---> 26ad39335d08
Step 3/4 : COPY . /usr/src/sentry
 ---> Using cache
 ---> d0490600bf2a
Step 4/4 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then pip install -r /usr/src/sentry/requirements.txt; fi
 ---> Using cache
Step 1/4 : ARG SENTRY_IMAGE ---> 43e3a23999e4


Step 2/4 : FROM ${SENTRY_IMAGE}Successfully built 43e3a23999e4

Successfully tagged sentry-onpremise-local:latest
 ---> 26ad39335d08
Step 3/4 : COPY . /usr/src/sentry
 ---> Using cache
 ---> d0490600bf2a
Building post-process-forwarder ... done
Step 4/4 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then pip install -r /usr/src/sentry/requirements.txt; fi
 ---> Using cache
 ---> 43e3a23999e4

Successfully built 43e3a23999e4
Successfully tagged sentry-onpremise-local:latest
Building cron                   ... done

Docker images built.
Removing network onpremise_default
Network onpremise_default not found.
Removing ltierrorreporting_clickhouse_1 ... 
Removing ltierrorreporting_clickhouse_1 ... done
Removing network ltierrorreporting_default
Creating network "ltierrorreporting_default" with the default driver
Creating ltierrorreporting_clickhouse_1 ... 
Creating ltierrorreporting_clickhouse_1 ... done
Trying again. Remaining tries #4
Trying again. Remaining tries #3
Trying again. Remaining tries #2
Trying again. Remaining tries #1
Clickhouse server failed to come up in 5 tries.

I have tried to figure this out myself first but I’m not having much luck. Any assistance in helping me solve issue would be greatly appreciated.

[BYK]

What do you get when you run docker-compose logs clickhouse?

[shaqaruden]

@BYK

docker-compose logs clickhouse

Thank you for your response. I also just finished testing setting up sentry on CentOS 7 inside a VM and it worked perfectly without fail

Here is the output of that command

clickhouse_1 | Include not found: clickhouse_remote_servers
clickhouse_1 | Include not found: clickhouse_compression
clickhouse_1 | Logging trace to /var/log/clickhouse-server/clickhouse-server.log
clickhouse_1 | Logging errors to /var/log/clickhouse-server/clickhouse-server.err.log
clickhouse_1 | Include not found: networks
clickhouse_1 | Include not found: clickhouse_remote_servers
clickhouse_1 | Include not found: clickhouse_compression

[BYK]

Hmm, the logs don’s show any errors so I’d make a guess and say you don’t have enough memory so the OOM-killer simply kills the container as it reaches the memory limit?

[shaqaruden]

The CentOS 7 VM I had tested on had only 4GB of RAM. This CentOS 8 server instance has 32GB of RAM

[shaqaruden]

Vcenter only shows a 6% memory utilization when running ./install.sh

[BYK]

Well, then I’d look somewhere else in the logs but not sure where TBH.

[shaqaruden]

I have been stepping through install.sh to see if I can find the problem. The clickhouse-server container does start just fine

docker-compose up -d clickhouse

when I run docker ps I see

7f8a555dd61a        yandex/clickhouse-server:19.17   "/entrypoint.sh"    12 minutes ago      Up 12 minutes       8123/tcp, 9000/tcp, 9009/tcp   sentry_onpremise_clickhouse_1

but when running this line
docker-compose run clickhouse clickhouse-client --host clickhouse -q "SELECT 1";

it returns this

Code: 210. DB::NetException: Net Exception: No route to host (clickhouse:9000)

[shaqaruden]

I figured it out! With CentOS 8 there is firewall rules that you need to set.

# Masquerading allows for docker ingress and egress (this is the juicy bit)
firewall-cmd --zone=public --add-masquerade --permanent

# Specifically allow incoming traffic on port 80/443 (nothing new here)
firewall-cmd --zone=public --add-port=80/tcp
firewall-cmd --zone=public --add-port=443/tcp

# Reload firewall to apply permanent rules
firewall-cmd --reload

cite: https://serverfault.com/questions/987686/no-network-connectivity-to-from-docker-ce-container-on-centos-8

[BYK]

This is quite peculiar as when using docker-compose run it should be using the project-specific private network and shouldn’t need any firewall exceptions.

Glad you found a solution though I am not sure if this is a general issue with CentOS 8.

[shaqaruden]

I would say so since this was the first thing I tried to configure on this server. It was just a standard minimal install of CentOS 8. The same install with CentOS 7 worked fine no issues.

[shaqaruden]

Update on this while attempting to get Sentry working with traefik the issue came back. At this point I rebuilt the server with CentOS 7 and moved forward.