Celery.worker.consumer.consumer: consumer: Connection to broker lost. Trying to re-establish the connection... worker_1Restoring 13 unacknowledged message(s)

On-Premise
1

Hello Team,

From past one week sentry was not able to show ant alerts and when i checked at worker i found the issue below
Traceback (most recent call last):
worker_1 | File “/usr/local/lib/python2.7/site-packages/celery/worker/consumer/consumer.py”, line 316, in start
worker_1 | blueprint.start(self)
worker_1 | File “/usr/local/lib/python2.7/site-packages/celery/bootsteps.py”, line 119, in start
worker_1 | step.start(parent)
worker_1 | File “/usr/local/lib/python2.7/site-packages/celery/worker/consumer/consumer.py”, line 592, in start
worker_1 | c.loop(*c.loop_args())
worker_1 | File “/usr/local/lib/python2.7/site-packages/celery/worker/loops.py”, line 91, in asynloop
worker_1 | next(loop)
worker_1 | File “/usr/local/lib/python2.7/site-packages/kombu/asynchronous/hub.py”, line 354, in create_loop
worker_1 | cb(*cbargs)
worker_1 | File “/usr/local/lib/python2.7/site-packages/kombu/transport/redis.py”, line 1047, in on_readable
worker_1 | self.cycle.on_readable(fileno)
worker_1 | File “/usr/local/lib/python2.7/site-packages/kombu/transport/redis.py”, line 344, in on_readable
worker_1 | chan.handlerstype
worker_1 | File “/usr/local/lib/python2.7/site-packages/kombu/transport/redis.py”, line 721, in _brpop_read
worker_1 | **options)
worker_1 | File “/usr/local/lib/python2.7/site-packages/redis/client.py”, line 680, in parse_response
worker_1 | response = connection.read_response()
worker_1 | File “/usr/local/lib/python2.7/site-packages/redis/connection.py”, line 624, in read_response
worker_1 | response = self._parser.read_response()
worker_1 | File “/usr/local/lib/python2.7/site-packages/redis/connection.py”, line 403, in read_response
worker_1 | (e.args,))
worker_1 | ConnectionError: Error while reading from socket: (‘Connection closed by server.’,)
worker_1 | 10:31:18 [WARNING] celery.worker.consumer.consumer: consumer: Connection to broker lost. Trying to re-establish the connection…
worker_1 | Restoring 13 unacknowledged message(s)

2

The last message implies that the worker was able to recover, is that correct? This may indicate your Redis service not keeping up with the event volume.

There’s also a known bug where sometimes worker services lose connection to Redis and never recover unless you restart them: https://github.com/getsentry/sentry/issues/21133

Also, which version of Sentry are you using?

3

Hello Byk,

I am using Sentry 20.8.0, when i restarted the services it didn’t work so i have installed ./install.sh and it worked for 4 days and has the issue again , i have seen the post looks like need to restart the service .
Thanks for your help.

4

@kkgamini we have a number of fixes after 20.8.0 so I strongly recommend trying the latest version 20.11.1 if you haven’t already.

5

Sure BYK i will do that . Thank you

1 Like
6

Hello Byk,

I am upgrading from sentry 20.8 to 20.11 , i am facing the issue below

Setting up database…
Creating sentry_onpremise_smtp_1 …
Starting sentry_onpremise_redis_1 …
Starting sentry_onpremise_redis_1 … done
Creating sentry_onpremise_symbolicator_1 …
Creating sentry_onpremise_memcached_1 …
Starting sentry_onpremise_clickhouse_1 …
Starting sentry_onpremise_clickhouse_1 … done
Starting sentry_onpremise_zookeeper_1 …
Creating sentry_onpremise_postgres_1 …
Starting sentry_onpremise_zookeeper_1 … done
Starting sentry_onpremise_kafka_1 …
Starting sentry_onpremise_kafka_1 … done
Creating sentry_onpremise_snuba-consumer_1 …
Creating sentry_onpremise_snuba-api_1 …
Creating sentry_onpremise_snuba-sessions-consumer_1 …
Creating sentry_onpremise_snuba-transactions-consumer_1 …
Creating sentry_onpremise_snuba-outcomes-consumer_1 …
Creating sentry_onpremise_snuba-replacer_1 …
Creating sentry_onpremise_memcached_1 … done
Creating sentry_onpremise_smtp_1 … done
Creating sentry_onpremise_snuba-sessions-consumer_1 … done
Creating sentry_onpremise_postgres_1 … done
Creating sentry_onpremise_snuba-consumer_1 … done
Creating sentry_onpremise_snuba-outcomes-consumer_1 … done
Creating sentry_onpremise_symbolicator_1 … done
Creating sentry_onpremise_snuba-replacer_1 … done
Creating sentry_onpremise_snuba-api_1 … done
Creating sentry_onpremise_snuba-transactions-consumer_1 … done
!! Configuration error: IOError: [Errno 13] Unable to load configuration file (Permission denied): ‘/etc/sentry/sentry.conf.py’

I have copied the old config.yml file to the new folder

7

Does the user have permission to “/etc/sentry/sentry.conf.py”?

1 Like
8

yes i am doing it from root

9

according to the error, theres permission issues. Double check them :slight_smile:

1 Like
10

Hello Alexander i have changed the export SENTRY_CONFIG_PY=/path/ it worked but had issue in bringing up nginx , worker and web it shows the below error

 Configuration file does not exist. Use 'sentry init' to initialize the file.
Cleaning up...
11

Hello Byk i have checked the other posts provided full access to sentry still i am facing the issue below
eb_1 | Error: Configuration file does not exist. Use ‘sentry init’ to initialize the file.
web_1 | Error: Configuration file does not exist. Use ‘sentry init’ to initialize the file.
web_1 | Error: Configuration file does not exist. Use ‘sentry init’ to initialize the file.

12

@kkgamini It’s weird to me that the web node can’t find a config file and thinks you need to re-init. Could I ask for a little more context about your installation procedure and setup? How closely did you follow Getting Started? What hosting provider are you using? What OS are you running on? Have you installed any other services on the same box?

13

Hello Chad, thanks for your reply
I am using ubuntu v18 , already sentry v20.8 is running on the host (Amazon Ec2). I have tested the same thing in the staging env by upgrading sentry v20.8 to v20.11 and it worked out fine. When i tired in production initially i am getting the error
!! Configuration error: IOError: [Errno 13] Unable to load configuration file (Permission denied): ‘/etc/sentry/sentry.conf.py’
After i change the export SENTRY_CONFIG_PY=/path/ it worked and also provided full permissions for sentry but nginx , worker and web keeps on restarting and when i checked the logs its same issue
Error: Configuration file does not exist. Use ‘sentry init’ to initialize the file.
There is one more error in relay credentials credentials were not found , i found the error IO in credentials.json file
IOError: [Errno 13] Unable to load configuration file (Permission denied): ‘/etc/sentry/sentry.conf.py

Installation: As we already have v20.8 i have downloaded v20.11 and run intsall.sh as we have to keep the old configuration i have copied the old config.yml file to new folder

14

Would you mind pasting your full install log?

15

Hello Chad please find the install log for permission issue
onpremise-20.11.1# cat sentry_install_log-2020-12-02_19-43-59.txt
Checking minimum requirements…

Creating volumes for persistent storage…
Created sentry-data.
Created sentry-postgres.
Created sentry-redis.
Created sentry-zookeeper.
Created sentry-kafka.
Created sentry-clickhouse.
Created sentry-symbolicator.

sentry/sentry.conf.py already exists, skipped creation.
sentry/config.yml already exists, skipped creation.
sentry/requirements.txt already exists, skipped creation.
symbolicator/config.yml already exists, skipped creation.
relay/config.yml already exists, skipped creation.

Fetching and updating Docker images…

Some service image(s) must be built from source by running:
docker-compose build snuba-cleanup cron post-process-forwarder ingest-consumer subscription-consumer-events symbolicator-cleanup sentry-cleanup web subscription-consumer-transactions worker
20.11.1: Pulling from getsentry/sentry
Digest: sha256:c44619851c548bbac43c7ac716e0b0404679344de36792d84717745d8e23517b
Status: Image is up to date for getsentry/sentry:20.11.1
docker.io/getsentry/sentry:20.11.1

Building and tagging Docker images…

Building web
Step 1/5 : ARG SENTRY_IMAGE
Step 2/5 : ARG SENTRY_PYTHON3
Step 3/5 : FROM {SENTRY_IMAGE}{SENTRY_PYTHON3:±py3}
—> 71ac5bcb4aaa
Step 4/5 : COPY . /usr/src/sentry
—> Using cache
—> 997c3f583107
Step 5/5 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then pip install -r /usr/src/sentry/requirements.txt; fi
—> Using cache
—> ba215304a1c4

Successfully built ba215304a1c4
Successfully tagged sentry-onpremise-local:latest
smtp uses an image, skipping
memcached uses an image, skipping
redis uses an image, skipping
postgres uses an image, skipping
zookeeper uses an image, skipping
kafka uses an image, skipping
clickhouse uses an image, skipping
snuba-api uses an image, skipping
snuba-consumer uses an image, skipping
snuba-outcomes-consumer uses an image, skipping
snuba-sessions-consumer uses an image, skipping
snuba-transactions-consumer uses an image, skipping
snuba-replacer uses an image, skipping
snuba-subscription-consumer-events uses an image, skipping
snuba-subscription-consumer-transactions uses an image, skipping
symbolicator uses an image, skipping
relay uses an image, skipping
nginx uses an image, skipping
Building snuba-cleanup …
Building symbolicator-cleanup …
Building web …
Building cron …
Building worker …
Building ingest-consumer …
Building post-process-forwarder …
Building subscription-consumer-events …
Building subscription-consumer-transactions …
Building sentry-cleanup …
Building snuba-cleanup
Building subscription-consumer-transactions
Building worker
Building subscription-consumer-events
Building web
Step 1/5 : ARG SENTRY_IMAGE
Step 2/5 : ARG SENTRY_PYTHON3
Step 3/5 : FROM {SENTRY_IMAGE}{SENTRY_PYTHON3:±py3}
—> 71ac5bcb4aaa
Step 4/5 : COPY . /usr/src/sentry
—> Using cache
—> 997c3f583107
Step 5/5 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then pip install -r /usr/src/sentry/requirements.txt; fi
—> Using cache
—> ba215304a1c4

Successfully built ba215304a1c4
Step 1/5 : ARG BASE_IMAGE
Step 2/5 : FROM ${BASE_IMAGE}
—> 7cf07ea0a0a6
Step 1/5 : ARG SENTRY_IMAGEStep 3/5 : RUN apt-get update && apt-get install -y --no-install-recommends cron && rm -r /var/lib/apt/lists/*

Step 2/5 : ARG SENTRY_PYTHON3 —> Using cache

Step 3/5 : FROM {SENTRY_IMAGE}{SENTRY_PYTHON3:±py3}
—> 782756782954
Step 4/5 : COPY entrypoint.sh /entrypoint.sh
—> Using cache
—> a712a730b127
Step 5/5 : ENTRYPOINT ["/entrypoint.sh"]
—> Using cache
—> c2c461fbc5af

Successfully built c2c461fbc5af
Successfully tagged snuba-cleanup-onpremise-local:latest
—> 71ac5bcb4aaa
Step 4/5 : COPY . /usr/src/sentry
—> Using cache
—> 997c3f583107
Successfully tagged sentry-onpremise-local:latest
Building cron
Step 5/5 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then pip install -r /usr/src/sentry/requirements.txt; fi
—> Using cache
—> ba215304a1c4

Successfully built ba215304a1c4
Successfully tagged sentry-onpremise-local:latest
Building snuba-cleanup … done
Building web … done
Building subscription-consumer-events … done
Building ingest-consumer
Step 1/5 : ARG SENTRY_IMAGE
Step 2/5 : ARG SENTRY_PYTHON3
Step 3/5 : FROM {SENTRY_IMAGE}{SENTRY_PYTHON3:±py3}
—> 71ac5bcb4aaa
Step 4/5 : COPY . /usr/src/sentry
Building symbolicator-cleanup
Step 1/5 : ARG SENTRY_IMAGE
Step 2/5 : ARG SENTRY_PYTHON3 —> Using cache
—> 997c3f583107
Step 5/5 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then pip install -r /usr/src/sentry/requirements.txt; fi
—> Using cache
—> ba215304a1c4

Successfully built ba215304a1c4

Step 3/5 : FROM {SENTRY_IMAGE}{SENTRY_PYTHON3:±py3}
Successfully tagged sentry-onpremise-local:latest
—> 71ac5bcb4aaa
Step 4/5 : COPY . /usr/src/sentryBuilding post-process-forwarder

Building worker … done
—> Using cache
—> 997c3f583107
Step 5/5 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then pip install -r /usr/src/sentry/requirements.txt; fi
—> Using cache
—> ba215304a1c4

Successfully built ba215304a1c4
Successfully tagged sentry-onpremise-local:latest
Building subscription-consumer-transactions … done
Building sentry-cleanup
Step 1/5 : ARG BASE_IMAGE
Step 2/5 : FROM ${BASE_IMAGE}
—> 3a8b8c057e0c
Step 3/5 : RUN apt-get update && apt-get install -y --no-install-recommends cron && rm -r /var/lib/apt/lists/*
—> Using cache
—> 2ebfcf06999f
Step 4/5 : COPY entrypoint.sh /entrypoint.sh
—> Using cache
—> 3653e8ba5907
Step 5/5 : ENTRYPOINT ["/entrypoint.sh"]
—> Using cache
—> 8b64df2bdd44

Successfully built 8b64df2bdd44
Successfully tagged symbolicator-cleanup-onpremise-local:latest
Building symbolicator-cleanup … done
Step 1/5 : ARG SENTRY_IMAGE
Step 2/5 : ARG SENTRY_PYTHON3
Step 3/5 : FROM {SENTRY_IMAGE}{SENTRY_PYTHON3:±py3}
—> 71ac5bcb4aaa
Step 4/5 : COPY . /usr/src/sentry
—> Using cache
—> 997c3f583107
Step 5/5 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then pip install -r /usr/src/sentry/requirements.txt; fi
—> Using cache
—> ba215304a1c4

Successfully built ba215304a1c4
Successfully tagged sentry-onpremise-local:latest
Building cron … done
Step 1/5 : ARG SENTRY_IMAGE
Step 2/5 : ARG SENTRY_PYTHON3
Step 3/5 : FROM {SENTRY_IMAGE}{SENTRY_PYTHON3:±py3}
—> 71ac5bcb4aaa
Step 4/5 : COPY . /usr/src/sentry
—> Using cache
—> 997c3f583107
Step 5/5 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then pip install -r /usr/src/sentry/requirements.txt; fi
—> Using cache
—> ba215304a1c4

Successfully built ba215304a1c4
Successfully tagged sentry-onpremise-local:latest
Building ingest-consumer … done
Step 1/5 : ARG SENTRY_IMAGE
Step 2/5 : ARG SENTRY_PYTHON3
Step 3/5 : FROM {SENTRY_IMAGE}{SENTRY_PYTHON3:±py3}
—> 71ac5bcb4aaa
Step 4/5 : COPY . /usr/src/sentry
—> Using cache
—> 997c3f583107
Step 5/5 : RUN if [ -s /usr/src/sentry/requirements.txt ]; then pip install -r /usr/src/sentry/requirements.txt; fi
—> Using cache
—> ba215304a1c4

Successfully built ba215304a1c4
Step 1/5 : ARG BASE_IMAGE
Step 2/5 : FROM ${BASE_IMAGE}
Successfully tagged sentry-onpremise-local:latest
Building post-process-forwarder … done
—> ba215304a1c4
Step 3/5 : RUN apt-get update && apt-get install -y --no-install-recommends cron && rm -r /var/lib/apt/lists/*
—> Using cache
—> ac2f1fb79655
Step 4/5 : COPY entrypoint.sh /entrypoint.sh
—> Using cache
—> b2228418c3d1
Step 5/5 : ENTRYPOINT ["/entrypoint.sh"]
—> Using cache
—> 64f547cf627f

Successfully built 64f547cf627f
Successfully tagged sentry-cleanup-onpremise-local:latest
Building sentry-cleanup … done

Docker images built.
Removing network onpremise_default
Network onpremise_default not found.
Removing sentry_onpremise_snuba-sessions-consumer_1 …
Removing sentry_onpremise_snuba-outcomes-consumer_1 …
Removing sentry_onpremise_snuba-consumer_1 …
Removing sentry_onpremise_snuba-transactions-consumer_1 …
Removing sentry_onpremise_snuba-api_1 …
Removing sentry_onpremise_snuba-replacer_1 …
Removing sentry_onpremise_smtp_1 …
Removing sentry_onpremise_memcached_1 …
Removing sentry_onpremise_symbolicator_1 …
Removing sentry_onpremise_postgres_1 …
Removing sentry_onpremise_kafka_1 …
Removing sentry_onpremise_redis_1 …
Removing sentry_onpremise_zookeeper_1 …
Removing sentry_onpremise_clickhouse_1 …
Removing sentry_onpremise_redis_1 … done
Removing sentry_onpremise_kafka_1 … done
Removing sentry_onpremise_snuba-sessions-consumer_1 … done
Removing sentry_onpremise_snuba-consumer_1 … done
Removing sentry_onpremise_symbolicator_1 … done
Removing sentry_onpremise_snuba-transactions-consumer_1 … done
Removing sentry_onpremise_clickhouse_1 … done
Removing sentry_onpremise_snuba-replacer_1 … done
Removing sentry_onpremise_smtp_1 … done
Removing sentry_onpremise_snuba-outcomes-consumer_1 … done
Removing sentry_onpremise_snuba-api_1 … done
Removing sentry_onpremise_memcached_1 … done
Removing sentry_onpremise_zookeeper_1 … done
Removing sentry_onpremise_postgres_1 … done
Removing network sentry_onpremise_default
Creating network “sentry_onpremise_default” with the default driver
Bootstrapping and migrating Snuba…
Creating sentry_onpremise_zookeeper_1 …
Creating sentry_onpremise_clickhouse_1 …
Creating sentry_onpremise_redis_1 …
Creating sentry_onpremise_zookeeper_1 … done
Creating sentry_onpremise_kafka_1 …
Creating sentry_onpremise_redis_1 … done
Creating sentry_onpremise_clickhouse_1 … done
Creating sentry_onpremise_kafka_1 … done

  • ‘[’ b = - ‘]’
  • snuba bootstrap --help
  • set – snuba bootstrap --no-migrate --force
  • set gosu snuba snuba bootstrap --no-migrate --force
  • exec gosu snuba snuba bootstrap --no-migrate --force
    %3|1606918473.390|FAIL|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#192.168.160.5:9092 failed: Connection refused (after 3ms in state CONNECT)
    2020-12-02 14:14:34,387 Connection to Kafka failed (attempt 0)
    Traceback (most recent call last):
    File “/usr/src/snuba/snuba/cli/bootstrap.py”, line 55, in bootstrap
    client.list_topics(timeout=1)
    cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str=“Failed to get metadata: Local: Broker transport failure”}
    %3|1606918474.388|FAIL|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#192.168.160.5:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
    %3|1606918475.389|FAIL|rdkafka#producer-2| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#192.168.160.5:9092 failed: Connection refused (after 0ms in state CONNECT)
    %3|1606918476.389|FAIL|rdkafka#producer-2| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#192.168.160.5:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
    2020-12-02 14:14:36,390 Connection to Kafka failed (attempt 1)
    Traceback (most recent call last):
    File “/usr/src/snuba/snuba/cli/bootstrap.py”, line 55, in bootstrap
    client.list_topics(timeout=1)
    cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str=“Failed to get metadata: Local: Broker transport failure”}
    2020-12-02 14:14:38,054 Failed to create topic cdc
    Traceback (most recent call last):
    File “/usr/src/snuba/snuba/cli/bootstrap.py”, line 91, in bootstrap
    future.result()
    File “/usr/local/lib/python3.8/concurrent/futures/_base.py”, line 439, in result
    return self.__get_result()
    File “/usr/local/lib/python3.8/concurrent/futures/_base.py”, line 388, in __get_result
    raise self._exception
    cimpl.KafkaException: KafkaError{code=TOPIC_ALREADY_EXISTS,val=36,str=“Topic ‘cdc’ already exists.”}
    2020-12-02 14:14:38,055 Failed to create topic outcomes
    Traceback (most recent call last):
    File “/usr/src/snuba/snuba/cli/bootstrap.py”, line 91, in bootstrap
    future.result()
    File “/usr/local/lib/python3.8/concurrent/futures/_base.py”, line 432, in result
    return self.__get_result()
    File “/usr/local/lib/python3.8/concurrent/futures/_base.py”, line 388, in __get_result
    raise self._exception
    cimpl.KafkaException: KafkaError{code=TOPIC_ALREADY_EXISTS,val=36,str=“Topic ‘outcomes’ already exists.”}
    2020-12-02 14:14:38,055 Failed to create topic events
    Traceback (most recent call last):
    File “/usr/src/snuba/snuba/cli/bootstrap.py”, line 91, in bootstrap
    future.result()
    File “/usr/local/lib/python3.8/concurrent/futures/_base.py”, line 432, in result
    return self.__get_result()
    File “/usr/local/lib/python3.8/concurrent/futures/_base.py”, line 388, in __get_result
    raise self._exception
    cimpl.KafkaException: KafkaError{code=TOPIC_ALREADY_EXISTS,val=36,str=“Topic ‘events’ already exists.”}
    2020-12-02 14:14:38,055 Failed to create topic event-replacements
    Traceback (most recent call last):
    File “/usr/src/snuba/snuba/cli/bootstrap.py”, line 91, in bootstrap
    future.result()
    File “/usr/local/lib/python3.8/concurrent/futures/_base.py”, line 432, in result
    return self.__get_result()
    File “/usr/local/lib/python3.8/concurrent/futures/_base.py”, line 388, in __get_result
    raise self._exception
    cimpl.KafkaException: KafkaError{code=TOPIC_ALREADY_EXISTS,val=36,str=“Topic ‘event-replacements’ already exists.”}
    2020-12-02 14:14:38,055 Failed to create topic snuba-commit-log
    Traceback (most recent call last):
    File “/usr/src/snuba/snuba/cli/bootstrap.py”, line 91, in bootstrap
    future.result()
    File “/usr/local/lib/python3.8/concurrent/futures/_base.py”, line 432, in result
    return self.__get_result()
    File “/usr/local/lib/python3.8/concurrent/futures/_base.py”, line 388, in __get_result
    raise self._exception
    cimpl.KafkaException: KafkaError{code=TOPIC_ALREADY_EXISTS,val=36,str=“Topic ‘snuba-commit-log’ already exists.”}
    2020-12-02 14:14:38,055 Failed to create topic ingest-sessions
    Traceback (most recent call last):
    File “/usr/src/snuba/snuba/cli/bootstrap.py”, line 91, in bootstrap
    future.result()
    File “/usr/local/lib/python3.8/concurrent/futures/_base.py”, line 432, in result
    return self.__get_result()
    File “/usr/local/lib/python3.8/concurrent/futures/_base.py”, line 388, in __get_result
    raise self._exception
    cimpl.KafkaException: KafkaError{code=TOPIC_ALREADY_EXISTS,val=36,str=“Topic ‘ingest-sessions’ already exists.”}
    2020-12-02 14:14:38,055 Failed to create topic errors-replacements
    Traceback (most recent call last):
    File “/usr/src/snuba/snuba/cli/bootstrap.py”, line 91, in bootstrap
    future.result()
    File “/usr/local/lib/python3.8/concurrent/futures/_base.py”, line 432, in result
    return self.__get_result()
    File “/usr/local/lib/python3.8/concurrent/futures/_base.py”, line 388, in __get_result
    raise self._exception
    cimpl.KafkaException: KafkaError{code=TOPIC_ALREADY_EXISTS,val=36,str=“Topic ‘errors-replacements’ already exists.”}
    Starting sentry_onpremise_clickhouse_1 …
    Starting sentry_onpremise_redis_1 …
    Starting sentry_onpremise_zookeeper_1 …
    Starting sentry_onpremise_clickhouse_1 … done
    Starting sentry_onpremise_redis_1 … done
    Starting sentry_onpremise_zookeeper_1 … done
    Starting sentry_onpremise_kafka_1 …
    Starting sentry_onpremise_kafka_1 … done
  • ‘[’ m = - ‘]’
  • snuba migrations --help
  • set – snuba migrations migrate --force
  • set gosu snuba snuba migrations migrate --force
  • exec gosu snuba snuba migrations migrate --force
    Finished running migrations

Setting up database…
Starting sentry_onpremise_clickhouse_1 …
Starting sentry_onpremise_clickhouse_1 … done
Creating sentry_onpremise_memcached_1 …
Starting sentry_onpremise_redis_1 …
Starting sentry_onpremise_redis_1 … done
Creating sentry_onpremise_smtp_1 …
Creating sentry_onpremise_symbolicator_1 …
Creating sentry_onpremise_postgres_1 …
Starting sentry_onpremise_zookeeper_1 …
Starting sentry_onpremise_zookeeper_1 … done
Starting sentry_onpremise_kafka_1 …
Starting sentry_onpremise_kafka_1 … done
Creating sentry_onpremise_snuba-sessions-consumer_1 …
Creating sentry_onpremise_snuba-replacer_1 …
Creating sentry_onpremise_snuba-outcomes-consumer_1 …
Creating sentry_onpremise_snuba-transactions-consumer_1 …
Creating sentry_onpremise_snuba-api_1 …
Creating sentry_onpremise_snuba-consumer_1 …
Creating sentry_onpremise_smtp_1 … done
Creating sentry_onpremise_memcached_1 … done
Creating sentry_onpremise_snuba-consumer_1 … done
Creating sentry_onpremise_snuba-api_1 … done
Creating sentry_onpremise_snuba-transactions-consumer_1 … done
Creating sentry_onpremise_symbolicator_1 … done
Creating sentry_onpremise_postgres_1 … done
Creating sentry_onpremise_snuba-sessions-consumer_1 … done
Creating sentry_onpremise_snuba-replacer_1 … done
Creating sentry_onpremise_snuba-outcomes-consumer_1 … done
!! Configuration error: IOError: [Errno 13] Unable to load configuration file (Permission denied): ‘/etc/sentry/sentry.conf.py’
An error occurred, caught SIGERR on line 303
Cleaning up…

16

Hello Chad,

Ihave installed it again and it worked fine but has the issue with relay this time its not generating the credentials please find the logs below
relay_1 | error: could not parse json config file (file /work/.relay/credentials.json)
relay_1 | caused by: expected value at line 1 column 1
relay_1 | error: could not parse json config file (file /work/.relay/credentials.json)
relay_1 | caused by: expected value at line 1 column 1
relay_1 | error: could not parse json config file (file /work/.relay/credentials.json)
relay_1 | caused by: expected value at line 1 column 1
relay_1 | error: could not parse json config file (file /work/.relay/credentials.json)
relay_1 | caused by: expected value at line 1 column 1
relay_1 | error: could not parse json config file (file /work/.relay/credentials.json)
relay_1 | caused by: expected value at line 1 column 1
relay_1 | error: could not parse json config file (file /work/.relay/credentials.json)
relay_1 | caused by: expected value at line 1 column 1
relay_1 | error: could not parse json config file (file /work/.relay/credentials.json)
relay_1 | caused by: expected value at line 1 column 1
relay_1 | error: could not parse json config file (file /work/.relay/credentials.json)
relay_1 | caused by: expected value at line 1 column 1
relay_1 | error: could not parse json config file (file /work/.relay/credentials.json)
relay_1 | caused by: expected value at line 1 column 1
relay_1 | error: could not parse json config file (file /work/.relay/credentials.json)
relay_1 | caused by: expected value at line 1 column 1
relay_1 | error: could not parse json config file (file /work/.relay/credentials.json)
relay_1 | caused by: expected value at line 1 column 1

I have checked solution for this and modified the code in install.sh
with
$dcr --no-deps --entrypoint /bin/bash relay -c “cp /work/.relay/config.yml /tmp/config.yml && /bin/relay --config /tmp credentials generate > /dev/null && cat /tmp/credentials.json” > “$RELAY_CREDENTIALS_JSON”

replacing old code
dcr --no-deps -v (pwd)/$RELAY_CONFIG_YML:/tmp/config.yml relay --config /tmp credentials generate --stdout > “$RELAY_CREDENTIALS_JSON”

can you help me on this please

17

Hello chad ,

Thanks for your time its working fine now just used manual creation of credentials

docker-compose run --no-deps -v $(pwd)/relay/config.yml:/work .relay/config.yml relay credentials generate --stdout

1 Like
18

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.