Sentry stopped receiving events

Following instances keep on restarting

sentry_onpremise_snuba-transactions-consumer_1
sentry_onpremise_snuba-consumer_1
sentry_onpremise_snuba-outcomes-consumer_1

CONTAINER ID   IMAGE                                  COMMAND                  CREATED       STATUS                          PORTS                          NAMES
43229be0c76c   nginx:1.16                             "nginx -g 'daemon of…"   6 weeks ago   Up 4 hours                      0.0.0.0:9000->80/tcp           sentry_onpremise_nginx_1
074d7d9ad7d6   getsentry/sentry:nightly               "/etc/sentry/entrypo…"   6 weeks ago   Up 3 hours                      9000/tcp                       sentry_onpremise_subscription-consumer-transactions_1
2c8356e4b17d   getsentry/sentry:nightly               "/etc/sentry/entrypo…"   6 weeks ago   Up 3 hours                      9000/tcp                       sentry_onpremise_ingest-consumer_1
574386c529ff   getsentry/sentry:nightly               "/etc/sentry/entrypo…"   6 weeks ago   Up 4 hours                      9000/tcp                       sentry_onpremise_worker_1
15b16cbb3890   getsentry/sentry:nightly               "/etc/sentry/entrypo…"   6 weeks ago   Up 4 hours                      9000/tcp                       sentry_onpremise_web_1
c7f2f90528f6   sentry-cleanup-onpremise-local         "/entrypoint.sh '0 0…"   6 weeks ago   Up 4 hours                      9000/tcp                       sentry_onpremise_sentry-cleanup_1
ff86fdc545dd   getsentry/sentry:nightly               "/etc/sentry/entrypo…"   6 weeks ago   Up 3 hours                      9000/tcp                       sentry_onpremise_subscription-consumer-events_1
b3c28f20d97a   getsentry/sentry:nightly               "/etc/sentry/entrypo…"   6 weeks ago   Up 4 hours                      9000/tcp                       sentry_onpremise_post-process-forwarder_1
a1ae2a844005   getsentry/sentry:nightly               "/etc/sentry/entrypo…"   6 weeks ago   Up 4 hours                      9000/tcp                       sentry_onpremise_cron_1
26c70390cee0   getsentry/snuba:nightly                "./docker_entrypoint…"   6 weeks ago   Up 3 hours                      1218/tcp                       sentry_onpremise_snuba-replacer_1
5f42ccf0d1d5   getsentry/snuba:nightly                "./docker_entrypoint…"   6 weeks ago   Restarting (1) 17 seconds ago                                  sentry_onpremise_snuba-transactions-consumer_1
5a4be50e9d6d   getsentry/snuba:nightly                "./docker_entrypoint…"   6 weeks ago   Restarting (1) 35 seconds ago                                  sentry_onpremise_snuba-consumer_1
8fc4efa51906   getsentry/snuba:nightly                "./docker_entrypoint…"   6 weeks ago   Up 4 seconds                    1218/tcp                       sentry_onpremise_snuba-subscription-consumer-events_1
596d3f39a9de   getsentry/snuba:nightly                "./docker_entrypoint…"   6 weeks ago   Up 4 hours                      1218/tcp                       sentry_onpremise_snuba-api_1
2e4c945f0772   getsentry/snuba:nightly                "./docker_entrypoint…"   6 weeks ago   Restarting (1) 56 seconds ago                                  sentry_onpremise_snuba-outcomes-consumer_1
fc5892e4d8f4   getsentry/snuba:nightly                "./docker_entrypoint…"   6 weeks ago   Up 4 hours                      1218/tcp                       sentry_onpremise_snuba-sessions-consumer_1
57b0cfa42a93   getsentry/relay:nightly                "/bin/bash /docker-e…"   6 weeks ago   Up 4 hours                      3000/tcp                       sentry_onpremise_relay_1
a9bbe7f19a26   snuba-cleanup-onpremise-local          "/entrypoint.sh '*/5…"   6 weeks ago   Up 4 hours                      1218/tcp                       sentry_onpremise_snuba-cleanup_1
753b533395c4   getsentry/snuba:nightly                "./docker_entrypoint…"   6 weeks ago   Up 4 seconds                    1218/tcp                       sentry_onpremise_snuba-subscription-consumer-transactions_1
0edd51811a14   confluentinc/cp-kafka:5.5.0            "/etc/confluent/dock…"   6 weeks ago   Up 4 hours                      9092/tcp                       sentry_onpremise_kafka_1
119ddb7c48e2   memcached:1.5-alpine                   "docker-entrypoint.s…"   6 weeks ago   Up 4 hours                      11211/tcp                      sentry_onpremise_memcached_1
3382fbbaca96   confluentinc/cp-zookeeper:5.5.0        "/etc/confluent/dock…"   6 weeks ago   Up 4 hours                      2181/tcp, 2888/tcp, 3888/tcp   sentry_onpremise_zookeeper_1
23a97ddb3b93   tianon/exim4                           "docker-entrypoint.s…"   6 weeks ago   Up 4 hours                      25/tcp                         sentry_onpremise_smtp_1
9e92b4b4ce91   symbolicator-cleanup-onpremise-local   "/entrypoint.sh '55 …"   6 weeks ago   Up 4 hours                      3021/tcp                       sentry_onpremise_symbolicator-cleanup_1
5d962c941344   getsentry/symbolicator:nightly         "/bin/bash /docker-e…"   6 weeks ago   Up 4 hours                      3021/tcp                       sentry_onpremise_symbolicator_1
f39ce15010a0   redis:5.0-alpine                       "docker-entrypoint.s…"   6 weeks ago   Up 4 hours                      6379/tcp                       sentry_onpremise_redis_1
c79ae4f98009   postgres:9.6                           "docker-entrypoint.s…"   6 weeks ago   Up 4 hours                      5432/tcp                       sentry_onpremise_postgres_1
e211522982e8   yandex/clickhouse-server:20.3.9.70     "/entrypoint.sh"         6 weeks ago   Up 4 hours                      8123/tcp, 9000/tcp, 9009/tcp   sentry_onpremise_clickhouse_1

Without the logs, we cannot say anything about why.

Thanks for the quick response.

Would you mind help me to get necessary logs that can help you to review?

Would https://develop.sentry.dev/self-hosted/troubleshooting/#general help?

I guess Kafka related issue

subscription-consumer-events_1              | RuntimeError: Timeout when waiting for Kafka topic 'events-subscription-results' to become available, last error: KafkaError{code=LEADER_NOT_AVAILABLE,val=5,str="Broker: Leader not available"}
snuba-subscription-consumer-transactions_1  | + set gosu snuba snuba subscriptions --auto-offset-reset=latest --consumer-group=snuba-transactions-subscriptions-consumers --topic=events --result-topic=transactions-subscription-results --dataset=transactions --commit-log-topic=snuba-commit-log --commit-log-group=transactions_group --delay-seconds=60 --schedule-ttl=60
web_1                                       | 09:32:50 [INFO] sentry.deletions.api: object.delete.queued (object_id=1 transaction_id='0e05fc5ab4354c47b55562cffc99d79a' model='Group')
snuba-transactions-consumer_1               | snuba.utils.streams.backends.abstract.OffsetOutOfRange: KafkaError{code=OFFSET_OUT_OF_RANGE,val=1,str="Broker: Offset out of range"}
symbolicator-cleanup_1                      | 2021-05-02T23:55:01Z [symbolicator::cache] INFO: Cleaning up cache: cficaches
symbolicator-cleanup_1                      | SHELL=/bin/bash
worker_1                                    | 13:55:13 [WARNING] sentry.tasks.release_registry: Release registry URL is not specified, skipping the task.
subscription-consumer-transactions_1        | RuntimeError: Timeout when waiting for Kafka topic 'transactions-subscription-results' to become available, last error: KafkaError{code=LEADER_NOT_AVAILABLE,val=5,str="Broker: Leader not available"}
snuba-subscription-consumer-events_1        |     raise OffsetOutOfRange(str(error))
subscription-consumer-events_1              | 09:30:52 [INFO] sentry.plugins.github: apps-not-configured
snuba-subscription-consumer-transactions_1  | + exec gosu snuba snuba subscriptions --auto-offset-reset=latest --consumer-group=snuba-transactions-subscriptions-consumers --topic=events --result-topic=transactions-subscription-results --dataset=transactions --commit-log-topic=snuba-commit-log --commit-log-group=transactions_group --delay-seconds=60 --schedule-ttl=60
web_1                                       | 09:33:01 [INFO] sentry.api: group_external_issue.get (user='amit@rentola.com')
symbolicator-cleanup_1                      | BASH_ENV=/container.env
zookeeper_1                                 | ===> Check if /var/lib/zookeeper/log is writable ...
worker_1                                    | 13:55:43 [INFO] sentry.tasks.update_user_reports: update_user_reports.records_updated (reports_to_update=0 reports_with_event=0 updated_reports=0)
subscription-consumer-transactions_1        | 09:30:51 [INFO] sentry.plugins.github: apps-not-configured
snuba-subscription-consumer-events_1        | snuba.utils.streams.backends.abstract.OffsetOutOfRange: KafkaError{code=OFFSET_OUT_OF_RANGE,val=1,str="Broker: Offset out of range"}
subscription-consumer-events_1              | 09:30:55 [INFO] sentry.snuba.query_subscription_consumer: query-subscription-consumer.on_assign (offsets='{0: None}' partitions='[TopicPartition{topic=events-subscription-results,partition=0,offset=-1001,error=None}]')
web_1                                       | 11:51:59 [INFO] sentry.api: group_external_issue.get (user='amit@rentola.com')
symbolicator-cleanup_1                      | 55 23 * * * gosu symbolicator symbolicator cleanup > /proc/1/fd/1 2>/proc/1/fd/2
zookeeper_1                                 | ===> Launching ... 
worker_1                                    | 14:00:13 [WARNING] sentry.tasks.release_registry: Release registry URL is not specified, skipping the task.
subscription-consumer-transactions_1        | 09:30:55 [INFO] sentry.snuba.query_subscription_consumer: query-subscription-consumer.on_assign (offsets='{0: None}' partitions='[TopicPartition{topic=transactions-subscription-results,partition=0,offset=-1001,error=None}]')
snuba-subscription-consumer-events_1        | + '[' s = - ']'
zookeeper_1                                 | ===> Launching zookeeper ... 
snuba-subscription-consumer-events_1        | + snuba subscriptions --help
zookeeper_1                                 | [2021-05-03 09:30:11,219] WARN Either no config or no quorum defined in config, running  in standalone mode (org.apache.zookeeper.server.quorum.QuorumPeerMain)
zookeeper_1                                 | [2021-05-03 09:30:11,533] WARN o.e.j.s.ServletContextHandler@4d95d2a2{/,null,UNAVAILABLE} contextPath ends with /* (org.eclipse.jetty.server.handler.ContextHandler)
zookeeper_1                                 | [2021-05-03 09:30:11,533] WARN Empty contextPath (org.eclipse.jetty.server.handler.ContextHandler)

Looks like it but the logs you share seem clipped so hard to be sure with this much.