Sentry stopped receiving events

amitsavani · May 3, 2021, 1:08pm

Following instances keep on restarting

sentry_onpremise_snuba-transactions-consumer_1
sentry_onpremise_snuba-consumer_1
sentry_onpremise_snuba-outcomes-consumer_1

CONTAINER ID   IMAGE                                  COMMAND                  CREATED       STATUS                          PORTS                          NAMES
43229be0c76c   nginx:1.16                             "nginx -g 'daemon of…"   6 weeks ago   Up 4 hours                      0.0.0.0:9000->80/tcp           sentry_onpremise_nginx_1
074d7d9ad7d6   getsentry/sentry:nightly               "/etc/sentry/entrypo…"   6 weeks ago   Up 3 hours                      9000/tcp                       sentry_onpremise_subscription-consumer-transactions_1
2c8356e4b17d   getsentry/sentry:nightly               "/etc/sentry/entrypo…"   6 weeks ago   Up 3 hours                      9000/tcp                       sentry_onpremise_ingest-consumer_1
574386c529ff   getsentry/sentry:nightly               "/etc/sentry/entrypo…"   6 weeks ago   Up 4 hours                      9000/tcp                       sentry_onpremise_worker_1
15b16cbb3890   getsentry/sentry:nightly               "/etc/sentry/entrypo…"   6 weeks ago   Up 4 hours                      9000/tcp                       sentry_onpremise_web_1
c7f2f90528f6   sentry-cleanup-onpremise-local         "/entrypoint.sh '0 0…"   6 weeks ago   Up 4 hours                      9000/tcp                       sentry_onpremise_sentry-cleanup_1
ff86fdc545dd   getsentry/sentry:nightly               "/etc/sentry/entrypo…"   6 weeks ago   Up 3 hours                      9000/tcp                       sentry_onpremise_subscription-consumer-events_1
b3c28f20d97a   getsentry/sentry:nightly               "/etc/sentry/entrypo…"   6 weeks ago   Up 4 hours                      9000/tcp                       sentry_onpremise_post-process-forwarder_1
a1ae2a844005   getsentry/sentry:nightly               "/etc/sentry/entrypo…"   6 weeks ago   Up 4 hours                      9000/tcp                       sentry_onpremise_cron_1
26c70390cee0   getsentry/snuba:nightly                "./docker_entrypoint…"   6 weeks ago   Up 3 hours                      1218/tcp                       sentry_onpremise_snuba-replacer_1
5f42ccf0d1d5   getsentry/snuba:nightly                "./docker_entrypoint…"   6 weeks ago   Restarting (1) 17 seconds ago                                  sentry_onpremise_snuba-transactions-consumer_1
5a4be50e9d6d   getsentry/snuba:nightly                "./docker_entrypoint…"   6 weeks ago   Restarting (1) 35 seconds ago                                  sentry_onpremise_snuba-consumer_1
8fc4efa51906   getsentry/snuba:nightly                "./docker_entrypoint…"   6 weeks ago   Up 4 seconds                    1218/tcp                       sentry_onpremise_snuba-subscription-consumer-events_1
596d3f39a9de   getsentry/snuba:nightly                "./docker_entrypoint…"   6 weeks ago   Up 4 hours                      1218/tcp                       sentry_onpremise_snuba-api_1
2e4c945f0772   getsentry/snuba:nightly                "./docker_entrypoint…"   6 weeks ago   Restarting (1) 56 seconds ago                                  sentry_onpremise_snuba-outcomes-consumer_1
fc5892e4d8f4   getsentry/snuba:nightly                "./docker_entrypoint…"   6 weeks ago   Up 4 hours                      1218/tcp                       sentry_onpremise_snuba-sessions-consumer_1
57b0cfa42a93   getsentry/relay:nightly                "/bin/bash /docker-e…"   6 weeks ago   Up 4 hours                      3000/tcp                       sentry_onpremise_relay_1
a9bbe7f19a26   snuba-cleanup-onpremise-local          "/entrypoint.sh '*/5…"   6 weeks ago   Up 4 hours                      1218/tcp                       sentry_onpremise_snuba-cleanup_1
753b533395c4   getsentry/snuba:nightly                "./docker_entrypoint…"   6 weeks ago   Up 4 seconds                    1218/tcp                       sentry_onpremise_snuba-subscription-consumer-transactions_1
0edd51811a14   confluentinc/cp-kafka:5.5.0            "/etc/confluent/dock…"   6 weeks ago   Up 4 hours                      9092/tcp                       sentry_onpremise_kafka_1
119ddb7c48e2   memcached:1.5-alpine                   "docker-entrypoint.s…"   6 weeks ago   Up 4 hours                      11211/tcp                      sentry_onpremise_memcached_1
3382fbbaca96   confluentinc/cp-zookeeper:5.5.0        "/etc/confluent/dock…"   6 weeks ago   Up 4 hours                      2181/tcp, 2888/tcp, 3888/tcp   sentry_onpremise_zookeeper_1
23a97ddb3b93   tianon/exim4                           "docker-entrypoint.s…"   6 weeks ago   Up 4 hours                      25/tcp                         sentry_onpremise_smtp_1
9e92b4b4ce91   symbolicator-cleanup-onpremise-local   "/entrypoint.sh '55 …"   6 weeks ago   Up 4 hours                      3021/tcp                       sentry_onpremise_symbolicator-cleanup_1
5d962c941344   getsentry/symbolicator:nightly         "/bin/bash /docker-e…"   6 weeks ago   Up 4 hours                      3021/tcp                       sentry_onpremise_symbolicator_1
f39ce15010a0   redis:5.0-alpine                       "docker-entrypoint.s…"   6 weeks ago   Up 4 hours                      6379/tcp                       sentry_onpremise_redis_1
c79ae4f98009   postgres:9.6                           "docker-entrypoint.s…"   6 weeks ago   Up 4 hours                      5432/tcp                       sentry_onpremise_postgres_1
e211522982e8   yandex/clickhouse-server:20.3.9.70     "/entrypoint.sh"         6 weeks ago   Up 4 hours                      8123/tcp, 9000/tcp, 9009/tcp   sentry_onpremise_clickhouse_1

BYK · May 3, 2021, 1:25pm

Without the logs, we cannot say anything about why.

amitsavani · May 3, 2021, 1:49pm

Thanks for the quick response.

Would you mind help me to get necessary logs that can help you to review?

BYK · May 3, 2021, 2:00pm

Would https://develop.sentry.dev/self-hosted/troubleshooting/#general help?

amitsavani · May 3, 2021, 2:06pm

I guess Kafka related issue

subscription-consumer-events_1              | RuntimeError: Timeout when waiting for Kafka topic 'events-subscription-results' to become available, last error: KafkaError{code=LEADER_NOT_AVAILABLE,val=5,str="Broker: Leader not available"}
snuba-subscription-consumer-transactions_1  | + set gosu snuba snuba subscriptions --auto-offset-reset=latest --consumer-group=snuba-transactions-subscriptions-consumers --topic=events --result-topic=transactions-subscription-results --dataset=transactions --commit-log-topic=snuba-commit-log --commit-log-group=transactions_group --delay-seconds=60 --schedule-ttl=60
web_1                                       | 09:32:50 [INFO] sentry.deletions.api: object.delete.queued (object_id=1 transaction_id='0e05fc5ab4354c47b55562cffc99d79a' model='Group')
snuba-transactions-consumer_1               | snuba.utils.streams.backends.abstract.OffsetOutOfRange: KafkaError{code=OFFSET_OUT_OF_RANGE,val=1,str="Broker: Offset out of range"}
symbolicator-cleanup_1                      | 2021-05-02T23:55:01Z [symbolicator::cache] INFO: Cleaning up cache: cficaches
symbolicator-cleanup_1                      | SHELL=/bin/bash
worker_1                                    | 13:55:13 [WARNING] sentry.tasks.release_registry: Release registry URL is not specified, skipping the task.
subscription-consumer-transactions_1        | RuntimeError: Timeout when waiting for Kafka topic 'transactions-subscription-results' to become available, last error: KafkaError{code=LEADER_NOT_AVAILABLE,val=5,str="Broker: Leader not available"}
snuba-subscription-consumer-events_1        |     raise OffsetOutOfRange(str(error))
subscription-consumer-events_1              | 09:30:52 [INFO] sentry.plugins.github: apps-not-configured
snuba-subscription-consumer-transactions_1  | + exec gosu snuba snuba subscriptions --auto-offset-reset=latest --consumer-group=snuba-transactions-subscriptions-consumers --topic=events --result-topic=transactions-subscription-results --dataset=transactions --commit-log-topic=snuba-commit-log --commit-log-group=transactions_group --delay-seconds=60 --schedule-ttl=60
web_1                                       | 09:33:01 [INFO] sentry.api: group_external_issue.get (user='amit@rentola.com')
symbolicator-cleanup_1                      | BASH_ENV=/container.env
zookeeper_1                                 | ===> Check if /var/lib/zookeeper/log is writable ...
worker_1                                    | 13:55:43 [INFO] sentry.tasks.update_user_reports: update_user_reports.records_updated (reports_to_update=0 reports_with_event=0 updated_reports=0)
subscription-consumer-transactions_1        | 09:30:51 [INFO] sentry.plugins.github: apps-not-configured
snuba-subscription-consumer-events_1        | snuba.utils.streams.backends.abstract.OffsetOutOfRange: KafkaError{code=OFFSET_OUT_OF_RANGE,val=1,str="Broker: Offset out of range"}
subscription-consumer-events_1              | 09:30:55 [INFO] sentry.snuba.query_subscription_consumer: query-subscription-consumer.on_assign (offsets='{0: None}' partitions='[TopicPartition{topic=events-subscription-results,partition=0,offset=-1001,error=None}]')
web_1                                       | 11:51:59 [INFO] sentry.api: group_external_issue.get (user='amit@rentola.com')
symbolicator-cleanup_1                      | 55 23 * * * gosu symbolicator symbolicator cleanup > /proc/1/fd/1 2>/proc/1/fd/2
zookeeper_1                                 | ===> Launching ... 
worker_1                                    | 14:00:13 [WARNING] sentry.tasks.release_registry: Release registry URL is not specified, skipping the task.
subscription-consumer-transactions_1        | 09:30:55 [INFO] sentry.snuba.query_subscription_consumer: query-subscription-consumer.on_assign (offsets='{0: None}' partitions='[TopicPartition{topic=transactions-subscription-results,partition=0,offset=-1001,error=None}]')
snuba-subscription-consumer-events_1        | + '[' s = - ']'
zookeeper_1                                 | ===> Launching zookeeper ... 
snuba-subscription-consumer-events_1        | + snuba subscriptions --help
zookeeper_1                                 | [2021-05-03 09:30:11,219] WARN Either no config or no quorum defined in config, running  in standalone mode (org.apache.zookeeper.server.quorum.QuorumPeerMain)
zookeeper_1                                 | [2021-05-03 09:30:11,533] WARN o.e.j.s.ServletContextHandler@4d95d2a2{/,null,UNAVAILABLE} contextPath ends with /* (org.eclipse.jetty.server.handler.ContextHandler)
zookeeper_1                                 | [2021-05-03 09:30:11,533] WARN Empty contextPath (org.eclipse.jetty.server.handler.ContextHandler)

BYK · May 3, 2021, 2:18pm

Looks like it but the logs you share seem clipped so hard to be sure with this much.

system · August 1, 2021, 2:19pm

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Sentry stops registering events after a while On-Premise	4	2375	July 2, 2021
Sentry is totally dead On-Premise	4	5999	May 14, 2020
Clients sending in events resulting in 502 error On-Premise	5	1949	October 11, 2021
Snuba stuck restarting, "no such option --dataset" On-Premise	12	4539	May 25, 2020
New events not processed or displayed On-Premise	3	2457	June 2, 2021

Sentry stopped receiving events

Related topics