Sentry 21.8.0 Helm Chart - Old events not showing up

Hello,
I’m trying to deploy Sentry Helm Chart v21.8.0 in our k8s cluster using our external PostgreSQL DB (already upgraded to 21.8.0 and all DB migrations have been executed). For some strange reason, the old events are not showing up on Sentry UI. I was able to deploy the chart successfully once and the old events showed up. I remember in that case after deploying the chart, I got this message on the UI - Background workers haven’t checked in recently. This is likely an issue with your configuration or the workers aren’t running, which prompted me to execute sentry run cron and sentry run web processes in sentry-web pod.
Now in order to ensure and test that what I did the first time works for us the second time also, I am trying to deploy Sentry again but the old events fail to show up. I executed snuba bootstrap --force inside the snuba-api pod, followed by snuba migrations migrate (not sure if the migrations migrate step is needed). The output of snuba bootstrap --force shows that the Kafka topics were created and some migrations have been run and I can see a few events on Sentry UI from the Internal project but not our old events.
There are 3 worker pods created, logs from worker-3 pod,


  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 384, in _make_request
    six.raise_from(e, None)
  File "<string>", line 2, in raise_from
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 380, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/local/lib/python3.6/site-packages/sentry_sdk/integrations/stdlib.py", line 102, in getresponse
    rv = real_getresponse(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/http/client.py", line 1379, in getresponse
    response.begin()
  File "/usr/local/lib/python3.6/http/client.py", line 311, in begin
    version, status, reason = self._read_status()
  File "/usr/local/lib/python3.6/http/client.py", line 272, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/local/lib/python3.6/socket.py", line 586, in readinto
    return self._sock.recv_into(b)
  File "/usr/local/lib/python3.6/ssl.py", line 1012, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/local/lib/python3.6/ssl.py", line 874, in read
    return self._sslobj.read(len, buffer)
  File "/usr/local/lib/python3.6/ssl.py", line 631, in read
    v = self._sslobj.read(len, buffer)
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
16:25:40 [ERROR] sentry_sdk.errors: Internal error in sentry_sdk
16:29:33 [WARNING] sentry.tasks.release_registry: Release registry URL is not specified, skipping the task.
16:29:33 [INFO] sentry.tasks.update_user_reports: update_user_reports.records_updated (reports_to_update=0 reports_with_event=0 updated_reports=0)
16:30:40 [WARNING] sentry.tasks.release_registry: Release registry URL is not specified, skipping the task.

Logs from 1 of the Kafka pods,

[2022-01-21 14:15:20,939] INFO [Log partition=__consumer_offsets-11, dir=/bitnami/kafka/data] Loading producer state till offset 0 with message format version 2 (kafka.log.Log)

[2022-01-21 14:15:20,940] INFO Created log for partition __consumer_offsets-11 in /bitnami/kafka/data/__consumer_offsets-11 with properties {compression.type -> producer, min.insync.replicas -> 1, message.downconversion.enable -> true, segment.jitter.ms -> 0, cleanup.policy -> compact, flush.ms -> 1000, retention.ms -> 604800000, segment.bytes -> 104857600, flush.messages -> 10000, message.format.version -> 2.6-IV0, max.compaction.lag.ms -> 9223372036854775807, file.delete.delay.ms -> 60000, max.message.bytes -> 50000000, min.compaction.lag.ms -> 0, message.timestamp.type -> CreateTime, preallocate -> false, index.interval.bytes -> 4096, min.cleanable.dirty.ratio -> 0.5, unclean.leader.election.enable -> false, retention.bytes -> 1073741824, delete.retention.ms -> 86400000, segment.ms -> 604800000, message.timestamp.difference.max.ms -> 9223372036854775807, segment.index.bytes -> 10485760}. (kafka.log.LogManager)

[2022-01-21 14:15:20,940] INFO [Partition __consumer_offsets-11 broker=1] No checkpointed highwatermark is found for partition __consumer_offsets-11 (kafka.cluster.Partition)

[2022-01-21 14:15:20,940] INFO [Partition __consumer_offsets-11 broker=1] Log loaded for partition __consumer_offsets-11 with initial high watermark 0 (kafka.cluster.Partition)

[2022-01-21 14:15:20,942] INFO [Log partition=__consumer_offsets-30, dir=/bitnami/kafka/data] Loading producer state till offset 0 with message format version 2 (kafka.log.Log)

[2022-01-21 14:15:20,943] INFO Created log for partition __consumer_offsets-30 in /bitnami/kafka/data/__consumer_offsets-30 with properties {compression.type -> producer, min.insync.replicas -> 1, message.downconversion.enable -> true, segment.jitter.ms -> 0, cleanup.policy -> compact, flush.ms -> 1000, retention.ms -> 604800000, segment.bytes -> 104857600, flush.messages -> 10000, message.format.version -> 2.6-IV0, max.compaction.lag.ms -> 9223372036854775807, file.delete.delay.ms -> 60000, max.message.bytes -> 50000000, min.compaction.lag.ms -> 0, message.timestamp.type -> CreateTime, preallocate -> false, index.interval.bytes -> 4096, min.cleanable.dirty.ratio -> 0.5, unclean.leader.election.enable -> false, retention.bytes -> 1073741824, delete.retention.ms -> 86400000, segment.ms -> 604800000, message.timestamp.difference.max.ms -> 9223372036854775807, segment.index.bytes -> 10485760}. (kafka.log.LogManager)

[2022-01-21 14:15:20,943] INFO [Partition __consumer_offsets-30 broker=1] No checkpointed highwatermark is found for partition __consumer_offsets-30 (kafka.cluster.Partition)

[2022-01-21 14:15:20,943] INFO [Partition __consumer_offsets-30 broker=1] Log loaded for partition __consumer_offsets-30 with initial high watermark 0 (kafka.cluster.Partition)

[2022-01-21 14:15:20,945] INFO [Log partition=__consumer_offsets-27, dir=/bitnami/kafka/data] Loading producer state till offset 0 with message format version 2 (kafka.log.Log)

[2022-01-21 14:15:20,946] INFO Created log for partition __consumer_offsets-27 in /bitnami/kafka/data/__consumer_offsets-27 with properties {compression.type -> producer, min.insync.replicas -> 1, message.downconversion.enable -> true, segment.jitter.ms -> 0, cleanup.policy -> compact, flush.ms -> 1000, retention.ms -> 604800000, segment.bytes -> 104857600, flush.messages -> 10000, message.format.version -> 2.6-IV0, max.compaction.lag.ms -> 9223372036854775807, file.delete.delay.ms -> 60000, max.message.bytes -> 50000000, min.compaction.lag.ms -> 0, message.timestamp.type -> CreateTime, preallocate -> false, index.interval.bytes -> 4096, min.cleanable.dirty.ratio -> 0.5, unclean.leader.election.enable -> false, retention.bytes -> 1073741824, delete.retention.ms -> 86400000, segment.ms -> 604800000, message.timestamp.difference.max.ms -> 9223372036854775807, segment.index.bytes -> 10485760}. (kafka.log.LogManager)

[2022-01-21 14:15:20,946] INFO [Partition __consumer_offsets-27 broker=1] No checkpointed highwatermark is found for partition __consumer_offsets-27 (kafka.cluster.Partition)

[2022-01-21 14:15:20,946] INFO [Partition __consumer_offsets-27 broker=1] Log loaded for partition __consumer_offsets-27 with initial high watermark 0 (kafka.cluster.Partition)

[2022-01-21 14:15:20,949] INFO [Log partition=__consumer_offsets-8, dir=/bitnami/kafka/data] Loading producer state till offset 0 with message format version 2 (kafka.log.Log)

[2022-01-21 14:15:20,950] INFO Created log for partition __consumer_offsets-8 in /bitnami/kafka/data/__consumer_offsets-8 with properties {compression.type -> producer, min.insync.replicas -> 1, message.downconversion.enable -> true, segment.jitter.ms -> 0, cleanup.policy -> compact, flush.ms -> 1000, retention.ms -> 604800000, segment.bytes -> 104857600, flush.messages -> 10000, message.format.version -> 2.6-IV0, max.compaction.lag.ms -> 9223372036854775807, file.delete.delay.ms -> 60000, max.message.bytes -> 50000000, min.compaction.lag.ms -> 0, message.timestamp.type -> CreateTime, preallocate -> false, index.interval.bytes -> 4096, min.cleanable.dirty.ratio -> 0.5, unclean.leader.election.enable -> false, retention.bytes -> 1073741824, delete.retention.ms -> 86400000, segment.ms -> 604800000, message.timestamp.difference.max.ms -> 9223372036854775807, segment.index.bytes -> 10485760}. (kafka.log.LogManager)

[2022-01-21 14:15:20,950] INFO [Partition __consumer_offsets-8 broker=1] No checkpointed highwatermark is found for partition __consumer_offsets-8 (kafka.cluster.Partition)

[2022-01-21 14:15:20,950] INFO [Partition __consumer_offsets-8 broker=1] Log loaded for partition __consumer_offsets-8 with initial high watermark 0 (kafka.cluster.Partition)

[2022-01-21 14:15:20,952] INFO [Log partition=__consumer_offsets-24, dir=/bitnami/kafka/data] Loading producer state till offset 0 with message format version 2 (kafka.log.Log)

[2022-01-21 14:15:20,953] INFO Created log for partition __consumer_offsets-24 in /bitnami/kafka/data/__consumer_offsets-24 with properties {compression.type -> producer, min.insync.replicas -> 1, message.downconversion.enable -> true, segment.jitter.ms -> 0, cleanup.policy -> compact, flush.ms -> 1000, retention.ms -> 604800000, segment.bytes -> 104857600, flush.messages -> 10000, message.format.version -> 2.6-IV0, max.compaction.lag.ms -> 9223372036854775807, file.delete.delay.ms -> 60000, max.message.bytes -> 50000000, min.compaction.lag.ms -> 0, message.timestamp.type -> CreateTime, preallocate -> false, index.interval.bytes -> 4096, min.cleanable.dirty.ratio -> 0.5, unclean.leader.election.enable -> false, retention.bytes -> 1073741824, delete.retention.ms -> 86400000, segment.ms -> 604800000, message.timestamp.difference.max.ms -> 9223372036854775807, segment.index.bytes -> 10485760}. (kafka.log.LogManager)

[2022-01-21 14:15:20,953] INFO [Partition __consumer_offsets-24 broker=1] No checkpointed highwatermark is found for partition __consumer_offsets-24 (kafka.cluster.Partition)

[2022-01-21 14:15:20,953] INFO [Partition __consumer_offsets-24 broker=1] Log loaded for partition __consumer_offsets-24 with initial high watermark 0 (kafka.cluster.Partition)

[2022-01-21 14:15:20,956] INFO [Log partition=__consumer_offsets-5, dir=/bitnami/kafka/data] Loading producer state till offset 0 with message format version 2 (kafka.log.Log)

[2022-01-21 14:15:20,957] INFO Created log for partition __consumer_offsets-5 in /bitnami/kafka/data/__consumer_offsets-5 with properties {compression.type -> producer, min.insync.replicas -> 1, message.downconversion.enable -> true, segment.jitter.ms -> 0, cleanup.policy -> compact, flush.ms -> 1000, retention.ms -> 604800000, segment.bytes -> 104857600, flush.messages -> 10000, message.format.version -> 2.6-IV0, max.compaction.lag.ms -> 9223372036854775807, file.delete.delay.ms -> 60000, max.message.bytes -> 50000000, min.compaction.lag.ms -> 0, message.timestamp.type -> CreateTime, preallocate -> false, index.interval.bytes -> 4096, min.cleanable.dirty.ratio -> 0.5, unclean.leader.election.enable -> false, retention.bytes -> 1073741824, delete.retention.ms -> 86400000, segment.ms -> 604800000, message.timestamp.difference.max.ms -> 9223372036854775807, segment.index.bytes -> 10485760}. (kafka.log.LogManager)

There is 1 ingest consumer pod,


14:15:22 [WARNING] batching-kafka-consumer: Topic 'ingest-transactions' or its partitions are not ready, retrying...

14:15:22 [WARNING] batching-kafka-consumer: Topic 'ingest-events' or its partitions are not ready, retrying...

14:15:22 [INFO] batching-kafka-consumer: New partitions assigned: [TopicPartition{topic=ingest-attachments,partition=0,offset=-1001,error=None}, TopicPartition{topic=ingest-events,partition=0,offset=-1001,error=None}, TopicPartition{topic=ingest-transactions,partition=0,offset=-1001,error=None}]

14:18:59 [INFO] batching-kafka-consumer: Flushing 1 items (from {('ingest-transactions', 0): [0, 0]}): forced:False size:False time:True

14:18:59 [INFO] batching-kafka-consumer: Worker flush took 64ms

14:19:54 [INFO] batching-kafka-consumer: Flushing 1 items (from {('ingest-events', 0): [0, 0]}): forced:False size:False time:True

14:19:54 [INFO] batching-kafka-consumer: Worker flush took 10ms

14:19:59 [INFO] batching-kafka-consumer: Flushing 1 items (from {('ingest-transactions', 0): [1, 1]}): forced:False size:False time:True

14:19:59 [INFO] batching-kafka-consumer: Worker flush took 8ms

14:20:59 [INFO] batching-kafka-consumer: Flushing 1 items (from {('ingest-transactions', 0): [2, 2]}): forced:False size:False time:True

14:20:59 [INFO] batching-kafka-consumer: Worker flush took 8ms

14:21:59 [INFO] batching-kafka-consumer: Flushing 1 items (from {('ingest-transactions', 0): [3, 3]}): forced:False size:False time:True

14:21:59 [INFO] batching-kafka-consumer: Worker flush took 8ms

14:22:59 [INFO] batching-kafka-consumer: Flushing 1 items (from {('ingest-transactions', 0): [4, 4]}): forced:False size:False time:True

14:22:59 [INFO] batching-kafka-consumer: Worker flush took 8ms

14:23:59 [INFO] batching-kafka-consumer: Flushing 1 items (from {('ingest-transactions', 0): [5, 5]}): forced:False size:False time:True

14:23:59 [INFO] batching-kafka-consumer: Worker flush took 8ms

Would really helpful if you could guide me in the right direction.

NOTE - Unable to edit the Category of this post to On-Premise. Don’t know why.

@BYK