Event from `ingest-events` topic not getting through to `events` topic

On-Premise
1

Hello,

I am trying to send event by sentry-cli to default internal project and see it in ui. I can see my event in ingest-events kafka topic but I do not see it coming though to events topic.

Sentry 20.9.0 - fresh install on Kubernetes (one worker, 2 clickhouse replicas, external postgres otherwise standard)

Event sent by ./sentry-cli-Linux-x86_64 send-event --no-environ -m "ahoj" --log-level=debug

“Sample event” added by UI to empty project is processed ok.

Potentially relevant entries from logs:

Sentry-worker

07:16:18 [INFO] sentry.tasks.update_user_reports: update_user_reports.records_updated (reports_with_event=0 updated_reports=0 reports_to_update=0)
07:16:18 [WARNING] sentry.tasks.release_registry: Release registry URL is not specified, skipping the task.
07:21:18 [WARNING] sentry.tasks.release_registry: Release registry URL is not specified, skipping the task.

Snuba-replacer

2020-10-22 16:36:12,443 New partitions assigned: {Partition(topic=Topic(name='event-replacements'), index=0): 0}
%4|1603427381.632|SESSTMOUT|rdkafka#consumer-1| [thrd:main]: Consumer group session timed out (in join-state started) after 10319 ms without a successful response from the group coordinator (broker 0, last error was Success): revoking assignment and rejoining group
2020-10-23 04:29:41,632 Partitions revoked: [Partition(topic=Topic(name='event-replacements'), index=0)]
%5|1603427385.201|REQTMOUT|rdkafka#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator/0: Timed out HeartbeatRequest in flight (after 10890ms, timeout #0)
%4|1603427385.201|REQTMOUT|rdkafka#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator/0: Timed out 1 in-flight, 0 retry-queued, 0 out-queue, 0 partially-sent requests
%3|1603427385.202|FAIL|rdkafka#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator: 1 request(s) timed out: disconnect (after 42812865ms in state UP)
2020-10-23 04:31:34,356 New partitions assigned: {Partition(topic=Topic(name='event-replacements'), index=0): 0}

Snuba-outcome-consumer (similar in snuba-consumer)

2020-10-22 16:36:20,879 New partitions assigned: {Partition(topic=Topic(name='outcomes'), index=0): 0}
2020-10-22 16:54:16,946 Error submitting packet, dropping the packet and closing the socket
2020-10-22 16:54:16,986 Completed processing <Batch: 1 message, open for 2.05 seconds>.

In snuba-api occasionally

2020-10-23 07:01:39,535 Error submitting packet, dropping the packet and closing the socket
10.244.1.99 - - [23/Oct/2020:07:01:39 +0000] "POST /query HTTP/1.1" 200 621 "tagstore.__get_tag_keys" "-"
2020-10-23 07:01:40,376 Error submitting packet, dropping the packet and closing the socket
10.244.1.99 - - [23/Oct/2020:07:01:40 +0000] "POST /query HTTP/1.1" 200 541 "search" "-"
2020-10-23 07:01:40,470 Error submitting packet, dropping the packet and closing the socket
10.244.1.99 - - [23/Oct/2020:07:01:40 +0000] "POST /query HTTP/1.1" 200 402 "tagstore.get_groups_user_counts" "-"
2020-10-23 07:01:40,513 Error submitting packet, dropping the packet and closing the socket
10.244.1.99 - - [23/Oct/2020:07:01:40 +0000] "POST /query HTTP/1.1" 200 584 "tagstore.get_group_seen_values_for_environments" "-"
2020-10-23 07:01:40,575 Error submitting packet, dropping the packet and closing the socket
10.244.1.99 - - [23/Oct/2020:07:01:40 +0000] "POST /query HTTP/1.1" 200 498 "tsdb" "-"
10.244.1.240 - - [23/Oct/2020:07:01:40 +0000] "GET / HTTP/1.1" 200 12113 "-" "kube-probe/1.18"
2020-10-23 07:01:40,686 Error submitting packet, dropping the packet and closing the socket
10.244.1.99 - - [23/Oct/2020:07:01:40 +0000] "POST /query HTTP/1.1" 200 402 "tagstore.get_groups_user_counts" "-"
2020-10-23 07:01:40,736 Error submitting packet, dropping the packet and closing the socket
10.244.1.99 - - [23/Oct/2020:07:01:40 +0000] "POST /query HTTP/1.1" 200 584 "tagstore.get_group_seen_values_for_environments" "-"
2020-10-23 07:01:40,797 Error submitting packet, dropping the packet and closing the socke

And in many places

[WARNING] sentry.utils.geo: settings.GEOIP_PATH_MMDB not configured.

Other logs looks fine to me. Does any of logs above sound relevant to my event not coming to events topic? What should I investigate? I am running out of ideas. Thank you!

2

OK I feel kinda lame because I am answering my own questions here and reasons are always trivial :grimacing:

The issue here was that sentry-ingest-consumer had no postgre password configured. I was not seeing this originally since pod immediatelly crashed and was restarted so log stream cleared. Colleague of mine introduced me to notion of “previous logs” where I found this issue.

Hope this helps someone else at least.

1 Like