Sentry stopped displaying events in UI

On-Premise
1

I’ve installed Sentry on premise with just the default options. Everything was working fine until a couple of days ago. Somehow, no event are being added or created in the frontend.

I can see in the docker logs that events are coming in and stored, but I also see Kafka errors. I’ve recreated the sentry-kafka volume but this didn’t help. Any ideas how to get things up and running again? Below an except of the log:

web_1                     | 172.18.0.1 - - [20/Mar/2020:07:28:24 +0000] "POST /api/12/store/     HTTP/1.0" 200 662 "-" "-"
worker_1                  | 07:28:24 [WARNING] sentry.eventstream.kafka.backend: Could not publish message (error: KafkaError{code=_MSG_TIMED_OUT,val=-192,str="Local: Message timed out"}): <cimpl.Message object at 0x7f1b804056d0>
worker_1                  | 07:28:24 [WARNING] sentry.eventstream.kafka.backend: Could not publish message (error: KafkaError{code=_MSG_TIMED_OUT,val=-192,str="Local: Message timed out"}): <cimpl.Message object at 0x7f1b804056d0>
worker_1                  | 07:28:24 [WARNING] sentry.eventstream.kafka.backend: Could not publish message (error: KafkaError{code=_MSG_TIMED_OUT,val=-192,str="Local: Message timed out"}): <cimpl.Message object at 0x7f1b804056d0>
worker_1                  | 07:28:24 [WARNING] sentry.eventstream.kafka.backend: Could not publish message (error: KafkaError{code=_MSG_TIMED_OUT,val=-192,str="Local: Message timed out"}): <cimpl.Message object at 0x7f1b804056d0>
snuba-cleanup_1           | 2020-03-20 07:30:02,723 Dropped 0 partitions on clickhouse
redis_1                   | 1:M 20 Mar 2020 07:30:51.023 * 100 changes in 300 seconds. Saving...
redis_1                   | 1:M 20 Mar 2020 07:30:51.025 * Background saving started by pid 15
redis_1                   | 15:C 20 Mar 2020 07:30:51.062 * DB saved on disk
redis_1                   | 15:C 20 Mar 2020 07:30:51.063 * RDB: 6 MB of memory used by copy-on-write
redis_1                   | 1:M 20 Mar 2020 07:30:51.125 * Background saving terminated with success
worker_1                  | 07:31:22 [INFO] sentry.tasks.update_user_reports: update_user_reports.records_updated (reports_with_event=0 updated_reports=0 reports_to_update=0)
web_1                     | 172.18.0.1 - - [20/Mar/2020:07:31:51 +0000] "POST /api/5/store/ HTTP/1.0" 200 662 "-" "-"
worker_1                  | 07:31:51 [WARNING] sentry.eventstream.kafka.backend: Could not publish message (error: KafkaError{code=_MSG_TIMED_OUT,val=-192,str="Local: Message timed out"}): <cimpl.Message object at 0x7f1b80453450>
web_1                     | 172.18.0.1 - - [20/Mar/2020:07:32:03 +0000] "POST /api/5/store/ HTTP/1.0" 200 662 "-" "-"
web_1                     | 172.18.0.1 - - [20/Mar/2020:07:32:15 +0000] "POST /api/5/store/ HTTP/1.0" 200 662 "-" "-"
worker_1                  | 07:32:15 [WARNING] sentry.eventstream.kafka.backend: Could not publish message (error: KafkaError{code=_MSG_TIMED_OUT,val=-192,str="Local: Message timed out"}): <cimpl.Message object at 0x7f1b804b33d0>
worker_1                  | 07:32:15 [WARNING] sentry.eventstream.kafka.backend: Could not publish message (error: KafkaError{code=_MSG_TIMED_OUT,val=-192,str="Local: Message timed out"}): <cimpl.Message object at 0x7f1b804b33d0>
worker_1                  | 07:32:15 [WARNING] sentry.eventstream.kafka.backend: Could not publish message (error: KafkaError{code=_MSG_TIMED_OUT,val=-192,str="Local: Message timed out"}): <cimpl.Message object at 0x7f1b804b33d0>
worker_1                  | 07:32:15 [WARNING] sentry.eventstream.kafka.backend: Could not publish message (error: KafkaError{code=_MSG_TIMED_OUT,val=-192,str="Local: Message timed out"}): <cimpl.Message object at 0x7f1b804b33d0>
worker_1                  | 07:32:15 [WARNING] sentry.eventstream.kafka.backend: Could not publish message (error: KafkaError{code=_MSG_TIMED_OUT,val=-192,str="Local: Message timed out"}): <cimpl.Message object at 0x7f1b804b33d0>
worker_1                  | 07:32:15 [WARNING] sentry.eventstream.kafka.backend: Could not publish message (error: KafkaError{code=_MSG_TIMED_OUT,val=-192,str="Local: Message timed out"}): <cimpl.Message object at 0x7f1b804b33d0>
worker_1                  | 07:32:15 [WARNING] sentry.eventstream.kafka.backend: Could not publish message (error: KafkaError{code=_MSG_TIMED_OUT,val=-192,str="Local: Message timed out"}): <cimpl.Message object at 0x7f1b804b33d0>
web_1                     | 172.18.0.1 - - [20/Mar/2020:07:32:27 +0000] "POST /api/5/store/ HTTP/1.0" 200 662 "-" "-"
web_1                     | 172.18.0.1 - - [20/Mar/2020:07:32:39 +0000] "POST /api/5/store/ HTTP/1.0" 200 662 "-" "-"
web_1                     | 172.18.0.1 - - [20/Mar/2020:07:32:51 +0000] "POST /api/5/store/ HTTP/1.0" 200 662 "-" "-

Thanks

2

Hi!

This seems to be the issue. Looks like you are having issues with your Kafka service.

3
  1. Delete stack via docker-compose down
  2. Delete Kafka volume’s _data content, except *. properties
  3. Run stack again: docker-compose up -d
1 Like
4

Im having the same issue.
sentry.eventstream.kafka.backend: Could not publish message (error: KafkaError{code=_MSG_TIMED_OUT,val=-192,str=“Local: Message timed out”

Any help ?

Thanks

5

Tried this, but to no avail. Did anyone have any luck with this yet?

6

@najam I recommend posting a new topic with specific details about your setup as if the solution did not work for you, you probably have a different issue.

7

Will do, thanks!