Events not displaying

On-Premise
1

Recently my docker based onpremise has stopped displaying events.

I can send events and get a response back, however no errors are displayed within Sentry…

Looking in the logs I see this:

snuba-consumer_1 | %3|1596588187.987|FAIL|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.22.0.10:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
snuba-consumer_1 | %3|1596588188.025|FAIL|rdkafka#consumer-2| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.22.0.10:9092 failed: Connection refused (after 14ms in state CONNECT, 1 identical error(s) suppressed)

Any ideas how to resolve this issue? I have re-run ./install.sh to update to the latest, and also flushed redis with no success.

2

Looks like your Snuba consumer cannot connect to Kafka. Either Kafka is not up and running or you have networking issues.

3

Thanks,

I think I will need to start clean again.

Is there a clean way to export the DSN’s and other meta data and re-import into a clean install?

I’m not really worried about historical data, but preserving the DSNs and some settings would be critical.

4

If you just keep the sentry-postgres volume and remove the others, it should keep this information.

Even better, I don’t think deleting anything in Clickhouse would help so instead of starting from scratch, I’d first try removing the following volumes:

  • sentry-kafka
  • sentry-zookeeper
  • sentry-zookeeper-log
  • sentry-kafka-log

You may loose some data but it should be minimal.

5

Hi @BYK

Thanks for this, unfortunately removing those volumes still doesn’t resolve the issue.

6

@turbo124 then I’d say this is very likely a network configuration issue. Running docker-compose down and then docker-compose up -d again should remove the network and then recreate it which may help.

7

I am running into the same issue. any further resolution on this. This happens when i rerun the .install.sh

  • exec gosu snuba snuba bootstrap --force
    %3|1597170250.847|FAIL|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.18.0.5:9092 failed: Connection refused (after 3ms in state CONNECT)
    %3|1597170251.843|FAIL|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.18.0.5:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
    2020-08-11 18:24:11,843 Connection to Kafka failed (attempt 0)
8

This is the same error I am seeing.

I’ve run the update script multiple times, deleted volumes nothing seems to help.

9

Same problem here (thread Sentry stops processing events after upgrade 10.0 => 20.8.0.dev0ba2aa70) . I tried deleting kafka and zookeeper and it didn’t help. It’s not networking issuse ( I tried adding container to network and I can ping kafka no problem).

Any recent commit that could cause this?

10

@hheexx @turbo124 have you also tried using the recently released 20.8.0 version?

We are investigating this issue but since we are not experiencing this ourselves, it is mostly a guessing with trial & error.

11

@BYK

Currently using Sentry 20.9.0.dev 026dc3ea

Seeing this only under heavy load. Light load it works perfectly, I am sure I can recreate this on any setup. If you want to send me a DSN to send test events to :slight_smile:

12

I stayed on 20.8.0.17e488c
Problem persists.

I can also confirm there is heavy load probably happening on my server from time to time…

13

If you are seeing issues under heavy load, I’d recommend investigating and implementing your own solutions to your specific setup and needs as I don’t think we can come up with a one-size-fits-all scalability solution, especially for on-premise.

If there are some obvious fixes you see, I’m willing to incorporate them into the default setup but right now our focus is making on-premise easy to setup and usable by most people instead of sharpening it for perfection under heavy loads.

14

Could anyone find a solution for the problem? Facing this after an update. Eventually I installed Sentry on a fresh Ubuntu 20.04 from the scratch. Same problem.