Events not displaying

On-Premise
#1

Recently my docker based onpremise has stopped displaying events.

I can send events and get a response back, however no errors are displayed within Sentry…

Looking in the logs I see this:

snuba-consumer_1 | %3|1596588187.987|FAIL|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.22.0.10:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
snuba-consumer_1 | %3|1596588188.025|FAIL|rdkafka#consumer-2| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.22.0.10:9092 failed: Connection refused (after 14ms in state CONNECT, 1 identical error(s) suppressed)

Any ideas how to resolve this issue? I have re-run ./install.sh to update to the latest, and also flushed redis with no success.

#2

Looks like your Snuba consumer cannot connect to Kafka. Either Kafka is not up and running or you have networking issues.

#3

Thanks,

I think I will need to start clean again.

Is there a clean way to export the DSN’s and other meta data and re-import into a clean install?

I’m not really worried about historical data, but preserving the DSNs and some settings would be critical.

#4

If you just keep the sentry-postgres volume and remove the others, it should keep this information.

Even better, I don’t think deleting anything in Clickhouse would help so instead of starting from scratch, I’d first try removing the following volumes:

  • sentry-kafka
  • sentry-zookeeper
  • sentry-zookeeper-log
  • sentry-kafka-log

You may loose some data but it should be minimal.

Sentry stops processing events after upgrade 10.0 => 20.8.0.dev0ba2aa70
#5

Hi @BYK

Thanks for this, unfortunately removing those volumes still doesn’t resolve the issue.

#6

@turbo124 then I’d say this is very likely a network configuration issue. Running docker-compose down and then docker-compose up -d again should remove the network and then recreate it which may help.

#7

I am running into the same issue. any further resolution on this. This happens when i rerun the .install.sh

  • exec gosu snuba snuba bootstrap --force
    %3|1597170250.847|FAIL|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.18.0.5:9092 failed: Connection refused (after 3ms in state CONNECT)
    %3|1597170251.843|FAIL|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.18.0.5:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
    2020-08-11 18:24:11,843 Connection to Kafka failed (attempt 0)
#8

This is the same error I am seeing.

I’ve run the update script multiple times, deleted volumes nothing seems to help.

#9

Same problem here (thread Sentry stops processing events after upgrade 10.0 => 20.8.0.dev0ba2aa70) . I tried deleting kafka and zookeeper and it didn’t help. It’s not networking issuse ( I tried adding container to network and I can ping kafka no problem).

Any recent commit that could cause this?

#10

@hheexx @turbo124 have you also tried using the recently released 20.8.0 version?

We are investigating this issue but since we are not experiencing this ourselves, it is mostly a guessing with trial & error.

#11

@BYK

Currently using Sentry 20.9.0.dev 026dc3ea

Seeing this only under heavy load. Light load it works perfectly, I am sure I can recreate this on any setup. If you want to send me a DSN to send test events to :slight_smile:

#12

I stayed on 20.8.0.17e488c
Problem persists.

I can also confirm there is heavy load probably happening on my server from time to time…

#13

If you are seeing issues under heavy load, I’d recommend investigating and implementing your own solutions to your specific setup and needs as I don’t think we can come up with a one-size-fits-all scalability solution, especially for on-premise.

If there are some obvious fixes you see, I’m willing to incorporate them into the default setup but right now our focus is making on-premise easy to setup and usable by most people instead of sharpening it for perfection under heavy loads.

#14

Could anyone find a solution for the problem? Facing this after an update. Eventually I installed Sentry on a fresh Ubuntu 20.04 from the scratch. Same problem.