Standard docker setup. After upgrade events were accepted for a few hours and then stopped. Started again after server restart and then stooped again. Server restart does not help anymore. I have kafka related errors in consumer logs: + '[' c = - ']' + snuba consumer --help + set -- snuba consume…

@fabriciols - interesting. That means redis is taking its time to get up and running and it takes longer than the sentry instance. This is definitely not ideal but can you try adding something like sleep 10 before that line 280 and see if it helps?

I face the same issue, after upgrading from 10 to 20.7.2 after a few hours, it stopped processing events. Then I went to upgrade to 20.8*-dev (master) to resolve issues with relay authentication. The relay errors are gone. I’ve reset my installation by calling: cd /var/www/equadrat/sentry.e-2.at…

If this is not a networking issue (and it doesn’t seem like it) then I’d suspect Kafka being overloaded or throttled by something and not being able to accept new consumer connections. Is your volume high? Or your NFS volume is slow maybe?

Sentry 9 & Sentry 10 were running with same load fine. Now I can see no errors/warnings on Kafka, but this time: worker_1 | 14:16:51 [WARNING] celery.worker.consumer.consumer: consumer: Connection to broker lost. Trying to re-establish the connection... Then I restarted kafka,…

Should we post this in Sentry github issues?

I’m not sure yet what to report? Kafka does not print any warning, worker & relay do not reconnect automatically, but why? Is there metrics on Kafka that might indicate some overload?

I honestly don’t know much about how to run Kafka @e2_robert so cannot really comment here until I learn more. Maybe @matt would have some ideas.

I deleted clickhouse-data together with kafka-data and zookeeper-data. I lost all events but it looks like new ones are arriving for now…

It worked for 12 hours and stopped again :sob:

I had the same result. But it stopped after 8 hours.

Sentry stops processing events after upgrade 10.0 => 20.8.0.dev0ba2aa70

On-Premise

hheexx August 17, 2020, 12:12pm 19

I understand.
I can provide access to my server to Sentry engineers if needed.

Topic		Replies	Views
Sentry no more catch errors On-Premise	11	10748	June 5, 2021
Sentry did not access new events On-Premise	8	5304	September 3, 2021
Kafka Error after Upgrading On-Premise	28	11855	February 9, 2021
Connection to Kafka failed When installing On-Premise	18	8184	May 23, 2020
No new events after update On-Premise	9	3588	March 5, 2021

Sentry stops processing events after upgrade 10.0 => 20.8.0.dev0ba2aa70

Related topics