Perfomance metric alarm related to Snuba-events-subscriptions-consumers

[seungjinlee]

Snuba-events-subscriptions-consumers of onpremise sentry built internally does not work properly. For this reason, the alarm according to the metric change in the performance tab does not work properly. I’ve already asked a couple of questions about this issue before. but failed to normalize

Kafka offset issue - snuba-subscription-consumer-events

Thank you for providing a great project. Currently, I am using this project to scale out. Some kafka topic events such as events, outcomes, and ingest-events are divided into 10 partitions, and each partition works well by snuba consumers and workers. But there is one thing that doesn’t work. In the case of snuba-subscription-consumer-events, rebalancing continues with an offset error. Caught OffsetOutOfRange('KafkaError{code=OFFSET_OUT_OF_RANGE,val=1,str="Broker: Offset out of range"}'), shutt…

Snuba-events-subscriptions-consumers is not working

Currently I am running the event topic divided into 10 partitions. In all other cases, there is no problem, only snuba-events-subscriptions-consumers are constantly rebalancing or the current offset remains at zero. When I first set it up, I didn’t have this problem. At the time, snuba-events-subscriptions-consumers also worked well. However, once the problem occurs, initializing the offset with “–reset-offsets --to-latest --execute” does not solve the problem. You need a solid solution.

Here is a summary of my experiences with kafka offset kinks:

1. When the inflow is much higher than the throughput
2. In the post-processing process, when clickhouse does not operate normally

What about Snuba-events-subscriptions-consumers? I wonder how this consumer works.

@BYK previously mentioned two people @markstory @fpacifici regarding this issue. If you know anything, please let me know.

[fpacifici]

Will move the conversation here:
re: how the snuba-events-subscriptions-consumer works.
IT consumes two topics. The main events topic (all events (errors/transactions) produced by sentry) is one and the other is the commit_log topic. The snuba consumer (that writes on clickhouse) writes a mark on the commit log telling the offset it committed for each partition.
The snuba events subscriptions consumer gets an offset from the commit log, then it consumes the events topic up to that offsset and, for every time tick (let’s say every minute to simplify) it executes all the clickhouse queries to provide metric alerts data to sentry. Then it pauses until the main consumer has made some progress.

There are multiple reasons why the subscriptions consumer may be rebalacing continuously depending how many replicas of that consumer you are running. I believe you have 10 partitions.

• Does one replica crash for some reason? That would immediately cause a rebalance. We saw OOM issues at times.
• Is one (or multiple) of the replicas so overwhelmed that it does not manage to poll from Kafka within the timeout, then the broker may be excluding the consumer and trigger a rebalance.

To nail down the cause we would need more information than those provided in this issue. Specifically:

• How many replicas of the consumer are you running ?
• how often does the rebalancing happens? The consumer log would show partition assignments. Would you mind providing logs on a longer period of time? What was provided in the other issue only contains an exception which is hard to contextualize.
• Also any chance you could run the consumer in debug mode -log-level DEBUG And provide the full log from the start to the crash ?
• Do you have a lot of metric alerts configured ? Maybe the bottleneck is running those queries against clickhouse. We recently made a change to even out the load on the DB which should help in those cases. Here and here

Hope this helps.
Filippo

[seungjinlee]

Thanks so much for the detailed explanation! I’m sure it will be of great help to us in resolving this issue. We are now configuring a separate RC environment for various tests including related issues. It seems that the process of log collection and testing you mentioned will be done after the RC environment is built. I will build an RC environment next week to check it out and leave a reply again. thank you @fpacifici