I want to scale out my consumption in kafka. I know snuba is in the consumer role. I know that Kafka has a partition for each topic, and it can consume as much as the number of consumers when partition >= consumers. So, after separating all the Snuba aggregates, I launched them on multiple instances. And I increased Kafka’s KAFKA_OFFSETS_TOPIC_NUM_PARTITIONS according to the number of snuba instances and then consumed it. However, only one of the snubas was consumed, and the other snubas only repeated “New partitions assigned: {}” and “Partitions revoked: ” and could not consume them. How can I increase consumption? If this is possible, I think event processing will be dramatically faster when I run install.sh.
Hi,
thanks for the question.
What is the specific problem you are trying to solve by scaling out consumption? Specifically are the consumers lagging? Asking this because depending on where your bottleneck is there are different roads to go through.
A few steps though could help inform such decision.
-
Sentry and Snuba partition events semantically and not in a round robin way (all the events for a single project are always produced to the same partition, so that we snuba can have read your write consistency). That also means you cannot scale out consumption of an individual project by adding consumers. If you add consumers the partition may be reassigned but it will always be entirely assigned to one and only one consumer.
-
We built a parallel snuba consumer to cope with that that uses python multi processing and python 3.8 shared memory to parallelize some heavy work in the consumer. This is controlled by these parameters that have to be set on the consumer: “processes” “input-block-size”, “output-block-size”. processes is the number of parallel processes to start, the other two are the size of the shared memory blocks to create. Tuning that depends on the number of processes, your memory and your event size, but very roughly you can do something above processes * event_size_bytes * events in batch (1000).
-
If you have many projects you can spread across partitions instead, the first thing to do is to increase the partitions of the existing events topic. If the topic is already created you need to manually add partitions. Something like
bin/kafka-topics.sh --zookeeper YOUR_ZOOKEEPER --alter --topic events --partitions XXXdepending on the kafka version. Though this is not it. You also need to updateTOPIC_PARTITION_COUNTSwith your number of partitions in the snuba config as this is required for subscriptions to work consistently.
There is a more complex discussion on how to scale out sentry and snuba, but that depends on how your data is partitioned to begin with.
Happy to provide more details
Filippo
2 Likes
Hello thank you for your detailed answer.
I want to multiplex single points of failure and bottlenecks to ensure HA.
So, I want to cluster kafka and cluster both producers and consumers corresponding to it.
So I want to cluster snuba too.
As you said, even if I create multiple partitions in kafka and create multiple snubas accordingly, I have confirmed that consumption does not occur in parallel.
As soon as one snuba made a connection, I noticed that the other snubas disconnected.
As you answered, even if I divide multiple partitions, it made sense that if the sentry side generates an event in only one partition, it would be useless to increase the snuba consumer.
So, fpacifici suggested two solutions, but I have a question.
First, the first solution is judged by increasing the process within snuba.
I understood that even if events are generated on only one partition per topic, if snuba’s process grows, it can handle write speeds in parallel. Is that right?
If so, can you give me information about the environment variables that can be used when launching the snuba container? I don’t know where the “processes” “input-block-size”, “output-block-size” are.
Also, even if I can increase snuba’s process on one node, I think running more than one snuba node guarantees HA. (Of course, only one snuba is assigned and the other snuba can be in hotstandby state)
So, I would like to know in more detail the solution that you presented second. Even if the contents are partitioned with an environment variable, parallel consumption is not possible, but it seems that it is possible with a direct partitioning method. Is it correct? Could you please explain in more detail. Thank you.
And what are the projects you’re talking about?
Are you referring to the project registered in sentry?
In sentry, you said that only one partition is assigned. Can I divide a partition if I register multiple projects?
https://github.com/getsentry/snuba/blob/6d86bd98ad2207f003db8e1d7cdaa86bc1f81ee6/snuba/settings.py
In the snuba setting, there is information about TOPIC_PARTITION_COUNTS, but there is no environment variable, so there is no way to access it except to fix the image directly? Is that right?
re: I want to multiplex single points of failure and bottlenecks to ensure HA.
Ok, this is a critical point. I originally thought your goal was to increase throughput by scaling out the consumers. If the goal is HA, the answer would be different.
Making snuba consumption resilient:
- The number of partition will not be very important.
- Kafka. You need to replicate Kafka on the number of replicas (I would advise an odd number) you like and increase the replication factor of the topic accordingly so that messages are appended and persisted by all nodes. Now you have durability of your messages in Kafka and your system is resilient to one kafka broker going away. This is widely documented on Kafka docs, I can give you pointers if you need.
- Snuba consumer. If you want to ensure the consumer is always up. Deploy as many as you want. Even if the partition is only one and consumption does not happen in parallel, when a consumer stops Kafka will rebalance the partitions and if there is another consumer waiting it would get the partition assigned and pick up from the last committed offset (we commit at every clickhouse write after writing to ensure durability).
- Clickhouse. You will need to build a clickhouse with Replicated tables. Unfortunately Snuba does not support this out of the box yet. You would have to do it manually following Clickhouse documentation.https://clickhouse.tech/docs/en/engines/table-engines/mergetree-family/replication/. Then you will have to handle your DDL change manually. I can fill in with more details when we get there.
- Snuba API nodes. You may want to have multiple Snuba API nodes that Query clickhouse in cases one crashes. This depends on your infrastructure, nodes are stateless you can start as many as you want for as long as you load balance the traffic from sentry to Snuba. Sentry does not provide anything out of the box, an example for doing this is using Envoy.
re: First, the first solution is judged by increasing the process within snuba.
This helps you with throughput on a single partition but does not do anything at all for HA since the node is still one. If the goal is HA, I would ignore this solution. Anyway, for your knowledge, those three parameters are command line parameters for the consumer script. How are you running snuba? Is it from the onpremise repo or from source? The way to customize those value depends on this.
re: And what are the projects you’re talking about? Are you referring to the project registered in sentry?
Yes those are the projects I am referring to. The partition picked for an event is some hash of the project id. So creating multiple projects would send events to multiple partitions. Again though, if your goal is HA and you do not have throughput problems you can still deal with one partition only. The way to change the settings depends, as well, on how you run Snuba (onpremise repo vs source and are you running the scripts as docker containers?), you should not need to patch the image but the way to solve the problem depends on the way you run snuba.
Hope this helps
Filippo
This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.