About the relationship between nodes

On-Premise
1

Could you please review in detail whether I understand the relationship between nodes?

I think you are using a total of two cache systems and one message queue.

  1. The first cache is memcached, which you use as a small cache. I think the web container uses memcached for the parts that need to share memory as a cache.

  2. The second cache is redis, which you use as the main cache.
    In 9.1.2 you used redis as a query cache as well as a message queue for reads and writes. However, since version 10, kafka is used as a message queue instead, and in redis, it is used for query caching for read operations, and in redis, it is only used for querying through snuba after creating a json event in case of a cache miss. These are the legacy paths that were left as they were from 9.1.2, right?

In other words, I think that all the work of writing events was transferred to kafka through relay. Other features remain in redis. Is that right?

Then, there will be only one image “worker” involved in the operation of redis as before, right?

If these assumptions are correct, can the “worker” be in charge of making a json event for the parts that redis could not cache, making a request to snuba or requesting a query to postgre? (Both operations correspond to read queries, of course, if you write to postgre as an exception, I think you will still use redis-worker as before)

  1. The next thing I’m curious about is the association of images involved in Message Queuing.
    You are upstreaming “relay” rather than “web” for “issue” related write processing in nginx.

Then, for kafka, the first producer is “relay”, the message queue is “kafka”, and the consumer is “ingest-consumer”. Is it correct?

And the event is processed again as “ingest-consumer” becomes producer, “kafka” becomes message queue, and consumer becomes “post-process-forwarder” (This part is probably not correct. It seems like it will go straight from the ingest-consumer to the post-process-forwarder.)

And once again, after “post-process-forwarder” becomes producer (probably the product is a json event at this time), this time the consumer becomes “snuba-consumer” created by topic. Are these courses all right?

In other words, I think you handle “relay”=>“kafka”=>“ingest_consumer”=> (“kafka” )=>“post_process_forwarder”=>“kafka”=>“snuba_consumer”. is this right?

I want to accurately understand the role of each image and know the relationship between them clearly and in detail. Thank you for always giving kind and detailed answers to me with many questions!

2

Conceptually, I currently classify the types of queries in sentry into 4 types.
Read, write to “issue”, and CRUD (get, post, update, delete) for anything other than “issue”
I think only the writing for “issue” is handled by kafka, and everything else is handled through redis.
Also, I think that writing to “issue” is handled only by the clickhouse, but queries for other than “issue” are handled using postgre or clickhouse, or both.
The reason I think this is because I think you do query caching through redis for read operations.
Am i right?

3

Hi, if you’d like to learn more about how Sentry processes events, I recommend reading/listening to the following:

4

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.