Sentry's RabbitMQ architecture?

ParthKolekar · January 26, 2020, 6:47am

Just curious how you have (were?) been running the RabbitMQ cluster.

I looked at the scale and code of Sentry, and I could not identify how Sentry achieved high-availability with RabbitMQ. Maybe you could make a blog or something describing the layout? I could not find any good resource online for a High Availability setup for RabbitMQ at scale, so I think it would be beneficial to the community as a proven design pattern.

No pressure in case this is a business secret.

matt · January 26, 2020, 7:33am

Nothing special, in production we just use federated queues.

https://www.rabbitmq.com/federation.html

ParthKolekar · January 26, 2020, 7:50am

Celery supports initializing a fallback broker inside the BROKER_URL if you specify the broker URL as a list of strings.

Sentry’s internal monitoring breaks with the lists, however, and we need to initialize it with a single broker url string.

github.com

getsentry/sentry/blob/releases/9.1.x/src/sentry/monitoring/queues.py#L82


        with self.get_conn() as conn:
            with conn.channel() as channel:
                return self._get_size_from_channel(channel, queue)


    def purge_queue(self, queue):
        with self.get_conn() as conn:
            with conn.channel() as channel:
                return channel.queue_purge(queue)




def get_backend_for_broker(broker_url):
    if broker_url is None:
        raise KeyError
    return backends[urlparse(broker_url).scheme](broker_url)




def get_queue_by_name(name):
    "Lookup a celery Queue object by it's name"
    for queue in settings.CELERY_QUEUES:
        if queue.name == name:
            return queue

Thankfully, celery also supports multiple brokers in a single string as long as they’re separated by a semi-colon.

We give the broker url as "amqp://user:password@host1/vhost;amqp://user:password@host2/vhost".

This causes Celery to be happy and start up. It causes Sentry’s “Monitoring” to think that we have

scheme: amqp
user: user
password: password
host: host1
url: vhost;amqp://user:password@host2/vhost

This causes us pain whenever we open the administrative UI, but generally the application works well.

I’m curious to know how the RabbitMQ was setup so that Sentry can continue to discover the nodes even when the broker crashes.

Do you have an additional load-balancer / high-availability tooling that keeps the singular broker url pointing to a single broker, active at all times?

matt · January 26, 2020, 8:02am

Yeah, we don’t use anything built into celery for this. We do routing through haproxy or envoy.

From application servers, they all have a local haproxy or envoy which is used for routing outbound connections. In this case, we just round robin between the brokers from there.

Ultimately our broker url is something over 127.0.0.1.

ParthKolekar · January 26, 2020, 11:54am

Ah gotcha… Yeah that seems about right. Thanks.

Topic		Replies	Views
Sentry.conf.py for RabbitMQ On-Premise	2	2456	November 29, 2016
On using Amazon MQ to back Sentry's workers	1	1298	November 26, 2019
Scaling sentry across multiple regions On-Premise	3	1580	June 11, 2018
Sentry worker performance On-Premise	1	2455	September 7, 2018
Sentry worker stop working (rabbitmq connection issue?)	2	3504	March 25, 2021

Sentry's RabbitMQ architecture?

Related Topics