I have a sentry installation with 2k events per minute. I have a dedicated machine for postgreSQL and one for the rest of the sentry(redis, web and workers as systemd services).
When we had 1.5k events everything was fine. We recently hit 2k and we encountered “preprocess.failed.empty”. Increasing the worker count increased the performance but it was not enough.Now I want to add two more servers with worker role.
I haven’t seen any documentation on this and I’m just guessing here. Since workers are stateless can I add two more machines , put redis and worker service on them point them to postgreSQL. I have an f5 load balancer and I plan distribute packages to workers based on health checks.
You can add as many worker hosts as you want, but you won’t want to carry along Redis with them. Assuming you’re using the default configuration here, Redis is your queue to do work. So your Redis needs to be shared. Other than that, you can easily split up all web and worker processes for more scale.
If the redis needs to be shared for queue, there is no need for a load balancer then.
Since workers are the ones consuming the queue, which part of sentry listens for incoming requests and pushes them to queue ?
EDIT: I got the cluster working by only changing redis URL in default config. I removed crontabs from workers and disabled the beat service(I’m using systemd). Sentry seems to be working fine.
I’m seeing the following error in the systemd logs on worker nodes roughly once for every 20 minutes.
process_pending.fail (error=UnableToAcquireLock(“Unable to acquire <Lock: ‘buffer:process_pending’> due to error: Could not set key: u’l:buffer:process_pending’”,))