Limit Kafka Worker and Cloud Server Worker Resources and Parallel Processes

I’m using the Open Source Self Hosted Sentry instance. It runs a Ubuntu 20 Server and I used the Docker installation. I see a lot of worker for Apache Kafka and Clickhouse Server projects, I’m searching for a config to avoid that to much of this worker are running in parallel to match my servers system resources. But could not yet find where this worker are configured or how to limit the memory for them.

Does anybody have an hint where this kind of things can be configured.

You may try using the deploy key to define the limits and then use compatibility mode or docker stack deploy command to make those values effective.

Alternatively, you may consider using Kubernetes through this community-maintained helm chart.

Thank you for the response, we will give it a try. If understand correctly the workers / threads are calculated by available resources and so don’t have a fixed value configured?

Not necessarily. They have some defaults and they just keep consuming more resources as they can get them :slight_smile:

You may wanna check some of these to get ideas on scaling up btw: https://develop.sentry.dev/self-hosted/troubleshooting/