Restrict Kafka disk usage

Migrated from v9 to v10 recently. Now, the Docker volume sentry-kafka was consuming 107G within just 4 days of operation. Is there a possibility to restrict it? This might be related to How to disable Kafka logs in on premise sentry

I’ve found this great article about limiting Kafka: http://iv-m.github.io/articles/kafka-limit-disk-space/
It suggests to add:

    "log.retention.hours": 24,
    "log.retention.bytes": 1073741824,
    "log.segment.bytes": 107374182,
    "log.retention.check.interval.ms": 20000,
    "log.segment.delete.delay.ms": 1000

However, where would I apply this to? I’ve tried sentry/sentry.conf.py, but its Kafka options are only for the producer end.
What options would make sense in terms of Sentry? Does it only temporarily store incoming requests before they are processed or is Kafka also used as permanent storage?
Even if I’d loose some events, I’m in favor of keeping this volume small.

2 Likes

You should be able to add these values to the kafka container in your docker-compose.yml file as environment variables. Just prefix them with KAFKA_ , use _ instead of . and all uppercase. See here: https://docs.confluent.io/current/installation/docker/config-reference.html#confluent-kafka-configuration

For your last questions I don’t have answers but I am interested in the answers too.