Restrict Kafka disk usage

On-Premise
5

Hi again, now our Kafka went crazy consuming nearly 100G.

What is the environment variables that you’ve configured for your instance? I’ve used 50G for retention size, 1G for segments and a bit higher intervals now:

    KAFKA_LOG_RETENTION_HOURS: 24
    KAFKA_LOG_RETENTION_BYTES: 53687091200
    KAFKA_LOG_SEGMENT_BYTES: 1073741824
    KAFKA_LOG_RETENTION_CHECK_INTERVAL_MS: 300000
    KAFKA_LOG_SEGMENT_DELETE_DELAY_MS: 60000

In the logs I could see the LogCleaner to be started:

kafka_1                    | [2020-07-27 09:06:10,351] INFO Starting the log cleaner (kafka.log.LogCleaner)
kafka_1                    | [2020-07-27 09:06:14,414] INFO [kafka-log-cleaner-thread-0]: Starting (kafka.log.LogCleaner)

After some minutes, disk usage dropped from 98G to 19G. I hope to see no further issues with e.g.: lost messages.