Kafka fails to start up after truncating logs, preallocate log size

[amit1]

In order to clean up logs, i truncated some logs in kafka .
root@871d7856246d:/var/lib/kafka/data/events-0# pwd
/var/lib/kafka/data/events-0

-rw-r–r-- 1 root root 0 Apr 5 13:42 00000000000018096585.log
-rw-r–r-- 1 root root 0 Apr 5 13:42 00000000000017849768.log
-rw-r–r-- 1 root root 0 Apr 5 13:42 00000000000017574826.log
-rw-r–r-- 1 root root 0 Apr 5 13:42 00000000000017307174.log
-rw-r–r-- 1 root root 10 Apr 5 10:27 00000000000018352981.snapshot
-rw-r–r-- 1 root root 1486168 Apr 5 10:27 00000000000018096585.index

After doing that . on startup i see the following errors . Does kafka need a preallocated log size on startup . Can this be fixed somehow @BYK

**ingest-consumer_1                           | 14:26:48 [INFO] batching-kafka-consumer: New partitions assigned: [TopicPartition{topic=ingest-attachments,partition=0,offset=-1001,error=None}, TopicPartition{topic=ingest-events,partition=0,offset=-1001,error=None}, TopicPartition{topic=ingest-transactions,partition=0,offset=-1001,error=None}]**
**kafka_1                                     | [2021-04-05 14:26:48,226] ERROR [ReplicaManager broker=1001] Error processing fetch with max size 1048576 from consumer on partition ingest-transactions-0: (fetchOffset=11632139, logStartOffset=-1, maxBytes=1048576, currentLeaderEpoch=Optional.empty) (kafka.server.ReplicaManager)**
**kafka_1                                     | org.apache.kafka.common.errors.CorruptRecordException: Found record size 0 smaller than minimum record overhead (14) in file /var/lib/kafka/data/ingest-transactions-0/00000000000011627018.log.**
**ingest-consumer_1                           | * Unknown config option found: 'slack.legacy-app'**

[BYK]

Sorry, I honestly have no idea what this is. Were you able to fix this on your own?

[amit1]

@BYK , yes i was able to fix this on my own.

Interestingly kafka log files should be maintained through KAFKA_LOG_RETENTION_HOURS: “24”, and any other log retention values, such as log.cleanup.policy and log.retention.bytes and maybe others mentioned in

Kafka Log Retention and Cleanup Policies

Are any of these attributes supported on the docker-compose.yml for sentry?

I solved this by removing my
kafka, kafka-log , zookeeper and zookeeper-log volumes and doing a .install.sh. That fixed the problem

[BYK]

Yup, the logic is prepending KAFKA_ to them, make them all caps and replace . with _. So setting KAFKA_LOG_CLEANUP_POLICY and KAFKA_LOG_RETENTION_BYTES should work.

If you think we should set these to certain values for smooth operation, feel free to submit a PR to the onpremise repo. If not you can create a docker-compose.override.yml file and set these there which docker-compose would automatically pick up and merge with the main config file.