Question about kafka log retention

[Xiaochi Weng <wx...@gmail.com>]

Hello,

I am currently using kafka 3.1.0 with java 1.8. I have set kafka log
retention policy in the server.properties like this:

log.retention.hours=6


log.retention.bytes=5368709120

log.segment.bytes=1073741824

log.retention.check.interval.ms=300000

log.cleanup.policy=delete

However, it seems that Kafka is still keeping the log opened and keeps
consuming the spaces. I have used the "df -h" and "du" commands and found
that the number of used spaces were different. When I checked with lsof |
grep "deleted", I found that kafka is still keeping deleted files.
Currently, it seems that the only way to resolve this issue is to kill the
kafka process and restart a new one.

Could you tell me whether there is an alternative way to resolve this issue?

Thanks,

Xiaochi Weng

Re: Question about kafka log retention

[Franz van Betteraey <fr...@van-betteraey.de>]

Hello Xiaochi ,

I am not sure if I have understood the problem correctly but beware the 
fact that only old log segments and not the current log segment are 
taken into account for deletions. So if you want the data to be deleted 
in a timely manner, you also need to configure a tighter interval for 
rotating the log files.

See 'log.roll.hours' (broker config):

https://kafka.apache.org/documentation.html#brokerconfigs_log.roll.hours

Kind regards
   Franz

Am 02.02.2022 um 20:37 schrieb Xiaochi Weng:
> Hello,
>
> I am currently using kafka 3.1.0 with java 1.8. I have set kafka log
> retention policy in the server.properties like this:
>
> log.retention.hours=6
>
>
> log.retention.bytes=5368709120
>
> log.segment.bytes=1073741824
>
> log.retention.check.interval.ms=300000
>
> log.cleanup.policy=delete
>
> However, it seems that Kafka is still keeping the log opened and keeps
> consuming the spaces. I have used the "df -h" and "du" commands and found
> that the number of used spaces were different. When I checked with lsof |
> grep "deleted", I found that kafka is still keeping deleted files.
> Currently, it seems that the only way to resolve this issue is to kill the
> kafka process and restart a new one.
>
> Could you tell me whether there is an alternative way to resolve this issue?
>
> Thanks,
>
> Xiaochi Weng
>