kafka log retention problem

["Ncir, Kawther (external - Service)" <k....@sap.com>]

Hi,
I am working on a project using kafka broker and I face a problem with kafka log retention ,
We need to delete logs every 24h  but  after the retention time the broker was stopped and make an issue like this :
"ERROR Uncaught exception in scheduled task 'kafka-log-retention' (kafka.utils.KafkaScheduler)
org.apache.kafka.common.errors.KafkaStorageException: Error while deleting segme
nts for test-1 in dir C:\tmp\log Caused by: java.nio.file.FileSystemException: C:\tmp\log\test-1\0000000000000000
0000.timeindex -> C:\tmp\log\test-1\00000000000000000000.timeindex.deleted: The process cannot access the file because it is being used by another process."

Could you please support me to find a solution.

Regards,
kawther

Re: kafka log retention problem

[Ricardo Ferreira <ri...@riferrei.com>]

It sounds like you are trying to forcibly delete the files that build up 
the segments used by the partitions. If that is the case then I would 
recommend not using external tools and leave to Kafka manage its 
filesystem. If you set the retention policy (either by size or time) in 
your topics the log cleaner threads will take care of purging any unused 
space from the segments automatically. No need to have a external 
process that clean those out.

In order to recover from this; kill any processes that might be acting 
upon those files and then restart your Kafka brokers once again. Only 
the broker JVM should have file handles to the segments to avoid any 
lock problems during file write/read.

Thanks,

-- Ricardo

On 6/15/20 4:50 AM, Ncir, Kawther (external - Service) wrote:
> Hi,
> I am working on a project using kafka broker and I face a problem with kafka log retention ,
> We need to delete logs every 24h  but  after the retention time the broker was stopped and make an issue like this :
> "ERROR Uncaught exception in scheduled task 'kafka-log-retention' (kafka.utils.KafkaScheduler)
> org.apache.kafka.common.errors.KafkaStorageException: Error while deleting segme
> nts for test-1 in dir C:\tmp\log Caused by: java.nio.file.FileSystemException: C:\tmp\log\test-1\0000000000000000
> 0000.timeindex -> C:\tmp\log\test-1\00000000000000000000.timeindex.deleted: The process cannot access the file because it is being used by another process."
>
> Could you please support me to find a solution.
>
> Regards,
> kawther
>
>