Log Retention Period of Kafka Messages

["Muruganandham, Ashokkumar" <as...@fluke.com>]

Hi Team,

Form the documentation I could see that log retention period is relative to the partition segments. So unless the segment is closed (1Gb/1 Week) the message will never be deleted.

Consider I have a very critical use case where I will need to delete the data every day and my segment reaches 1GB after three days. So according the above statement, then there is no way I could configure Kafka to delete the data in one day.

Is my understanding correct ?

Regards,
AK
________________________________

Please be advised that this email may contain confidential information. If you are not the intended recipient, please notify us by email by replying to the sender and delete this message. The sender disclaims that the content of this email constitutes an offer to enter into, or the acceptance of, any agreement; provided that the foregoing does not invalidate the binding effect of any digital or other electronic reproduction of a manual signature that is included in any attachment.

Re: Log Retention Period of Kafka Messages

[James Cheng <wu...@gmail.com>]

> On Mar 12, 2018, at 10:13 AM, Kyle Tinker <kt...@workforcesoftware.com> wrote:
> 
> You have a couple options:
> 1) You can adjust log.segment.bytes to make the segments smaller so that individual segments can be trimmed
> 2) You can set log.roll.hours, which will roll to a new log segment even if the size hasn't been reached
> Note: #1 and #2 also have per-topic setting controls
> 

About #2, it's possible that segments will only roll when a message has been received, after log.roll.hours (or log.roll.ms, etc). That is, if the time has passed but no new message has been received, the log roll might not happen.

You should do some testing to double check.

-James


> https://kafka.apache.org/08/documentation/#brokerconfigs
> 
> Thanks,
> --
> 
> 
> KYLE TINKER
> Lead Architect/Team Lead | WorkForce Software 
> 
> T: +1 734 742 2616 | E: ktinker@workforcesoftware.com
> 
> 
> -----Original Message-----
> From: Muruganandham, Ashokkumar [mailto:ashokkumar.muruganandham@fluke.com] 
> Sent: Monday, March 12, 2018 1:19 AM
> To: dev@kafka.apache.org
> Subject: Log Retention Period of Kafka Messages
> 
> Hi Team,
> 
> Form the documentation I could see that log retention period is relative to the partition segments. So unless the segment is closed (1Gb/1 Week) the message will never be deleted.
> 
> Consider I have a very critical use case where I will need to delete the data every day and my segment reaches 1GB after three days. So according the above statement, then there is no way I could configure Kafka to delete the data in one day.
> 
> Is my understanding correct ?
> 
> Regards,
> AK
> ________________________________
> 
> Please be advised that this email may contain confidential information. If you are not the intended recipient, please notify us by email by replying to the sender and delete this message. The sender disclaims that the content of this email constitutes an offer to enter into, or the acceptance of, any agreement; provided that the foregoing does not invalidate the binding effect of any digital or other electronic reproduction of a manual signature that is included in any attachment.
> 
> 
> 
> This message is intended exclusively for the individual or entity to which it is addressed. This communication may contain information that is proprietary, privileged, confidential or otherwise legally exempt from disclosure. If you are not the named addressee, or have been inadvertently and erroneously referenced in the address line, you are not authorized to read, print, retain, copy or disseminate this message or any part of it. If you have received this message in error, please notify the sender immediately by e-mail and delete all copies of the message. (ID m031214)

RE: Log Retention Period of Kafka Messages

[Kyle Tinker <kt...@workforcesoftware.com>]

You have a couple options:
1) You can adjust log.segment.bytes to make the segments smaller so that individual segments can be trimmed
2) You can set log.roll.hours, which will roll to a new log segment even if the size hasn't been reached
Note: #1 and #2 also have per-topic setting controls

https://kafka.apache.org/08/documentation/#brokerconfigs

Thanks,
--


KYLE TINKER
Lead Architect/Team Lead | WorkForce Software 

T: +1 734 742 2616 | E: ktinker@workforcesoftware.com


-----Original Message-----
From: Muruganandham, Ashokkumar [mailto:ashokkumar.muruganandham@fluke.com] 
Sent: Monday, March 12, 2018 1:19 AM
To: dev@kafka.apache.org
Subject: Log Retention Period of Kafka Messages

Hi Team,

Form the documentation I could see that log retention period is relative to the partition segments. So unless the segment is closed (1Gb/1 Week) the message will never be deleted.

Consider I have a very critical use case where I will need to delete the data every day and my segment reaches 1GB after three days. So according the above statement, then there is no way I could configure Kafka to delete the data in one day.

Is my understanding correct ?

Regards,
AK
________________________________

Please be advised that this email may contain confidential information. If you are not the intended recipient, please notify us by email by replying to the sender and delete this message. The sender disclaims that the content of this email constitutes an offer to enter into, or the acceptance of, any agreement; provided that the foregoing does not invalidate the binding effect of any digital or other electronic reproduction of a manual signature that is included in any attachment.



This message is intended exclusively for the individual or entity to which it is addressed. This communication may contain information that is proprietary, privileged, confidential or otherwise legally exempt from disclosure. If you are not the named addressee, or have been inadvertently and erroneously referenced in the address line, you are not authorized to read, print, retain, copy or disseminate this message or any part of it. If you have received this message in error, please notify the sender immediately by e-mail and delete all copies of the message. (ID m031214)