You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by "Nayak, Soumya R." <sn...@firstam.com> on 2019/07/30 10:29:27 UTC
Log Retention
Hi Team,
There are two variables in Kafka - log.retention.hours and log.retention.bytes.
What is the value has to be set for the above two variables so that there is no limit of hours or bytes, the messages stays forever? Is it a production level set up ?
Will it impact kafka performance and also use much of disk space or can we in between change the above variable values and restart the brokers?
Why I am asking the above is in hyperledger fabric blockchain framework kafka and zookeeper is used as a messaging service but recently many people have complained about its pruning of data for unknown reasons.
So just wanted to know how the best can the kafka be configured so that there is no issues of pruning. Currently zookeeper version - 3.4.10 and kafka version - 1.0.0 is used. Will the issues be resolved migrating to latest versions?
Regards,
******************************************************************************************
This message may contain confidential or proprietary information intended only for the use of the
addressee(s) named above or may contain information that is legally privileged. If you are
not the intended addressee, or the person responsible for delivering it to the intended addressee,
you are hereby notified that reading, disseminating, distributing or copying this message is strictly
prohibited. If you have received this message by mistake, please immediately notify us by
replying to the message and delete the original message and any copies immediately thereafter.
If you received this email as a commercial message and would like to opt out of future commercial
messages, please let us know and we will remove you from our distribution list.
Thank you.~
******************************************************************************************
FAFLD
Log Retention
Posted by "Nayak, Soumya R." <sn...@firstam.com>.
Hi Manna,
Thanks for the update. Will try with the log.cleaner.enable-false .
Will the above parameter if false - will override the log.retention.hours and log.retention.bytes properties? And vice versa when true.
Regards,
Soumya
-----Original Message-----
From: M. Manna <ma...@gmail.com>
Sent: Tuesday, July 30, 2019 4:46 PM
To: Kafka Users <us...@kafka.apache.org>
Subject: Re: Log Retention
Hi,
instead of tweaking retention time/size, you can try using log.cleaner.enable=false. It's true by default as of 0.9.0.1. However, i haven't tried that myself.
There is always going to be an impact since it takes up resources. And in production, the general idea is to do housekeeping at an acceptable rate.
That again, depends on how much cost you can bear.
Thanks,
On Tue, 30 Jul 2019 at 11:29, Nayak, Soumya R. <sn...@firstam.com> wrote:
> Hi Team,
>
> There are two variables in Kafka - log.retention.hours and
> log.retention.bytes.
> What is the value has to be set for the above two variables so that
> there is no limit of hours or bytes, the messages stays forever? Is it
> a production level set up ?
> Will it impact kafka performance and also use much of disk space or
> can we in between change the above variable values and restart the brokers?
>
> Why I am asking the above is in hyperledger fabric blockchain
> framework kafka and zookeeper is used as a messaging service but
> recently many people have complained about its pruning of data for unknown reasons.
> So just wanted to know how the best can the kafka be configured so
> that there is no issues of pruning. Currently zookeeper version -
> 3.4.10 and kafka version - 1.0.0 is used. Will the issues be resolved
> migrating to latest versions?
>
> Regards,
>
>
>
> **********************************************************************
> ******************** This message may contain confidential or
> proprietary information intended only for the use of the
> addressee(s) named above or may contain information that is legally
> privileged. If you are not the intended addressee, or the person
> responsible for delivering it to the intended addressee, you are
> hereby notified that reading, disseminating, distributing or copying
> this message is strictly prohibited. If you have received this message
> by mistake, please immediately notify us by replying to the message
> and delete the original message and any copies immediately thereafter.
>
> If you received this email as a commercial message and would like to
> opt out of future commercial messages, please let us know and we will
> remove you from our distribution list.
>
> Thank you.~
>
> **********************************************************************
> ********************
> FAFLD
>
Re: Log Retention
Posted by "M. Manna" <ma...@gmail.com>.
Hi,
instead of tweaking retention time/size, you can try using
log.cleaner.enable=false. It's true by default as of 0.9.0.1. However, i
haven't tried that myself.
There is always going to be an impact since it takes up resources. And in
production, the general idea is to do housekeeping at an acceptable rate.
That again, depends on how much cost you can bear.
Thanks,
On Tue, 30 Jul 2019 at 11:29, Nayak, Soumya R. <sn...@firstam.com> wrote:
> Hi Team,
>
> There are two variables in Kafka - log.retention.hours and
> log.retention.bytes.
> What is the value has to be set for the above two variables so that there
> is no limit of hours or bytes, the messages stays forever? Is it a
> production level set up ?
> Will it impact kafka performance and also use much of disk space or can we
> in between change the above variable values and restart the brokers?
>
> Why I am asking the above is in hyperledger fabric blockchain framework
> kafka and zookeeper is used as a messaging service but recently many people
> have complained about its pruning of data for unknown reasons.
> So just wanted to know how the best can the kafka be configured so that
> there is no issues of pruning. Currently zookeeper version - 3.4.10 and
> kafka version - 1.0.0 is used. Will the issues be resolved migrating to
> latest versions?
>
> Regards,
>
>
>
> ******************************************************************************************
> This message may contain confidential or proprietary information intended
> only for the use of the
> addressee(s) named above or may contain information that is legally
> privileged. If you are
> not the intended addressee, or the person responsible for delivering it to
> the intended addressee,
> you are hereby notified that reading, disseminating, distributing or
> copying this message is strictly
> prohibited. If you have received this message by mistake, please
> immediately notify us by
> replying to the message and delete the original message and any copies
> immediately thereafter.
>
> If you received this email as a commercial message and would like to opt
> out of future commercial
> messages, please let us know and we will remove you from our distribution
> list.
>
> Thank you.~
>
> ******************************************************************************************
> FAFLD
>