You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by François Langelier <f....@gmail.com> on 2014/08/15 21:32:05 UTC
Re: kafka 0.8.1.1 log.retention.minutes NOT being honored
UP! Can someone add the log.retention.hours in the documentation?
And is the retention.hours or retention.minutes available for the by-topic
configuration?
François Langelier
Étudiant en génie Logiciel - École de Technologie Supérieure
<http://www.etsmtl.ca/>
Capitaine Club Capra <http://capra.etsmtl.ca/>
VP-Communication - CS Games <http://csgames.org> 2014
Jeux de Génie <http://www.jdgets.com/> 2011 à 2014
Magistrat Fraternité du Piranha <http://fraternitedupiranha.com/>
Comité Organisateur Olympiades ÉTS 2012
Compétition Québécoise d'Ingénierie 2012 - Compétition Senior
On Thu, Jul 10, 2014 at 11:16 PM, Virendra Pratap Singh <
vpsingh@yahoo-inc.com.invalid> wrote:
> No worries, its working. My deployment was flawed and it was running
> 0.8.0, while I was under the impression that its running 0.8.1.1.
> After going to proper 0.8.1.1, the parameter started working properly.
>
> On 7/10/14, 2:18 PM, "Virendra Pratap Singh" <vp...@yahoo-inc.com>
> wrote:
>
> >Has someone really used : log.retention.minutes in kafka 0.8.1.1.
> >I have my full cluster running on 0.8.1.1 and the logs data is just not
> >getting cleaned up.
> >
> >And I see this message in kafka server.log
> >...
> >[2014-07-10 20:49:43,786] WARN Property log.retention.minutes is not valid
> >(kafka.utils.VerifiableProperties)
> >[2014-07-10 20:49:43,786] WARN Property log.retention.minutes is not valid
> >(kafka.utils.VerifiableProperties)
> >....
> >
> >Virendra
> >
> >On 7/10/14, 10:07 AM, "Virendra Pratap Singh"
> ><vp...@yahoo-inc.com.INVALID> wrote:
> >
> >>Thank you for pointing out to JIRA.
> >>
> >>On 7/9/14, 6:38 PM, "Jun Rao" <ju...@gmail.com> wrote:
> >>
> >>>This is being worked on in
> >>>https://issues.apache.org/jira/browse/KAFKA-1325
> >>>
> >>>Thanks,
> >>>
> >>>Jun
> >>>
> >>>
> >>>On Wed, Jul 9, 2014 at 11:42 AM, Virendra Pratap Singh <
> >>>vpsingh@yahoo-inc.com.invalid> wrote:
> >>>
> >>>> Well currently the log rollup is controlled via
> >>>>
> >>>> log.roll.hours
> >>>> Or
> >>>> log.segment.bytes
> >>>>
> >>>> Given that we now have support to log retention in minutes, I guess it
> >>>> would be apt to have rollup also have capability to be available in
> >>>> minutes. Whom/where should I ask to have that coded in.
> >>>>
> >>>> One a similar note, I would genuinely want all the size and time based
> >>>> parameters to be defined at bytes and ms level. This would make it
> >>>>generic
> >>>> enough to have user choose and define kind of setting they want and
> >>>>not
> >>>> require someone to go and change the code to support a new use case.
> >>>> Having hours/minutes in config will just cause proliferation/bloating
> >>>>of
> >>>> config parameters. What is a good place to air this suggestion.
> >>>>Anyone?
> >>>>
> >>>> Virendra
> >>>>
> >>>> On 7/9/14, 7:21 AM, "Jun Rao" <ju...@gmail.com> wrote:
> >>>>
> >>>> >Actually, Kafka only removes old segments. The last (active) segment
> >>>>is
> >>>> >never removed. So, f you want to have a 10 min retention, you need to
> >>>> >configure log rolling such that log segments are rolled at least
> >>>>every
> >>>>10
> >>>> >mins.
> >>>> >
> >>>> >Thanks,
> >>>> >
> >>>> >Jun
> >>>> >
> >>>> >
> >>>> >On Tue, Jul 8, 2014 at 10:04 PM, Virendra Pratap Singh <
> >>>> >vpsingh@yahoo-inc.com.invalid> wrote:
> >>>> >
> >>>> >> That's correct. The server where in I was running 0.8.1.1 was not
> >>>> >>honoring
> >>>> >> this parameter, despite the fact it was set in it
> >>>>server.properties.
> >>>> >> Not sure if this fact would play any role, the server which was
> >>>>running
> >>>> >> 0.8.0 was the leader for all the topics and partition in my setup.
> >>>>And
> >>>> >>the
> >>>> >> second server running 0.8.1.1 has all the replicas (follower).
> >>>> >>
> >>>> >> Virendra
> >>>> >>
> >>>> >> On 7/8/14, 12:54 PM, "Guozhang Wang" <wa...@gmail.com> wrote:
> >>>> >>
> >>>> >> >Server properties should affect on only the local instance
> >>>>separately.
> >>>> >>Are
> >>>> >> >you saying the property is not honored even on the 0.8.1 machines?
> >>>> >> >
> >>>> >> >Guozhang
> >>>> >> >
> >>>> >> >On Mon, Jul 7, 2014 at 3:55 PM, Virendra Pratap Singh <
> >>>> >> >vpsingh@yahoo-inc.com.invalid> wrote:
> >>>> >> >
> >>>> >> >> By setting this property
> >>>> >> >> log.retention.mins=10
> >>>> >> >> in the server.properties file, which is passed as argument when
> >>>> >>starting
> >>>> >> >> the broker.
> >>>> >> >>
> >>>> >> >> Virendra
> >>>> >> >>
> >>>> >> >> On 7/7/14, 3:31 PM, "Guozhang Wang" <wa...@gmail.com> wrote:
> >>>> >> >>
> >>>> >> >> >How do you set the retention.minutes property? Is it through
> >>>> >>zk-based
> >>>> >> >> >topics tool?
> >>>> >> >> >
> >>>> >> >> >Guozhang
> >>>> >> >> >
> >>>> >> >> >
> >>>> >> >> >On Mon, Jul 7, 2014 at 3:07 PM, Virendra Pratap Singh <
> >>>> >> >> >vpsingh@yahoo-inc.com.invalid> wrote:
> >>>> >> >> >
> >>>> >> >> >> I am running a mixed cluster as I mentioned earlier. 1 broker
> >>>> >>0.8.0
> >>>> >> >>and
> >>>> >> >> >> the other 0.8.1.1. Should the retention of topics for
> >>>>partitions
> >>>> >> >> >> owned/replicated by the broker running 0.8.1.1 not enforce
> >>>>the
> >>>> >>server
> >>>> >> >> >> properties settings as defined for that server.
> >>>> >> >> >>
> >>>> >> >> >> So this brings an interesting question, in case of
> >>>>heterogeneous
> >>>> >> >> >> environment (as is in my case, which system parameters will
> >>>>take
> >>>> >> >> >> preference/precedence).
> >>>> >> >> >>
> >>>> >> >> >> Virendra
> >>>> >> >> >>
> >>>> >> >> >> On 6/30/14, 9:19 AM, "Guozhang Wang" <wa...@gmail.com>
> >>>>wrote:
> >>>> >> >> >>
> >>>> >> >> >> >The retention.minute property is only introduced in 0.8.1:
> >>>> >> >> >> >
> >>>> >> >> >> >https://issues.apache.org/jira/browse/KAFKA-918
> >>>> >> >> >> >
> >>>> >> >> >> >if you are running 0.8.0 then it will not be recognized.
> >>>> >> >> >> >
> >>>> >> >> >> >Guozhang
> >>>> >> >> >> >
> >>>> >> >> >> >
> >>>> >> >> >> >
> >>>> >> >> >> >On Fri, Jun 27, 2014 at 2:13 PM, Virendra Pratap Singh <
> >>>> >> >> >> >vpsingh@yahoo-inc.com.invalid> wrote:
> >>>> >> >> >> >
> >>>> >> >> >> >> Running a mixed 2 broker cluster. Mixed as in one of the
> >>>> >>broker1
> >>>> >> >>is
> >>>> >> >> >> >> running 0.8.0 and broker2 one 0.8.1.1 (from the apache
> >>>>release
> >>>> >> >>link.
> >>>> >> >> >> >> Directly using the tar ball, no local build used).
> >>>> >> >> >> >>
> >>>> >> >> >> >> I have set the log.retention.minutes=10. However the
> >>>>broker
> >>>>is
> >>>> >>not
> >>>> >> >> >> >> honoring the setting. I see its not cleaning the log.dir
> >>>>at
> >>>> >>all.
> >>>> >> >> >> >>
> >>>> >> >> >> >> However when I set the log.retention.hours=1, then it
> >>>>starts
> >>>> >> >>cleaning
> >>>> >> >> >> >>the
> >>>> >> >> >> >> log.
> >>>> >> >> >> >>
> >>>> >> >> >> >> When I have the log.retention.minutes set in the
> >>>> >>server.properties
> >>>> >> >> >>then
> >>>> >> >> >> >>I
> >>>> >> >> >> >> see this logged in server.log:
> >>>> >> >> >> >>
> >>>> >> >> >> >> Š..
> >>>> >> >> >> >> [2014-06-27 19:21:06,633] WARN Property
> >>>>log.retention.minutes
> >>>> >>is
> >>>> >> >>not
> >>>> >> >> >> >>valid
> >>>> >> >> >> >> (kafka.utils.VerifiableProperties)
> >>>> >> >> >> >> [2014-06-27 19:21:06,633] WARN Property
> >>>>log.retention.minutes
> >>>> >>is
> >>>> >> >>not
> >>>> >> >> >> >>valid
> >>>> >> >> >> >> (kafka.utils.VerifiableProperties)
> >>>> >> >> >> >> ŠŠ
> >>>> >> >> >> >>
> >>>> >> >> >> >>
> >>>> >> >> >> >> I have set these properties too:
> >>>> >> >> >> >>
> >>>> >> >> >> >> log.cleaner.enable=true
> >>>> >> >> >> >> log.cleanup.policy=delete
> >>>> >> >> >> >>
> >>>> >> >> >> >>
> >>>> >> >> >> >> But I see similar warning logged for these properties too.
> >>>> >> >> >> >>
> >>>> >> >> >> >> Regards,
> >>>> >> >> >> >> Virendra
> >>>> >> >> >> >>
> >>>> >> >> >> >>
> >>>> >> >> >> >
> >>>> >> >> >> >
> >>>> >> >> >> >--
> >>>> >> >> >> >-- Guozhang
> >>>> >> >> >>
> >>>> >> >> >>
> >>>> >> >> >
> >>>> >> >> >
> >>>> >> >> >--
> >>>> >> >> >-- Guozhang
> >>>> >> >>
> >>>> >> >>
> >>>> >> >
> >>>> >> >
> >>>> >> >--
> >>>> >> >-- Guozhang
> >>>> >>
> >>>> >>
> >>>>
> >>>>
> >>
> >
>
>
Re: kafka 0.8.1.1 log.retention.minutes NOT being honored
Posted by Jun Rao <ju...@gmail.com>.
http://kafka.apache.org/documentation.html#configuration now has the
following entry.
log.retention.{minutes,hours}
Thanks,
Jun
On Thu, Aug 21, 2014 at 7:44 AM, François Langelier <f....@gmail.com>
wrote:
> Thanks Jun,
>
> I don't see changes in the documentation :(
> Just to be sure,
>
> We can either use "*log.retention.hours*" (giving the number of *hours* you
> want the retention to last) or "*log.retention.minutes*" (giving the number
> of *minutes* you want the retention to last) for all the topics (in
> server.properties)
> And we can use "*retention.ms <http://retention.ms>*" (giving the number
> of *milliseconds* you want the retention to last) for a specific topic.
>
> If so, why log.retention.hours isn't written here
> http://kafka.apache.org/documentation.html#configuration
>
> if not, there is a example using "log.retention.hours"
> http://kafka.apache.org/documentation.html#config and also in the example
> of the server.properties on github.
>
>
>
> François Langelier
> Étudiant en génie Logiciel - École de Technologie Supérieure
> <http://www.etsmtl.ca/>
> Capitaine Club Capra <http://capra.etsmtl.ca/>
> VP-Communication - CS Games <http://csgames.org> 2014
> Jeux de Génie <http://www.jdgets.com/> 2011 à 2014
> Magistrat Fraternité du Piranha <http://fraternitedupiranha.com/>
> Comité Organisateur Olympiades ÉTS 2012
> Compétition Québécoise d'Ingénierie 2012 - Compétition Senior
>
>
> On Thu, Aug 21, 2014 at 12:50 AM, Jun Rao <ju...@gmail.com> wrote:
>
> > Added to the documentation. Topic level retentions is called
> retention.ms
> > and can be found in
> > http://kafka.apache.org/documentation.html#configuration
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Fri, Aug 15, 2014 at 12:32 PM, François Langelier <
> > f.langelier@gmail.com>
> > wrote:
> >
> > > UP! Can someone add the log.retention.hours in the documentation?
> > >
> > > And is the retention.hours or retention.minutes available for the
> > by-topic
> > > configuration?
> > >
> > >
> > >
> > > François Langelier
> > > Étudiant en génie Logiciel - École de Technologie Supérieure
> > > <http://www.etsmtl.ca/>
> > > Capitaine Club Capra <http://capra.etsmtl.ca/>
> > > VP-Communication - CS Games <http://csgames.org> 2014
> > > Jeux de Génie <http://www.jdgets.com/> 2011 à 2014
> > > Magistrat Fraternité du Piranha <http://fraternitedupiranha.com/>
> > > Comité Organisateur Olympiades ÉTS 2012
> > > Compétition Québécoise d'Ingénierie 2012 - Compétition Senior
> > >
> > >
> > > On Thu, Jul 10, 2014 at 11:16 PM, Virendra Pratap Singh <
> > > vpsingh@yahoo-inc.com.invalid> wrote:
> > >
> > > > No worries, its working. My deployment was flawed and it was running
> > > > 0.8.0, while I was under the impression that its running 0.8.1.1.
> > > > After going to proper 0.8.1.1, the parameter started working
> properly.
> > > >
> > > > On 7/10/14, 2:18 PM, "Virendra Pratap Singh" <vp...@yahoo-inc.com>
> > > > wrote:
> > > >
> > > > >Has someone really used : log.retention.minutes in kafka 0.8.1.1.
> > > > >I have my full cluster running on 0.8.1.1 and the logs data is just
> > not
> > > > >getting cleaned up.
> > > > >
> > > > >And I see this message in kafka server.log
> > > > >...
> > > > >[2014-07-10 20:49:43,786] WARN Property log.retention.minutes is not
> > > valid
> > > > >(kafka.utils.VerifiableProperties)
> > > > >[2014-07-10 20:49:43,786] WARN Property log.retention.minutes is not
> > > valid
> > > > >(kafka.utils.VerifiableProperties)
> > > > >....
> > > > >
> > > > >Virendra
> > > > >
> > > > >On 7/10/14, 10:07 AM, "Virendra Pratap Singh"
> > > > ><vp...@yahoo-inc.com.INVALID> wrote:
> > > > >
> > > > >>Thank you for pointing out to JIRA.
> > > > >>
> > > > >>On 7/9/14, 6:38 PM, "Jun Rao" <ju...@gmail.com> wrote:
> > > > >>
> > > > >>>This is being worked on in
> > > > >>>https://issues.apache.org/jira/browse/KAFKA-1325
> > > > >>>
> > > > >>>Thanks,
> > > > >>>
> > > > >>>Jun
> > > > >>>
> > > > >>>
> > > > >>>On Wed, Jul 9, 2014 at 11:42 AM, Virendra Pratap Singh <
> > > > >>>vpsingh@yahoo-inc.com.invalid> wrote:
> > > > >>>
> > > > >>>> Well currently the log rollup is controlled via
> > > > >>>>
> > > > >>>> log.roll.hours
> > > > >>>> Or
> > > > >>>> log.segment.bytes
> > > > >>>>
> > > > >>>> Given that we now have support to log retention in minutes, I
> > guess
> > > it
> > > > >>>> would be apt to have rollup also have capability to be available
> > in
> > > > >>>> minutes. Whom/where should I ask to have that coded in.
> > > > >>>>
> > > > >>>> One a similar note, I would genuinely want all the size and time
> > > based
> > > > >>>> parameters to be defined at bytes and ms level. This would make
> it
> > > > >>>>generic
> > > > >>>> enough to have user choose and define kind of setting they want
> > and
> > > > >>>>not
> > > > >>>> require someone to go and change the code to support a new use
> > case.
> > > > >>>> Having hours/minutes in config will just cause
> > > proliferation/bloating
> > > > >>>>of
> > > > >>>> config parameters. What is a good place to air this suggestion.
> > > > >>>>Anyone?
> > > > >>>>
> > > > >>>> Virendra
> > > > >>>>
> > > > >>>> On 7/9/14, 7:21 AM, "Jun Rao" <ju...@gmail.com> wrote:
> > > > >>>>
> > > > >>>> >Actually, Kafka only removes old segments. The last (active)
> > > segment
> > > > >>>>is
> > > > >>>> >never removed. So, f you want to have a 10 min retention, you
> > need
> > > to
> > > > >>>> >configure log rolling such that log segments are rolled at
> least
> > > > >>>>every
> > > > >>>>10
> > > > >>>> >mins.
> > > > >>>> >
> > > > >>>> >Thanks,
> > > > >>>> >
> > > > >>>> >Jun
> > > > >>>> >
> > > > >>>> >
> > > > >>>> >On Tue, Jul 8, 2014 at 10:04 PM, Virendra Pratap Singh <
> > > > >>>> >vpsingh@yahoo-inc.com.invalid> wrote:
> > > > >>>> >
> > > > >>>> >> That's correct. The server where in I was running 0.8.1.1 was
> > not
> > > > >>>> >>honoring
> > > > >>>> >> this parameter, despite the fact it was set in it
> > > > >>>>server.properties.
> > > > >>>> >> Not sure if this fact would play any role, the server which
> was
> > > > >>>>running
> > > > >>>> >> 0.8.0 was the leader for all the topics and partition in my
> > > setup.
> > > > >>>>And
> > > > >>>> >>the
> > > > >>>> >> second server running 0.8.1.1 has all the replicas
> (follower).
> > > > >>>> >>
> > > > >>>> >> Virendra
> > > > >>>> >>
> > > > >>>> >> On 7/8/14, 12:54 PM, "Guozhang Wang" <wa...@gmail.com>
> > wrote:
> > > > >>>> >>
> > > > >>>> >> >Server properties should affect on only the local instance
> > > > >>>>separately.
> > > > >>>> >>Are
> > > > >>>> >> >you saying the property is not honored even on the 0.8.1
> > > machines?
> > > > >>>> >> >
> > > > >>>> >> >Guozhang
> > > > >>>> >> >
> > > > >>>> >> >On Mon, Jul 7, 2014 at 3:55 PM, Virendra Pratap Singh <
> > > > >>>> >> >vpsingh@yahoo-inc.com.invalid> wrote:
> > > > >>>> >> >
> > > > >>>> >> >> By setting this property
> > > > >>>> >> >> log.retention.mins=10
> > > > >>>> >> >> in the server.properties file, which is passed as argument
> > > when
> > > > >>>> >>starting
> > > > >>>> >> >> the broker.
> > > > >>>> >> >>
> > > > >>>> >> >> Virendra
> > > > >>>> >> >>
> > > > >>>> >> >> On 7/7/14, 3:31 PM, "Guozhang Wang" <wa...@gmail.com>
> > > wrote:
> > > > >>>> >> >>
> > > > >>>> >> >> >How do you set the retention.minutes property? Is it
> > through
> > > > >>>> >>zk-based
> > > > >>>> >> >> >topics tool?
> > > > >>>> >> >> >
> > > > >>>> >> >> >Guozhang
> > > > >>>> >> >> >
> > > > >>>> >> >> >
> > > > >>>> >> >> >On Mon, Jul 7, 2014 at 3:07 PM, Virendra Pratap Singh <
> > > > >>>> >> >> >vpsingh@yahoo-inc.com.invalid> wrote:
> > > > >>>> >> >> >
> > > > >>>> >> >> >> I am running a mixed cluster as I mentioned earlier. 1
> > > broker
> > > > >>>> >>0.8.0
> > > > >>>> >> >>and
> > > > >>>> >> >> >> the other 0.8.1.1. Should the retention of topics for
> > > > >>>>partitions
> > > > >>>> >> >> >> owned/replicated by the broker running 0.8.1.1 not
> > enforce
> > > > >>>>the
> > > > >>>> >>server
> > > > >>>> >> >> >> properties settings as defined for that server.
> > > > >>>> >> >> >>
> > > > >>>> >> >> >> So this brings an interesting question, in case of
> > > > >>>>heterogeneous
> > > > >>>> >> >> >> environment (as is in my case, which system parameters
> > will
> > > > >>>>take
> > > > >>>> >> >> >> preference/precedence).
> > > > >>>> >> >> >>
> > > > >>>> >> >> >> Virendra
> > > > >>>> >> >> >>
> > > > >>>> >> >> >> On 6/30/14, 9:19 AM, "Guozhang Wang" <
> wangguoz@gmail.com
> > >
> > > > >>>>wrote:
> > > > >>>> >> >> >>
> > > > >>>> >> >> >> >The retention.minute property is only introduced in
> > 0.8.1:
> > > > >>>> >> >> >> >
> > > > >>>> >> >> >> >https://issues.apache.org/jira/browse/KAFKA-918
> > > > >>>> >> >> >> >
> > > > >>>> >> >> >> >if you are running 0.8.0 then it will not be
> recognized.
> > > > >>>> >> >> >> >
> > > > >>>> >> >> >> >Guozhang
> > > > >>>> >> >> >> >
> > > > >>>> >> >> >> >
> > > > >>>> >> >> >> >
> > > > >>>> >> >> >> >On Fri, Jun 27, 2014 at 2:13 PM, Virendra Pratap
> Singh <
> > > > >>>> >> >> >> >vpsingh@yahoo-inc.com.invalid> wrote:
> > > > >>>> >> >> >> >
> > > > >>>> >> >> >> >> Running a mixed 2 broker cluster. Mixed as in one of
> > the
> > > > >>>> >>broker1
> > > > >>>> >> >>is
> > > > >>>> >> >> >> >> running 0.8.0 and broker2 one 0.8.1.1 (from the
> apache
> > > > >>>>release
> > > > >>>> >> >>link.
> > > > >>>> >> >> >> >> Directly using the tar ball, no local build used).
> > > > >>>> >> >> >> >>
> > > > >>>> >> >> >> >> I have set the log.retention.minutes=10. However the
> > > > >>>>broker
> > > > >>>>is
> > > > >>>> >>not
> > > > >>>> >> >> >> >> honoring the setting. I see its not cleaning the
> > log.dir
> > > > >>>>at
> > > > >>>> >>all.
> > > > >>>> >> >> >> >>
> > > > >>>> >> >> >> >> However when I set the log.retention.hours=1, then
> it
> > > > >>>>starts
> > > > >>>> >> >>cleaning
> > > > >>>> >> >> >> >>the
> > > > >>>> >> >> >> >> log.
> > > > >>>> >> >> >> >>
> > > > >>>> >> >> >> >> When I have the log.retention.minutes set in the
> > > > >>>> >>server.properties
> > > > >>>> >> >> >>then
> > > > >>>> >> >> >> >>I
> > > > >>>> >> >> >> >> see this logged in server.log:
> > > > >>>> >> >> >> >>
> > > > >>>> >> >> >> >> Š..
> > > > >>>> >> >> >> >> [2014-06-27 19:21:06,633] WARN Property
> > > > >>>>log.retention.minutes
> > > > >>>> >>is
> > > > >>>> >> >>not
> > > > >>>> >> >> >> >>valid
> > > > >>>> >> >> >> >> (kafka.utils.VerifiableProperties)
> > > > >>>> >> >> >> >> [2014-06-27 19:21:06,633] WARN Property
> > > > >>>>log.retention.minutes
> > > > >>>> >>is
> > > > >>>> >> >>not
> > > > >>>> >> >> >> >>valid
> > > > >>>> >> >> >> >> (kafka.utils.VerifiableProperties)
> > > > >>>> >> >> >> >> ŠŠ
> > > > >>>> >> >> >> >>
> > > > >>>> >> >> >> >>
> > > > >>>> >> >> >> >> I have set these properties too:
> > > > >>>> >> >> >> >>
> > > > >>>> >> >> >> >> log.cleaner.enable=true
> > > > >>>> >> >> >> >> log.cleanup.policy=delete
> > > > >>>> >> >> >> >>
> > > > >>>> >> >> >> >>
> > > > >>>> >> >> >> >> But I see similar warning logged for these
> properties
> > > too.
> > > > >>>> >> >> >> >>
> > > > >>>> >> >> >> >> Regards,
> > > > >>>> >> >> >> >> Virendra
> > > > >>>> >> >> >> >>
> > > > >>>> >> >> >> >>
> > > > >>>> >> >> >> >
> > > > >>>> >> >> >> >
> > > > >>>> >> >> >> >--
> > > > >>>> >> >> >> >-- Guozhang
> > > > >>>> >> >> >>
> > > > >>>> >> >> >>
> > > > >>>> >> >> >
> > > > >>>> >> >> >
> > > > >>>> >> >> >--
> > > > >>>> >> >> >-- Guozhang
> > > > >>>> >> >>
> > > > >>>> >> >>
> > > > >>>> >> >
> > > > >>>> >> >
> > > > >>>> >> >--
> > > > >>>> >> >-- Guozhang
> > > > >>>> >>
> > > > >>>> >>
> > > > >>>>
> > > > >>>>
> > > > >>
> > > > >
> > > >
> > > >
> > >
> >
>
Re: kafka 0.8.1.1 log.retention.minutes NOT being honored
Posted by François Langelier <f....@gmail.com>.
Thanks Jun,
I don't see changes in the documentation :(
Just to be sure,
We can either use "*log.retention.hours*" (giving the number of *hours* you
want the retention to last) or "*log.retention.minutes*" (giving the number
of *minutes* you want the retention to last) for all the topics (in
server.properties)
And we can use "*retention.ms <http://retention.ms>*" (giving the number
of *milliseconds* you want the retention to last) for a specific topic.
If so, why log.retention.hours isn't written here
http://kafka.apache.org/documentation.html#configuration
if not, there is a example using "log.retention.hours"
http://kafka.apache.org/documentation.html#config and also in the example
of the server.properties on github.
François Langelier
Étudiant en génie Logiciel - École de Technologie Supérieure
<http://www.etsmtl.ca/>
Capitaine Club Capra <http://capra.etsmtl.ca/>
VP-Communication - CS Games <http://csgames.org> 2014
Jeux de Génie <http://www.jdgets.com/> 2011 à 2014
Magistrat Fraternité du Piranha <http://fraternitedupiranha.com/>
Comité Organisateur Olympiades ÉTS 2012
Compétition Québécoise d'Ingénierie 2012 - Compétition Senior
On Thu, Aug 21, 2014 at 12:50 AM, Jun Rao <ju...@gmail.com> wrote:
> Added to the documentation. Topic level retentions is called retention.ms
> and can be found in
> http://kafka.apache.org/documentation.html#configuration
>
> Thanks,
>
> Jun
>
>
> On Fri, Aug 15, 2014 at 12:32 PM, François Langelier <
> f.langelier@gmail.com>
> wrote:
>
> > UP! Can someone add the log.retention.hours in the documentation?
> >
> > And is the retention.hours or retention.minutes available for the
> by-topic
> > configuration?
> >
> >
> >
> > François Langelier
> > Étudiant en génie Logiciel - École de Technologie Supérieure
> > <http://www.etsmtl.ca/>
> > Capitaine Club Capra <http://capra.etsmtl.ca/>
> > VP-Communication - CS Games <http://csgames.org> 2014
> > Jeux de Génie <http://www.jdgets.com/> 2011 à 2014
> > Magistrat Fraternité du Piranha <http://fraternitedupiranha.com/>
> > Comité Organisateur Olympiades ÉTS 2012
> > Compétition Québécoise d'Ingénierie 2012 - Compétition Senior
> >
> >
> > On Thu, Jul 10, 2014 at 11:16 PM, Virendra Pratap Singh <
> > vpsingh@yahoo-inc.com.invalid> wrote:
> >
> > > No worries, its working. My deployment was flawed and it was running
> > > 0.8.0, while I was under the impression that its running 0.8.1.1.
> > > After going to proper 0.8.1.1, the parameter started working properly.
> > >
> > > On 7/10/14, 2:18 PM, "Virendra Pratap Singh" <vp...@yahoo-inc.com>
> > > wrote:
> > >
> > > >Has someone really used : log.retention.minutes in kafka 0.8.1.1.
> > > >I have my full cluster running on 0.8.1.1 and the logs data is just
> not
> > > >getting cleaned up.
> > > >
> > > >And I see this message in kafka server.log
> > > >...
> > > >[2014-07-10 20:49:43,786] WARN Property log.retention.minutes is not
> > valid
> > > >(kafka.utils.VerifiableProperties)
> > > >[2014-07-10 20:49:43,786] WARN Property log.retention.minutes is not
> > valid
> > > >(kafka.utils.VerifiableProperties)
> > > >....
> > > >
> > > >Virendra
> > > >
> > > >On 7/10/14, 10:07 AM, "Virendra Pratap Singh"
> > > ><vp...@yahoo-inc.com.INVALID> wrote:
> > > >
> > > >>Thank you for pointing out to JIRA.
> > > >>
> > > >>On 7/9/14, 6:38 PM, "Jun Rao" <ju...@gmail.com> wrote:
> > > >>
> > > >>>This is being worked on in
> > > >>>https://issues.apache.org/jira/browse/KAFKA-1325
> > > >>>
> > > >>>Thanks,
> > > >>>
> > > >>>Jun
> > > >>>
> > > >>>
> > > >>>On Wed, Jul 9, 2014 at 11:42 AM, Virendra Pratap Singh <
> > > >>>vpsingh@yahoo-inc.com.invalid> wrote:
> > > >>>
> > > >>>> Well currently the log rollup is controlled via
> > > >>>>
> > > >>>> log.roll.hours
> > > >>>> Or
> > > >>>> log.segment.bytes
> > > >>>>
> > > >>>> Given that we now have support to log retention in minutes, I
> guess
> > it
> > > >>>> would be apt to have rollup also have capability to be available
> in
> > > >>>> minutes. Whom/where should I ask to have that coded in.
> > > >>>>
> > > >>>> One a similar note, I would genuinely want all the size and time
> > based
> > > >>>> parameters to be defined at bytes and ms level. This would make it
> > > >>>>generic
> > > >>>> enough to have user choose and define kind of setting they want
> and
> > > >>>>not
> > > >>>> require someone to go and change the code to support a new use
> case.
> > > >>>> Having hours/minutes in config will just cause
> > proliferation/bloating
> > > >>>>of
> > > >>>> config parameters. What is a good place to air this suggestion.
> > > >>>>Anyone?
> > > >>>>
> > > >>>> Virendra
> > > >>>>
> > > >>>> On 7/9/14, 7:21 AM, "Jun Rao" <ju...@gmail.com> wrote:
> > > >>>>
> > > >>>> >Actually, Kafka only removes old segments. The last (active)
> > segment
> > > >>>>is
> > > >>>> >never removed. So, f you want to have a 10 min retention, you
> need
> > to
> > > >>>> >configure log rolling such that log segments are rolled at least
> > > >>>>every
> > > >>>>10
> > > >>>> >mins.
> > > >>>> >
> > > >>>> >Thanks,
> > > >>>> >
> > > >>>> >Jun
> > > >>>> >
> > > >>>> >
> > > >>>> >On Tue, Jul 8, 2014 at 10:04 PM, Virendra Pratap Singh <
> > > >>>> >vpsingh@yahoo-inc.com.invalid> wrote:
> > > >>>> >
> > > >>>> >> That's correct. The server where in I was running 0.8.1.1 was
> not
> > > >>>> >>honoring
> > > >>>> >> this parameter, despite the fact it was set in it
> > > >>>>server.properties.
> > > >>>> >> Not sure if this fact would play any role, the server which was
> > > >>>>running
> > > >>>> >> 0.8.0 was the leader for all the topics and partition in my
> > setup.
> > > >>>>And
> > > >>>> >>the
> > > >>>> >> second server running 0.8.1.1 has all the replicas (follower).
> > > >>>> >>
> > > >>>> >> Virendra
> > > >>>> >>
> > > >>>> >> On 7/8/14, 12:54 PM, "Guozhang Wang" <wa...@gmail.com>
> wrote:
> > > >>>> >>
> > > >>>> >> >Server properties should affect on only the local instance
> > > >>>>separately.
> > > >>>> >>Are
> > > >>>> >> >you saying the property is not honored even on the 0.8.1
> > machines?
> > > >>>> >> >
> > > >>>> >> >Guozhang
> > > >>>> >> >
> > > >>>> >> >On Mon, Jul 7, 2014 at 3:55 PM, Virendra Pratap Singh <
> > > >>>> >> >vpsingh@yahoo-inc.com.invalid> wrote:
> > > >>>> >> >
> > > >>>> >> >> By setting this property
> > > >>>> >> >> log.retention.mins=10
> > > >>>> >> >> in the server.properties file, which is passed as argument
> > when
> > > >>>> >>starting
> > > >>>> >> >> the broker.
> > > >>>> >> >>
> > > >>>> >> >> Virendra
> > > >>>> >> >>
> > > >>>> >> >> On 7/7/14, 3:31 PM, "Guozhang Wang" <wa...@gmail.com>
> > wrote:
> > > >>>> >> >>
> > > >>>> >> >> >How do you set the retention.minutes property? Is it
> through
> > > >>>> >>zk-based
> > > >>>> >> >> >topics tool?
> > > >>>> >> >> >
> > > >>>> >> >> >Guozhang
> > > >>>> >> >> >
> > > >>>> >> >> >
> > > >>>> >> >> >On Mon, Jul 7, 2014 at 3:07 PM, Virendra Pratap Singh <
> > > >>>> >> >> >vpsingh@yahoo-inc.com.invalid> wrote:
> > > >>>> >> >> >
> > > >>>> >> >> >> I am running a mixed cluster as I mentioned earlier. 1
> > broker
> > > >>>> >>0.8.0
> > > >>>> >> >>and
> > > >>>> >> >> >> the other 0.8.1.1. Should the retention of topics for
> > > >>>>partitions
> > > >>>> >> >> >> owned/replicated by the broker running 0.8.1.1 not
> enforce
> > > >>>>the
> > > >>>> >>server
> > > >>>> >> >> >> properties settings as defined for that server.
> > > >>>> >> >> >>
> > > >>>> >> >> >> So this brings an interesting question, in case of
> > > >>>>heterogeneous
> > > >>>> >> >> >> environment (as is in my case, which system parameters
> will
> > > >>>>take
> > > >>>> >> >> >> preference/precedence).
> > > >>>> >> >> >>
> > > >>>> >> >> >> Virendra
> > > >>>> >> >> >>
> > > >>>> >> >> >> On 6/30/14, 9:19 AM, "Guozhang Wang" <wangguoz@gmail.com
> >
> > > >>>>wrote:
> > > >>>> >> >> >>
> > > >>>> >> >> >> >The retention.minute property is only introduced in
> 0.8.1:
> > > >>>> >> >> >> >
> > > >>>> >> >> >> >https://issues.apache.org/jira/browse/KAFKA-918
> > > >>>> >> >> >> >
> > > >>>> >> >> >> >if you are running 0.8.0 then it will not be recognized.
> > > >>>> >> >> >> >
> > > >>>> >> >> >> >Guozhang
> > > >>>> >> >> >> >
> > > >>>> >> >> >> >
> > > >>>> >> >> >> >
> > > >>>> >> >> >> >On Fri, Jun 27, 2014 at 2:13 PM, Virendra Pratap Singh <
> > > >>>> >> >> >> >vpsingh@yahoo-inc.com.invalid> wrote:
> > > >>>> >> >> >> >
> > > >>>> >> >> >> >> Running a mixed 2 broker cluster. Mixed as in one of
> the
> > > >>>> >>broker1
> > > >>>> >> >>is
> > > >>>> >> >> >> >> running 0.8.0 and broker2 one 0.8.1.1 (from the apache
> > > >>>>release
> > > >>>> >> >>link.
> > > >>>> >> >> >> >> Directly using the tar ball, no local build used).
> > > >>>> >> >> >> >>
> > > >>>> >> >> >> >> I have set the log.retention.minutes=10. However the
> > > >>>>broker
> > > >>>>is
> > > >>>> >>not
> > > >>>> >> >> >> >> honoring the setting. I see its not cleaning the
> log.dir
> > > >>>>at
> > > >>>> >>all.
> > > >>>> >> >> >> >>
> > > >>>> >> >> >> >> However when I set the log.retention.hours=1, then it
> > > >>>>starts
> > > >>>> >> >>cleaning
> > > >>>> >> >> >> >>the
> > > >>>> >> >> >> >> log.
> > > >>>> >> >> >> >>
> > > >>>> >> >> >> >> When I have the log.retention.minutes set in the
> > > >>>> >>server.properties
> > > >>>> >> >> >>then
> > > >>>> >> >> >> >>I
> > > >>>> >> >> >> >> see this logged in server.log:
> > > >>>> >> >> >> >>
> > > >>>> >> >> >> >> Š..
> > > >>>> >> >> >> >> [2014-06-27 19:21:06,633] WARN Property
> > > >>>>log.retention.minutes
> > > >>>> >>is
> > > >>>> >> >>not
> > > >>>> >> >> >> >>valid
> > > >>>> >> >> >> >> (kafka.utils.VerifiableProperties)
> > > >>>> >> >> >> >> [2014-06-27 19:21:06,633] WARN Property
> > > >>>>log.retention.minutes
> > > >>>> >>is
> > > >>>> >> >>not
> > > >>>> >> >> >> >>valid
> > > >>>> >> >> >> >> (kafka.utils.VerifiableProperties)
> > > >>>> >> >> >> >> ŠŠ
> > > >>>> >> >> >> >>
> > > >>>> >> >> >> >>
> > > >>>> >> >> >> >> I have set these properties too:
> > > >>>> >> >> >> >>
> > > >>>> >> >> >> >> log.cleaner.enable=true
> > > >>>> >> >> >> >> log.cleanup.policy=delete
> > > >>>> >> >> >> >>
> > > >>>> >> >> >> >>
> > > >>>> >> >> >> >> But I see similar warning logged for these properties
> > too.
> > > >>>> >> >> >> >>
> > > >>>> >> >> >> >> Regards,
> > > >>>> >> >> >> >> Virendra
> > > >>>> >> >> >> >>
> > > >>>> >> >> >> >>
> > > >>>> >> >> >> >
> > > >>>> >> >> >> >
> > > >>>> >> >> >> >--
> > > >>>> >> >> >> >-- Guozhang
> > > >>>> >> >> >>
> > > >>>> >> >> >>
> > > >>>> >> >> >
> > > >>>> >> >> >
> > > >>>> >> >> >--
> > > >>>> >> >> >-- Guozhang
> > > >>>> >> >>
> > > >>>> >> >>
> > > >>>> >> >
> > > >>>> >> >
> > > >>>> >> >--
> > > >>>> >> >-- Guozhang
> > > >>>> >>
> > > >>>> >>
> > > >>>>
> > > >>>>
> > > >>
> > > >
> > >
> > >
> >
>
Re: kafka 0.8.1.1 log.retention.minutes NOT being honored
Posted by Jun Rao <ju...@gmail.com>.
Added to the documentation. Topic level retentions is called retention.ms
and can be found in http://kafka.apache.org/documentation.html#configuration
Thanks,
Jun
On Fri, Aug 15, 2014 at 12:32 PM, François Langelier <f....@gmail.com>
wrote:
> UP! Can someone add the log.retention.hours in the documentation?
>
> And is the retention.hours or retention.minutes available for the by-topic
> configuration?
>
>
>
> François Langelier
> Étudiant en génie Logiciel - École de Technologie Supérieure
> <http://www.etsmtl.ca/>
> Capitaine Club Capra <http://capra.etsmtl.ca/>
> VP-Communication - CS Games <http://csgames.org> 2014
> Jeux de Génie <http://www.jdgets.com/> 2011 à 2014
> Magistrat Fraternité du Piranha <http://fraternitedupiranha.com/>
> Comité Organisateur Olympiades ÉTS 2012
> Compétition Québécoise d'Ingénierie 2012 - Compétition Senior
>
>
> On Thu, Jul 10, 2014 at 11:16 PM, Virendra Pratap Singh <
> vpsingh@yahoo-inc.com.invalid> wrote:
>
> > No worries, its working. My deployment was flawed and it was running
> > 0.8.0, while I was under the impression that its running 0.8.1.1.
> > After going to proper 0.8.1.1, the parameter started working properly.
> >
> > On 7/10/14, 2:18 PM, "Virendra Pratap Singh" <vp...@yahoo-inc.com>
> > wrote:
> >
> > >Has someone really used : log.retention.minutes in kafka 0.8.1.1.
> > >I have my full cluster running on 0.8.1.1 and the logs data is just not
> > >getting cleaned up.
> > >
> > >And I see this message in kafka server.log
> > >...
> > >[2014-07-10 20:49:43,786] WARN Property log.retention.minutes is not
> valid
> > >(kafka.utils.VerifiableProperties)
> > >[2014-07-10 20:49:43,786] WARN Property log.retention.minutes is not
> valid
> > >(kafka.utils.VerifiableProperties)
> > >....
> > >
> > >Virendra
> > >
> > >On 7/10/14, 10:07 AM, "Virendra Pratap Singh"
> > ><vp...@yahoo-inc.com.INVALID> wrote:
> > >
> > >>Thank you for pointing out to JIRA.
> > >>
> > >>On 7/9/14, 6:38 PM, "Jun Rao" <ju...@gmail.com> wrote:
> > >>
> > >>>This is being worked on in
> > >>>https://issues.apache.org/jira/browse/KAFKA-1325
> > >>>
> > >>>Thanks,
> > >>>
> > >>>Jun
> > >>>
> > >>>
> > >>>On Wed, Jul 9, 2014 at 11:42 AM, Virendra Pratap Singh <
> > >>>vpsingh@yahoo-inc.com.invalid> wrote:
> > >>>
> > >>>> Well currently the log rollup is controlled via
> > >>>>
> > >>>> log.roll.hours
> > >>>> Or
> > >>>> log.segment.bytes
> > >>>>
> > >>>> Given that we now have support to log retention in minutes, I guess
> it
> > >>>> would be apt to have rollup also have capability to be available in
> > >>>> minutes. Whom/where should I ask to have that coded in.
> > >>>>
> > >>>> One a similar note, I would genuinely want all the size and time
> based
> > >>>> parameters to be defined at bytes and ms level. This would make it
> > >>>>generic
> > >>>> enough to have user choose and define kind of setting they want and
> > >>>>not
> > >>>> require someone to go and change the code to support a new use case.
> > >>>> Having hours/minutes in config will just cause
> proliferation/bloating
> > >>>>of
> > >>>> config parameters. What is a good place to air this suggestion.
> > >>>>Anyone?
> > >>>>
> > >>>> Virendra
> > >>>>
> > >>>> On 7/9/14, 7:21 AM, "Jun Rao" <ju...@gmail.com> wrote:
> > >>>>
> > >>>> >Actually, Kafka only removes old segments. The last (active)
> segment
> > >>>>is
> > >>>> >never removed. So, f you want to have a 10 min retention, you need
> to
> > >>>> >configure log rolling such that log segments are rolled at least
> > >>>>every
> > >>>>10
> > >>>> >mins.
> > >>>> >
> > >>>> >Thanks,
> > >>>> >
> > >>>> >Jun
> > >>>> >
> > >>>> >
> > >>>> >On Tue, Jul 8, 2014 at 10:04 PM, Virendra Pratap Singh <
> > >>>> >vpsingh@yahoo-inc.com.invalid> wrote:
> > >>>> >
> > >>>> >> That's correct. The server where in I was running 0.8.1.1 was not
> > >>>> >>honoring
> > >>>> >> this parameter, despite the fact it was set in it
> > >>>>server.properties.
> > >>>> >> Not sure if this fact would play any role, the server which was
> > >>>>running
> > >>>> >> 0.8.0 was the leader for all the topics and partition in my
> setup.
> > >>>>And
> > >>>> >>the
> > >>>> >> second server running 0.8.1.1 has all the replicas (follower).
> > >>>> >>
> > >>>> >> Virendra
> > >>>> >>
> > >>>> >> On 7/8/14, 12:54 PM, "Guozhang Wang" <wa...@gmail.com> wrote:
> > >>>> >>
> > >>>> >> >Server properties should affect on only the local instance
> > >>>>separately.
> > >>>> >>Are
> > >>>> >> >you saying the property is not honored even on the 0.8.1
> machines?
> > >>>> >> >
> > >>>> >> >Guozhang
> > >>>> >> >
> > >>>> >> >On Mon, Jul 7, 2014 at 3:55 PM, Virendra Pratap Singh <
> > >>>> >> >vpsingh@yahoo-inc.com.invalid> wrote:
> > >>>> >> >
> > >>>> >> >> By setting this property
> > >>>> >> >> log.retention.mins=10
> > >>>> >> >> in the server.properties file, which is passed as argument
> when
> > >>>> >>starting
> > >>>> >> >> the broker.
> > >>>> >> >>
> > >>>> >> >> Virendra
> > >>>> >> >>
> > >>>> >> >> On 7/7/14, 3:31 PM, "Guozhang Wang" <wa...@gmail.com>
> wrote:
> > >>>> >> >>
> > >>>> >> >> >How do you set the retention.minutes property? Is it through
> > >>>> >>zk-based
> > >>>> >> >> >topics tool?
> > >>>> >> >> >
> > >>>> >> >> >Guozhang
> > >>>> >> >> >
> > >>>> >> >> >
> > >>>> >> >> >On Mon, Jul 7, 2014 at 3:07 PM, Virendra Pratap Singh <
> > >>>> >> >> >vpsingh@yahoo-inc.com.invalid> wrote:
> > >>>> >> >> >
> > >>>> >> >> >> I am running a mixed cluster as I mentioned earlier. 1
> broker
> > >>>> >>0.8.0
> > >>>> >> >>and
> > >>>> >> >> >> the other 0.8.1.1. Should the retention of topics for
> > >>>>partitions
> > >>>> >> >> >> owned/replicated by the broker running 0.8.1.1 not enforce
> > >>>>the
> > >>>> >>server
> > >>>> >> >> >> properties settings as defined for that server.
> > >>>> >> >> >>
> > >>>> >> >> >> So this brings an interesting question, in case of
> > >>>>heterogeneous
> > >>>> >> >> >> environment (as is in my case, which system parameters will
> > >>>>take
> > >>>> >> >> >> preference/precedence).
> > >>>> >> >> >>
> > >>>> >> >> >> Virendra
> > >>>> >> >> >>
> > >>>> >> >> >> On 6/30/14, 9:19 AM, "Guozhang Wang" <wa...@gmail.com>
> > >>>>wrote:
> > >>>> >> >> >>
> > >>>> >> >> >> >The retention.minute property is only introduced in 0.8.1:
> > >>>> >> >> >> >
> > >>>> >> >> >> >https://issues.apache.org/jira/browse/KAFKA-918
> > >>>> >> >> >> >
> > >>>> >> >> >> >if you are running 0.8.0 then it will not be recognized.
> > >>>> >> >> >> >
> > >>>> >> >> >> >Guozhang
> > >>>> >> >> >> >
> > >>>> >> >> >> >
> > >>>> >> >> >> >
> > >>>> >> >> >> >On Fri, Jun 27, 2014 at 2:13 PM, Virendra Pratap Singh <
> > >>>> >> >> >> >vpsingh@yahoo-inc.com.invalid> wrote:
> > >>>> >> >> >> >
> > >>>> >> >> >> >> Running a mixed 2 broker cluster. Mixed as in one of the
> > >>>> >>broker1
> > >>>> >> >>is
> > >>>> >> >> >> >> running 0.8.0 and broker2 one 0.8.1.1 (from the apache
> > >>>>release
> > >>>> >> >>link.
> > >>>> >> >> >> >> Directly using the tar ball, no local build used).
> > >>>> >> >> >> >>
> > >>>> >> >> >> >> I have set the log.retention.minutes=10. However the
> > >>>>broker
> > >>>>is
> > >>>> >>not
> > >>>> >> >> >> >> honoring the setting. I see its not cleaning the log.dir
> > >>>>at
> > >>>> >>all.
> > >>>> >> >> >> >>
> > >>>> >> >> >> >> However when I set the log.retention.hours=1, then it
> > >>>>starts
> > >>>> >> >>cleaning
> > >>>> >> >> >> >>the
> > >>>> >> >> >> >> log.
> > >>>> >> >> >> >>
> > >>>> >> >> >> >> When I have the log.retention.minutes set in the
> > >>>> >>server.properties
> > >>>> >> >> >>then
> > >>>> >> >> >> >>I
> > >>>> >> >> >> >> see this logged in server.log:
> > >>>> >> >> >> >>
> > >>>> >> >> >> >> Š..
> > >>>> >> >> >> >> [2014-06-27 19:21:06,633] WARN Property
> > >>>>log.retention.minutes
> > >>>> >>is
> > >>>> >> >>not
> > >>>> >> >> >> >>valid
> > >>>> >> >> >> >> (kafka.utils.VerifiableProperties)
> > >>>> >> >> >> >> [2014-06-27 19:21:06,633] WARN Property
> > >>>>log.retention.minutes
> > >>>> >>is
> > >>>> >> >>not
> > >>>> >> >> >> >>valid
> > >>>> >> >> >> >> (kafka.utils.VerifiableProperties)
> > >>>> >> >> >> >> ŠŠ
> > >>>> >> >> >> >>
> > >>>> >> >> >> >>
> > >>>> >> >> >> >> I have set these properties too:
> > >>>> >> >> >> >>
> > >>>> >> >> >> >> log.cleaner.enable=true
> > >>>> >> >> >> >> log.cleanup.policy=delete
> > >>>> >> >> >> >>
> > >>>> >> >> >> >>
> > >>>> >> >> >> >> But I see similar warning logged for these properties
> too.
> > >>>> >> >> >> >>
> > >>>> >> >> >> >> Regards,
> > >>>> >> >> >> >> Virendra
> > >>>> >> >> >> >>
> > >>>> >> >> >> >>
> > >>>> >> >> >> >
> > >>>> >> >> >> >
> > >>>> >> >> >> >--
> > >>>> >> >> >> >-- Guozhang
> > >>>> >> >> >>
> > >>>> >> >> >>
> > >>>> >> >> >
> > >>>> >> >> >
> > >>>> >> >> >--
> > >>>> >> >> >-- Guozhang
> > >>>> >> >>
> > >>>> >> >>
> > >>>> >> >
> > >>>> >> >
> > >>>> >> >--
> > >>>> >> >-- Guozhang
> > >>>> >>
> > >>>> >>
> > >>>>
> > >>>>
> > >>
> > >
> >
> >
>
Re: kafka 0.8.1.1 log.retention.minutes NOT being honored
Posted by François Langelier <f....@gmail.com>.
up?
François Langelier
Étudiant en génie Logiciel - École de Technologie Supérieure
<http://www.etsmtl.ca/>
Capitaine Club Capra <http://capra.etsmtl.ca/>
VP-Communication - CS Games <http://csgames.org> 2014
Jeux de Génie <http://www.jdgets.com/> 2011 à 2014
Magistrat Fraternité du Piranha <http://fraternitedupiranha.com/>
Comité Organisateur Olympiades ÉTS 2012
Compétition Québécoise d'Ingénierie 2012 - Compétition Senior
On Fri, Aug 15, 2014 at 3:32 PM, François Langelier <f....@gmail.com>
wrote:
> UP! Can someone add the log.retention.hours in the documentation?
>
> And is the retention.hours or retention.minutes available for the by-topic
> configuration?
>
>
>
> François Langelier
> Étudiant en génie Logiciel - École de Technologie Supérieure
> <http://www.etsmtl.ca/>
> Capitaine Club Capra <http://capra.etsmtl.ca/>
> VP-Communication - CS Games <http://csgames.org> 2014
> Jeux de Génie <http://www.jdgets.com/> 2011 à 2014
> Magistrat Fraternité du Piranha <http://fraternitedupiranha.com/>
> Comité Organisateur Olympiades ÉTS 2012
> Compétition Québécoise d'Ingénierie 2012 - Compétition Senior
>
>
> On Thu, Jul 10, 2014 at 11:16 PM, Virendra Pratap Singh <
> vpsingh@yahoo-inc.com.invalid> wrote:
>
>> No worries, its working. My deployment was flawed and it was running
>> 0.8.0, while I was under the impression that its running 0.8.1.1.
>> After going to proper 0.8.1.1, the parameter started working properly.
>>
>> On 7/10/14, 2:18 PM, "Virendra Pratap Singh" <vp...@yahoo-inc.com>
>> wrote:
>>
>> >Has someone really used : log.retention.minutes in kafka 0.8.1.1.
>> >I have my full cluster running on 0.8.1.1 and the logs data is just not
>> >getting cleaned up.
>> >
>> >And I see this message in kafka server.log
>> >...
>> >[2014-07-10 20:49:43,786] WARN Property log.retention.minutes is not
>> valid
>> >(kafka.utils.VerifiableProperties)
>> >[2014-07-10 20:49:43,786] WARN Property log.retention.minutes is not
>> valid
>> >(kafka.utils.VerifiableProperties)
>> >....
>> >
>> >Virendra
>> >
>> >On 7/10/14, 10:07 AM, "Virendra Pratap Singh"
>> ><vp...@yahoo-inc.com.INVALID> wrote:
>> >
>> >>Thank you for pointing out to JIRA.
>> >>
>> >>On 7/9/14, 6:38 PM, "Jun Rao" <ju...@gmail.com> wrote:
>> >>
>> >>>This is being worked on in
>> >>>https://issues.apache.org/jira/browse/KAFKA-1325
>> >>>
>> >>>Thanks,
>> >>>
>> >>>Jun
>> >>>
>> >>>
>> >>>On Wed, Jul 9, 2014 at 11:42 AM, Virendra Pratap Singh <
>> >>>vpsingh@yahoo-inc.com.invalid> wrote:
>> >>>
>> >>>> Well currently the log rollup is controlled via
>> >>>>
>> >>>> log.roll.hours
>> >>>> Or
>> >>>> log.segment.bytes
>> >>>>
>> >>>> Given that we now have support to log retention in minutes, I guess
>> it
>> >>>> would be apt to have rollup also have capability to be available in
>> >>>> minutes. Whom/where should I ask to have that coded in.
>> >>>>
>> >>>> One a similar note, I would genuinely want all the size and time
>> based
>> >>>> parameters to be defined at bytes and ms level. This would make it
>> >>>>generic
>> >>>> enough to have user choose and define kind of setting they want and
>> >>>>not
>> >>>> require someone to go and change the code to support a new use case.
>> >>>> Having hours/minutes in config will just cause proliferation/bloating
>> >>>>of
>> >>>> config parameters. What is a good place to air this suggestion.
>> >>>>Anyone?
>> >>>>
>> >>>> Virendra
>> >>>>
>> >>>> On 7/9/14, 7:21 AM, "Jun Rao" <ju...@gmail.com> wrote:
>> >>>>
>> >>>> >Actually, Kafka only removes old segments. The last (active) segment
>> >>>>is
>> >>>> >never removed. So, f you want to have a 10 min retention, you need
>> to
>> >>>> >configure log rolling such that log segments are rolled at least
>> >>>>every
>> >>>>10
>> >>>> >mins.
>> >>>> >
>> >>>> >Thanks,
>> >>>> >
>> >>>> >Jun
>> >>>> >
>> >>>> >
>> >>>> >On Tue, Jul 8, 2014 at 10:04 PM, Virendra Pratap Singh <
>> >>>> >vpsingh@yahoo-inc.com.invalid> wrote:
>> >>>> >
>> >>>> >> That's correct. The server where in I was running 0.8.1.1 was not
>> >>>> >>honoring
>> >>>> >> this parameter, despite the fact it was set in it
>> >>>>server.properties.
>> >>>> >> Not sure if this fact would play any role, the server which was
>> >>>>running
>> >>>> >> 0.8.0 was the leader for all the topics and partition in my setup.
>> >>>>And
>> >>>> >>the
>> >>>> >> second server running 0.8.1.1 has all the replicas (follower).
>> >>>> >>
>> >>>> >> Virendra
>> >>>> >>
>> >>>> >> On 7/8/14, 12:54 PM, "Guozhang Wang" <wa...@gmail.com> wrote:
>> >>>> >>
>> >>>> >> >Server properties should affect on only the local instance
>> >>>>separately.
>> >>>> >>Are
>> >>>> >> >you saying the property is not honored even on the 0.8.1
>> machines?
>> >>>> >> >
>> >>>> >> >Guozhang
>> >>>> >> >
>> >>>> >> >On Mon, Jul 7, 2014 at 3:55 PM, Virendra Pratap Singh <
>> >>>> >> >vpsingh@yahoo-inc.com.invalid> wrote:
>> >>>> >> >
>> >>>> >> >> By setting this property
>> >>>> >> >> log.retention.mins=10
>> >>>> >> >> in the server.properties file, which is passed as argument when
>> >>>> >>starting
>> >>>> >> >> the broker.
>> >>>> >> >>
>> >>>> >> >> Virendra
>> >>>> >> >>
>> >>>> >> >> On 7/7/14, 3:31 PM, "Guozhang Wang" <wa...@gmail.com>
>> wrote:
>> >>>> >> >>
>> >>>> >> >> >How do you set the retention.minutes property? Is it through
>> >>>> >>zk-based
>> >>>> >> >> >topics tool?
>> >>>> >> >> >
>> >>>> >> >> >Guozhang
>> >>>> >> >> >
>> >>>> >> >> >
>> >>>> >> >> >On Mon, Jul 7, 2014 at 3:07 PM, Virendra Pratap Singh <
>> >>>> >> >> >vpsingh@yahoo-inc.com.invalid> wrote:
>> >>>> >> >> >
>> >>>> >> >> >> I am running a mixed cluster as I mentioned earlier. 1
>> broker
>> >>>> >>0.8.0
>> >>>> >> >>and
>> >>>> >> >> >> the other 0.8.1.1. Should the retention of topics for
>> >>>>partitions
>> >>>> >> >> >> owned/replicated by the broker running 0.8.1.1 not enforce
>> >>>>the
>> >>>> >>server
>> >>>> >> >> >> properties settings as defined for that server.
>> >>>> >> >> >>
>> >>>> >> >> >> So this brings an interesting question, in case of
>> >>>>heterogeneous
>> >>>> >> >> >> environment (as is in my case, which system parameters will
>> >>>>take
>> >>>> >> >> >> preference/precedence).
>> >>>> >> >> >>
>> >>>> >> >> >> Virendra
>> >>>> >> >> >>
>> >>>> >> >> >> On 6/30/14, 9:19 AM, "Guozhang Wang" <wa...@gmail.com>
>> >>>>wrote:
>> >>>> >> >> >>
>> >>>> >> >> >> >The retention.minute property is only introduced in 0.8.1:
>> >>>> >> >> >> >
>> >>>> >> >> >> >https://issues.apache.org/jira/browse/KAFKA-918
>> >>>> >> >> >> >
>> >>>> >> >> >> >if you are running 0.8.0 then it will not be recognized.
>> >>>> >> >> >> >
>> >>>> >> >> >> >Guozhang
>> >>>> >> >> >> >
>> >>>> >> >> >> >
>> >>>> >> >> >> >
>> >>>> >> >> >> >On Fri, Jun 27, 2014 at 2:13 PM, Virendra Pratap Singh <
>> >>>> >> >> >> >vpsingh@yahoo-inc.com.invalid> wrote:
>> >>>> >> >> >> >
>> >>>> >> >> >> >> Running a mixed 2 broker cluster. Mixed as in one of the
>> >>>> >>broker1
>> >>>> >> >>is
>> >>>> >> >> >> >> running 0.8.0 and broker2 one 0.8.1.1 (from the apache
>> >>>>release
>> >>>> >> >>link.
>> >>>> >> >> >> >> Directly using the tar ball, no local build used).
>> >>>> >> >> >> >>
>> >>>> >> >> >> >> I have set the log.retention.minutes=10. However the
>> >>>>broker
>> >>>>is
>> >>>> >>not
>> >>>> >> >> >> >> honoring the setting. I see its not cleaning the log.dir
>> >>>>at
>> >>>> >>all.
>> >>>> >> >> >> >>
>> >>>> >> >> >> >> However when I set the log.retention.hours=1, then it
>> >>>>starts
>> >>>> >> >>cleaning
>> >>>> >> >> >> >>the
>> >>>> >> >> >> >> log.
>> >>>> >> >> >> >>
>> >>>> >> >> >> >> When I have the log.retention.minutes set in the
>> >>>> >>server.properties
>> >>>> >> >> >>then
>> >>>> >> >> >> >>I
>> >>>> >> >> >> >> see this logged in server.log:
>> >>>> >> >> >> >>
>> >>>> >> >> >> >> Š..
>> >>>> >> >> >> >> [2014-06-27 19:21:06,633] WARN Property
>> >>>>log.retention.minutes
>> >>>> >>is
>> >>>> >> >>not
>> >>>> >> >> >> >>valid
>> >>>> >> >> >> >> (kafka.utils.VerifiableProperties)
>> >>>> >> >> >> >> [2014-06-27 19:21:06,633] WARN Property
>> >>>>log.retention.minutes
>> >>>> >>is
>> >>>> >> >>not
>> >>>> >> >> >> >>valid
>> >>>> >> >> >> >> (kafka.utils.VerifiableProperties)
>> >>>> >> >> >> >> ŠŠ
>> >>>> >> >> >> >>
>> >>>> >> >> >> >>
>> >>>> >> >> >> >> I have set these properties too:
>> >>>> >> >> >> >>
>> >>>> >> >> >> >> log.cleaner.enable=true
>> >>>> >> >> >> >> log.cleanup.policy=delete
>> >>>> >> >> >> >>
>> >>>> >> >> >> >>
>> >>>> >> >> >> >> But I see similar warning logged for these properties
>> too.
>> >>>> >> >> >> >>
>> >>>> >> >> >> >> Regards,
>> >>>> >> >> >> >> Virendra
>> >>>> >> >> >> >>
>> >>>> >> >> >> >>
>> >>>> >> >> >> >
>> >>>> >> >> >> >
>> >>>> >> >> >> >--
>> >>>> >> >> >> >-- Guozhang
>> >>>> >> >> >>
>> >>>> >> >> >>
>> >>>> >> >> >
>> >>>> >> >> >
>> >>>> >> >> >--
>> >>>> >> >> >-- Guozhang
>> >>>> >> >>
>> >>>> >> >>
>> >>>> >> >
>> >>>> >> >
>> >>>> >> >--
>> >>>> >> >-- Guozhang
>> >>>> >>
>> >>>> >>
>> >>>>
>> >>>>
>> >>
>> >
>>
>>
>