You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Virendra Pratap Singh <vp...@yahoo-inc.com.INVALID> on 2014/06/27 23:13:43 UTC
kafka 0.8.1.1 log.retention.minutes NOT being honored
Running a mixed 2 broker cluster. Mixed as in one of the broker1 is
running 0.8.0 and broker2 one 0.8.1.1 (from the apache release link.
Directly using the tar ball, no local build used).
I have set the log.retention.minutes=10. However the broker is not
honoring the setting. I see its not cleaning the log.dir at all.
However when I set the log.retention.hours=1, then it starts cleaning the
log.
When I have the log.retention.minutes set in the server.properties then I
see this logged in server.log:
Š..
[2014-06-27 19:21:06,633] WARN Property log.retention.minutes is not valid
(kafka.utils.VerifiableProperties)
[2014-06-27 19:21:06,633] WARN Property log.retention.minutes is not valid
(kafka.utils.VerifiableProperties)
ŠŠ
I have set these properties too:
log.cleaner.enable=true
log.cleanup.policy=delete
But I see similar warning logged for these properties too.
Regards,
Virendra
Re: kafka 0.8.1.1 log.retention.minutes NOT being honored
Posted by Jun Rao <ju...@gmail.com>.
log.retention.minutes is only available in 0.8.1.*.
Thanks,
Jun
On Fri, Jun 27, 2014 at 2:13 PM, Virendra Pratap Singh <
vpsingh@yahoo-inc.com.invalid> wrote:
> Running a mixed 2 broker cluster. Mixed as in one of the broker1 is
> running 0.8.0 and broker2 one 0.8.1.1 (from the apache release link.
> Directly using the tar ball, no local build used).
>
> I have set the log.retention.minutes=10. However the broker is not
> honoring the setting. I see its not cleaning the log.dir at all.
>
> However when I set the log.retention.hours=1, then it starts cleaning the
> log.
>
> When I have the log.retention.minutes set in the server.properties then I
> see this logged in server.log:
>
> Š..
> [2014-06-27 19:21:06,633] WARN Property log.retention.minutes is not valid
> (kafka.utils.VerifiableProperties)
> [2014-06-27 19:21:06,633] WARN Property log.retention.minutes is not valid
> (kafka.utils.VerifiableProperties)
> ŠŠ
>
>
> I have set these properties too:
>
> log.cleaner.enable=true
> log.cleanup.policy=delete
>
>
> But I see similar warning logged for these properties too.
>
> Regards,
> Virendra
>
>
Re: kafka 0.8.1.1 log.retention.minutes NOT being honored
Posted by Jun Rao <ju...@gmail.com>.
http://kafka.apache.org/documentation.html#configuration now has the
following entry.
log.retention.{minutes,hours}
Thanks,
Jun
On Thu, Aug 21, 2014 at 7:44 AM, François Langelier <f....@gmail.com>
wrote:
> Thanks Jun,
>
> I don't see changes in the documentation :(
> Just to be sure,
>
> We can either use "*log.retention.hours*" (giving the number of *hours* you
> want the retention to last) or "*log.retention.minutes*" (giving the number
> of *minutes* you want the retention to last) for all the topics (in
> server.properties)
> And we can use "*retention.ms <http://retention.ms>*" (giving the number
> of *milliseconds* you want the retention to last) for a specific topic.
>
> If so, why log.retention.hours isn't written here
> http://kafka.apache.org/documentation.html#configuration
>
> if not, there is a example using "log.retention.hours"
> http://kafka.apache.org/documentation.html#config and also in the example
> of the server.properties on github.
>
>
>
> François Langelier
> Étudiant en génie Logiciel - École de Technologie Supérieure
> <http://www.etsmtl.ca/>
> Capitaine Club Capra <http://capra.etsmtl.ca/>
> VP-Communication - CS Games <http://csgames.org> 2014
> Jeux de Génie <http://www.jdgets.com/> 2011 à 2014
> Magistrat Fraternité du Piranha <http://fraternitedupiranha.com/>
> Comité Organisateur Olympiades ÉTS 2012
> Compétition Québécoise d'Ingénierie 2012 - Compétition Senior
>
>
> On Thu, Aug 21, 2014 at 12:50 AM, Jun Rao <ju...@gmail.com> wrote:
>
> > Added to the documentation. Topic level retentions is called
> retention.ms
> > and can be found in
> > http://kafka.apache.org/documentation.html#configuration
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Fri, Aug 15, 2014 at 12:32 PM, François Langelier <
> > f.langelier@gmail.com>
> > wrote:
> >
> > > UP! Can someone add the log.retention.hours in the documentation?
> > >
> > > And is the retention.hours or retention.minutes available for the
> > by-topic
> > > configuration?
> > >
> > >
> > >
> > > François Langelier
> > > Étudiant en génie Logiciel - École de Technologie Supérieure
> > > <http://www.etsmtl.ca/>
> > > Capitaine Club Capra <http://capra.etsmtl.ca/>
> > > VP-Communication - CS Games <http://csgames.org> 2014
> > > Jeux de Génie <http://www.jdgets.com/> 2011 à 2014
> > > Magistrat Fraternité du Piranha <http://fraternitedupiranha.com/>
> > > Comité Organisateur Olympiades ÉTS 2012
> > > Compétition Québécoise d'Ingénierie 2012 - Compétition Senior
> > >
> > >
> > > On Thu, Jul 10, 2014 at 11:16 PM, Virendra Pratap Singh <
> > > vpsingh@yahoo-inc.com.invalid> wrote:
> > >
> > > > No worries, its working. My deployment was flawed and it was running
> > > > 0.8.0, while I was under the impression that its running 0.8.1.1.
> > > > After going to proper 0.8.1.1, the parameter started working
> properly.
> > > >
> > > > On 7/10/14, 2:18 PM, "Virendra Pratap Singh" <vp...@yahoo-inc.com>
> > > > wrote:
> > > >
> > > > >Has someone really used : log.retention.minutes in kafka 0.8.1.1.
> > > > >I have my full cluster running on 0.8.1.1 and the logs data is just
> > not
> > > > >getting cleaned up.
> > > > >
> > > > >And I see this message in kafka server.log
> > > > >...
> > > > >[2014-07-10 20:49:43,786] WARN Property log.retention.minutes is not
> > > valid
> > > > >(kafka.utils.VerifiableProperties)
> > > > >[2014-07-10 20:49:43,786] WARN Property log.retention.minutes is not
> > > valid
> > > > >(kafka.utils.VerifiableProperties)
> > > > >....
> > > > >
> > > > >Virendra
> > > > >
> > > > >On 7/10/14, 10:07 AM, "Virendra Pratap Singh"
> > > > ><vp...@yahoo-inc.com.INVALID> wrote:
> > > > >
> > > > >>Thank you for pointing out to JIRA.
> > > > >>
> > > > >>On 7/9/14, 6:38 PM, "Jun Rao" <ju...@gmail.com> wrote:
> > > > >>
> > > > >>>This is being worked on in
> > > > >>>https://issues.apache.org/jira/browse/KAFKA-1325
> > > > >>>
> > > > >>>Thanks,
> > > > >>>
> > > > >>>Jun
> > > > >>>
> > > > >>>
> > > > >>>On Wed, Jul 9, 2014 at 11:42 AM, Virendra Pratap Singh <
> > > > >>>vpsingh@yahoo-inc.com.invalid> wrote:
> > > > >>>
> > > > >>>> Well currently the log rollup is controlled via
> > > > >>>>
> > > > >>>> log.roll.hours
> > > > >>>> Or
> > > > >>>> log.segment.bytes
> > > > >>>>
> > > > >>>> Given that we now have support to log retention in minutes, I
> > guess
> > > it
> > > > >>>> would be apt to have rollup also have capability to be available
> > in
> > > > >>>> minutes. Whom/where should I ask to have that coded in.
> > > > >>>>
> > > > >>>> One a similar note, I would genuinely want all the size and time
> > > based
> > > > >>>> parameters to be defined at bytes and ms level. This would make
> it
> > > > >>>>generic
> > > > >>>> enough to have user choose and define kind of setting they want
> > and
> > > > >>>>not
> > > > >>>> require someone to go and change the code to support a new use
> > case.
> > > > >>>> Having hours/minutes in config will just cause
> > > proliferation/bloating
> > > > >>>>of
> > > > >>>> config parameters. What is a good place to air this suggestion.
> > > > >>>>Anyone?
> > > > >>>>
> > > > >>>> Virendra
> > > > >>>>
> > > > >>>> On 7/9/14, 7:21 AM, "Jun Rao" <ju...@gmail.com> wrote:
> > > > >>>>
> > > > >>>> >Actually, Kafka only removes old segments. The last (active)
> > > segment
> > > > >>>>is
> > > > >>>> >never removed. So, f you want to have a 10 min retention, you
> > need
> > > to
> > > > >>>> >configure log rolling such that log segments are rolled at
> least
> > > > >>>>every
> > > > >>>>10
> > > > >>>> >mins.
> > > > >>>> >
> > > > >>>> >Thanks,
> > > > >>>> >
> > > > >>>> >Jun
> > > > >>>> >
> > > > >>>> >
> > > > >>>> >On Tue, Jul 8, 2014 at 10:04 PM, Virendra Pratap Singh <
> > > > >>>> >vpsingh@yahoo-inc.com.invalid> wrote:
> > > > >>>> >
> > > > >>>> >> That's correct. The server where in I was running 0.8.1.1 was
> > not
> > > > >>>> >>honoring
> > > > >>>> >> this parameter, despite the fact it was set in it
> > > > >>>>server.properties.
> > > > >>>> >> Not sure if this fact would play any role, the server which
> was
> > > > >>>>running
> > > > >>>> >> 0.8.0 was the leader for all the topics and partition in my
> > > setup.
> > > > >>>>And
> > > > >>>> >>the
> > > > >>>> >> second server running 0.8.1.1 has all the replicas
> (follower).
> > > > >>>> >>
> > > > >>>> >> Virendra
> > > > >>>> >>
> > > > >>>> >> On 7/8/14, 12:54 PM, "Guozhang Wang" <wa...@gmail.com>
> > wrote:
> > > > >>>> >>
> > > > >>>> >> >Server properties should affect on only the local instance
> > > > >>>>separately.
> > > > >>>> >>Are
> > > > >>>> >> >you saying the property is not honored even on the 0.8.1
> > > machines?
> > > > >>>> >> >
> > > > >>>> >> >Guozhang
> > > > >>>> >> >
> > > > >>>> >> >On Mon, Jul 7, 2014 at 3:55 PM, Virendra Pratap Singh <
> > > > >>>> >> >vpsingh@yahoo-inc.com.invalid> wrote:
> > > > >>>> >> >
> > > > >>>> >> >> By setting this property
> > > > >>>> >> >> log.retention.mins=10
> > > > >>>> >> >> in the server.properties file, which is passed as argument
> > > when
> > > > >>>> >>starting
> > > > >>>> >> >> the broker.
> > > > >>>> >> >>
> > > > >>>> >> >> Virendra
> > > > >>>> >> >>
> > > > >>>> >> >> On 7/7/14, 3:31 PM, "Guozhang Wang" <wa...@gmail.com>
> > > wrote:
> > > > >>>> >> >>
> > > > >>>> >> >> >How do you set the retention.minutes property? Is it
> > through
> > > > >>>> >>zk-based
> > > > >>>> >> >> >topics tool?
> > > > >>>> >> >> >
> > > > >>>> >> >> >Guozhang
> > > > >>>> >> >> >
> > > > >>>> >> >> >
> > > > >>>> >> >> >On Mon, Jul 7, 2014 at 3:07 PM, Virendra Pratap Singh <
> > > > >>>> >> >> >vpsingh@yahoo-inc.com.invalid> wrote:
> > > > >>>> >> >> >
> > > > >>>> >> >> >> I am running a mixed cluster as I mentioned earlier. 1
> > > broker
> > > > >>>> >>0.8.0
> > > > >>>> >> >>and
> > > > >>>> >> >> >> the other 0.8.1.1. Should the retention of topics for
> > > > >>>>partitions
> > > > >>>> >> >> >> owned/replicated by the broker running 0.8.1.1 not
> > enforce
> > > > >>>>the
> > > > >>>> >>server
> > > > >>>> >> >> >> properties settings as defined for that server.
> > > > >>>> >> >> >>
> > > > >>>> >> >> >> So this brings an interesting question, in case of
> > > > >>>>heterogeneous
> > > > >>>> >> >> >> environment (as is in my case, which system parameters
> > will
> > > > >>>>take
> > > > >>>> >> >> >> preference/precedence).
> > > > >>>> >> >> >>
> > > > >>>> >> >> >> Virendra
> > > > >>>> >> >> >>
> > > > >>>> >> >> >> On 6/30/14, 9:19 AM, "Guozhang Wang" <
> wangguoz@gmail.com
> > >
> > > > >>>>wrote:
> > > > >>>> >> >> >>
> > > > >>>> >> >> >> >The retention.minute property is only introduced in
> > 0.8.1:
> > > > >>>> >> >> >> >
> > > > >>>> >> >> >> >https://issues.apache.org/jira/browse/KAFKA-918
> > > > >>>> >> >> >> >
> > > > >>>> >> >> >> >if you are running 0.8.0 then it will not be
> recognized.
> > > > >>>> >> >> >> >
> > > > >>>> >> >> >> >Guozhang
> > > > >>>> >> >> >> >
> > > > >>>> >> >> >> >
> > > > >>>> >> >> >> >
> > > > >>>> >> >> >> >On Fri, Jun 27, 2014 at 2:13 PM, Virendra Pratap
> Singh <
> > > > >>>> >> >> >> >vpsingh@yahoo-inc.com.invalid> wrote:
> > > > >>>> >> >> >> >
> > > > >>>> >> >> >> >> Running a mixed 2 broker cluster. Mixed as in one of
> > the
> > > > >>>> >>broker1
> > > > >>>> >> >>is
> > > > >>>> >> >> >> >> running 0.8.0 and broker2 one 0.8.1.1 (from the
> apache
> > > > >>>>release
> > > > >>>> >> >>link.
> > > > >>>> >> >> >> >> Directly using the tar ball, no local build used).
> > > > >>>> >> >> >> >>
> > > > >>>> >> >> >> >> I have set the log.retention.minutes=10. However the
> > > > >>>>broker
> > > > >>>>is
> > > > >>>> >>not
> > > > >>>> >> >> >> >> honoring the setting. I see its not cleaning the
> > log.dir
> > > > >>>>at
> > > > >>>> >>all.
> > > > >>>> >> >> >> >>
> > > > >>>> >> >> >> >> However when I set the log.retention.hours=1, then
> it
> > > > >>>>starts
> > > > >>>> >> >>cleaning
> > > > >>>> >> >> >> >>the
> > > > >>>> >> >> >> >> log.
> > > > >>>> >> >> >> >>
> > > > >>>> >> >> >> >> When I have the log.retention.minutes set in the
> > > > >>>> >>server.properties
> > > > >>>> >> >> >>then
> > > > >>>> >> >> >> >>I
> > > > >>>> >> >> >> >> see this logged in server.log:
> > > > >>>> >> >> >> >>
> > > > >>>> >> >> >> >> Š..
> > > > >>>> >> >> >> >> [2014-06-27 19:21:06,633] WARN Property
> > > > >>>>log.retention.minutes
> > > > >>>> >>is
> > > > >>>> >> >>not
> > > > >>>> >> >> >> >>valid
> > > > >>>> >> >> >> >> (kafka.utils.VerifiableProperties)
> > > > >>>> >> >> >> >> [2014-06-27 19:21:06,633] WARN Property
> > > > >>>>log.retention.minutes
> > > > >>>> >>is
> > > > >>>> >> >>not
> > > > >>>> >> >> >> >>valid
> > > > >>>> >> >> >> >> (kafka.utils.VerifiableProperties)
> > > > >>>> >> >> >> >> ŠŠ
> > > > >>>> >> >> >> >>
> > > > >>>> >> >> >> >>
> > > > >>>> >> >> >> >> I have set these properties too:
> > > > >>>> >> >> >> >>
> > > > >>>> >> >> >> >> log.cleaner.enable=true
> > > > >>>> >> >> >> >> log.cleanup.policy=delete
> > > > >>>> >> >> >> >>
> > > > >>>> >> >> >> >>
> > > > >>>> >> >> >> >> But I see similar warning logged for these
> properties
> > > too.
> > > > >>>> >> >> >> >>
> > > > >>>> >> >> >> >> Regards,
> > > > >>>> >> >> >> >> Virendra
> > > > >>>> >> >> >> >>
> > > > >>>> >> >> >> >>
> > > > >>>> >> >> >> >
> > > > >>>> >> >> >> >
> > > > >>>> >> >> >> >--
> > > > >>>> >> >> >> >-- Guozhang
> > > > >>>> >> >> >>
> > > > >>>> >> >> >>
> > > > >>>> >> >> >
> > > > >>>> >> >> >
> > > > >>>> >> >> >--
> > > > >>>> >> >> >-- Guozhang
> > > > >>>> >> >>
> > > > >>>> >> >>
> > > > >>>> >> >
> > > > >>>> >> >
> > > > >>>> >> >--
> > > > >>>> >> >-- Guozhang
> > > > >>>> >>
> > > > >>>> >>
> > > > >>>>
> > > > >>>>
> > > > >>
> > > > >
> > > >
> > > >
> > >
> >
>
Re: kafka 0.8.1.1 log.retention.minutes NOT being honored
Posted by François Langelier <f....@gmail.com>.
Thanks Jun,
I don't see changes in the documentation :(
Just to be sure,
We can either use "*log.retention.hours*" (giving the number of *hours* you
want the retention to last) or "*log.retention.minutes*" (giving the number
of *minutes* you want the retention to last) for all the topics (in
server.properties)
And we can use "*retention.ms <http://retention.ms>*" (giving the number
of *milliseconds* you want the retention to last) for a specific topic.
If so, why log.retention.hours isn't written here
http://kafka.apache.org/documentation.html#configuration
if not, there is a example using "log.retention.hours"
http://kafka.apache.org/documentation.html#config and also in the example
of the server.properties on github.
François Langelier
Étudiant en génie Logiciel - École de Technologie Supérieure
<http://www.etsmtl.ca/>
Capitaine Club Capra <http://capra.etsmtl.ca/>
VP-Communication - CS Games <http://csgames.org> 2014
Jeux de Génie <http://www.jdgets.com/> 2011 à 2014
Magistrat Fraternité du Piranha <http://fraternitedupiranha.com/>
Comité Organisateur Olympiades ÉTS 2012
Compétition Québécoise d'Ingénierie 2012 - Compétition Senior
On Thu, Aug 21, 2014 at 12:50 AM, Jun Rao <ju...@gmail.com> wrote:
> Added to the documentation. Topic level retentions is called retention.ms
> and can be found in
> http://kafka.apache.org/documentation.html#configuration
>
> Thanks,
>
> Jun
>
>
> On Fri, Aug 15, 2014 at 12:32 PM, François Langelier <
> f.langelier@gmail.com>
> wrote:
>
> > UP! Can someone add the log.retention.hours in the documentation?
> >
> > And is the retention.hours or retention.minutes available for the
> by-topic
> > configuration?
> >
> >
> >
> > François Langelier
> > Étudiant en génie Logiciel - École de Technologie Supérieure
> > <http://www.etsmtl.ca/>
> > Capitaine Club Capra <http://capra.etsmtl.ca/>
> > VP-Communication - CS Games <http://csgames.org> 2014
> > Jeux de Génie <http://www.jdgets.com/> 2011 à 2014
> > Magistrat Fraternité du Piranha <http://fraternitedupiranha.com/>
> > Comité Organisateur Olympiades ÉTS 2012
> > Compétition Québécoise d'Ingénierie 2012 - Compétition Senior
> >
> >
> > On Thu, Jul 10, 2014 at 11:16 PM, Virendra Pratap Singh <
> > vpsingh@yahoo-inc.com.invalid> wrote:
> >
> > > No worries, its working. My deployment was flawed and it was running
> > > 0.8.0, while I was under the impression that its running 0.8.1.1.
> > > After going to proper 0.8.1.1, the parameter started working properly.
> > >
> > > On 7/10/14, 2:18 PM, "Virendra Pratap Singh" <vp...@yahoo-inc.com>
> > > wrote:
> > >
> > > >Has someone really used : log.retention.minutes in kafka 0.8.1.1.
> > > >I have my full cluster running on 0.8.1.1 and the logs data is just
> not
> > > >getting cleaned up.
> > > >
> > > >And I see this message in kafka server.log
> > > >...
> > > >[2014-07-10 20:49:43,786] WARN Property log.retention.minutes is not
> > valid
> > > >(kafka.utils.VerifiableProperties)
> > > >[2014-07-10 20:49:43,786] WARN Property log.retention.minutes is not
> > valid
> > > >(kafka.utils.VerifiableProperties)
> > > >....
> > > >
> > > >Virendra
> > > >
> > > >On 7/10/14, 10:07 AM, "Virendra Pratap Singh"
> > > ><vp...@yahoo-inc.com.INVALID> wrote:
> > > >
> > > >>Thank you for pointing out to JIRA.
> > > >>
> > > >>On 7/9/14, 6:38 PM, "Jun Rao" <ju...@gmail.com> wrote:
> > > >>
> > > >>>This is being worked on in
> > > >>>https://issues.apache.org/jira/browse/KAFKA-1325
> > > >>>
> > > >>>Thanks,
> > > >>>
> > > >>>Jun
> > > >>>
> > > >>>
> > > >>>On Wed, Jul 9, 2014 at 11:42 AM, Virendra Pratap Singh <
> > > >>>vpsingh@yahoo-inc.com.invalid> wrote:
> > > >>>
> > > >>>> Well currently the log rollup is controlled via
> > > >>>>
> > > >>>> log.roll.hours
> > > >>>> Or
> > > >>>> log.segment.bytes
> > > >>>>
> > > >>>> Given that we now have support to log retention in minutes, I
> guess
> > it
> > > >>>> would be apt to have rollup also have capability to be available
> in
> > > >>>> minutes. Whom/where should I ask to have that coded in.
> > > >>>>
> > > >>>> One a similar note, I would genuinely want all the size and time
> > based
> > > >>>> parameters to be defined at bytes and ms level. This would make it
> > > >>>>generic
> > > >>>> enough to have user choose and define kind of setting they want
> and
> > > >>>>not
> > > >>>> require someone to go and change the code to support a new use
> case.
> > > >>>> Having hours/minutes in config will just cause
> > proliferation/bloating
> > > >>>>of
> > > >>>> config parameters. What is a good place to air this suggestion.
> > > >>>>Anyone?
> > > >>>>
> > > >>>> Virendra
> > > >>>>
> > > >>>> On 7/9/14, 7:21 AM, "Jun Rao" <ju...@gmail.com> wrote:
> > > >>>>
> > > >>>> >Actually, Kafka only removes old segments. The last (active)
> > segment
> > > >>>>is
> > > >>>> >never removed. So, f you want to have a 10 min retention, you
> need
> > to
> > > >>>> >configure log rolling such that log segments are rolled at least
> > > >>>>every
> > > >>>>10
> > > >>>> >mins.
> > > >>>> >
> > > >>>> >Thanks,
> > > >>>> >
> > > >>>> >Jun
> > > >>>> >
> > > >>>> >
> > > >>>> >On Tue, Jul 8, 2014 at 10:04 PM, Virendra Pratap Singh <
> > > >>>> >vpsingh@yahoo-inc.com.invalid> wrote:
> > > >>>> >
> > > >>>> >> That's correct. The server where in I was running 0.8.1.1 was
> not
> > > >>>> >>honoring
> > > >>>> >> this parameter, despite the fact it was set in it
> > > >>>>server.properties.
> > > >>>> >> Not sure if this fact would play any role, the server which was
> > > >>>>running
> > > >>>> >> 0.8.0 was the leader for all the topics and partition in my
> > setup.
> > > >>>>And
> > > >>>> >>the
> > > >>>> >> second server running 0.8.1.1 has all the replicas (follower).
> > > >>>> >>
> > > >>>> >> Virendra
> > > >>>> >>
> > > >>>> >> On 7/8/14, 12:54 PM, "Guozhang Wang" <wa...@gmail.com>
> wrote:
> > > >>>> >>
> > > >>>> >> >Server properties should affect on only the local instance
> > > >>>>separately.
> > > >>>> >>Are
> > > >>>> >> >you saying the property is not honored even on the 0.8.1
> > machines?
> > > >>>> >> >
> > > >>>> >> >Guozhang
> > > >>>> >> >
> > > >>>> >> >On Mon, Jul 7, 2014 at 3:55 PM, Virendra Pratap Singh <
> > > >>>> >> >vpsingh@yahoo-inc.com.invalid> wrote:
> > > >>>> >> >
> > > >>>> >> >> By setting this property
> > > >>>> >> >> log.retention.mins=10
> > > >>>> >> >> in the server.properties file, which is passed as argument
> > when
> > > >>>> >>starting
> > > >>>> >> >> the broker.
> > > >>>> >> >>
> > > >>>> >> >> Virendra
> > > >>>> >> >>
> > > >>>> >> >> On 7/7/14, 3:31 PM, "Guozhang Wang" <wa...@gmail.com>
> > wrote:
> > > >>>> >> >>
> > > >>>> >> >> >How do you set the retention.minutes property? Is it
> through
> > > >>>> >>zk-based
> > > >>>> >> >> >topics tool?
> > > >>>> >> >> >
> > > >>>> >> >> >Guozhang
> > > >>>> >> >> >
> > > >>>> >> >> >
> > > >>>> >> >> >On Mon, Jul 7, 2014 at 3:07 PM, Virendra Pratap Singh <
> > > >>>> >> >> >vpsingh@yahoo-inc.com.invalid> wrote:
> > > >>>> >> >> >
> > > >>>> >> >> >> I am running a mixed cluster as I mentioned earlier. 1
> > broker
> > > >>>> >>0.8.0
> > > >>>> >> >>and
> > > >>>> >> >> >> the other 0.8.1.1. Should the retention of topics for
> > > >>>>partitions
> > > >>>> >> >> >> owned/replicated by the broker running 0.8.1.1 not
> enforce
> > > >>>>the
> > > >>>> >>server
> > > >>>> >> >> >> properties settings as defined for that server.
> > > >>>> >> >> >>
> > > >>>> >> >> >> So this brings an interesting question, in case of
> > > >>>>heterogeneous
> > > >>>> >> >> >> environment (as is in my case, which system parameters
> will
> > > >>>>take
> > > >>>> >> >> >> preference/precedence).
> > > >>>> >> >> >>
> > > >>>> >> >> >> Virendra
> > > >>>> >> >> >>
> > > >>>> >> >> >> On 6/30/14, 9:19 AM, "Guozhang Wang" <wangguoz@gmail.com
> >
> > > >>>>wrote:
> > > >>>> >> >> >>
> > > >>>> >> >> >> >The retention.minute property is only introduced in
> 0.8.1:
> > > >>>> >> >> >> >
> > > >>>> >> >> >> >https://issues.apache.org/jira/browse/KAFKA-918
> > > >>>> >> >> >> >
> > > >>>> >> >> >> >if you are running 0.8.0 then it will not be recognized.
> > > >>>> >> >> >> >
> > > >>>> >> >> >> >Guozhang
> > > >>>> >> >> >> >
> > > >>>> >> >> >> >
> > > >>>> >> >> >> >
> > > >>>> >> >> >> >On Fri, Jun 27, 2014 at 2:13 PM, Virendra Pratap Singh <
> > > >>>> >> >> >> >vpsingh@yahoo-inc.com.invalid> wrote:
> > > >>>> >> >> >> >
> > > >>>> >> >> >> >> Running a mixed 2 broker cluster. Mixed as in one of
> the
> > > >>>> >>broker1
> > > >>>> >> >>is
> > > >>>> >> >> >> >> running 0.8.0 and broker2 one 0.8.1.1 (from the apache
> > > >>>>release
> > > >>>> >> >>link.
> > > >>>> >> >> >> >> Directly using the tar ball, no local build used).
> > > >>>> >> >> >> >>
> > > >>>> >> >> >> >> I have set the log.retention.minutes=10. However the
> > > >>>>broker
> > > >>>>is
> > > >>>> >>not
> > > >>>> >> >> >> >> honoring the setting. I see its not cleaning the
> log.dir
> > > >>>>at
> > > >>>> >>all.
> > > >>>> >> >> >> >>
> > > >>>> >> >> >> >> However when I set the log.retention.hours=1, then it
> > > >>>>starts
> > > >>>> >> >>cleaning
> > > >>>> >> >> >> >>the
> > > >>>> >> >> >> >> log.
> > > >>>> >> >> >> >>
> > > >>>> >> >> >> >> When I have the log.retention.minutes set in the
> > > >>>> >>server.properties
> > > >>>> >> >> >>then
> > > >>>> >> >> >> >>I
> > > >>>> >> >> >> >> see this logged in server.log:
> > > >>>> >> >> >> >>
> > > >>>> >> >> >> >> Š..
> > > >>>> >> >> >> >> [2014-06-27 19:21:06,633] WARN Property
> > > >>>>log.retention.minutes
> > > >>>> >>is
> > > >>>> >> >>not
> > > >>>> >> >> >> >>valid
> > > >>>> >> >> >> >> (kafka.utils.VerifiableProperties)
> > > >>>> >> >> >> >> [2014-06-27 19:21:06,633] WARN Property
> > > >>>>log.retention.minutes
> > > >>>> >>is
> > > >>>> >> >>not
> > > >>>> >> >> >> >>valid
> > > >>>> >> >> >> >> (kafka.utils.VerifiableProperties)
> > > >>>> >> >> >> >> ŠŠ
> > > >>>> >> >> >> >>
> > > >>>> >> >> >> >>
> > > >>>> >> >> >> >> I have set these properties too:
> > > >>>> >> >> >> >>
> > > >>>> >> >> >> >> log.cleaner.enable=true
> > > >>>> >> >> >> >> log.cleanup.policy=delete
> > > >>>> >> >> >> >>
> > > >>>> >> >> >> >>
> > > >>>> >> >> >> >> But I see similar warning logged for these properties
> > too.
> > > >>>> >> >> >> >>
> > > >>>> >> >> >> >> Regards,
> > > >>>> >> >> >> >> Virendra
> > > >>>> >> >> >> >>
> > > >>>> >> >> >> >>
> > > >>>> >> >> >> >
> > > >>>> >> >> >> >
> > > >>>> >> >> >> >--
> > > >>>> >> >> >> >-- Guozhang
> > > >>>> >> >> >>
> > > >>>> >> >> >>
> > > >>>> >> >> >
> > > >>>> >> >> >
> > > >>>> >> >> >--
> > > >>>> >> >> >-- Guozhang
> > > >>>> >> >>
> > > >>>> >> >>
> > > >>>> >> >
> > > >>>> >> >
> > > >>>> >> >--
> > > >>>> >> >-- Guozhang
> > > >>>> >>
> > > >>>> >>
> > > >>>>
> > > >>>>
> > > >>
> > > >
> > >
> > >
> >
>
Re: kafka 0.8.1.1 log.retention.minutes NOT being honored
Posted by Jun Rao <ju...@gmail.com>.
Added to the documentation. Topic level retentions is called retention.ms
and can be found in http://kafka.apache.org/documentation.html#configuration
Thanks,
Jun
On Fri, Aug 15, 2014 at 12:32 PM, François Langelier <f....@gmail.com>
wrote:
> UP! Can someone add the log.retention.hours in the documentation?
>
> And is the retention.hours or retention.minutes available for the by-topic
> configuration?
>
>
>
> François Langelier
> Étudiant en génie Logiciel - École de Technologie Supérieure
> <http://www.etsmtl.ca/>
> Capitaine Club Capra <http://capra.etsmtl.ca/>
> VP-Communication - CS Games <http://csgames.org> 2014
> Jeux de Génie <http://www.jdgets.com/> 2011 à 2014
> Magistrat Fraternité du Piranha <http://fraternitedupiranha.com/>
> Comité Organisateur Olympiades ÉTS 2012
> Compétition Québécoise d'Ingénierie 2012 - Compétition Senior
>
>
> On Thu, Jul 10, 2014 at 11:16 PM, Virendra Pratap Singh <
> vpsingh@yahoo-inc.com.invalid> wrote:
>
> > No worries, its working. My deployment was flawed and it was running
> > 0.8.0, while I was under the impression that its running 0.8.1.1.
> > After going to proper 0.8.1.1, the parameter started working properly.
> >
> > On 7/10/14, 2:18 PM, "Virendra Pratap Singh" <vp...@yahoo-inc.com>
> > wrote:
> >
> > >Has someone really used : log.retention.minutes in kafka 0.8.1.1.
> > >I have my full cluster running on 0.8.1.1 and the logs data is just not
> > >getting cleaned up.
> > >
> > >And I see this message in kafka server.log
> > >...
> > >[2014-07-10 20:49:43,786] WARN Property log.retention.minutes is not
> valid
> > >(kafka.utils.VerifiableProperties)
> > >[2014-07-10 20:49:43,786] WARN Property log.retention.minutes is not
> valid
> > >(kafka.utils.VerifiableProperties)
> > >....
> > >
> > >Virendra
> > >
> > >On 7/10/14, 10:07 AM, "Virendra Pratap Singh"
> > ><vp...@yahoo-inc.com.INVALID> wrote:
> > >
> > >>Thank you for pointing out to JIRA.
> > >>
> > >>On 7/9/14, 6:38 PM, "Jun Rao" <ju...@gmail.com> wrote:
> > >>
> > >>>This is being worked on in
> > >>>https://issues.apache.org/jira/browse/KAFKA-1325
> > >>>
> > >>>Thanks,
> > >>>
> > >>>Jun
> > >>>
> > >>>
> > >>>On Wed, Jul 9, 2014 at 11:42 AM, Virendra Pratap Singh <
> > >>>vpsingh@yahoo-inc.com.invalid> wrote:
> > >>>
> > >>>> Well currently the log rollup is controlled via
> > >>>>
> > >>>> log.roll.hours
> > >>>> Or
> > >>>> log.segment.bytes
> > >>>>
> > >>>> Given that we now have support to log retention in minutes, I guess
> it
> > >>>> would be apt to have rollup also have capability to be available in
> > >>>> minutes. Whom/where should I ask to have that coded in.
> > >>>>
> > >>>> One a similar note, I would genuinely want all the size and time
> based
> > >>>> parameters to be defined at bytes and ms level. This would make it
> > >>>>generic
> > >>>> enough to have user choose and define kind of setting they want and
> > >>>>not
> > >>>> require someone to go and change the code to support a new use case.
> > >>>> Having hours/minutes in config will just cause
> proliferation/bloating
> > >>>>of
> > >>>> config parameters. What is a good place to air this suggestion.
> > >>>>Anyone?
> > >>>>
> > >>>> Virendra
> > >>>>
> > >>>> On 7/9/14, 7:21 AM, "Jun Rao" <ju...@gmail.com> wrote:
> > >>>>
> > >>>> >Actually, Kafka only removes old segments. The last (active)
> segment
> > >>>>is
> > >>>> >never removed. So, f you want to have a 10 min retention, you need
> to
> > >>>> >configure log rolling such that log segments are rolled at least
> > >>>>every
> > >>>>10
> > >>>> >mins.
> > >>>> >
> > >>>> >Thanks,
> > >>>> >
> > >>>> >Jun
> > >>>> >
> > >>>> >
> > >>>> >On Tue, Jul 8, 2014 at 10:04 PM, Virendra Pratap Singh <
> > >>>> >vpsingh@yahoo-inc.com.invalid> wrote:
> > >>>> >
> > >>>> >> That's correct. The server where in I was running 0.8.1.1 was not
> > >>>> >>honoring
> > >>>> >> this parameter, despite the fact it was set in it
> > >>>>server.properties.
> > >>>> >> Not sure if this fact would play any role, the server which was
> > >>>>running
> > >>>> >> 0.8.0 was the leader for all the topics and partition in my
> setup.
> > >>>>And
> > >>>> >>the
> > >>>> >> second server running 0.8.1.1 has all the replicas (follower).
> > >>>> >>
> > >>>> >> Virendra
> > >>>> >>
> > >>>> >> On 7/8/14, 12:54 PM, "Guozhang Wang" <wa...@gmail.com> wrote:
> > >>>> >>
> > >>>> >> >Server properties should affect on only the local instance
> > >>>>separately.
> > >>>> >>Are
> > >>>> >> >you saying the property is not honored even on the 0.8.1
> machines?
> > >>>> >> >
> > >>>> >> >Guozhang
> > >>>> >> >
> > >>>> >> >On Mon, Jul 7, 2014 at 3:55 PM, Virendra Pratap Singh <
> > >>>> >> >vpsingh@yahoo-inc.com.invalid> wrote:
> > >>>> >> >
> > >>>> >> >> By setting this property
> > >>>> >> >> log.retention.mins=10
> > >>>> >> >> in the server.properties file, which is passed as argument
> when
> > >>>> >>starting
> > >>>> >> >> the broker.
> > >>>> >> >>
> > >>>> >> >> Virendra
> > >>>> >> >>
> > >>>> >> >> On 7/7/14, 3:31 PM, "Guozhang Wang" <wa...@gmail.com>
> wrote:
> > >>>> >> >>
> > >>>> >> >> >How do you set the retention.minutes property? Is it through
> > >>>> >>zk-based
> > >>>> >> >> >topics tool?
> > >>>> >> >> >
> > >>>> >> >> >Guozhang
> > >>>> >> >> >
> > >>>> >> >> >
> > >>>> >> >> >On Mon, Jul 7, 2014 at 3:07 PM, Virendra Pratap Singh <
> > >>>> >> >> >vpsingh@yahoo-inc.com.invalid> wrote:
> > >>>> >> >> >
> > >>>> >> >> >> I am running a mixed cluster as I mentioned earlier. 1
> broker
> > >>>> >>0.8.0
> > >>>> >> >>and
> > >>>> >> >> >> the other 0.8.1.1. Should the retention of topics for
> > >>>>partitions
> > >>>> >> >> >> owned/replicated by the broker running 0.8.1.1 not enforce
> > >>>>the
> > >>>> >>server
> > >>>> >> >> >> properties settings as defined for that server.
> > >>>> >> >> >>
> > >>>> >> >> >> So this brings an interesting question, in case of
> > >>>>heterogeneous
> > >>>> >> >> >> environment (as is in my case, which system parameters will
> > >>>>take
> > >>>> >> >> >> preference/precedence).
> > >>>> >> >> >>
> > >>>> >> >> >> Virendra
> > >>>> >> >> >>
> > >>>> >> >> >> On 6/30/14, 9:19 AM, "Guozhang Wang" <wa...@gmail.com>
> > >>>>wrote:
> > >>>> >> >> >>
> > >>>> >> >> >> >The retention.minute property is only introduced in 0.8.1:
> > >>>> >> >> >> >
> > >>>> >> >> >> >https://issues.apache.org/jira/browse/KAFKA-918
> > >>>> >> >> >> >
> > >>>> >> >> >> >if you are running 0.8.0 then it will not be recognized.
> > >>>> >> >> >> >
> > >>>> >> >> >> >Guozhang
> > >>>> >> >> >> >
> > >>>> >> >> >> >
> > >>>> >> >> >> >
> > >>>> >> >> >> >On Fri, Jun 27, 2014 at 2:13 PM, Virendra Pratap Singh <
> > >>>> >> >> >> >vpsingh@yahoo-inc.com.invalid> wrote:
> > >>>> >> >> >> >
> > >>>> >> >> >> >> Running a mixed 2 broker cluster. Mixed as in one of the
> > >>>> >>broker1
> > >>>> >> >>is
> > >>>> >> >> >> >> running 0.8.0 and broker2 one 0.8.1.1 (from the apache
> > >>>>release
> > >>>> >> >>link.
> > >>>> >> >> >> >> Directly using the tar ball, no local build used).
> > >>>> >> >> >> >>
> > >>>> >> >> >> >> I have set the log.retention.minutes=10. However the
> > >>>>broker
> > >>>>is
> > >>>> >>not
> > >>>> >> >> >> >> honoring the setting. I see its not cleaning the log.dir
> > >>>>at
> > >>>> >>all.
> > >>>> >> >> >> >>
> > >>>> >> >> >> >> However when I set the log.retention.hours=1, then it
> > >>>>starts
> > >>>> >> >>cleaning
> > >>>> >> >> >> >>the
> > >>>> >> >> >> >> log.
> > >>>> >> >> >> >>
> > >>>> >> >> >> >> When I have the log.retention.minutes set in the
> > >>>> >>server.properties
> > >>>> >> >> >>then
> > >>>> >> >> >> >>I
> > >>>> >> >> >> >> see this logged in server.log:
> > >>>> >> >> >> >>
> > >>>> >> >> >> >> Š..
> > >>>> >> >> >> >> [2014-06-27 19:21:06,633] WARN Property
> > >>>>log.retention.minutes
> > >>>> >>is
> > >>>> >> >>not
> > >>>> >> >> >> >>valid
> > >>>> >> >> >> >> (kafka.utils.VerifiableProperties)
> > >>>> >> >> >> >> [2014-06-27 19:21:06,633] WARN Property
> > >>>>log.retention.minutes
> > >>>> >>is
> > >>>> >> >>not
> > >>>> >> >> >> >>valid
> > >>>> >> >> >> >> (kafka.utils.VerifiableProperties)
> > >>>> >> >> >> >> ŠŠ
> > >>>> >> >> >> >>
> > >>>> >> >> >> >>
> > >>>> >> >> >> >> I have set these properties too:
> > >>>> >> >> >> >>
> > >>>> >> >> >> >> log.cleaner.enable=true
> > >>>> >> >> >> >> log.cleanup.policy=delete
> > >>>> >> >> >> >>
> > >>>> >> >> >> >>
> > >>>> >> >> >> >> But I see similar warning logged for these properties
> too.
> > >>>> >> >> >> >>
> > >>>> >> >> >> >> Regards,
> > >>>> >> >> >> >> Virendra
> > >>>> >> >> >> >>
> > >>>> >> >> >> >>
> > >>>> >> >> >> >
> > >>>> >> >> >> >
> > >>>> >> >> >> >--
> > >>>> >> >> >> >-- Guozhang
> > >>>> >> >> >>
> > >>>> >> >> >>
> > >>>> >> >> >
> > >>>> >> >> >
> > >>>> >> >> >--
> > >>>> >> >> >-- Guozhang
> > >>>> >> >>
> > >>>> >> >>
> > >>>> >> >
> > >>>> >> >
> > >>>> >> >--
> > >>>> >> >-- Guozhang
> > >>>> >>
> > >>>> >>
> > >>>>
> > >>>>
> > >>
> > >
> >
> >
>
Re: kafka 0.8.1.1 log.retention.minutes NOT being honored
Posted by François Langelier <f....@gmail.com>.
up?
François Langelier
Étudiant en génie Logiciel - École de Technologie Supérieure
<http://www.etsmtl.ca/>
Capitaine Club Capra <http://capra.etsmtl.ca/>
VP-Communication - CS Games <http://csgames.org> 2014
Jeux de Génie <http://www.jdgets.com/> 2011 à 2014
Magistrat Fraternité du Piranha <http://fraternitedupiranha.com/>
Comité Organisateur Olympiades ÉTS 2012
Compétition Québécoise d'Ingénierie 2012 - Compétition Senior
On Fri, Aug 15, 2014 at 3:32 PM, François Langelier <f....@gmail.com>
wrote:
> UP! Can someone add the log.retention.hours in the documentation?
>
> And is the retention.hours or retention.minutes available for the by-topic
> configuration?
>
>
>
> François Langelier
> Étudiant en génie Logiciel - École de Technologie Supérieure
> <http://www.etsmtl.ca/>
> Capitaine Club Capra <http://capra.etsmtl.ca/>
> VP-Communication - CS Games <http://csgames.org> 2014
> Jeux de Génie <http://www.jdgets.com/> 2011 à 2014
> Magistrat Fraternité du Piranha <http://fraternitedupiranha.com/>
> Comité Organisateur Olympiades ÉTS 2012
> Compétition Québécoise d'Ingénierie 2012 - Compétition Senior
>
>
> On Thu, Jul 10, 2014 at 11:16 PM, Virendra Pratap Singh <
> vpsingh@yahoo-inc.com.invalid> wrote:
>
>> No worries, its working. My deployment was flawed and it was running
>> 0.8.0, while I was under the impression that its running 0.8.1.1.
>> After going to proper 0.8.1.1, the parameter started working properly.
>>
>> On 7/10/14, 2:18 PM, "Virendra Pratap Singh" <vp...@yahoo-inc.com>
>> wrote:
>>
>> >Has someone really used : log.retention.minutes in kafka 0.8.1.1.
>> >I have my full cluster running on 0.8.1.1 and the logs data is just not
>> >getting cleaned up.
>> >
>> >And I see this message in kafka server.log
>> >...
>> >[2014-07-10 20:49:43,786] WARN Property log.retention.minutes is not
>> valid
>> >(kafka.utils.VerifiableProperties)
>> >[2014-07-10 20:49:43,786] WARN Property log.retention.minutes is not
>> valid
>> >(kafka.utils.VerifiableProperties)
>> >....
>> >
>> >Virendra
>> >
>> >On 7/10/14, 10:07 AM, "Virendra Pratap Singh"
>> ><vp...@yahoo-inc.com.INVALID> wrote:
>> >
>> >>Thank you for pointing out to JIRA.
>> >>
>> >>On 7/9/14, 6:38 PM, "Jun Rao" <ju...@gmail.com> wrote:
>> >>
>> >>>This is being worked on in
>> >>>https://issues.apache.org/jira/browse/KAFKA-1325
>> >>>
>> >>>Thanks,
>> >>>
>> >>>Jun
>> >>>
>> >>>
>> >>>On Wed, Jul 9, 2014 at 11:42 AM, Virendra Pratap Singh <
>> >>>vpsingh@yahoo-inc.com.invalid> wrote:
>> >>>
>> >>>> Well currently the log rollup is controlled via
>> >>>>
>> >>>> log.roll.hours
>> >>>> Or
>> >>>> log.segment.bytes
>> >>>>
>> >>>> Given that we now have support to log retention in minutes, I guess
>> it
>> >>>> would be apt to have rollup also have capability to be available in
>> >>>> minutes. Whom/where should I ask to have that coded in.
>> >>>>
>> >>>> One a similar note, I would genuinely want all the size and time
>> based
>> >>>> parameters to be defined at bytes and ms level. This would make it
>> >>>>generic
>> >>>> enough to have user choose and define kind of setting they want and
>> >>>>not
>> >>>> require someone to go and change the code to support a new use case.
>> >>>> Having hours/minutes in config will just cause proliferation/bloating
>> >>>>of
>> >>>> config parameters. What is a good place to air this suggestion.
>> >>>>Anyone?
>> >>>>
>> >>>> Virendra
>> >>>>
>> >>>> On 7/9/14, 7:21 AM, "Jun Rao" <ju...@gmail.com> wrote:
>> >>>>
>> >>>> >Actually, Kafka only removes old segments. The last (active) segment
>> >>>>is
>> >>>> >never removed. So, f you want to have a 10 min retention, you need
>> to
>> >>>> >configure log rolling such that log segments are rolled at least
>> >>>>every
>> >>>>10
>> >>>> >mins.
>> >>>> >
>> >>>> >Thanks,
>> >>>> >
>> >>>> >Jun
>> >>>> >
>> >>>> >
>> >>>> >On Tue, Jul 8, 2014 at 10:04 PM, Virendra Pratap Singh <
>> >>>> >vpsingh@yahoo-inc.com.invalid> wrote:
>> >>>> >
>> >>>> >> That's correct. The server where in I was running 0.8.1.1 was not
>> >>>> >>honoring
>> >>>> >> this parameter, despite the fact it was set in it
>> >>>>server.properties.
>> >>>> >> Not sure if this fact would play any role, the server which was
>> >>>>running
>> >>>> >> 0.8.0 was the leader for all the topics and partition in my setup.
>> >>>>And
>> >>>> >>the
>> >>>> >> second server running 0.8.1.1 has all the replicas (follower).
>> >>>> >>
>> >>>> >> Virendra
>> >>>> >>
>> >>>> >> On 7/8/14, 12:54 PM, "Guozhang Wang" <wa...@gmail.com> wrote:
>> >>>> >>
>> >>>> >> >Server properties should affect on only the local instance
>> >>>>separately.
>> >>>> >>Are
>> >>>> >> >you saying the property is not honored even on the 0.8.1
>> machines?
>> >>>> >> >
>> >>>> >> >Guozhang
>> >>>> >> >
>> >>>> >> >On Mon, Jul 7, 2014 at 3:55 PM, Virendra Pratap Singh <
>> >>>> >> >vpsingh@yahoo-inc.com.invalid> wrote:
>> >>>> >> >
>> >>>> >> >> By setting this property
>> >>>> >> >> log.retention.mins=10
>> >>>> >> >> in the server.properties file, which is passed as argument when
>> >>>> >>starting
>> >>>> >> >> the broker.
>> >>>> >> >>
>> >>>> >> >> Virendra
>> >>>> >> >>
>> >>>> >> >> On 7/7/14, 3:31 PM, "Guozhang Wang" <wa...@gmail.com>
>> wrote:
>> >>>> >> >>
>> >>>> >> >> >How do you set the retention.minutes property? Is it through
>> >>>> >>zk-based
>> >>>> >> >> >topics tool?
>> >>>> >> >> >
>> >>>> >> >> >Guozhang
>> >>>> >> >> >
>> >>>> >> >> >
>> >>>> >> >> >On Mon, Jul 7, 2014 at 3:07 PM, Virendra Pratap Singh <
>> >>>> >> >> >vpsingh@yahoo-inc.com.invalid> wrote:
>> >>>> >> >> >
>> >>>> >> >> >> I am running a mixed cluster as I mentioned earlier. 1
>> broker
>> >>>> >>0.8.0
>> >>>> >> >>and
>> >>>> >> >> >> the other 0.8.1.1. Should the retention of topics for
>> >>>>partitions
>> >>>> >> >> >> owned/replicated by the broker running 0.8.1.1 not enforce
>> >>>>the
>> >>>> >>server
>> >>>> >> >> >> properties settings as defined for that server.
>> >>>> >> >> >>
>> >>>> >> >> >> So this brings an interesting question, in case of
>> >>>>heterogeneous
>> >>>> >> >> >> environment (as is in my case, which system parameters will
>> >>>>take
>> >>>> >> >> >> preference/precedence).
>> >>>> >> >> >>
>> >>>> >> >> >> Virendra
>> >>>> >> >> >>
>> >>>> >> >> >> On 6/30/14, 9:19 AM, "Guozhang Wang" <wa...@gmail.com>
>> >>>>wrote:
>> >>>> >> >> >>
>> >>>> >> >> >> >The retention.minute property is only introduced in 0.8.1:
>> >>>> >> >> >> >
>> >>>> >> >> >> >https://issues.apache.org/jira/browse/KAFKA-918
>> >>>> >> >> >> >
>> >>>> >> >> >> >if you are running 0.8.0 then it will not be recognized.
>> >>>> >> >> >> >
>> >>>> >> >> >> >Guozhang
>> >>>> >> >> >> >
>> >>>> >> >> >> >
>> >>>> >> >> >> >
>> >>>> >> >> >> >On Fri, Jun 27, 2014 at 2:13 PM, Virendra Pratap Singh <
>> >>>> >> >> >> >vpsingh@yahoo-inc.com.invalid> wrote:
>> >>>> >> >> >> >
>> >>>> >> >> >> >> Running a mixed 2 broker cluster. Mixed as in one of the
>> >>>> >>broker1
>> >>>> >> >>is
>> >>>> >> >> >> >> running 0.8.0 and broker2 one 0.8.1.1 (from the apache
>> >>>>release
>> >>>> >> >>link.
>> >>>> >> >> >> >> Directly using the tar ball, no local build used).
>> >>>> >> >> >> >>
>> >>>> >> >> >> >> I have set the log.retention.minutes=10. However the
>> >>>>broker
>> >>>>is
>> >>>> >>not
>> >>>> >> >> >> >> honoring the setting. I see its not cleaning the log.dir
>> >>>>at
>> >>>> >>all.
>> >>>> >> >> >> >>
>> >>>> >> >> >> >> However when I set the log.retention.hours=1, then it
>> >>>>starts
>> >>>> >> >>cleaning
>> >>>> >> >> >> >>the
>> >>>> >> >> >> >> log.
>> >>>> >> >> >> >>
>> >>>> >> >> >> >> When I have the log.retention.minutes set in the
>> >>>> >>server.properties
>> >>>> >> >> >>then
>> >>>> >> >> >> >>I
>> >>>> >> >> >> >> see this logged in server.log:
>> >>>> >> >> >> >>
>> >>>> >> >> >> >> Š..
>> >>>> >> >> >> >> [2014-06-27 19:21:06,633] WARN Property
>> >>>>log.retention.minutes
>> >>>> >>is
>> >>>> >> >>not
>> >>>> >> >> >> >>valid
>> >>>> >> >> >> >> (kafka.utils.VerifiableProperties)
>> >>>> >> >> >> >> [2014-06-27 19:21:06,633] WARN Property
>> >>>>log.retention.minutes
>> >>>> >>is
>> >>>> >> >>not
>> >>>> >> >> >> >>valid
>> >>>> >> >> >> >> (kafka.utils.VerifiableProperties)
>> >>>> >> >> >> >> ŠŠ
>> >>>> >> >> >> >>
>> >>>> >> >> >> >>
>> >>>> >> >> >> >> I have set these properties too:
>> >>>> >> >> >> >>
>> >>>> >> >> >> >> log.cleaner.enable=true
>> >>>> >> >> >> >> log.cleanup.policy=delete
>> >>>> >> >> >> >>
>> >>>> >> >> >> >>
>> >>>> >> >> >> >> But I see similar warning logged for these properties
>> too.
>> >>>> >> >> >> >>
>> >>>> >> >> >> >> Regards,
>> >>>> >> >> >> >> Virendra
>> >>>> >> >> >> >>
>> >>>> >> >> >> >>
>> >>>> >> >> >> >
>> >>>> >> >> >> >
>> >>>> >> >> >> >--
>> >>>> >> >> >> >-- Guozhang
>> >>>> >> >> >>
>> >>>> >> >> >>
>> >>>> >> >> >
>> >>>> >> >> >
>> >>>> >> >> >--
>> >>>> >> >> >-- Guozhang
>> >>>> >> >>
>> >>>> >> >>
>> >>>> >> >
>> >>>> >> >
>> >>>> >> >--
>> >>>> >> >-- Guozhang
>> >>>> >>
>> >>>> >>
>> >>>>
>> >>>>
>> >>
>> >
>>
>>
>
Re: kafka 0.8.1.1 log.retention.minutes NOT being honored
Posted by François Langelier <f....@gmail.com>.
UP! Can someone add the log.retention.hours in the documentation?
And is the retention.hours or retention.minutes available for the by-topic
configuration?
François Langelier
Étudiant en génie Logiciel - École de Technologie Supérieure
<http://www.etsmtl.ca/>
Capitaine Club Capra <http://capra.etsmtl.ca/>
VP-Communication - CS Games <http://csgames.org> 2014
Jeux de Génie <http://www.jdgets.com/> 2011 à 2014
Magistrat Fraternité du Piranha <http://fraternitedupiranha.com/>
Comité Organisateur Olympiades ÉTS 2012
Compétition Québécoise d'Ingénierie 2012 - Compétition Senior
On Thu, Jul 10, 2014 at 11:16 PM, Virendra Pratap Singh <
vpsingh@yahoo-inc.com.invalid> wrote:
> No worries, its working. My deployment was flawed and it was running
> 0.8.0, while I was under the impression that its running 0.8.1.1.
> After going to proper 0.8.1.1, the parameter started working properly.
>
> On 7/10/14, 2:18 PM, "Virendra Pratap Singh" <vp...@yahoo-inc.com>
> wrote:
>
> >Has someone really used : log.retention.minutes in kafka 0.8.1.1.
> >I have my full cluster running on 0.8.1.1 and the logs data is just not
> >getting cleaned up.
> >
> >And I see this message in kafka server.log
> >...
> >[2014-07-10 20:49:43,786] WARN Property log.retention.minutes is not valid
> >(kafka.utils.VerifiableProperties)
> >[2014-07-10 20:49:43,786] WARN Property log.retention.minutes is not valid
> >(kafka.utils.VerifiableProperties)
> >....
> >
> >Virendra
> >
> >On 7/10/14, 10:07 AM, "Virendra Pratap Singh"
> ><vp...@yahoo-inc.com.INVALID> wrote:
> >
> >>Thank you for pointing out to JIRA.
> >>
> >>On 7/9/14, 6:38 PM, "Jun Rao" <ju...@gmail.com> wrote:
> >>
> >>>This is being worked on in
> >>>https://issues.apache.org/jira/browse/KAFKA-1325
> >>>
> >>>Thanks,
> >>>
> >>>Jun
> >>>
> >>>
> >>>On Wed, Jul 9, 2014 at 11:42 AM, Virendra Pratap Singh <
> >>>vpsingh@yahoo-inc.com.invalid> wrote:
> >>>
> >>>> Well currently the log rollup is controlled via
> >>>>
> >>>> log.roll.hours
> >>>> Or
> >>>> log.segment.bytes
> >>>>
> >>>> Given that we now have support to log retention in minutes, I guess it
> >>>> would be apt to have rollup also have capability to be available in
> >>>> minutes. Whom/where should I ask to have that coded in.
> >>>>
> >>>> One a similar note, I would genuinely want all the size and time based
> >>>> parameters to be defined at bytes and ms level. This would make it
> >>>>generic
> >>>> enough to have user choose and define kind of setting they want and
> >>>>not
> >>>> require someone to go and change the code to support a new use case.
> >>>> Having hours/minutes in config will just cause proliferation/bloating
> >>>>of
> >>>> config parameters. What is a good place to air this suggestion.
> >>>>Anyone?
> >>>>
> >>>> Virendra
> >>>>
> >>>> On 7/9/14, 7:21 AM, "Jun Rao" <ju...@gmail.com> wrote:
> >>>>
> >>>> >Actually, Kafka only removes old segments. The last (active) segment
> >>>>is
> >>>> >never removed. So, f you want to have a 10 min retention, you need to
> >>>> >configure log rolling such that log segments are rolled at least
> >>>>every
> >>>>10
> >>>> >mins.
> >>>> >
> >>>> >Thanks,
> >>>> >
> >>>> >Jun
> >>>> >
> >>>> >
> >>>> >On Tue, Jul 8, 2014 at 10:04 PM, Virendra Pratap Singh <
> >>>> >vpsingh@yahoo-inc.com.invalid> wrote:
> >>>> >
> >>>> >> That's correct. The server where in I was running 0.8.1.1 was not
> >>>> >>honoring
> >>>> >> this parameter, despite the fact it was set in it
> >>>>server.properties.
> >>>> >> Not sure if this fact would play any role, the server which was
> >>>>running
> >>>> >> 0.8.0 was the leader for all the topics and partition in my setup.
> >>>>And
> >>>> >>the
> >>>> >> second server running 0.8.1.1 has all the replicas (follower).
> >>>> >>
> >>>> >> Virendra
> >>>> >>
> >>>> >> On 7/8/14, 12:54 PM, "Guozhang Wang" <wa...@gmail.com> wrote:
> >>>> >>
> >>>> >> >Server properties should affect on only the local instance
> >>>>separately.
> >>>> >>Are
> >>>> >> >you saying the property is not honored even on the 0.8.1 machines?
> >>>> >> >
> >>>> >> >Guozhang
> >>>> >> >
> >>>> >> >On Mon, Jul 7, 2014 at 3:55 PM, Virendra Pratap Singh <
> >>>> >> >vpsingh@yahoo-inc.com.invalid> wrote:
> >>>> >> >
> >>>> >> >> By setting this property
> >>>> >> >> log.retention.mins=10
> >>>> >> >> in the server.properties file, which is passed as argument when
> >>>> >>starting
> >>>> >> >> the broker.
> >>>> >> >>
> >>>> >> >> Virendra
> >>>> >> >>
> >>>> >> >> On 7/7/14, 3:31 PM, "Guozhang Wang" <wa...@gmail.com> wrote:
> >>>> >> >>
> >>>> >> >> >How do you set the retention.minutes property? Is it through
> >>>> >>zk-based
> >>>> >> >> >topics tool?
> >>>> >> >> >
> >>>> >> >> >Guozhang
> >>>> >> >> >
> >>>> >> >> >
> >>>> >> >> >On Mon, Jul 7, 2014 at 3:07 PM, Virendra Pratap Singh <
> >>>> >> >> >vpsingh@yahoo-inc.com.invalid> wrote:
> >>>> >> >> >
> >>>> >> >> >> I am running a mixed cluster as I mentioned earlier. 1 broker
> >>>> >>0.8.0
> >>>> >> >>and
> >>>> >> >> >> the other 0.8.1.1. Should the retention of topics for
> >>>>partitions
> >>>> >> >> >> owned/replicated by the broker running 0.8.1.1 not enforce
> >>>>the
> >>>> >>server
> >>>> >> >> >> properties settings as defined for that server.
> >>>> >> >> >>
> >>>> >> >> >> So this brings an interesting question, in case of
> >>>>heterogeneous
> >>>> >> >> >> environment (as is in my case, which system parameters will
> >>>>take
> >>>> >> >> >> preference/precedence).
> >>>> >> >> >>
> >>>> >> >> >> Virendra
> >>>> >> >> >>
> >>>> >> >> >> On 6/30/14, 9:19 AM, "Guozhang Wang" <wa...@gmail.com>
> >>>>wrote:
> >>>> >> >> >>
> >>>> >> >> >> >The retention.minute property is only introduced in 0.8.1:
> >>>> >> >> >> >
> >>>> >> >> >> >https://issues.apache.org/jira/browse/KAFKA-918
> >>>> >> >> >> >
> >>>> >> >> >> >if you are running 0.8.0 then it will not be recognized.
> >>>> >> >> >> >
> >>>> >> >> >> >Guozhang
> >>>> >> >> >> >
> >>>> >> >> >> >
> >>>> >> >> >> >
> >>>> >> >> >> >On Fri, Jun 27, 2014 at 2:13 PM, Virendra Pratap Singh <
> >>>> >> >> >> >vpsingh@yahoo-inc.com.invalid> wrote:
> >>>> >> >> >> >
> >>>> >> >> >> >> Running a mixed 2 broker cluster. Mixed as in one of the
> >>>> >>broker1
> >>>> >> >>is
> >>>> >> >> >> >> running 0.8.0 and broker2 one 0.8.1.1 (from the apache
> >>>>release
> >>>> >> >>link.
> >>>> >> >> >> >> Directly using the tar ball, no local build used).
> >>>> >> >> >> >>
> >>>> >> >> >> >> I have set the log.retention.minutes=10. However the
> >>>>broker
> >>>>is
> >>>> >>not
> >>>> >> >> >> >> honoring the setting. I see its not cleaning the log.dir
> >>>>at
> >>>> >>all.
> >>>> >> >> >> >>
> >>>> >> >> >> >> However when I set the log.retention.hours=1, then it
> >>>>starts
> >>>> >> >>cleaning
> >>>> >> >> >> >>the
> >>>> >> >> >> >> log.
> >>>> >> >> >> >>
> >>>> >> >> >> >> When I have the log.retention.minutes set in the
> >>>> >>server.properties
> >>>> >> >> >>then
> >>>> >> >> >> >>I
> >>>> >> >> >> >> see this logged in server.log:
> >>>> >> >> >> >>
> >>>> >> >> >> >> Š..
> >>>> >> >> >> >> [2014-06-27 19:21:06,633] WARN Property
> >>>>log.retention.minutes
> >>>> >>is
> >>>> >> >>not
> >>>> >> >> >> >>valid
> >>>> >> >> >> >> (kafka.utils.VerifiableProperties)
> >>>> >> >> >> >> [2014-06-27 19:21:06,633] WARN Property
> >>>>log.retention.minutes
> >>>> >>is
> >>>> >> >>not
> >>>> >> >> >> >>valid
> >>>> >> >> >> >> (kafka.utils.VerifiableProperties)
> >>>> >> >> >> >> ŠŠ
> >>>> >> >> >> >>
> >>>> >> >> >> >>
> >>>> >> >> >> >> I have set these properties too:
> >>>> >> >> >> >>
> >>>> >> >> >> >> log.cleaner.enable=true
> >>>> >> >> >> >> log.cleanup.policy=delete
> >>>> >> >> >> >>
> >>>> >> >> >> >>
> >>>> >> >> >> >> But I see similar warning logged for these properties too.
> >>>> >> >> >> >>
> >>>> >> >> >> >> Regards,
> >>>> >> >> >> >> Virendra
> >>>> >> >> >> >>
> >>>> >> >> >> >>
> >>>> >> >> >> >
> >>>> >> >> >> >
> >>>> >> >> >> >--
> >>>> >> >> >> >-- Guozhang
> >>>> >> >> >>
> >>>> >> >> >>
> >>>> >> >> >
> >>>> >> >> >
> >>>> >> >> >--
> >>>> >> >> >-- Guozhang
> >>>> >> >>
> >>>> >> >>
> >>>> >> >
> >>>> >> >
> >>>> >> >--
> >>>> >> >-- Guozhang
> >>>> >>
> >>>> >>
> >>>>
> >>>>
> >>
> >
>
>
Re: kafka 0.8.1.1 log.retention.minutes NOT being honored
Posted by Virendra Pratap Singh <vp...@yahoo-inc.com.INVALID>.
No worries, its working. My deployment was flawed and it was running
0.8.0, while I was under the impression that its running 0.8.1.1.
After going to proper 0.8.1.1, the parameter started working properly.
On 7/10/14, 2:18 PM, "Virendra Pratap Singh" <vp...@yahoo-inc.com> wrote:
>Has someone really used : log.retention.minutes in kafka 0.8.1.1.
>I have my full cluster running on 0.8.1.1 and the logs data is just not
>getting cleaned up.
>
>And I see this message in kafka server.log
>...
>[2014-07-10 20:49:43,786] WARN Property log.retention.minutes is not valid
>(kafka.utils.VerifiableProperties)
>[2014-07-10 20:49:43,786] WARN Property log.retention.minutes is not valid
>(kafka.utils.VerifiableProperties)
>....
>
>Virendra
>
>On 7/10/14, 10:07 AM, "Virendra Pratap Singh"
><vp...@yahoo-inc.com.INVALID> wrote:
>
>>Thank you for pointing out to JIRA.
>>
>>On 7/9/14, 6:38 PM, "Jun Rao" <ju...@gmail.com> wrote:
>>
>>>This is being worked on in
>>>https://issues.apache.org/jira/browse/KAFKA-1325
>>>
>>>Thanks,
>>>
>>>Jun
>>>
>>>
>>>On Wed, Jul 9, 2014 at 11:42 AM, Virendra Pratap Singh <
>>>vpsingh@yahoo-inc.com.invalid> wrote:
>>>
>>>> Well currently the log rollup is controlled via
>>>>
>>>> log.roll.hours
>>>> Or
>>>> log.segment.bytes
>>>>
>>>> Given that we now have support to log retention in minutes, I guess it
>>>> would be apt to have rollup also have capability to be available in
>>>> minutes. Whom/where should I ask to have that coded in.
>>>>
>>>> One a similar note, I would genuinely want all the size and time based
>>>> parameters to be defined at bytes and ms level. This would make it
>>>>generic
>>>> enough to have user choose and define kind of setting they want and
>>>>not
>>>> require someone to go and change the code to support a new use case.
>>>> Having hours/minutes in config will just cause proliferation/bloating
>>>>of
>>>> config parameters. What is a good place to air this suggestion.
>>>>Anyone?
>>>>
>>>> Virendra
>>>>
>>>> On 7/9/14, 7:21 AM, "Jun Rao" <ju...@gmail.com> wrote:
>>>>
>>>> >Actually, Kafka only removes old segments. The last (active) segment
>>>>is
>>>> >never removed. So, f you want to have a 10 min retention, you need to
>>>> >configure log rolling such that log segments are rolled at least
>>>>every
>>>>10
>>>> >mins.
>>>> >
>>>> >Thanks,
>>>> >
>>>> >Jun
>>>> >
>>>> >
>>>> >On Tue, Jul 8, 2014 at 10:04 PM, Virendra Pratap Singh <
>>>> >vpsingh@yahoo-inc.com.invalid> wrote:
>>>> >
>>>> >> That's correct. The server where in I was running 0.8.1.1 was not
>>>> >>honoring
>>>> >> this parameter, despite the fact it was set in it
>>>>server.properties.
>>>> >> Not sure if this fact would play any role, the server which was
>>>>running
>>>> >> 0.8.0 was the leader for all the topics and partition in my setup.
>>>>And
>>>> >>the
>>>> >> second server running 0.8.1.1 has all the replicas (follower).
>>>> >>
>>>> >> Virendra
>>>> >>
>>>> >> On 7/8/14, 12:54 PM, "Guozhang Wang" <wa...@gmail.com> wrote:
>>>> >>
>>>> >> >Server properties should affect on only the local instance
>>>>separately.
>>>> >>Are
>>>> >> >you saying the property is not honored even on the 0.8.1 machines?
>>>> >> >
>>>> >> >Guozhang
>>>> >> >
>>>> >> >On Mon, Jul 7, 2014 at 3:55 PM, Virendra Pratap Singh <
>>>> >> >vpsingh@yahoo-inc.com.invalid> wrote:
>>>> >> >
>>>> >> >> By setting this property
>>>> >> >> log.retention.mins=10
>>>> >> >> in the server.properties file, which is passed as argument when
>>>> >>starting
>>>> >> >> the broker.
>>>> >> >>
>>>> >> >> Virendra
>>>> >> >>
>>>> >> >> On 7/7/14, 3:31 PM, "Guozhang Wang" <wa...@gmail.com> wrote:
>>>> >> >>
>>>> >> >> >How do you set the retention.minutes property? Is it through
>>>> >>zk-based
>>>> >> >> >topics tool?
>>>> >> >> >
>>>> >> >> >Guozhang
>>>> >> >> >
>>>> >> >> >
>>>> >> >> >On Mon, Jul 7, 2014 at 3:07 PM, Virendra Pratap Singh <
>>>> >> >> >vpsingh@yahoo-inc.com.invalid> wrote:
>>>> >> >> >
>>>> >> >> >> I am running a mixed cluster as I mentioned earlier. 1 broker
>>>> >>0.8.0
>>>> >> >>and
>>>> >> >> >> the other 0.8.1.1. Should the retention of topics for
>>>>partitions
>>>> >> >> >> owned/replicated by the broker running 0.8.1.1 not enforce
>>>>the
>>>> >>server
>>>> >> >> >> properties settings as defined for that server.
>>>> >> >> >>
>>>> >> >> >> So this brings an interesting question, in case of
>>>>heterogeneous
>>>> >> >> >> environment (as is in my case, which system parameters will
>>>>take
>>>> >> >> >> preference/precedence).
>>>> >> >> >>
>>>> >> >> >> Virendra
>>>> >> >> >>
>>>> >> >> >> On 6/30/14, 9:19 AM, "Guozhang Wang" <wa...@gmail.com>
>>>>wrote:
>>>> >> >> >>
>>>> >> >> >> >The retention.minute property is only introduced in 0.8.1:
>>>> >> >> >> >
>>>> >> >> >> >https://issues.apache.org/jira/browse/KAFKA-918
>>>> >> >> >> >
>>>> >> >> >> >if you are running 0.8.0 then it will not be recognized.
>>>> >> >> >> >
>>>> >> >> >> >Guozhang
>>>> >> >> >> >
>>>> >> >> >> >
>>>> >> >> >> >
>>>> >> >> >> >On Fri, Jun 27, 2014 at 2:13 PM, Virendra Pratap Singh <
>>>> >> >> >> >vpsingh@yahoo-inc.com.invalid> wrote:
>>>> >> >> >> >
>>>> >> >> >> >> Running a mixed 2 broker cluster. Mixed as in one of the
>>>> >>broker1
>>>> >> >>is
>>>> >> >> >> >> running 0.8.0 and broker2 one 0.8.1.1 (from the apache
>>>>release
>>>> >> >>link.
>>>> >> >> >> >> Directly using the tar ball, no local build used).
>>>> >> >> >> >>
>>>> >> >> >> >> I have set the log.retention.minutes=10. However the
>>>>broker
>>>>is
>>>> >>not
>>>> >> >> >> >> honoring the setting. I see its not cleaning the log.dir
>>>>at
>>>> >>all.
>>>> >> >> >> >>
>>>> >> >> >> >> However when I set the log.retention.hours=1, then it
>>>>starts
>>>> >> >>cleaning
>>>> >> >> >> >>the
>>>> >> >> >> >> log.
>>>> >> >> >> >>
>>>> >> >> >> >> When I have the log.retention.minutes set in the
>>>> >>server.properties
>>>> >> >> >>then
>>>> >> >> >> >>I
>>>> >> >> >> >> see this logged in server.log:
>>>> >> >> >> >>
>>>> >> >> >> >> Š..
>>>> >> >> >> >> [2014-06-27 19:21:06,633] WARN Property
>>>>log.retention.minutes
>>>> >>is
>>>> >> >>not
>>>> >> >> >> >>valid
>>>> >> >> >> >> (kafka.utils.VerifiableProperties)
>>>> >> >> >> >> [2014-06-27 19:21:06,633] WARN Property
>>>>log.retention.minutes
>>>> >>is
>>>> >> >>not
>>>> >> >> >> >>valid
>>>> >> >> >> >> (kafka.utils.VerifiableProperties)
>>>> >> >> >> >> ŠŠ
>>>> >> >> >> >>
>>>> >> >> >> >>
>>>> >> >> >> >> I have set these properties too:
>>>> >> >> >> >>
>>>> >> >> >> >> log.cleaner.enable=true
>>>> >> >> >> >> log.cleanup.policy=delete
>>>> >> >> >> >>
>>>> >> >> >> >>
>>>> >> >> >> >> But I see similar warning logged for these properties too.
>>>> >> >> >> >>
>>>> >> >> >> >> Regards,
>>>> >> >> >> >> Virendra
>>>> >> >> >> >>
>>>> >> >> >> >>
>>>> >> >> >> >
>>>> >> >> >> >
>>>> >> >> >> >--
>>>> >> >> >> >-- Guozhang
>>>> >> >> >>
>>>> >> >> >>
>>>> >> >> >
>>>> >> >> >
>>>> >> >> >--
>>>> >> >> >-- Guozhang
>>>> >> >>
>>>> >> >>
>>>> >> >
>>>> >> >
>>>> >> >--
>>>> >> >-- Guozhang
>>>> >>
>>>> >>
>>>>
>>>>
>>
>
Re: kafka 0.8.1.1 log.retention.minutes NOT being honored
Posted by Virendra Pratap Singh <vp...@yahoo-inc.com.INVALID>.
Has someone really used : log.retention.minutes in kafka 0.8.1.1.
I have my full cluster running on 0.8.1.1 and the logs data is just not
getting cleaned up.
And I see this message in kafka server.log
...
[2014-07-10 20:49:43,786] WARN Property log.retention.minutes is not valid
(kafka.utils.VerifiableProperties)
[2014-07-10 20:49:43,786] WARN Property log.retention.minutes is not valid
(kafka.utils.VerifiableProperties)
....
Virendra
On 7/10/14, 10:07 AM, "Virendra Pratap Singh"
<vp...@yahoo-inc.com.INVALID> wrote:
>Thank you for pointing out to JIRA.
>
>On 7/9/14, 6:38 PM, "Jun Rao" <ju...@gmail.com> wrote:
>
>>This is being worked on in
>>https://issues.apache.org/jira/browse/KAFKA-1325
>>
>>Thanks,
>>
>>Jun
>>
>>
>>On Wed, Jul 9, 2014 at 11:42 AM, Virendra Pratap Singh <
>>vpsingh@yahoo-inc.com.invalid> wrote:
>>
>>> Well currently the log rollup is controlled via
>>>
>>> log.roll.hours
>>> Or
>>> log.segment.bytes
>>>
>>> Given that we now have support to log retention in minutes, I guess it
>>> would be apt to have rollup also have capability to be available in
>>> minutes. Whom/where should I ask to have that coded in.
>>>
>>> One a similar note, I would genuinely want all the size and time based
>>> parameters to be defined at bytes and ms level. This would make it
>>>generic
>>> enough to have user choose and define kind of setting they want and not
>>> require someone to go and change the code to support a new use case.
>>> Having hours/minutes in config will just cause proliferation/bloating
>>>of
>>> config parameters. What is a good place to air this suggestion. Anyone?
>>>
>>> Virendra
>>>
>>> On 7/9/14, 7:21 AM, "Jun Rao" <ju...@gmail.com> wrote:
>>>
>>> >Actually, Kafka only removes old segments. The last (active) segment
>>>is
>>> >never removed. So, f you want to have a 10 min retention, you need to
>>> >configure log rolling such that log segments are rolled at least every
>>>10
>>> >mins.
>>> >
>>> >Thanks,
>>> >
>>> >Jun
>>> >
>>> >
>>> >On Tue, Jul 8, 2014 at 10:04 PM, Virendra Pratap Singh <
>>> >vpsingh@yahoo-inc.com.invalid> wrote:
>>> >
>>> >> That's correct. The server where in I was running 0.8.1.1 was not
>>> >>honoring
>>> >> this parameter, despite the fact it was set in it server.properties.
>>> >> Not sure if this fact would play any role, the server which was
>>>running
>>> >> 0.8.0 was the leader for all the topics and partition in my setup.
>>>And
>>> >>the
>>> >> second server running 0.8.1.1 has all the replicas (follower).
>>> >>
>>> >> Virendra
>>> >>
>>> >> On 7/8/14, 12:54 PM, "Guozhang Wang" <wa...@gmail.com> wrote:
>>> >>
>>> >> >Server properties should affect on only the local instance
>>>separately.
>>> >>Are
>>> >> >you saying the property is not honored even on the 0.8.1 machines?
>>> >> >
>>> >> >Guozhang
>>> >> >
>>> >> >On Mon, Jul 7, 2014 at 3:55 PM, Virendra Pratap Singh <
>>> >> >vpsingh@yahoo-inc.com.invalid> wrote:
>>> >> >
>>> >> >> By setting this property
>>> >> >> log.retention.mins=10
>>> >> >> in the server.properties file, which is passed as argument when
>>> >>starting
>>> >> >> the broker.
>>> >> >>
>>> >> >> Virendra
>>> >> >>
>>> >> >> On 7/7/14, 3:31 PM, "Guozhang Wang" <wa...@gmail.com> wrote:
>>> >> >>
>>> >> >> >How do you set the retention.minutes property? Is it through
>>> >>zk-based
>>> >> >> >topics tool?
>>> >> >> >
>>> >> >> >Guozhang
>>> >> >> >
>>> >> >> >
>>> >> >> >On Mon, Jul 7, 2014 at 3:07 PM, Virendra Pratap Singh <
>>> >> >> >vpsingh@yahoo-inc.com.invalid> wrote:
>>> >> >> >
>>> >> >> >> I am running a mixed cluster as I mentioned earlier. 1 broker
>>> >>0.8.0
>>> >> >>and
>>> >> >> >> the other 0.8.1.1. Should the retention of topics for
>>>partitions
>>> >> >> >> owned/replicated by the broker running 0.8.1.1 not enforce the
>>> >>server
>>> >> >> >> properties settings as defined for that server.
>>> >> >> >>
>>> >> >> >> So this brings an interesting question, in case of
>>>heterogeneous
>>> >> >> >> environment (as is in my case, which system parameters will
>>>take
>>> >> >> >> preference/precedence).
>>> >> >> >>
>>> >> >> >> Virendra
>>> >> >> >>
>>> >> >> >> On 6/30/14, 9:19 AM, "Guozhang Wang" <wa...@gmail.com>
>>>wrote:
>>> >> >> >>
>>> >> >> >> >The retention.minute property is only introduced in 0.8.1:
>>> >> >> >> >
>>> >> >> >> >https://issues.apache.org/jira/browse/KAFKA-918
>>> >> >> >> >
>>> >> >> >> >if you are running 0.8.0 then it will not be recognized.
>>> >> >> >> >
>>> >> >> >> >Guozhang
>>> >> >> >> >
>>> >> >> >> >
>>> >> >> >> >
>>> >> >> >> >On Fri, Jun 27, 2014 at 2:13 PM, Virendra Pratap Singh <
>>> >> >> >> >vpsingh@yahoo-inc.com.invalid> wrote:
>>> >> >> >> >
>>> >> >> >> >> Running a mixed 2 broker cluster. Mixed as in one of the
>>> >>broker1
>>> >> >>is
>>> >> >> >> >> running 0.8.0 and broker2 one 0.8.1.1 (from the apache
>>>release
>>> >> >>link.
>>> >> >> >> >> Directly using the tar ball, no local build used).
>>> >> >> >> >>
>>> >> >> >> >> I have set the log.retention.minutes=10. However the broker
>>>is
>>> >>not
>>> >> >> >> >> honoring the setting. I see its not cleaning the log.dir at
>>> >>all.
>>> >> >> >> >>
>>> >> >> >> >> However when I set the log.retention.hours=1, then it
>>>starts
>>> >> >>cleaning
>>> >> >> >> >>the
>>> >> >> >> >> log.
>>> >> >> >> >>
>>> >> >> >> >> When I have the log.retention.minutes set in the
>>> >>server.properties
>>> >> >> >>then
>>> >> >> >> >>I
>>> >> >> >> >> see this logged in server.log:
>>> >> >> >> >>
>>> >> >> >> >> Š..
>>> >> >> >> >> [2014-06-27 19:21:06,633] WARN Property
>>>log.retention.minutes
>>> >>is
>>> >> >>not
>>> >> >> >> >>valid
>>> >> >> >> >> (kafka.utils.VerifiableProperties)
>>> >> >> >> >> [2014-06-27 19:21:06,633] WARN Property
>>>log.retention.minutes
>>> >>is
>>> >> >>not
>>> >> >> >> >>valid
>>> >> >> >> >> (kafka.utils.VerifiableProperties)
>>> >> >> >> >> ŠŠ
>>> >> >> >> >>
>>> >> >> >> >>
>>> >> >> >> >> I have set these properties too:
>>> >> >> >> >>
>>> >> >> >> >> log.cleaner.enable=true
>>> >> >> >> >> log.cleanup.policy=delete
>>> >> >> >> >>
>>> >> >> >> >>
>>> >> >> >> >> But I see similar warning logged for these properties too.
>>> >> >> >> >>
>>> >> >> >> >> Regards,
>>> >> >> >> >> Virendra
>>> >> >> >> >>
>>> >> >> >> >>
>>> >> >> >> >
>>> >> >> >> >
>>> >> >> >> >--
>>> >> >> >> >-- Guozhang
>>> >> >> >>
>>> >> >> >>
>>> >> >> >
>>> >> >> >
>>> >> >> >--
>>> >> >> >-- Guozhang
>>> >> >>
>>> >> >>
>>> >> >
>>> >> >
>>> >> >--
>>> >> >-- Guozhang
>>> >>
>>> >>
>>>
>>>
>
Re: kafka 0.8.1.1 log.retention.minutes NOT being honored
Posted by Virendra Pratap Singh <vp...@yahoo-inc.com.INVALID>.
Thank you for pointing out to JIRA.
On 7/9/14, 6:38 PM, "Jun Rao" <ju...@gmail.com> wrote:
>This is being worked on in
>https://issues.apache.org/jira/browse/KAFKA-1325
>
>Thanks,
>
>Jun
>
>
>On Wed, Jul 9, 2014 at 11:42 AM, Virendra Pratap Singh <
>vpsingh@yahoo-inc.com.invalid> wrote:
>
>> Well currently the log rollup is controlled via
>>
>> log.roll.hours
>> Or
>> log.segment.bytes
>>
>> Given that we now have support to log retention in minutes, I guess it
>> would be apt to have rollup also have capability to be available in
>> minutes. Whom/where should I ask to have that coded in.
>>
>> One a similar note, I would genuinely want all the size and time based
>> parameters to be defined at bytes and ms level. This would make it
>>generic
>> enough to have user choose and define kind of setting they want and not
>> require someone to go and change the code to support a new use case.
>> Having hours/minutes in config will just cause proliferation/bloating of
>> config parameters. What is a good place to air this suggestion. Anyone?
>>
>> Virendra
>>
>> On 7/9/14, 7:21 AM, "Jun Rao" <ju...@gmail.com> wrote:
>>
>> >Actually, Kafka only removes old segments. The last (active) segment is
>> >never removed. So, f you want to have a 10 min retention, you need to
>> >configure log rolling such that log segments are rolled at least every
>>10
>> >mins.
>> >
>> >Thanks,
>> >
>> >Jun
>> >
>> >
>> >On Tue, Jul 8, 2014 at 10:04 PM, Virendra Pratap Singh <
>> >vpsingh@yahoo-inc.com.invalid> wrote:
>> >
>> >> That's correct. The server where in I was running 0.8.1.1 was not
>> >>honoring
>> >> this parameter, despite the fact it was set in it server.properties.
>> >> Not sure if this fact would play any role, the server which was
>>running
>> >> 0.8.0 was the leader for all the topics and partition in my setup.
>>And
>> >>the
>> >> second server running 0.8.1.1 has all the replicas (follower).
>> >>
>> >> Virendra
>> >>
>> >> On 7/8/14, 12:54 PM, "Guozhang Wang" <wa...@gmail.com> wrote:
>> >>
>> >> >Server properties should affect on only the local instance
>>separately.
>> >>Are
>> >> >you saying the property is not honored even on the 0.8.1 machines?
>> >> >
>> >> >Guozhang
>> >> >
>> >> >On Mon, Jul 7, 2014 at 3:55 PM, Virendra Pratap Singh <
>> >> >vpsingh@yahoo-inc.com.invalid> wrote:
>> >> >
>> >> >> By setting this property
>> >> >> log.retention.mins=10
>> >> >> in the server.properties file, which is passed as argument when
>> >>starting
>> >> >> the broker.
>> >> >>
>> >> >> Virendra
>> >> >>
>> >> >> On 7/7/14, 3:31 PM, "Guozhang Wang" <wa...@gmail.com> wrote:
>> >> >>
>> >> >> >How do you set the retention.minutes property? Is it through
>> >>zk-based
>> >> >> >topics tool?
>> >> >> >
>> >> >> >Guozhang
>> >> >> >
>> >> >> >
>> >> >> >On Mon, Jul 7, 2014 at 3:07 PM, Virendra Pratap Singh <
>> >> >> >vpsingh@yahoo-inc.com.invalid> wrote:
>> >> >> >
>> >> >> >> I am running a mixed cluster as I mentioned earlier. 1 broker
>> >>0.8.0
>> >> >>and
>> >> >> >> the other 0.8.1.1. Should the retention of topics for
>>partitions
>> >> >> >> owned/replicated by the broker running 0.8.1.1 not enforce the
>> >>server
>> >> >> >> properties settings as defined for that server.
>> >> >> >>
>> >> >> >> So this brings an interesting question, in case of
>>heterogeneous
>> >> >> >> environment (as is in my case, which system parameters will
>>take
>> >> >> >> preference/precedence).
>> >> >> >>
>> >> >> >> Virendra
>> >> >> >>
>> >> >> >> On 6/30/14, 9:19 AM, "Guozhang Wang" <wa...@gmail.com>
>>wrote:
>> >> >> >>
>> >> >> >> >The retention.minute property is only introduced in 0.8.1:
>> >> >> >> >
>> >> >> >> >https://issues.apache.org/jira/browse/KAFKA-918
>> >> >> >> >
>> >> >> >> >if you are running 0.8.0 then it will not be recognized.
>> >> >> >> >
>> >> >> >> >Guozhang
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >On Fri, Jun 27, 2014 at 2:13 PM, Virendra Pratap Singh <
>> >> >> >> >vpsingh@yahoo-inc.com.invalid> wrote:
>> >> >> >> >
>> >> >> >> >> Running a mixed 2 broker cluster. Mixed as in one of the
>> >>broker1
>> >> >>is
>> >> >> >> >> running 0.8.0 and broker2 one 0.8.1.1 (from the apache
>>release
>> >> >>link.
>> >> >> >> >> Directly using the tar ball, no local build used).
>> >> >> >> >>
>> >> >> >> >> I have set the log.retention.minutes=10. However the broker
>>is
>> >>not
>> >> >> >> >> honoring the setting. I see its not cleaning the log.dir at
>> >>all.
>> >> >> >> >>
>> >> >> >> >> However when I set the log.retention.hours=1, then it starts
>> >> >>cleaning
>> >> >> >> >>the
>> >> >> >> >> log.
>> >> >> >> >>
>> >> >> >> >> When I have the log.retention.minutes set in the
>> >>server.properties
>> >> >> >>then
>> >> >> >> >>I
>> >> >> >> >> see this logged in server.log:
>> >> >> >> >>
>> >> >> >> >> Š..
>> >> >> >> >> [2014-06-27 19:21:06,633] WARN Property
>>log.retention.minutes
>> >>is
>> >> >>not
>> >> >> >> >>valid
>> >> >> >> >> (kafka.utils.VerifiableProperties)
>> >> >> >> >> [2014-06-27 19:21:06,633] WARN Property
>>log.retention.minutes
>> >>is
>> >> >>not
>> >> >> >> >>valid
>> >> >> >> >> (kafka.utils.VerifiableProperties)
>> >> >> >> >> ŠŠ
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >> I have set these properties too:
>> >> >> >> >>
>> >> >> >> >> log.cleaner.enable=true
>> >> >> >> >> log.cleanup.policy=delete
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >> But I see similar warning logged for these properties too.
>> >> >> >> >>
>> >> >> >> >> Regards,
>> >> >> >> >> Virendra
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >--
>> >> >> >> >-- Guozhang
>> >> >> >>
>> >> >> >>
>> >> >> >
>> >> >> >
>> >> >> >--
>> >> >> >-- Guozhang
>> >> >>
>> >> >>
>> >> >
>> >> >
>> >> >--
>> >> >-- Guozhang
>> >>
>> >>
>>
>>
Re: kafka 0.8.1.1 log.retention.minutes NOT being honored
Posted by Jun Rao <ju...@gmail.com>.
This is being worked on in https://issues.apache.org/jira/browse/KAFKA-1325
Thanks,
Jun
On Wed, Jul 9, 2014 at 11:42 AM, Virendra Pratap Singh <
vpsingh@yahoo-inc.com.invalid> wrote:
> Well currently the log rollup is controlled via
>
> log.roll.hours
> Or
> log.segment.bytes
>
> Given that we now have support to log retention in minutes, I guess it
> would be apt to have rollup also have capability to be available in
> minutes. Whom/where should I ask to have that coded in.
>
> One a similar note, I would genuinely want all the size and time based
> parameters to be defined at bytes and ms level. This would make it generic
> enough to have user choose and define kind of setting they want and not
> require someone to go and change the code to support a new use case.
> Having hours/minutes in config will just cause proliferation/bloating of
> config parameters. What is a good place to air this suggestion. Anyone?
>
> Virendra
>
> On 7/9/14, 7:21 AM, "Jun Rao" <ju...@gmail.com> wrote:
>
> >Actually, Kafka only removes old segments. The last (active) segment is
> >never removed. So, f you want to have a 10 min retention, you need to
> >configure log rolling such that log segments are rolled at least every 10
> >mins.
> >
> >Thanks,
> >
> >Jun
> >
> >
> >On Tue, Jul 8, 2014 at 10:04 PM, Virendra Pratap Singh <
> >vpsingh@yahoo-inc.com.invalid> wrote:
> >
> >> That's correct. The server where in I was running 0.8.1.1 was not
> >>honoring
> >> this parameter, despite the fact it was set in it server.properties.
> >> Not sure if this fact would play any role, the server which was running
> >> 0.8.0 was the leader for all the topics and partition in my setup. And
> >>the
> >> second server running 0.8.1.1 has all the replicas (follower).
> >>
> >> Virendra
> >>
> >> On 7/8/14, 12:54 PM, "Guozhang Wang" <wa...@gmail.com> wrote:
> >>
> >> >Server properties should affect on only the local instance separately.
> >>Are
> >> >you saying the property is not honored even on the 0.8.1 machines?
> >> >
> >> >Guozhang
> >> >
> >> >On Mon, Jul 7, 2014 at 3:55 PM, Virendra Pratap Singh <
> >> >vpsingh@yahoo-inc.com.invalid> wrote:
> >> >
> >> >> By setting this property
> >> >> log.retention.mins=10
> >> >> in the server.properties file, which is passed as argument when
> >>starting
> >> >> the broker.
> >> >>
> >> >> Virendra
> >> >>
> >> >> On 7/7/14, 3:31 PM, "Guozhang Wang" <wa...@gmail.com> wrote:
> >> >>
> >> >> >How do you set the retention.minutes property? Is it through
> >>zk-based
> >> >> >topics tool?
> >> >> >
> >> >> >Guozhang
> >> >> >
> >> >> >
> >> >> >On Mon, Jul 7, 2014 at 3:07 PM, Virendra Pratap Singh <
> >> >> >vpsingh@yahoo-inc.com.invalid> wrote:
> >> >> >
> >> >> >> I am running a mixed cluster as I mentioned earlier. 1 broker
> >>0.8.0
> >> >>and
> >> >> >> the other 0.8.1.1. Should the retention of topics for partitions
> >> >> >> owned/replicated by the broker running 0.8.1.1 not enforce the
> >>server
> >> >> >> properties settings as defined for that server.
> >> >> >>
> >> >> >> So this brings an interesting question, in case of heterogeneous
> >> >> >> environment (as is in my case, which system parameters will take
> >> >> >> preference/precedence).
> >> >> >>
> >> >> >> Virendra
> >> >> >>
> >> >> >> On 6/30/14, 9:19 AM, "Guozhang Wang" <wa...@gmail.com> wrote:
> >> >> >>
> >> >> >> >The retention.minute property is only introduced in 0.8.1:
> >> >> >> >
> >> >> >> >https://issues.apache.org/jira/browse/KAFKA-918
> >> >> >> >
> >> >> >> >if you are running 0.8.0 then it will not be recognized.
> >> >> >> >
> >> >> >> >Guozhang
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> >On Fri, Jun 27, 2014 at 2:13 PM, Virendra Pratap Singh <
> >> >> >> >vpsingh@yahoo-inc.com.invalid> wrote:
> >> >> >> >
> >> >> >> >> Running a mixed 2 broker cluster. Mixed as in one of the
> >>broker1
> >> >>is
> >> >> >> >> running 0.8.0 and broker2 one 0.8.1.1 (from the apache release
> >> >>link.
> >> >> >> >> Directly using the tar ball, no local build used).
> >> >> >> >>
> >> >> >> >> I have set the log.retention.minutes=10. However the broker is
> >>not
> >> >> >> >> honoring the setting. I see its not cleaning the log.dir at
> >>all.
> >> >> >> >>
> >> >> >> >> However when I set the log.retention.hours=1, then it starts
> >> >>cleaning
> >> >> >> >>the
> >> >> >> >> log.
> >> >> >> >>
> >> >> >> >> When I have the log.retention.minutes set in the
> >>server.properties
> >> >> >>then
> >> >> >> >>I
> >> >> >> >> see this logged in server.log:
> >> >> >> >>
> >> >> >> >> Š..
> >> >> >> >> [2014-06-27 19:21:06,633] WARN Property log.retention.minutes
> >>is
> >> >>not
> >> >> >> >>valid
> >> >> >> >> (kafka.utils.VerifiableProperties)
> >> >> >> >> [2014-06-27 19:21:06,633] WARN Property log.retention.minutes
> >>is
> >> >>not
> >> >> >> >>valid
> >> >> >> >> (kafka.utils.VerifiableProperties)
> >> >> >> >> ŠŠ
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> I have set these properties too:
> >> >> >> >>
> >> >> >> >> log.cleaner.enable=true
> >> >> >> >> log.cleanup.policy=delete
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> But I see similar warning logged for these properties too.
> >> >> >> >>
> >> >> >> >> Regards,
> >> >> >> >> Virendra
> >> >> >> >>
> >> >> >> >>
> >> >> >> >
> >> >> >> >
> >> >> >> >--
> >> >> >> >-- Guozhang
> >> >> >>
> >> >> >>
> >> >> >
> >> >> >
> >> >> >--
> >> >> >-- Guozhang
> >> >>
> >> >>
> >> >
> >> >
> >> >--
> >> >-- Guozhang
> >>
> >>
>
>
Re: kafka 0.8.1.1 log.retention.minutes NOT being honored
Posted by Virendra Pratap Singh <vp...@yahoo-inc.com.INVALID>.
Well currently the log rollup is controlled via
log.roll.hours
Or
log.segment.bytes
Given that we now have support to log retention in minutes, I guess it
would be apt to have rollup also have capability to be available in
minutes. Whom/where should I ask to have that coded in.
One a similar note, I would genuinely want all the size and time based
parameters to be defined at bytes and ms level. This would make it generic
enough to have user choose and define kind of setting they want and not
require someone to go and change the code to support a new use case.
Having hours/minutes in config will just cause proliferation/bloating of
config parameters. What is a good place to air this suggestion. Anyone?
Virendra
On 7/9/14, 7:21 AM, "Jun Rao" <ju...@gmail.com> wrote:
>Actually, Kafka only removes old segments. The last (active) segment is
>never removed. So, f you want to have a 10 min retention, you need to
>configure log rolling such that log segments are rolled at least every 10
>mins.
>
>Thanks,
>
>Jun
>
>
>On Tue, Jul 8, 2014 at 10:04 PM, Virendra Pratap Singh <
>vpsingh@yahoo-inc.com.invalid> wrote:
>
>> That's correct. The server where in I was running 0.8.1.1 was not
>>honoring
>> this parameter, despite the fact it was set in it server.properties.
>> Not sure if this fact would play any role, the server which was running
>> 0.8.0 was the leader for all the topics and partition in my setup. And
>>the
>> second server running 0.8.1.1 has all the replicas (follower).
>>
>> Virendra
>>
>> On 7/8/14, 12:54 PM, "Guozhang Wang" <wa...@gmail.com> wrote:
>>
>> >Server properties should affect on only the local instance separately.
>>Are
>> >you saying the property is not honored even on the 0.8.1 machines?
>> >
>> >Guozhang
>> >
>> >On Mon, Jul 7, 2014 at 3:55 PM, Virendra Pratap Singh <
>> >vpsingh@yahoo-inc.com.invalid> wrote:
>> >
>> >> By setting this property
>> >> log.retention.mins=10
>> >> in the server.properties file, which is passed as argument when
>>starting
>> >> the broker.
>> >>
>> >> Virendra
>> >>
>> >> On 7/7/14, 3:31 PM, "Guozhang Wang" <wa...@gmail.com> wrote:
>> >>
>> >> >How do you set the retention.minutes property? Is it through
>>zk-based
>> >> >topics tool?
>> >> >
>> >> >Guozhang
>> >> >
>> >> >
>> >> >On Mon, Jul 7, 2014 at 3:07 PM, Virendra Pratap Singh <
>> >> >vpsingh@yahoo-inc.com.invalid> wrote:
>> >> >
>> >> >> I am running a mixed cluster as I mentioned earlier. 1 broker
>>0.8.0
>> >>and
>> >> >> the other 0.8.1.1. Should the retention of topics for partitions
>> >> >> owned/replicated by the broker running 0.8.1.1 not enforce the
>>server
>> >> >> properties settings as defined for that server.
>> >> >>
>> >> >> So this brings an interesting question, in case of heterogeneous
>> >> >> environment (as is in my case, which system parameters will take
>> >> >> preference/precedence).
>> >> >>
>> >> >> Virendra
>> >> >>
>> >> >> On 6/30/14, 9:19 AM, "Guozhang Wang" <wa...@gmail.com> wrote:
>> >> >>
>> >> >> >The retention.minute property is only introduced in 0.8.1:
>> >> >> >
>> >> >> >https://issues.apache.org/jira/browse/KAFKA-918
>> >> >> >
>> >> >> >if you are running 0.8.0 then it will not be recognized.
>> >> >> >
>> >> >> >Guozhang
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> >On Fri, Jun 27, 2014 at 2:13 PM, Virendra Pratap Singh <
>> >> >> >vpsingh@yahoo-inc.com.invalid> wrote:
>> >> >> >
>> >> >> >> Running a mixed 2 broker cluster. Mixed as in one of the
>>broker1
>> >>is
>> >> >> >> running 0.8.0 and broker2 one 0.8.1.1 (from the apache release
>> >>link.
>> >> >> >> Directly using the tar ball, no local build used).
>> >> >> >>
>> >> >> >> I have set the log.retention.minutes=10. However the broker is
>>not
>> >> >> >> honoring the setting. I see its not cleaning the log.dir at
>>all.
>> >> >> >>
>> >> >> >> However when I set the log.retention.hours=1, then it starts
>> >>cleaning
>> >> >> >>the
>> >> >> >> log.
>> >> >> >>
>> >> >> >> When I have the log.retention.minutes set in the
>>server.properties
>> >> >>then
>> >> >> >>I
>> >> >> >> see this logged in server.log:
>> >> >> >>
>> >> >> >> Š..
>> >> >> >> [2014-06-27 19:21:06,633] WARN Property log.retention.minutes
>>is
>> >>not
>> >> >> >>valid
>> >> >> >> (kafka.utils.VerifiableProperties)
>> >> >> >> [2014-06-27 19:21:06,633] WARN Property log.retention.minutes
>>is
>> >>not
>> >> >> >>valid
>> >> >> >> (kafka.utils.VerifiableProperties)
>> >> >> >> ŠŠ
>> >> >> >>
>> >> >> >>
>> >> >> >> I have set these properties too:
>> >> >> >>
>> >> >> >> log.cleaner.enable=true
>> >> >> >> log.cleanup.policy=delete
>> >> >> >>
>> >> >> >>
>> >> >> >> But I see similar warning logged for these properties too.
>> >> >> >>
>> >> >> >> Regards,
>> >> >> >> Virendra
>> >> >> >>
>> >> >> >>
>> >> >> >
>> >> >> >
>> >> >> >--
>> >> >> >-- Guozhang
>> >> >>
>> >> >>
>> >> >
>> >> >
>> >> >--
>> >> >-- Guozhang
>> >>
>> >>
>> >
>> >
>> >--
>> >-- Guozhang
>>
>>
Re: kafka 0.8.1.1 log.retention.minutes NOT being honored
Posted by Jun Rao <ju...@gmail.com>.
Actually, Kafka only removes old segments. The last (active) segment is
never removed. So, f you want to have a 10 min retention, you need to
configure log rolling such that log segments are rolled at least every 10
mins.
Thanks,
Jun
On Tue, Jul 8, 2014 at 10:04 PM, Virendra Pratap Singh <
vpsingh@yahoo-inc.com.invalid> wrote:
> That's correct. The server where in I was running 0.8.1.1 was not honoring
> this parameter, despite the fact it was set in it server.properties.
> Not sure if this fact would play any role, the server which was running
> 0.8.0 was the leader for all the topics and partition in my setup. And the
> second server running 0.8.1.1 has all the replicas (follower).
>
> Virendra
>
> On 7/8/14, 12:54 PM, "Guozhang Wang" <wa...@gmail.com> wrote:
>
> >Server properties should affect on only the local instance separately. Are
> >you saying the property is not honored even on the 0.8.1 machines?
> >
> >Guozhang
> >
> >On Mon, Jul 7, 2014 at 3:55 PM, Virendra Pratap Singh <
> >vpsingh@yahoo-inc.com.invalid> wrote:
> >
> >> By setting this property
> >> log.retention.mins=10
> >> in the server.properties file, which is passed as argument when starting
> >> the broker.
> >>
> >> Virendra
> >>
> >> On 7/7/14, 3:31 PM, "Guozhang Wang" <wa...@gmail.com> wrote:
> >>
> >> >How do you set the retention.minutes property? Is it through zk-based
> >> >topics tool?
> >> >
> >> >Guozhang
> >> >
> >> >
> >> >On Mon, Jul 7, 2014 at 3:07 PM, Virendra Pratap Singh <
> >> >vpsingh@yahoo-inc.com.invalid> wrote:
> >> >
> >> >> I am running a mixed cluster as I mentioned earlier. 1 broker 0.8.0
> >>and
> >> >> the other 0.8.1.1. Should the retention of topics for partitions
> >> >> owned/replicated by the broker running 0.8.1.1 not enforce the server
> >> >> properties settings as defined for that server.
> >> >>
> >> >> So this brings an interesting question, in case of heterogeneous
> >> >> environment (as is in my case, which system parameters will take
> >> >> preference/precedence).
> >> >>
> >> >> Virendra
> >> >>
> >> >> On 6/30/14, 9:19 AM, "Guozhang Wang" <wa...@gmail.com> wrote:
> >> >>
> >> >> >The retention.minute property is only introduced in 0.8.1:
> >> >> >
> >> >> >https://issues.apache.org/jira/browse/KAFKA-918
> >> >> >
> >> >> >if you are running 0.8.0 then it will not be recognized.
> >> >> >
> >> >> >Guozhang
> >> >> >
> >> >> >
> >> >> >
> >> >> >On Fri, Jun 27, 2014 at 2:13 PM, Virendra Pratap Singh <
> >> >> >vpsingh@yahoo-inc.com.invalid> wrote:
> >> >> >
> >> >> >> Running a mixed 2 broker cluster. Mixed as in one of the broker1
> >>is
> >> >> >> running 0.8.0 and broker2 one 0.8.1.1 (from the apache release
> >>link.
> >> >> >> Directly using the tar ball, no local build used).
> >> >> >>
> >> >> >> I have set the log.retention.minutes=10. However the broker is not
> >> >> >> honoring the setting. I see its not cleaning the log.dir at all.
> >> >> >>
> >> >> >> However when I set the log.retention.hours=1, then it starts
> >>cleaning
> >> >> >>the
> >> >> >> log.
> >> >> >>
> >> >> >> When I have the log.retention.minutes set in the server.properties
> >> >>then
> >> >> >>I
> >> >> >> see this logged in server.log:
> >> >> >>
> >> >> >> Š..
> >> >> >> [2014-06-27 19:21:06,633] WARN Property log.retention.minutes is
> >>not
> >> >> >>valid
> >> >> >> (kafka.utils.VerifiableProperties)
> >> >> >> [2014-06-27 19:21:06,633] WARN Property log.retention.minutes is
> >>not
> >> >> >>valid
> >> >> >> (kafka.utils.VerifiableProperties)
> >> >> >> ŠŠ
> >> >> >>
> >> >> >>
> >> >> >> I have set these properties too:
> >> >> >>
> >> >> >> log.cleaner.enable=true
> >> >> >> log.cleanup.policy=delete
> >> >> >>
> >> >> >>
> >> >> >> But I see similar warning logged for these properties too.
> >> >> >>
> >> >> >> Regards,
> >> >> >> Virendra
> >> >> >>
> >> >> >>
> >> >> >
> >> >> >
> >> >> >--
> >> >> >-- Guozhang
> >> >>
> >> >>
> >> >
> >> >
> >> >--
> >> >-- Guozhang
> >>
> >>
> >
> >
> >--
> >-- Guozhang
>
>
Re: kafka 0.8.1.1 log.retention.minutes NOT being honored
Posted by Virendra Pratap Singh <vp...@yahoo-inc.com.INVALID>.
That's correct. The server where in I was running 0.8.1.1 was not honoring
this parameter, despite the fact it was set in it server.properties.
Not sure if this fact would play any role, the server which was running
0.8.0 was the leader for all the topics and partition in my setup. And the
second server running 0.8.1.1 has all the replicas (follower).
Virendra
On 7/8/14, 12:54 PM, "Guozhang Wang" <wa...@gmail.com> wrote:
>Server properties should affect on only the local instance separately. Are
>you saying the property is not honored even on the 0.8.1 machines?
>
>Guozhang
>
>On Mon, Jul 7, 2014 at 3:55 PM, Virendra Pratap Singh <
>vpsingh@yahoo-inc.com.invalid> wrote:
>
>> By setting this property
>> log.retention.mins=10
>> in the server.properties file, which is passed as argument when starting
>> the broker.
>>
>> Virendra
>>
>> On 7/7/14, 3:31 PM, "Guozhang Wang" <wa...@gmail.com> wrote:
>>
>> >How do you set the retention.minutes property? Is it through zk-based
>> >topics tool?
>> >
>> >Guozhang
>> >
>> >
>> >On Mon, Jul 7, 2014 at 3:07 PM, Virendra Pratap Singh <
>> >vpsingh@yahoo-inc.com.invalid> wrote:
>> >
>> >> I am running a mixed cluster as I mentioned earlier. 1 broker 0.8.0
>>and
>> >> the other 0.8.1.1. Should the retention of topics for partitions
>> >> owned/replicated by the broker running 0.8.1.1 not enforce the server
>> >> properties settings as defined for that server.
>> >>
>> >> So this brings an interesting question, in case of heterogeneous
>> >> environment (as is in my case, which system parameters will take
>> >> preference/precedence).
>> >>
>> >> Virendra
>> >>
>> >> On 6/30/14, 9:19 AM, "Guozhang Wang" <wa...@gmail.com> wrote:
>> >>
>> >> >The retention.minute property is only introduced in 0.8.1:
>> >> >
>> >> >https://issues.apache.org/jira/browse/KAFKA-918
>> >> >
>> >> >if you are running 0.8.0 then it will not be recognized.
>> >> >
>> >> >Guozhang
>> >> >
>> >> >
>> >> >
>> >> >On Fri, Jun 27, 2014 at 2:13 PM, Virendra Pratap Singh <
>> >> >vpsingh@yahoo-inc.com.invalid> wrote:
>> >> >
>> >> >> Running a mixed 2 broker cluster. Mixed as in one of the broker1
>>is
>> >> >> running 0.8.0 and broker2 one 0.8.1.1 (from the apache release
>>link.
>> >> >> Directly using the tar ball, no local build used).
>> >> >>
>> >> >> I have set the log.retention.minutes=10. However the broker is not
>> >> >> honoring the setting. I see its not cleaning the log.dir at all.
>> >> >>
>> >> >> However when I set the log.retention.hours=1, then it starts
>>cleaning
>> >> >>the
>> >> >> log.
>> >> >>
>> >> >> When I have the log.retention.minutes set in the server.properties
>> >>then
>> >> >>I
>> >> >> see this logged in server.log:
>> >> >>
>> >> >> Š..
>> >> >> [2014-06-27 19:21:06,633] WARN Property log.retention.minutes is
>>not
>> >> >>valid
>> >> >> (kafka.utils.VerifiableProperties)
>> >> >> [2014-06-27 19:21:06,633] WARN Property log.retention.minutes is
>>not
>> >> >>valid
>> >> >> (kafka.utils.VerifiableProperties)
>> >> >> ŠŠ
>> >> >>
>> >> >>
>> >> >> I have set these properties too:
>> >> >>
>> >> >> log.cleaner.enable=true
>> >> >> log.cleanup.policy=delete
>> >> >>
>> >> >>
>> >> >> But I see similar warning logged for these properties too.
>> >> >>
>> >> >> Regards,
>> >> >> Virendra
>> >> >>
>> >> >>
>> >> >
>> >> >
>> >> >--
>> >> >-- Guozhang
>> >>
>> >>
>> >
>> >
>> >--
>> >-- Guozhang
>>
>>
>
>
>--
>-- Guozhang
Re: kafka 0.8.1.1 log.retention.minutes NOT being honored
Posted by Guozhang Wang <wa...@gmail.com>.
Yeah we probably should do that. I am not a committer though so someone
else may help fix this issue?
On Tue, Jul 8, 2014 at 2:10 PM, Jason Rosenberg <jb...@squareup.com> wrote:
> Ah, ok.....it's just no longer documented as such in the config docs?
>
>
> On Tue, Jul 8, 2014 at 4:46 PM, Guozhang Wang <wa...@gmail.com> wrote:
>
> > Jason,
> >
> > getLogRetentionTimeMillis() take either "log.retention.minutes" or
> > "log.retention.hours" and transform the value into milis. So you can
> > specify using either granularity.
> >
> > Guozhang
> >
> >
> > On Tue, Jul 8, 2014 at 1:11 PM, Jason Rosenberg <jb...@squareup.com>
> wrote:
> >
> > > On a related note, in doing the upgrade from 0.8.0, I noticed that the
> > > config property changed from 'log.retention.hours' to
> > > 'log.retention.minutes'. Would it have made more sense to deprecate
> > rather
> > > than replace there?
> > >
> > > Also, I notice that internally, in the KafkaConfig class, it's
> > represented
> > > as logRetentionTimeMillis() (e.g. not hours or minutes). And the
> > per-topic
> > > version is in ms and not minutes. So, it all seems a bit confusing
> there
> > > (is there a reason for this)?
> > >
> > > Jason
> > >
> > >
> > > On Tue, Jul 8, 2014 at 3:54 PM, Guozhang Wang <wa...@gmail.com>
> > wrote:
> > >
> > > > Server properties should affect on only the local instance
> separately.
> > > Are
> > > > you saying the property is not honored even on the 0.8.1 machines?
> > > >
> > > > Guozhang
> > > >
> > > > On Mon, Jul 7, 2014 at 3:55 PM, Virendra Pratap Singh <
> > > > vpsingh@yahoo-inc.com.invalid> wrote:
> > > >
> > > > > By setting this property
> > > > > log.retention.mins=10
> > > > > in the server.properties file, which is passed as argument when
> > > starting
> > > > > the broker.
> > > > >
> > > > > Virendra
> > > > >
> > > > > On 7/7/14, 3:31 PM, "Guozhang Wang" <wa...@gmail.com> wrote:
> > > > >
> > > > > >How do you set the retention.minutes property? Is it through
> > zk-based
> > > > > >topics tool?
> > > > > >
> > > > > >Guozhang
> > > > > >
> > > > > >
> > > > > >On Mon, Jul 7, 2014 at 3:07 PM, Virendra Pratap Singh <
> > > > > >vpsingh@yahoo-inc.com.invalid> wrote:
> > > > > >
> > > > > >> I am running a mixed cluster as I mentioned earlier. 1 broker
> > 0.8.0
> > > > and
> > > > > >> the other 0.8.1.1. Should the retention of topics for partitions
> > > > > >> owned/replicated by the broker running 0.8.1.1 not enforce the
> > > server
> > > > > >> properties settings as defined for that server.
> > > > > >>
> > > > > >> So this brings an interesting question, in case of heterogeneous
> > > > > >> environment (as is in my case, which system parameters will take
> > > > > >> preference/precedence).
> > > > > >>
> > > > > >> Virendra
> > > > > >>
> > > > > >> On 6/30/14, 9:19 AM, "Guozhang Wang" <wa...@gmail.com>
> wrote:
> > > > > >>
> > > > > >> >The retention.minute property is only introduced in 0.8.1:
> > > > > >> >
> > > > > >> >https://issues.apache.org/jira/browse/KAFKA-918
> > > > > >> >
> > > > > >> >if you are running 0.8.0 then it will not be recognized.
> > > > > >> >
> > > > > >> >Guozhang
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> >On Fri, Jun 27, 2014 at 2:13 PM, Virendra Pratap Singh <
> > > > > >> >vpsingh@yahoo-inc.com.invalid> wrote:
> > > > > >> >
> > > > > >> >> Running a mixed 2 broker cluster. Mixed as in one of the
> > broker1
> > > is
> > > > > >> >> running 0.8.0 and broker2 one 0.8.1.1 (from the apache
> release
> > > > link.
> > > > > >> >> Directly using the tar ball, no local build used).
> > > > > >> >>
> > > > > >> >> I have set the log.retention.minutes=10. However the broker
> is
> > > not
> > > > > >> >> honoring the setting. I see its not cleaning the log.dir at
> > all.
> > > > > >> >>
> > > > > >> >> However when I set the log.retention.hours=1, then it starts
> > > > cleaning
> > > > > >> >>the
> > > > > >> >> log.
> > > > > >> >>
> > > > > >> >> When I have the log.retention.minutes set in the
> > > server.properties
> > > > > >>then
> > > > > >> >>I
> > > > > >> >> see this logged in server.log:
> > > > > >> >>
> > > > > >> >> Š..
> > > > > >> >> [2014-06-27 19:21:06,633] WARN Property log.retention.minutes
> > is
> > > > not
> > > > > >> >>valid
> > > > > >> >> (kafka.utils.VerifiableProperties)
> > > > > >> >> [2014-06-27 19:21:06,633] WARN Property log.retention.minutes
> > is
> > > > not
> > > > > >> >>valid
> > > > > >> >> (kafka.utils.VerifiableProperties)
> > > > > >> >> ŠŠ
> > > > > >> >>
> > > > > >> >>
> > > > > >> >> I have set these properties too:
> > > > > >> >>
> > > > > >> >> log.cleaner.enable=true
> > > > > >> >> log.cleanup.policy=delete
> > > > > >> >>
> > > > > >> >>
> > > > > >> >> But I see similar warning logged for these properties too.
> > > > > >> >>
> > > > > >> >> Regards,
> > > > > >> >> Virendra
> > > > > >> >>
> > > > > >> >>
> > > > > >> >
> > > > > >> >
> > > > > >> >--
> > > > > >> >-- Guozhang
> > > > > >>
> > > > > >>
> > > > > >
> > > > > >
> > > > > >--
> > > > > >-- Guozhang
> > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > -- Guozhang
> > > >
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
--
-- Guozhang
Re: kafka 0.8.1.1 log.retention.minutes NOT being honored
Posted by Jason Rosenberg <jb...@squareup.com>.
Ah, ok.....it's just no longer documented as such in the config docs?
On Tue, Jul 8, 2014 at 4:46 PM, Guozhang Wang <wa...@gmail.com> wrote:
> Jason,
>
> getLogRetentionTimeMillis() take either "log.retention.minutes" or
> "log.retention.hours" and transform the value into milis. So you can
> specify using either granularity.
>
> Guozhang
>
>
> On Tue, Jul 8, 2014 at 1:11 PM, Jason Rosenberg <jb...@squareup.com> wrote:
>
> > On a related note, in doing the upgrade from 0.8.0, I noticed that the
> > config property changed from 'log.retention.hours' to
> > 'log.retention.minutes'. Would it have made more sense to deprecate
> rather
> > than replace there?
> >
> > Also, I notice that internally, in the KafkaConfig class, it's
> represented
> > as logRetentionTimeMillis() (e.g. not hours or minutes). And the
> per-topic
> > version is in ms and not minutes. So, it all seems a bit confusing there
> > (is there a reason for this)?
> >
> > Jason
> >
> >
> > On Tue, Jul 8, 2014 at 3:54 PM, Guozhang Wang <wa...@gmail.com>
> wrote:
> >
> > > Server properties should affect on only the local instance separately.
> > Are
> > > you saying the property is not honored even on the 0.8.1 machines?
> > >
> > > Guozhang
> > >
> > > On Mon, Jul 7, 2014 at 3:55 PM, Virendra Pratap Singh <
> > > vpsingh@yahoo-inc.com.invalid> wrote:
> > >
> > > > By setting this property
> > > > log.retention.mins=10
> > > > in the server.properties file, which is passed as argument when
> > starting
> > > > the broker.
> > > >
> > > > Virendra
> > > >
> > > > On 7/7/14, 3:31 PM, "Guozhang Wang" <wa...@gmail.com> wrote:
> > > >
> > > > >How do you set the retention.minutes property? Is it through
> zk-based
> > > > >topics tool?
> > > > >
> > > > >Guozhang
> > > > >
> > > > >
> > > > >On Mon, Jul 7, 2014 at 3:07 PM, Virendra Pratap Singh <
> > > > >vpsingh@yahoo-inc.com.invalid> wrote:
> > > > >
> > > > >> I am running a mixed cluster as I mentioned earlier. 1 broker
> 0.8.0
> > > and
> > > > >> the other 0.8.1.1. Should the retention of topics for partitions
> > > > >> owned/replicated by the broker running 0.8.1.1 not enforce the
> > server
> > > > >> properties settings as defined for that server.
> > > > >>
> > > > >> So this brings an interesting question, in case of heterogeneous
> > > > >> environment (as is in my case, which system parameters will take
> > > > >> preference/precedence).
> > > > >>
> > > > >> Virendra
> > > > >>
> > > > >> On 6/30/14, 9:19 AM, "Guozhang Wang" <wa...@gmail.com> wrote:
> > > > >>
> > > > >> >The retention.minute property is only introduced in 0.8.1:
> > > > >> >
> > > > >> >https://issues.apache.org/jira/browse/KAFKA-918
> > > > >> >
> > > > >> >if you are running 0.8.0 then it will not be recognized.
> > > > >> >
> > > > >> >Guozhang
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> >On Fri, Jun 27, 2014 at 2:13 PM, Virendra Pratap Singh <
> > > > >> >vpsingh@yahoo-inc.com.invalid> wrote:
> > > > >> >
> > > > >> >> Running a mixed 2 broker cluster. Mixed as in one of the
> broker1
> > is
> > > > >> >> running 0.8.0 and broker2 one 0.8.1.1 (from the apache release
> > > link.
> > > > >> >> Directly using the tar ball, no local build used).
> > > > >> >>
> > > > >> >> I have set the log.retention.minutes=10. However the broker is
> > not
> > > > >> >> honoring the setting. I see its not cleaning the log.dir at
> all.
> > > > >> >>
> > > > >> >> However when I set the log.retention.hours=1, then it starts
> > > cleaning
> > > > >> >>the
> > > > >> >> log.
> > > > >> >>
> > > > >> >> When I have the log.retention.minutes set in the
> > server.properties
> > > > >>then
> > > > >> >>I
> > > > >> >> see this logged in server.log:
> > > > >> >>
> > > > >> >> Š..
> > > > >> >> [2014-06-27 19:21:06,633] WARN Property log.retention.minutes
> is
> > > not
> > > > >> >>valid
> > > > >> >> (kafka.utils.VerifiableProperties)
> > > > >> >> [2014-06-27 19:21:06,633] WARN Property log.retention.minutes
> is
> > > not
> > > > >> >>valid
> > > > >> >> (kafka.utils.VerifiableProperties)
> > > > >> >> ŠŠ
> > > > >> >>
> > > > >> >>
> > > > >> >> I have set these properties too:
> > > > >> >>
> > > > >> >> log.cleaner.enable=true
> > > > >> >> log.cleanup.policy=delete
> > > > >> >>
> > > > >> >>
> > > > >> >> But I see similar warning logged for these properties too.
> > > > >> >>
> > > > >> >> Regards,
> > > > >> >> Virendra
> > > > >> >>
> > > > >> >>
> > > > >> >
> > > > >> >
> > > > >> >--
> > > > >> >-- Guozhang
> > > > >>
> > > > >>
> > > > >
> > > > >
> > > > >--
> > > > >-- Guozhang
> > > >
> > > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
>
>
>
> --
> -- Guozhang
>
Re: kafka 0.8.1.1 log.retention.minutes NOT being honored
Posted by Guozhang Wang <wa...@gmail.com>.
Jason,
getLogRetentionTimeMillis() take either "log.retention.minutes" or
"log.retention.hours" and transform the value into milis. So you can
specify using either granularity.
Guozhang
On Tue, Jul 8, 2014 at 1:11 PM, Jason Rosenberg <jb...@squareup.com> wrote:
> On a related note, in doing the upgrade from 0.8.0, I noticed that the
> config property changed from 'log.retention.hours' to
> 'log.retention.minutes'. Would it have made more sense to deprecate rather
> than replace there?
>
> Also, I notice that internally, in the KafkaConfig class, it's represented
> as logRetentionTimeMillis() (e.g. not hours or minutes). And the per-topic
> version is in ms and not minutes. So, it all seems a bit confusing there
> (is there a reason for this)?
>
> Jason
>
>
> On Tue, Jul 8, 2014 at 3:54 PM, Guozhang Wang <wa...@gmail.com> wrote:
>
> > Server properties should affect on only the local instance separately.
> Are
> > you saying the property is not honored even on the 0.8.1 machines?
> >
> > Guozhang
> >
> > On Mon, Jul 7, 2014 at 3:55 PM, Virendra Pratap Singh <
> > vpsingh@yahoo-inc.com.invalid> wrote:
> >
> > > By setting this property
> > > log.retention.mins=10
> > > in the server.properties file, which is passed as argument when
> starting
> > > the broker.
> > >
> > > Virendra
> > >
> > > On 7/7/14, 3:31 PM, "Guozhang Wang" <wa...@gmail.com> wrote:
> > >
> > > >How do you set the retention.minutes property? Is it through zk-based
> > > >topics tool?
> > > >
> > > >Guozhang
> > > >
> > > >
> > > >On Mon, Jul 7, 2014 at 3:07 PM, Virendra Pratap Singh <
> > > >vpsingh@yahoo-inc.com.invalid> wrote:
> > > >
> > > >> I am running a mixed cluster as I mentioned earlier. 1 broker 0.8.0
> > and
> > > >> the other 0.8.1.1. Should the retention of topics for partitions
> > > >> owned/replicated by the broker running 0.8.1.1 not enforce the
> server
> > > >> properties settings as defined for that server.
> > > >>
> > > >> So this brings an interesting question, in case of heterogeneous
> > > >> environment (as is in my case, which system parameters will take
> > > >> preference/precedence).
> > > >>
> > > >> Virendra
> > > >>
> > > >> On 6/30/14, 9:19 AM, "Guozhang Wang" <wa...@gmail.com> wrote:
> > > >>
> > > >> >The retention.minute property is only introduced in 0.8.1:
> > > >> >
> > > >> >https://issues.apache.org/jira/browse/KAFKA-918
> > > >> >
> > > >> >if you are running 0.8.0 then it will not be recognized.
> > > >> >
> > > >> >Guozhang
> > > >> >
> > > >> >
> > > >> >
> > > >> >On Fri, Jun 27, 2014 at 2:13 PM, Virendra Pratap Singh <
> > > >> >vpsingh@yahoo-inc.com.invalid> wrote:
> > > >> >
> > > >> >> Running a mixed 2 broker cluster. Mixed as in one of the broker1
> is
> > > >> >> running 0.8.0 and broker2 one 0.8.1.1 (from the apache release
> > link.
> > > >> >> Directly using the tar ball, no local build used).
> > > >> >>
> > > >> >> I have set the log.retention.minutes=10. However the broker is
> not
> > > >> >> honoring the setting. I see its not cleaning the log.dir at all.
> > > >> >>
> > > >> >> However when I set the log.retention.hours=1, then it starts
> > cleaning
> > > >> >>the
> > > >> >> log.
> > > >> >>
> > > >> >> When I have the log.retention.minutes set in the
> server.properties
> > > >>then
> > > >> >>I
> > > >> >> see this logged in server.log:
> > > >> >>
> > > >> >> Š..
> > > >> >> [2014-06-27 19:21:06,633] WARN Property log.retention.minutes is
> > not
> > > >> >>valid
> > > >> >> (kafka.utils.VerifiableProperties)
> > > >> >> [2014-06-27 19:21:06,633] WARN Property log.retention.minutes is
> > not
> > > >> >>valid
> > > >> >> (kafka.utils.VerifiableProperties)
> > > >> >> ŠŠ
> > > >> >>
> > > >> >>
> > > >> >> I have set these properties too:
> > > >> >>
> > > >> >> log.cleaner.enable=true
> > > >> >> log.cleanup.policy=delete
> > > >> >>
> > > >> >>
> > > >> >> But I see similar warning logged for these properties too.
> > > >> >>
> > > >> >> Regards,
> > > >> >> Virendra
> > > >> >>
> > > >> >>
> > > >> >
> > > >> >
> > > >> >--
> > > >> >-- Guozhang
> > > >>
> > > >>
> > > >
> > > >
> > > >--
> > > >-- Guozhang
> > >
> > >
> >
> >
> > --
> > -- Guozhang
> >
>
--
-- Guozhang
Re: kafka 0.8.1.1 log.retention.minutes NOT being honored
Posted by Jason Rosenberg <jb...@squareup.com>.
On a related note, in doing the upgrade from 0.8.0, I noticed that the
config property changed from 'log.retention.hours' to
'log.retention.minutes'. Would it have made more sense to deprecate rather
than replace there?
Also, I notice that internally, in the KafkaConfig class, it's represented
as logRetentionTimeMillis() (e.g. not hours or minutes). And the per-topic
version is in ms and not minutes. So, it all seems a bit confusing there
(is there a reason for this)?
Jason
On Tue, Jul 8, 2014 at 3:54 PM, Guozhang Wang <wa...@gmail.com> wrote:
> Server properties should affect on only the local instance separately. Are
> you saying the property is not honored even on the 0.8.1 machines?
>
> Guozhang
>
> On Mon, Jul 7, 2014 at 3:55 PM, Virendra Pratap Singh <
> vpsingh@yahoo-inc.com.invalid> wrote:
>
> > By setting this property
> > log.retention.mins=10
> > in the server.properties file, which is passed as argument when starting
> > the broker.
> >
> > Virendra
> >
> > On 7/7/14, 3:31 PM, "Guozhang Wang" <wa...@gmail.com> wrote:
> >
> > >How do you set the retention.minutes property? Is it through zk-based
> > >topics tool?
> > >
> > >Guozhang
> > >
> > >
> > >On Mon, Jul 7, 2014 at 3:07 PM, Virendra Pratap Singh <
> > >vpsingh@yahoo-inc.com.invalid> wrote:
> > >
> > >> I am running a mixed cluster as I mentioned earlier. 1 broker 0.8.0
> and
> > >> the other 0.8.1.1. Should the retention of topics for partitions
> > >> owned/replicated by the broker running 0.8.1.1 not enforce the server
> > >> properties settings as defined for that server.
> > >>
> > >> So this brings an interesting question, in case of heterogeneous
> > >> environment (as is in my case, which system parameters will take
> > >> preference/precedence).
> > >>
> > >> Virendra
> > >>
> > >> On 6/30/14, 9:19 AM, "Guozhang Wang" <wa...@gmail.com> wrote:
> > >>
> > >> >The retention.minute property is only introduced in 0.8.1:
> > >> >
> > >> >https://issues.apache.org/jira/browse/KAFKA-918
> > >> >
> > >> >if you are running 0.8.0 then it will not be recognized.
> > >> >
> > >> >Guozhang
> > >> >
> > >> >
> > >> >
> > >> >On Fri, Jun 27, 2014 at 2:13 PM, Virendra Pratap Singh <
> > >> >vpsingh@yahoo-inc.com.invalid> wrote:
> > >> >
> > >> >> Running a mixed 2 broker cluster. Mixed as in one of the broker1 is
> > >> >> running 0.8.0 and broker2 one 0.8.1.1 (from the apache release
> link.
> > >> >> Directly using the tar ball, no local build used).
> > >> >>
> > >> >> I have set the log.retention.minutes=10. However the broker is not
> > >> >> honoring the setting. I see its not cleaning the log.dir at all.
> > >> >>
> > >> >> However when I set the log.retention.hours=1, then it starts
> cleaning
> > >> >>the
> > >> >> log.
> > >> >>
> > >> >> When I have the log.retention.minutes set in the server.properties
> > >>then
> > >> >>I
> > >> >> see this logged in server.log:
> > >> >>
> > >> >> Š..
> > >> >> [2014-06-27 19:21:06,633] WARN Property log.retention.minutes is
> not
> > >> >>valid
> > >> >> (kafka.utils.VerifiableProperties)
> > >> >> [2014-06-27 19:21:06,633] WARN Property log.retention.minutes is
> not
> > >> >>valid
> > >> >> (kafka.utils.VerifiableProperties)
> > >> >> ŠŠ
> > >> >>
> > >> >>
> > >> >> I have set these properties too:
> > >> >>
> > >> >> log.cleaner.enable=true
> > >> >> log.cleanup.policy=delete
> > >> >>
> > >> >>
> > >> >> But I see similar warning logged for these properties too.
> > >> >>
> > >> >> Regards,
> > >> >> Virendra
> > >> >>
> > >> >>
> > >> >
> > >> >
> > >> >--
> > >> >-- Guozhang
> > >>
> > >>
> > >
> > >
> > >--
> > >-- Guozhang
> >
> >
>
>
> --
> -- Guozhang
>
Re: kafka 0.8.1.1 log.retention.minutes NOT being honored
Posted by Guozhang Wang <wa...@gmail.com>.
Server properties should affect on only the local instance separately. Are
you saying the property is not honored even on the 0.8.1 machines?
Guozhang
On Mon, Jul 7, 2014 at 3:55 PM, Virendra Pratap Singh <
vpsingh@yahoo-inc.com.invalid> wrote:
> By setting this property
> log.retention.mins=10
> in the server.properties file, which is passed as argument when starting
> the broker.
>
> Virendra
>
> On 7/7/14, 3:31 PM, "Guozhang Wang" <wa...@gmail.com> wrote:
>
> >How do you set the retention.minutes property? Is it through zk-based
> >topics tool?
> >
> >Guozhang
> >
> >
> >On Mon, Jul 7, 2014 at 3:07 PM, Virendra Pratap Singh <
> >vpsingh@yahoo-inc.com.invalid> wrote:
> >
> >> I am running a mixed cluster as I mentioned earlier. 1 broker 0.8.0 and
> >> the other 0.8.1.1. Should the retention of topics for partitions
> >> owned/replicated by the broker running 0.8.1.1 not enforce the server
> >> properties settings as defined for that server.
> >>
> >> So this brings an interesting question, in case of heterogeneous
> >> environment (as is in my case, which system parameters will take
> >> preference/precedence).
> >>
> >> Virendra
> >>
> >> On 6/30/14, 9:19 AM, "Guozhang Wang" <wa...@gmail.com> wrote:
> >>
> >> >The retention.minute property is only introduced in 0.8.1:
> >> >
> >> >https://issues.apache.org/jira/browse/KAFKA-918
> >> >
> >> >if you are running 0.8.0 then it will not be recognized.
> >> >
> >> >Guozhang
> >> >
> >> >
> >> >
> >> >On Fri, Jun 27, 2014 at 2:13 PM, Virendra Pratap Singh <
> >> >vpsingh@yahoo-inc.com.invalid> wrote:
> >> >
> >> >> Running a mixed 2 broker cluster. Mixed as in one of the broker1 is
> >> >> running 0.8.0 and broker2 one 0.8.1.1 (from the apache release link.
> >> >> Directly using the tar ball, no local build used).
> >> >>
> >> >> I have set the log.retention.minutes=10. However the broker is not
> >> >> honoring the setting. I see its not cleaning the log.dir at all.
> >> >>
> >> >> However when I set the log.retention.hours=1, then it starts cleaning
> >> >>the
> >> >> log.
> >> >>
> >> >> When I have the log.retention.minutes set in the server.properties
> >>then
> >> >>I
> >> >> see this logged in server.log:
> >> >>
> >> >> Š..
> >> >> [2014-06-27 19:21:06,633] WARN Property log.retention.minutes is not
> >> >>valid
> >> >> (kafka.utils.VerifiableProperties)
> >> >> [2014-06-27 19:21:06,633] WARN Property log.retention.minutes is not
> >> >>valid
> >> >> (kafka.utils.VerifiableProperties)
> >> >> ŠŠ
> >> >>
> >> >>
> >> >> I have set these properties too:
> >> >>
> >> >> log.cleaner.enable=true
> >> >> log.cleanup.policy=delete
> >> >>
> >> >>
> >> >> But I see similar warning logged for these properties too.
> >> >>
> >> >> Regards,
> >> >> Virendra
> >> >>
> >> >>
> >> >
> >> >
> >> >--
> >> >-- Guozhang
> >>
> >>
> >
> >
> >--
> >-- Guozhang
>
>
--
-- Guozhang
Re: kafka 0.8.1.1 log.retention.minutes NOT being honored
Posted by Virendra Pratap Singh <vp...@yahoo-inc.com.INVALID>.
By setting this property
log.retention.mins=10
in the server.properties file, which is passed as argument when starting
the broker.
Virendra
On 7/7/14, 3:31 PM, "Guozhang Wang" <wa...@gmail.com> wrote:
>How do you set the retention.minutes property? Is it through zk-based
>topics tool?
>
>Guozhang
>
>
>On Mon, Jul 7, 2014 at 3:07 PM, Virendra Pratap Singh <
>vpsingh@yahoo-inc.com.invalid> wrote:
>
>> I am running a mixed cluster as I mentioned earlier. 1 broker 0.8.0 and
>> the other 0.8.1.1. Should the retention of topics for partitions
>> owned/replicated by the broker running 0.8.1.1 not enforce the server
>> properties settings as defined for that server.
>>
>> So this brings an interesting question, in case of heterogeneous
>> environment (as is in my case, which system parameters will take
>> preference/precedence).
>>
>> Virendra
>>
>> On 6/30/14, 9:19 AM, "Guozhang Wang" <wa...@gmail.com> wrote:
>>
>> >The retention.minute property is only introduced in 0.8.1:
>> >
>> >https://issues.apache.org/jira/browse/KAFKA-918
>> >
>> >if you are running 0.8.0 then it will not be recognized.
>> >
>> >Guozhang
>> >
>> >
>> >
>> >On Fri, Jun 27, 2014 at 2:13 PM, Virendra Pratap Singh <
>> >vpsingh@yahoo-inc.com.invalid> wrote:
>> >
>> >> Running a mixed 2 broker cluster. Mixed as in one of the broker1 is
>> >> running 0.8.0 and broker2 one 0.8.1.1 (from the apache release link.
>> >> Directly using the tar ball, no local build used).
>> >>
>> >> I have set the log.retention.minutes=10. However the broker is not
>> >> honoring the setting. I see its not cleaning the log.dir at all.
>> >>
>> >> However when I set the log.retention.hours=1, then it starts cleaning
>> >>the
>> >> log.
>> >>
>> >> When I have the log.retention.minutes set in the server.properties
>>then
>> >>I
>> >> see this logged in server.log:
>> >>
>> >> Š..
>> >> [2014-06-27 19:21:06,633] WARN Property log.retention.minutes is not
>> >>valid
>> >> (kafka.utils.VerifiableProperties)
>> >> [2014-06-27 19:21:06,633] WARN Property log.retention.minutes is not
>> >>valid
>> >> (kafka.utils.VerifiableProperties)
>> >> ŠŠ
>> >>
>> >>
>> >> I have set these properties too:
>> >>
>> >> log.cleaner.enable=true
>> >> log.cleanup.policy=delete
>> >>
>> >>
>> >> But I see similar warning logged for these properties too.
>> >>
>> >> Regards,
>> >> Virendra
>> >>
>> >>
>> >
>> >
>> >--
>> >-- Guozhang
>>
>>
>
>
>--
>-- Guozhang
Re: kafka 0.8.1.1 log.retention.minutes NOT being honored
Posted by Guozhang Wang <wa...@gmail.com>.
How do you set the retention.minutes property? Is it through zk-based
topics tool?
Guozhang
On Mon, Jul 7, 2014 at 3:07 PM, Virendra Pratap Singh <
vpsingh@yahoo-inc.com.invalid> wrote:
> I am running a mixed cluster as I mentioned earlier. 1 broker 0.8.0 and
> the other 0.8.1.1. Should the retention of topics for partitions
> owned/replicated by the broker running 0.8.1.1 not enforce the server
> properties settings as defined for that server.
>
> So this brings an interesting question, in case of heterogeneous
> environment (as is in my case, which system parameters will take
> preference/precedence).
>
> Virendra
>
> On 6/30/14, 9:19 AM, "Guozhang Wang" <wa...@gmail.com> wrote:
>
> >The retention.minute property is only introduced in 0.8.1:
> >
> >https://issues.apache.org/jira/browse/KAFKA-918
> >
> >if you are running 0.8.0 then it will not be recognized.
> >
> >Guozhang
> >
> >
> >
> >On Fri, Jun 27, 2014 at 2:13 PM, Virendra Pratap Singh <
> >vpsingh@yahoo-inc.com.invalid> wrote:
> >
> >> Running a mixed 2 broker cluster. Mixed as in one of the broker1 is
> >> running 0.8.0 and broker2 one 0.8.1.1 (from the apache release link.
> >> Directly using the tar ball, no local build used).
> >>
> >> I have set the log.retention.minutes=10. However the broker is not
> >> honoring the setting. I see its not cleaning the log.dir at all.
> >>
> >> However when I set the log.retention.hours=1, then it starts cleaning
> >>the
> >> log.
> >>
> >> When I have the log.retention.minutes set in the server.properties then
> >>I
> >> see this logged in server.log:
> >>
> >> Š..
> >> [2014-06-27 19:21:06,633] WARN Property log.retention.minutes is not
> >>valid
> >> (kafka.utils.VerifiableProperties)
> >> [2014-06-27 19:21:06,633] WARN Property log.retention.minutes is not
> >>valid
> >> (kafka.utils.VerifiableProperties)
> >> ŠŠ
> >>
> >>
> >> I have set these properties too:
> >>
> >> log.cleaner.enable=true
> >> log.cleanup.policy=delete
> >>
> >>
> >> But I see similar warning logged for these properties too.
> >>
> >> Regards,
> >> Virendra
> >>
> >>
> >
> >
> >--
> >-- Guozhang
>
>
--
-- Guozhang
Re: kafka 0.8.1.1 log.retention.minutes NOT being honored
Posted by Virendra Pratap Singh <vp...@yahoo-inc.com.INVALID>.
I am running a mixed cluster as I mentioned earlier. 1 broker 0.8.0 and
the other 0.8.1.1. Should the retention of topics for partitions
owned/replicated by the broker running 0.8.1.1 not enforce the server
properties settings as defined for that server.
So this brings an interesting question, in case of heterogeneous
environment (as is in my case, which system parameters will take
preference/precedence).
Virendra
On 6/30/14, 9:19 AM, "Guozhang Wang" <wa...@gmail.com> wrote:
>The retention.minute property is only introduced in 0.8.1:
>
>https://issues.apache.org/jira/browse/KAFKA-918
>
>if you are running 0.8.0 then it will not be recognized.
>
>Guozhang
>
>
>
>On Fri, Jun 27, 2014 at 2:13 PM, Virendra Pratap Singh <
>vpsingh@yahoo-inc.com.invalid> wrote:
>
>> Running a mixed 2 broker cluster. Mixed as in one of the broker1 is
>> running 0.8.0 and broker2 one 0.8.1.1 (from the apache release link.
>> Directly using the tar ball, no local build used).
>>
>> I have set the log.retention.minutes=10. However the broker is not
>> honoring the setting. I see its not cleaning the log.dir at all.
>>
>> However when I set the log.retention.hours=1, then it starts cleaning
>>the
>> log.
>>
>> When I have the log.retention.minutes set in the server.properties then
>>I
>> see this logged in server.log:
>>
>> Š..
>> [2014-06-27 19:21:06,633] WARN Property log.retention.minutes is not
>>valid
>> (kafka.utils.VerifiableProperties)
>> [2014-06-27 19:21:06,633] WARN Property log.retention.minutes is not
>>valid
>> (kafka.utils.VerifiableProperties)
>> ŠŠ
>>
>>
>> I have set these properties too:
>>
>> log.cleaner.enable=true
>> log.cleanup.policy=delete
>>
>>
>> But I see similar warning logged for these properties too.
>>
>> Regards,
>> Virendra
>>
>>
>
>
>--
>-- Guozhang
Re: kafka 0.8.1.1 log.retention.minutes NOT being honored
Posted by Guozhang Wang <wa...@gmail.com>.
The retention.minute property is only introduced in 0.8.1:
https://issues.apache.org/jira/browse/KAFKA-918
if you are running 0.8.0 then it will not be recognized.
Guozhang
On Fri, Jun 27, 2014 at 2:13 PM, Virendra Pratap Singh <
vpsingh@yahoo-inc.com.invalid> wrote:
> Running a mixed 2 broker cluster. Mixed as in one of the broker1 is
> running 0.8.0 and broker2 one 0.8.1.1 (from the apache release link.
> Directly using the tar ball, no local build used).
>
> I have set the log.retention.minutes=10. However the broker is not
> honoring the setting. I see its not cleaning the log.dir at all.
>
> However when I set the log.retention.hours=1, then it starts cleaning the
> log.
>
> When I have the log.retention.minutes set in the server.properties then I
> see this logged in server.log:
>
> Š..
> [2014-06-27 19:21:06,633] WARN Property log.retention.minutes is not valid
> (kafka.utils.VerifiableProperties)
> [2014-06-27 19:21:06,633] WARN Property log.retention.minutes is not valid
> (kafka.utils.VerifiableProperties)
> ŠŠ
>
>
> I have set these properties too:
>
> log.cleaner.enable=true
> log.cleanup.policy=delete
>
>
> But I see similar warning logged for these properties too.
>
> Regards,
> Virendra
>
>
--
-- Guozhang