You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Ben Summer <bs...@gnipcentral.com> on 2014/01/17 00:15:06 UTC
log.retention.bytes.per.topic does not trigger deletion
I tried using the following two retention properties
log.retention.bytes=3221225472
log.retention.bytes.per.topic=first_topic:1099511627776,second_topic:1099511627776
which I interpret to mean "by default, keep 3GB per topic partition, except
for first_topic and second_topic, which should retain 1TB each (across all
their partitions).
I've noticed that the data for first_topic and second_topic grow way beyond
1TB and no deletions occur. Is the syntax correct?
Thanks in advance,
Ben
Re: log.retention.bytes.per.topic does not trigger deletion
Posted by Guozhang Wang <wa...@gmail.com>.
Yes, once 0.8.1 is released we will add a separate quick-start page.
On Sun, Jan 19, 2014 at 10:57 PM, Joe Stein <jo...@stealth.ly> wrote:
> There are a lot of nuance changes to the QuickStart and other changes for
> 0.8.1 that will need a separate page... Not just modifications.
>
>
> /*******************************************
> Joe Stein
> Founder, Principal Consultant
> Big Data Open Source Security LLC
> http://www.stealth.ly
> Twitter: @allthingshadoop
> ********************************************/
>
>
> On Jan 19, 2014, at 10:19 PM, Jason Rosenberg <jb...@squareup.com> wrote:
>
> > Please be sure to update the online config docs with this change! The
> > per topic options are still listed there....
> >
> > Jason
> >
> > On Thu, Jan 16, 2014 at 9:57 PM, Ben Summer <bs...@gnipcentral.com>
> wrote:
> >> I see. I don't have version 0.8.1 yet. We just updated to 0.8.0 from
> beta
> >> after it became the "stable version".
> >> Good to know there is a fix for this. I'll start trying it out in some
> >> non-production environments.
> >>
> >> Thanks,
> >> Ben
> >>
> >>
> >> On Thu, Jan 16, 2014 at 7:42 PM, Guozhang Wang <wa...@gmail.com>
> wrote:
> >>
> >>> In the latest version, per-topic configs have been moved to Zookeeper
> you
> >>> and set them using admin tools instead of writing the config files.
> Could
> >>> you try trunk HEAD and see if this issue has already been resolved:
> >>>
> >>> https://issues.apache.org/jira/browse/KAFKA-554
> >>>
> >>> Guozhang
> >>>
> >>>
> >>> On Thu, Jan 16, 2014 at 6:27 PM, Ben Summer <bs...@gnipcentral.com>
> >>> wrote:
> >>>
> >>>> v0.8.0, (non-beta) downloaded from the website. Let me know if there
> is a
> >>>> file I can check in the installation footprint that can give you an
> >>> actual
> >>>> change id.
> >>>>
> >>>> Is there a fix to this particular feature that you know was made in a
> >>> later
> >>>> build?
> >>>>
> >>>> Thanks for the quick response,
> >>>> Ben
> >>>>
> >>>>
> >>>> On Thu, Jan 16, 2014 at 6:22 PM, Guozhang Wang <wa...@gmail.com>
> >>> wrote:
> >>>>
> >>>>> Hello Ben,
> >>>>>
> >>>>> Which version are you using?
> >>>>>
> >>>>> Guozhang
> >>>>>
> >>>>>
> >>>>> On Thu, Jan 16, 2014 at 3:15 PM, Ben Summer <bsummer@gnipcentral.com
> >
> >>>>> wrote:
> >>>>>
> >>>>>> I tried using the following two retention properties
> >>>>>>
> >>>>>> log.retention.bytes=3221225472
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> log.retention.bytes.per.topic=first_topic:1099511627776,second_topic:1099511627776
> >>>>>>
> >>>>>> which I interpret to mean "by default, keep 3GB per topic partition,
> >>>>> except
> >>>>>> for first_topic and second_topic, which should retain 1TB each
> >>> (across
> >>>>> all
> >>>>>> their partitions).
> >>>>>>
> >>>>>> I've noticed that the data for first_topic and second_topic grow way
> >>>>> beyond
> >>>>>> 1TB and no deletions occur. Is the syntax correct?
> >>>>>>
> >>>>>> Thanks in advance,
> >>>>>> Ben
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> -- Guozhang
> >>>>>
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> -- Guozhang
> >>>
>
--
-- Guozhang
Re: log.retention.bytes.per.topic does not trigger deletion
Posted by Joe Stein <jo...@stealth.ly>.
There are a lot of nuance changes to the QuickStart and other changes for 0.8.1 that will need a separate page... Not just modifications.
/*******************************************
Joe Stein
Founder, Principal Consultant
Big Data Open Source Security LLC
http://www.stealth.ly
Twitter: @allthingshadoop
********************************************/
On Jan 19, 2014, at 10:19 PM, Jason Rosenberg <jb...@squareup.com> wrote:
> Please be sure to update the online config docs with this change! The
> per topic options are still listed there....
>
> Jason
>
> On Thu, Jan 16, 2014 at 9:57 PM, Ben Summer <bs...@gnipcentral.com> wrote:
>> I see. I don't have version 0.8.1 yet. We just updated to 0.8.0 from beta
>> after it became the "stable version".
>> Good to know there is a fix for this. I'll start trying it out in some
>> non-production environments.
>>
>> Thanks,
>> Ben
>>
>>
>> On Thu, Jan 16, 2014 at 7:42 PM, Guozhang Wang <wa...@gmail.com> wrote:
>>
>>> In the latest version, per-topic configs have been moved to Zookeeper you
>>> and set them using admin tools instead of writing the config files. Could
>>> you try trunk HEAD and see if this issue has already been resolved:
>>>
>>> https://issues.apache.org/jira/browse/KAFKA-554
>>>
>>> Guozhang
>>>
>>>
>>> On Thu, Jan 16, 2014 at 6:27 PM, Ben Summer <bs...@gnipcentral.com>
>>> wrote:
>>>
>>>> v0.8.0, (non-beta) downloaded from the website. Let me know if there is a
>>>> file I can check in the installation footprint that can give you an
>>> actual
>>>> change id.
>>>>
>>>> Is there a fix to this particular feature that you know was made in a
>>> later
>>>> build?
>>>>
>>>> Thanks for the quick response,
>>>> Ben
>>>>
>>>>
>>>> On Thu, Jan 16, 2014 at 6:22 PM, Guozhang Wang <wa...@gmail.com>
>>> wrote:
>>>>
>>>>> Hello Ben,
>>>>>
>>>>> Which version are you using?
>>>>>
>>>>> Guozhang
>>>>>
>>>>>
>>>>> On Thu, Jan 16, 2014 at 3:15 PM, Ben Summer <bs...@gnipcentral.com>
>>>>> wrote:
>>>>>
>>>>>> I tried using the following two retention properties
>>>>>>
>>>>>> log.retention.bytes=3221225472
>>>>>>
>>>>>>
>>>>>
>>>>
>>> log.retention.bytes.per.topic=first_topic:1099511627776,second_topic:1099511627776
>>>>>>
>>>>>> which I interpret to mean "by default, keep 3GB per topic partition,
>>>>> except
>>>>>> for first_topic and second_topic, which should retain 1TB each
>>> (across
>>>>> all
>>>>>> their partitions).
>>>>>>
>>>>>> I've noticed that the data for first_topic and second_topic grow way
>>>>> beyond
>>>>>> 1TB and no deletions occur. Is the syntax correct?
>>>>>>
>>>>>> Thanks in advance,
>>>>>> Ben
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> -- Guozhang
>>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> -- Guozhang
>>>
Re: log.retention.bytes.per.topic does not trigger deletion
Posted by Jason Rosenberg <jb...@squareup.com>.
Please be sure to update the online config docs with this change! The
per topic options are still listed there....
Jason
On Thu, Jan 16, 2014 at 9:57 PM, Ben Summer <bs...@gnipcentral.com> wrote:
> I see. I don't have version 0.8.1 yet. We just updated to 0.8.0 from beta
> after it became the "stable version".
> Good to know there is a fix for this. I'll start trying it out in some
> non-production environments.
>
> Thanks,
> Ben
>
>
> On Thu, Jan 16, 2014 at 7:42 PM, Guozhang Wang <wa...@gmail.com> wrote:
>
>> In the latest version, per-topic configs have been moved to Zookeeper you
>> and set them using admin tools instead of writing the config files. Could
>> you try trunk HEAD and see if this issue has already been resolved:
>>
>> https://issues.apache.org/jira/browse/KAFKA-554
>>
>> Guozhang
>>
>>
>> On Thu, Jan 16, 2014 at 6:27 PM, Ben Summer <bs...@gnipcentral.com>
>> wrote:
>>
>> > v0.8.0, (non-beta) downloaded from the website. Let me know if there is a
>> > file I can check in the installation footprint that can give you an
>> actual
>> > change id.
>> >
>> > Is there a fix to this particular feature that you know was made in a
>> later
>> > build?
>> >
>> > Thanks for the quick response,
>> > Ben
>> >
>> >
>> > On Thu, Jan 16, 2014 at 6:22 PM, Guozhang Wang <wa...@gmail.com>
>> wrote:
>> >
>> > > Hello Ben,
>> > >
>> > > Which version are you using?
>> > >
>> > > Guozhang
>> > >
>> > >
>> > > On Thu, Jan 16, 2014 at 3:15 PM, Ben Summer <bs...@gnipcentral.com>
>> > > wrote:
>> > >
>> > > > I tried using the following two retention properties
>> > > >
>> > > > log.retention.bytes=3221225472
>> > > >
>> > > >
>> > >
>> >
>> log.retention.bytes.per.topic=first_topic:1099511627776,second_topic:1099511627776
>> > > >
>> > > > which I interpret to mean "by default, keep 3GB per topic partition,
>> > > except
>> > > > for first_topic and second_topic, which should retain 1TB each
>> (across
>> > > all
>> > > > their partitions).
>> > > >
>> > > > I've noticed that the data for first_topic and second_topic grow way
>> > > beyond
>> > > > 1TB and no deletions occur. Is the syntax correct?
>> > > >
>> > > > Thanks in advance,
>> > > > Ben
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > > -- Guozhang
>> > >
>> >
>>
>>
>>
>> --
>> -- Guozhang
>>
Re: log.retention.bytes.per.topic does not trigger deletion
Posted by Ben Summer <bs...@gnipcentral.com>.
I see. I don't have version 0.8.1 yet. We just updated to 0.8.0 from beta
after it became the "stable version".
Good to know there is a fix for this. I'll start trying it out in some
non-production environments.
Thanks,
Ben
On Thu, Jan 16, 2014 at 7:42 PM, Guozhang Wang <wa...@gmail.com> wrote:
> In the latest version, per-topic configs have been moved to Zookeeper you
> and set them using admin tools instead of writing the config files. Could
> you try trunk HEAD and see if this issue has already been resolved:
>
> https://issues.apache.org/jira/browse/KAFKA-554
>
> Guozhang
>
>
> On Thu, Jan 16, 2014 at 6:27 PM, Ben Summer <bs...@gnipcentral.com>
> wrote:
>
> > v0.8.0, (non-beta) downloaded from the website. Let me know if there is a
> > file I can check in the installation footprint that can give you an
> actual
> > change id.
> >
> > Is there a fix to this particular feature that you know was made in a
> later
> > build?
> >
> > Thanks for the quick response,
> > Ben
> >
> >
> > On Thu, Jan 16, 2014 at 6:22 PM, Guozhang Wang <wa...@gmail.com>
> wrote:
> >
> > > Hello Ben,
> > >
> > > Which version are you using?
> > >
> > > Guozhang
> > >
> > >
> > > On Thu, Jan 16, 2014 at 3:15 PM, Ben Summer <bs...@gnipcentral.com>
> > > wrote:
> > >
> > > > I tried using the following two retention properties
> > > >
> > > > log.retention.bytes=3221225472
> > > >
> > > >
> > >
> >
> log.retention.bytes.per.topic=first_topic:1099511627776,second_topic:1099511627776
> > > >
> > > > which I interpret to mean "by default, keep 3GB per topic partition,
> > > except
> > > > for first_topic and second_topic, which should retain 1TB each
> (across
> > > all
> > > > their partitions).
> > > >
> > > > I've noticed that the data for first_topic and second_topic grow way
> > > beyond
> > > > 1TB and no deletions occur. Is the syntax correct?
> > > >
> > > > Thanks in advance,
> > > > Ben
> > > >
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
>
>
>
> --
> -- Guozhang
>
Re: log.retention.bytes.per.topic does not trigger deletion
Posted by Guozhang Wang <wa...@gmail.com>.
In the latest version, per-topic configs have been moved to Zookeeper you
and set them using admin tools instead of writing the config files. Could
you try trunk HEAD and see if this issue has already been resolved:
https://issues.apache.org/jira/browse/KAFKA-554
Guozhang
On Thu, Jan 16, 2014 at 6:27 PM, Ben Summer <bs...@gnipcentral.com> wrote:
> v0.8.0, (non-beta) downloaded from the website. Let me know if there is a
> file I can check in the installation footprint that can give you an actual
> change id.
>
> Is there a fix to this particular feature that you know was made in a later
> build?
>
> Thanks for the quick response,
> Ben
>
>
> On Thu, Jan 16, 2014 at 6:22 PM, Guozhang Wang <wa...@gmail.com> wrote:
>
> > Hello Ben,
> >
> > Which version are you using?
> >
> > Guozhang
> >
> >
> > On Thu, Jan 16, 2014 at 3:15 PM, Ben Summer <bs...@gnipcentral.com>
> > wrote:
> >
> > > I tried using the following two retention properties
> > >
> > > log.retention.bytes=3221225472
> > >
> > >
> >
> log.retention.bytes.per.topic=first_topic:1099511627776,second_topic:1099511627776
> > >
> > > which I interpret to mean "by default, keep 3GB per topic partition,
> > except
> > > for first_topic and second_topic, which should retain 1TB each (across
> > all
> > > their partitions).
> > >
> > > I've noticed that the data for first_topic and second_topic grow way
> > beyond
> > > 1TB and no deletions occur. Is the syntax correct?
> > >
> > > Thanks in advance,
> > > Ben
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
--
-- Guozhang
Re: log.retention.bytes.per.topic does not trigger deletion
Posted by Ben Summer <bs...@gnipcentral.com>.
v0.8.0, (non-beta) downloaded from the website. Let me know if there is a
file I can check in the installation footprint that can give you an actual
change id.
Is there a fix to this particular feature that you know was made in a later
build?
Thanks for the quick response,
Ben
On Thu, Jan 16, 2014 at 6:22 PM, Guozhang Wang <wa...@gmail.com> wrote:
> Hello Ben,
>
> Which version are you using?
>
> Guozhang
>
>
> On Thu, Jan 16, 2014 at 3:15 PM, Ben Summer <bs...@gnipcentral.com>
> wrote:
>
> > I tried using the following two retention properties
> >
> > log.retention.bytes=3221225472
> >
> >
> log.retention.bytes.per.topic=first_topic:1099511627776,second_topic:1099511627776
> >
> > which I interpret to mean "by default, keep 3GB per topic partition,
> except
> > for first_topic and second_topic, which should retain 1TB each (across
> all
> > their partitions).
> >
> > I've noticed that the data for first_topic and second_topic grow way
> beyond
> > 1TB and no deletions occur. Is the syntax correct?
> >
> > Thanks in advance,
> > Ben
> >
>
>
>
> --
> -- Guozhang
>
Re: log.retention.bytes.per.topic does not trigger deletion
Posted by Guozhang Wang <wa...@gmail.com>.
Hello Ben,
Which version are you using?
Guozhang
On Thu, Jan 16, 2014 at 3:15 PM, Ben Summer <bs...@gnipcentral.com> wrote:
> I tried using the following two retention properties
>
> log.retention.bytes=3221225472
>
> log.retention.bytes.per.topic=first_topic:1099511627776,second_topic:1099511627776
>
> which I interpret to mean "by default, keep 3GB per topic partition, except
> for first_topic and second_topic, which should retain 1TB each (across all
> their partitions).
>
> I've noticed that the data for first_topic and second_topic grow way beyond
> 1TB and no deletions occur. Is the syntax correct?
>
> Thanks in advance,
> Ben
>
--
-- Guozhang