You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Jun Rao <ju...@gmail.com> on 2014/03/25 17:18:19 UTC

Re: dynamic configure per topic problem

The unit is milli-sec. The deletion only happens when the log segment is
rolled though.

Thanks,

Jun


On Tue, Mar 25, 2014 at 12:59 AM, 陈小军 <ch...@nhn.com> wrote:

> Hi,
>    Today I test the dynamic configure for each topic, I set the '
> retention.ms' attribution for one topic,
>         [irteam@xseed170.kdev bin]$ ./kafka-topics.sh --describe
> --zookeeper 10.96.250.215:10013/nelo2-kafka --topic nelo2-crash-logs
> Topic:nelo2-crash-logs  PartitionCount:6        ReplicationFactor:2
> Configs:retention.ms=2880
>         Topic: nelo2-crash-logs Partition: 0    Leader: 0       Replicas:
> 0,2   Isr: 2,0
>         Topic: nelo2-crash-logs Partition: 1    Leader: 1       Replicas:
> 1,0   Isr: 0,1
>         Topic: nelo2-crash-logs Partition: 2    Leader: 2       Replicas:
> 2,1   Isr: 2,1
>         Topic: nelo2-crash-logs Partition: 3    Leader: 1       Replicas:
> 1,2   Isr: 1,2
>         Topic: nelo2-crash-logs Partition: 4    Leader: 2       Replicas:
> 2,1   Isr: 2,1
>         Topic: nelo2-crash-logs Partition: 5    Leader: 0       Replicas:
> 0,2   Isr: 0,2
>
> in my mind it should delete my logs per 48 hours, actually it will delete
> my logs per 5 minutes. What's the unit of 'retention.ms', second, minute
> or this is a bug?
>
> Thanks~!
>
> Best Regards
> Jerry
>

Re: dynamic configure per topic problem

Posted by 陈小军 <ch...@nhn.com>.
Thanks. I got it. 

Best Regards
Jerry
-----Original Message-----
From: "Neha Narkhede"&lt;neha.narkhede@gmail.com&gt; 
To: "users@kafka.apache.org"&lt;users@kafka.apache.org&gt;; 
Cc: "陈小军"&lt;chenxiaojun@nhn.com&gt;; 
Sent: 2014-03-26 (星期三) 02:08:46
Subject: Re: dynamic configure per topic problem
 
Filed https://issues.apache.org/jira/browse/KAFKA-1325 to track this. 

On Tue, Mar 25, 2014 at 9:31 AM, Neha Narkhede &lt;neha.narkhede@gmail.com&gt; wrote:

What's the unit of 'retention.ms', second, minute or this is a bug?

 Our documentation is a little confusing on the log configs. See this-



 The log property is in ms but the server default it maps to is in minutes. Same is true for segment.ms as well. We could either improve the docs or change the per-topic configs to be consistent with the server defaults. 




On Tue, Mar 25, 2014 at 9:18 AM, Jun Rao &lt;junrao@gmail.com&gt; wrote:


The unit is milli-sec. The deletion only happens when the log segment is

rolled though.



Thanks,



Jun





On Tue, Mar 25, 2014 at 12:59 AM, 陈小军 &lt;chenxiaojun@nhn.com&gt; wrote:



&gt; Hi,

&gt;    Today I test the dynamic configure for each topic, I set the '

&gt; retention.ms' attribution for one topic,

&gt;         [irteam@xseed170.kdev bin]$ ./kafka-topics.sh --describe

&gt; --zookeeper 10.96.250.215:10013/nelo2-kafka --topic nelo2-crash-logs

&gt; Topic:nelo2-crash-logs  PartitionCount:6        ReplicationFactor:2

&gt; Configs:retention.ms=2880

&gt;         Topic: nelo2-crash-logs Partition: 0    Leader: 0       Replicas:

&gt; 0,2   Isr: 2,0

&gt;         Topic: nelo2-crash-logs Partition: 1    Leader: 1       Replicas:

&gt; 1,0   Isr: 0,1

&gt;         Topic: nelo2-crash-logs Partition: 2    Leader: 2       Replicas:

&gt; 2,1   Isr: 2,1

&gt;         Topic: nelo2-crash-logs Partition: 3    Leader: 1       Replicas:

&gt; 1,2   Isr: 1,2

&gt;         Topic: nelo2-crash-logs Partition: 4    Leader: 2       Replicas:

&gt; 2,1   Isr: 2,1

&gt;         Topic: nelo2-crash-logs Partition: 5    Leader: 0       Replicas:

&gt; 0,2   Isr: 0,2

&gt;

&gt; in my mind it should delete my logs per 48 hours, actually it will delete

&gt; my logs per 5 minutes. What's the unit of 'retention.ms', second, minute

&gt; or this is a bug?

&gt;

&gt; Thanks~!

&gt;

&gt; Best Regards

&gt; Jerry

&gt;

 
 



Re: dynamic configure per topic problem

Posted by Neha Narkhede <ne...@gmail.com>.
Filed https://issues.apache.org/jira/browse/KAFKA-1325 to track this.


On Tue, Mar 25, 2014 at 9:31 AM, Neha Narkhede <ne...@gmail.com>wrote:

> What's the unit of 'retention.ms', second, minute or this is a bug?
>
> Our documentation is a little confusing on the log configs. See this-
>
> [image: Inline image 1]
>
> The log property is in ms but the server default it maps to is in minutes.
> Same is true for segment.ms as well. We could either improve the docs or
> change the per-topic configs to be consistent with the server defaults.
>
>
> On Tue, Mar 25, 2014 at 9:18 AM, Jun Rao <ju...@gmail.com> wrote:
>
>> The unit is milli-sec. The deletion only happens when the log segment is
>> rolled though.
>>
>> Thanks,
>>
>> Jun
>>
>>
>> On Tue, Mar 25, 2014 at 12:59 AM, 陈小军 <ch...@nhn.com> wrote:
>>
>> > Hi,
>> >    Today I test the dynamic configure for each topic, I set the '
>> > retention.ms' attribution for one topic,
>> >         [irteam@xseed170.kdev bin]$ ./kafka-topics.sh --describe
>> > --zookeeper 10.96.250.215:10013/nelo2-kafka --topic nelo2-crash-logs
>> > Topic:nelo2-crash-logs  PartitionCount:6        ReplicationFactor:2
>> > Configs:retention.ms=2880
>> >         Topic: nelo2-crash-logs Partition: 0    Leader: 0
>> Replicas:
>> > 0,2   Isr: 2,0
>> >         Topic: nelo2-crash-logs Partition: 1    Leader: 1
>> Replicas:
>> > 1,0   Isr: 0,1
>> >         Topic: nelo2-crash-logs Partition: 2    Leader: 2
>> Replicas:
>> > 2,1   Isr: 2,1
>> >         Topic: nelo2-crash-logs Partition: 3    Leader: 1
>> Replicas:
>> > 1,2   Isr: 1,2
>> >         Topic: nelo2-crash-logs Partition: 4    Leader: 2
>> Replicas:
>> > 2,1   Isr: 2,1
>> >         Topic: nelo2-crash-logs Partition: 5    Leader: 0
>> Replicas:
>> > 0,2   Isr: 0,2
>> >
>> > in my mind it should delete my logs per 48 hours, actually it will
>> delete
>> > my logs per 5 minutes. What's the unit of 'retention.ms', second,
>> minute
>> > or this is a bug?
>> >
>> > Thanks~!
>> >
>> > Best Regards
>> > Jerry
>> >
>>
>
>

Re: dynamic configure per topic problem

Posted by Neha Narkhede <ne...@gmail.com>.
What's the unit of 'retention.ms', second, minute or this is a bug?

Our documentation is a little confusing on the log configs. See this-

[image: Inline image 1]

The log property is in ms but the server default it maps to is in minutes.
Same is true for segment.ms as well. We could either improve the docs or
change the per-topic configs to be consistent with the server defaults.


On Tue, Mar 25, 2014 at 9:18 AM, Jun Rao <ju...@gmail.com> wrote:

> The unit is milli-sec. The deletion only happens when the log segment is
> rolled though.
>
> Thanks,
>
> Jun
>
>
> On Tue, Mar 25, 2014 at 12:59 AM, 陈小军 <ch...@nhn.com> wrote:
>
> > Hi,
> >    Today I test the dynamic configure for each topic, I set the '
> > retention.ms' attribution for one topic,
> >         [irteam@xseed170.kdev bin]$ ./kafka-topics.sh --describe
> > --zookeeper 10.96.250.215:10013/nelo2-kafka --topic nelo2-crash-logs
> > Topic:nelo2-crash-logs  PartitionCount:6        ReplicationFactor:2
> > Configs:retention.ms=2880
> >         Topic: nelo2-crash-logs Partition: 0    Leader: 0       Replicas:
> > 0,2   Isr: 2,0
> >         Topic: nelo2-crash-logs Partition: 1    Leader: 1       Replicas:
> > 1,0   Isr: 0,1
> >         Topic: nelo2-crash-logs Partition: 2    Leader: 2       Replicas:
> > 2,1   Isr: 2,1
> >         Topic: nelo2-crash-logs Partition: 3    Leader: 1       Replicas:
> > 1,2   Isr: 1,2
> >         Topic: nelo2-crash-logs Partition: 4    Leader: 2       Replicas:
> > 2,1   Isr: 2,1
> >         Topic: nelo2-crash-logs Partition: 5    Leader: 0       Replicas:
> > 0,2   Isr: 0,2
> >
> > in my mind it should delete my logs per 48 hours, actually it will delete
> > my logs per 5 minutes. What's the unit of 'retention.ms', second, minute
> > or this is a bug?
> >
> > Thanks~!
> >
> > Best Regards
> > Jerry
> >
>