You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Shantanu Deshmukh <sh...@gmail.com> on 2018/05/28 11:48:14 UTC
Effect of settings segment.ms and retention.ms not accurate
I have a topic otp-sms. I want that retention of this topic should be 5
minutes as OTPs are invalid post that amount of time. So I set
retention.ms=300000.
However, this was not working. So reading more in depth in Kafka
configuration document I found another topic level setting that can be
tuned for topic retention to work properly. So I set segment.ms=300000 as
well.
After these changes I saw that old logs go deleted. Still in the topic I
could see one record which is more than 15 minutes old and not getting
deleted. What does one have to do to actually delete messages generated n
minutes ago?
Re: Effect of settings segment.ms and retention.ms not accurate
Posted by Shantanu Deshmukh <sh...@gmail.com>.
This is helpful. Thanks a lot :-)
On Tue, May 29, 2018 at 11:47 PM Matthias J. Sax <ma...@confluent.io>
wrote:
> ConsumerRecord#timestamp()
>
> similar to ConsumerRecord#key() and ConsumerRecord#value()
>
>
> -Matthias
>
> On 5/28/18 11:22 PM, Shantanu Deshmukh wrote:
> > But then I wonder, why such things are not mentioned anywhere in Kafka
> > configuration document? I relied on that setting and it caused us some
> > issues. If it is mentioned clearly then everyone will be aware. Could you
> > please point in right direction about reading timestamp of log message? I
> > will see about implementing that solution in code.
> >
> > On Tue, May 29, 2018 at 11:37 AM Matthias J. Sax <ma...@confluent.io>
> > wrote:
> >
> >> Retention time is a lower bound for how long it is guaranteed that data
> >> will be stored. This guarantee work "one way" only. There is no
> >> guarantee when data will be deleted after the bound passed.
> >>
> >> However, client side, you can always check the record timestamp and just
> >> drop older data that is still in the topic.
> >>
> >> Hope this helps.
> >>
> >>
> >> -Matthias
> >>
> >>
> >> On 5/28/18 9:18 PM, Shantanu Deshmukh wrote:
> >>> Please help.
> >>>
> >>> On Mon, May 28, 2018 at 5:18 PM Shantanu Deshmukh <
> shantanu88d@gmail.com
> >>>
> >>> wrote:
> >>>
> >>>> I have a topic otp-sms. I want that retention of this topic should
> be 5
> >>>> minutes as OTPs are invalid post that amount of time. So I set
> >>>> retention.ms=300000. However, this was not working. So reading more
> in
> >>>> depth in Kafka configuration document I found another topic level
> >> setting
> >>>> that can be tuned for topic retention to work properly. So I set
> >>>> segment.ms=300000 as well.
> >>>>
> >>>> After these changes I saw that old logs go deleted. Still in the
> topic I
> >>>> could see one record which is more than 15 minutes old and not getting
> >>>> deleted. What does one have to do to actually delete messages
> generated
> >> n
> >>>> minutes ago?
> >>>>
> >>>
> >>
> >>
> >
>
>
Re: Effect of settings segment.ms and retention.ms not accurate
Posted by "Matthias J. Sax" <ma...@confluent.io>.
ConsumerRecord#timestamp()
similar to ConsumerRecord#key() and ConsumerRecord#value()
-Matthias
On 5/28/18 11:22 PM, Shantanu Deshmukh wrote:
> But then I wonder, why such things are not mentioned anywhere in Kafka
> configuration document? I relied on that setting and it caused us some
> issues. If it is mentioned clearly then everyone will be aware. Could you
> please point in right direction about reading timestamp of log message? I
> will see about implementing that solution in code.
>
> On Tue, May 29, 2018 at 11:37 AM Matthias J. Sax <ma...@confluent.io>
> wrote:
>
>> Retention time is a lower bound for how long it is guaranteed that data
>> will be stored. This guarantee work "one way" only. There is no
>> guarantee when data will be deleted after the bound passed.
>>
>> However, client side, you can always check the record timestamp and just
>> drop older data that is still in the topic.
>>
>> Hope this helps.
>>
>>
>> -Matthias
>>
>>
>> On 5/28/18 9:18 PM, Shantanu Deshmukh wrote:
>>> Please help.
>>>
>>> On Mon, May 28, 2018 at 5:18 PM Shantanu Deshmukh <shantanu88d@gmail.com
>>>
>>> wrote:
>>>
>>>> I have a topic otp-sms. I want that retention of this topic should be 5
>>>> minutes as OTPs are invalid post that amount of time. So I set
>>>> retention.ms=300000. However, this was not working. So reading more in
>>>> depth in Kafka configuration document I found another topic level
>> setting
>>>> that can be tuned for topic retention to work properly. So I set
>>>> segment.ms=300000 as well.
>>>>
>>>> After these changes I saw that old logs go deleted. Still in the topic I
>>>> could see one record which is more than 15 minutes old and not getting
>>>> deleted. What does one have to do to actually delete messages generated
>> n
>>>> minutes ago?
>>>>
>>>
>>
>>
>
Re: Effect of settings segment.ms and retention.ms not accurate
Posted by Shantanu Deshmukh <sh...@gmail.com>.
But then I wonder, why such things are not mentioned anywhere in Kafka
configuration document? I relied on that setting and it caused us some
issues. If it is mentioned clearly then everyone will be aware. Could you
please point in right direction about reading timestamp of log message? I
will see about implementing that solution in code.
On Tue, May 29, 2018 at 11:37 AM Matthias J. Sax <ma...@confluent.io>
wrote:
> Retention time is a lower bound for how long it is guaranteed that data
> will be stored. This guarantee work "one way" only. There is no
> guarantee when data will be deleted after the bound passed.
>
> However, client side, you can always check the record timestamp and just
> drop older data that is still in the topic.
>
> Hope this helps.
>
>
> -Matthias
>
>
> On 5/28/18 9:18 PM, Shantanu Deshmukh wrote:
> > Please help.
> >
> > On Mon, May 28, 2018 at 5:18 PM Shantanu Deshmukh <shantanu88d@gmail.com
> >
> > wrote:
> >
> >> I have a topic otp-sms. I want that retention of this topic should be 5
> >> minutes as OTPs are invalid post that amount of time. So I set
> >> retention.ms=300000. However, this was not working. So reading more in
> >> depth in Kafka configuration document I found another topic level
> setting
> >> that can be tuned for topic retention to work properly. So I set
> >> segment.ms=300000 as well.
> >>
> >> After these changes I saw that old logs go deleted. Still in the topic I
> >> could see one record which is more than 15 minutes old and not getting
> >> deleted. What does one have to do to actually delete messages generated
> n
> >> minutes ago?
> >>
> >
>
>
Re: Effect of settings segment.ms and retention.ms not accurate
Posted by "Matthias J. Sax" <ma...@confluent.io>.
Retention time is a lower bound for how long it is guaranteed that data
will be stored. This guarantee work "one way" only. There is no
guarantee when data will be deleted after the bound passed.
However, client side, you can always check the record timestamp and just
drop older data that is still in the topic.
Hope this helps.
-Matthias
On 5/28/18 9:18 PM, Shantanu Deshmukh wrote:
> Please help.
>
> On Mon, May 28, 2018 at 5:18 PM Shantanu Deshmukh <sh...@gmail.com>
> wrote:
>
>> I have a topic otp-sms. I want that retention of this topic should be 5
>> minutes as OTPs are invalid post that amount of time. So I set
>> retention.ms=300000. However, this was not working. So reading more in
>> depth in Kafka configuration document I found another topic level setting
>> that can be tuned for topic retention to work properly. So I set
>> segment.ms=300000 as well.
>>
>> After these changes I saw that old logs go deleted. Still in the topic I
>> could see one record which is more than 15 minutes old and not getting
>> deleted. What does one have to do to actually delete messages generated n
>> minutes ago?
>>
>
Re: Effect of settings segment.ms and retention.ms not accurate
Posted by Shantanu Deshmukh <sh...@gmail.com>.
Please help.
On Mon, May 28, 2018 at 5:18 PM Shantanu Deshmukh <sh...@gmail.com>
wrote:
> I have a topic otp-sms. I want that retention of this topic should be 5
> minutes as OTPs are invalid post that amount of time. So I set
> retention.ms=300000. However, this was not working. So reading more in
> depth in Kafka configuration document I found another topic level setting
> that can be tuned for topic retention to work properly. So I set
> segment.ms=300000 as well.
>
> After these changes I saw that old logs go deleted. Still in the topic I
> could see one record which is more than 15 minutes old and not getting
> deleted. What does one have to do to actually delete messages generated n
> minutes ago?
>