You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by "Lian, Li" <ll...@ebay.com> on 2014/06/24 14:20:26 UTC
What's the behavior when Kafka is deleting messages and consumers
are still reading
When Kafka broker is trying to delete message according to the log retention setting, either triggered by log age or topic partition size, if the same time there are still consumers reading the topic, what will happen?
Best regards,
Lex Lian
Email: llian@ebay.com
Re: What's the behavior when Kafka is deleting messages and consumers
are still reading
Posted by Guozhang Wang <wa...@gmail.com>.
I think we can probably update the documentation page for this update:
https://kafka.apache.org/documentation.html#compaction
On Tue, Jun 24, 2014 at 3:54 PM, Lian, Li <ll...@ebay.com> wrote:
> GuoZhang,
>
> Thanks for explaining. I thought there might be such kind of lock
> mechanism but cannot confirm it in any documentation on Kafka website.
>
> It will be better if this could be written down in some Wiki or FAQ.
>
>
> Best regards,
> Lex Lian
>
> Email: llian@ebay.com
>
>
>
>
>
> On 6/24/14, 11:17 PM, "Guozhang Wang" <wa...@gmail.com> wrote:
>
> >Hi Li,
> >
> >The log operations are protected by a lock, so if there is a concurrent
> >read on this partition it will not be deleted. But then when it is deleted
> >the next fetch/read will result in a OffsetOutOfRange exception and the
> >consumer needs to restart from a offset reset value.
> >
> >Guozhang
> >
> >
> >On Tue, Jun 24, 2014 at 5:20 AM, Lian, Li <ll...@ebay.com> wrote:
> >
> >> When Kafka broker is trying to delete message according to the log
> >> retention setting, either triggered by log age or topic partition size,
> >>if
> >> the same time there are still consumers reading the topic, what will
> >>happen?
> >>
> >> Best regards,
> >> Lex Lian
> >>
> >> Email: llian@ebay.com
> >>
> >>
> >
> >
> >--
> >-- Guozhang
>
>
--
-- Guozhang
Re: What's the behavior when Kafka is deleting messages and
consumers are still reading
Posted by "Lian, Li" <ll...@ebay.com>.
GuoZhang,
Thanks for explaining. I thought there might be such kind of lock
mechanism but cannot confirm it in any documentation on Kafka website.
It will be better if this could be written down in some Wiki or FAQ.
Best regards,
Lex Lian
Email: llian@ebay.com
On 6/24/14, 11:17 PM, "Guozhang Wang" <wa...@gmail.com> wrote:
>Hi Li,
>
>The log operations are protected by a lock, so if there is a concurrent
>read on this partition it will not be deleted. But then when it is deleted
>the next fetch/read will result in a OffsetOutOfRange exception and the
>consumer needs to restart from a offset reset value.
>
>Guozhang
>
>
>On Tue, Jun 24, 2014 at 5:20 AM, Lian, Li <ll...@ebay.com> wrote:
>
>> When Kafka broker is trying to delete message according to the log
>> retention setting, either triggered by log age or topic partition size,
>>if
>> the same time there are still consumers reading the topic, what will
>>happen?
>>
>> Best regards,
>> Lex Lian
>>
>> Email: llian@ebay.com
>>
>>
>
>
>--
>-- Guozhang
Re: What's the behavior when Kafka is deleting messages and consumers
are still reading
Posted by Neha Narkhede <ne...@gmail.com>.
The behavior on the consumer in this case is governed by the value of the
"auto.offset.reset" config. Depending on this config, it will reset it's
offset to either the earliest or the latest in the log.
Thanks,
Neha
On Tue, Jun 24, 2014 at 8:17 AM, Guozhang Wang <wa...@gmail.com> wrote:
> Hi Li,
>
> The log operations are protected by a lock, so if there is a concurrent
> read on this partition it will not be deleted. But then when it is deleted
> the next fetch/read will result in a OffsetOutOfRange exception and the
> consumer needs to restart from a offset reset value.
>
> Guozhang
>
>
> On Tue, Jun 24, 2014 at 5:20 AM, Lian, Li <ll...@ebay.com> wrote:
>
> > When Kafka broker is trying to delete message according to the log
> > retention setting, either triggered by log age or topic partition size,
> if
> > the same time there are still consumers reading the topic, what will
> happen?
> >
> > Best regards,
> > Lex Lian
> >
> > Email: llian@ebay.com
> >
> >
>
>
> --
> -- Guozhang
>
Re: What's the behavior when Kafka is deleting messages and consumers
are still reading
Posted by Guozhang Wang <wa...@gmail.com>.
Hi Li,
The log operations are protected by a lock, so if there is a concurrent
read on this partition it will not be deleted. But then when it is deleted
the next fetch/read will result in a OffsetOutOfRange exception and the
consumer needs to restart from a offset reset value.
Guozhang
On Tue, Jun 24, 2014 at 5:20 AM, Lian, Li <ll...@ebay.com> wrote:
> When Kafka broker is trying to delete message according to the log
> retention setting, either triggered by log age or topic partition size, if
> the same time there are still consumers reading the topic, what will happen?
>
> Best regards,
> Lex Lian
>
> Email: llian@ebay.com
>
>
--
-- Guozhang