You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@kafka.apache.org by Haruki Okada <oc...@gmail.com> on 2021/02/19 06:23:36 UTC

Splitting partition may cause message loss for consumers

Hi, Kafka.

Recently I noticed that splitting partition may cause message delivery loss
for consumers with auto.offset.reset=latest.

I described details in https://issues.apache.org/jira/browse/KAFKA-12261 .

Since delivery loss is undesired in most cases, I think this should be
described in ConsumerConfig.AUTO_OFFSET_RESET_DOC (or somewhere) at least.

What do you think?


Thanks,
-- 
========================
Okada Haruki
ocadaruma@gmail.com
========================

Re: Splitting partition may cause message loss for consumers

Posted by Haruki Okada <oc...@gmail.com>.

Thanks for your quick response.
Yeah, agree with that. (also replied on the issue)

2021年2月19日(金) 16:25 Luke Chen <sh...@gmail.com>:

> Hi Okada san,
> Yes, I agree the "latest" setting in this situation is not good, and we
> should document it.
> But I don't think we should change the default auto.offset.reset setting to
> the earliest.
> The auto.offset.reset setting starts before kafka V1.0, which means, there
> are already a lot of users using it and get used to it now.
>
> Thanks.
> Luke
>
> On Fri, Feb 19, 2021 at 2:24 PM Haruki Okada <oc...@gmail.com> wrote:
>
> > Hi, Kafka.
> >
> > Recently I noticed that splitting partition may cause message delivery
> loss
> > for consumers with auto.offset.reset=latest.
> >
> > I described details in https://issues.apache.org/jira/browse/KAFKA-12261
> .
> >
> > Since delivery loss is undesired in most cases, I think this should be
> > described in ConsumerConfig.AUTO_OFFSET_RESET_DOC (or somewhere) at
> least.
> >
> > What do you think?
> >
> >
> > Thanks,
> > --
> > ========================
> > Okada Haruki
> > ocadaruma@gmail.com
> > ========================
> >
>


-- 
========================
Okada Haruki
ocadaruma@gmail.com
========================

Re: Splitting partition may cause message loss for consumers

Posted by Luke Chen <sh...@gmail.com>.

Hi Okada san,
Yes, I agree the "latest" setting in this situation is not good, and we
should document it.
But I don't think we should change the default auto.offset.reset setting to
the earliest.
The auto.offset.reset setting starts before kafka V1.0, which means, there
are already a lot of users using it and get used to it now.

Thanks.
Luke

On Fri, Feb 19, 2021 at 2:24 PM Haruki Okada <oc...@gmail.com> wrote:

> Hi, Kafka.
>
> Recently I noticed that splitting partition may cause message delivery loss
> for consumers with auto.offset.reset=latest.
>
> I described details in https://issues.apache.org/jira/browse/KAFKA-12261 .
>
> Since delivery loss is undesired in most cases, I think this should be
> described in ConsumerConfig.AUTO_OFFSET_RESET_DOC (or somewhere) at least.
>
> What do you think?
>
>
> Thanks,
> --
> ========================
> Okada Haruki
> ocadaruma@gmail.com
> ========================
>