You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Adam Bellemare <ad...@gmail.com> on 2019/01/29 20:53:40 UTC
Why is enable.auto.commit=true the default value for consumer?
As the question indicates.
Should this not be default false? I think this is a bit nefarious to
someone launching their application into production without testing it
extensively around failure modes. I can see a scenario where a consumer
polls for events, processes them, produces to output topic, and commits the
offsets. Say it takes 30 seconds for a batch. If it fails halfway through,
upon restarting it will skip everything that was unprocessed/unpublished up
to the committed offset.
Is there a historic reason why it's set to default true? Is it because to
change it to default false it could affect the upgrade path of previous
implementations?
Adam
Re: Why is enable.auto.commit=true the default value for consumer?
Posted by Colin McCabe <cm...@apache.org>.
This is a very fair question. I suspect the answer is that people feel that automatic offset commit is easier to get started with, even though it has the error handling shortcomings you mention. It would be very hard to change the default now, since it would break such a huge amount of code. Perhaps if we create a whole new consumer interface.
best.
Colin
On Tue, Jan 29, 2019, at 12:53, Adam Bellemare wrote:
> As the question indicates.
>
> Should this not be default false? I think this is a bit nefarious to
> someone launching their application into production without testing it
> extensively around failure modes. I can see a scenario where a consumer
> polls for events, processes them, produces to output topic, and commits the
> offsets. Say it takes 30 seconds for a batch. If it fails halfway through,
> upon restarting it will skip everything that was unprocessed/unpublished up
> to the committed offset.
>
> Is there a historic reason why it's set to default true? Is it because to
> change it to default false it could affect the upgrade path of previous
> implementations?
>
> Adam
>
RE: Why is enable.auto.commit=true the default value for consumer?
Posted by "Pellerin, Clement" <Cl...@ibi.com>.
Yes, that sentence is the source of all that confusion.
I learned it was misleading when reading part of the source code.
There are also some stack overflow answers that clarify auto-commit.
I think you are right, the documentation probably does not say
you must process all records in the batch when enabling auto-commit.
This is a best-practice you can infer from all the examples.
The examples are always single-threaded though.
We should mention the requirement explicitly as that part of the
contract the application must obey to use auto-commit safely.
That would cover the multi-threaded case too.
-----Original Message-----
From: Adam Bellemare [mailto:adam.bellemare@gmail.com]
Sent: Friday, February 1, 2019 8:42 AM
To: dev@kafka.apache.org
Subject: Re: Why is enable.auto.commit=true the default value for consumer?
Hi Clement & Colin
Thanks for the discussion on this.
I think the biggest confusion is how the enable.auto.commit documentation
says "If true the consumer's offset will be periodically committed in the
background."
@Clement, you mention that there is documentation about what you must do
with auto-commit. I am looking at https://kafka.apache.org/documentation
and I cannot seem to locate it. Could you point me to it?
I would be fine with making a simple PR to improve the documentation on
this - I think it will help straighten things out.
Thanks again,
Adam
Re: Why is enable.auto.commit=true the default value for consumer?
Posted by Adam Bellemare <ad...@gmail.com>.
Hi Clement & Colin
Thanks for the discussion on this.
I think the biggest confusion is how the enable.auto.commit documentation
says "If true the consumer's offset will be periodically committed in the
background."
@Clement, you mention that there is documentation about what you must do
with auto-commit. I am looking at https://kafka.apache.org/documentation
and I cannot seem to locate it. Could you point me to it?
I would be fine with making a simple PR to improve the documentation on
this - I think it will help straighten things out.
Thanks again,
Adam
On Tue, Jan 29, 2019 at 7:35 PM Pellerin, Clement <Cl...@ibi.com>
wrote:
> Indeed, but this is what the documentation says you must do with
> auto-commit.
> I say it is a user error if you don't.
>
> Regardless, I think this is a fairly common misconception, so it would not
> hurt
> to debunk it explicitly in the documentation.
>
> -----Original Message-----
> From: Colin McCabe [mailto:cmccabe@apache.org]
> Sent: Tuesday, January 29, 2019 7:26 PM
> To: dev@kafka.apache.org
> Subject: Re: Why is enable.auto.commit=true the default value for consumer?
>
> Hi Clement,
>
> You are assuming that the client application is single-threaded-- or at
> least processes all the records before polling for more. This may or may
> not be the case. But that is a fair point-- in this case, auto-commit
> would be safe.
>
> best,
> Colin
>
> On Tue, Jan 29, 2019, at 16:23, Pellerin, Clement wrote:
> > I had the same impression at some point but this is not how auto-commit
> works.
> > Auto-commit can only commit when the application comes back to poll and
> > if it decides to commit at that time, it will only commit the previous
> batch.
> > In your example, the app might come back and have to re-execute all
> > the records in the uncommitted batch but it will never skip over
> > unprocessed records.
> >
> > -----Original Message-----
> > From: Adam Bellemare [mailto:adam.bellemare@gmail.com]
> > Sent: Tuesday, January 29, 2019 3:54 PM
> > To: dev@kafka.apache.org
> > Subject: Why is enable.auto.commit=true the default value for consumer?
> >
> > As the question indicates.
> >
> > Should this not be default false? I think this is a bit nefarious to
> > someone launching their application into production without testing it
> > extensively around failure modes. I can see a scenario where a consumer
> > polls for events, processes them, produces to output topic, and commits
> the
> > offsets. Say it takes 30 seconds for a batch. If it fails halfway
> through,
> > upon restarting it will skip everything that was unprocessed/unpublished
> up
> > to the committed offset.
> >
> > Is there a historic reason why it's set to default true? Is it because to
> > change it to default false it could affect the upgrade path of previous
> > implementations?
> >
> > Adam
> >
>
RE: Why is enable.auto.commit=true the default value for consumer?
Posted by "Pellerin, Clement" <Cl...@ibi.com>.
Indeed, but this is what the documentation says you must do with auto-commit.
I say it is a user error if you don't.
Regardless, I think this is a fairly common misconception, so it would not hurt
to debunk it explicitly in the documentation.
-----Original Message-----
From: Colin McCabe [mailto:cmccabe@apache.org]
Sent: Tuesday, January 29, 2019 7:26 PM
To: dev@kafka.apache.org
Subject: Re: Why is enable.auto.commit=true the default value for consumer?
Hi Clement,
You are assuming that the client application is single-threaded-- or at least processes all the records before polling for more. This may or may not be the case. But that is a fair point-- in this case, auto-commit would be safe.
best,
Colin
On Tue, Jan 29, 2019, at 16:23, Pellerin, Clement wrote:
> I had the same impression at some point but this is not how auto-commit works.
> Auto-commit can only commit when the application comes back to poll and
> if it decides to commit at that time, it will only commit the previous batch.
> In your example, the app might come back and have to re-execute all
> the records in the uncommitted batch but it will never skip over
> unprocessed records.
>
> -----Original Message-----
> From: Adam Bellemare [mailto:adam.bellemare@gmail.com]
> Sent: Tuesday, January 29, 2019 3:54 PM
> To: dev@kafka.apache.org
> Subject: Why is enable.auto.commit=true the default value for consumer?
>
> As the question indicates.
>
> Should this not be default false? I think this is a bit nefarious to
> someone launching their application into production without testing it
> extensively around failure modes. I can see a scenario where a consumer
> polls for events, processes them, produces to output topic, and commits the
> offsets. Say it takes 30 seconds for a batch. If it fails halfway through,
> upon restarting it will skip everything that was unprocessed/unpublished up
> to the committed offset.
>
> Is there a historic reason why it's set to default true? Is it because to
> change it to default false it could affect the upgrade path of previous
> implementations?
>
> Adam
>
Re: Why is enable.auto.commit=true the default value for consumer?
Posted by Colin McCabe <cm...@apache.org>.
Hi Clement,
You are assuming that the client application is single-threaded-- or at least processes all the records before polling for more. This may or may not be the case. But that is a fair point-- in this case, auto-commit would be safe.
best,
Colin
On Tue, Jan 29, 2019, at 16:23, Pellerin, Clement wrote:
> I had the same impression at some point but this is not how auto-commit works.
> Auto-commit can only commit when the application comes back to poll and
> if it decides to commit at that time, it will only commit the previous batch.
> In your example, the app might come back and have to re-execute all
> the records in the uncommitted batch but it will never skip over
> unprocessed records.
>
> -----Original Message-----
> From: Adam Bellemare [mailto:adam.bellemare@gmail.com]
> Sent: Tuesday, January 29, 2019 3:54 PM
> To: dev@kafka.apache.org
> Subject: Why is enable.auto.commit=true the default value for consumer?
>
> As the question indicates.
>
> Should this not be default false? I think this is a bit nefarious to
> someone launching their application into production without testing it
> extensively around failure modes. I can see a scenario where a consumer
> polls for events, processes them, produces to output topic, and commits the
> offsets. Say it takes 30 seconds for a batch. If it fails halfway through,
> upon restarting it will skip everything that was unprocessed/unpublished up
> to the committed offset.
>
> Is there a historic reason why it's set to default true? Is it because to
> change it to default false it could affect the upgrade path of previous
> implementations?
>
> Adam
>
RE: Why is enable.auto.commit=true the default value for consumer?
Posted by "Pellerin, Clement" <Cl...@ibi.com>.
I had the same impression at some point but this is not how auto-commit works.
Auto-commit can only commit when the application comes back to poll and
if it decides to commit at that time, it will only commit the previous batch.
In your example, the app might come back and have to re-execute all
the records in the uncommitted batch but it will never skip over
unprocessed records.
-----Original Message-----
From: Adam Bellemare [mailto:adam.bellemare@gmail.com]
Sent: Tuesday, January 29, 2019 3:54 PM
To: dev@kafka.apache.org
Subject: Why is enable.auto.commit=true the default value for consumer?
As the question indicates.
Should this not be default false? I think this is a bit nefarious to
someone launching their application into production without testing it
extensively around failure modes. I can see a scenario where a consumer
polls for events, processes them, produces to output topic, and commits the
offsets. Say it takes 30 seconds for a batch. If it fails halfway through,
upon restarting it will skip everything that was unprocessed/unpublished up
to the committed offset.
Is there a historic reason why it's set to default true? Is it because to
change it to default false it could affect the upgrade path of previous
implementations?
Adam