You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by John Roesler <jo...@confluent.io> on 2018/04/17 18:08:22 UTC

[DISCUSS] KIP-288: Consumer poll timeout change and new waitForAssignment method

Hello all,

I am proposing KIP-288 to improve the semantics of Consumer.poll() w.r.t.
the timeout parameter and to add a new method for blocking for assignment
only.

Please find the details here:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-288%3A+Consumer.poll%28%29+timeout+semantic+change+and+new+waitForAssignment+method

Please let me know what you think!

Thanks,
-John

Re: [DISCUSS] KIP-288: Consumer poll timeout change and new waitForAssignment method

Posted by John Roesler <jo...@confluent.io>.
Hi all,

I'm closing this discussion and marking KIP-288 as discarded. Please pick
up the discussion on the KIP-266 thread.

Thanks,
-John

On Tue, Apr 17, 2018 at 2:07 PM, John Roesler <jo...@confluent.io> wrote:

> Ah, I see that this KIP is a sub-kip of KIP-266. Sorry about that; I've
> sent a message to the discussion thread of KIP-266 to work this out.
>
> Thanks,
> -John
>
> On Tue, Apr 17, 2018 at 1:29 PM, John Roesler <jo...@confluent.io> wrote:
>
>> Thanks, Richard, I'll check it out.
>>
>> On Tue, Apr 17, 2018 at 1:19 PM, Richard Yu <yo...@gmail.com>
>> wrote:
>>
>>> Hi John,
>>>
>>> bq. #1 (wait for metadata) is infinite.
>>>
>>> Some of what you stated in this KIP has already been previously discussed
>>> in a older KIP. (KIP-266)
>>> Just for your reference.
>>>
>>> Thanks, Richard
>>>
>>> On Tue, Apr 17, 2018 at 11:08 AM, John Roesler <jo...@confluent.io>
>>> wrote:
>>>
>>> > Hello all,
>>> >
>>> > I am proposing KIP-288 to improve the semantics of Consumer.poll()
>>> w.r.t.
>>> > the timeout parameter and to add a new method for blocking for
>>> assignment
>>> > only.
>>> >
>>> > Please find the details here:
>>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>>> > 288%3A+Consumer.poll%28%29+timeout+semantic+change+and+
>>> > new+waitForAssignment+method
>>> >
>>> > Please let me know what you think!
>>> >
>>> > Thanks,
>>> > -John
>>> >
>>>
>>
>>
>

Re: [DISCUSS] KIP-288: Consumer poll timeout change and new waitForAssignment method

Posted by John Roesler <jo...@confluent.io>.
Ah, I see that this KIP is a sub-kip of KIP-266. Sorry about that; I've
sent a message to the discussion thread of KIP-266 to work this out.

Thanks,
-John

On Tue, Apr 17, 2018 at 1:29 PM, John Roesler <jo...@confluent.io> wrote:

> Thanks, Richard, I'll check it out.
>
> On Tue, Apr 17, 2018 at 1:19 PM, Richard Yu <yo...@gmail.com>
> wrote:
>
>> Hi John,
>>
>> bq. #1 (wait for metadata) is infinite.
>>
>> Some of what you stated in this KIP has already been previously discussed
>> in a older KIP. (KIP-266)
>> Just for your reference.
>>
>> Thanks, Richard
>>
>> On Tue, Apr 17, 2018 at 11:08 AM, John Roesler <jo...@confluent.io> wrote:
>>
>> > Hello all,
>> >
>> > I am proposing KIP-288 to improve the semantics of Consumer.poll()
>> w.r.t.
>> > the timeout parameter and to add a new method for blocking for
>> assignment
>> > only.
>> >
>> > Please find the details here:
>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> > 288%3A+Consumer.poll%28%29+timeout+semantic+change+and+
>> > new+waitForAssignment+method
>> >
>> > Please let me know what you think!
>> >
>> > Thanks,
>> > -John
>> >
>>
>
>

Re: [DISCUSS] KIP-288: Consumer poll timeout change and new waitForAssignment method

Posted by John Roesler <jo...@confluent.io>.
Thanks, Richard, I'll check it out.

On Tue, Apr 17, 2018 at 1:19 PM, Richard Yu <yo...@gmail.com>
wrote:

> Hi John,
>
> bq. #1 (wait for metadata) is infinite.
>
> Some of what you stated in this KIP has already been previously discussed
> in a older KIP. (KIP-266)
> Just for your reference.
>
> Thanks, Richard
>
> On Tue, Apr 17, 2018 at 11:08 AM, John Roesler <jo...@confluent.io> wrote:
>
> > Hello all,
> >
> > I am proposing KIP-288 to improve the semantics of Consumer.poll() w.r.t.
> > the timeout parameter and to add a new method for blocking for assignment
> > only.
> >
> > Please find the details here:
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 288%3A+Consumer.poll%28%29+timeout+semantic+change+and+
> > new+waitForAssignment+method
> >
> > Please let me know what you think!
> >
> > Thanks,
> > -John
> >
>

Re: [DISCUSS] KIP-288: Consumer poll timeout change and new waitForAssignment method

Posted by Richard Yu <yo...@gmail.com>.
Hi John,

bq. #1 (wait for metadata) is infinite.

Some of what you stated in this KIP has already been previously discussed
in a older KIP. (KIP-266)
Just for your reference.

Thanks, Richard

On Tue, Apr 17, 2018 at 11:08 AM, John Roesler <jo...@confluent.io> wrote:

> Hello all,
>
> I am proposing KIP-288 to improve the semantics of Consumer.poll() w.r.t.
> the timeout parameter and to add a new method for blocking for assignment
> only.
>
> Please find the details here:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 288%3A+Consumer.poll%28%29+timeout+semantic+change+and+
> new+waitForAssignment+method
>
> Please let me know what you think!
>
> Thanks,
> -John
>

Re: [DISCUSS] KIP-288: Consumer poll timeout change and new waitForAssignment method

Posted by John Roesler <jo...@confluent.io>.
Hi Ted,

Thanks for the review.

During phase #2, the consumer continuously polls until it either gets some
responses or spends the whole timeout. If it reaches the timeout, it
doesn't mean that the operation "timed out" in an exceptional way, it just
means we determined that there are no records available for our consumer.
Throwing when we reach the timeout wold make the method contract "Returns a
non-empty collection or throws a TimeoutException".

Compared to phase #1, in which a timeout means we actually didn't get to do
the requested operation (get an assignment), it seems like returning an
empty collection is appropriate when our observation is that there are no
records to process.

Throwing an exception also means that any consumer that is keeping updated
is going to throw an exception on most calls to poll, which suggests the
condition really isn't that exceptional.

I agree that it's a little odd to have this caveat, and I actually was
thinking of the opposite approach: never throwing an exception and always
returning an empty collection when it can't finish in the timeout.

But I'm worried that folks may not deeply read the release notes and
realize they need to change their calling behavior w.r.t. poll. Just
returning an empty collection may lead them to believe that (like today)
after the call to poll, the consumer has an assignment, which would no
longer be correct. So throwing an exception in this case is mostly ensuring
callers don't have their expectations silently broken.

Thanks,
-John


On Tue, Apr 17, 2018 at 1:22 PM, Ted Yu <yu...@gmail.com> wrote:

> bq. This method will continue to return an empty collection if it doesn't
> complete phase #2 within the timeout. If phase #1 times out, poll will
> throw a TimeoutException.
>
> It seems response from the new poll method depends on which phase it is in
> prior to the timeout.
> Wouldn't it be cleaner if TimeoutException is thrown regardless of the
> phase it is in ?
>
> Cheers
>
> On Tue, Apr 17, 2018 at 11:08 AM, John Roesler <jo...@confluent.io> wrote:
>
> > Hello all,
> >
> > I am proposing KIP-288 to improve the semantics of Consumer.poll() w.r.t.
> > the timeout parameter and to add a new method for blocking for assignment
> > only.
> >
> > Please find the details here:
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 288%3A+Consumer.poll%28%29+timeout+semantic+change+and+
> > new+waitForAssignment+method
> >
> > Please let me know what you think!
> >
> > Thanks,
> > -John
> >
>

Re: [DISCUSS] KIP-288: Consumer poll timeout change and new waitForAssignment method

Posted by Ted Yu <yu...@gmail.com>.
bq. This method will continue to return an empty collection if it doesn't
complete phase #2 within the timeout. If phase #1 times out, poll will
throw a TimeoutException.

It seems response from the new poll method depends on which phase it is in
prior to the timeout.
Wouldn't it be cleaner if TimeoutException is thrown regardless of the
phase it is in ?

Cheers

On Tue, Apr 17, 2018 at 11:08 AM, John Roesler <jo...@confluent.io> wrote:

> Hello all,
>
> I am proposing KIP-288 to improve the semantics of Consumer.poll() w.r.t.
> the timeout parameter and to add a new method for blocking for assignment
> only.
>
> Please find the details here:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 288%3A+Consumer.poll%28%29+timeout+semantic+change+and+
> new+waitForAssignment+method
>
> Please let me know what you think!
>
> Thanks,
> -John
>