You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Yan Wang <yw...@xmatters.com> on 2017/06/23 20:43:28 UTC
Pause and max.poll.records
How the pause influences max.poll.records? If I have two partitions, say p1 and p2. p1 has number local fetched records > max.poll.records. p2 has no data in local but in the server (not fetched yet). Now, if I pause P1. Does the next poll send fetch request to the server so that I can get data for p2 from the server or we get nothing from p2 till we resume p1 and drain p1 till total local fetched records < max.poll.records?
Thanks a lot!
RE: Pause and max.poll.records
Posted by Yan Wang <yw...@xmatters.com>.
Sorry, seems it works as you said. System.out.print("+") in my test is buffered. It does not show the "+" immediately till I print a new line. Weird.
Thanks a lot!
Yan
-----Original Message-----
From: Yan Wang
Sent: Sunday, June 25, 2017 11:18 PM
To: dev@kafka.apache.org
Subject: RE: Pause and max.poll.records
As I tested, seems if I pause p1 but do not call pause/resume again on p1, I cannot pull any message from p2. But if I pause p1, I find I get one message from p2 each time I call pause on p1 ( I mean I keep calling consumer.pause(p1) before consumer.poll).
Is it a bug or right behavior?
Thanks a lot!
Yan
-----Original Message-----
From: Jason Gustafson [mailto:jason@confluent.io]
Sent: Friday, June 23, 2017 3:08 PM
To: dev@kafka.apache.org
Subject: Re: Pause and max.poll.records
Fetches for each partition are sent once all the pending data for that partition has been consumed. The only difference that pause makes is that we do not bother fetching paused partitions. So in your example, p2 would be fetched even if p1 is paused and has fetched records > max.poll.records pending.
-Jason
On Fri, Jun 23, 2017 at 1:43 PM, Yan Wang <yw...@xmatters.com> wrote:
> How the pause influences max.poll.records? If I have two partitions,
> say
> p1 and p2. p1 has number local fetched records > max.poll.records. p2
> has no data in local but in the server (not fetched yet). Now, if I pause P1.
> Does the next poll send fetch request to the server so that I can get
> data for p2 from the server or we get nothing from p2 till we resume
> p1 and drain p1 till total local fetched records < max.poll.records?
>
>
> Thanks a lot!
>
RE: Pause and max.poll.records
Posted by Yan Wang <yw...@xmatters.com>.
As I tested, seems if I pause p1 but do not call pause/resume again on p1, I cannot pull any message from p2. But if I pause p1, I find I get one message from p2 each time I call pause on p1 ( I mean I keep calling consumer.pause(p1) before consumer.poll).
Is it a bug or right behavior?
Thanks a lot!
Yan
-----Original Message-----
From: Jason Gustafson [mailto:jason@confluent.io]
Sent: Friday, June 23, 2017 3:08 PM
To: dev@kafka.apache.org
Subject: Re: Pause and max.poll.records
Fetches for each partition are sent once all the pending data for that partition has been consumed. The only difference that pause makes is that we do not bother fetching paused partitions. So in your example, p2 would be fetched even if p1 is paused and has fetched records > max.poll.records pending.
-Jason
On Fri, Jun 23, 2017 at 1:43 PM, Yan Wang <yw...@xmatters.com> wrote:
> How the pause influences max.poll.records? If I have two partitions,
> say
> p1 and p2. p1 has number local fetched records > max.poll.records. p2
> has no data in local but in the server (not fetched yet). Now, if I pause P1.
> Does the next poll send fetch request to the server so that I can get
> data for p2 from the server or we get nothing from p2 till we resume
> p1 and drain p1 till total local fetched records < max.poll.records?
>
>
> Thanks a lot!
>
Re: Pause and max.poll.records
Posted by Jason Gustafson <ja...@confluent.io>.
Fetches for each partition are sent once all the pending data for that
partition has been consumed. The only difference that pause makes is that
we do not bother fetching paused partitions. So in your example, p2 would
be fetched even if p1 is paused and has fetched records > max.poll.records
pending.
-Jason
On Fri, Jun 23, 2017 at 1:43 PM, Yan Wang <yw...@xmatters.com> wrote:
> How the pause influences max.poll.records? If I have two partitions, say
> p1 and p2. p1 has number local fetched records > max.poll.records. p2 has
> no data in local but in the server (not fetched yet). Now, if I pause P1.
> Does the next poll send fetch request to the server so that I can get data
> for p2 from the server or we get nothing from p2 till we resume p1 and
> drain p1 till total local fetched records < max.poll.records?
>
>
> Thanks a lot!
>