You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Krzysztof Ciesielski <kr...@softwaremill.com> on 2016/01/21 19:46:44 UTC
Kafka 0.9 -> consumer.poll() occasionally returns 0 elements
Hello, I'm running into an issue with the new consumer in Kafka 0.9.0.0.
Here's a runnable gist illustrating the problem:
https://gist.github.com/kciesielski/054bb4359a318aa17561 (requires Kafka on
localhost:9092)
Scenario description:
First, a producer writes 500000 elements into a topic
Then, a consumer starts to read, polling in a loop.
When "max.partition.fetch.bytes" is set to a relatively small value, each
"consumer.poll()" returns a batch of messages.
If this value is left as default, the output tends to look like this:
Poll returned 13793 elements
Poll returned 13793 elements
Poll returned 13793 elements
Poll returned 13793 elements
Poll returned 0 elements
Poll returned 0 elements
Poll returned 0 elements
Poll returned 0 elements
Poll returned 13793 elements
Poll returned 13793 elements
As we can see, there are weird "gaps" when poll returns 0 elements for some
time. What is the reason for that? Maybe there are some good practices
about setting "max.partition.fetch.bytes" which I don't follow?
Bests,
Chris
Re: Kafka 0.9 -> consumer.poll() occasionally returns 0 elements
Posted by tao xiao <xi...@gmail.com>.
No. With receive.buffer.bytes setting to 64K I am unable to reproduce the
error. BTW I set the message size to 10K when testing with 64K buffer size
On Thu, 28 Jan 2016 at 01:35 Jason Gustafson <ja...@confluent.io> wrote:
> Hey Tao,
>
> If you increase "receive.buffer.bytes" to 64K, can you still reproduce the
> problem?
>
> -Jason
>
> On Tue, Jan 26, 2016 at 11:18 PM, Krzysztof Ciesielski <
> krzysztof.ciesielski@softwaremill.pl> wrote:
>
> > Jason,
> >
> > My os/vm is OSX 10.11.3, JDK 1.8.0.40
> >
> > —
> > Krzysztof
> > On 26 January 2016 at 19:04:58, Jason Gustafson (jason@confluent.io)
> > wrote:
> >
> > Hey Krzysztof,
> >
> > So far I haven't had any luck figuring out the cause of the 5 second
> pause,
> > but I've reproduced it with the old consumer on 0.8.2, so that rules out
> > anything specific to the new consumer. Can you tell me which os/jvm
> you're
> > seeing it with? Also, can you try changing the "receive.buffer.bytes"
> > consumer configuration to "65536"? This was the default for the old
> > consumer and I haven't been able to reproduce the problem when it is set.
> >
> > -Jason
> >
>
Re: Kafka 0.9 -> consumer.poll() occasionally returns 0 elements
Posted by Jason Gustafson <ja...@confluent.io>.
Hey Tao,
If you increase "receive.buffer.bytes" to 64K, can you still reproduce the
problem?
-Jason
On Tue, Jan 26, 2016 at 11:18 PM, Krzysztof Ciesielski <
krzysztof.ciesielski@softwaremill.pl> wrote:
> Jason,
>
> My os/vm is OSX 10.11.3, JDK 1.8.0.40
>
> —
> Krzysztof
> On 26 January 2016 at 19:04:58, Jason Gustafson (jason@confluent.io)
> wrote:
>
> Hey Krzysztof,
>
> So far I haven't had any luck figuring out the cause of the 5 second pause,
> but I've reproduced it with the old consumer on 0.8.2, so that rules out
> anything specific to the new consumer. Can you tell me which os/jvm you're
> seeing it with? Also, can you try changing the "receive.buffer.bytes"
> consumer configuration to "65536"? This was the default for the old
> consumer and I haven't been able to reproduce the problem when it is set.
>
> -Jason
>
Re: Kafka 0.9 -> consumer.poll() occasionally returns 0
elements
Posted by Krzysztof Ciesielski <kr...@softwaremill.pl>.
Jason,
My os/vm is OSX 10.11.3, JDK 1.8.0.40
—
Krzysztof
On 26 January 2016 at 19:04:58, Jason Gustafson (jason@confluent.io) wrote:
Hey Krzysztof,
So far I haven't had any luck figuring out the cause of the 5 second pause,
but I've reproduced it with the old consumer on 0.8.2, so that rules out
anything specific to the new consumer. Can you tell me which os/jvm you're
seeing it with? Also, can you try changing the "receive.buffer.bytes"
consumer configuration to "65536"? This was the default for the old
consumer and I haven't been able to reproduce the problem when it is set.
-Jason
Re: Kafka 0.9 -> consumer.poll() occasionally returns 0 elements
Posted by tao xiao <xi...@gmail.com>.
I managed to reproduce this issue on my mac with receive.buffer.bytes
setting to new consumer default value. My JVM is hotspot 64 bit 1.7.0_60
and mac 10.10.5
On Wed, 27 Jan 2016 at 02:54 Krzysztof Ciesielski <
krzysztof.ciesielski@softwaremill.pl> wrote:
> Hi Jason,
>
> Lowering "receive.buffer.bytes” helps, but when the message size gets
> bigger - it comes back again.
> I will test with 65536 and check how big the message has to be to make the
> issue reappear with this value (I suspect that quite big).
>
> --
> Krzysztof Ciesielski
> SoftwareMill
>
> On 26 January 2016 at 19:04:58, Jason Gustafson (jason@confluent.io)
> wrote:
>
> Hey Krzysztof,
>
> So far I haven't had any luck figuring out the cause of the 5 second pause,
> but I've reproduced it with the old consumer on 0.8.2, so that rules out
> anything specific to the new consumer. Can you tell me which os/jvm you're
> seeing it with? Also, can you try changing the "receive.buffer.bytes"
> consumer configuration to "65536"? This was the default for the old
> consumer and I haven't been able to reproduce the problem when it is set.
>
> -Jason
>
Re: Kafka 0.9 -> consumer.poll() occasionally returns 0
elements
Posted by Krzysztof Ciesielski <kr...@softwaremill.pl>.
Hi Jason,
Lowering "receive.buffer.bytes” helps, but when the message size gets bigger - it comes back again.
I will test with 65536 and check how big the message has to be to make the issue reappear with this value (I suspect that quite big).
--
Krzysztof Ciesielski
SoftwareMill
On 26 January 2016 at 19:04:58, Jason Gustafson (jason@confluent.io) wrote:
Hey Krzysztof,
So far I haven't had any luck figuring out the cause of the 5 second pause,
but I've reproduced it with the old consumer on 0.8.2, so that rules out
anything specific to the new consumer. Can you tell me which os/jvm you're
seeing it with? Also, can you try changing the "receive.buffer.bytes"
consumer configuration to "65536"? This was the default for the old
consumer and I haven't been able to reproduce the problem when it is set.
-Jason
Re: Kafka 0.9 -> consumer.poll() occasionally returns 0 elements
Posted by Jason Gustafson <ja...@confluent.io>.
Hey Krzysztof,
So far I haven't had any luck figuring out the cause of the 5 second pause,
but I've reproduced it with the old consumer on 0.8.2, so that rules out
anything specific to the new consumer. Can you tell me which os/jvm you're
seeing it with? Also, can you try changing the "receive.buffer.bytes"
consumer configuration to "65536"? This was the default for the old
consumer and I haven't been able to reproduce the problem when it is set.
-Jason
Re: Kafka 0.9 -> consumer.poll() occasionally returns 0 elements
Posted by Ismael Juma <is...@juma.me.uk>.
Hi,
On Sun, Jan 24, 2016 at 7:17 PM, Krzysztof Ciesielski <
krzysztof.ciesielski@softwaremill.com> wrote:
> Yes, it's exactly 5 seconds on every machine. Sure, I'll open a JIRA
>
Jason already did so:
https://issues.apache.org/jira/browse/KAFKA-3135
Feel free to add relevant information there.
Ismael
> On Jan 22, 2016 7:24 AM, "Jason Gustafson" <ja...@confluent.io> wrote:
>
> > Hi Krzysztof,
> >
> > This is definitely weird. I see the data in the broker's send queue, but
> > there's a delay of 5 seconds before it's sent to the client. Can you
> create
> > a JIRA?
> >
> > Thanks,
> > Jason
> >
> >
> >
> > On Thu, Jan 21, 2016 at 11:30 AM, Samya Maiti <samya.maiti2012@gmail.com
> >
> > wrote:
> >
> > > +1, facing same issue.
> > > -Sam
> > > > On 22-Jan-2016, at 12:16 am, Krzysztof Ciesielski <
> > > krzysztof.ciesielski@softwaremill.com> wrote:
> > > >
> > > > Hello, I'm running into an issue with the new consumer in Kafka
> > 0.9.0.0.
> > > > Here's a runnable gist illustrating the problem:
> > > > https://gist.github.com/kciesielski/054bb4359a318aa17561 (requires
> > > Kafka on
> > > > localhost:9092)
> > > >
> > > > Scenario description:
> > > > First, a producer writes 500000 elements into a topic
> > > > Then, a consumer starts to read, polling in a loop.
> > > > When "max.partition.fetch.bytes" is set to a relatively small value,
> > each
> > > > "consumer.poll()" returns a batch of messages.
> > > > If this value is left as default, the output tends to look like this:
> > > >
> > > > Poll returned 13793 elements
> > > > Poll returned 13793 elements
> > > > Poll returned 13793 elements
> > > > Poll returned 13793 elements
> > > > Poll returned 0 elements
> > > > Poll returned 0 elements
> > > > Poll returned 0 elements
> > > > Poll returned 0 elements
> > > > Poll returned 13793 elements
> > > > Poll returned 13793 elements
> > > >
> > > > As we can see, there are weird "gaps" when poll returns 0 elements
> for
> > > some
> > > > time. What is the reason for that? Maybe there are some good
> practices
> > > > about setting "max.partition.fetch.bytes" which I don't follow?
> > > >
> > > > Bests,
> > > > Chris
> > >
> > >
> >
>
Re: Kafka 0.9 -> consumer.poll() occasionally returns 0 elements
Posted by Krzysztof Ciesielski <kr...@softwaremill.com>.
Yes, it's exactly 5 seconds on every machine. Sure, I'll open a JIRA
On Jan 22, 2016 7:24 AM, "Jason Gustafson" <ja...@confluent.io> wrote:
> Hi Krzysztof,
>
> This is definitely weird. I see the data in the broker's send queue, but
> there's a delay of 5 seconds before it's sent to the client. Can you create
> a JIRA?
>
> Thanks,
> Jason
>
>
>
> On Thu, Jan 21, 2016 at 11:30 AM, Samya Maiti <sa...@gmail.com>
> wrote:
>
> > +1, facing same issue.
> > -Sam
> > > On 22-Jan-2016, at 12:16 am, Krzysztof Ciesielski <
> > krzysztof.ciesielski@softwaremill.com> wrote:
> > >
> > > Hello, I'm running into an issue with the new consumer in Kafka
> 0.9.0.0.
> > > Here's a runnable gist illustrating the problem:
> > > https://gist.github.com/kciesielski/054bb4359a318aa17561 (requires
> > Kafka on
> > > localhost:9092)
> > >
> > > Scenario description:
> > > First, a producer writes 500000 elements into a topic
> > > Then, a consumer starts to read, polling in a loop.
> > > When "max.partition.fetch.bytes" is set to a relatively small value,
> each
> > > "consumer.poll()" returns a batch of messages.
> > > If this value is left as default, the output tends to look like this:
> > >
> > > Poll returned 13793 elements
> > > Poll returned 13793 elements
> > > Poll returned 13793 elements
> > > Poll returned 13793 elements
> > > Poll returned 0 elements
> > > Poll returned 0 elements
> > > Poll returned 0 elements
> > > Poll returned 0 elements
> > > Poll returned 13793 elements
> > > Poll returned 13793 elements
> > >
> > > As we can see, there are weird "gaps" when poll returns 0 elements for
> > some
> > > time. What is the reason for that? Maybe there are some good practices
> > > about setting "max.partition.fetch.bytes" which I don't follow?
> > >
> > > Bests,
> > > Chris
> >
> >
>
Re: Kafka 0.9 -> consumer.poll() occasionally returns 0 elements
Posted by Jason Gustafson <ja...@confluent.io>.
I see the pause regardless of the client's poll timeout. I thought it might
be caused by the quota manager (which is supposed to delay fetches if a
quota is exceeded), but this is looking more like a problem in the server's
network layer. I went ahead and opened KAFKA-3135, so we can move
investigation there.
-Jason
On Thu, Jan 21, 2016 at 10:52 PM, Samya Maiti <sa...@gmail.com>
wrote:
> Providing a non-Zero value (Zero is default), for
>
> public ConsumerRecords<K, V> poll(long timeout)
>
> Works fine for me with no gaps,but said so is just a work around.
> Consumer is definitely picking up messages with some delay.
>
> -Sam
>
>
> > On 22-Jan-2016, at 11:54 am, Jason Gustafson <ja...@confluent.io> wrote:
> >
> > Hi Krzysztof,
> >
> > This is definitely weird. I see the data in the broker's send queue, but
> > there's a delay of 5 seconds before it's sent to the client. Can you
> create
> > a JIRA?
> >
> > Thanks,
> > Jason
> >
> >
> >
> > On Thu, Jan 21, 2016 at 11:30 AM, Samya Maiti <samya.maiti2012@gmail.com
> >
> > wrote:
> >
> >> +1, facing same issue.
> >> -Sam
> >>> On 22-Jan-2016, at 12:16 am, Krzysztof Ciesielski <
> >> krzysztof.ciesielski@softwaremill.com> wrote:
> >>>
> >>> Hello, I'm running into an issue with the new consumer in Kafka
> 0.9.0.0.
> >>> Here's a runnable gist illustrating the problem:
> >>> https://gist.github.com/kciesielski/054bb4359a318aa17561 (requires
> >> Kafka on
> >>> localhost:9092)
> >>>
> >>> Scenario description:
> >>> First, a producer writes 500000 elements into a topic
> >>> Then, a consumer starts to read, polling in a loop.
> >>> When "max.partition.fetch.bytes" is set to a relatively small value,
> each
> >>> "consumer.poll()" returns a batch of messages.
> >>> If this value is left as default, the output tends to look like this:
> >>>
> >>> Poll returned 13793 elements
> >>> Poll returned 13793 elements
> >>> Poll returned 13793 elements
> >>> Poll returned 13793 elements
> >>> Poll returned 0 elements
> >>> Poll returned 0 elements
> >>> Poll returned 0 elements
> >>> Poll returned 0 elements
> >>> Poll returned 13793 elements
> >>> Poll returned 13793 elements
> >>>
> >>> As we can see, there are weird "gaps" when poll returns 0 elements for
> >> some
> >>> time. What is the reason for that? Maybe there are some good practices
> >>> about setting "max.partition.fetch.bytes" which I don't follow?
> >>>
> >>> Bests,
> >>> Chris
> >>
> >>
>
>
Re: Kafka 0.9 -> consumer.poll() occasionally returns 0 elements
Posted by Samya Maiti <sa...@gmail.com>.
Providing a non-Zero value (Zero is default), for
public ConsumerRecords<K, V> poll(long timeout)
Works fine for me with no gaps,but said so is just a work around.
Consumer is definitely picking up messages with some delay.
-Sam
> On 22-Jan-2016, at 11:54 am, Jason Gustafson <ja...@confluent.io> wrote:
>
> Hi Krzysztof,
>
> This is definitely weird. I see the data in the broker's send queue, but
> there's a delay of 5 seconds before it's sent to the client. Can you create
> a JIRA?
>
> Thanks,
> Jason
>
>
>
> On Thu, Jan 21, 2016 at 11:30 AM, Samya Maiti <sa...@gmail.com>
> wrote:
>
>> +1, facing same issue.
>> -Sam
>>> On 22-Jan-2016, at 12:16 am, Krzysztof Ciesielski <
>> krzysztof.ciesielski@softwaremill.com> wrote:
>>>
>>> Hello, I'm running into an issue with the new consumer in Kafka 0.9.0.0.
>>> Here's a runnable gist illustrating the problem:
>>> https://gist.github.com/kciesielski/054bb4359a318aa17561 (requires
>> Kafka on
>>> localhost:9092)
>>>
>>> Scenario description:
>>> First, a producer writes 500000 elements into a topic
>>> Then, a consumer starts to read, polling in a loop.
>>> When "max.partition.fetch.bytes" is set to a relatively small value, each
>>> "consumer.poll()" returns a batch of messages.
>>> If this value is left as default, the output tends to look like this:
>>>
>>> Poll returned 13793 elements
>>> Poll returned 13793 elements
>>> Poll returned 13793 elements
>>> Poll returned 13793 elements
>>> Poll returned 0 elements
>>> Poll returned 0 elements
>>> Poll returned 0 elements
>>> Poll returned 0 elements
>>> Poll returned 13793 elements
>>> Poll returned 13793 elements
>>>
>>> As we can see, there are weird "gaps" when poll returns 0 elements for
>> some
>>> time. What is the reason for that? Maybe there are some good practices
>>> about setting "max.partition.fetch.bytes" which I don't follow?
>>>
>>> Bests,
>>> Chris
>>
>>
Re: Kafka 0.9 -> consumer.poll() occasionally returns 0 elements
Posted by Jason Gustafson <ja...@confluent.io>.
Hi Krzysztof,
This is definitely weird. I see the data in the broker's send queue, but
there's a delay of 5 seconds before it's sent to the client. Can you create
a JIRA?
Thanks,
Jason
On Thu, Jan 21, 2016 at 11:30 AM, Samya Maiti <sa...@gmail.com>
wrote:
> +1, facing same issue.
> -Sam
> > On 22-Jan-2016, at 12:16 am, Krzysztof Ciesielski <
> krzysztof.ciesielski@softwaremill.com> wrote:
> >
> > Hello, I'm running into an issue with the new consumer in Kafka 0.9.0.0.
> > Here's a runnable gist illustrating the problem:
> > https://gist.github.com/kciesielski/054bb4359a318aa17561 (requires
> Kafka on
> > localhost:9092)
> >
> > Scenario description:
> > First, a producer writes 500000 elements into a topic
> > Then, a consumer starts to read, polling in a loop.
> > When "max.partition.fetch.bytes" is set to a relatively small value, each
> > "consumer.poll()" returns a batch of messages.
> > If this value is left as default, the output tends to look like this:
> >
> > Poll returned 13793 elements
> > Poll returned 13793 elements
> > Poll returned 13793 elements
> > Poll returned 13793 elements
> > Poll returned 0 elements
> > Poll returned 0 elements
> > Poll returned 0 elements
> > Poll returned 0 elements
> > Poll returned 13793 elements
> > Poll returned 13793 elements
> >
> > As we can see, there are weird "gaps" when poll returns 0 elements for
> some
> > time. What is the reason for that? Maybe there are some good practices
> > about setting "max.partition.fetch.bytes" which I don't follow?
> >
> > Bests,
> > Chris
>
>
Re: Kafka 0.9 -> consumer.poll() occasionally returns 0 elements
Posted by Samya Maiti <sa...@gmail.com>.
+1, facing same issue.
-Sam
> On 22-Jan-2016, at 12:16 am, Krzysztof Ciesielski <kr...@softwaremill.com> wrote:
>
> Hello, I'm running into an issue with the new consumer in Kafka 0.9.0.0.
> Here's a runnable gist illustrating the problem:
> https://gist.github.com/kciesielski/054bb4359a318aa17561 (requires Kafka on
> localhost:9092)
>
> Scenario description:
> First, a producer writes 500000 elements into a topic
> Then, a consumer starts to read, polling in a loop.
> When "max.partition.fetch.bytes" is set to a relatively small value, each
> "consumer.poll()" returns a batch of messages.
> If this value is left as default, the output tends to look like this:
>
> Poll returned 13793 elements
> Poll returned 13793 elements
> Poll returned 13793 elements
> Poll returned 13793 elements
> Poll returned 0 elements
> Poll returned 0 elements
> Poll returned 0 elements
> Poll returned 0 elements
> Poll returned 13793 elements
> Poll returned 13793 elements
>
> As we can see, there are weird "gaps" when poll returns 0 elements for some
> time. What is the reason for that? Maybe there are some good practices
> about setting "max.partition.fetch.bytes" which I don't follow?
>
> Bests,
> Chris