You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Gowtham S <go...@gmail.com> on 2020/09/17 07:37:31 UTC
Poll Records Return Zero Records
Hello All,
We are consuming a topic with a single partition, the consumer.poll(1000)
returns "0" records mostly event if we have more than 10000 records in lag.
In which case it will behave like this.
We are using Kafka-2.4.0 client and 2.4.0 broker. The single record size
is 100Kb.
Consumer configuration
- fetch.max.bytes=102428800
- fetch.min.bytes=1
- receive.buffer.bytes=-1
- max.partition.fetch.bytes=102428800
- max.poll.records=1000
Can anyone help us to resolve this?
Thanks and regards,
Gowtham S, MCA
Re: Poll Records Return Zero Records
Posted by Gowtham S <go...@gmail.com>.
>
> Hi Gowtham, that output indicates that there is no lag for the consumer
> group involved. What makes you think there is?
>
Yes, there is no lag now, when I had huge numbers of lags, I saw that most
numbers of the poll calls return zero records, so we thought if we can
solve this we can able to increase consumer throughput.
With regards,
Gowtham S, MCA
On Fri, 18 Sep 2020 at 14:07, 子明小友 <xi...@gmail.com> wrote:
> Interesting!
>
> Liam Clarke-Hutchinson <li...@adscale.co.nz> 于 2020年9月18日周五 15:23写道：
>
> > Hi Gowtham, that output indicates that there is no lag for the consumer
> > group involved. What makes you think there is?
> >
> > On Fri, Sep 18, 2020 at 7:21 PM Gowtham S <go...@gmail.com>
> > wrote:
> >
> > > Thanks for the reply Liam Clarke-Hutchinson, Shaohan Yin
> > >
> > > > Hi, what is the output of kafka-consumer-groups.sh --describe for
> that
> > > > consumer's group please?
> > >
> > > GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID
> > HOST
> > > CLIENT-ID
> > >
> > > oneg one 0 4838037 4838037 0 fa9d1602
> > > localhost
> > > fa9d1602
> > >
> > >
> > > The poll doesn't return always zero if we make 100 poll calls in 100
> > > seconds it returns 0 records on 70% of poll calls.
> > >
> > >
> > > You could check if there are any throttles on the broker. The
> > > > metric kafka_server_quota_throttle_time might do you a favor.
> > >
> > >
> > > There are no quota throttles. We are using the default quota
> > configuration.
> > > Please let us know if we miss any configuration/metrics to
> change/listen?
> > >
> > >
> > > With regards,
> > > Gowtham S, MCA
> > >
> > >
> > >
> > > On Fri, 18 Sep 2020 at 09:51, Shaohan Yin <sh...@gmail.com>
> wrote:
> > >
> > > > Hi Gowtham,
> > > >
> > > > You could check if there are any throttles on the broker. The
> > > > metric kafka_server_quota_throttle_time might do you a favor.
> > > >
> > > > Cheers
> > > >
> > > > On Fri, 18 Sep 2020 at 07:20, Liam Clarke-Hutchinson <
> > > > liam.clarke@adscale.co.nz> wrote:
> > > >
> > > > > Hi, what is the output of kafka-consumer-groups.sh --describe for
> > that
> > > > > consumer's group please?
> > > > >
> > > > > On Thu, 17 Sep. 2020, 7:37 pm Gowtham S, <gowtham.co.inc@gmail.com
> >
> > > > wrote:
> > > > >
> > > > > > Hello All,
> > > > > > We are consuming a topic with a single partition, the
> > > > consumer.poll(1000)
> > > > > > returns "0" records mostly event if we have more than 10000
> records
> > > in
> > > > > lag.
> > > > > > In which case it will behave like this.
> > > > > > We are using Kafka-2.4.0 client and 2.4.0 broker. The single
> > record
> > > > size
> > > > > > is 100Kb.
> > > > > >
> > > > > > Consumer configuration
> > > > > >
> > > > > > - fetch.max.bytes=102428800
> > > > > > - fetch.min.bytes=1
> > > > > > - receive.buffer.bytes=-1
> > > > > > - max.partition.fetch.bytes=102428800
> > > > > > - max.poll.records=1000
> > > > > >
> > > > > > Can anyone help us to resolve this?
> > > > > >
> > > > > > Thanks and regards,
> > > > > > Gowtham S, MCA
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Poll Records Return Zero Records
Posted by 子明小友 <xi...@gmail.com>.
Interesting!
Liam Clarke-Hutchinson <li...@adscale.co.nz> 于 2020年9月18日周五 15:23写道：
> Hi Gowtham, that output indicates that there is no lag for the consumer
> group involved. What makes you think there is?
>
> On Fri, Sep 18, 2020 at 7:21 PM Gowtham S <go...@gmail.com>
> wrote:
>
> > Thanks for the reply Liam Clarke-Hutchinson, Shaohan Yin
> >
> > > Hi, what is the output of kafka-consumer-groups.sh --describe for that
> > > consumer's group please?
> >
> > GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID
> HOST
> > CLIENT-ID
> >
> > oneg one 0 4838037 4838037 0 fa9d1602
> > localhost
> > fa9d1602
> >
> >
> > The poll doesn't return always zero if we make 100 poll calls in 100
> > seconds it returns 0 records on 70% of poll calls.
> >
> >
> > You could check if there are any throttles on the broker. The
> > > metric kafka_server_quota_throttle_time might do you a favor.
> >
> >
> > There are no quota throttles. We are using the default quota
> configuration.
> > Please let us know if we miss any configuration/metrics to change/listen?
> >
> >
> > With regards,
> > Gowtham S, MCA
> >
> >
> >
> > On Fri, 18 Sep 2020 at 09:51, Shaohan Yin <sh...@gmail.com> wrote:
> >
> > > Hi Gowtham,
> > >
> > > You could check if there are any throttles on the broker. The
> > > metric kafka_server_quota_throttle_time might do you a favor.
> > >
> > > Cheers
> > >
> > > On Fri, 18 Sep 2020 at 07:20, Liam Clarke-Hutchinson <
> > > liam.clarke@adscale.co.nz> wrote:
> > >
> > > > Hi, what is the output of kafka-consumer-groups.sh --describe for
> that
> > > > consumer's group please?
> > > >
> > > > On Thu, 17 Sep. 2020, 7:37 pm Gowtham S, <go...@gmail.com>
> > > wrote:
> > > >
> > > > > Hello All,
> > > > > We are consuming a topic with a single partition, the
> > > consumer.poll(1000)
> > > > > returns "0" records mostly event if we have more than 10000 records
> > in
> > > > lag.
> > > > > In which case it will behave like this.
> > > > > We are using Kafka-2.4.0 client and 2.4.0 broker. The single
> record
> > > size
> > > > > is 100Kb.
> > > > >
> > > > > Consumer configuration
> > > > >
> > > > > - fetch.max.bytes=102428800
> > > > > - fetch.min.bytes=1
> > > > > - receive.buffer.bytes=-1
> > > > > - max.partition.fetch.bytes=102428800
> > > > > - max.poll.records=1000
> > > > >
> > > > > Can anyone help us to resolve this?
> > > > >
> > > > > Thanks and regards,
> > > > > Gowtham S, MCA
> > > > >
> > > >
> > >
> >
>
Re: Poll Records Return Zero Records
Posted by Liam Clarke-Hutchinson <li...@adscale.co.nz>.
Hi Gowtham, that output indicates that there is no lag for the consumer
group involved. What makes you think there is?
On Fri, Sep 18, 2020 at 7:21 PM Gowtham S <go...@gmail.com> wrote:
> Thanks for the reply Liam Clarke-Hutchinson, Shaohan Yin
>
> > Hi, what is the output of kafka-consumer-groups.sh --describe for that
> > consumer's group please?
>
> GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST
> CLIENT-ID
>
> oneg one 0 4838037 4838037 0 fa9d1602
> localhost
> fa9d1602
>
>
> The poll doesn't return always zero if we make 100 poll calls in 100
> seconds it returns 0 records on 70% of poll calls.
>
>
> You could check if there are any throttles on the broker. The
> > metric kafka_server_quota_throttle_time might do you a favor.
>
>
> There are no quota throttles. We are using the default quota configuration.
> Please let us know if we miss any configuration/metrics to change/listen?
>
>
> With regards,
> Gowtham S, MCA
>
>
>
> On Fri, 18 Sep 2020 at 09:51, Shaohan Yin <sh...@gmail.com> wrote:
>
> > Hi Gowtham,
> >
> > You could check if there are any throttles on the broker. The
> > metric kafka_server_quota_throttle_time might do you a favor.
> >
> > Cheers
> >
> > On Fri, 18 Sep 2020 at 07:20, Liam Clarke-Hutchinson <
> > liam.clarke@adscale.co.nz> wrote:
> >
> > > Hi, what is the output of kafka-consumer-groups.sh --describe for that
> > > consumer's group please?
> > >
> > > On Thu, 17 Sep. 2020, 7:37 pm Gowtham S, <go...@gmail.com>
> > wrote:
> > >
> > > > Hello All,
> > > > We are consuming a topic with a single partition, the
> > consumer.poll(1000)
> > > > returns "0" records mostly event if we have more than 10000 records
> in
> > > lag.
> > > > In which case it will behave like this.
> > > > We are using Kafka-2.4.0 client and 2.4.0 broker. The single record
> > size
> > > > is 100Kb.
> > > >
> > > > Consumer configuration
> > > >
> > > > - fetch.max.bytes=102428800
> > > > - fetch.min.bytes=1
> > > > - receive.buffer.bytes=-1
> > > > - max.partition.fetch.bytes=102428800
> > > > - max.poll.records=1000
> > > >
> > > > Can anyone help us to resolve this?
> > > >
> > > > Thanks and regards,
> > > > Gowtham S, MCA
> > > >
> > >
> >
>
Re: Poll Records Return Zero Records
Posted by Gowtham S <go...@gmail.com>.
Thanks for the reply Liam Clarke-Hutchinson, Shaohan Yin
> Hi, what is the output of kafka-consumer-groups.sh --describe for that
> consumer's group please?
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST
CLIENT-ID
oneg one 0 4838037 4838037 0 fa9d1602 localhost
fa9d1602
The poll doesn't return always zero if we make 100 poll calls in 100
seconds it returns 0 records on 70% of poll calls.
You could check if there are any throttles on the broker. The
> metric kafka_server_quota_throttle_time might do you a favor.
There are no quota throttles. We are using the default quota configuration.
Please let us know if we miss any configuration/metrics to change/listen?
With regards,
Gowtham S, MCA
On Fri, 18 Sep 2020 at 09:51, Shaohan Yin <sh...@gmail.com> wrote:
> Hi Gowtham,
>
> You could check if there are any throttles on the broker. The
> metric kafka_server_quota_throttle_time might do you a favor.
>
> Cheers
>
> On Fri, 18 Sep 2020 at 07:20, Liam Clarke-Hutchinson <
> liam.clarke@adscale.co.nz> wrote:
>
> > Hi, what is the output of kafka-consumer-groups.sh --describe for that
> > consumer's group please?
> >
> > On Thu, 17 Sep. 2020, 7:37 pm Gowtham S, <go...@gmail.com>
> wrote:
> >
> > > Hello All,
> > > We are consuming a topic with a single partition, the
> consumer.poll(1000)
> > > returns "0" records mostly event if we have more than 10000 records in
> > lag.
> > > In which case it will behave like this.
> > > We are using Kafka-2.4.0 client and 2.4.0 broker. The single record
> size
> > > is 100Kb.
> > >
> > > Consumer configuration
> > >
> > > - fetch.max.bytes=102428800
> > > - fetch.min.bytes=1
> > > - receive.buffer.bytes=-1
> > > - max.partition.fetch.bytes=102428800
> > > - max.poll.records=1000
> > >
> > > Can anyone help us to resolve this?
> > >
> > > Thanks and regards,
> > > Gowtham S, MCA
> > >
> >
>
Re: Poll Records Return Zero Records
Posted by Shaohan Yin <sh...@gmail.com>.
Hi Gowtham,
You could check if there are any throttles on the broker. The
metric kafka_server_quota_throttle_time might do you a favor.
Cheers
On Fri, 18 Sep 2020 at 07:20, Liam Clarke-Hutchinson <
liam.clarke@adscale.co.nz> wrote:
> Hi, what is the output of kafka-consumer-groups.sh --describe for that
> consumer's group please?
>
> On Thu, 17 Sep. 2020, 7:37 pm Gowtham S, <go...@gmail.com> wrote:
>
> > Hello All,
> > We are consuming a topic with a single partition, the consumer.poll(1000)
> > returns "0" records mostly event if we have more than 10000 records in
> lag.
> > In which case it will behave like this.
> > We are using Kafka-2.4.0 client and 2.4.0 broker. The single record size
> > is 100Kb.
> >
> > Consumer configuration
> >
> > - fetch.max.bytes=102428800
> > - fetch.min.bytes=1
> > - receive.buffer.bytes=-1
> > - max.partition.fetch.bytes=102428800
> > - max.poll.records=1000
> >
> > Can anyone help us to resolve this?
> >
> > Thanks and regards,
> > Gowtham S, MCA
> >
>
Re: Poll Records Return Zero Records
Posted by Liam Clarke-Hutchinson <li...@adscale.co.nz>.
Hi, what is the output of kafka-consumer-groups.sh --describe for that
consumer's group please?
On Thu, 17 Sep. 2020, 7:37 pm Gowtham S, <go...@gmail.com> wrote:
> Hello All,
> We are consuming a topic with a single partition, the consumer.poll(1000)
> returns "0" records mostly event if we have more than 10000 records in lag.
> In which case it will behave like this.
> We are using Kafka-2.4.0 client and 2.4.0 broker. The single record size
> is 100Kb.
>
> Consumer configuration
>
> - fetch.max.bytes=102428800
> - fetch.min.bytes=1
> - receive.buffer.bytes=-1
> - max.partition.fetch.bytes=102428800
> - max.poll.records=1000
>
> Can anyone help us to resolve this?
>
> Thanks and regards,
> Gowtham S, MCA
>