specifying exact number of records to fetch

["Ramanan, Buvana (Nokia - US)" <bu...@nokia-bell-labs.com>]

Hello,

Do Kafka protocol & KafkaConsumer (java) client address the following need?

The caller specifies that it needs N number of records with a  max wait time of Tn milliseconds. If N records are available within Tn, the records are returned to the caller. If Tn expires, then the caller gets whatever number of records (< N) that were available.

If this functionality not readily supported by KafkaConsumer, can someone please give me the logic I can use at the caller to simulate the scenario. I need to execute an application as soon as N records (or less, in case Tn is reached) become available (at the broker itself - that is, N records since the time of last fetch).

Kafka Consumer properties allows us to specify fetch.max.wait.ms and fetch.min.bytes, with which you can indirectly achieve the above functionality in case you know the periodicity & the record size of the topic of interest. But it is too restrictive and I am looking for alternatives.

Thanks,
Buvana

RE: specifying exact number of records to fetch

["Ramanan, Buvana (Nokia - US)" <bu...@nokia-bell-labs.com>]

Thanks, Manikumar! I find that max.poll.records feature is introduced in 0.10 release.
And we are still working with Kafka 0.9.

-----Original Message-----
From: Manikumar [mailto:manikumar.reddy@gmail.com] 
Sent: Friday, September 23, 2016 1:31 PM
To: users@kafka.apache.org
Subject: Re: specifying exact number of records to fetch

"max.poll.records" config property can be used to limit the number of records returned in each consumer poll() method call.

On Fri, Sep 23, 2016 at 10:49 PM, Ramanan, Buvana (Nokia - US) < buvana.ramanan@nokia-bell-labs.com> wrote:

> Hello,
>
> Do Kafka protocol & KafkaConsumer (java) client address the following need?
>
> The caller specifies that it needs N number of records with a  max 
> wait time of Tn milliseconds. If N records are available within Tn, 
> the records are returned to the caller. If Tn expires, then the caller 
> gets whatever number of records (< N) that were available.
>
> If this functionality not readily supported by KafkaConsumer, can 
> someone please give me the logic I can use at the caller to simulate the scenario.
> I need to execute an application as soon as N records (or less, in 
> case Tn is reached) become available (at the broker itself - that is, 
> N records since the time of last fetch).
>
> Kafka Consumer properties allows us to specify fetch.max.wait.ms and 
> fetch.min.bytes, with which you can indirectly achieve the above 
> functionality in case you know the periodicity & the record size of 
> the topic of interest. But it is too restrictive and I am looking for 
> alternatives.
>
> Thanks,
> Buvana
>

Re: specifying exact number of records to fetch

[Manikumar <ma...@gmail.com>]

"max.poll.records" config property can be used to limit the number of
records returned
in each consumer poll() method call.

On Fri, Sep 23, 2016 at 10:49 PM, Ramanan, Buvana (Nokia - US) <
buvana.ramanan@nokia-bell-labs.com> wrote:

> Hello,
>
> Do Kafka protocol & KafkaConsumer (java) client address the following need?
>
> The caller specifies that it needs N number of records with a  max wait
> time of Tn milliseconds. If N records are available within Tn, the records
> are returned to the caller. If Tn expires, then the caller gets whatever
> number of records (< N) that were available.
>
> If this functionality not readily supported by KafkaConsumer, can someone
> please give me the logic I can use at the caller to simulate the scenario.
> I need to execute an application as soon as N records (or less, in case Tn
> is reached) become available (at the broker itself - that is, N records
> since the time of last fetch).
>
> Kafka Consumer properties allows us to specify fetch.max.wait.ms and
> fetch.min.bytes, with which you can indirectly achieve the above
> functionality in case you know the periodicity & the record size of the
> topic of interest. But it is too restrictive and I am looking for
> alternatives.
>
> Thanks,
> Buvana
>