Non-blocking High-Level Consumer

[Ivan Balashov <ib...@gmail.com>]

Hi,

Is it possible to read all available messages with HLC in a
non-blocking way? E.g. read all messages and not wait for more
messages to appear in the topic.

As far as I understand, currently one has to keep high-level consumer
in a separate thread until it is shut down explicitly, but how can one
check if all available messages are in fact consumed?

Thanks,

Re: Non-blocking High-Level Consumer

[Gwen Shapira <gs...@cloudera.com>]

For Fluffka, I created a wrapping function:

  IterStatus timedHasNext() {
    try {
      long startTime = System.currentTimeMillis();
      it.hasNext();
      long endTime = System.currentTimeMillis();
      return new IterStatus(true,endTime-startTime);
    } catch (ConsumerTimeoutException e) {
      return new IterStatus(false,consumerTimeout);
    }
  }

IterStatus is basically a pair of whether there is data and how long we waited.

Calling this in a loop like that:
      while (timeWaited < timeUpperLimit) {
        iterStatus = timedHasNext();
        if (iterStatus.hasData()) {
          .... // handling data
        }
        timeWaited += iterStatus.getWaitTime();
      }

Allows us to read all messages that are available within a certain
amount of time.

You can do other cool stuff (exit after certain number of messages,
when you waited X amount of time with 0 new data, etc).

Gwen

On Mon, Sep 15, 2014 at 8:54 AM, Neha Narkhede <ne...@gmail.com> wrote:
> There isn't a very clean way to do this in the 0.8.x high level consumer.
> You can configure consumer.timeout.ms to a certain value so that the
> consumer's blocking iterator returns if no message arrives for
> consumer.timeout.ms.
>
> Thanks
> Neha
>
> On Mon, Sep 15, 2014 at 6:33 AM, Ivan Balashov <ib...@gmail.com> wrote:
>
>> Hi,
>>
>> Is it possible to read all available messages with HLC in a
>> non-blocking way? E.g. read all messages and not wait for more
>> messages to appear in the topic.
>>
>> As far as I understand, currently one has to keep high-level consumer
>> in a separate thread until it is shut down explicitly, but how can one
>> check if all available messages are in fact consumed?
>>
>> Thanks,
>>

Re: Non-blocking High-Level Consumer

[Neha Narkhede <ne...@gmail.com>]

There isn't a very clean way to do this in the 0.8.x high level consumer.
You can configure consumer.timeout.ms to a certain value so that the
consumer's blocking iterator returns if no message arrives for
consumer.timeout.ms.

Thanks
Neha

On Mon, Sep 15, 2014 at 6:33 AM, Ivan Balashov <ib...@gmail.com> wrote:

> Hi,
>
> Is it possible to read all available messages with HLC in a
> non-blocking way? E.g. read all messages and not wait for more
> messages to appear in the topic.
>
> As far as I understand, currently one has to keep high-level consumer
> in a separate thread until it is shut down explicitly, but how can one
> check if all available messages are in fact consumed?
>
> Thanks,
>