Consumption: poll one message at a time

[André <ol...@gmail.com>]

Hi there

I've just started evaluating Kafka as an additional message broker to be
supported by the platform I work for. Currently, we support AMQP and JMS.

One of our use-cases for messaging is to use queues for distributing tasks
to workers as they become available.

I've noticed that by using a common group id and multiple partitions, the
messages are fairly distributed among consumers of a given topic, but still
messages might be consumed in batches and we would prefer to poll only one
message at a time from the broker, which is the behavior we currently
expect while working with the other message brokers we support.

Which leads me to ask: Is there a way of determining that a Kafka consumer
will poll only one message at a time?

Cheers
Andre

Re: Consumption: poll one message at a time

[Ismael Juma <is...@juma.me.uk>]

Hi Andre,

Please see KIP-41:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-41%3A+KafkaConsumer+Max+Records

The aim is to include this in the next release of Kafka.

Ismael

On Mon, Feb 15, 2016 at 12:21 PM, André <ol...@gmail.com> wrote:

> Hi there
>
> I've just started evaluating Kafka as an additional message broker to be
> supported by the platform I work for. Currently, we support AMQP and JMS.
>
> One of our use-cases for messaging is to use queues for distributing tasks
> to workers as they become available.
>
> I've noticed that by using a common group id and multiple partitions, the
> messages are fairly distributed among consumers of a given topic, but still
> messages might be consumed in batches and we would prefer to poll only one
> message at a time from the broker, which is the behavior we currently
> expect while working with the other message brokers we support.
>
> Which leads me to ask: Is there a way of determining that a Kafka consumer
> will poll only one message at a time?
>
> Cheers
> Andre
>