Having multi-threaded Kafka Consumer per partition, is it possible and recommended, if so any sample snippet?

["sagarcasual ." <sa...@gmail.com>]

Hello,

We are using Kafka 0.9 version and are having large number of messages
pushed to specific partition within a kafka topic. And there are multiple
such partitions within this topic. We have one consumer assigned per
partition within this topic, and we are maintaining offset manually within
the topic-partition in an outside datastore. I wanted to know if we start
getting really large number of messages in a topic-partition, is it
possible to have consumer dealing with a topic-partition to be
multi-threaded. Because it might not be possible that a consumer instance
assigned to a partition is able to finish processing all the records in the
time-span we want it finish. Is it possible to have such kind of
multi-threaded consumer with a partition. Is it recommended? Also if the
answer is YES, then how multiple threads can do offset management, because
all these threads are dealing with messages within the same partition. Any
sample snippet available?

Please note: I am asking about "*consumer dealing with a single partition
within a topic*", I am *not *looking at the consumer group for a topic
across partitions in it.

Re: Having multi-threaded Kafka Consumer per partition, is it possible and recommended, if so any sample snippet?

[Ewen Cheslack-Postava <ew...@confluent.io>]

Hi,

The new consumer is single threaded. You can layer multi-threaded
processing on top of it, but you'll definitely need to be careful about how
offset commits are handled to ensure a) processing of a message is actually
*complete* not just passed off to another thread before committing an
offset and b) that all threads have finished processing up to the offset to
commit (e.g. message 10 may take a long time to process, and completing
processing message 11 in another thread should *not* allow you to commit
offset 11). And, of course, as soon as you process messages in multiple
threads, you're going to lose ordering guarantees.

Given that this can get relatively complicated and has some significant
tradeoffs, this definitely isn't recommended for normal usage but could be
reasonable in certain situations. Depending on where time is being spent,
you might be able to use a slightly simpler approach where you let the
consumer run in one thread and only have one other thread that processes
messages in order. This requires some coordination, but is less complicated
and still preserves ordering. It is useful if a significant chunk of time
is spent in the consumer itself (and, e.g., the deserializer).

-Ewen

On Wed, Jan 20, 2016 at 1:45 PM, sagarcasual . <sa...@gmail.com>
wrote:

> Hello,
>
> We are using Kafka 0.9 version and are having large number of messages
> pushed to specific partition within a kafka topic. And there are multiple
> such partitions within this topic. We have one consumer assigned per
> partition within this topic, and we are maintaining offset manually within
> the topic-partition in an outside datastore. I wanted to know if we start
> getting really large number of messages in a topic-partition, is it
> possible to have consumer dealing with a topic-partition to be
> multi-threaded. Because it might not be possible that a consumer instance
> assigned to a partition is able to finish processing all the records in the
> time-span we want it finish. Is it possible to have such kind of
> multi-threaded consumer with a partition. Is it recommended? Also if the
> answer is YES, then how multiple threads can do offset management, because
> all these threads are dealing with messages within the same partition. Any
> sample snippet available?
>
> Please note: I am asking about "*consumer dealing with a single partition
> within a topic*", I am *not *looking at the consumer group for a topic
> across partitions in it.
>



-- 
Thanks,
Ewen