Concurrency and KafkaConsumer?

["Richard L. Burton III" <mr...@gmail.com>]

I have a case in which I'm iterating over the results of a poll(..)
in KafkaConsumer. While I'm iterating over the records, the thread is
blocked. e.g., I'm writing to a BlockingQueue that's full.

Will this have any impact on the KafkaConsumer?

I have one more follow up question.

Let's say I have 20 topics. I plan to spawn 20 KafkaConsumer's to process
each topic because I'm looking to rate limit the speed at which the records
are processed to avoid overloading external systems.

The proper usage of KafkaConsumer in this case would be:


   - Create a thread with a new instance of KafkaConsumer used in it.
   - Ensure the group id for that consumer is unique
   - Ensure only one consumer for each topic is running to avoid
   re-balancing



-- 
-Richard L. Burton III
@rburton

Re: Concurrency and KafkaConsumer?

[Jens Rantil <je...@tink.se>]

Hi Richard,

> Will this have any impact on the KafkaConsumer?

Make sure you are calling poll() more often than session.timeout.ms,
otherwise Kafka will initiate a rebalance of the consumers for that topic.
An alternative is that you pause() your consumer while being blocked.

You might want to look at
https://cwiki.apache.org/confluence/display/KAFKA/KIP-41%3A+KafkaConsumer+Max+Records
which
will be included in the next release. This will allow you to rate limit the
number of incoming messages from the poll call instead of rate limit the
messages you are sending outgoing. The advantage of that will also be that
the consumer will make sure you don't loose messages if your application
crashes when messages are consumed, but lingering in your BlockingQueue.

> The proper usage of KafkaConsumer in this case would be:
>
>    - Create a thread with a new instance of KafkaConsumer used in it.
>    - Ensure the group id for that consumer is unique
>    - Ensure only one consumer for each topic is running to avoid
>    re-balancing

Sounds sane to me. Note that a single consumer also can subscibe to
multiple topics, but that would obviously make your application single
threaded...

Hope this helps.

Cheers,
Jens

On Fri, Apr 22, 2016 at 2:33 AM Richard L. Burton III <mr...@gmail.com>
wrote:

> I have a case in which I'm iterating over the results of a poll(..)
> in KafkaConsumer. While I'm iterating over the records, the thread is
> blocked. e.g., I'm writing to a BlockingQueue that's full.
>
> Will this have any impact on the KafkaConsumer?
>
> I have one more follow up question.
>
> Let's say I have 20 topics. I plan to spawn 20 KafkaConsumer's to process
> each topic because I'm looking to rate limit the speed at which the records
> are processed to avoid overloading external systems.
>
> The proper usage of KafkaConsumer in this case would be:
>
>
>    - Create a thread with a new instance of KafkaConsumer used in it.
>    - Ensure the group id for that consumer is unique
>    - Ensure only one consumer for each topic is running to avoid
>    re-balancing
>
>
>
> --
> -Richard L. Burton III
> @rburton
>
-- 

Jens Rantil
Backend Developer @ Tink

Tink AB, Wallingatan 5, 111 60 Stockholm, Sweden
For urgent matters you can reach me at +46-708-84 18 32.