High Level Consumer use case

[Sreenivasulu NALLAPATI <sr...@amadeus.com>]

Hello,

I would like to introduce myself first. I am Sreeni working as a technical lead with Amadeus. We planned to use Kafka for our billing messages.

Here is my use case:
We wanted to run a batch job every 15min with High Level Consumer by changing auto.commit.enable=false.
For now, we have a topic with 20 partitions and we would like to start 20 consumers one per topic. In the future the partition count may go up and for scalability we want to use High Level Consumer.
We were unable to find a best way to do this with High Level Consumer.
Can you please suggest how to map one consumer to one/more partitions? We are using Java Consumer API for this activity.

Thanks
Sreenivasulu Nallapati
Senior Member Technical Staff
R&D-ALB-AQG-DUI-LBD
Amadeus Software Labs, India Pvt. Ltd
T: 4914 6827
M: +919902065578
sreenivasulu.nallapati@amadeus.com
[cid:image001.png@01D0C460.B984FF70]

Re: High Level Consumer use case

[Stevo Slavić <ss...@gmail.com>]

Hello Sreenivasulu,

It's automatic, just start them, and as each HLC starts it registers in ZK,
rebalancing of the HLC to partition assignments happens.

Be gentle when starting consumers, there is a bug reported that if multiple
HLCs are started in short time, some of them may end up without any
partition assigned, while others could have more than one assigned (see
https://issues.apache.org/jira/browse/KAFKA-2329 ).

Kind regards,
Stevo Slavic.

On Wed, Jul 22, 2015 at 6:02 AM, Sreenivasulu NALLAPATI <
sreenivasulu.nallapati@amadeus.com> wrote:

>  Hello,
>
>
>
> I would like to introduce myself first. I am Sreeni working as a technical
> lead with Amadeus. We planned to use Kafka for our billing messages.
>
>
>
> Here is my use case:
>
> We wanted to run a batch job every 15min with High Level Consumer by
> changing *auto.commit.enable=false. *
>
> For now, we have a topic with 20 partitions and we would like to start 20
> consumers one per topic. In the future the partition count may go up and
> for scalability we want to use High Level Consumer.
>
> We were unable to find a best way to do this with High Level Consumer.
>
> Can you please suggest how to map one consumer to one/more partitions? We
> are using Java Consumer API for this activity.
>
>
>
> Thanks
> *Sreenivasulu Nallapati*
> * Senior Member Technical Staff*
> R&D-ALB-AQG-DUI-LBD
> Amadeus Software Labs, India Pvt. Ltd
> T: 4914 6827
> M: +919902065578
> sreenivasulu.nallapati@amadeus.com
>
>
>