Re: Kafka Consumer manual partition assignment

[Sagar <sa...@gmail.com>]

Hi,

Any help here would be highly appreciated :)

Sagar.

On Wed, 27 Dec 2017 at 07:53, Sagar <sa...@gmail.com> wrote:

> We have a use case where in we want to assign partitions manually for a
> set of topics to allow fine grained control of the records we are fetching.
>
> Basically what we are trying to achieve is that a group of messages which
> logically belong to a particular entity should be sent to the same
> partition . So, Topic A and topic B both have a certain field which are
> unique and common( say some id = 101) so we want to send both of those
> records to partition 1 ( 101 % 10 is our simple partition assignment
> strategy written in both producer and Kafka Connect interceptor.
>
> Fro what I understood, if I want to have a consumer which listens to
> partition 1 for both Topic A and Topic B, then we need to use the assign
> method.
>
> I have been reading up a bit and what has been mentioned everywhere is
> that we won't have any rebalancing triggered. Also, I tried a simple use
> case where in I didn't poll for more than the value configured in
> group.max.session.timeout.ms but it didn't die. Is it because its not
> part of a consumer group per se?
>
> So just wanted to understand what points should we declare a consumer is
> dead so that we can spin up a new consumer for the same topic partition. We
> are using AWS ECS for running our consumers so target group would spin up a
> new consumer based upon health checks.
>
> Any examples + guidelines around this would be highly appreciated.
>
> Thanks!
> Sagar.
>