managing a Kafka consumer group using Helix

["vlad.gm@gmail.com" <vl...@gmail.com>]

Dear all,

I am sure the following question appeared inside Linkedin before :)

We would like to manage a Kafka consumer group using Helix, that is have
multiple consumer instances and assign topics and their partitions among
the consumers automatically. The consumer group would use a whitelist to
select the topics, which means that the topic/partitions list is dynamic
and can change quite frequently. I can see each topic mapping to a Helix
resource or, alternatively, using a single Helix resource to handle all
topics. We are most likely to use a custom rebalancer in order to use
throughput metrics in order to balance traffic, not partition count.

Here are a few questions:
1) If we are to use a resource per topic, would we be able to later on
jointly rebalance multiple resources at once? The current rebalancer
callback seems to handle a single resource. Would we have to actually
manage the multiple resources in the background and just use the callback
when we are asked what to do with that resource?
2) If we are to put all topics and their partitions in a single resource we
are likely to quickly go over the amount of data that can be stored in a ZK
node. I remember that buckets can help with that problem. Can the number of
buckets increase dynamically with the number of partitions?
3) How big of a problem would it be to have an environment in which the
group of administered partitions changes quite often? I guess that with one
resource per topic this would not be a big issue, however it might be a
problem with a single resource for all topics.
4) Is there a limit on the number of resources that can be stored in a
single cluster, because of the amount of data that can be stored in a
single ZK node?

Regards,
Vlad

Re: managing a Kafka consumer group using Helix

[kishore g <g....@gmail.com>]

you bet :-)

Kafka had its own consumer group implementation before Helix was
implemented.

We have implemented a similar consumer group pattern for Databus, so its
definitely possible to extend it to Kafka.

Regarding your questions.

1. Yes, current rebalancer handles one resource at a time, we had some
discussion about having a composite rebalancer that rebalances multiple
resources at once. Even without this feature, its possible to read the
entire cluster data from the rebalancer and make the decision considering
other resources.
2. Probably not a good idea to put all topics in one resource even with
bucketing support.
3. Theoretically, group of changes does not matter if its within a resource
v/s across resource. Its the overall change that matters.
4. We have gone up to 2000 resources across the cluster. This is not using
one ZNode. I dont think its a good idea to have one ZNode for all resources.


thanks,
Kishore G



On Wed, Dec 3, 2014 at 10:57 AM, vlad.gm@gmail.com <vl...@gmail.com>
wrote:

>
> Dear all,
>
> I am sure the following question appeared inside Linkedin before :)
>
> We would like to manage a Kafka consumer group using Helix, that is have
> multiple consumer instances and assign topics and their partitions among
> the consumers automatically. The consumer group would use a whitelist to
> select the topics, which means that the topic/partitions list is dynamic
> and can change quite frequently. I can see each topic mapping to a Helix
> resource or, alternatively, using a single Helix resource to handle all
> topics. We are most likely to use a custom rebalancer in order to use
> throughput metrics in order to balance traffic, not partition count.
>
> Here are a few questions:
> 1) If we are to use a resource per topic, would we be able to later on
> jointly rebalance multiple resources at once? The current rebalancer
> callback seems to handle a single resource. Would we have to actually
> manage the multiple resources in the background and just use the callback
> when we are asked what to do with that resource?
> 2) If we are to put all topics and their partitions in a single resource
> we are likely to quickly go over the amount of data that can be stored in a
> ZK node. I remember that buckets can help with that problem. Can the number
> of buckets increase dynamically with the number of partitions?
> 3) How big of a problem would it be to have an environment in which the
> group of administered partitions changes quite often? I guess that with one
> resource per topic this would not be a big issue, however it might be a
> problem with a single resource for all topics.
> 4) Is there a limit on the number of resources that can be stored in a
> single cluster, because of the amount of data that can be stored in a
> single ZK node?
>
> Regards,
> Vlad
>