Concumer with topicFilter

[Neil Obeid <ne...@gmail.com>]

Hi everyone,

I have a use-case where I need to use many topics (about 500) each topic
have different number of partitions, some as low as 10, and some as many as
20K partitions. The reasons this is a requirement is data sensitivity, we
cannot put data from different sources in same toppar.

My Kafka cluster is version 0.8.2.1 and I am using high-level consumer with
TopicFilter to target multiple topics like this:

Consumer-Group-A = (consumers a1, a2, a3) => topicFilter = "A1|A2|A3"
Consumer-Group-B = (consumers b1, b2, b3) => topicFilter = "B1|...|B10"
...
Consumer-Group-Z = (consumer z1, z2, z3) => topicFilter = "Z1|...|Z50"


Two observations:

*Issue1*: If Topic A1 was created after I started my consumers in
Consumer-Group-A, it is not picked up, I do not receive messages from A1
until I restart consumers in that group.

*Issue2*: This arrangement is working fine for topics that have less than
5K partitions, I noticed for all my topics that have 5K+ partitions, the
consumer is unstable, sometimes it cannot connect to the cluster, long
communications with Zookeeper, and I see many instances of the error
ArrayIndexOutOfBoundsException as in here KAFKA-2174


Anyone have a similar design ran through these issues?

Thank you
Neil