Question on Kafka Consumer

[Govindarajan Srinivasaraghavan <go...@gmail.com>]

Hello All,

I am working on building an application on cloud and below are the high
level requirements.

- Ingest data into Kafka with say 100 partitions (Incoming rate - 100,000
msgs/sec)
- Read data from Kafka and process each data (Do some computation, compare
with old data, find location etc) real time

I need some suggestions on how to distribute and manage kafka consumers for
my scenario.

- If I deploy my application with kafka consumer threads then how can I
distribute load and rebalance it if I add more partitions or during failure.
- Should I be using some kind of data streaming platform like storm, flink,
samza to get the data from kafka and stream it realtime?

Thanks a bunch and appreciate all your help.

Regards,
Govind