how to get parallel processing with kafka and akka

[Laxmi Narayan <ni...@gmail.com>]

Hi,

 Kafka has partitions and akka can do parallel processing.
I have one perfect use-case where I have to read data in parallel.
But seems like partitions does not give me any extra info other than
partition number and how do i make sure that data_x should always go to
x-partition next time  and I want to keep it consistent , means if new
data_k comes that should create one more partition k, is there is any way
of associating some meta data so that next time I should push it back in
same and if not present , i can create one more.

may be I can keep created 10,000 partitions and keep associating over and
over until
my all partitions are consumed.

Example :
eg: I have twitter handles, how do I make sure to feed all tweets from
same handle go to same partition and It should not affect once a new handle
gets added.

2. In akka I can keep ready my consumers inside actors and keep listing to
partitions in parallel. But for newly added handle how do I create one more
actor-child to start listening, I do not want to scan meta data entirely
(seems like , after 1mints I can do
scan is only way to get this done.) .


Keep learning keep moving .....