Single thread, Multiple partitions

["Sudarshan Kadambi (BLOOMBERG/ 731 LEXIN)" <sk...@bloomberg.net>]

Let's say I've a single consumer thread reading off multiple partitions (I'll have around 10K partitions). As per the documentation on https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example, there are no guarantees on the order in which messages are read off the set of partitions. If I wanted to enforce priority-weighted round robin reads off the partitions, could I get a pointer on what code to fiddle with? Thanks!

RE: Single thread, Multiple partitions

["Seshadri, Balaji" <Ba...@dish.com>]

I think you are looking for accessing messages from set of partitions by your own policy.You should use simple consumers in 0.8 and maintain the offsets you have read.

https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example

If it is 0.9 I'm yet to come upto speed.

Thanks,

Balaji


-----Original Message-----
From: Sudarshan Kadambi (BLOOMBERG/ 731 LEXIN) [mailto:skadambi@bloomberg.net] 
Sent: Tuesday, April 08, 2014 11:58 AM
To: users@kafka.apache.org
Subject: Single thread, Multiple partitions

Let's say I've a single consumer thread reading off multiple partitions (I'll have around 10K partitions). As per the documentation on https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example, there are no guarantees on the order in which messages are read off the set of partitions. If I wanted to enforce priority-weighted round robin reads off the partitions, could I get a pointer on what code to fiddle with? Thanks!