Java Consumers Threads vs Topics

[Riccardo Ferrari <fe...@gmail.com>]

Hi list,

I need some input on best practices on wiritng Java Kafka (0.10.1.0)
consumers.

*The scenario:*
A java distributed system sending/receving messages, currently based on
Akka + RabbitMQ.
A reasonably low number of channels (~dozen) (mapped to Kafka topics)
however it can potentially grow to a high number (~thousands) as the system
scale.

*The requirement:*
Use Kafka as replacement for RabbitMQ (basically as a queue).
Disabled offset auto-commit.

According to their Javadoc there are mainly two options:
https://kafka.apache.org/0101/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#multithreaded
#1. One thread per consumer (and per topic)
#2. Decouple message consuming from processing

*The problem with #1:*
While this solution is very easy to implement, it seems that every Kafka
Consumer is adding (too much) load to the system.
System load get very high (7.0+ on a dual-core vm). This means 99% cpu
utilization with NO io wait.

I actually tried 2 different implementation:
1) Main Thread.run() polling from Kafka and putting messages into a local
concurrent queue. a commodity method (getMessages) to retrieve messages
from the locally populated queue.
2) Put the consumer.poll() logic straight in getMessages call.

No big difference. Taking thread dumps and checking them against system
threads all seems to point the finger agains the consumer.poll() logic.

Are there any server/client side tuning that can help?
Any suggestion on what to investigate further in order to get a clear
answer on why those few threads are adding so much CPU usage?

*Before trying #2:*
I have not tried this solution yet. My main concern is that my consumer
threads will have to handle multiple topics.
Is there a "best practice" or limit in terms of topics per consumer thread?

Any help is much appreciated