KafkaProducer (0.9): Single producer multiple topics.

["Naik, Ninad" <ni...@epsilon.com>]

Hello,
We’re currently using one producer per topic. We’re considering switching to one producer for multiple topics for obvious reasons.

Now, is there a case where one topic might be slower for some reason, and not the others? And if there is such a case, how would it play out with a single producer producing to multiple topics? Would it affect throughput of other topics?

Thanks,
Ninad Naik.

Re: KafkaProducer (0.9): Single producer multiple topics.

["Naik, Ninad" <ni...@epsilon.com>]

Hello Guys,
Any update on this? Here’s what we’ve tried so far:

We briefly looked at the code and here's what we understand: Kafka server accepts a a batch of messages for each topic partition. Map[TopicAndPartition, MessageSet]

It tries to process the entire batch within the configured timeout, and sends back a response containing successful and failed records.

Now let's look at what can go wrong on a topic level, and how that would affect the entire batch:

1) Not all replicas available for a topic: Let's say a batch contains data for three topics: test1, test2, test3. For test3, not all replicas are available. This case will be detected by the leader instantly, and it will fail the records. In this case, the batch won't be delayed, and throughput for test1, and test2 won't be affected.

2) Replicas for a topic are slow. Let's say that the replicas for only test3 are slow. In this case, replicating messages for test3 will take time, which in turn will delay the entire batch, thus affecting the throughput of topics test1 and test2.

What else can go wrong?

Thanks,
Ninad Naik.

From: Ninad Naik <ni...@epsilon.com>>
Date: Friday, August 4, 2017 at 12:45 PM
To: "users@kafka.apache.org<ma...@kafka.apache.org>" <us...@kafka.apache.org>>
Subject: KafkaProducer (0.9): Single producer multiple topics.

Hello,
We’re currently using one producer per topic. We’re considering switching to one producer for multiple topics for obvious reasons.

Now, is there a case where one topic might be slower for some reason, and not the others? And if there is such a case, how would it play out with a single producer producing to multiple topics? Would it affect throughput of other topics?

Thanks,
Ninad Naik.