Should there be separate Kafka for producers and consumers

[nitin agarwal <ni...@gmail.com>]

Hi All,

We have Camus consumer which reads data from Kafka and writes to HDFS.
Whenever this consumer runs, there is impact on producer latencies. We
don't want producer latencies to be increased in anyway.

I want to know which design pattern is more suited for supporting this use
case.
1) Single Kafka cluster with efficient throttling limits for consumers.
2) Separate Kafka cluster for producers and consumers. Data is replicated
using MirrorMaker.

Thanks,
Nitin

Re: Should there be separate Kafka for producers and consumers

[Ryanne Dolan <ry...@gmail.com>]

Hey Nitin, it's not uncommon to use separate clusters and MirrorMaker for
this purpose, e.g. separate ingest and ETL clusters. That way your ETL
cluster has only one producer (MM) and only one consumer (Camus).

It's also possible you just need more brokers and/or more partitions.

Ryanne

On Sat, May 23, 2020, 2:38 AM nitin agarwal <ni...@gmail.com> wrote:

> Hi All,
>
> We have Camus consumer which reads data from Kafka and writes to HDFS.
> Whenever this consumer runs, there is impact on producer latencies. We
> don't want producer latencies to be increased in anyway.
>
> I want to know which design pattern is more suited for supporting this use
> case.
> 1) Single Kafka cluster with efficient throttling limits for consumers.
> 2) Separate Kafka cluster for producers and consumers. Data is replicated
> using MirrorMaker.
>
> Thanks,
> Nitin
>