You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Cyrille Karmann <cy...@nnamrak.org> on 2020/02/13 16:52:07 UTC

Streams - multiple clusters support

Hello,

We are trying to create a streaming pipeline of data between different
Kafka clusters. Our users send data to the input Kafka cluster, and we want
to process this data and send the result to topics on another Kafka cluster.

We have different reasons for this setup, but mainly it's for isolation:
the two clusters don't have to have the same configuration and the first
"input" Kafka cluster is critical: we want to be able to do maintenance on
the second cluster without impacting the first one. Also we have more than
a thousand topics on each side so managing them separately is easier.

We are investigating different technologies for the processing part, and
Kafka Streams looked promising except it is apparently not supporting to
write in a different cluster as the one it is reading from.

I saw people on forums suggesting to write in the first cluster and use
MirrorMaker to channel the data to the output cluster. This breaks our
isolation requirements and add more latency so we don't want to do that.

I have two questions:

- Is there a reason behind the constraint that Kafka Streams can not
produce to a different cluster? I see that Kafka Streams allow to specify
different configuration for the producer but it explicitly disallow it for
ProducerConfig.BOOTSTRAP_SERVERS_CONFIG so it definitely something the
developers did not want to support (
https://kafka.apache.org/20/javadoc/org/apache/kafka/streams/StreamsConfig.html#getMainConsumerConfigs-java.lang.String-java.lang.String-)
but I am not clear why it is so.

- At the same time, there is the KafkaClientSupplier mechanism that allows
to inject our own KafkaProducer. I was actually successful in injecting
such a KafkaProducer that connects to a different cluster. The fact that I
am able to do, using a not-very documented API, something that other parts
of the Kafka Streams library try to prevent me to do, makes me wonder if I
am breaking something while doing this? In particular one thing important
to me is exactly-once processing so I want to be sure it would still work.

Thanks,
Cyrille

Re: Streams - multiple clusters support

Posted by "Matthias J. Sax" <mj...@apache.org>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

There are two reasons (some would be easier to address than others):

1) a client can only connect to one cluster; to allow data
repartitioning the producer must write the repartition data into the
input cluster, otherwise the consumer cannot read the data back

- -> we would need to introduce multiple consumers/producers to allow
for this case

  - there is also the question to which cluster repartition/changelog
topics should be placed?
  - some people might even want to have a third cluster for this
making it even more complex
  - while possible in general, it's a non-trivial architectural change


2) Kafka's exactly-once guarantees only work on a single cluster (this
is hard to change)


What kind of processing do you want to do? Kafka Connect supports
single-message-transforms than can carry you a long way already.


- -Matthias

On 2/13/20 9:15 AM, Ryanne Dolan wrote:
> Cyrille, I don't see why using MM1/2 would break your isolation
> requirement. But if you can't mirror topics for some reason
> consider Flink instead of Kafka Streams.
>
> Ryanne
>
> On Thu, Feb 13, 2020 at 10:52 AM Cyrille Karmann
> <cy...@nnamrak.org> wrote:
>
>> Hello,
>>
>> We are trying to create a streaming pipeline of data between
>> different Kafka clusters. Our users send data to the input Kafka
>> cluster, and we want to process this data and send the result to
>> topics on another Kafka cluster.
>>
>> We have different reasons for this setup, but mainly it's for
>> isolation: the two clusters don't have to have the same
>> configuration and the first "input" Kafka cluster is critical: we
>> want to be able to do maintenance on the second cluster without
>> impacting the first one. Also we have more than a thousand topics
>> on each side so managing them separately is easier.
>>
>> We are investigating different technologies for the processing
>> part, and Kafka Streams looked promising except it is apparently
>> not supporting to write in a different cluster as the one it is
>> reading from.
>>
>> I saw people on forums suggesting to write in the first cluster
>> and use MirrorMaker to channel the data to the output cluster.
>> This breaks our isolation requirements and add more latency so we
>> don't want to do that.
>>
>> I have two questions:
>>
>> - Is there a reason behind the constraint that Kafka Streams can
>> not produce to a different cluster? I see that Kafka Streams
>> allow to specify different configuration for the producer but it
>> explicitly disallow it for
>> ProducerConfig.BOOTSTRAP_SERVERS_CONFIG so it definitely
>> something the developers did not want to support (
>>
>> https://kafka.apache.org/20/javadoc/org/apache/kafka/streams/StreamsC
onfig.html#getMainConsumerConfigs-java.lang.String-java.lang.String-
>>
>>
)
>> but I am not clear why it is so.
>>
>> - At the same time, there is the KafkaClientSupplier mechanism
>> that allows to inject our own KafkaProducer. I was actually
>> successful in injecting such a KafkaProducer that connects to a
>> different cluster. The fact that I am able to do, using a
>> not-very documented API, something that other parts of the Kafka
>> Streams library try to prevent me to do, makes me wonder if I am
>> breaking something while doing this? In particular one thing
>> important to me is exactly-once processing so I want to be sure
>> it would still work.
>>
>> Thanks, Cyrille
>>
>
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEI8mthP+5zxXZZdDSO4miYXKq/OgFAl5Fi+4ACgkQO4miYXKq
/Ogxhw/+MLEVYn6cJSBL9BCwA3goeRttJO1aZXbDuM4KJHNsnioWNbtm+A102MKy
ED4a2/Hvuc9ShfhpMqYFGR0NROenTXH8F8KqXfxcavXQZPbOxZRq/VI3ppcMb3nG
KqM9PxkyVNponY/XvTRU6hYPqaJ47SXZXLENumT5F0PbhkTyNjkfuEoxNzA2C364
3nB1QkYVtPYWdGOSxyCJmFfYhrYws92NW5ytixp3b57SlItGVkGMLBZgd18l+Gm/
Dzvtmj5gBHmr8ncoum9UvUCgNBTLPlv2WrKo4wIEdjXJFnsK0UsXDHmCoO54Z55O
xik2+dIPo/x31Zyq4DOJIVCvaOOQviGwuoJPKCUhbYwtLsoIKqlE9AgPlPi25T/k
ifJ9+MxxlNjtkAKaVhVXxY9FAIrqcev5dQZA1C34YfNFt/SrMKZmf1nudX3b1thE
8KDrPucymgZTZLjP2ITgrrKb6j2OCzNS9CfKGC+qN/jH2SM3U6oAUhbgLmK2uMCT
sdTaUu0Gp+BY4C0+WKcrYapuj5r46sBqYdH5h9gbzlwpU6ghwFFrDjgA00Z3Xh9v
5Ku30/w0ZKzRZuhRpwJpygNTqzV3vf1aXyflPdXlyS8pqmf5xKoa0b6dvsG4t/z+
o9ceDgoIErpL9vnIKqeUPwt8wLFZoiMwYaH4zw87cQ72E8svLrU=
=XqS6
-----END PGP SIGNATURE-----

Re: Streams - multiple clusters support

Posted by Ryanne Dolan <ry...@gmail.com>.
Cyrille, I don't see why using MM1/2 would break your isolation
requirement. But if you can't mirror topics for some reason consider Flink
instead of Kafka Streams.

Ryanne

On Thu, Feb 13, 2020 at 10:52 AM Cyrille Karmann <cy...@nnamrak.org>
wrote:

> Hello,
>
> We are trying to create a streaming pipeline of data between different
> Kafka clusters. Our users send data to the input Kafka cluster, and we want
> to process this data and send the result to topics on another Kafka
> cluster.
>
> We have different reasons for this setup, but mainly it's for isolation:
> the two clusters don't have to have the same configuration and the first
> "input" Kafka cluster is critical: we want to be able to do maintenance on
> the second cluster without impacting the first one. Also we have more than
> a thousand topics on each side so managing them separately is easier.
>
> We are investigating different technologies for the processing part, and
> Kafka Streams looked promising except it is apparently not supporting to
> write in a different cluster as the one it is reading from.
>
> I saw people on forums suggesting to write in the first cluster and use
> MirrorMaker to channel the data to the output cluster. This breaks our
> isolation requirements and add more latency so we don't want to do that.
>
> I have two questions:
>
> - Is there a reason behind the constraint that Kafka Streams can not
> produce to a different cluster? I see that Kafka Streams allow to specify
> different configuration for the producer but it explicitly disallow it for
> ProducerConfig.BOOTSTRAP_SERVERS_CONFIG so it definitely something the
> developers did not want to support (
>
> https://kafka.apache.org/20/javadoc/org/apache/kafka/streams/StreamsConfig.html#getMainConsumerConfigs-java.lang.String-java.lang.String-
> )
> but I am not clear why it is so.
>
> - At the same time, there is the KafkaClientSupplier mechanism that allows
> to inject our own KafkaProducer. I was actually successful in injecting
> such a KafkaProducer that connects to a different cluster. The fact that I
> am able to do, using a not-very documented API, something that other parts
> of the Kafka Streams library try to prevent me to do, makes me wonder if I
> am breaking something while doing this? In particular one thing important
> to me is exactly-once processing so I want to be sure it would still work.
>
> Thanks,
> Cyrille
>