Moving data from one cluster to another with Kafka Streams

[Geoffrey Holmes <gh...@pinsightmedia.com>]

I need to get messages from a topic in one Kafka cluster, transform the message payload, and put the messages into topics in another Kafka cluster. Is it possible to do this with Kafka Streams? I don’t see how I can configure the stream to use one cluster for the input and another cluster for output.

Re: Moving data from one cluster to another with Kafka Streams

["Matthias J. Sax" <ma...@confluent.io>]

Kafka Streams only work with a single cluster.

Thus, you would need to either transform the data first and replicate
the output topic to the target cluster, or replicate first and transform
within the target cluster.

Note, for the "intermediate" topic you need, you can set a low retention
time to reduce storage footprint, as it only acts as a temporal topic
and the actual data is safely stored in the original source and final
target topic.

As an alternative, you might want to check out
"single-message-transforms" (SMT) using Kafka Connect. Those allow you
to do simple transformation on the fly while copying data around. If you
don't need advanced transformations like aggregations or joins, SMT
might be sufficient and you don't need to use Kafka Streams.


-Matthias

On 2/2/18 2:54 PM, Geoffrey Holmes wrote:
> I need to get messages from a topic in one Kafka cluster, transform the message payload, and put the messages into topics in another Kafka cluster. Is it possible to do this with Kafka Streams? I don’t see how I can configure the stream to use one cluster for the input and another cluster for output.
> 
>