You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Eric Beabes <ma...@gmail.com> on 2020/12/01 18:58:55 UTC
Kafka Streams: Read from one Bootstrap server & write to other
I need to read from a topic in one bootstrap server & write it to another
topic in another bootstrap server. Since there's only one
StreamsConfig.BOOTSTRAP_SERVERS_CONFIG property, I am wondering how to
accomplish this?
Do I need to create 2 different KafkaStreams objects? One for reading & the
other for writing?
Re: Kafka Streams: Read from one Bootstrap server & write to other
Posted by "Matthias J. Sax" <mj...@apache.org>.
Eric,
this idea came up multiple time already, but it is much more difficult
to build than it seems, especially for supporting "exactly once". Kafka
transaction only work on a single cluster...
Maybe we will support it at some point, but the "workaround" to either
replicate the input or output topic(s) is fairly simple and thus it
seems not to be a "killer feature" but more like a "nice to have" (just
my personal opinion).
-Matthias
On 12/1/20 2:08 PM, Malcolm McFarland wrote:
> How about a lightweight MirrorMaker instance for just this topic?
>
> Malcolm McFarland
> Cavulus
>
>
> This correspondence is from HealthPlanCRM, LLC, d/b/a Cavulus. Any
> unauthorized or improper disclosure, copying, distribution, or use of the
> contents of this message is prohibited. The information contained in this
> message is intended only for the personal and confidential use of the
> recipient(s) named above. If you have received this message in error,
> please notify the sender immediately and delete the original message.
>
>
> On Tue, Dec 1, 2020 at 1:18 PM Eric Beabes <ma...@gmail.com> wrote:
>
>> Hmm.. Ok. Maybe this could be a feature request for a future release. This
>> can be accomplished easily in Spark Structured Streaming... just saying :)
>>
>> In Spark Structured Streaming we can have separate configurations for
>> 'readStream' & 'writeStream'. I am a bit surprised this is not available in
>> Kafka Streams.
>>
>> Even during development sometimes we want to read from a topic in a QA
>> cluster & write it to a topic on the Local cluster.
>>
>>
>>
>> On Tue, Dec 1, 2020 at 11:33 AM Matthias J. Sax <mj...@apache.org> wrote:
>>
>>> KafkaStreams can only connect to a single cluster.
>>>
>>> If you really need to read from one cluster and write to another, you
>>> have 3 main options:
>>>
>>> - use KafkaStreams on the source cluster and mirror the output topic
>>> from the source to the target cluster
>>>
>>> - mirror the input topic from the source cluster to the target cluster
>>> and use KafkaStreams on the target cluster
>>>
>>> - don't use KafkaStreams but plain consumer/producer
>>>
>>>
>>>
>>> -Matthias
>>>
>>> On 12/1/20 10:58 AM, Eric Beabes wrote:
>>>> I need to read from a topic in one bootstrap server & write it to
>> another
>>>> topic in another bootstrap server. Since there's only one
>>>> StreamsConfig.BOOTSTRAP_SERVERS_CONFIG property, I am wondering how to
>>>> accomplish this?
>>>>
>>>> Do I need to create 2 different KafkaStreams objects? One for reading &
>>> the
>>>> other for writing?
>>>>
>>>
>>
>
Re: Kafka Streams: Read from one Bootstrap server & write to other
Posted by Malcolm McFarland <mm...@cavulus.com>.
How about a lightweight MirrorMaker instance for just this topic?
Malcolm McFarland
Cavulus
This correspondence is from HealthPlanCRM, LLC, d/b/a Cavulus. Any
unauthorized or improper disclosure, copying, distribution, or use of the
contents of this message is prohibited. The information contained in this
message is intended only for the personal and confidential use of the
recipient(s) named above. If you have received this message in error,
please notify the sender immediately and delete the original message.
On Tue, Dec 1, 2020 at 1:18 PM Eric Beabes <ma...@gmail.com> wrote:
> Hmm.. Ok. Maybe this could be a feature request for a future release. This
> can be accomplished easily in Spark Structured Streaming... just saying :)
>
> In Spark Structured Streaming we can have separate configurations for
> 'readStream' & 'writeStream'. I am a bit surprised this is not available in
> Kafka Streams.
>
> Even during development sometimes we want to read from a topic in a QA
> cluster & write it to a topic on the Local cluster.
>
>
>
> On Tue, Dec 1, 2020 at 11:33 AM Matthias J. Sax <mj...@apache.org> wrote:
>
> > KafkaStreams can only connect to a single cluster.
> >
> > If you really need to read from one cluster and write to another, you
> > have 3 main options:
> >
> > - use KafkaStreams on the source cluster and mirror the output topic
> > from the source to the target cluster
> >
> > - mirror the input topic from the source cluster to the target cluster
> > and use KafkaStreams on the target cluster
> >
> > - don't use KafkaStreams but plain consumer/producer
> >
> >
> >
> > -Matthias
> >
> > On 12/1/20 10:58 AM, Eric Beabes wrote:
> > > I need to read from a topic in one bootstrap server & write it to
> another
> > > topic in another bootstrap server. Since there's only one
> > > StreamsConfig.BOOTSTRAP_SERVERS_CONFIG property, I am wondering how to
> > > accomplish this?
> > >
> > > Do I need to create 2 different KafkaStreams objects? One for reading &
> > the
> > > other for writing?
> > >
> >
>
Re: Kafka Streams: Read from one Bootstrap server & write to other
Posted by Eric Beabes <ma...@gmail.com>.
Hmm.. Ok. Maybe this could be a feature request for a future release. This
can be accomplished easily in Spark Structured Streaming... just saying :)
In Spark Structured Streaming we can have separate configurations for
'readStream' & 'writeStream'. I am a bit surprised this is not available in
Kafka Streams.
Even during development sometimes we want to read from a topic in a QA
cluster & write it to a topic on the Local cluster.
On Tue, Dec 1, 2020 at 11:33 AM Matthias J. Sax <mj...@apache.org> wrote:
> KafkaStreams can only connect to a single cluster.
>
> If you really need to read from one cluster and write to another, you
> have 3 main options:
>
> - use KafkaStreams on the source cluster and mirror the output topic
> from the source to the target cluster
>
> - mirror the input topic from the source cluster to the target cluster
> and use KafkaStreams on the target cluster
>
> - don't use KafkaStreams but plain consumer/producer
>
>
>
> -Matthias
>
> On 12/1/20 10:58 AM, Eric Beabes wrote:
> > I need to read from a topic in one bootstrap server & write it to another
> > topic in another bootstrap server. Since there's only one
> > StreamsConfig.BOOTSTRAP_SERVERS_CONFIG property, I am wondering how to
> > accomplish this?
> >
> > Do I need to create 2 different KafkaStreams objects? One for reading &
> the
> > other for writing?
> >
>
Re: Kafka Streams: Read from one Bootstrap server & write to other
Posted by "Matthias J. Sax" <mj...@apache.org>.
KafkaStreams can only connect to a single cluster.
If you really need to read from one cluster and write to another, you
have 3 main options:
- use KafkaStreams on the source cluster and mirror the output topic
from the source to the target cluster
- mirror the input topic from the source cluster to the target cluster
and use KafkaStreams on the target cluster
- don't use KafkaStreams but plain consumer/producer
-Matthias
On 12/1/20 10:58 AM, Eric Beabes wrote:
> I need to read from a topic in one bootstrap server & write it to another
> topic in another bootstrap server. Since there's only one
> StreamsConfig.BOOTSTRAP_SERVERS_CONFIG property, I am wondering how to
> accomplish this?
>
> Do I need to create 2 different KafkaStreams objects? One for reading & the
> other for writing?
>