You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by vishnu murali <vi...@gmail.com> on 2020/04/29 18:47:19 UTC
Apache Kafka cluster to cluster
Hi Guys,
I am having two separate Kafka cluster running in two independent zookeeper
I need to send a set of data from one topic from cluster A to cluster B
with the same topic name with all data also..
How can I achieve this
Done anyone have any idea ??
Re: Apache Kafka cluster to cluster
Posted by vishnu murali <vi...@gmail.com>.
Hi Clark
My source cluster is on localhost:9092
My Target cluster was localhost:9091
I am not aware of something like Mirror maker 1 and 2
Because I am using mirror maker which come default from Apache Kafka 2.3.0
On Thu, Apr 30, 2020, 09:43 Liam Clarke-Hutchinson <
liam.clarke@adscale.co.nz> wrote:
> Hi Vishnu,
>
> I can see you're using Mirror Maker 1, unlike Mirror Maker 2, this doesn't
> auto-create topics on the target cluster. Does your target cluster on port
> 9091 have the same topics created on it that exist on the source cluster on
> port 9092?
>
> Also note that a Kafka consumer defaults to "latest" for auto offset reset
> - so when starting MM 1 it won't automatically replicate all existing data,
> only data created after it begins consuming. You can override this by
> setting the consumer property "auto.offset.reset" to "earliest".
>
> Hope that helps,
>
> Liam Clarke-Hutchinson
>
>
> On Thu, Apr 30, 2020 at 3:56 PM vishnu murali <vi...@gmail.com>
> wrote:
>
> > i start the kafka mirror using the below configuration
> >
> > .\bin\windows\kafka-mirror-maker.bat --consumer.config
> > .\config\consumer.properties --producer.config
> .\config\producer.properties
> > --whitelist=".*"
> >
> > *Consumer.properties:*
> > # format: host1:port1,host2:port2 ...
> > bootstrap.servers=localhost:9092
> >
> > # consumer group id
> > group.id=test-consumer-group
> >
> > *Producer.properties:*
> > # format: host1:port1,host2:port2 ...
> > bootstrap.servers=localhost:9091
> >
> > # specify the compression codec for all data generated: none, gzip,
> snappy,
> > lz4, zstd
> > compression.type=none
> >
> >
> > I get this kind of warning statement and it will be still present in
> that
> > statement of more than 30 minutes.No data will be transfered
> >
> > WARNING: The default partition assignment strategy of the mirror maker
> will
> > change from 'range' to 'roundrobin' in an upcoming release (so that
> better
> > load balancing can be achieved). If you prefer to make this switch in
> > advance of that release add the following to the corresponding config:
> >
> >
> 'partition.assignment.strategy=org.apache.kafka.clients.consumer.RoundRobinAssignor'
> >
> > can anyone clarify why i am getting this and what i am doing wrong?
> >
> > On Thu, Apr 30, 2020 at 12:22 AM vishnu murali <
> vishnumurali9762@gmail.com
> > >
> > wrote:
> >
> > > Thanks Blake..ππ
> > >
> > > More over can we use any connector types for this situation?
> > >
> > > Like source is a topic and sink is also an another topic
> > >
> > > Is this possible...
> > >
> > > On Thu, Apr 30, 2020, 00:19 Blake Miller <bl...@gmail.com>
> wrote:
> > >
> > >> Hi Vishnu,
> > >>
> > >> Check out MirrorMaker
> > >>
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=27846330
> > >>
> > >> This can do what you want. Note that the offsets are not copied, nor
> are
> > >> the message timestamps.
> > >>
> > >> HTH
> > >>
> > >>
> > >> On Wed, Apr 29, 2020 at 6:47 PM vishnu murali <
> > vishnumurali9762@gmail.com
> > >> >
> > >> wrote:
> > >>
> > >> > Hi Guys,
> > >> >
> > >> > I am having two separate Kafka cluster running in two independent
> > >> zookeeper
> > >> >
> > >> > I need to send a set of data from one topic from cluster A to
> cluster
> > B
> > >> > with the same topic name with all data also..
> > >> >
> > >> > How can I achieve this
> > >> >
> > >> > Done anyone have any idea ??
> > >> >
> > >>
> > >
> >
>
Re: Apache Kafka cluster to cluster
Posted by Liam Clarke-Hutchinson <li...@adscale.co.nz>.
Hi Vishnu,
I can see you're using Mirror Maker 1, unlike Mirror Maker 2, this doesn't
auto-create topics on the target cluster. Does your target cluster on port
9091 have the same topics created on it that exist on the source cluster on
port 9092?
Also note that a Kafka consumer defaults to "latest" for auto offset reset
- so when starting MM 1 it won't automatically replicate all existing data,
only data created after it begins consuming. You can override this by
setting the consumer property "auto.offset.reset" to "earliest".
Hope that helps,
Liam Clarke-Hutchinson
On Thu, Apr 30, 2020 at 3:56 PM vishnu murali <vi...@gmail.com>
wrote:
> i start the kafka mirror using the below configuration
>
> .\bin\windows\kafka-mirror-maker.bat --consumer.config
> .\config\consumer.properties --producer.config .\config\producer.properties
> --whitelist=".*"
>
> *Consumer.properties:*
> # format: host1:port1,host2:port2 ...
> bootstrap.servers=localhost:9092
>
> # consumer group id
> group.id=test-consumer-group
>
> *Producer.properties:*
> # format: host1:port1,host2:port2 ...
> bootstrap.servers=localhost:9091
>
> # specify the compression codec for all data generated: none, gzip, snappy,
> lz4, zstd
> compression.type=none
>
>
> I get this kind of warning statement and it will be still present in that
> statement of more than 30 minutes.No data will be transfered
>
> WARNING: The default partition assignment strategy of the mirror maker will
> change from 'range' to 'roundrobin' in an upcoming release (so that better
> load balancing can be achieved). If you prefer to make this switch in
> advance of that release add the following to the corresponding config:
>
> 'partition.assignment.strategy=org.apache.kafka.clients.consumer.RoundRobinAssignor'
>
> can anyone clarify why i am getting this and what i am doing wrong?
>
> On Thu, Apr 30, 2020 at 12:22 AM vishnu murali <vishnumurali9762@gmail.com
> >
> wrote:
>
> > Thanks Blake..ππ
> >
> > More over can we use any connector types for this situation?
> >
> > Like source is a topic and sink is also an another topic
> >
> > Is this possible...
> >
> > On Thu, Apr 30, 2020, 00:19 Blake Miller <bl...@gmail.com> wrote:
> >
> >> Hi Vishnu,
> >>
> >> Check out MirrorMaker
> >>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=27846330
> >>
> >> This can do what you want. Note that the offsets are not copied, nor are
> >> the message timestamps.
> >>
> >> HTH
> >>
> >>
> >> On Wed, Apr 29, 2020 at 6:47 PM vishnu murali <
> vishnumurali9762@gmail.com
> >> >
> >> wrote:
> >>
> >> > Hi Guys,
> >> >
> >> > I am having two separate Kafka cluster running in two independent
> >> zookeeper
> >> >
> >> > I need to send a set of data from one topic from cluster A to cluster
> B
> >> > with the same topic name with all data also..
> >> >
> >> > How can I achieve this
> >> >
> >> > Done anyone have any idea ??
> >> >
> >>
> >
>
Re: Apache Kafka cluster to cluster
Posted by vishnu murali <vi...@gmail.com>.
i start the kafka mirror using the below configuration
.\bin\windows\kafka-mirror-maker.bat --consumer.config
.\config\consumer.properties --producer.config .\config\producer.properties
--whitelist=".*"
*Consumer.properties:*
# format: host1:port1,host2:port2 ...
bootstrap.servers=localhost:9092
# consumer group id
group.id=test-consumer-group
*Producer.properties:*
# format: host1:port1,host2:port2 ...
bootstrap.servers=localhost:9091
# specify the compression codec for all data generated: none, gzip, snappy,
lz4, zstd
compression.type=none
I get this kind of warning statement and it will be still present in that
statement of more than 30 minutes.No data will be transfered
WARNING: The default partition assignment strategy of the mirror maker will
change from 'range' to 'roundrobin' in an upcoming release (so that better
load balancing can be achieved). If you prefer to make this switch in
advance of that release add the following to the corresponding config:
'partition.assignment.strategy=org.apache.kafka.clients.consumer.RoundRobinAssignor'
can anyone clarify why i am getting this and what i am doing wrong?
On Thu, Apr 30, 2020 at 12:22 AM vishnu murali <vi...@gmail.com>
wrote:
> Thanks Blake..ππ
>
> More over can we use any connector types for this situation?
>
> Like source is a topic and sink is also an another topic
>
> Is this possible...
>
> On Thu, Apr 30, 2020, 00:19 Blake Miller <bl...@gmail.com> wrote:
>
>> Hi Vishnu,
>>
>> Check out MirrorMaker
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=27846330
>>
>> This can do what you want. Note that the offsets are not copied, nor are
>> the message timestamps.
>>
>> HTH
>>
>>
>> On Wed, Apr 29, 2020 at 6:47 PM vishnu murali <vishnumurali9762@gmail.com
>> >
>> wrote:
>>
>> > Hi Guys,
>> >
>> > I am having two separate Kafka cluster running in two independent
>> zookeeper
>> >
>> > I need to send a set of data from one topic from cluster A to cluster B
>> > with the same topic name with all data also..
>> >
>> > How can I achieve this
>> >
>> > Done anyone have any idea ??
>> >
>>
>
Re: Apache Kafka cluster to cluster
Posted by vishnu murali <vi...@gmail.com>.
Thanks Blake..ππ
More over can we use any connector types for this situation?
Like source is a topic and sink is also an another topic
Is this possible...
On Thu, Apr 30, 2020, 00:19 Blake Miller <bl...@gmail.com> wrote:
> Hi Vishnu,
>
> Check out MirrorMaker
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=27846330
>
> This can do what you want. Note that the offsets are not copied, nor are
> the message timestamps.
>
> HTH
>
>
> On Wed, Apr 29, 2020 at 6:47 PM vishnu murali <vi...@gmail.com>
> wrote:
>
> > Hi Guys,
> >
> > I am having two separate Kafka cluster running in two independent
> zookeeper
> >
> > I need to send a set of data from one topic from cluster A to cluster B
> > with the same topic name with all data also..
> >
> > How can I achieve this
> >
> > Done anyone have any idea ??
> >
>
Re: Apache Kafka cluster to cluster
Posted by Ma...@cognizant.com.
Is there documentation or example for mirror maker 2.0 ?
ο»ΏOn 4/29/20, 9:04 PM, "Liam Clarke-Hutchinson" <li...@adscale.co.nz> wrote:
[External]
Hi Blake,
Replicator is, AFAIK, not FOSS - however, Mirror Maker 2.0, which is built
along very similar lines (i.e., on top of Kafka Connect) is, as is Mirror
Maker 1.0.
On Thu, Apr 30, 2020 at 6:51 AM Blake Miller <bl...@gmail.com> wrote:
> Oh, and it looks like Confluent has released a newer replacement for
> MirrorMaker called Replicator
>
>
> https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.confluent.io%2Fcurrent%2Fmulti-dc-deployments%2Freplicator%2Fmigrate-replicator.html&data=02%7C01%7CManoj.Agrawal2%40cognizant.com%7Cdd5c2b32b1aa4567421e08d7ecbb9def%7Cde08c40719b9427d9fe8edf254300ca7%7C0%7C0%7C637238162871927627&sdata=f6zWwsX%2FEjBxwpiT%2FuaOBB5RSi6HnFcN13S9iukAIqA%3D&reserved=0
>
>
>
> On Wed, Apr 29, 2020 at 6:49 PM Blake Miller <bl...@gmail.com>
> wrote:
>
> > Hi Vishnu,
> >
> > Check out MirrorMaker
> >
> https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fpages%2Fviewpage.action%3FpageId%3D27846330&data=02%7C01%7CManoj.Agrawal2%40cognizant.com%7Cdd5c2b32b1aa4567421e08d7ecbb9def%7Cde08c40719b9427d9fe8edf254300ca7%7C0%7C0%7C637238162871927627&sdata=ey5jIPUG%2BVgEGCshvlMlXdmU%2B54E787oWiF6AZ6jj%2FU%3D&reserved=0
> >
> > This can do what you want. Note that the offsets are not copied, nor are
> > the message timestamps.
> >
> > HTH
> >
> >
> > On Wed, Apr 29, 2020 at 6:47 PM vishnu murali <
> vishnumurali9762@gmail.com>
> > wrote:
> >
> >> Hi Guys,
> >>
> >> I am having two separate Kafka cluster running in two independent
> >> zookeeper
> >>
> >> I need to send a set of data from one topic from cluster A to cluster B
> >> with the same topic name with all data also..
> >>
> >> How can I achieve this
> >>
> >> Done anyone have any idea ??
> >>
> >
>
This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient(s), please reply to the sender and destroy all copies of the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email, and/or any action taken in reliance on the contents of this e-mail is strictly prohibited and may be unlawful. Where permitted by applicable law, this e-mail and other e-mail communications sent to and from Cognizant e-mail addresses may be monitored.
Re: Apache Kafka cluster to cluster
Posted by Liam Clarke-Hutchinson <li...@adscale.co.nz>.
Hi Blake,
Replicator is, AFAIK, not FOSS - however, Mirror Maker 2.0, which is built
along very similar lines (i.e., on top of Kafka Connect) is, as is Mirror
Maker 1.0.
On Thu, Apr 30, 2020 at 6:51 AM Blake Miller <bl...@gmail.com> wrote:
> Oh, and it looks like Confluent has released a newer replacement for
> MirrorMaker called Replicator
>
>
> https://docs.confluent.io/current/multi-dc-deployments/replicator/migrate-replicator.html
>
>
>
> On Wed, Apr 29, 2020 at 6:49 PM Blake Miller <bl...@gmail.com>
> wrote:
>
> > Hi Vishnu,
> >
> > Check out MirrorMaker
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=27846330
> >
> > This can do what you want. Note that the offsets are not copied, nor are
> > the message timestamps.
> >
> > HTH
> >
> >
> > On Wed, Apr 29, 2020 at 6:47 PM vishnu murali <
> vishnumurali9762@gmail.com>
> > wrote:
> >
> >> Hi Guys,
> >>
> >> I am having two separate Kafka cluster running in two independent
> >> zookeeper
> >>
> >> I need to send a set of data from one topic from cluster A to cluster B
> >> with the same topic name with all data also..
> >>
> >> How can I achieve this
> >>
> >> Done anyone have any idea ??
> >>
> >
>
Re: Apache Kafka cluster to cluster
Posted by Blake Miller <bl...@gmail.com>.
Oh, and it looks like Confluent has released a newer replacement for
MirrorMaker called Replicator
https://docs.confluent.io/current/multi-dc-deployments/replicator/migrate-replicator.html
On Wed, Apr 29, 2020 at 6:49 PM Blake Miller <bl...@gmail.com> wrote:
> Hi Vishnu,
>
> Check out MirrorMaker
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=27846330
>
> This can do what you want. Note that the offsets are not copied, nor are
> the message timestamps.
>
> HTH
>
>
> On Wed, Apr 29, 2020 at 6:47 PM vishnu murali <vi...@gmail.com>
> wrote:
>
>> Hi Guys,
>>
>> I am having two separate Kafka cluster running in two independent
>> zookeeper
>>
>> I need to send a set of data from one topic from cluster A to cluster B
>> with the same topic name with all data also..
>>
>> How can I achieve this
>>
>> Done anyone have any idea ??
>>
>
Re: Apache Kafka cluster to cluster
Posted by Blake Miller <bl...@gmail.com>.
Hi Vishnu,
Check out MirrorMaker
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=27846330
This can do what you want. Note that the offsets are not copied, nor are
the message timestamps.
HTH
On Wed, Apr 29, 2020 at 6:47 PM vishnu murali <vi...@gmail.com>
wrote:
> Hi Guys,
>
> I am having two separate Kafka cluster running in two independent zookeeper
>
> I need to send a set of data from one topic from cluster A to cluster B
> with the same topic name with all data also..
>
> How can I achieve this
>
> Done anyone have any idea ??
>