You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Miguel Ángel Fernández Fernández <mi...@gmail.com> on 2023/03/20 21:34:32 UTC
MirrorMaker 2.0 question
Hello,
I'm doing some tests with MirrorMaker 2 but I'm stuck. I have a couple of
kafka clusters, I think everything is set up correctly. However, when I run
/bin/connect-mirror-maker /var/lib/kafka/data/mm2.properties
the result I get is the creation of the topics
mm2-configs.A.internal,
mm2-offsets.A.internal,
mm2-status.A.internal
but not of the specific topics found in cluster A (topic1).
The mm2.properties configuration is basically a copy of
https://github.com/apache/kafka/blob/trunk/config/connect-mirror-maker.properties
modifying bootstrap servers
A.bootstrap.servers = 127.0.0.1:29092
B.bootstrap.servers = 127.0.0.1:29093
Can anyone advise me with this problem? Is it possible that some
configuration in mm2.properties is missing? Are there any real requirements
regarding the configuration of the kafka clusters that may be overlooked?
Thank you for your help,
Miguel Ángel
Re: MirrorMaker 2.0 question
Posted by Miguel Ángel Fernández Fernández <mi...@gmail.com>.
I got it to work by raising a kafka connect from which to launch the mm2.
Silly question.
El lun, 20 mar 2023 a las 23:09, Miguel Ángel Fernández Fernández (<
miguelangelprogramacion@gmail.com>) escribió:
> I have two clusters up on the same machine with docker-compose
>
> services:
> zookeeper-lab:
> image: "bitnami/zookeeper:3.8.1"
> restart: always
> environment:
> ZOO_PORT_NUMBER: 2183
> ALLOW_ANONYMOUS_LOGIN: "yes"
> ports:
> - "2183:2183"
> - "2886:2888"
> - "3886:3888"
> kafka-confluent:
> image: "confluentinc/cp-kafka:6.2.0"
> restart: always
> environment:
> - KAFKA_ZOOKEEPER_CONNECT=localhost:2183
> - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:29092
> network_mode: "host"
> volumes:
> -
> "/home/miguelangel/Documents/KafkaLab/mm2.properties:/var/lib/kafka/data/mm2.properties"
>
> and mm2 is running on one of them.
>
> Regarding the log, I don't see anything that seems suspicious to me. Not
> ERROR logs.
>
> [2023-03-20 22:03:14,290] WARN The configuration '
> metrics.context.connect.group.id' was supplied but isn't a known config.
> (org.apache.kafka.clients.producer.ProducerConfig:380)
> [2023-03-20 22:03:14,290] INFO Created topic (name=mm2-configs.B.internal,
> numPartitions=1, replicationFactor=1, replicasAssignments=null,
> configs={cleanup.policy=compact}) on brokers at 127.0.0.1:29092
> (org.apache.kafka.connect.util.TopicAdmin:389)
> [2023-03-20 22:03:14,290] WARN The configuration 'group.id' was supplied
> but isn't a known config.
> (org.apache.kafka.clients.producer.ProducerConfig:380)
> [2023-03-20 22:03:14,290] WARN The configuration 'status.storage.topic'
> was supplied but isn't a known config.
> (org.apache.kafka.clients.producer.ProducerConfig:380)
>
> Cheers,
>
> Miguel Ángel
>
>
> El lun, 20 mar 2023 a las 22:44, Chris Egerton (<ch...@aiven.io.invalid>)
> escribió:
>
>> Mi Miguel,
>>
>> How many nodes are you running MM2 with? Just one?
>>
>> Separately, do you notice anything at ERROR level in the logs?
>>
>> Cheers,
>>
>> Chris
>>
>> On Mon, Mar 20, 2023 at 5:35 PM Miguel Ángel Fernández Fernández <
>> miguelangelprogramacion@gmail.com> wrote:
>>
>> > Hello,
>> >
>> > I'm doing some tests with MirrorMaker 2 but I'm stuck. I have a couple
>> of
>> > kafka clusters, I think everything is set up correctly. However, when I
>> run
>> >
>> > /bin/connect-mirror-maker /var/lib/kafka/data/mm2.properties
>> >
>> > the result I get is the creation of the topics
>> >
>> > mm2-configs.A.internal,
>> > mm2-offsets.A.internal,
>> > mm2-status.A.internal
>> >
>> > but not of the specific topics found in cluster A (topic1).
>> >
>> > The mm2.properties configuration is basically a copy of
>> >
>> >
>> >
>> https://github.com/apache/kafka/blob/trunk/config/connect-mirror-maker.properties
>> >
>> > modifying bootstrap servers
>> >
>> > A.bootstrap.servers = 127.0.0.1:29092
>> > B.bootstrap.servers = 127.0.0.1:29093
>> >
>> > Can anyone advise me with this problem? Is it possible that some
>> > configuration in mm2.properties is missing? Are there any real
>> requirements
>> > regarding the configuration of the kafka clusters that may be
>> overlooked?
>> >
>> > Thank you for your help,
>> > Miguel Ángel
>> >
>>
>
Re: MirrorMaker 2.0 question
Posted by Miguel Ángel Fernández Fernández <mi...@gmail.com>.
I have two clusters up on the same machine with docker-compose
services:
zookeeper-lab:
image: "bitnami/zookeeper:3.8.1"
restart: always
environment:
ZOO_PORT_NUMBER: 2183
ALLOW_ANONYMOUS_LOGIN: "yes"
ports:
- "2183:2183"
- "2886:2888"
- "3886:3888"
kafka-confluent:
image: "confluentinc/cp-kafka:6.2.0"
restart: always
environment:
- KAFKA_ZOOKEEPER_CONNECT=localhost:2183
- KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:29092
network_mode: "host"
volumes:
-
"/home/miguelangel/Documents/KafkaLab/mm2.properties:/var/lib/kafka/data/mm2.properties"
and mm2 is running on one of them.
Regarding the log, I don't see anything that seems suspicious to me. Not
ERROR logs.
[2023-03-20 22:03:14,290] WARN The configuration '
metrics.context.connect.group.id' was supplied but isn't a known config.
(org.apache.kafka.clients.producer.ProducerConfig:380)
[2023-03-20 22:03:14,290] INFO Created topic (name=mm2-configs.B.internal,
numPartitions=1, replicationFactor=1, replicasAssignments=null,
configs={cleanup.policy=compact}) on brokers at 127.0.0.1:29092
(org.apache.kafka.connect.util.TopicAdmin:389)
[2023-03-20 22:03:14,290] WARN The configuration 'group.id' was supplied
but isn't a known config.
(org.apache.kafka.clients.producer.ProducerConfig:380)
[2023-03-20 22:03:14,290] WARN The configuration 'status.storage.topic' was
supplied but isn't a known config.
(org.apache.kafka.clients.producer.ProducerConfig:380)
Cheers,
Miguel Ángel
El lun, 20 mar 2023 a las 22:44, Chris Egerton (<ch...@aiven.io.invalid>)
escribió:
> Mi Miguel,
>
> How many nodes are you running MM2 with? Just one?
>
> Separately, do you notice anything at ERROR level in the logs?
>
> Cheers,
>
> Chris
>
> On Mon, Mar 20, 2023 at 5:35 PM Miguel Ángel Fernández Fernández <
> miguelangelprogramacion@gmail.com> wrote:
>
> > Hello,
> >
> > I'm doing some tests with MirrorMaker 2 but I'm stuck. I have a couple of
> > kafka clusters, I think everything is set up correctly. However, when I
> run
> >
> > /bin/connect-mirror-maker /var/lib/kafka/data/mm2.properties
> >
> > the result I get is the creation of the topics
> >
> > mm2-configs.A.internal,
> > mm2-offsets.A.internal,
> > mm2-status.A.internal
> >
> > but not of the specific topics found in cluster A (topic1).
> >
> > The mm2.properties configuration is basically a copy of
> >
> >
> >
> https://github.com/apache/kafka/blob/trunk/config/connect-mirror-maker.properties
> >
> > modifying bootstrap servers
> >
> > A.bootstrap.servers = 127.0.0.1:29092
> > B.bootstrap.servers = 127.0.0.1:29093
> >
> > Can anyone advise me with this problem? Is it possible that some
> > configuration in mm2.properties is missing? Are there any real
> requirements
> > regarding the configuration of the kafka clusters that may be overlooked?
> >
> > Thank you for your help,
> > Miguel Ángel
> >
>
Re: MirrorMaker 2.0 question
Posted by Chris Egerton <ch...@aiven.io.INVALID>.
Mi Miguel,
How many nodes are you running MM2 with? Just one?
Separately, do you notice anything at ERROR level in the logs?
Cheers,
Chris
On Mon, Mar 20, 2023 at 5:35 PM Miguel Ángel Fernández Fernández <
miguelangelprogramacion@gmail.com> wrote:
> Hello,
>
> I'm doing some tests with MirrorMaker 2 but I'm stuck. I have a couple of
> kafka clusters, I think everything is set up correctly. However, when I run
>
> /bin/connect-mirror-maker /var/lib/kafka/data/mm2.properties
>
> the result I get is the creation of the topics
>
> mm2-configs.A.internal,
> mm2-offsets.A.internal,
> mm2-status.A.internal
>
> but not of the specific topics found in cluster A (topic1).
>
> The mm2.properties configuration is basically a copy of
>
>
> https://github.com/apache/kafka/blob/trunk/config/connect-mirror-maker.properties
>
> modifying bootstrap servers
>
> A.bootstrap.servers = 127.0.0.1:29092
> B.bootstrap.servers = 127.0.0.1:29093
>
> Can anyone advise me with this problem? Is it possible that some
> configuration in mm2.properties is missing? Are there any real requirements
> regarding the configuration of the kafka clusters that may be overlooked?
>
> Thank you for your help,
> Miguel Ángel
>