You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Miguel Ángel Fernández Fernández <mi...@gmail.com> on 2023/03/20 21:34:32 UTC

MirrorMaker 2.0 question

Hello,

I'm doing some tests with MirrorMaker 2 but I'm stuck. I have a couple of
kafka clusters, I think everything is set up correctly. However, when I run

/bin/connect-mirror-maker /var/lib/kafka/data/mm2.properties

the result I get is the creation of the topics

mm2-configs.A.internal,
mm2-offsets.A.internal,
mm2-status.A.internal

but not of the specific topics found in cluster A (topic1).

The mm2.properties configuration is basically a copy of

https://github.com/apache/kafka/blob/trunk/config/connect-mirror-maker.properties

modifying bootstrap servers

A.bootstrap.servers = 127.0.0.1:29092
B.bootstrap.servers = 127.0.0.1:29093

Can anyone advise me with this problem? Is it possible that some
configuration in mm2.properties is missing? Are there any real requirements
regarding the configuration of the kafka clusters that may be overlooked?

Thank you for your help,
Miguel Ángel

Re: MirrorMaker 2.0 question

Posted by Miguel Ángel Fernández Fernández <mi...@gmail.com>.
I got it to work by raising a kafka connect from which to launch the mm2.
Silly question.

El lun, 20 mar 2023 a las 23:09, Miguel Ángel Fernández Fernández (<
miguelangelprogramacion@gmail.com>) escribió:

> I have two clusters up on the same machine with docker-compose
>
> services:
>   zookeeper-lab:
>     image: "bitnami/zookeeper:3.8.1"
>     restart: always
>     environment:
>       ZOO_PORT_NUMBER: 2183
>       ALLOW_ANONYMOUS_LOGIN: "yes"
>     ports:
>     - "2183:2183"
>     - "2886:2888"
>     - "3886:3888"
>   kafka-confluent:
>     image: "confluentinc/cp-kafka:6.2.0"
>     restart: always
>     environment:
>     - KAFKA_ZOOKEEPER_CONNECT=localhost:2183
>     - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:29092
>     network_mode: "host"
>     volumes:
>       -
> "/home/miguelangel/Documents/KafkaLab/mm2.properties:/var/lib/kafka/data/mm2.properties"
>
> and mm2 is running on one of them.
>
> Regarding the log, I don't see anything that seems suspicious to me. Not
> ERROR logs.
>
> [2023-03-20 22:03:14,290] WARN The configuration '
> metrics.context.connect.group.id' was supplied but isn't a known config.
> (org.apache.kafka.clients.producer.ProducerConfig:380)
> [2023-03-20 22:03:14,290] INFO Created topic (name=mm2-configs.B.internal,
> numPartitions=1, replicationFactor=1, replicasAssignments=null,
> configs={cleanup.policy=compact}) on brokers at 127.0.0.1:29092
> (org.apache.kafka.connect.util.TopicAdmin:389)
> [2023-03-20 22:03:14,290] WARN The configuration 'group.id' was supplied
> but isn't a known config.
> (org.apache.kafka.clients.producer.ProducerConfig:380)
> [2023-03-20 22:03:14,290] WARN The configuration 'status.storage.topic'
> was supplied but isn't a known config.
> (org.apache.kafka.clients.producer.ProducerConfig:380)
>
> Cheers,
>
> Miguel Ángel
>
>
> El lun, 20 mar 2023 a las 22:44, Chris Egerton (<ch...@aiven.io.invalid>)
> escribió:
>
>> Mi Miguel,
>>
>> How many nodes are you running MM2 with? Just one?
>>
>> Separately, do you notice anything at ERROR level in the logs?
>>
>> Cheers,
>>
>> Chris
>>
>> On Mon, Mar 20, 2023 at 5:35 PM Miguel Ángel Fernández Fernández <
>> miguelangelprogramacion@gmail.com> wrote:
>>
>> > Hello,
>> >
>> > I'm doing some tests with MirrorMaker 2 but I'm stuck. I have a couple
>> of
>> > kafka clusters, I think everything is set up correctly. However, when I
>> run
>> >
>> > /bin/connect-mirror-maker /var/lib/kafka/data/mm2.properties
>> >
>> > the result I get is the creation of the topics
>> >
>> > mm2-configs.A.internal,
>> > mm2-offsets.A.internal,
>> > mm2-status.A.internal
>> >
>> > but not of the specific topics found in cluster A (topic1).
>> >
>> > The mm2.properties configuration is basically a copy of
>> >
>> >
>> >
>> https://github.com/apache/kafka/blob/trunk/config/connect-mirror-maker.properties
>> >
>> > modifying bootstrap servers
>> >
>> > A.bootstrap.servers = 127.0.0.1:29092
>> > B.bootstrap.servers = 127.0.0.1:29093
>> >
>> > Can anyone advise me with this problem? Is it possible that some
>> > configuration in mm2.properties is missing? Are there any real
>> requirements
>> > regarding the configuration of the kafka clusters that may be
>> overlooked?
>> >
>> > Thank you for your help,
>> > Miguel Ángel
>> >
>>
>

Re: MirrorMaker 2.0 question

Posted by Miguel Ángel Fernández Fernández <mi...@gmail.com>.
I have two clusters up on the same machine with docker-compose

services:
  zookeeper-lab:
    image: "bitnami/zookeeper:3.8.1"
    restart: always
    environment:
      ZOO_PORT_NUMBER: 2183
      ALLOW_ANONYMOUS_LOGIN: "yes"
    ports:
    - "2183:2183"
    - "2886:2888"
    - "3886:3888"
  kafka-confluent:
    image: "confluentinc/cp-kafka:6.2.0"
    restart: always
    environment:
    - KAFKA_ZOOKEEPER_CONNECT=localhost:2183
    - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:29092
    network_mode: "host"
    volumes:
      -
"/home/miguelangel/Documents/KafkaLab/mm2.properties:/var/lib/kafka/data/mm2.properties"

and mm2 is running on one of them.

Regarding the log, I don't see anything that seems suspicious to me. Not
ERROR logs.

[2023-03-20 22:03:14,290] WARN The configuration '
metrics.context.connect.group.id' was supplied but isn't a known config.
(org.apache.kafka.clients.producer.ProducerConfig:380)
[2023-03-20 22:03:14,290] INFO Created topic (name=mm2-configs.B.internal,
numPartitions=1, replicationFactor=1, replicasAssignments=null,
configs={cleanup.policy=compact}) on brokers at 127.0.0.1:29092
(org.apache.kafka.connect.util.TopicAdmin:389)
[2023-03-20 22:03:14,290] WARN The configuration 'group.id' was supplied
but isn't a known config.
(org.apache.kafka.clients.producer.ProducerConfig:380)
[2023-03-20 22:03:14,290] WARN The configuration 'status.storage.topic' was
supplied but isn't a known config.
(org.apache.kafka.clients.producer.ProducerConfig:380)

Cheers,

Miguel Ángel


El lun, 20 mar 2023 a las 22:44, Chris Egerton (<ch...@aiven.io.invalid>)
escribió:

> Mi Miguel,
>
> How many nodes are you running MM2 with? Just one?
>
> Separately, do you notice anything at ERROR level in the logs?
>
> Cheers,
>
> Chris
>
> On Mon, Mar 20, 2023 at 5:35 PM Miguel Ángel Fernández Fernández <
> miguelangelprogramacion@gmail.com> wrote:
>
> > Hello,
> >
> > I'm doing some tests with MirrorMaker 2 but I'm stuck. I have a couple of
> > kafka clusters, I think everything is set up correctly. However, when I
> run
> >
> > /bin/connect-mirror-maker /var/lib/kafka/data/mm2.properties
> >
> > the result I get is the creation of the topics
> >
> > mm2-configs.A.internal,
> > mm2-offsets.A.internal,
> > mm2-status.A.internal
> >
> > but not of the specific topics found in cluster A (topic1).
> >
> > The mm2.properties configuration is basically a copy of
> >
> >
> >
> https://github.com/apache/kafka/blob/trunk/config/connect-mirror-maker.properties
> >
> > modifying bootstrap servers
> >
> > A.bootstrap.servers = 127.0.0.1:29092
> > B.bootstrap.servers = 127.0.0.1:29093
> >
> > Can anyone advise me with this problem? Is it possible that some
> > configuration in mm2.properties is missing? Are there any real
> requirements
> > regarding the configuration of the kafka clusters that may be overlooked?
> >
> > Thank you for your help,
> > Miguel Ángel
> >
>

Re: MirrorMaker 2.0 question

Posted by Chris Egerton <ch...@aiven.io.INVALID>.
Mi Miguel,

How many nodes are you running MM2 with? Just one?

Separately, do you notice anything at ERROR level in the logs?

Cheers,

Chris

On Mon, Mar 20, 2023 at 5:35 PM Miguel Ángel Fernández Fernández <
miguelangelprogramacion@gmail.com> wrote:

> Hello,
>
> I'm doing some tests with MirrorMaker 2 but I'm stuck. I have a couple of
> kafka clusters, I think everything is set up correctly. However, when I run
>
> /bin/connect-mirror-maker /var/lib/kafka/data/mm2.properties
>
> the result I get is the creation of the topics
>
> mm2-configs.A.internal,
> mm2-offsets.A.internal,
> mm2-status.A.internal
>
> but not of the specific topics found in cluster A (topic1).
>
> The mm2.properties configuration is basically a copy of
>
>
> https://github.com/apache/kafka/blob/trunk/config/connect-mirror-maker.properties
>
> modifying bootstrap servers
>
> A.bootstrap.servers = 127.0.0.1:29092
> B.bootstrap.servers = 127.0.0.1:29093
>
> Can anyone advise me with this problem? Is it possible that some
> configuration in mm2.properties is missing? Are there any real requirements
> regarding the configuration of the kafka clusters that may be overlooked?
>
> Thank you for your help,
> Miguel Ángel
>