You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Sebastian Schmitz <se...@propellerhead.co.nz> on 2019/12/23 18:57:39 UTC
Mirrormaker 2.0
Hello,
I'm currently trying to implement the new Kafka 2.4.0 and the new MM2.
However, it looks like the only documentation available is the KIP-382,
and the documentation
(https://kafka.apache.org/documentation/#basic_ops_mirror_maker) for the
MM isn't yet updated, and the documentation in the KIP seems to be
missing some stuff as I get a lot of errors and warning when starting
the MM2 as connect-mirror, and it doesn't mirror, so I probably have
some mistakes in my configuration, but can't confirm this as it's the
same as in the KIP.
Any plans when the documentation will be updated?
Thanks
Sebastian
--
DISCLAIMER
This email contains information that is confidential and which
may be
legally privileged. If you have received this email in error please
notify the sender immediately and delete the email.
This email is intended
solely for the use of the intended recipient and you may not use or
disclose this email in any way.
Re: Mirrormaker 2.0
Posted by Ryanne Dolan <ry...@gmail.com>.
Sebastian, there are multiple ways to run MM2. One way is to start the
individual Connectors (MirrorSourceConnector, MirrorCheckpointConnector,
and MirrorHeartbeatConnector) on an existing Connect cluster, if you have
one. Some of the configuration properties you've listed, e.g. "name" and
"connector.class" are only relevant when configuring individual Connectors
in this way.
The ./bin/connect-mirror.sh script, on the other hand, has its own
high-level configuration file. This is where properties like "a->b.topics"
come into play. The high-level configuration file is used to generate the
low-level configurations for each internal Connector.
Generally you'd use the high-level approach, unless you already have a
bunch of Connect clusters you want to leverage. Keep this distinction in
mind when looking at example configurations, and hopefully things will be
clearer.
Ryanne
On Mon, Dec 23, 2019 at 1:34 PM Sebastian Schmitz <
sebastian.schmitz@propellerhead.co.nz> wrote:
> Hello,
>
> I tried running this connect-mirror-config:
>
> <snip>
> name = $MIRROR_NAME
> clusters = source, target
> source.bootstrap.servers = $SOURCE_SERVERS
> target.bootstrap.servers = $TARGET_SERVERS
> source->target.topics = $SOURCE_TARGET_TOPICS
> target->source.topics = $TARGET_SOURCE_TOPICS
> source->target.emit.heartbeats.enabled = true
> target->source.emit.heartbeats.enabled = true
> connector.class = org.apache.kafka.connect.mirror.MirrorSourceConnector
>
> # disable some new features
> refresh.topics.enabled = false
> refresh.groups.enabled = false
> emit.checkpoints.enables = true
> emit.heartbeats.enabled = true
> sync.topic.configs.enabled = false
> sync.topic.acls.enabled = false
> </snip>
>
> SOURCE_SERVERS and TARGET_SERVERS are a comma-separated list of three
> brokers with ports.
> The TOPICS are |-separated lists of topics.
>
> I get these warning during startup which is a bit weird as I never
> supplied any of those settings, but maybe I should?
>
> [2019-12-23 00:36:25,918] WARN The configuration 'config.storage.topic'
> was supplied but isn't a known config.
> (org.apache.kafka.clients.producer.ProducerConfig:355)
> [2019-12-23 00:36:25,918] WARN The configuration
> 'producer.bootstrap.servers' was supplied but isn't a known config.
> (org.apache.kafka.clients.producer.ProducerConfig:355)
> [2019-12-23 00:36:25,918] WARN The configuration 'group.id' was supplied
> but isn't a known config.
> (org.apache.kafka.clients.producer.ProducerConfig:355)
> [2019-12-23 00:36:25,919] WARN The configuration 'status.storage.topic'
> was supplied but isn't a known config.
> (org.apache.kafka.clients.producer.ProducerConfig:355)
> [2019-12-23 00:36:25,919] WARN The configuration 'header.converter' was
> supplied but isn't a known config.
> (org.apache.kafka.clients.producer.ProducerConfig:355)
> [2019-12-23 00:36:25,919] WARN The configuration
> 'consumer.bootstrap.servers' was supplied but isn't a known config.
> (org.apache.kafka.clients.producer.ProducerConfig:355)
> [2019-12-23 00:36:25,919] WARN The configuration 'offset.storage.topic'
> was supplied but isn't a known config.
> (org.apache.kafka.clients.producer.ProducerConfig:355)
> [2019-12-23 00:36:25,919] WARN The configuration 'value.converter' was
> supplied but isn't a known config.
> (org.apache.kafka.clients.producer.ProducerConfig:355)
> [2019-12-23 00:36:25,919] WARN The configuration 'key.converter' was
> supplied but isn't a known config.
> (org.apache.kafka.clients.producer.ProducerConfig:355)
> [2019-12-23 00:36:25,919] WARN The configuration
> 'admin.bootstrap.servers' was supplied but isn't a known config.
> (org.apache.kafka.clients.producer.ProducerConfig:355)
>
> And this error:
>
> [2019-12-23 00:36:29,320] ERROR Plugin class loader for connector:
> 'org.apache.kafka.connect.mirror.MirrorSourceConnector' was not found.
> Returning:
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader@5c316230
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:165)
>
> First I tried the config mentioned in the KIP for "MirrorMaker Clusters"
> which didn't work and I found removing the "cluster." from the
> bootstrap-servers made it work a bit more, at least it didn't complain
> about not having any servers in the config.
> So, I checked the "Running a dedicated MirrorMaker cluster"from the KIP,
> which is basically more or less the same, but without the "cluster." for
> the servers and it does at least start and it looks like all the three
> MMs find each other, but no mirroring taking place.
>
> Running the legacy-config from the old MM is working fine though. I'll
> try to do some more digging today, so if you need some of those very
> verbose logs or something else just let me know. I am sure that I can
> figure this out and just wanted to know if the documentation will get
> extended as the new MM2 has a lot of features and is a bit more
> complicated than the old one...
>
> Thanks
>
> Sebastian
>
> On 24-Dec-19 8:06 AM, Ryanne Dolan wrote:
> > Hello Sebastian, please let us know what issues you are facing and we can
> > probably help. Which config from the KIP are you referencing? Also check
> > out the readme under ./connect/mirror for more examples.
> >
> > Ryanne
> >
> > On Mon, Dec 23, 2019, 12:58 PM Sebastian Schmitz <
> > sebastian.schmitz@propellerhead.co.nz> wrote:
> >
> >> Hello,
> >>
> >> I'm currently trying to implement the new Kafka 2.4.0 and the new MM2.
> >>
> >> However, it looks like the only documentation available is the KIP-382,
> >> and the documentation
> >> (https://kafka.apache.org/documentation/#basic_ops_mirror_maker) for
> the
> >> MM isn't yet updated, and the documentation in the KIP seems to be
> >> missing some stuff as I get a lot of errors and warning when starting
> >> the MM2 as connect-mirror, and it doesn't mirror, so I probably have
> >> some mistakes in my configuration, but can't confirm this as it's the
> >> same as in the KIP.
> >>
> >> Any plans when the documentation will be updated?
> >>
> >> Thanks
> >>
> >> Sebastian
> >>
> >>
> >> --
> >> DISCLAIMER
> >> This email contains information that is confidential and which
> >> may be
> >> legally privileged. If you have received this email in error please
> >>
> >> notify the sender immediately and delete the email.
> >> This email is intended
> >> solely for the use of the intended recipient and you may not use or
> >> disclose this email in any way.
> >>
>
> --
> DISCLAIMER
> This email contains information that is confidential and which
> may be
> legally privileged. If you have received this email in error please
>
> notify the sender immediately and delete the email.
> This email is intended
> solely for the use of the intended recipient and you may not use or
> disclose this email in any way.
>
Re: Mirrormaker 2.0
Posted by Sebastian Schmitz <se...@propellerhead.co.nz>.
Hello Ryanne,
thank you, that helps to get a better understanding.
We'll just wait until something better is available and until then use
the legacy-mode of MM2...
Best regards
Sebastian
On 30-Dec-19 7:04 PM, Ryanne Dolan wrote:
>> Is there a way to prevent that from happening?
> Unfortunately there is no tooling (yet?) to manipulate Connect's offsets,
> so it's difficult to force MM2 to skip ahead, reset, etc.
>
> One approach is to use Connect's Simple Message Transform feature. This
> enables you to filter the messages being replicated, e.g. based on
> timestamps, s.t. only recent messages are ever replicated. It's possible to
> configure MM2 to use an existing SMT or you can write your own as a plug-in.
>
> Ryanne
>
> On Thu, Dec 26, 2019, 12:25 PM Sebastian Schmitz <
> sebastian.schmitz@propellerhead.co.nz> wrote:
>
>> Hello Ryanne,
>>
>> Is there a way to prevent that from happening? We have two separate
>> clusters with some topics being replicated to the second one for
>> reporting. If we replicate everything again that reporting would
>> probably have some problems.
>>
>> Yes, I wondered when the Networking-guys would come and complain about
>> me using too much bandwidth on the VPN-Link ;)
>>
>> Thanks
>>
>> Sebastian
>>
>> On 24-Dec-19 1:11 PM, Ryanne Dolan wrote:
>>> Glad to hear you are replicating now :)
>>>
>>>> it probably started mirroring the last seven days as there was no offset
>>> for the new consumer-group.
>>>
>>> That's correct -- MM2 will replicate the entire topic, as far back as the
>>> retention period. However, technically there are no consumer groups in
>> MM2!
>>> 550MB/s in a test cluster sounds pretty good to me. Try increasing
>>> "tasks.max" and adding additional nodes.
>>>
>>> Ryanne
>>>
>>>
>>> On Mon, Dec 23, 2019 at 5:40 PM Sebastian Schmitz <
>>> sebastian.schmitz@propellerhead.co.nz> wrote:
>>>
>>>> Hello again!
>>>>
>>>> Some probably important configs I found out:
>>>>
>>>> We need this to enable mirroring as it seems to disabled by default?
>>>>
>>>> source->target.enabled = true
>>>> target->source.enabled = true
>>>>
>>>> Also, the Client-IDs can be configured using:
>>>>
>>>> source.client.id = my_cool_id
>>>> target.client.id = my_cooler_id
>>>>
>>>> I configured them to include the ID of the server and the name of the
>>>> environment to have separate IDs per mirror-node.
>>>>
>>>> After adding these two, it looks a bit better than before, but still not
>>>> satisfied as it started to mirror from my prod to test with 550MB/s as
>>>> it probably started mirroring the last seven days as there was no offset
>>>> for the new consumer-group. That's next on my list to solve.
>>>>
>>>> Best regards
>>>>
>>>> Sebastian
>>>>
>>>> On 24-Dec-19 8:34 AM, Sebastian Schmitz wrote:
>>>>> Hello,
>>>>>
>>>>> I tried running this connect-mirror-config:
>>>>>
>>>>> <snip>
>>>>> name = $MIRROR_NAME
>>>>> clusters = source, target
>>>>> source.bootstrap.servers = $SOURCE_SERVERS
>>>>> target.bootstrap.servers = $TARGET_SERVERS
>>>>> source->target.topics = $SOURCE_TARGET_TOPICS
>>>>> target->source.topics = $TARGET_SOURCE_TOPICS
>>>>> source->target.emit.heartbeats.enabled = true
>>>>> target->source.emit.heartbeats.enabled = true
>>>>> connector.class = org.apache.kafka.connect.mirror.MirrorSourceConnector
>>>>>
>>>>> # disable some new features
>>>>> refresh.topics.enabled = false
>>>>> refresh.groups.enabled = false
>>>>> emit.checkpoints.enables = true
>>>>> emit.heartbeats.enabled = true
>>>>> sync.topic.configs.enabled = false
>>>>> sync.topic.acls.enabled = false
>>>>> </snip>
>>>>>
>>>>> SOURCE_SERVERS and TARGET_SERVERS are a comma-separated list of three
>>>>> brokers with ports.
>>>>> The TOPICS are |-separated lists of topics.
>>>>>
>>>>> I get these warning during startup which is a bit weird as I never
>>>>> supplied any of those settings, but maybe I should?
>>>>>
>>>>> [2019-12-23 00:36:25,918] WARN The configuration
>>>>> 'config.storage.topic' was supplied but isn't a known config.
>>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
>>>>> [2019-12-23 00:36:25,918] WARN The configuration
>>>>> 'producer.bootstrap.servers' was supplied but isn't a known config.
>>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
>>>>> [2019-12-23 00:36:25,918] WARN The configuration 'group.id' was
>>>>> supplied but isn't a known config.
>>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
>>>>> [2019-12-23 00:36:25,919] WARN The configuration
>>>>> 'status.storage.topic' was supplied but isn't a known config.
>>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
>>>>> [2019-12-23 00:36:25,919] WARN The configuration 'header.converter'
>>>>> was supplied but isn't a known config.
>>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
>>>>> [2019-12-23 00:36:25,919] WARN The configuration
>>>>> 'consumer.bootstrap.servers' was supplied but isn't a known config.
>>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
>>>>> [2019-12-23 00:36:25,919] WARN The configuration
>>>>> 'offset.storage.topic' was supplied but isn't a known config.
>>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
>>>>> [2019-12-23 00:36:25,919] WARN The configuration 'value.converter' was
>>>>> supplied but isn't a known config.
>>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
>>>>> [2019-12-23 00:36:25,919] WARN The configuration 'key.converter' was
>>>>> supplied but isn't a known config.
>>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
>>>>> [2019-12-23 00:36:25,919] WARN The configuration
>>>>> 'admin.bootstrap.servers' was supplied but isn't a known config.
>>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
>>>>>
>>>>> And this error:
>>>>>
>>>>> [2019-12-23 00:36:29,320] ERROR Plugin class loader for connector:
>>>>> 'org.apache.kafka.connect.mirror.MirrorSourceConnector' was not found.
>>>>> Returning:
>>>>>
>> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader@5c316230
>>>>> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:165)
>>>>>
>>>>> First I tried the config mentioned in the KIP for "MirrorMaker
>>>>> Clusters" which didn't work and I found removing the "cluster." from
>>>>> the bootstrap-servers made it work a bit more, at least it didn't
>>>>> complain about not having any servers in the config.
>>>>> So, I checked the "Running a dedicated MirrorMaker cluster"from the
>>>>> KIP, which is basically more or less the same, but without the
>>>>> "cluster." for the servers and it does at least start and it looks
>>>>> like all the three MMs find each other, but no mirroring taking place.
>>>>>
>>>>> Running the legacy-config from the old MM is working fine though. I'll
>>>>> try to do some more digging today, so if you need some of those very
>>>>> verbose logs or something else just let me know. I am sure that I can
>>>>> figure this out and just wanted to know if the documentation will get
>>>>> extended as the new MM2 has a lot of features and is a bit more
>>>>> complicated than the old one...
>>>>>
>>>>> Thanks
>>>>>
>>>>> Sebastian
>>>>>
>>>>> On 24-Dec-19 8:06 AM, Ryanne Dolan wrote:
>>>>>> Hello Sebastian, please let us know what issues you are facing and we
>>>>>> can
>>>>>> probably help. Which config from the KIP are you referencing? Also
>> check
>>>>>> out the readme under ./connect/mirror for more examples.
>>>>>>
>>>>>> Ryanne
>>>>>>
>>>>>> On Mon, Dec 23, 2019, 12:58 PM Sebastian Schmitz <
>>>>>> sebastian.schmitz@propellerhead.co.nz> wrote:
>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> I'm currently trying to implement the new Kafka 2.4.0 and the new
>> MM2.
>>>>>>> However, it looks like the only documentation available is the
>> KIP-382,
>>>>>>> and the documentation
>>>>>>> (https://kafka.apache.org/documentation/#basic_ops_mirror_maker) for
>>>>>>> the
>>>>>>> MM isn't yet updated, and the documentation in the KIP seems to be
>>>>>>> missing some stuff as I get a lot of errors and warning when starting
>>>>>>> the MM2 as connect-mirror, and it doesn't mirror, so I probably have
>>>>>>> some mistakes in my configuration, but can't confirm this as it's the
>>>>>>> same as in the KIP.
>>>>>>>
>>>>>>> Any plans when the documentation will be updated?
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>> Sebastian
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> DISCLAIMER
>>>>>>> This email contains information that is confidential and which
>>>>>>> may be
>>>>>>> legally privileged. If you have received this email in error please
>>>>>>>
>>>>>>> notify the sender immediately and delete the email.
>>>>>>> This email is intended
>>>>>>> solely for the use of the intended recipient and you may not use or
>>>>>>> disclose this email in any way.
>>>>>>>
>>>> --
>>>> DISCLAIMER
>>>> This email contains information that is confidential and which
>>>> may be
>>>> legally privileged. If you have received this email in error please
>>>>
>>>> notify the sender immediately and delete the email.
>>>> This email is intended
>>>> solely for the use of the intended recipient and you may not use or
>>>> disclose this email in any way.
>>>>
>> --
>> DISCLAIMER
>> This email contains information that is confidential and which
>> may be
>> legally privileged. If you have received this email in error please
>>
>> notify the sender immediately and delete the email.
>> This email is intended
>> solely for the use of the intended recipient and you may not use or
>> disclose this email in any way.
>>
--
DISCLAIMER
This email contains information that is confidential and which
may be
legally privileged. If you have received this email in error please
notify the sender immediately and delete the email.
This email is intended
solely for the use of the intended recipient and you may not use or
disclose this email in any way.
Re: Mirrormaker 2.0
Posted by Ryanne Dolan <ry...@gmail.com>.
> Is there a way to prevent that from happening?
Unfortunately there is no tooling (yet?) to manipulate Connect's offsets,
so it's difficult to force MM2 to skip ahead, reset, etc.
One approach is to use Connect's Simple Message Transform feature. This
enables you to filter the messages being replicated, e.g. based on
timestamps, s.t. only recent messages are ever replicated. It's possible to
configure MM2 to use an existing SMT or you can write your own as a plug-in.
Ryanne
On Thu, Dec 26, 2019, 12:25 PM Sebastian Schmitz <
sebastian.schmitz@propellerhead.co.nz> wrote:
> Hello Ryanne,
>
> Is there a way to prevent that from happening? We have two separate
> clusters with some topics being replicated to the second one for
> reporting. If we replicate everything again that reporting would
> probably have some problems.
>
> Yes, I wondered when the Networking-guys would come and complain about
> me using too much bandwidth on the VPN-Link ;)
>
> Thanks
>
> Sebastian
>
> On 24-Dec-19 1:11 PM, Ryanne Dolan wrote:
> > Glad to hear you are replicating now :)
> >
> >> it probably started mirroring the last seven days as there was no offset
> > for the new consumer-group.
> >
> > That's correct -- MM2 will replicate the entire topic, as far back as the
> > retention period. However, technically there are no consumer groups in
> MM2!
> >
> > 550MB/s in a test cluster sounds pretty good to me. Try increasing
> > "tasks.max" and adding additional nodes.
> >
> > Ryanne
> >
> >
> > On Mon, Dec 23, 2019 at 5:40 PM Sebastian Schmitz <
> > sebastian.schmitz@propellerhead.co.nz> wrote:
> >
> >> Hello again!
> >>
> >> Some probably important configs I found out:
> >>
> >> We need this to enable mirroring as it seems to disabled by default?
> >>
> >> source->target.enabled = true
> >> target->source.enabled = true
> >>
> >> Also, the Client-IDs can be configured using:
> >>
> >> source.client.id = my_cool_id
> >> target.client.id = my_cooler_id
> >>
> >> I configured them to include the ID of the server and the name of the
> >> environment to have separate IDs per mirror-node.
> >>
> >> After adding these two, it looks a bit better than before, but still not
> >> satisfied as it started to mirror from my prod to test with 550MB/s as
> >> it probably started mirroring the last seven days as there was no offset
> >> for the new consumer-group. That's next on my list to solve.
> >>
> >> Best regards
> >>
> >> Sebastian
> >>
> >> On 24-Dec-19 8:34 AM, Sebastian Schmitz wrote:
> >>> Hello,
> >>>
> >>> I tried running this connect-mirror-config:
> >>>
> >>> <snip>
> >>> name = $MIRROR_NAME
> >>> clusters = source, target
> >>> source.bootstrap.servers = $SOURCE_SERVERS
> >>> target.bootstrap.servers = $TARGET_SERVERS
> >>> source->target.topics = $SOURCE_TARGET_TOPICS
> >>> target->source.topics = $TARGET_SOURCE_TOPICS
> >>> source->target.emit.heartbeats.enabled = true
> >>> target->source.emit.heartbeats.enabled = true
> >>> connector.class = org.apache.kafka.connect.mirror.MirrorSourceConnector
> >>>
> >>> # disable some new features
> >>> refresh.topics.enabled = false
> >>> refresh.groups.enabled = false
> >>> emit.checkpoints.enables = true
> >>> emit.heartbeats.enabled = true
> >>> sync.topic.configs.enabled = false
> >>> sync.topic.acls.enabled = false
> >>> </snip>
> >>>
> >>> SOURCE_SERVERS and TARGET_SERVERS are a comma-separated list of three
> >>> brokers with ports.
> >>> The TOPICS are |-separated lists of topics.
> >>>
> >>> I get these warning during startup which is a bit weird as I never
> >>> supplied any of those settings, but maybe I should?
> >>>
> >>> [2019-12-23 00:36:25,918] WARN The configuration
> >>> 'config.storage.topic' was supplied but isn't a known config.
> >>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> >>> [2019-12-23 00:36:25,918] WARN The configuration
> >>> 'producer.bootstrap.servers' was supplied but isn't a known config.
> >>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> >>> [2019-12-23 00:36:25,918] WARN The configuration 'group.id' was
> >>> supplied but isn't a known config.
> >>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> >>> [2019-12-23 00:36:25,919] WARN The configuration
> >>> 'status.storage.topic' was supplied but isn't a known config.
> >>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> >>> [2019-12-23 00:36:25,919] WARN The configuration 'header.converter'
> >>> was supplied but isn't a known config.
> >>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> >>> [2019-12-23 00:36:25,919] WARN The configuration
> >>> 'consumer.bootstrap.servers' was supplied but isn't a known config.
> >>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> >>> [2019-12-23 00:36:25,919] WARN The configuration
> >>> 'offset.storage.topic' was supplied but isn't a known config.
> >>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> >>> [2019-12-23 00:36:25,919] WARN The configuration 'value.converter' was
> >>> supplied but isn't a known config.
> >>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> >>> [2019-12-23 00:36:25,919] WARN The configuration 'key.converter' was
> >>> supplied but isn't a known config.
> >>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> >>> [2019-12-23 00:36:25,919] WARN The configuration
> >>> 'admin.bootstrap.servers' was supplied but isn't a known config.
> >>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> >>>
> >>> And this error:
> >>>
> >>> [2019-12-23 00:36:29,320] ERROR Plugin class loader for connector:
> >>> 'org.apache.kafka.connect.mirror.MirrorSourceConnector' was not found.
> >>> Returning:
> >>>
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader@5c316230
> >>> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:165)
> >>>
> >>> First I tried the config mentioned in the KIP for "MirrorMaker
> >>> Clusters" which didn't work and I found removing the "cluster." from
> >>> the bootstrap-servers made it work a bit more, at least it didn't
> >>> complain about not having any servers in the config.
> >>> So, I checked the "Running a dedicated MirrorMaker cluster"from the
> >>> KIP, which is basically more or less the same, but without the
> >>> "cluster." for the servers and it does at least start and it looks
> >>> like all the three MMs find each other, but no mirroring taking place.
> >>>
> >>> Running the legacy-config from the old MM is working fine though. I'll
> >>> try to do some more digging today, so if you need some of those very
> >>> verbose logs or something else just let me know. I am sure that I can
> >>> figure this out and just wanted to know if the documentation will get
> >>> extended as the new MM2 has a lot of features and is a bit more
> >>> complicated than the old one...
> >>>
> >>> Thanks
> >>>
> >>> Sebastian
> >>>
> >>> On 24-Dec-19 8:06 AM, Ryanne Dolan wrote:
> >>>> Hello Sebastian, please let us know what issues you are facing and we
> >>>> can
> >>>> probably help. Which config from the KIP are you referencing? Also
> check
> >>>> out the readme under ./connect/mirror for more examples.
> >>>>
> >>>> Ryanne
> >>>>
> >>>> On Mon, Dec 23, 2019, 12:58 PM Sebastian Schmitz <
> >>>> sebastian.schmitz@propellerhead.co.nz> wrote:
> >>>>
> >>>>> Hello,
> >>>>>
> >>>>> I'm currently trying to implement the new Kafka 2.4.0 and the new
> MM2.
> >>>>>
> >>>>> However, it looks like the only documentation available is the
> KIP-382,
> >>>>> and the documentation
> >>>>> (https://kafka.apache.org/documentation/#basic_ops_mirror_maker) for
> >>>>> the
> >>>>> MM isn't yet updated, and the documentation in the KIP seems to be
> >>>>> missing some stuff as I get a lot of errors and warning when starting
> >>>>> the MM2 as connect-mirror, and it doesn't mirror, so I probably have
> >>>>> some mistakes in my configuration, but can't confirm this as it's the
> >>>>> same as in the KIP.
> >>>>>
> >>>>> Any plans when the documentation will be updated?
> >>>>>
> >>>>> Thanks
> >>>>>
> >>>>> Sebastian
> >>>>>
> >>>>>
> >>>>> --
> >>>>> DISCLAIMER
> >>>>> This email contains information that is confidential and which
> >>>>> may be
> >>>>> legally privileged. If you have received this email in error please
> >>>>>
> >>>>> notify the sender immediately and delete the email.
> >>>>> This email is intended
> >>>>> solely for the use of the intended recipient and you may not use or
> >>>>> disclose this email in any way.
> >>>>>
> >> --
> >> DISCLAIMER
> >> This email contains information that is confidential and which
> >> may be
> >> legally privileged. If you have received this email in error please
> >>
> >> notify the sender immediately and delete the email.
> >> This email is intended
> >> solely for the use of the intended recipient and you may not use or
> >> disclose this email in any way.
> >>
>
> --
> DISCLAIMER
> This email contains information that is confidential and which
> may be
> legally privileged. If you have received this email in error please
>
> notify the sender immediately and delete the email.
> This email is intended
> solely for the use of the intended recipient and you may not use or
> disclose this email in any way.
>
Re: Mirrormaker 2.0
Posted by Péter Sinóros-Szabó <pe...@transferwise.com.INVALID>.
Hello Ryanne,
Are there any plans to implement an easy to use throttling to be a little
more kind with the cluster that we start to replicate?
I guess it is possible to use the existing throttling in the source and
destination clusters, but it is not really easy to use.
Also maybe an option to start mirroring from "the latest" messages?
Thanks,
Peter
On Thu, 26 Dec 2019 at 19:24, Sebastian Schmitz <
sebastian.schmitz@propellerhead.co.nz> wrote:
> Hello Ryanne,
>
> Is there a way to prevent that from happening? We have two separate
> clusters with some topics being replicated to the second one for
> reporting. If we replicate everything again that reporting would
> probably have some problems.
>
> Yes, I wondered when the Networking-guys would come and complain about
> me using too much bandwidth on the VPN-Link ;)
>
> Thanks
>
> Sebastian
>
> On 24-Dec-19 1:11 PM, Ryanne Dolan wrote:
> > Glad to hear you are replicating now :)
> >
> >> it probably started mirroring the last seven days as there was no offset
> > for the new consumer-group.
> >
> > That's correct -- MM2 will replicate the entire topic, as far back as the
> > retention period. However, technically there are no consumer groups in
> MM2!
> >
> > 550MB/s in a test cluster sounds pretty good to me. Try increasing
> > "tasks.max" and adding additional nodes.
> >
> > Ryanne
> >
> >
> > On Mon, Dec 23, 2019 at 5:40 PM Sebastian Schmitz <
> > sebastian.schmitz@propellerhead.co.nz> wrote:
> >
> >> Hello again!
> >>
> >> Some probably important configs I found out:
> >>
> >> We need this to enable mirroring as it seems to disabled by default?
> >>
> >> source->target.enabled = true
> >> target->source.enabled = true
> >>
> >> Also, the Client-IDs can be configured using:
> >>
> >> source.client.id = my_cool_id
> >> target.client.id = my_cooler_id
> >>
> >> I configured them to include the ID of the server and the name of the
> >> environment to have separate IDs per mirror-node.
> >>
> >> After adding these two, it looks a bit better than before, but still not
> >> satisfied as it started to mirror from my prod to test with 550MB/s as
> >> it probably started mirroring the last seven days as there was no offset
> >> for the new consumer-group. That's next on my list to solve.
> >>
> >> Best regards
> >>
> >> Sebastian
> >>
> >> On 24-Dec-19 8:34 AM, Sebastian Schmitz wrote:
> >>> Hello,
> >>>
> >>> I tried running this connect-mirror-config:
> >>>
> >>> <snip>
> >>> name = $MIRROR_NAME
> >>> clusters = source, target
> >>> source.bootstrap.servers = $SOURCE_SERVERS
> >>> target.bootstrap.servers = $TARGET_SERVERS
> >>> source->target.topics = $SOURCE_TARGET_TOPICS
> >>> target->source.topics = $TARGET_SOURCE_TOPICS
> >>> source->target.emit.heartbeats.enabled = true
> >>> target->source.emit.heartbeats.enabled = true
> >>> connector.class = org.apache.kafka.connect.mirror.MirrorSourceConnector
> >>>
> >>> # disable some new features
> >>> refresh.topics.enabled = false
> >>> refresh.groups.enabled = false
> >>> emit.checkpoints.enables = true
> >>> emit.heartbeats.enabled = true
> >>> sync.topic.configs.enabled = false
> >>> sync.topic.acls.enabled = false
> >>> </snip>
> >>>
> >>> SOURCE_SERVERS and TARGET_SERVERS are a comma-separated list of three
> >>> brokers with ports.
> >>> The TOPICS are |-separated lists of topics.
> >>>
> >>> I get these warning during startup which is a bit weird as I never
> >>> supplied any of those settings, but maybe I should?
> >>>
> >>> [2019-12-23 00:36:25,918] WARN The configuration
> >>> 'config.storage.topic' was supplied but isn't a known config.
> >>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> >>> [2019-12-23 00:36:25,918] WARN The configuration
> >>> 'producer.bootstrap.servers' was supplied but isn't a known config.
> >>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> >>> [2019-12-23 00:36:25,918] WARN The configuration 'group.id' was
> >>> supplied but isn't a known config.
> >>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> >>> [2019-12-23 00:36:25,919] WARN The configuration
> >>> 'status.storage.topic' was supplied but isn't a known config.
> >>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> >>> [2019-12-23 00:36:25,919] WARN The configuration 'header.converter'
> >>> was supplied but isn't a known config.
> >>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> >>> [2019-12-23 00:36:25,919] WARN The configuration
> >>> 'consumer.bootstrap.servers' was supplied but isn't a known config.
> >>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> >>> [2019-12-23 00:36:25,919] WARN The configuration
> >>> 'offset.storage.topic' was supplied but isn't a known config.
> >>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> >>> [2019-12-23 00:36:25,919] WARN The configuration 'value.converter' was
> >>> supplied but isn't a known config.
> >>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> >>> [2019-12-23 00:36:25,919] WARN The configuration 'key.converter' was
> >>> supplied but isn't a known config.
> >>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> >>> [2019-12-23 00:36:25,919] WARN The configuration
> >>> 'admin.bootstrap.servers' was supplied but isn't a known config.
> >>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> >>>
> >>> And this error:
> >>>
> >>> [2019-12-23 00:36:29,320] ERROR Plugin class loader for connector:
> >>> 'org.apache.kafka.connect.mirror.MirrorSourceConnector' was not found.
> >>> Returning:
> >>>
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader@5c316230
> >>> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:165)
> >>>
> >>> First I tried the config mentioned in the KIP for "MirrorMaker
> >>> Clusters" which didn't work and I found removing the "cluster." from
> >>> the bootstrap-servers made it work a bit more, at least it didn't
> >>> complain about not having any servers in the config.
> >>> So, I checked the "Running a dedicated MirrorMaker cluster"from the
> >>> KIP, which is basically more or less the same, but without the
> >>> "cluster." for the servers and it does at least start and it looks
> >>> like all the three MMs find each other, but no mirroring taking place.
> >>>
> >>> Running the legacy-config from the old MM is working fine though. I'll
> >>> try to do some more digging today, so if you need some of those very
> >>> verbose logs or something else just let me know. I am sure that I can
> >>> figure this out and just wanted to know if the documentation will get
> >>> extended as the new MM2 has a lot of features and is a bit more
> >>> complicated than the old one...
> >>>
> >>> Thanks
> >>>
> >>> Sebastian
> >>>
> >>> On 24-Dec-19 8:06 AM, Ryanne Dolan wrote:
> >>>> Hello Sebastian, please let us know what issues you are facing and we
> >>>> can
> >>>> probably help. Which config from the KIP are you referencing? Also
> check
> >>>> out the readme under ./connect/mirror for more examples.
> >>>>
> >>>> Ryanne
> >>>>
> >>>> On Mon, Dec 23, 2019, 12:58 PM Sebastian Schmitz <
> >>>> sebastian.schmitz@propellerhead.co.nz> wrote:
> >>>>
> >>>>> Hello,
> >>>>>
> >>>>> I'm currently trying to implement the new Kafka 2.4.0 and the new
> MM2.
> >>>>>
> >>>>> However, it looks like the only documentation available is the
> KIP-382,
> >>>>> and the documentation
> >>>>> (https://kafka.apache.org/documentation/#basic_ops_mirror_maker) for
> >>>>> the
> >>>>> MM isn't yet updated, and the documentation in the KIP seems to be
> >>>>> missing some stuff as I get a lot of errors and warning when starting
> >>>>> the MM2 as connect-mirror, and it doesn't mirror, so I probably have
> >>>>> some mistakes in my configuration, but can't confirm this as it's the
> >>>>> same as in the KIP.
> >>>>>
> >>>>> Any plans when the documentation will be updated?
> >>>>>
> >>>>> Thanks
> >>>>>
> >>>>> Sebastian
> >>>>>
> >>>>>
> >>>>> --
> >>>>> DISCLAIMER
> >>>>> This email contains information that is confidential and which
> >>>>> may be
> >>>>> legally privileged. If you have received this email in error please
> >>>>>
> >>>>> notify the sender immediately and delete the email.
> >>>>> This email is intended
> >>>>> solely for the use of the intended recipient and you may not use or
> >>>>> disclose this email in any way.
> >>>>>
> >> --
> >> DISCLAIMER
> >> This email contains information that is confidential and which
> >> may be
> >> legally privileged. If you have received this email in error please
> >>
> >> notify the sender immediately and delete the email.
> >> This email is intended
> >> solely for the use of the intended recipient and you may not use or
> >> disclose this email in any way.
> >>
>
> --
> DISCLAIMER
> This email contains information that is confidential and which
> may be
> legally privileged. If you have received this email in error please
>
> notify the sender immediately and delete the email.
> This email is intended
> solely for the use of the intended recipient and you may not use or
> disclose this email in any way.
>
--
- Sini
Re: Mirrormaker 2.0
Posted by Ryanne Dolan <ry...@gmail.com>.
Peter, that's right. So long as ReplicationPolicy is implemented with
proper semantics (i.e. the methods do what they say they should do) any
naming convention will work. You can also use something like double
underscore "__" as a separator with DefaultReplicationPolicy -- it doesn't
need to be a single character.
Ryanne
On Thu, Jan 9, 2020, 7:24 AM Péter Sinóros-Szabó
<pe...@transferwise.com.invalid> wrote:
> Hi Ryanne,
>
> Am I right that as far as I implement ReplicationPolicy properly, those
> features you just mentioned will work fine?
>
> Asking because we already use dot(.) underscore(_) and even hyphen(-)
> characters in not replicated topics :D , so it seems to be that we will
> need a more advanced renaming convention if we plan to use those mentioned
> features. Or we'll need to rename topics, but that's a huge task I'd like
> to avoid...
>
> Thanks,
> Peter
>
Re: Mirrormaker 2.0
Posted by Péter Sinóros-Szabó <pe...@transferwise.com.INVALID>.
Hi Ryanne,
Am I right that as far as I implement ReplicationPolicy properly, those
features you just mentioned will work fine?
Asking because we already use dot(.) underscore(_) and even hyphen(-)
characters in not replicated topics :D , so it seems to be that we will
need a more advanced renaming convention if we plan to use those mentioned
features. Or we'll need to rename topics, but that's a huge task I'd like
to avoid...
Thanks,
Peter
Re: Mirrormaker 2.0
Posted by Péter Sinóros-Szabó <pe...@transferwise.com.INVALID>.
Ok, I see. I almost started to work on it, but figured out that we do not
need it now.
Thanks for the help around this topic :)
Peter
On Tue, 21 Jan 2020 at 21:04, Ryanne Dolan <ry...@gmail.com> wrote:
> Peter, the LegacyReplicationPolicy class is described in the existing
> KIP-382 and is a requirement for the deprecation of MM1. I was planning to
> implement it but would love the help if you're interested.
>
> Ryanne
>
> On Tue, Jan 21, 2020, 8:25 AM Péter Sinóros-Szabó
> <pe...@transferwise.com.invalid> wrote:
>
> > Ryanne,
> >
> > I didn't do much work yet, just checked the Interface to see if it is
> easy
> > to implement or not.
> >
> > > The PR for LegacyReplicationPolicy should include any relevant fixes to
> > get it to run without crashing
> > Do you mean that there is already a PR for LegacyReplicationPolicy? If
> > there is, please link it here, I could not find it.
> >
> > Thanks,
> > Peter
> >
> >
> >
> >
> > On Fri, 17 Jan 2020 at 20:58, Ryanne Dolan <ry...@gmail.com>
> wrote:
> >
> > > Peter, KIP-382 includes LegacyReplicationPolicy for this purpose, but
> no,
> > > it has not been implemented yet. If you are interested in writing the
> PR,
> > > it would not require a separate KIP before merging. Looks like you are
> > > already doing the work :)
> > >
> > > It is possible, as you point out, that returning nulls like that will
> > break
> > > parts of MM2. The PR for LegacyReplicationPolicy should include any
> > > relevant fixes to get it to run without crashing.
> > >
> > > Certainly some parts of MM2 will not work as intended when used with
> > > LegacyReplicationPolicy, e.g. MM2 would not be able to prevent cycles.
> > But
> > > of course it should still _run_ and should work roughly the same as
> MM1.
> > >
> > > I'm happy to work with you on the PR if you are interested.
> > >
> > > Ryanne
> > >
> > > On Fri, Jan 17, 2020 at 9:34 AM Péter Sinóros-Szabó
> > > <pe...@transferwise.com.invalid> wrote:
> > >
> > > > Hi Sebastian & Ryanne,
> > > >
> > > > do you have maybe an implementation of this is just some ideas about
> > how
> > > to
> > > > implement the policy that does not rename topics?
> > > > I am checking the ReplicationPolicy interface and don't really know
> > what
> > > > the impact will be if I implement this:
> > > >
> > > > public String formatRemoteTopic(String sourceClusterAlias, String
> > topic)
> > > {
> > > > return topic; }
> > > > public String topicSource(String topic) { return null; // well, I do
> > not
> > > > really know if this were a mirrored topic or not }
> > > > public String upstreamTopic(String topic) { return topic; // well, I
> > do
> > > > not really know if this were a mirrored topic or not}
> > > > public String originalTopic(String topic) { return topic; }
> > > >
> > > > Thanks,
> > > > Peter
> > > >
> > > > On Mon, 30 Dec 2019 at 06:57, Ryanne Dolan <ry...@gmail.com>
> > > wrote:
> > > >
> > > > > Sebastian, you can drop in a custom jar in the "Connect plug-in
> path"
> > > and
> > > > > MM2 will be able to load it. That enables you to implement your own
> > > > > ReplicationPolicy (and other pluggable interfaces) without
> compiling
> > > > > everything.
> > > > >
> > > > > In an upcoming release we'll have a "LegacyReplicationPolicy" that
> > does
> > > > not
> > > > > rename topics. It's possible "SimpleReplicationPolicy" is a better
> > > name.
> > > > >
> > > > > Be advised that some features depend on correct ReplicationPolicy
> > > > > semantics, which LegacyReplicationPolicy will explicitly break. For
> > > > > example, MM2 cannot prevent cycles if topics are not renamed (or
> some
> > > > other
> > > > > similar mechanism is used).
> > > > >
> > > > > Ryanne
> > > > >
> > > > > On Sun, Dec 29, 2019, 7:41 PM Sebastian Schmitz <
> > > > > sebastian.schmitz@propellerhead.co.nz> wrote:
> > > > >
> > > > > > Hello,
> > > > > >
> > > > > > I found that it's using the DefaultReplicationPolicy that always
> > > > returns
> > > > > > "sourceClusterAlias + separator + topic" with only the separator
> > > being
> > > > > > configurable in the configuration-file with
> > > > REPLICATION_POLICY_SEPARATOR.
> > > > > >
> > > > > > It seems like I need a different ReplicationPolicy, like a
> > > > > > SimpleReplicationPolicy which always returns "topic" for the
> > > > > > formatRemoteTopic, then. But that would mean that I can't
> download
> > > the
> > > > > > Binaries and have to build the whole thing myself after adding
> the
> > > new
> > > > > > Policy-file!?
> > > > > > Or I could create a PR for a SimpleReplicationPolicy to be in
> some
> > > > > > future build...
> > > > > >
> > > > > > Any suggestions for this?
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > > Sebastian
> > > > > >
> > > > > >
> > > > > > On 30-Dec-19 1:39 PM, Sebastian Schmitz wrote:
> > > > > > > Hello,
> > > > > > >
> > > > > > > another thing I found and didn't find any configuration in the
> > KIP
> > > > yet
> > > > > > > was that if I have two clusters (source and target) and a topic
> > > > > > > "replicateme" on the source-cluster it will get replicated to
> the
> > > > > > > target-cluster as "source.replicateme".
> > > > > > >
> > > > > > > How can I stop it from adding the cluster-name in front of the
> > > > > > > topic-name on target-cluster?
> > > > > > >
> > > > > > > Thanks
> > > > > > >
> > > > > > > Sebastian
> > > > > > >
> > > > > > > On 27-Dec-19 7:24 AM, Sebastian Schmitz wrote:
> > > > > > >> Hello Ryanne,
> > > > > > >>
> > > > > > >> Is there a way to prevent that from happening? We have two
> > > separate
> > > > > > >> clusters with some topics being replicated to the second one
> for
> > > > > > >> reporting. If we replicate everything again that reporting
> would
> > > > > > >> probably have some problems.
> > > > > > >>
> > > > > > >> Yes, I wondered when the Networking-guys would come and
> complain
> > > > > > >> about me using too much bandwidth on the VPN-Link ;)
> > > > > > >>
> > > > > > >> Thanks
> > > > > > >>
> > > > > > >> Sebastian
> > > > > > >>
> > > > > > >> On 24-Dec-19 1:11 PM, Ryanne Dolan wrote:
> > > > > > >>> Glad to hear you are replicating now :)
> > > > > > >>>
> > > > > > >>>> it probably started mirroring the last seven days as there
> was
> > > no
> > > > > > >>>> offset
> > > > > > >>> for the new consumer-group.
> > > > > > >>>
> > > > > > >>> That's correct -- MM2 will replicate the entire topic, as far
> > > back
> > > > > > >>> as the
> > > > > > >>> retention period. However, technically there are no consumer
> > > groups
> > > > > > >>> in MM2!
> > > > > > >>>
> > > > > > >>> 550MB/s in a test cluster sounds pretty good to me. Try
> > > increasing
> > > > > > >>> "tasks.max" and adding additional nodes.
> > > > > > >>>
> > > > > > >>> Ryanne
> > > > > > >>>
> > > > > > >>>
> > > > > > >>> On Mon, Dec 23, 2019 at 5:40 PM Sebastian Schmitz <
> > > > > > >>> sebastian.schmitz@propellerhead.co.nz> wrote:
> > > > > > >>>
> > > > > > >>>> Hello again!
> > > > > > >>>>
> > > > > > >>>> Some probably important configs I found out:
> > > > > > >>>>
> > > > > > >>>> We need this to enable mirroring as it seems to disabled by
> > > > default?
> > > > > > >>>>
> > > > > > >>>> source->target.enabled = true
> > > > > > >>>> target->source.enabled = true
> > > > > > >>>>
> > > > > > >>>> Also, the Client-IDs can be configured using:
> > > > > > >>>>
> > > > > > >>>> source.client.id = my_cool_id
> > > > > > >>>> target.client.id = my_cooler_id
> > > > > > >>>>
> > > > > > >>>> I configured them to include the ID of the server and the
> name
> > > of
> > > > > the
> > > > > > >>>> environment to have separate IDs per mirror-node.
> > > > > > >>>>
> > > > > > >>>> After adding these two, it looks a bit better than before,
> but
> > > > > > >>>> still not
> > > > > > >>>> satisfied as it started to mirror from my prod to test with
> > > > 550MB/s
> > > > > as
> > > > > > >>>> it probably started mirroring the last seven days as there
> was
> > > no
> > > > > > >>>> offset
> > > > > > >>>> for the new consumer-group. That's next on my list to solve.
> > > > > > >>>>
> > > > > > >>>> Best regards
> > > > > > >>>>
> > > > > > >>>> Sebastian
> > > > > > >>>>
> > > > > > >>>> On 24-Dec-19 8:34 AM, Sebastian Schmitz wrote:
> > > > > > >>>>> Hello,
> > > > > > >>>>>
> > > > > > >>>>> I tried running this connect-mirror-config:
> > > > > > >>>>>
> > > > > > >>>>> <snip>
> > > > > > >>>>> name = $MIRROR_NAME
> > > > > > >>>>> clusters = source, target
> > > > > > >>>>> source.bootstrap.servers = $SOURCE_SERVERS
> > > > > > >>>>> target.bootstrap.servers = $TARGET_SERVERS
> > > > > > >>>>> source->target.topics = $SOURCE_TARGET_TOPICS
> > > > > > >>>>> target->source.topics = $TARGET_SOURCE_TOPICS
> > > > > > >>>>> source->target.emit.heartbeats.enabled = true
> > > > > > >>>>> target->source.emit.heartbeats.enabled = true
> > > > > > >>>>> connector.class =
> > > > > > >>>>> org.apache.kafka.connect.mirror.MirrorSourceConnector
> > > > > > >>>>>
> > > > > > >>>>> # disable some new features
> > > > > > >>>>> refresh.topics.enabled = false
> > > > > > >>>>> refresh.groups.enabled = false
> > > > > > >>>>> emit.checkpoints.enables = true
> > > > > > >>>>> emit.heartbeats.enabled = true
> > > > > > >>>>> sync.topic.configs.enabled = false
> > > > > > >>>>> sync.topic.acls.enabled = false
> > > > > > >>>>> </snip>
> > > > > > >>>>>
> > > > > > >>>>> SOURCE_SERVERS and TARGET_SERVERS are a comma-separated
> list
> > of
> > > > > three
> > > > > > >>>>> brokers with ports.
> > > > > > >>>>> The TOPICS are |-separated lists of topics.
> > > > > > >>>>>
> > > > > > >>>>> I get these warning during startup which is a bit weird as
> I
> > > > never
> > > > > > >>>>> supplied any of those settings, but maybe I should?
> > > > > > >>>>>
> > > > > > >>>>> [2019-12-23 00:36:25,918] WARN The configuration
> > > > > > >>>>> 'config.storage.topic' was supplied but isn't a known
> config.
> > > > > > >>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> > > > > > >>>>> [2019-12-23 00:36:25,918] WARN The configuration
> > > > > > >>>>> 'producer.bootstrap.servers' was supplied but isn't a known
> > > > config.
> > > > > > >>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> > > > > > >>>>> [2019-12-23 00:36:25,918] WARN The configuration 'group.id
> '
> > > was
> > > > > > >>>>> supplied but isn't a known config.
> > > > > > >>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> > > > > > >>>>> [2019-12-23 00:36:25,919] WARN The configuration
> > > > > > >>>>> 'status.storage.topic' was supplied but isn't a known
> config.
> > > > > > >>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> > > > > > >>>>> [2019-12-23 00:36:25,919] WARN The configuration
> > > > 'header.converter'
> > > > > > >>>>> was supplied but isn't a known config.
> > > > > > >>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> > > > > > >>>>> [2019-12-23 00:36:25,919] WARN The configuration
> > > > > > >>>>> 'consumer.bootstrap.servers' was supplied but isn't a known
> > > > config.
> > > > > > >>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> > > > > > >>>>> [2019-12-23 00:36:25,919] WARN The configuration
> > > > > > >>>>> 'offset.storage.topic' was supplied but isn't a known
> config.
> > > > > > >>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> > > > > > >>>>> [2019-12-23 00:36:25,919] WARN The configuration
> > > > 'value.converter'
> > > > > > >>>>> was
> > > > > > >>>>> supplied but isn't a known config.
> > > > > > >>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> > > > > > >>>>> [2019-12-23 00:36:25,919] WARN The configuration
> > > 'key.converter'
> > > > > was
> > > > > > >>>>> supplied but isn't a known config.
> > > > > > >>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> > > > > > >>>>> [2019-12-23 00:36:25,919] WARN The configuration
> > > > > > >>>>> 'admin.bootstrap.servers' was supplied but isn't a known
> > > config.
> > > > > > >>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> > > > > > >>>>>
> > > > > > >>>>> And this error:
> > > > > > >>>>>
> > > > > > >>>>> [2019-12-23 00:36:29,320] ERROR Plugin class loader for
> > > > connector:
> > > > > > >>>>> 'org.apache.kafka.connect.mirror.MirrorSourceConnector' was
> > not
> > > > > > >>>>> found.
> > > > > > >>>>> Returning:
> > > > > > >>>>>
> > > > > >
> > > >
> > org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader@5c316230
> > > > > > >>>>>
> > > > > > >>>>>
> > > > > >
> > > (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:165)
> > > > > > >>>>>
> > > > > > >>>>>
> > > > > > >>>>> First I tried the config mentioned in the KIP for
> > "MirrorMaker
> > > > > > >>>>> Clusters" which didn't work and I found removing the
> > "cluster."
> > > > > from
> > > > > > >>>>> the bootstrap-servers made it work a bit more, at least it
> > > didn't
> > > > > > >>>>> complain about not having any servers in the config.
> > > > > > >>>>> So, I checked the "Running a dedicated MirrorMaker
> > cluster"from
> > > > the
> > > > > > >>>>> KIP, which is basically more or less the same, but without
> > the
> > > > > > >>>>> "cluster." for the servers and it does at least start and
> it
> > > > looks
> > > > > > >>>>> like all the three MMs find each other, but no mirroring
> > taking
> > > > > > >>>>> place.
> > > > > > >>>>>
> > > > > > >>>>> Running the legacy-config from the old MM is working fine
> > > though.
> > > > > > >>>>> I'll
> > > > > > >>>>> try to do some more digging today, so if you need some of
> > those
> > > > > very
> > > > > > >>>>> verbose logs or something else just let me know. I am sure
> > > that I
> > > > > can
> > > > > > >>>>> figure this out and just wanted to know if the
> documentation
> > > will
> > > > > get
> > > > > > >>>>> extended as the new MM2 has a lot of features and is a bit
> > more
> > > > > > >>>>> complicated than the old one...
> > > > > > >>>>>
> > > > > > >>>>> Thanks
> > > > > > >>>>>
> > > > > > >>>>> Sebastian
> > > > > > >>>>>
> > > > > > >>>>> On 24-Dec-19 8:06 AM, Ryanne Dolan wrote:
> > > > > > >>>>>> Hello Sebastian, please let us know what issues you are
> > facing
> > > > > > >>>>>> and we
> > > > > > >>>>>> can
> > > > > > >>>>>> probably help. Which config from the KIP are you
> > referencing?
> > > > > > >>>>>> Also check
> > > > > > >>>>>> out the readme under ./connect/mirror for more examples.
> > > > > > >>>>>>
> > > > > > >>>>>> Ryanne
> > > > > > >>>>>>
> > > > > > >>>>>> On Mon, Dec 23, 2019, 12:58 PM Sebastian Schmitz <
> > > > > > >>>>>> sebastian.schmitz@propellerhead.co.nz> wrote:
> > > > > > >>>>>>
> > > > > > >>>>>>> Hello,
> > > > > > >>>>>>>
> > > > > > >>>>>>> I'm currently trying to implement the new Kafka 2.4.0 and
> > the
> > > > > > >>>>>>> new MM2.
> > > > > > >>>>>>>
> > > > > > >>>>>>> However, it looks like the only documentation available
> is
> > > the
> > > > > > >>>>>>> KIP-382,
> > > > > > >>>>>>> and the documentation
> > > > > > >>>>>>> (
> > > > https://kafka.apache.org/documentation/#basic_ops_mirror_maker)
> > > > > > >>>>>>> for
> > > > > > >>>>>>> the
> > > > > > >>>>>>> MM isn't yet updated, and the documentation in the KIP
> > seems
> > > to
> > > > > be
> > > > > > >>>>>>> missing some stuff as I get a lot of errors and warning
> > when
> > > > > > >>>>>>> starting
> > > > > > >>>>>>> the MM2 as connect-mirror, and it doesn't mirror, so I
> > > probably
> > > > > > >>>>>>> have
> > > > > > >>>>>>> some mistakes in my configuration, but can't confirm this
> > as
> > > > > > >>>>>>> it's the
> > > > > > >>>>>>> same as in the KIP.
> > > > > > >>>>>>>
> > > > > > >>>>>>> Any plans when the documentation will be updated?
> > > > > > >>>>>>>
> > > > > > >>>>>>> Thanks
> > > > > > >>>>>>>
> > > > > > >>>>>>> Sebastian
> > > > > > >>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>>>>> --
> > > > > > >>>>>>> DISCLAIMER
> > > > > > >>>>>>> This email contains information that is confidential and
> > > which
> > > > > > >>>>>>> may be
> > > > > > >>>>>>> legally privileged. If you have received this email in
> > error
> > > > > please
> > > > > > >>>>>>>
> > > > > > >>>>>>> notify the sender immediately and delete the email.
> > > > > > >>>>>>> This email is intended
> > > > > > >>>>>>> solely for the use of the intended recipient and you may
> > not
> > > > use
> > > > > or
> > > > > > >>>>>>> disclose this email in any way.
> > > > > > >>>>>>>
> > > > > > >>>> --
> > > > > > >>>> DISCLAIMER
> > > > > > >>>> This email contains information that is confidential and
> which
> > > > > > >>>> may be
> > > > > > >>>> legally privileged. If you have received this email in error
> > > > please
> > > > > > >>>>
> > > > > > >>>> notify the sender immediately and delete the email.
> > > > > > >>>> This email is intended
> > > > > > >>>> solely for the use of the intended recipient and you may not
> > use
> > > > or
> > > > > > >>>> disclose this email in any way.
> > > > > > >>>>
> > > > > >
> > > > > > --
> > > > > > DISCLAIMER
> > > > > > This email contains information that is confidential and which
> > > > > > may be
> > > > > > legally privileged. If you have received this email in error
> please
> > > > > >
> > > > > > notify the sender immediately and delete the email.
> > > > > > This email is intended
> > > > > > solely for the use of the intended recipient and you may not use
> or
> > > > > > disclose this email in any way.
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Mirrormaker 2.0
Posted by Ryanne Dolan <ry...@gmail.com>.
Peter, the LegacyReplicationPolicy class is described in the existing
KIP-382 and is a requirement for the deprecation of MM1. I was planning to
implement it but would love the help if you're interested.
Ryanne
On Tue, Jan 21, 2020, 8:25 AM Péter Sinóros-Szabó
<pe...@transferwise.com.invalid> wrote:
> Ryanne,
>
> I didn't do much work yet, just checked the Interface to see if it is easy
> to implement or not.
>
> > The PR for LegacyReplicationPolicy should include any relevant fixes to
> get it to run without crashing
> Do you mean that there is already a PR for LegacyReplicationPolicy? If
> there is, please link it here, I could not find it.
>
> Thanks,
> Peter
>
>
>
>
> On Fri, 17 Jan 2020 at 20:58, Ryanne Dolan <ry...@gmail.com> wrote:
>
> > Peter, KIP-382 includes LegacyReplicationPolicy for this purpose, but no,
> > it has not been implemented yet. If you are interested in writing the PR,
> > it would not require a separate KIP before merging. Looks like you are
> > already doing the work :)
> >
> > It is possible, as you point out, that returning nulls like that will
> break
> > parts of MM2. The PR for LegacyReplicationPolicy should include any
> > relevant fixes to get it to run without crashing.
> >
> > Certainly some parts of MM2 will not work as intended when used with
> > LegacyReplicationPolicy, e.g. MM2 would not be able to prevent cycles.
> But
> > of course it should still _run_ and should work roughly the same as MM1.
> >
> > I'm happy to work with you on the PR if you are interested.
> >
> > Ryanne
> >
> > On Fri, Jan 17, 2020 at 9:34 AM Péter Sinóros-Szabó
> > <pe...@transferwise.com.invalid> wrote:
> >
> > > Hi Sebastian & Ryanne,
> > >
> > > do you have maybe an implementation of this is just some ideas about
> how
> > to
> > > implement the policy that does not rename topics?
> > > I am checking the ReplicationPolicy interface and don't really know
> what
> > > the impact will be if I implement this:
> > >
> > > public String formatRemoteTopic(String sourceClusterAlias, String
> topic)
> > {
> > > return topic; }
> > > public String topicSource(String topic) { return null; // well, I do
> not
> > > really know if this were a mirrored topic or not }
> > > public String upstreamTopic(String topic) { return topic; // well, I
> do
> > > not really know if this were a mirrored topic or not}
> > > public String originalTopic(String topic) { return topic; }
> > >
> > > Thanks,
> > > Peter
> > >
> > > On Mon, 30 Dec 2019 at 06:57, Ryanne Dolan <ry...@gmail.com>
> > wrote:
> > >
> > > > Sebastian, you can drop in a custom jar in the "Connect plug-in path"
> > and
> > > > MM2 will be able to load it. That enables you to implement your own
> > > > ReplicationPolicy (and other pluggable interfaces) without compiling
> > > > everything.
> > > >
> > > > In an upcoming release we'll have a "LegacyReplicationPolicy" that
> does
> > > not
> > > > rename topics. It's possible "SimpleReplicationPolicy" is a better
> > name.
> > > >
> > > > Be advised that some features depend on correct ReplicationPolicy
> > > > semantics, which LegacyReplicationPolicy will explicitly break. For
> > > > example, MM2 cannot prevent cycles if topics are not renamed (or some
> > > other
> > > > similar mechanism is used).
> > > >
> > > > Ryanne
> > > >
> > > > On Sun, Dec 29, 2019, 7:41 PM Sebastian Schmitz <
> > > > sebastian.schmitz@propellerhead.co.nz> wrote:
> > > >
> > > > > Hello,
> > > > >
> > > > > I found that it's using the DefaultReplicationPolicy that always
> > > returns
> > > > > "sourceClusterAlias + separator + topic" with only the separator
> > being
> > > > > configurable in the configuration-file with
> > > REPLICATION_POLICY_SEPARATOR.
> > > > >
> > > > > It seems like I need a different ReplicationPolicy, like a
> > > > > SimpleReplicationPolicy which always returns "topic" for the
> > > > > formatRemoteTopic, then. But that would mean that I can't download
> > the
> > > > > Binaries and have to build the whole thing myself after adding the
> > new
> > > > > Policy-file!?
> > > > > Or I could create a PR for a SimpleReplicationPolicy to be in some
> > > > > future build...
> > > > >
> > > > > Any suggestions for this?
> > > > >
> > > > > Thanks
> > > > >
> > > > > Sebastian
> > > > >
> > > > >
> > > > > On 30-Dec-19 1:39 PM, Sebastian Schmitz wrote:
> > > > > > Hello,
> > > > > >
> > > > > > another thing I found and didn't find any configuration in the
> KIP
> > > yet
> > > > > > was that if I have two clusters (source and target) and a topic
> > > > > > "replicateme" on the source-cluster it will get replicated to the
> > > > > > target-cluster as "source.replicateme".
> > > > > >
> > > > > > How can I stop it from adding the cluster-name in front of the
> > > > > > topic-name on target-cluster?
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > > Sebastian
> > > > > >
> > > > > > On 27-Dec-19 7:24 AM, Sebastian Schmitz wrote:
> > > > > >> Hello Ryanne,
> > > > > >>
> > > > > >> Is there a way to prevent that from happening? We have two
> > separate
> > > > > >> clusters with some topics being replicated to the second one for
> > > > > >> reporting. If we replicate everything again that reporting would
> > > > > >> probably have some problems.
> > > > > >>
> > > > > >> Yes, I wondered when the Networking-guys would come and complain
> > > > > >> about me using too much bandwidth on the VPN-Link ;)
> > > > > >>
> > > > > >> Thanks
> > > > > >>
> > > > > >> Sebastian
> > > > > >>
> > > > > >> On 24-Dec-19 1:11 PM, Ryanne Dolan wrote:
> > > > > >>> Glad to hear you are replicating now :)
> > > > > >>>
> > > > > >>>> it probably started mirroring the last seven days as there was
> > no
> > > > > >>>> offset
> > > > > >>> for the new consumer-group.
> > > > > >>>
> > > > > >>> That's correct -- MM2 will replicate the entire topic, as far
> > back
> > > > > >>> as the
> > > > > >>> retention period. However, technically there are no consumer
> > groups
> > > > > >>> in MM2!
> > > > > >>>
> > > > > >>> 550MB/s in a test cluster sounds pretty good to me. Try
> > increasing
> > > > > >>> "tasks.max" and adding additional nodes.
> > > > > >>>
> > > > > >>> Ryanne
> > > > > >>>
> > > > > >>>
> > > > > >>> On Mon, Dec 23, 2019 at 5:40 PM Sebastian Schmitz <
> > > > > >>> sebastian.schmitz@propellerhead.co.nz> wrote:
> > > > > >>>
> > > > > >>>> Hello again!
> > > > > >>>>
> > > > > >>>> Some probably important configs I found out:
> > > > > >>>>
> > > > > >>>> We need this to enable mirroring as it seems to disabled by
> > > default?
> > > > > >>>>
> > > > > >>>> source->target.enabled = true
> > > > > >>>> target->source.enabled = true
> > > > > >>>>
> > > > > >>>> Also, the Client-IDs can be configured using:
> > > > > >>>>
> > > > > >>>> source.client.id = my_cool_id
> > > > > >>>> target.client.id = my_cooler_id
> > > > > >>>>
> > > > > >>>> I configured them to include the ID of the server and the name
> > of
> > > > the
> > > > > >>>> environment to have separate IDs per mirror-node.
> > > > > >>>>
> > > > > >>>> After adding these two, it looks a bit better than before, but
> > > > > >>>> still not
> > > > > >>>> satisfied as it started to mirror from my prod to test with
> > > 550MB/s
> > > > as
> > > > > >>>> it probably started mirroring the last seven days as there was
> > no
> > > > > >>>> offset
> > > > > >>>> for the new consumer-group. That's next on my list to solve.
> > > > > >>>>
> > > > > >>>> Best regards
> > > > > >>>>
> > > > > >>>> Sebastian
> > > > > >>>>
> > > > > >>>> On 24-Dec-19 8:34 AM, Sebastian Schmitz wrote:
> > > > > >>>>> Hello,
> > > > > >>>>>
> > > > > >>>>> I tried running this connect-mirror-config:
> > > > > >>>>>
> > > > > >>>>> <snip>
> > > > > >>>>> name = $MIRROR_NAME
> > > > > >>>>> clusters = source, target
> > > > > >>>>> source.bootstrap.servers = $SOURCE_SERVERS
> > > > > >>>>> target.bootstrap.servers = $TARGET_SERVERS
> > > > > >>>>> source->target.topics = $SOURCE_TARGET_TOPICS
> > > > > >>>>> target->source.topics = $TARGET_SOURCE_TOPICS
> > > > > >>>>> source->target.emit.heartbeats.enabled = true
> > > > > >>>>> target->source.emit.heartbeats.enabled = true
> > > > > >>>>> connector.class =
> > > > > >>>>> org.apache.kafka.connect.mirror.MirrorSourceConnector
> > > > > >>>>>
> > > > > >>>>> # disable some new features
> > > > > >>>>> refresh.topics.enabled = false
> > > > > >>>>> refresh.groups.enabled = false
> > > > > >>>>> emit.checkpoints.enables = true
> > > > > >>>>> emit.heartbeats.enabled = true
> > > > > >>>>> sync.topic.configs.enabled = false
> > > > > >>>>> sync.topic.acls.enabled = false
> > > > > >>>>> </snip>
> > > > > >>>>>
> > > > > >>>>> SOURCE_SERVERS and TARGET_SERVERS are a comma-separated list
> of
> > > > three
> > > > > >>>>> brokers with ports.
> > > > > >>>>> The TOPICS are |-separated lists of topics.
> > > > > >>>>>
> > > > > >>>>> I get these warning during startup which is a bit weird as I
> > > never
> > > > > >>>>> supplied any of those settings, but maybe I should?
> > > > > >>>>>
> > > > > >>>>> [2019-12-23 00:36:25,918] WARN The configuration
> > > > > >>>>> 'config.storage.topic' was supplied but isn't a known config.
> > > > > >>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> > > > > >>>>> [2019-12-23 00:36:25,918] WARN The configuration
> > > > > >>>>> 'producer.bootstrap.servers' was supplied but isn't a known
> > > config.
> > > > > >>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> > > > > >>>>> [2019-12-23 00:36:25,918] WARN The configuration 'group.id'
> > was
> > > > > >>>>> supplied but isn't a known config.
> > > > > >>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> > > > > >>>>> [2019-12-23 00:36:25,919] WARN The configuration
> > > > > >>>>> 'status.storage.topic' was supplied but isn't a known config.
> > > > > >>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> > > > > >>>>> [2019-12-23 00:36:25,919] WARN The configuration
> > > 'header.converter'
> > > > > >>>>> was supplied but isn't a known config.
> > > > > >>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> > > > > >>>>> [2019-12-23 00:36:25,919] WARN The configuration
> > > > > >>>>> 'consumer.bootstrap.servers' was supplied but isn't a known
> > > config.
> > > > > >>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> > > > > >>>>> [2019-12-23 00:36:25,919] WARN The configuration
> > > > > >>>>> 'offset.storage.topic' was supplied but isn't a known config.
> > > > > >>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> > > > > >>>>> [2019-12-23 00:36:25,919] WARN The configuration
> > > 'value.converter'
> > > > > >>>>> was
> > > > > >>>>> supplied but isn't a known config.
> > > > > >>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> > > > > >>>>> [2019-12-23 00:36:25,919] WARN The configuration
> > 'key.converter'
> > > > was
> > > > > >>>>> supplied but isn't a known config.
> > > > > >>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> > > > > >>>>> [2019-12-23 00:36:25,919] WARN The configuration
> > > > > >>>>> 'admin.bootstrap.servers' was supplied but isn't a known
> > config.
> > > > > >>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> > > > > >>>>>
> > > > > >>>>> And this error:
> > > > > >>>>>
> > > > > >>>>> [2019-12-23 00:36:29,320] ERROR Plugin class loader for
> > > connector:
> > > > > >>>>> 'org.apache.kafka.connect.mirror.MirrorSourceConnector' was
> not
> > > > > >>>>> found.
> > > > > >>>>> Returning:
> > > > > >>>>>
> > > > >
> > >
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader@5c316230
> > > > > >>>>>
> > > > > >>>>>
> > > > >
> > (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:165)
> > > > > >>>>>
> > > > > >>>>>
> > > > > >>>>> First I tried the config mentioned in the KIP for
> "MirrorMaker
> > > > > >>>>> Clusters" which didn't work and I found removing the
> "cluster."
> > > > from
> > > > > >>>>> the bootstrap-servers made it work a bit more, at least it
> > didn't
> > > > > >>>>> complain about not having any servers in the config.
> > > > > >>>>> So, I checked the "Running a dedicated MirrorMaker
> cluster"from
> > > the
> > > > > >>>>> KIP, which is basically more or less the same, but without
> the
> > > > > >>>>> "cluster." for the servers and it does at least start and it
> > > looks
> > > > > >>>>> like all the three MMs find each other, but no mirroring
> taking
> > > > > >>>>> place.
> > > > > >>>>>
> > > > > >>>>> Running the legacy-config from the old MM is working fine
> > though.
> > > > > >>>>> I'll
> > > > > >>>>> try to do some more digging today, so if you need some of
> those
> > > > very
> > > > > >>>>> verbose logs or something else just let me know. I am sure
> > that I
> > > > can
> > > > > >>>>> figure this out and just wanted to know if the documentation
> > will
> > > > get
> > > > > >>>>> extended as the new MM2 has a lot of features and is a bit
> more
> > > > > >>>>> complicated than the old one...
> > > > > >>>>>
> > > > > >>>>> Thanks
> > > > > >>>>>
> > > > > >>>>> Sebastian
> > > > > >>>>>
> > > > > >>>>> On 24-Dec-19 8:06 AM, Ryanne Dolan wrote:
> > > > > >>>>>> Hello Sebastian, please let us know what issues you are
> facing
> > > > > >>>>>> and we
> > > > > >>>>>> can
> > > > > >>>>>> probably help. Which config from the KIP are you
> referencing?
> > > > > >>>>>> Also check
> > > > > >>>>>> out the readme under ./connect/mirror for more examples.
> > > > > >>>>>>
> > > > > >>>>>> Ryanne
> > > > > >>>>>>
> > > > > >>>>>> On Mon, Dec 23, 2019, 12:58 PM Sebastian Schmitz <
> > > > > >>>>>> sebastian.schmitz@propellerhead.co.nz> wrote:
> > > > > >>>>>>
> > > > > >>>>>>> Hello,
> > > > > >>>>>>>
> > > > > >>>>>>> I'm currently trying to implement the new Kafka 2.4.0 and
> the
> > > > > >>>>>>> new MM2.
> > > > > >>>>>>>
> > > > > >>>>>>> However, it looks like the only documentation available is
> > the
> > > > > >>>>>>> KIP-382,
> > > > > >>>>>>> and the documentation
> > > > > >>>>>>> (
> > > https://kafka.apache.org/documentation/#basic_ops_mirror_maker)
> > > > > >>>>>>> for
> > > > > >>>>>>> the
> > > > > >>>>>>> MM isn't yet updated, and the documentation in the KIP
> seems
> > to
> > > > be
> > > > > >>>>>>> missing some stuff as I get a lot of errors and warning
> when
> > > > > >>>>>>> starting
> > > > > >>>>>>> the MM2 as connect-mirror, and it doesn't mirror, so I
> > probably
> > > > > >>>>>>> have
> > > > > >>>>>>> some mistakes in my configuration, but can't confirm this
> as
> > > > > >>>>>>> it's the
> > > > > >>>>>>> same as in the KIP.
> > > > > >>>>>>>
> > > > > >>>>>>> Any plans when the documentation will be updated?
> > > > > >>>>>>>
> > > > > >>>>>>> Thanks
> > > > > >>>>>>>
> > > > > >>>>>>> Sebastian
> > > > > >>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>> --
> > > > > >>>>>>> DISCLAIMER
> > > > > >>>>>>> This email contains information that is confidential and
> > which
> > > > > >>>>>>> may be
> > > > > >>>>>>> legally privileged. If you have received this email in
> error
> > > > please
> > > > > >>>>>>>
> > > > > >>>>>>> notify the sender immediately and delete the email.
> > > > > >>>>>>> This email is intended
> > > > > >>>>>>> solely for the use of the intended recipient and you may
> not
> > > use
> > > > or
> > > > > >>>>>>> disclose this email in any way.
> > > > > >>>>>>>
> > > > > >>>> --
> > > > > >>>> DISCLAIMER
> > > > > >>>> This email contains information that is confidential and which
> > > > > >>>> may be
> > > > > >>>> legally privileged. If you have received this email in error
> > > please
> > > > > >>>>
> > > > > >>>> notify the sender immediately and delete the email.
> > > > > >>>> This email is intended
> > > > > >>>> solely for the use of the intended recipient and you may not
> use
> > > or
> > > > > >>>> disclose this email in any way.
> > > > > >>>>
> > > > >
> > > > > --
> > > > > DISCLAIMER
> > > > > This email contains information that is confidential and which
> > > > > may be
> > > > > legally privileged. If you have received this email in error please
> > > > >
> > > > > notify the sender immediately and delete the email.
> > > > > This email is intended
> > > > > solely for the use of the intended recipient and you may not use or
> > > > > disclose this email in any way.
> > > > >
> > > >
> > >
> >
>
Re: Mirrormaker 2.0
Posted by Péter Sinóros-Szabó <pe...@transferwise.com.INVALID>.
Ryanne,
I didn't do much work yet, just checked the Interface to see if it is easy
to implement or not.
> The PR for LegacyReplicationPolicy should include any relevant fixes to
get it to run without crashing
Do you mean that there is already a PR for LegacyReplicationPolicy? If
there is, please link it here, I could not find it.
Thanks,
Peter
On Fri, 17 Jan 2020 at 20:58, Ryanne Dolan <ry...@gmail.com> wrote:
> Peter, KIP-382 includes LegacyReplicationPolicy for this purpose, but no,
> it has not been implemented yet. If you are interested in writing the PR,
> it would not require a separate KIP before merging. Looks like you are
> already doing the work :)
>
> It is possible, as you point out, that returning nulls like that will break
> parts of MM2. The PR for LegacyReplicationPolicy should include any
> relevant fixes to get it to run without crashing.
>
> Certainly some parts of MM2 will not work as intended when used with
> LegacyReplicationPolicy, e.g. MM2 would not be able to prevent cycles. But
> of course it should still _run_ and should work roughly the same as MM1.
>
> I'm happy to work with you on the PR if you are interested.
>
> Ryanne
>
> On Fri, Jan 17, 2020 at 9:34 AM Péter Sinóros-Szabó
> <pe...@transferwise.com.invalid> wrote:
>
> > Hi Sebastian & Ryanne,
> >
> > do you have maybe an implementation of this is just some ideas about how
> to
> > implement the policy that does not rename topics?
> > I am checking the ReplicationPolicy interface and don't really know what
> > the impact will be if I implement this:
> >
> > public String formatRemoteTopic(String sourceClusterAlias, String topic)
> {
> > return topic; }
> > public String topicSource(String topic) { return null; // well, I do not
> > really know if this were a mirrored topic or not }
> > public String upstreamTopic(String topic) { return topic; // well, I do
> > not really know if this were a mirrored topic or not}
> > public String originalTopic(String topic) { return topic; }
> >
> > Thanks,
> > Peter
> >
> > On Mon, 30 Dec 2019 at 06:57, Ryanne Dolan <ry...@gmail.com>
> wrote:
> >
> > > Sebastian, you can drop in a custom jar in the "Connect plug-in path"
> and
> > > MM2 will be able to load it. That enables you to implement your own
> > > ReplicationPolicy (and other pluggable interfaces) without compiling
> > > everything.
> > >
> > > In an upcoming release we'll have a "LegacyReplicationPolicy" that does
> > not
> > > rename topics. It's possible "SimpleReplicationPolicy" is a better
> name.
> > >
> > > Be advised that some features depend on correct ReplicationPolicy
> > > semantics, which LegacyReplicationPolicy will explicitly break. For
> > > example, MM2 cannot prevent cycles if topics are not renamed (or some
> > other
> > > similar mechanism is used).
> > >
> > > Ryanne
> > >
> > > On Sun, Dec 29, 2019, 7:41 PM Sebastian Schmitz <
> > > sebastian.schmitz@propellerhead.co.nz> wrote:
> > >
> > > > Hello,
> > > >
> > > > I found that it's using the DefaultReplicationPolicy that always
> > returns
> > > > "sourceClusterAlias + separator + topic" with only the separator
> being
> > > > configurable in the configuration-file with
> > REPLICATION_POLICY_SEPARATOR.
> > > >
> > > > It seems like I need a different ReplicationPolicy, like a
> > > > SimpleReplicationPolicy which always returns "topic" for the
> > > > formatRemoteTopic, then. But that would mean that I can't download
> the
> > > > Binaries and have to build the whole thing myself after adding the
> new
> > > > Policy-file!?
> > > > Or I could create a PR for a SimpleReplicationPolicy to be in some
> > > > future build...
> > > >
> > > > Any suggestions for this?
> > > >
> > > > Thanks
> > > >
> > > > Sebastian
> > > >
> > > >
> > > > On 30-Dec-19 1:39 PM, Sebastian Schmitz wrote:
> > > > > Hello,
> > > > >
> > > > > another thing I found and didn't find any configuration in the KIP
> > yet
> > > > > was that if I have two clusters (source and target) and a topic
> > > > > "replicateme" on the source-cluster it will get replicated to the
> > > > > target-cluster as "source.replicateme".
> > > > >
> > > > > How can I stop it from adding the cluster-name in front of the
> > > > > topic-name on target-cluster?
> > > > >
> > > > > Thanks
> > > > >
> > > > > Sebastian
> > > > >
> > > > > On 27-Dec-19 7:24 AM, Sebastian Schmitz wrote:
> > > > >> Hello Ryanne,
> > > > >>
> > > > >> Is there a way to prevent that from happening? We have two
> separate
> > > > >> clusters with some topics being replicated to the second one for
> > > > >> reporting. If we replicate everything again that reporting would
> > > > >> probably have some problems.
> > > > >>
> > > > >> Yes, I wondered when the Networking-guys would come and complain
> > > > >> about me using too much bandwidth on the VPN-Link ;)
> > > > >>
> > > > >> Thanks
> > > > >>
> > > > >> Sebastian
> > > > >>
> > > > >> On 24-Dec-19 1:11 PM, Ryanne Dolan wrote:
> > > > >>> Glad to hear you are replicating now :)
> > > > >>>
> > > > >>>> it probably started mirroring the last seven days as there was
> no
> > > > >>>> offset
> > > > >>> for the new consumer-group.
> > > > >>>
> > > > >>> That's correct -- MM2 will replicate the entire topic, as far
> back
> > > > >>> as the
> > > > >>> retention period. However, technically there are no consumer
> groups
> > > > >>> in MM2!
> > > > >>>
> > > > >>> 550MB/s in a test cluster sounds pretty good to me. Try
> increasing
> > > > >>> "tasks.max" and adding additional nodes.
> > > > >>>
> > > > >>> Ryanne
> > > > >>>
> > > > >>>
> > > > >>> On Mon, Dec 23, 2019 at 5:40 PM Sebastian Schmitz <
> > > > >>> sebastian.schmitz@propellerhead.co.nz> wrote:
> > > > >>>
> > > > >>>> Hello again!
> > > > >>>>
> > > > >>>> Some probably important configs I found out:
> > > > >>>>
> > > > >>>> We need this to enable mirroring as it seems to disabled by
> > default?
> > > > >>>>
> > > > >>>> source->target.enabled = true
> > > > >>>> target->source.enabled = true
> > > > >>>>
> > > > >>>> Also, the Client-IDs can be configured using:
> > > > >>>>
> > > > >>>> source.client.id = my_cool_id
> > > > >>>> target.client.id = my_cooler_id
> > > > >>>>
> > > > >>>> I configured them to include the ID of the server and the name
> of
> > > the
> > > > >>>> environment to have separate IDs per mirror-node.
> > > > >>>>
> > > > >>>> After adding these two, it looks a bit better than before, but
> > > > >>>> still not
> > > > >>>> satisfied as it started to mirror from my prod to test with
> > 550MB/s
> > > as
> > > > >>>> it probably started mirroring the last seven days as there was
> no
> > > > >>>> offset
> > > > >>>> for the new consumer-group. That's next on my list to solve.
> > > > >>>>
> > > > >>>> Best regards
> > > > >>>>
> > > > >>>> Sebastian
> > > > >>>>
> > > > >>>> On 24-Dec-19 8:34 AM, Sebastian Schmitz wrote:
> > > > >>>>> Hello,
> > > > >>>>>
> > > > >>>>> I tried running this connect-mirror-config:
> > > > >>>>>
> > > > >>>>> <snip>
> > > > >>>>> name = $MIRROR_NAME
> > > > >>>>> clusters = source, target
> > > > >>>>> source.bootstrap.servers = $SOURCE_SERVERS
> > > > >>>>> target.bootstrap.servers = $TARGET_SERVERS
> > > > >>>>> source->target.topics = $SOURCE_TARGET_TOPICS
> > > > >>>>> target->source.topics = $TARGET_SOURCE_TOPICS
> > > > >>>>> source->target.emit.heartbeats.enabled = true
> > > > >>>>> target->source.emit.heartbeats.enabled = true
> > > > >>>>> connector.class =
> > > > >>>>> org.apache.kafka.connect.mirror.MirrorSourceConnector
> > > > >>>>>
> > > > >>>>> # disable some new features
> > > > >>>>> refresh.topics.enabled = false
> > > > >>>>> refresh.groups.enabled = false
> > > > >>>>> emit.checkpoints.enables = true
> > > > >>>>> emit.heartbeats.enabled = true
> > > > >>>>> sync.topic.configs.enabled = false
> > > > >>>>> sync.topic.acls.enabled = false
> > > > >>>>> </snip>
> > > > >>>>>
> > > > >>>>> SOURCE_SERVERS and TARGET_SERVERS are a comma-separated list of
> > > three
> > > > >>>>> brokers with ports.
> > > > >>>>> The TOPICS are |-separated lists of topics.
> > > > >>>>>
> > > > >>>>> I get these warning during startup which is a bit weird as I
> > never
> > > > >>>>> supplied any of those settings, but maybe I should?
> > > > >>>>>
> > > > >>>>> [2019-12-23 00:36:25,918] WARN The configuration
> > > > >>>>> 'config.storage.topic' was supplied but isn't a known config.
> > > > >>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> > > > >>>>> [2019-12-23 00:36:25,918] WARN The configuration
> > > > >>>>> 'producer.bootstrap.servers' was supplied but isn't a known
> > config.
> > > > >>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> > > > >>>>> [2019-12-23 00:36:25,918] WARN The configuration 'group.id'
> was
> > > > >>>>> supplied but isn't a known config.
> > > > >>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> > > > >>>>> [2019-12-23 00:36:25,919] WARN The configuration
> > > > >>>>> 'status.storage.topic' was supplied but isn't a known config.
> > > > >>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> > > > >>>>> [2019-12-23 00:36:25,919] WARN The configuration
> > 'header.converter'
> > > > >>>>> was supplied but isn't a known config.
> > > > >>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> > > > >>>>> [2019-12-23 00:36:25,919] WARN The configuration
> > > > >>>>> 'consumer.bootstrap.servers' was supplied but isn't a known
> > config.
> > > > >>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> > > > >>>>> [2019-12-23 00:36:25,919] WARN The configuration
> > > > >>>>> 'offset.storage.topic' was supplied but isn't a known config.
> > > > >>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> > > > >>>>> [2019-12-23 00:36:25,919] WARN The configuration
> > 'value.converter'
> > > > >>>>> was
> > > > >>>>> supplied but isn't a known config.
> > > > >>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> > > > >>>>> [2019-12-23 00:36:25,919] WARN The configuration
> 'key.converter'
> > > was
> > > > >>>>> supplied but isn't a known config.
> > > > >>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> > > > >>>>> [2019-12-23 00:36:25,919] WARN The configuration
> > > > >>>>> 'admin.bootstrap.servers' was supplied but isn't a known
> config.
> > > > >>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> > > > >>>>>
> > > > >>>>> And this error:
> > > > >>>>>
> > > > >>>>> [2019-12-23 00:36:29,320] ERROR Plugin class loader for
> > connector:
> > > > >>>>> 'org.apache.kafka.connect.mirror.MirrorSourceConnector' was not
> > > > >>>>> found.
> > > > >>>>> Returning:
> > > > >>>>>
> > > >
> > org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader@5c316230
> > > > >>>>>
> > > > >>>>>
> > > >
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:165)
> > > > >>>>>
> > > > >>>>>
> > > > >>>>> First I tried the config mentioned in the KIP for "MirrorMaker
> > > > >>>>> Clusters" which didn't work and I found removing the "cluster."
> > > from
> > > > >>>>> the bootstrap-servers made it work a bit more, at least it
> didn't
> > > > >>>>> complain about not having any servers in the config.
> > > > >>>>> So, I checked the "Running a dedicated MirrorMaker cluster"from
> > the
> > > > >>>>> KIP, which is basically more or less the same, but without the
> > > > >>>>> "cluster." for the servers and it does at least start and it
> > looks
> > > > >>>>> like all the three MMs find each other, but no mirroring taking
> > > > >>>>> place.
> > > > >>>>>
> > > > >>>>> Running the legacy-config from the old MM is working fine
> though.
> > > > >>>>> I'll
> > > > >>>>> try to do some more digging today, so if you need some of those
> > > very
> > > > >>>>> verbose logs or something else just let me know. I am sure
> that I
> > > can
> > > > >>>>> figure this out and just wanted to know if the documentation
> will
> > > get
> > > > >>>>> extended as the new MM2 has a lot of features and is a bit more
> > > > >>>>> complicated than the old one...
> > > > >>>>>
> > > > >>>>> Thanks
> > > > >>>>>
> > > > >>>>> Sebastian
> > > > >>>>>
> > > > >>>>> On 24-Dec-19 8:06 AM, Ryanne Dolan wrote:
> > > > >>>>>> Hello Sebastian, please let us know what issues you are facing
> > > > >>>>>> and we
> > > > >>>>>> can
> > > > >>>>>> probably help. Which config from the KIP are you referencing?
> > > > >>>>>> Also check
> > > > >>>>>> out the readme under ./connect/mirror for more examples.
> > > > >>>>>>
> > > > >>>>>> Ryanne
> > > > >>>>>>
> > > > >>>>>> On Mon, Dec 23, 2019, 12:58 PM Sebastian Schmitz <
> > > > >>>>>> sebastian.schmitz@propellerhead.co.nz> wrote:
> > > > >>>>>>
> > > > >>>>>>> Hello,
> > > > >>>>>>>
> > > > >>>>>>> I'm currently trying to implement the new Kafka 2.4.0 and the
> > > > >>>>>>> new MM2.
> > > > >>>>>>>
> > > > >>>>>>> However, it looks like the only documentation available is
> the
> > > > >>>>>>> KIP-382,
> > > > >>>>>>> and the documentation
> > > > >>>>>>> (
> > https://kafka.apache.org/documentation/#basic_ops_mirror_maker)
> > > > >>>>>>> for
> > > > >>>>>>> the
> > > > >>>>>>> MM isn't yet updated, and the documentation in the KIP seems
> to
> > > be
> > > > >>>>>>> missing some stuff as I get a lot of errors and warning when
> > > > >>>>>>> starting
> > > > >>>>>>> the MM2 as connect-mirror, and it doesn't mirror, so I
> probably
> > > > >>>>>>> have
> > > > >>>>>>> some mistakes in my configuration, but can't confirm this as
> > > > >>>>>>> it's the
> > > > >>>>>>> same as in the KIP.
> > > > >>>>>>>
> > > > >>>>>>> Any plans when the documentation will be updated?
> > > > >>>>>>>
> > > > >>>>>>> Thanks
> > > > >>>>>>>
> > > > >>>>>>> Sebastian
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>> --
> > > > >>>>>>> DISCLAIMER
> > > > >>>>>>> This email contains information that is confidential and
> which
> > > > >>>>>>> may be
> > > > >>>>>>> legally privileged. If you have received this email in error
> > > please
> > > > >>>>>>>
> > > > >>>>>>> notify the sender immediately and delete the email.
> > > > >>>>>>> This email is intended
> > > > >>>>>>> solely for the use of the intended recipient and you may not
> > use
> > > or
> > > > >>>>>>> disclose this email in any way.
> > > > >>>>>>>
> > > > >>>> --
> > > > >>>> DISCLAIMER
> > > > >>>> This email contains information that is confidential and which
> > > > >>>> may be
> > > > >>>> legally privileged. If you have received this email in error
> > please
> > > > >>>>
> > > > >>>> notify the sender immediately and delete the email.
> > > > >>>> This email is intended
> > > > >>>> solely for the use of the intended recipient and you may not use
> > or
> > > > >>>> disclose this email in any way.
> > > > >>>>
> > > >
> > > > --
> > > > DISCLAIMER
> > > > This email contains information that is confidential and which
> > > > may be
> > > > legally privileged. If you have received this email in error please
> > > >
> > > > notify the sender immediately and delete the email.
> > > > This email is intended
> > > > solely for the use of the intended recipient and you may not use or
> > > > disclose this email in any way.
> > > >
> > >
> >
>
Re: Mirrormaker 2.0
Posted by Ryanne Dolan <ry...@gmail.com>.
Peter, KIP-382 includes LegacyReplicationPolicy for this purpose, but no,
it has not been implemented yet. If you are interested in writing the PR,
it would not require a separate KIP before merging. Looks like you are
already doing the work :)
It is possible, as you point out, that returning nulls like that will break
parts of MM2. The PR for LegacyReplicationPolicy should include any
relevant fixes to get it to run without crashing.
Certainly some parts of MM2 will not work as intended when used with
LegacyReplicationPolicy, e.g. MM2 would not be able to prevent cycles. But
of course it should still _run_ and should work roughly the same as MM1.
I'm happy to work with you on the PR if you are interested.
Ryanne
On Fri, Jan 17, 2020 at 9:34 AM Péter Sinóros-Szabó
<pe...@transferwise.com.invalid> wrote:
> Hi Sebastian & Ryanne,
>
> do you have maybe an implementation of this is just some ideas about how to
> implement the policy that does not rename topics?
> I am checking the ReplicationPolicy interface and don't really know what
> the impact will be if I implement this:
>
> public String formatRemoteTopic(String sourceClusterAlias, String topic) {
> return topic; }
> public String topicSource(String topic) { return null; // well, I do not
> really know if this were a mirrored topic or not }
> public String upstreamTopic(String topic) { return topic; // well, I do
> not really know if this were a mirrored topic or not}
> public String originalTopic(String topic) { return topic; }
>
> Thanks,
> Peter
>
> On Mon, 30 Dec 2019 at 06:57, Ryanne Dolan <ry...@gmail.com> wrote:
>
> > Sebastian, you can drop in a custom jar in the "Connect plug-in path" and
> > MM2 will be able to load it. That enables you to implement your own
> > ReplicationPolicy (and other pluggable interfaces) without compiling
> > everything.
> >
> > In an upcoming release we'll have a "LegacyReplicationPolicy" that does
> not
> > rename topics. It's possible "SimpleReplicationPolicy" is a better name.
> >
> > Be advised that some features depend on correct ReplicationPolicy
> > semantics, which LegacyReplicationPolicy will explicitly break. For
> > example, MM2 cannot prevent cycles if topics are not renamed (or some
> other
> > similar mechanism is used).
> >
> > Ryanne
> >
> > On Sun, Dec 29, 2019, 7:41 PM Sebastian Schmitz <
> > sebastian.schmitz@propellerhead.co.nz> wrote:
> >
> > > Hello,
> > >
> > > I found that it's using the DefaultReplicationPolicy that always
> returns
> > > "sourceClusterAlias + separator + topic" with only the separator being
> > > configurable in the configuration-file with
> REPLICATION_POLICY_SEPARATOR.
> > >
> > > It seems like I need a different ReplicationPolicy, like a
> > > SimpleReplicationPolicy which always returns "topic" for the
> > > formatRemoteTopic, then. But that would mean that I can't download the
> > > Binaries and have to build the whole thing myself after adding the new
> > > Policy-file!?
> > > Or I could create a PR for a SimpleReplicationPolicy to be in some
> > > future build...
> > >
> > > Any suggestions for this?
> > >
> > > Thanks
> > >
> > > Sebastian
> > >
> > >
> > > On 30-Dec-19 1:39 PM, Sebastian Schmitz wrote:
> > > > Hello,
> > > >
> > > > another thing I found and didn't find any configuration in the KIP
> yet
> > > > was that if I have two clusters (source and target) and a topic
> > > > "replicateme" on the source-cluster it will get replicated to the
> > > > target-cluster as "source.replicateme".
> > > >
> > > > How can I stop it from adding the cluster-name in front of the
> > > > topic-name on target-cluster?
> > > >
> > > > Thanks
> > > >
> > > > Sebastian
> > > >
> > > > On 27-Dec-19 7:24 AM, Sebastian Schmitz wrote:
> > > >> Hello Ryanne,
> > > >>
> > > >> Is there a way to prevent that from happening? We have two separate
> > > >> clusters with some topics being replicated to the second one for
> > > >> reporting. If we replicate everything again that reporting would
> > > >> probably have some problems.
> > > >>
> > > >> Yes, I wondered when the Networking-guys would come and complain
> > > >> about me using too much bandwidth on the VPN-Link ;)
> > > >>
> > > >> Thanks
> > > >>
> > > >> Sebastian
> > > >>
> > > >> On 24-Dec-19 1:11 PM, Ryanne Dolan wrote:
> > > >>> Glad to hear you are replicating now :)
> > > >>>
> > > >>>> it probably started mirroring the last seven days as there was no
> > > >>>> offset
> > > >>> for the new consumer-group.
> > > >>>
> > > >>> That's correct -- MM2 will replicate the entire topic, as far back
> > > >>> as the
> > > >>> retention period. However, technically there are no consumer groups
> > > >>> in MM2!
> > > >>>
> > > >>> 550MB/s in a test cluster sounds pretty good to me. Try increasing
> > > >>> "tasks.max" and adding additional nodes.
> > > >>>
> > > >>> Ryanne
> > > >>>
> > > >>>
> > > >>> On Mon, Dec 23, 2019 at 5:40 PM Sebastian Schmitz <
> > > >>> sebastian.schmitz@propellerhead.co.nz> wrote:
> > > >>>
> > > >>>> Hello again!
> > > >>>>
> > > >>>> Some probably important configs I found out:
> > > >>>>
> > > >>>> We need this to enable mirroring as it seems to disabled by
> default?
> > > >>>>
> > > >>>> source->target.enabled = true
> > > >>>> target->source.enabled = true
> > > >>>>
> > > >>>> Also, the Client-IDs can be configured using:
> > > >>>>
> > > >>>> source.client.id = my_cool_id
> > > >>>> target.client.id = my_cooler_id
> > > >>>>
> > > >>>> I configured them to include the ID of the server and the name of
> > the
> > > >>>> environment to have separate IDs per mirror-node.
> > > >>>>
> > > >>>> After adding these two, it looks a bit better than before, but
> > > >>>> still not
> > > >>>> satisfied as it started to mirror from my prod to test with
> 550MB/s
> > as
> > > >>>> it probably started mirroring the last seven days as there was no
> > > >>>> offset
> > > >>>> for the new consumer-group. That's next on my list to solve.
> > > >>>>
> > > >>>> Best regards
> > > >>>>
> > > >>>> Sebastian
> > > >>>>
> > > >>>> On 24-Dec-19 8:34 AM, Sebastian Schmitz wrote:
> > > >>>>> Hello,
> > > >>>>>
> > > >>>>> I tried running this connect-mirror-config:
> > > >>>>>
> > > >>>>> <snip>
> > > >>>>> name = $MIRROR_NAME
> > > >>>>> clusters = source, target
> > > >>>>> source.bootstrap.servers = $SOURCE_SERVERS
> > > >>>>> target.bootstrap.servers = $TARGET_SERVERS
> > > >>>>> source->target.topics = $SOURCE_TARGET_TOPICS
> > > >>>>> target->source.topics = $TARGET_SOURCE_TOPICS
> > > >>>>> source->target.emit.heartbeats.enabled = true
> > > >>>>> target->source.emit.heartbeats.enabled = true
> > > >>>>> connector.class =
> > > >>>>> org.apache.kafka.connect.mirror.MirrorSourceConnector
> > > >>>>>
> > > >>>>> # disable some new features
> > > >>>>> refresh.topics.enabled = false
> > > >>>>> refresh.groups.enabled = false
> > > >>>>> emit.checkpoints.enables = true
> > > >>>>> emit.heartbeats.enabled = true
> > > >>>>> sync.topic.configs.enabled = false
> > > >>>>> sync.topic.acls.enabled = false
> > > >>>>> </snip>
> > > >>>>>
> > > >>>>> SOURCE_SERVERS and TARGET_SERVERS are a comma-separated list of
> > three
> > > >>>>> brokers with ports.
> > > >>>>> The TOPICS are |-separated lists of topics.
> > > >>>>>
> > > >>>>> I get these warning during startup which is a bit weird as I
> never
> > > >>>>> supplied any of those settings, but maybe I should?
> > > >>>>>
> > > >>>>> [2019-12-23 00:36:25,918] WARN The configuration
> > > >>>>> 'config.storage.topic' was supplied but isn't a known config.
> > > >>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> > > >>>>> [2019-12-23 00:36:25,918] WARN The configuration
> > > >>>>> 'producer.bootstrap.servers' was supplied but isn't a known
> config.
> > > >>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> > > >>>>> [2019-12-23 00:36:25,918] WARN The configuration 'group.id' was
> > > >>>>> supplied but isn't a known config.
> > > >>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> > > >>>>> [2019-12-23 00:36:25,919] WARN The configuration
> > > >>>>> 'status.storage.topic' was supplied but isn't a known config.
> > > >>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> > > >>>>> [2019-12-23 00:36:25,919] WARN The configuration
> 'header.converter'
> > > >>>>> was supplied but isn't a known config.
> > > >>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> > > >>>>> [2019-12-23 00:36:25,919] WARN The configuration
> > > >>>>> 'consumer.bootstrap.servers' was supplied but isn't a known
> config.
> > > >>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> > > >>>>> [2019-12-23 00:36:25,919] WARN The configuration
> > > >>>>> 'offset.storage.topic' was supplied but isn't a known config.
> > > >>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> > > >>>>> [2019-12-23 00:36:25,919] WARN The configuration
> 'value.converter'
> > > >>>>> was
> > > >>>>> supplied but isn't a known config.
> > > >>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> > > >>>>> [2019-12-23 00:36:25,919] WARN The configuration 'key.converter'
> > was
> > > >>>>> supplied but isn't a known config.
> > > >>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> > > >>>>> [2019-12-23 00:36:25,919] WARN The configuration
> > > >>>>> 'admin.bootstrap.servers' was supplied but isn't a known config.
> > > >>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> > > >>>>>
> > > >>>>> And this error:
> > > >>>>>
> > > >>>>> [2019-12-23 00:36:29,320] ERROR Plugin class loader for
> connector:
> > > >>>>> 'org.apache.kafka.connect.mirror.MirrorSourceConnector' was not
> > > >>>>> found.
> > > >>>>> Returning:
> > > >>>>>
> > >
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader@5c316230
> > > >>>>>
> > > >>>>>
> > > (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:165)
> > > >>>>>
> > > >>>>>
> > > >>>>> First I tried the config mentioned in the KIP for "MirrorMaker
> > > >>>>> Clusters" which didn't work and I found removing the "cluster."
> > from
> > > >>>>> the bootstrap-servers made it work a bit more, at least it didn't
> > > >>>>> complain about not having any servers in the config.
> > > >>>>> So, I checked the "Running a dedicated MirrorMaker cluster"from
> the
> > > >>>>> KIP, which is basically more or less the same, but without the
> > > >>>>> "cluster." for the servers and it does at least start and it
> looks
> > > >>>>> like all the three MMs find each other, but no mirroring taking
> > > >>>>> place.
> > > >>>>>
> > > >>>>> Running the legacy-config from the old MM is working fine though.
> > > >>>>> I'll
> > > >>>>> try to do some more digging today, so if you need some of those
> > very
> > > >>>>> verbose logs or something else just let me know. I am sure that I
> > can
> > > >>>>> figure this out and just wanted to know if the documentation will
> > get
> > > >>>>> extended as the new MM2 has a lot of features and is a bit more
> > > >>>>> complicated than the old one...
> > > >>>>>
> > > >>>>> Thanks
> > > >>>>>
> > > >>>>> Sebastian
> > > >>>>>
> > > >>>>> On 24-Dec-19 8:06 AM, Ryanne Dolan wrote:
> > > >>>>>> Hello Sebastian, please let us know what issues you are facing
> > > >>>>>> and we
> > > >>>>>> can
> > > >>>>>> probably help. Which config from the KIP are you referencing?
> > > >>>>>> Also check
> > > >>>>>> out the readme under ./connect/mirror for more examples.
> > > >>>>>>
> > > >>>>>> Ryanne
> > > >>>>>>
> > > >>>>>> On Mon, Dec 23, 2019, 12:58 PM Sebastian Schmitz <
> > > >>>>>> sebastian.schmitz@propellerhead.co.nz> wrote:
> > > >>>>>>
> > > >>>>>>> Hello,
> > > >>>>>>>
> > > >>>>>>> I'm currently trying to implement the new Kafka 2.4.0 and the
> > > >>>>>>> new MM2.
> > > >>>>>>>
> > > >>>>>>> However, it looks like the only documentation available is the
> > > >>>>>>> KIP-382,
> > > >>>>>>> and the documentation
> > > >>>>>>> (
> https://kafka.apache.org/documentation/#basic_ops_mirror_maker)
> > > >>>>>>> for
> > > >>>>>>> the
> > > >>>>>>> MM isn't yet updated, and the documentation in the KIP seems to
> > be
> > > >>>>>>> missing some stuff as I get a lot of errors and warning when
> > > >>>>>>> starting
> > > >>>>>>> the MM2 as connect-mirror, and it doesn't mirror, so I probably
> > > >>>>>>> have
> > > >>>>>>> some mistakes in my configuration, but can't confirm this as
> > > >>>>>>> it's the
> > > >>>>>>> same as in the KIP.
> > > >>>>>>>
> > > >>>>>>> Any plans when the documentation will be updated?
> > > >>>>>>>
> > > >>>>>>> Thanks
> > > >>>>>>>
> > > >>>>>>> Sebastian
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> --
> > > >>>>>>> DISCLAIMER
> > > >>>>>>> This email contains information that is confidential and which
> > > >>>>>>> may be
> > > >>>>>>> legally privileged. If you have received this email in error
> > please
> > > >>>>>>>
> > > >>>>>>> notify the sender immediately and delete the email.
> > > >>>>>>> This email is intended
> > > >>>>>>> solely for the use of the intended recipient and you may not
> use
> > or
> > > >>>>>>> disclose this email in any way.
> > > >>>>>>>
> > > >>>> --
> > > >>>> DISCLAIMER
> > > >>>> This email contains information that is confidential and which
> > > >>>> may be
> > > >>>> legally privileged. If you have received this email in error
> please
> > > >>>>
> > > >>>> notify the sender immediately and delete the email.
> > > >>>> This email is intended
> > > >>>> solely for the use of the intended recipient and you may not use
> or
> > > >>>> disclose this email in any way.
> > > >>>>
> > >
> > > --
> > > DISCLAIMER
> > > This email contains information that is confidential and which
> > > may be
> > > legally privileged. If you have received this email in error please
> > >
> > > notify the sender immediately and delete the email.
> > > This email is intended
> > > solely for the use of the intended recipient and you may not use or
> > > disclose this email in any way.
> > >
> >
>
Re: Mirrormaker 2.0
Posted by Péter Sinóros-Szabó <pe...@transferwise.com.INVALID>.
Hi Sebastian & Ryanne,
do you have maybe an implementation of this is just some ideas about how to
implement the policy that does not rename topics?
I am checking the ReplicationPolicy interface and don't really know what
the impact will be if I implement this:
public String formatRemoteTopic(String sourceClusterAlias, String topic) {
return topic; }
public String topicSource(String topic) { return null; // well, I do not
really know if this were a mirrored topic or not }
public String upstreamTopic(String topic) { return topic; // well, I do
not really know if this were a mirrored topic or not}
public String originalTopic(String topic) { return topic; }
Thanks,
Peter
On Mon, 30 Dec 2019 at 06:57, Ryanne Dolan <ry...@gmail.com> wrote:
> Sebastian, you can drop in a custom jar in the "Connect plug-in path" and
> MM2 will be able to load it. That enables you to implement your own
> ReplicationPolicy (and other pluggable interfaces) without compiling
> everything.
>
> In an upcoming release we'll have a "LegacyReplicationPolicy" that does not
> rename topics. It's possible "SimpleReplicationPolicy" is a better name.
>
> Be advised that some features depend on correct ReplicationPolicy
> semantics, which LegacyReplicationPolicy will explicitly break. For
> example, MM2 cannot prevent cycles if topics are not renamed (or some other
> similar mechanism is used).
>
> Ryanne
>
> On Sun, Dec 29, 2019, 7:41 PM Sebastian Schmitz <
> sebastian.schmitz@propellerhead.co.nz> wrote:
>
> > Hello,
> >
> > I found that it's using the DefaultReplicationPolicy that always returns
> > "sourceClusterAlias + separator + topic" with only the separator being
> > configurable in the configuration-file with REPLICATION_POLICY_SEPARATOR.
> >
> > It seems like I need a different ReplicationPolicy, like a
> > SimpleReplicationPolicy which always returns "topic" for the
> > formatRemoteTopic, then. But that would mean that I can't download the
> > Binaries and have to build the whole thing myself after adding the new
> > Policy-file!?
> > Or I could create a PR for a SimpleReplicationPolicy to be in some
> > future build...
> >
> > Any suggestions for this?
> >
> > Thanks
> >
> > Sebastian
> >
> >
> > On 30-Dec-19 1:39 PM, Sebastian Schmitz wrote:
> > > Hello,
> > >
> > > another thing I found and didn't find any configuration in the KIP yet
> > > was that if I have two clusters (source and target) and a topic
> > > "replicateme" on the source-cluster it will get replicated to the
> > > target-cluster as "source.replicateme".
> > >
> > > How can I stop it from adding the cluster-name in front of the
> > > topic-name on target-cluster?
> > >
> > > Thanks
> > >
> > > Sebastian
> > >
> > > On 27-Dec-19 7:24 AM, Sebastian Schmitz wrote:
> > >> Hello Ryanne,
> > >>
> > >> Is there a way to prevent that from happening? We have two separate
> > >> clusters with some topics being replicated to the second one for
> > >> reporting. If we replicate everything again that reporting would
> > >> probably have some problems.
> > >>
> > >> Yes, I wondered when the Networking-guys would come and complain
> > >> about me using too much bandwidth on the VPN-Link ;)
> > >>
> > >> Thanks
> > >>
> > >> Sebastian
> > >>
> > >> On 24-Dec-19 1:11 PM, Ryanne Dolan wrote:
> > >>> Glad to hear you are replicating now :)
> > >>>
> > >>>> it probably started mirroring the last seven days as there was no
> > >>>> offset
> > >>> for the new consumer-group.
> > >>>
> > >>> That's correct -- MM2 will replicate the entire topic, as far back
> > >>> as the
> > >>> retention period. However, technically there are no consumer groups
> > >>> in MM2!
> > >>>
> > >>> 550MB/s in a test cluster sounds pretty good to me. Try increasing
> > >>> "tasks.max" and adding additional nodes.
> > >>>
> > >>> Ryanne
> > >>>
> > >>>
> > >>> On Mon, Dec 23, 2019 at 5:40 PM Sebastian Schmitz <
> > >>> sebastian.schmitz@propellerhead.co.nz> wrote:
> > >>>
> > >>>> Hello again!
> > >>>>
> > >>>> Some probably important configs I found out:
> > >>>>
> > >>>> We need this to enable mirroring as it seems to disabled by default?
> > >>>>
> > >>>> source->target.enabled = true
> > >>>> target->source.enabled = true
> > >>>>
> > >>>> Also, the Client-IDs can be configured using:
> > >>>>
> > >>>> source.client.id = my_cool_id
> > >>>> target.client.id = my_cooler_id
> > >>>>
> > >>>> I configured them to include the ID of the server and the name of
> the
> > >>>> environment to have separate IDs per mirror-node.
> > >>>>
> > >>>> After adding these two, it looks a bit better than before, but
> > >>>> still not
> > >>>> satisfied as it started to mirror from my prod to test with 550MB/s
> as
> > >>>> it probably started mirroring the last seven days as there was no
> > >>>> offset
> > >>>> for the new consumer-group. That's next on my list to solve.
> > >>>>
> > >>>> Best regards
> > >>>>
> > >>>> Sebastian
> > >>>>
> > >>>> On 24-Dec-19 8:34 AM, Sebastian Schmitz wrote:
> > >>>>> Hello,
> > >>>>>
> > >>>>> I tried running this connect-mirror-config:
> > >>>>>
> > >>>>> <snip>
> > >>>>> name = $MIRROR_NAME
> > >>>>> clusters = source, target
> > >>>>> source.bootstrap.servers = $SOURCE_SERVERS
> > >>>>> target.bootstrap.servers = $TARGET_SERVERS
> > >>>>> source->target.topics = $SOURCE_TARGET_TOPICS
> > >>>>> target->source.topics = $TARGET_SOURCE_TOPICS
> > >>>>> source->target.emit.heartbeats.enabled = true
> > >>>>> target->source.emit.heartbeats.enabled = true
> > >>>>> connector.class =
> > >>>>> org.apache.kafka.connect.mirror.MirrorSourceConnector
> > >>>>>
> > >>>>> # disable some new features
> > >>>>> refresh.topics.enabled = false
> > >>>>> refresh.groups.enabled = false
> > >>>>> emit.checkpoints.enables = true
> > >>>>> emit.heartbeats.enabled = true
> > >>>>> sync.topic.configs.enabled = false
> > >>>>> sync.topic.acls.enabled = false
> > >>>>> </snip>
> > >>>>>
> > >>>>> SOURCE_SERVERS and TARGET_SERVERS are a comma-separated list of
> three
> > >>>>> brokers with ports.
> > >>>>> The TOPICS are |-separated lists of topics.
> > >>>>>
> > >>>>> I get these warning during startup which is a bit weird as I never
> > >>>>> supplied any of those settings, but maybe I should?
> > >>>>>
> > >>>>> [2019-12-23 00:36:25,918] WARN The configuration
> > >>>>> 'config.storage.topic' was supplied but isn't a known config.
> > >>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> > >>>>> [2019-12-23 00:36:25,918] WARN The configuration
> > >>>>> 'producer.bootstrap.servers' was supplied but isn't a known config.
> > >>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> > >>>>> [2019-12-23 00:36:25,918] WARN The configuration 'group.id' was
> > >>>>> supplied but isn't a known config.
> > >>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> > >>>>> [2019-12-23 00:36:25,919] WARN The configuration
> > >>>>> 'status.storage.topic' was supplied but isn't a known config.
> > >>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> > >>>>> [2019-12-23 00:36:25,919] WARN The configuration 'header.converter'
> > >>>>> was supplied but isn't a known config.
> > >>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> > >>>>> [2019-12-23 00:36:25,919] WARN The configuration
> > >>>>> 'consumer.bootstrap.servers' was supplied but isn't a known config.
> > >>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> > >>>>> [2019-12-23 00:36:25,919] WARN The configuration
> > >>>>> 'offset.storage.topic' was supplied but isn't a known config.
> > >>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> > >>>>> [2019-12-23 00:36:25,919] WARN The configuration 'value.converter'
> > >>>>> was
> > >>>>> supplied but isn't a known config.
> > >>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> > >>>>> [2019-12-23 00:36:25,919] WARN The configuration 'key.converter'
> was
> > >>>>> supplied but isn't a known config.
> > >>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> > >>>>> [2019-12-23 00:36:25,919] WARN The configuration
> > >>>>> 'admin.bootstrap.servers' was supplied but isn't a known config.
> > >>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> > >>>>>
> > >>>>> And this error:
> > >>>>>
> > >>>>> [2019-12-23 00:36:29,320] ERROR Plugin class loader for connector:
> > >>>>> 'org.apache.kafka.connect.mirror.MirrorSourceConnector' was not
> > >>>>> found.
> > >>>>> Returning:
> > >>>>>
> > org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader@5c316230
> > >>>>>
> > >>>>>
> > (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:165)
> > >>>>>
> > >>>>>
> > >>>>> First I tried the config mentioned in the KIP for "MirrorMaker
> > >>>>> Clusters" which didn't work and I found removing the "cluster."
> from
> > >>>>> the bootstrap-servers made it work a bit more, at least it didn't
> > >>>>> complain about not having any servers in the config.
> > >>>>> So, I checked the "Running a dedicated MirrorMaker cluster"from the
> > >>>>> KIP, which is basically more or less the same, but without the
> > >>>>> "cluster." for the servers and it does at least start and it looks
> > >>>>> like all the three MMs find each other, but no mirroring taking
> > >>>>> place.
> > >>>>>
> > >>>>> Running the legacy-config from the old MM is working fine though.
> > >>>>> I'll
> > >>>>> try to do some more digging today, so if you need some of those
> very
> > >>>>> verbose logs or something else just let me know. I am sure that I
> can
> > >>>>> figure this out and just wanted to know if the documentation will
> get
> > >>>>> extended as the new MM2 has a lot of features and is a bit more
> > >>>>> complicated than the old one...
> > >>>>>
> > >>>>> Thanks
> > >>>>>
> > >>>>> Sebastian
> > >>>>>
> > >>>>> On 24-Dec-19 8:06 AM, Ryanne Dolan wrote:
> > >>>>>> Hello Sebastian, please let us know what issues you are facing
> > >>>>>> and we
> > >>>>>> can
> > >>>>>> probably help. Which config from the KIP are you referencing?
> > >>>>>> Also check
> > >>>>>> out the readme under ./connect/mirror for more examples.
> > >>>>>>
> > >>>>>> Ryanne
> > >>>>>>
> > >>>>>> On Mon, Dec 23, 2019, 12:58 PM Sebastian Schmitz <
> > >>>>>> sebastian.schmitz@propellerhead.co.nz> wrote:
> > >>>>>>
> > >>>>>>> Hello,
> > >>>>>>>
> > >>>>>>> I'm currently trying to implement the new Kafka 2.4.0 and the
> > >>>>>>> new MM2.
> > >>>>>>>
> > >>>>>>> However, it looks like the only documentation available is the
> > >>>>>>> KIP-382,
> > >>>>>>> and the documentation
> > >>>>>>> (https://kafka.apache.org/documentation/#basic_ops_mirror_maker)
> > >>>>>>> for
> > >>>>>>> the
> > >>>>>>> MM isn't yet updated, and the documentation in the KIP seems to
> be
> > >>>>>>> missing some stuff as I get a lot of errors and warning when
> > >>>>>>> starting
> > >>>>>>> the MM2 as connect-mirror, and it doesn't mirror, so I probably
> > >>>>>>> have
> > >>>>>>> some mistakes in my configuration, but can't confirm this as
> > >>>>>>> it's the
> > >>>>>>> same as in the KIP.
> > >>>>>>>
> > >>>>>>> Any plans when the documentation will be updated?
> > >>>>>>>
> > >>>>>>> Thanks
> > >>>>>>>
> > >>>>>>> Sebastian
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> --
> > >>>>>>> DISCLAIMER
> > >>>>>>> This email contains information that is confidential and which
> > >>>>>>> may be
> > >>>>>>> legally privileged. If you have received this email in error
> please
> > >>>>>>>
> > >>>>>>> notify the sender immediately and delete the email.
> > >>>>>>> This email is intended
> > >>>>>>> solely for the use of the intended recipient and you may not use
> or
> > >>>>>>> disclose this email in any way.
> > >>>>>>>
> > >>>> --
> > >>>> DISCLAIMER
> > >>>> This email contains information that is confidential and which
> > >>>> may be
> > >>>> legally privileged. If you have received this email in error please
> > >>>>
> > >>>> notify the sender immediately and delete the email.
> > >>>> This email is intended
> > >>>> solely for the use of the intended recipient and you may not use or
> > >>>> disclose this email in any way.
> > >>>>
> >
> > --
> > DISCLAIMER
> > This email contains information that is confidential and which
> > may be
> > legally privileged. If you have received this email in error please
> >
> > notify the sender immediately and delete the email.
> > This email is intended
> > solely for the use of the intended recipient and you may not use or
> > disclose this email in any way.
> >
>
Re: Mirrormaker 2.0
Posted by Ryanne Dolan <ry...@gmail.com>.
Sebastian, you can drop in a custom jar in the "Connect plug-in path" and
MM2 will be able to load it. That enables you to implement your own
ReplicationPolicy (and other pluggable interfaces) without compiling
everything.
In an upcoming release we'll have a "LegacyReplicationPolicy" that does not
rename topics. It's possible "SimpleReplicationPolicy" is a better name.
Be advised that some features depend on correct ReplicationPolicy
semantics, which LegacyReplicationPolicy will explicitly break. For
example, MM2 cannot prevent cycles if topics are not renamed (or some other
similar mechanism is used).
Ryanne
On Sun, Dec 29, 2019, 7:41 PM Sebastian Schmitz <
sebastian.schmitz@propellerhead.co.nz> wrote:
> Hello,
>
> I found that it's using the DefaultReplicationPolicy that always returns
> "sourceClusterAlias + separator + topic" with only the separator being
> configurable in the configuration-file with REPLICATION_POLICY_SEPARATOR.
>
> It seems like I need a different ReplicationPolicy, like a
> SimpleReplicationPolicy which always returns "topic" for the
> formatRemoteTopic, then. But that would mean that I can't download the
> Binaries and have to build the whole thing myself after adding the new
> Policy-file!?
> Or I could create a PR for a SimpleReplicationPolicy to be in some
> future build...
>
> Any suggestions for this?
>
> Thanks
>
> Sebastian
>
>
> On 30-Dec-19 1:39 PM, Sebastian Schmitz wrote:
> > Hello,
> >
> > another thing I found and didn't find any configuration in the KIP yet
> > was that if I have two clusters (source and target) and a topic
> > "replicateme" on the source-cluster it will get replicated to the
> > target-cluster as "source.replicateme".
> >
> > How can I stop it from adding the cluster-name in front of the
> > topic-name on target-cluster?
> >
> > Thanks
> >
> > Sebastian
> >
> > On 27-Dec-19 7:24 AM, Sebastian Schmitz wrote:
> >> Hello Ryanne,
> >>
> >> Is there a way to prevent that from happening? We have two separate
> >> clusters with some topics being replicated to the second one for
> >> reporting. If we replicate everything again that reporting would
> >> probably have some problems.
> >>
> >> Yes, I wondered when the Networking-guys would come and complain
> >> about me using too much bandwidth on the VPN-Link ;)
> >>
> >> Thanks
> >>
> >> Sebastian
> >>
> >> On 24-Dec-19 1:11 PM, Ryanne Dolan wrote:
> >>> Glad to hear you are replicating now :)
> >>>
> >>>> it probably started mirroring the last seven days as there was no
> >>>> offset
> >>> for the new consumer-group.
> >>>
> >>> That's correct -- MM2 will replicate the entire topic, as far back
> >>> as the
> >>> retention period. However, technically there are no consumer groups
> >>> in MM2!
> >>>
> >>> 550MB/s in a test cluster sounds pretty good to me. Try increasing
> >>> "tasks.max" and adding additional nodes.
> >>>
> >>> Ryanne
> >>>
> >>>
> >>> On Mon, Dec 23, 2019 at 5:40 PM Sebastian Schmitz <
> >>> sebastian.schmitz@propellerhead.co.nz> wrote:
> >>>
> >>>> Hello again!
> >>>>
> >>>> Some probably important configs I found out:
> >>>>
> >>>> We need this to enable mirroring as it seems to disabled by default?
> >>>>
> >>>> source->target.enabled = true
> >>>> target->source.enabled = true
> >>>>
> >>>> Also, the Client-IDs can be configured using:
> >>>>
> >>>> source.client.id = my_cool_id
> >>>> target.client.id = my_cooler_id
> >>>>
> >>>> I configured them to include the ID of the server and the name of the
> >>>> environment to have separate IDs per mirror-node.
> >>>>
> >>>> After adding these two, it looks a bit better than before, but
> >>>> still not
> >>>> satisfied as it started to mirror from my prod to test with 550MB/s as
> >>>> it probably started mirroring the last seven days as there was no
> >>>> offset
> >>>> for the new consumer-group. That's next on my list to solve.
> >>>>
> >>>> Best regards
> >>>>
> >>>> Sebastian
> >>>>
> >>>> On 24-Dec-19 8:34 AM, Sebastian Schmitz wrote:
> >>>>> Hello,
> >>>>>
> >>>>> I tried running this connect-mirror-config:
> >>>>>
> >>>>> <snip>
> >>>>> name = $MIRROR_NAME
> >>>>> clusters = source, target
> >>>>> source.bootstrap.servers = $SOURCE_SERVERS
> >>>>> target.bootstrap.servers = $TARGET_SERVERS
> >>>>> source->target.topics = $SOURCE_TARGET_TOPICS
> >>>>> target->source.topics = $TARGET_SOURCE_TOPICS
> >>>>> source->target.emit.heartbeats.enabled = true
> >>>>> target->source.emit.heartbeats.enabled = true
> >>>>> connector.class =
> >>>>> org.apache.kafka.connect.mirror.MirrorSourceConnector
> >>>>>
> >>>>> # disable some new features
> >>>>> refresh.topics.enabled = false
> >>>>> refresh.groups.enabled = false
> >>>>> emit.checkpoints.enables = true
> >>>>> emit.heartbeats.enabled = true
> >>>>> sync.topic.configs.enabled = false
> >>>>> sync.topic.acls.enabled = false
> >>>>> </snip>
> >>>>>
> >>>>> SOURCE_SERVERS and TARGET_SERVERS are a comma-separated list of three
> >>>>> brokers with ports.
> >>>>> The TOPICS are |-separated lists of topics.
> >>>>>
> >>>>> I get these warning during startup which is a bit weird as I never
> >>>>> supplied any of those settings, but maybe I should?
> >>>>>
> >>>>> [2019-12-23 00:36:25,918] WARN The configuration
> >>>>> 'config.storage.topic' was supplied but isn't a known config.
> >>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> >>>>> [2019-12-23 00:36:25,918] WARN The configuration
> >>>>> 'producer.bootstrap.servers' was supplied but isn't a known config.
> >>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> >>>>> [2019-12-23 00:36:25,918] WARN The configuration 'group.id' was
> >>>>> supplied but isn't a known config.
> >>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> >>>>> [2019-12-23 00:36:25,919] WARN The configuration
> >>>>> 'status.storage.topic' was supplied but isn't a known config.
> >>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> >>>>> [2019-12-23 00:36:25,919] WARN The configuration 'header.converter'
> >>>>> was supplied but isn't a known config.
> >>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> >>>>> [2019-12-23 00:36:25,919] WARN The configuration
> >>>>> 'consumer.bootstrap.servers' was supplied but isn't a known config.
> >>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> >>>>> [2019-12-23 00:36:25,919] WARN The configuration
> >>>>> 'offset.storage.topic' was supplied but isn't a known config.
> >>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> >>>>> [2019-12-23 00:36:25,919] WARN The configuration 'value.converter'
> >>>>> was
> >>>>> supplied but isn't a known config.
> >>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> >>>>> [2019-12-23 00:36:25,919] WARN The configuration 'key.converter' was
> >>>>> supplied but isn't a known config.
> >>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> >>>>> [2019-12-23 00:36:25,919] WARN The configuration
> >>>>> 'admin.bootstrap.servers' was supplied but isn't a known config.
> >>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
> >>>>>
> >>>>> And this error:
> >>>>>
> >>>>> [2019-12-23 00:36:29,320] ERROR Plugin class loader for connector:
> >>>>> 'org.apache.kafka.connect.mirror.MirrorSourceConnector' was not
> >>>>> found.
> >>>>> Returning:
> >>>>>
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader@5c316230
> >>>>>
> >>>>>
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:165)
> >>>>>
> >>>>>
> >>>>> First I tried the config mentioned in the KIP for "MirrorMaker
> >>>>> Clusters" which didn't work and I found removing the "cluster." from
> >>>>> the bootstrap-servers made it work a bit more, at least it didn't
> >>>>> complain about not having any servers in the config.
> >>>>> So, I checked the "Running a dedicated MirrorMaker cluster"from the
> >>>>> KIP, which is basically more or less the same, but without the
> >>>>> "cluster." for the servers and it does at least start and it looks
> >>>>> like all the three MMs find each other, but no mirroring taking
> >>>>> place.
> >>>>>
> >>>>> Running the legacy-config from the old MM is working fine though.
> >>>>> I'll
> >>>>> try to do some more digging today, so if you need some of those very
> >>>>> verbose logs or something else just let me know. I am sure that I can
> >>>>> figure this out and just wanted to know if the documentation will get
> >>>>> extended as the new MM2 has a lot of features and is a bit more
> >>>>> complicated than the old one...
> >>>>>
> >>>>> Thanks
> >>>>>
> >>>>> Sebastian
> >>>>>
> >>>>> On 24-Dec-19 8:06 AM, Ryanne Dolan wrote:
> >>>>>> Hello Sebastian, please let us know what issues you are facing
> >>>>>> and we
> >>>>>> can
> >>>>>> probably help. Which config from the KIP are you referencing?
> >>>>>> Also check
> >>>>>> out the readme under ./connect/mirror for more examples.
> >>>>>>
> >>>>>> Ryanne
> >>>>>>
> >>>>>> On Mon, Dec 23, 2019, 12:58 PM Sebastian Schmitz <
> >>>>>> sebastian.schmitz@propellerhead.co.nz> wrote:
> >>>>>>
> >>>>>>> Hello,
> >>>>>>>
> >>>>>>> I'm currently trying to implement the new Kafka 2.4.0 and the
> >>>>>>> new MM2.
> >>>>>>>
> >>>>>>> However, it looks like the only documentation available is the
> >>>>>>> KIP-382,
> >>>>>>> and the documentation
> >>>>>>> (https://kafka.apache.org/documentation/#basic_ops_mirror_maker)
> >>>>>>> for
> >>>>>>> the
> >>>>>>> MM isn't yet updated, and the documentation in the KIP seems to be
> >>>>>>> missing some stuff as I get a lot of errors and warning when
> >>>>>>> starting
> >>>>>>> the MM2 as connect-mirror, and it doesn't mirror, so I probably
> >>>>>>> have
> >>>>>>> some mistakes in my configuration, but can't confirm this as
> >>>>>>> it's the
> >>>>>>> same as in the KIP.
> >>>>>>>
> >>>>>>> Any plans when the documentation will be updated?
> >>>>>>>
> >>>>>>> Thanks
> >>>>>>>
> >>>>>>> Sebastian
> >>>>>>>
> >>>>>>>
> >>>>>>> --
> >>>>>>> DISCLAIMER
> >>>>>>> This email contains information that is confidential and which
> >>>>>>> may be
> >>>>>>> legally privileged. If you have received this email in error please
> >>>>>>>
> >>>>>>> notify the sender immediately and delete the email.
> >>>>>>> This email is intended
> >>>>>>> solely for the use of the intended recipient and you may not use or
> >>>>>>> disclose this email in any way.
> >>>>>>>
> >>>> --
> >>>> DISCLAIMER
> >>>> This email contains information that is confidential and which
> >>>> may be
> >>>> legally privileged. If you have received this email in error please
> >>>>
> >>>> notify the sender immediately and delete the email.
> >>>> This email is intended
> >>>> solely for the use of the intended recipient and you may not use or
> >>>> disclose this email in any way.
> >>>>
>
> --
> DISCLAIMER
> This email contains information that is confidential and which
> may be
> legally privileged. If you have received this email in error please
>
> notify the sender immediately and delete the email.
> This email is intended
> solely for the use of the intended recipient and you may not use or
> disclose this email in any way.
>
Re: Mirrormaker 2.0
Posted by Sebastian Schmitz <se...@propellerhead.co.nz>.
Hello,
I found that it's using the DefaultReplicationPolicy that always returns
"sourceClusterAlias + separator + topic" with only the separator being
configurable in the configuration-file with REPLICATION_POLICY_SEPARATOR.
It seems like I need a different ReplicationPolicy, like a
SimpleReplicationPolicy which always returns "topic" for the
formatRemoteTopic, then. But that would mean that I can't download the
Binaries and have to build the whole thing myself after adding the new
Policy-file!?
Or I could create a PR for a SimpleReplicationPolicy to be in some
future build...
Any suggestions for this?
Thanks
Sebastian
On 30-Dec-19 1:39 PM, Sebastian Schmitz wrote:
> Hello,
>
> another thing I found and didn't find any configuration in the KIP yet
> was that if I have two clusters (source and target) and a topic
> "replicateme" on the source-cluster it will get replicated to the
> target-cluster as "source.replicateme".
>
> How can I stop it from adding the cluster-name in front of the
> topic-name on target-cluster?
>
> Thanks
>
> Sebastian
>
> On 27-Dec-19 7:24 AM, Sebastian Schmitz wrote:
>> Hello Ryanne,
>>
>> Is there a way to prevent that from happening? We have two separate
>> clusters with some topics being replicated to the second one for
>> reporting. If we replicate everything again that reporting would
>> probably have some problems.
>>
>> Yes, I wondered when the Networking-guys would come and complain
>> about me using too much bandwidth on the VPN-Link ;)
>>
>> Thanks
>>
>> Sebastian
>>
>> On 24-Dec-19 1:11 PM, Ryanne Dolan wrote:
>>> Glad to hear you are replicating now :)
>>>
>>>> it probably started mirroring the last seven days as there was no
>>>> offset
>>> for the new consumer-group.
>>>
>>> That's correct -- MM2 will replicate the entire topic, as far back
>>> as the
>>> retention period. However, technically there are no consumer groups
>>> in MM2!
>>>
>>> 550MB/s in a test cluster sounds pretty good to me. Try increasing
>>> "tasks.max" and adding additional nodes.
>>>
>>> Ryanne
>>>
>>>
>>> On Mon, Dec 23, 2019 at 5:40 PM Sebastian Schmitz <
>>> sebastian.schmitz@propellerhead.co.nz> wrote:
>>>
>>>> Hello again!
>>>>
>>>> Some probably important configs I found out:
>>>>
>>>> We need this to enable mirroring as it seems to disabled by default?
>>>>
>>>> source->target.enabled = true
>>>> target->source.enabled = true
>>>>
>>>> Also, the Client-IDs can be configured using:
>>>>
>>>> source.client.id = my_cool_id
>>>> target.client.id = my_cooler_id
>>>>
>>>> I configured them to include the ID of the server and the name of the
>>>> environment to have separate IDs per mirror-node.
>>>>
>>>> After adding these two, it looks a bit better than before, but
>>>> still not
>>>> satisfied as it started to mirror from my prod to test with 550MB/s as
>>>> it probably started mirroring the last seven days as there was no
>>>> offset
>>>> for the new consumer-group. That's next on my list to solve.
>>>>
>>>> Best regards
>>>>
>>>> Sebastian
>>>>
>>>> On 24-Dec-19 8:34 AM, Sebastian Schmitz wrote:
>>>>> Hello,
>>>>>
>>>>> I tried running this connect-mirror-config:
>>>>>
>>>>> <snip>
>>>>> name = $MIRROR_NAME
>>>>> clusters = source, target
>>>>> source.bootstrap.servers = $SOURCE_SERVERS
>>>>> target.bootstrap.servers = $TARGET_SERVERS
>>>>> source->target.topics = $SOURCE_TARGET_TOPICS
>>>>> target->source.topics = $TARGET_SOURCE_TOPICS
>>>>> source->target.emit.heartbeats.enabled = true
>>>>> target->source.emit.heartbeats.enabled = true
>>>>> connector.class =
>>>>> org.apache.kafka.connect.mirror.MirrorSourceConnector
>>>>>
>>>>> # disable some new features
>>>>> refresh.topics.enabled = false
>>>>> refresh.groups.enabled = false
>>>>> emit.checkpoints.enables = true
>>>>> emit.heartbeats.enabled = true
>>>>> sync.topic.configs.enabled = false
>>>>> sync.topic.acls.enabled = false
>>>>> </snip>
>>>>>
>>>>> SOURCE_SERVERS and TARGET_SERVERS are a comma-separated list of three
>>>>> brokers with ports.
>>>>> The TOPICS are |-separated lists of topics.
>>>>>
>>>>> I get these warning during startup which is a bit weird as I never
>>>>> supplied any of those settings, but maybe I should?
>>>>>
>>>>> [2019-12-23 00:36:25,918] WARN The configuration
>>>>> 'config.storage.topic' was supplied but isn't a known config.
>>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
>>>>> [2019-12-23 00:36:25,918] WARN The configuration
>>>>> 'producer.bootstrap.servers' was supplied but isn't a known config.
>>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
>>>>> [2019-12-23 00:36:25,918] WARN The configuration 'group.id' was
>>>>> supplied but isn't a known config.
>>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
>>>>> [2019-12-23 00:36:25,919] WARN The configuration
>>>>> 'status.storage.topic' was supplied but isn't a known config.
>>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
>>>>> [2019-12-23 00:36:25,919] WARN The configuration 'header.converter'
>>>>> was supplied but isn't a known config.
>>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
>>>>> [2019-12-23 00:36:25,919] WARN The configuration
>>>>> 'consumer.bootstrap.servers' was supplied but isn't a known config.
>>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
>>>>> [2019-12-23 00:36:25,919] WARN The configuration
>>>>> 'offset.storage.topic' was supplied but isn't a known config.
>>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
>>>>> [2019-12-23 00:36:25,919] WARN The configuration 'value.converter'
>>>>> was
>>>>> supplied but isn't a known config.
>>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
>>>>> [2019-12-23 00:36:25,919] WARN The configuration 'key.converter' was
>>>>> supplied but isn't a known config.
>>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
>>>>> [2019-12-23 00:36:25,919] WARN The configuration
>>>>> 'admin.bootstrap.servers' was supplied but isn't a known config.
>>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
>>>>>
>>>>> And this error:
>>>>>
>>>>> [2019-12-23 00:36:29,320] ERROR Plugin class loader for connector:
>>>>> 'org.apache.kafka.connect.mirror.MirrorSourceConnector' was not
>>>>> found.
>>>>> Returning:
>>>>> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader@5c316230
>>>>>
>>>>> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:165)
>>>>>
>>>>>
>>>>> First I tried the config mentioned in the KIP for "MirrorMaker
>>>>> Clusters" which didn't work and I found removing the "cluster." from
>>>>> the bootstrap-servers made it work a bit more, at least it didn't
>>>>> complain about not having any servers in the config.
>>>>> So, I checked the "Running a dedicated MirrorMaker cluster"from the
>>>>> KIP, which is basically more or less the same, but without the
>>>>> "cluster." for the servers and it does at least start and it looks
>>>>> like all the three MMs find each other, but no mirroring taking
>>>>> place.
>>>>>
>>>>> Running the legacy-config from the old MM is working fine though.
>>>>> I'll
>>>>> try to do some more digging today, so if you need some of those very
>>>>> verbose logs or something else just let me know. I am sure that I can
>>>>> figure this out and just wanted to know if the documentation will get
>>>>> extended as the new MM2 has a lot of features and is a bit more
>>>>> complicated than the old one...
>>>>>
>>>>> Thanks
>>>>>
>>>>> Sebastian
>>>>>
>>>>> On 24-Dec-19 8:06 AM, Ryanne Dolan wrote:
>>>>>> Hello Sebastian, please let us know what issues you are facing
>>>>>> and we
>>>>>> can
>>>>>> probably help. Which config from the KIP are you referencing?
>>>>>> Also check
>>>>>> out the readme under ./connect/mirror for more examples.
>>>>>>
>>>>>> Ryanne
>>>>>>
>>>>>> On Mon, Dec 23, 2019, 12:58 PM Sebastian Schmitz <
>>>>>> sebastian.schmitz@propellerhead.co.nz> wrote:
>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> I'm currently trying to implement the new Kafka 2.4.0 and the
>>>>>>> new MM2.
>>>>>>>
>>>>>>> However, it looks like the only documentation available is the
>>>>>>> KIP-382,
>>>>>>> and the documentation
>>>>>>> (https://kafka.apache.org/documentation/#basic_ops_mirror_maker)
>>>>>>> for
>>>>>>> the
>>>>>>> MM isn't yet updated, and the documentation in the KIP seems to be
>>>>>>> missing some stuff as I get a lot of errors and warning when
>>>>>>> starting
>>>>>>> the MM2 as connect-mirror, and it doesn't mirror, so I probably
>>>>>>> have
>>>>>>> some mistakes in my configuration, but can't confirm this as
>>>>>>> it's the
>>>>>>> same as in the KIP.
>>>>>>>
>>>>>>> Any plans when the documentation will be updated?
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>> Sebastian
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> DISCLAIMER
>>>>>>> This email contains information that is confidential and which
>>>>>>> may be
>>>>>>> legally privileged. If you have received this email in error please
>>>>>>>
>>>>>>> notify the sender immediately and delete the email.
>>>>>>> This email is intended
>>>>>>> solely for the use of the intended recipient and you may not use or
>>>>>>> disclose this email in any way.
>>>>>>>
>>>> --
>>>> DISCLAIMER
>>>> This email contains information that is confidential and which
>>>> may be
>>>> legally privileged. If you have received this email in error please
>>>>
>>>> notify the sender immediately and delete the email.
>>>> This email is intended
>>>> solely for the use of the intended recipient and you may not use or
>>>> disclose this email in any way.
>>>>
--
DISCLAIMER
This email contains information that is confidential and which
may be
legally privileged. If you have received this email in error please
notify the sender immediately and delete the email.
This email is intended
solely for the use of the intended recipient and you may not use or
disclose this email in any way.
Re: Mirrormaker 2.0
Posted by Sebastian Schmitz <se...@propellerhead.co.nz>.
Hello,
another thing I found and didn't find any configuration in the KIP yet
was that if I have two clusters (source and target) and a topic
"replicateme" on the source-cluster it will get replicated to the
target-cluster as "source.replicateme".
How can I stop it from adding the cluster-name in front of the
topic-name on target-cluster?
Thanks
Sebastian
On 27-Dec-19 7:24 AM, Sebastian Schmitz wrote:
> Hello Ryanne,
>
> Is there a way to prevent that from happening? We have two separate
> clusters with some topics being replicated to the second one for
> reporting. If we replicate everything again that reporting would
> probably have some problems.
>
> Yes, I wondered when the Networking-guys would come and complain about
> me using too much bandwidth on the VPN-Link ;)
>
> Thanks
>
> Sebastian
>
> On 24-Dec-19 1:11 PM, Ryanne Dolan wrote:
>> Glad to hear you are replicating now :)
>>
>>> it probably started mirroring the last seven days as there was no
>>> offset
>> for the new consumer-group.
>>
>> That's correct -- MM2 will replicate the entire topic, as far back as
>> the
>> retention period. However, technically there are no consumer groups
>> in MM2!
>>
>> 550MB/s in a test cluster sounds pretty good to me. Try increasing
>> "tasks.max" and adding additional nodes.
>>
>> Ryanne
>>
>>
>> On Mon, Dec 23, 2019 at 5:40 PM Sebastian Schmitz <
>> sebastian.schmitz@propellerhead.co.nz> wrote:
>>
>>> Hello again!
>>>
>>> Some probably important configs I found out:
>>>
>>> We need this to enable mirroring as it seems to disabled by default?
>>>
>>> source->target.enabled = true
>>> target->source.enabled = true
>>>
>>> Also, the Client-IDs can be configured using:
>>>
>>> source.client.id = my_cool_id
>>> target.client.id = my_cooler_id
>>>
>>> I configured them to include the ID of the server and the name of the
>>> environment to have separate IDs per mirror-node.
>>>
>>> After adding these two, it looks a bit better than before, but still
>>> not
>>> satisfied as it started to mirror from my prod to test with 550MB/s as
>>> it probably started mirroring the last seven days as there was no
>>> offset
>>> for the new consumer-group. That's next on my list to solve.
>>>
>>> Best regards
>>>
>>> Sebastian
>>>
>>> On 24-Dec-19 8:34 AM, Sebastian Schmitz wrote:
>>>> Hello,
>>>>
>>>> I tried running this connect-mirror-config:
>>>>
>>>> <snip>
>>>> name = $MIRROR_NAME
>>>> clusters = source, target
>>>> source.bootstrap.servers = $SOURCE_SERVERS
>>>> target.bootstrap.servers = $TARGET_SERVERS
>>>> source->target.topics = $SOURCE_TARGET_TOPICS
>>>> target->source.topics = $TARGET_SOURCE_TOPICS
>>>> source->target.emit.heartbeats.enabled = true
>>>> target->source.emit.heartbeats.enabled = true
>>>> connector.class =
>>>> org.apache.kafka.connect.mirror.MirrorSourceConnector
>>>>
>>>> # disable some new features
>>>> refresh.topics.enabled = false
>>>> refresh.groups.enabled = false
>>>> emit.checkpoints.enables = true
>>>> emit.heartbeats.enabled = true
>>>> sync.topic.configs.enabled = false
>>>> sync.topic.acls.enabled = false
>>>> </snip>
>>>>
>>>> SOURCE_SERVERS and TARGET_SERVERS are a comma-separated list of three
>>>> brokers with ports.
>>>> The TOPICS are |-separated lists of topics.
>>>>
>>>> I get these warning during startup which is a bit weird as I never
>>>> supplied any of those settings, but maybe I should?
>>>>
>>>> [2019-12-23 00:36:25,918] WARN The configuration
>>>> 'config.storage.topic' was supplied but isn't a known config.
>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
>>>> [2019-12-23 00:36:25,918] WARN The configuration
>>>> 'producer.bootstrap.servers' was supplied but isn't a known config.
>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
>>>> [2019-12-23 00:36:25,918] WARN The configuration 'group.id' was
>>>> supplied but isn't a known config.
>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
>>>> [2019-12-23 00:36:25,919] WARN The configuration
>>>> 'status.storage.topic' was supplied but isn't a known config.
>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
>>>> [2019-12-23 00:36:25,919] WARN The configuration 'header.converter'
>>>> was supplied but isn't a known config.
>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
>>>> [2019-12-23 00:36:25,919] WARN The configuration
>>>> 'consumer.bootstrap.servers' was supplied but isn't a known config.
>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
>>>> [2019-12-23 00:36:25,919] WARN The configuration
>>>> 'offset.storage.topic' was supplied but isn't a known config.
>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
>>>> [2019-12-23 00:36:25,919] WARN The configuration 'value.converter' was
>>>> supplied but isn't a known config.
>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
>>>> [2019-12-23 00:36:25,919] WARN The configuration 'key.converter' was
>>>> supplied but isn't a known config.
>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
>>>> [2019-12-23 00:36:25,919] WARN The configuration
>>>> 'admin.bootstrap.servers' was supplied but isn't a known config.
>>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
>>>>
>>>> And this error:
>>>>
>>>> [2019-12-23 00:36:29,320] ERROR Plugin class loader for connector:
>>>> 'org.apache.kafka.connect.mirror.MirrorSourceConnector' was not found.
>>>> Returning:
>>>> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader@5c316230
>>>>
>>>> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:165)
>>>>
>>>> First I tried the config mentioned in the KIP for "MirrorMaker
>>>> Clusters" which didn't work and I found removing the "cluster." from
>>>> the bootstrap-servers made it work a bit more, at least it didn't
>>>> complain about not having any servers in the config.
>>>> So, I checked the "Running a dedicated MirrorMaker cluster"from the
>>>> KIP, which is basically more or less the same, but without the
>>>> "cluster." for the servers and it does at least start and it looks
>>>> like all the three MMs find each other, but no mirroring taking place.
>>>>
>>>> Running the legacy-config from the old MM is working fine though. I'll
>>>> try to do some more digging today, so if you need some of those very
>>>> verbose logs or something else just let me know. I am sure that I can
>>>> figure this out and just wanted to know if the documentation will get
>>>> extended as the new MM2 has a lot of features and is a bit more
>>>> complicated than the old one...
>>>>
>>>> Thanks
>>>>
>>>> Sebastian
>>>>
>>>> On 24-Dec-19 8:06 AM, Ryanne Dolan wrote:
>>>>> Hello Sebastian, please let us know what issues you are facing and we
>>>>> can
>>>>> probably help. Which config from the KIP are you referencing? Also
>>>>> check
>>>>> out the readme under ./connect/mirror for more examples.
>>>>>
>>>>> Ryanne
>>>>>
>>>>> On Mon, Dec 23, 2019, 12:58 PM Sebastian Schmitz <
>>>>> sebastian.schmitz@propellerhead.co.nz> wrote:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> I'm currently trying to implement the new Kafka 2.4.0 and the new
>>>>>> MM2.
>>>>>>
>>>>>> However, it looks like the only documentation available is the
>>>>>> KIP-382,
>>>>>> and the documentation
>>>>>> (https://kafka.apache.org/documentation/#basic_ops_mirror_maker) for
>>>>>> the
>>>>>> MM isn't yet updated, and the documentation in the KIP seems to be
>>>>>> missing some stuff as I get a lot of errors and warning when
>>>>>> starting
>>>>>> the MM2 as connect-mirror, and it doesn't mirror, so I probably have
>>>>>> some mistakes in my configuration, but can't confirm this as it's
>>>>>> the
>>>>>> same as in the KIP.
>>>>>>
>>>>>> Any plans when the documentation will be updated?
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> Sebastian
>>>>>>
>>>>>>
>>>>>> --
>>>>>> DISCLAIMER
>>>>>> This email contains information that is confidential and which
>>>>>> may be
>>>>>> legally privileged. If you have received this email in error please
>>>>>>
>>>>>> notify the sender immediately and delete the email.
>>>>>> This email is intended
>>>>>> solely for the use of the intended recipient and you may not use or
>>>>>> disclose this email in any way.
>>>>>>
>>> --
>>> DISCLAIMER
>>> This email contains information that is confidential and which
>>> may be
>>> legally privileged. If you have received this email in error please
>>>
>>> notify the sender immediately and delete the email.
>>> This email is intended
>>> solely for the use of the intended recipient and you may not use or
>>> disclose this email in any way.
>>>
--
DISCLAIMER
This email contains information that is confidential and which
may be
legally privileged. If you have received this email in error please
notify the sender immediately and delete the email.
This email is intended
solely for the use of the intended recipient and you may not use or
disclose this email in any way.
Re: Mirrormaker 2.0
Posted by Sebastian Schmitz <se...@propellerhead.co.nz>.
Hello Ryanne,
Is there a way to prevent that from happening? We have two separate
clusters with some topics being replicated to the second one for
reporting. If we replicate everything again that reporting would
probably have some problems.
Yes, I wondered when the Networking-guys would come and complain about
me using too much bandwidth on the VPN-Link ;)
Thanks
Sebastian
On 24-Dec-19 1:11 PM, Ryanne Dolan wrote:
> Glad to hear you are replicating now :)
>
>> it probably started mirroring the last seven days as there was no offset
> for the new consumer-group.
>
> That's correct -- MM2 will replicate the entire topic, as far back as the
> retention period. However, technically there are no consumer groups in MM2!
>
> 550MB/s in a test cluster sounds pretty good to me. Try increasing
> "tasks.max" and adding additional nodes.
>
> Ryanne
>
>
> On Mon, Dec 23, 2019 at 5:40 PM Sebastian Schmitz <
> sebastian.schmitz@propellerhead.co.nz> wrote:
>
>> Hello again!
>>
>> Some probably important configs I found out:
>>
>> We need this to enable mirroring as it seems to disabled by default?
>>
>> source->target.enabled = true
>> target->source.enabled = true
>>
>> Also, the Client-IDs can be configured using:
>>
>> source.client.id = my_cool_id
>> target.client.id = my_cooler_id
>>
>> I configured them to include the ID of the server and the name of the
>> environment to have separate IDs per mirror-node.
>>
>> After adding these two, it looks a bit better than before, but still not
>> satisfied as it started to mirror from my prod to test with 550MB/s as
>> it probably started mirroring the last seven days as there was no offset
>> for the new consumer-group. That's next on my list to solve.
>>
>> Best regards
>>
>> Sebastian
>>
>> On 24-Dec-19 8:34 AM, Sebastian Schmitz wrote:
>>> Hello,
>>>
>>> I tried running this connect-mirror-config:
>>>
>>> <snip>
>>> name = $MIRROR_NAME
>>> clusters = source, target
>>> source.bootstrap.servers = $SOURCE_SERVERS
>>> target.bootstrap.servers = $TARGET_SERVERS
>>> source->target.topics = $SOURCE_TARGET_TOPICS
>>> target->source.topics = $TARGET_SOURCE_TOPICS
>>> source->target.emit.heartbeats.enabled = true
>>> target->source.emit.heartbeats.enabled = true
>>> connector.class = org.apache.kafka.connect.mirror.MirrorSourceConnector
>>>
>>> # disable some new features
>>> refresh.topics.enabled = false
>>> refresh.groups.enabled = false
>>> emit.checkpoints.enables = true
>>> emit.heartbeats.enabled = true
>>> sync.topic.configs.enabled = false
>>> sync.topic.acls.enabled = false
>>> </snip>
>>>
>>> SOURCE_SERVERS and TARGET_SERVERS are a comma-separated list of three
>>> brokers with ports.
>>> The TOPICS are |-separated lists of topics.
>>>
>>> I get these warning during startup which is a bit weird as I never
>>> supplied any of those settings, but maybe I should?
>>>
>>> [2019-12-23 00:36:25,918] WARN The configuration
>>> 'config.storage.topic' was supplied but isn't a known config.
>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
>>> [2019-12-23 00:36:25,918] WARN The configuration
>>> 'producer.bootstrap.servers' was supplied but isn't a known config.
>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
>>> [2019-12-23 00:36:25,918] WARN The configuration 'group.id' was
>>> supplied but isn't a known config.
>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
>>> [2019-12-23 00:36:25,919] WARN The configuration
>>> 'status.storage.topic' was supplied but isn't a known config.
>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
>>> [2019-12-23 00:36:25,919] WARN The configuration 'header.converter'
>>> was supplied but isn't a known config.
>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
>>> [2019-12-23 00:36:25,919] WARN The configuration
>>> 'consumer.bootstrap.servers' was supplied but isn't a known config.
>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
>>> [2019-12-23 00:36:25,919] WARN The configuration
>>> 'offset.storage.topic' was supplied but isn't a known config.
>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
>>> [2019-12-23 00:36:25,919] WARN The configuration 'value.converter' was
>>> supplied but isn't a known config.
>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
>>> [2019-12-23 00:36:25,919] WARN The configuration 'key.converter' was
>>> supplied but isn't a known config.
>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
>>> [2019-12-23 00:36:25,919] WARN The configuration
>>> 'admin.bootstrap.servers' was supplied but isn't a known config.
>>> (org.apache.kafka.clients.producer.ProducerConfig:355)
>>>
>>> And this error:
>>>
>>> [2019-12-23 00:36:29,320] ERROR Plugin class loader for connector:
>>> 'org.apache.kafka.connect.mirror.MirrorSourceConnector' was not found.
>>> Returning:
>>> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader@5c316230
>>> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:165)
>>>
>>> First I tried the config mentioned in the KIP for "MirrorMaker
>>> Clusters" which didn't work and I found removing the "cluster." from
>>> the bootstrap-servers made it work a bit more, at least it didn't
>>> complain about not having any servers in the config.
>>> So, I checked the "Running a dedicated MirrorMaker cluster"from the
>>> KIP, which is basically more or less the same, but without the
>>> "cluster." for the servers and it does at least start and it looks
>>> like all the three MMs find each other, but no mirroring taking place.
>>>
>>> Running the legacy-config from the old MM is working fine though. I'll
>>> try to do some more digging today, so if you need some of those very
>>> verbose logs or something else just let me know. I am sure that I can
>>> figure this out and just wanted to know if the documentation will get
>>> extended as the new MM2 has a lot of features and is a bit more
>>> complicated than the old one...
>>>
>>> Thanks
>>>
>>> Sebastian
>>>
>>> On 24-Dec-19 8:06 AM, Ryanne Dolan wrote:
>>>> Hello Sebastian, please let us know what issues you are facing and we
>>>> can
>>>> probably help. Which config from the KIP are you referencing? Also check
>>>> out the readme under ./connect/mirror for more examples.
>>>>
>>>> Ryanne
>>>>
>>>> On Mon, Dec 23, 2019, 12:58 PM Sebastian Schmitz <
>>>> sebastian.schmitz@propellerhead.co.nz> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> I'm currently trying to implement the new Kafka 2.4.0 and the new MM2.
>>>>>
>>>>> However, it looks like the only documentation available is the KIP-382,
>>>>> and the documentation
>>>>> (https://kafka.apache.org/documentation/#basic_ops_mirror_maker) for
>>>>> the
>>>>> MM isn't yet updated, and the documentation in the KIP seems to be
>>>>> missing some stuff as I get a lot of errors and warning when starting
>>>>> the MM2 as connect-mirror, and it doesn't mirror, so I probably have
>>>>> some mistakes in my configuration, but can't confirm this as it's the
>>>>> same as in the KIP.
>>>>>
>>>>> Any plans when the documentation will be updated?
>>>>>
>>>>> Thanks
>>>>>
>>>>> Sebastian
>>>>>
>>>>>
>>>>> --
>>>>> DISCLAIMER
>>>>> This email contains information that is confidential and which
>>>>> may be
>>>>> legally privileged. If you have received this email in error please
>>>>>
>>>>> notify the sender immediately and delete the email.
>>>>> This email is intended
>>>>> solely for the use of the intended recipient and you may not use or
>>>>> disclose this email in any way.
>>>>>
>> --
>> DISCLAIMER
>> This email contains information that is confidential and which
>> may be
>> legally privileged. If you have received this email in error please
>>
>> notify the sender immediately and delete the email.
>> This email is intended
>> solely for the use of the intended recipient and you may not use or
>> disclose this email in any way.
>>
--
DISCLAIMER
This email contains information that is confidential and which
may be
legally privileged. If you have received this email in error please
notify the sender immediately and delete the email.
This email is intended
solely for the use of the intended recipient and you may not use or
disclose this email in any way.
Re: Mirrormaker 2.0
Posted by Ryanne Dolan <ry...@gmail.com>.
Glad to hear you are replicating now :)
> it probably started mirroring the last seven days as there was no offset
for the new consumer-group.
That's correct -- MM2 will replicate the entire topic, as far back as the
retention period. However, technically there are no consumer groups in MM2!
550MB/s in a test cluster sounds pretty good to me. Try increasing
"tasks.max" and adding additional nodes.
Ryanne
On Mon, Dec 23, 2019 at 5:40 PM Sebastian Schmitz <
sebastian.schmitz@propellerhead.co.nz> wrote:
> Hello again!
>
> Some probably important configs I found out:
>
> We need this to enable mirroring as it seems to disabled by default?
>
> source->target.enabled = true
> target->source.enabled = true
>
> Also, the Client-IDs can be configured using:
>
> source.client.id = my_cool_id
> target.client.id = my_cooler_id
>
> I configured them to include the ID of the server and the name of the
> environment to have separate IDs per mirror-node.
>
> After adding these two, it looks a bit better than before, but still not
> satisfied as it started to mirror from my prod to test with 550MB/s as
> it probably started mirroring the last seven days as there was no offset
> for the new consumer-group. That's next on my list to solve.
>
> Best regards
>
> Sebastian
>
> On 24-Dec-19 8:34 AM, Sebastian Schmitz wrote:
> > Hello,
> >
> > I tried running this connect-mirror-config:
> >
> > <snip>
> > name = $MIRROR_NAME
> > clusters = source, target
> > source.bootstrap.servers = $SOURCE_SERVERS
> > target.bootstrap.servers = $TARGET_SERVERS
> > source->target.topics = $SOURCE_TARGET_TOPICS
> > target->source.topics = $TARGET_SOURCE_TOPICS
> > source->target.emit.heartbeats.enabled = true
> > target->source.emit.heartbeats.enabled = true
> > connector.class = org.apache.kafka.connect.mirror.MirrorSourceConnector
> >
> > # disable some new features
> > refresh.topics.enabled = false
> > refresh.groups.enabled = false
> > emit.checkpoints.enables = true
> > emit.heartbeats.enabled = true
> > sync.topic.configs.enabled = false
> > sync.topic.acls.enabled = false
> > </snip>
> >
> > SOURCE_SERVERS and TARGET_SERVERS are a comma-separated list of three
> > brokers with ports.
> > The TOPICS are |-separated lists of topics.
> >
> > I get these warning during startup which is a bit weird as I never
> > supplied any of those settings, but maybe I should?
> >
> > [2019-12-23 00:36:25,918] WARN The configuration
> > 'config.storage.topic' was supplied but isn't a known config.
> > (org.apache.kafka.clients.producer.ProducerConfig:355)
> > [2019-12-23 00:36:25,918] WARN The configuration
> > 'producer.bootstrap.servers' was supplied but isn't a known config.
> > (org.apache.kafka.clients.producer.ProducerConfig:355)
> > [2019-12-23 00:36:25,918] WARN The configuration 'group.id' was
> > supplied but isn't a known config.
> > (org.apache.kafka.clients.producer.ProducerConfig:355)
> > [2019-12-23 00:36:25,919] WARN The configuration
> > 'status.storage.topic' was supplied but isn't a known config.
> > (org.apache.kafka.clients.producer.ProducerConfig:355)
> > [2019-12-23 00:36:25,919] WARN The configuration 'header.converter'
> > was supplied but isn't a known config.
> > (org.apache.kafka.clients.producer.ProducerConfig:355)
> > [2019-12-23 00:36:25,919] WARN The configuration
> > 'consumer.bootstrap.servers' was supplied but isn't a known config.
> > (org.apache.kafka.clients.producer.ProducerConfig:355)
> > [2019-12-23 00:36:25,919] WARN The configuration
> > 'offset.storage.topic' was supplied but isn't a known config.
> > (org.apache.kafka.clients.producer.ProducerConfig:355)
> > [2019-12-23 00:36:25,919] WARN The configuration 'value.converter' was
> > supplied but isn't a known config.
> > (org.apache.kafka.clients.producer.ProducerConfig:355)
> > [2019-12-23 00:36:25,919] WARN The configuration 'key.converter' was
> > supplied but isn't a known config.
> > (org.apache.kafka.clients.producer.ProducerConfig:355)
> > [2019-12-23 00:36:25,919] WARN The configuration
> > 'admin.bootstrap.servers' was supplied but isn't a known config.
> > (org.apache.kafka.clients.producer.ProducerConfig:355)
> >
> > And this error:
> >
> > [2019-12-23 00:36:29,320] ERROR Plugin class loader for connector:
> > 'org.apache.kafka.connect.mirror.MirrorSourceConnector' was not found.
> > Returning:
> > org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader@5c316230
> > (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:165)
> >
> > First I tried the config mentioned in the KIP for "MirrorMaker
> > Clusters" which didn't work and I found removing the "cluster." from
> > the bootstrap-servers made it work a bit more, at least it didn't
> > complain about not having any servers in the config.
> > So, I checked the "Running a dedicated MirrorMaker cluster"from the
> > KIP, which is basically more or less the same, but without the
> > "cluster." for the servers and it does at least start and it looks
> > like all the three MMs find each other, but no mirroring taking place.
> >
> > Running the legacy-config from the old MM is working fine though. I'll
> > try to do some more digging today, so if you need some of those very
> > verbose logs or something else just let me know. I am sure that I can
> > figure this out and just wanted to know if the documentation will get
> > extended as the new MM2 has a lot of features and is a bit more
> > complicated than the old one...
> >
> > Thanks
> >
> > Sebastian
> >
> > On 24-Dec-19 8:06 AM, Ryanne Dolan wrote:
> >> Hello Sebastian, please let us know what issues you are facing and we
> >> can
> >> probably help. Which config from the KIP are you referencing? Also check
> >> out the readme under ./connect/mirror for more examples.
> >>
> >> Ryanne
> >>
> >> On Mon, Dec 23, 2019, 12:58 PM Sebastian Schmitz <
> >> sebastian.schmitz@propellerhead.co.nz> wrote:
> >>
> >>> Hello,
> >>>
> >>> I'm currently trying to implement the new Kafka 2.4.0 and the new MM2.
> >>>
> >>> However, it looks like the only documentation available is the KIP-382,
> >>> and the documentation
> >>> (https://kafka.apache.org/documentation/#basic_ops_mirror_maker) for
> >>> the
> >>> MM isn't yet updated, and the documentation in the KIP seems to be
> >>> missing some stuff as I get a lot of errors and warning when starting
> >>> the MM2 as connect-mirror, and it doesn't mirror, so I probably have
> >>> some mistakes in my configuration, but can't confirm this as it's the
> >>> same as in the KIP.
> >>>
> >>> Any plans when the documentation will be updated?
> >>>
> >>> Thanks
> >>>
> >>> Sebastian
> >>>
> >>>
> >>> --
> >>> DISCLAIMER
> >>> This email contains information that is confidential and which
> >>> may be
> >>> legally privileged. If you have received this email in error please
> >>>
> >>> notify the sender immediately and delete the email.
> >>> This email is intended
> >>> solely for the use of the intended recipient and you may not use or
> >>> disclose this email in any way.
> >>>
>
> --
> DISCLAIMER
> This email contains information that is confidential and which
> may be
> legally privileged. If you have received this email in error please
>
> notify the sender immediately and delete the email.
> This email is intended
> solely for the use of the intended recipient and you may not use or
> disclose this email in any way.
>
Re: Mirrormaker 2.0
Posted by Sebastian Schmitz <se...@propellerhead.co.nz>.
Hello again!
Some probably important configs I found out:
We need this to enable mirroring as it seems to disabled by default?
source->target.enabled = true
target->source.enabled = true
Also, the Client-IDs can be configured using:
source.client.id = my_cool_id
target.client.id = my_cooler_id
I configured them to include the ID of the server and the name of the
environment to have separate IDs per mirror-node.
After adding these two, it looks a bit better than before, but still not
satisfied as it started to mirror from my prod to test with 550MB/s as
it probably started mirroring the last seven days as there was no offset
for the new consumer-group. That's next on my list to solve.
Best regards
Sebastian
On 24-Dec-19 8:34 AM, Sebastian Schmitz wrote:
> Hello,
>
> I tried running this connect-mirror-config:
>
> <snip>
> name = $MIRROR_NAME
> clusters = source, target
> source.bootstrap.servers = $SOURCE_SERVERS
> target.bootstrap.servers = $TARGET_SERVERS
> source->target.topics = $SOURCE_TARGET_TOPICS
> target->source.topics = $TARGET_SOURCE_TOPICS
> source->target.emit.heartbeats.enabled = true
> target->source.emit.heartbeats.enabled = true
> connector.class = org.apache.kafka.connect.mirror.MirrorSourceConnector
>
> # disable some new features
> refresh.topics.enabled = false
> refresh.groups.enabled = false
> emit.checkpoints.enables = true
> emit.heartbeats.enabled = true
> sync.topic.configs.enabled = false
> sync.topic.acls.enabled = false
> </snip>
>
> SOURCE_SERVERS and TARGET_SERVERS are a comma-separated list of three
> brokers with ports.
> The TOPICS are |-separated lists of topics.
>
> I get these warning during startup which is a bit weird as I never
> supplied any of those settings, but maybe I should?
>
> [2019-12-23 00:36:25,918] WARN The configuration
> 'config.storage.topic' was supplied but isn't a known config.
> (org.apache.kafka.clients.producer.ProducerConfig:355)
> [2019-12-23 00:36:25,918] WARN The configuration
> 'producer.bootstrap.servers' was supplied but isn't a known config.
> (org.apache.kafka.clients.producer.ProducerConfig:355)
> [2019-12-23 00:36:25,918] WARN The configuration 'group.id' was
> supplied but isn't a known config.
> (org.apache.kafka.clients.producer.ProducerConfig:355)
> [2019-12-23 00:36:25,919] WARN The configuration
> 'status.storage.topic' was supplied but isn't a known config.
> (org.apache.kafka.clients.producer.ProducerConfig:355)
> [2019-12-23 00:36:25,919] WARN The configuration 'header.converter'
> was supplied but isn't a known config.
> (org.apache.kafka.clients.producer.ProducerConfig:355)
> [2019-12-23 00:36:25,919] WARN The configuration
> 'consumer.bootstrap.servers' was supplied but isn't a known config.
> (org.apache.kafka.clients.producer.ProducerConfig:355)
> [2019-12-23 00:36:25,919] WARN The configuration
> 'offset.storage.topic' was supplied but isn't a known config.
> (org.apache.kafka.clients.producer.ProducerConfig:355)
> [2019-12-23 00:36:25,919] WARN The configuration 'value.converter' was
> supplied but isn't a known config.
> (org.apache.kafka.clients.producer.ProducerConfig:355)
> [2019-12-23 00:36:25,919] WARN The configuration 'key.converter' was
> supplied but isn't a known config.
> (org.apache.kafka.clients.producer.ProducerConfig:355)
> [2019-12-23 00:36:25,919] WARN The configuration
> 'admin.bootstrap.servers' was supplied but isn't a known config.
> (org.apache.kafka.clients.producer.ProducerConfig:355)
>
> And this error:
>
> [2019-12-23 00:36:29,320] ERROR Plugin class loader for connector:
> 'org.apache.kafka.connect.mirror.MirrorSourceConnector' was not found.
> Returning:
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader@5c316230
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:165)
>
> First I tried the config mentioned in the KIP for "MirrorMaker
> Clusters" which didn't work and I found removing the "cluster." from
> the bootstrap-servers made it work a bit more, at least it didn't
> complain about not having any servers in the config.
> So, I checked the "Running a dedicated MirrorMaker cluster"from the
> KIP, which is basically more or less the same, but without the
> "cluster." for the servers and it does at least start and it looks
> like all the three MMs find each other, but no mirroring taking place.
>
> Running the legacy-config from the old MM is working fine though. I'll
> try to do some more digging today, so if you need some of those very
> verbose logs or something else just let me know. I am sure that I can
> figure this out and just wanted to know if the documentation will get
> extended as the new MM2 has a lot of features and is a bit more
> complicated than the old one...
>
> Thanks
>
> Sebastian
>
> On 24-Dec-19 8:06 AM, Ryanne Dolan wrote:
>> Hello Sebastian, please let us know what issues you are facing and we
>> can
>> probably help. Which config from the KIP are you referencing? Also check
>> out the readme under ./connect/mirror for more examples.
>>
>> Ryanne
>>
>> On Mon, Dec 23, 2019, 12:58 PM Sebastian Schmitz <
>> sebastian.schmitz@propellerhead.co.nz> wrote:
>>
>>> Hello,
>>>
>>> I'm currently trying to implement the new Kafka 2.4.0 and the new MM2.
>>>
>>> However, it looks like the only documentation available is the KIP-382,
>>> and the documentation
>>> (https://kafka.apache.org/documentation/#basic_ops_mirror_maker) for
>>> the
>>> MM isn't yet updated, and the documentation in the KIP seems to be
>>> missing some stuff as I get a lot of errors and warning when starting
>>> the MM2 as connect-mirror, and it doesn't mirror, so I probably have
>>> some mistakes in my configuration, but can't confirm this as it's the
>>> same as in the KIP.
>>>
>>> Any plans when the documentation will be updated?
>>>
>>> Thanks
>>>
>>> Sebastian
>>>
>>>
>>> --
>>> DISCLAIMER
>>> This email contains information that is confidential and which
>>> may be
>>> legally privileged. If you have received this email in error please
>>>
>>> notify the sender immediately and delete the email.
>>> This email is intended
>>> solely for the use of the intended recipient and you may not use or
>>> disclose this email in any way.
>>>
--
DISCLAIMER
This email contains information that is confidential and which
may be
legally privileged. If you have received this email in error please
notify the sender immediately and delete the email.
This email is intended
solely for the use of the intended recipient and you may not use or
disclose this email in any way.
Re: Mirrormaker 2.0
Posted by Sebastian Schmitz <se...@propellerhead.co.nz>.
Hello,
I tried running this connect-mirror-config:
<snip>
name = $MIRROR_NAME
clusters = source, target
source.bootstrap.servers = $SOURCE_SERVERS
target.bootstrap.servers = $TARGET_SERVERS
source->target.topics = $SOURCE_TARGET_TOPICS
target->source.topics = $TARGET_SOURCE_TOPICS
source->target.emit.heartbeats.enabled = true
target->source.emit.heartbeats.enabled = true
connector.class = org.apache.kafka.connect.mirror.MirrorSourceConnector
# disable some new features
refresh.topics.enabled = false
refresh.groups.enabled = false
emit.checkpoints.enables = true
emit.heartbeats.enabled = true
sync.topic.configs.enabled = false
sync.topic.acls.enabled = false
</snip>
SOURCE_SERVERS and TARGET_SERVERS are a comma-separated list of three
brokers with ports.
The TOPICS are |-separated lists of topics.
I get these warning during startup which is a bit weird as I never
supplied any of those settings, but maybe I should?
[2019-12-23 00:36:25,918] WARN The configuration 'config.storage.topic'
was supplied but isn't a known config.
(org.apache.kafka.clients.producer.ProducerConfig:355)
[2019-12-23 00:36:25,918] WARN The configuration
'producer.bootstrap.servers' was supplied but isn't a known config.
(org.apache.kafka.clients.producer.ProducerConfig:355)
[2019-12-23 00:36:25,918] WARN The configuration 'group.id' was supplied
but isn't a known config.
(org.apache.kafka.clients.producer.ProducerConfig:355)
[2019-12-23 00:36:25,919] WARN The configuration 'status.storage.topic'
was supplied but isn't a known config.
(org.apache.kafka.clients.producer.ProducerConfig:355)
[2019-12-23 00:36:25,919] WARN The configuration 'header.converter' was
supplied but isn't a known config.
(org.apache.kafka.clients.producer.ProducerConfig:355)
[2019-12-23 00:36:25,919] WARN The configuration
'consumer.bootstrap.servers' was supplied but isn't a known config.
(org.apache.kafka.clients.producer.ProducerConfig:355)
[2019-12-23 00:36:25,919] WARN The configuration 'offset.storage.topic'
was supplied but isn't a known config.
(org.apache.kafka.clients.producer.ProducerConfig:355)
[2019-12-23 00:36:25,919] WARN The configuration 'value.converter' was
supplied but isn't a known config.
(org.apache.kafka.clients.producer.ProducerConfig:355)
[2019-12-23 00:36:25,919] WARN The configuration 'key.converter' was
supplied but isn't a known config.
(org.apache.kafka.clients.producer.ProducerConfig:355)
[2019-12-23 00:36:25,919] WARN The configuration
'admin.bootstrap.servers' was supplied but isn't a known config.
(org.apache.kafka.clients.producer.ProducerConfig:355)
And this error:
[2019-12-23 00:36:29,320] ERROR Plugin class loader for connector:
'org.apache.kafka.connect.mirror.MirrorSourceConnector' was not found.
Returning:
org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader@5c316230
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:165)
First I tried the config mentioned in the KIP for "MirrorMaker Clusters"
which didn't work and I found removing the "cluster." from the
bootstrap-servers made it work a bit more, at least it didn't complain
about not having any servers in the config.
So, I checked the "Running a dedicated MirrorMaker cluster"from the KIP,
which is basically more or less the same, but without the "cluster." for
the servers and it does at least start and it looks like all the three
MMs find each other, but no mirroring taking place.
Running the legacy-config from the old MM is working fine though. I'll
try to do some more digging today, so if you need some of those very
verbose logs or something else just let me know. I am sure that I can
figure this out and just wanted to know if the documentation will get
extended as the new MM2 has a lot of features and is a bit more
complicated than the old one...
Thanks
Sebastian
On 24-Dec-19 8:06 AM, Ryanne Dolan wrote:
> Hello Sebastian, please let us know what issues you are facing and we can
> probably help. Which config from the KIP are you referencing? Also check
> out the readme under ./connect/mirror for more examples.
>
> Ryanne
>
> On Mon, Dec 23, 2019, 12:58 PM Sebastian Schmitz <
> sebastian.schmitz@propellerhead.co.nz> wrote:
>
>> Hello,
>>
>> I'm currently trying to implement the new Kafka 2.4.0 and the new MM2.
>>
>> However, it looks like the only documentation available is the KIP-382,
>> and the documentation
>> (https://kafka.apache.org/documentation/#basic_ops_mirror_maker) for the
>> MM isn't yet updated, and the documentation in the KIP seems to be
>> missing some stuff as I get a lot of errors and warning when starting
>> the MM2 as connect-mirror, and it doesn't mirror, so I probably have
>> some mistakes in my configuration, but can't confirm this as it's the
>> same as in the KIP.
>>
>> Any plans when the documentation will be updated?
>>
>> Thanks
>>
>> Sebastian
>>
>>
>> --
>> DISCLAIMER
>> This email contains information that is confidential and which
>> may be
>> legally privileged. If you have received this email in error please
>>
>> notify the sender immediately and delete the email.
>> This email is intended
>> solely for the use of the intended recipient and you may not use or
>> disclose this email in any way.
>>
--
DISCLAIMER
This email contains information that is confidential and which
may be
legally privileged. If you have received this email in error please
notify the sender immediately and delete the email.
This email is intended
solely for the use of the intended recipient and you may not use or
disclose this email in any way.
Re: Mirrormaker 2.0
Posted by Ryanne Dolan <ry...@gmail.com>.
Hello Sebastian, please let us know what issues you are facing and we can
probably help. Which config from the KIP are you referencing? Also check
out the readme under ./connect/mirror for more examples.
Ryanne
On Mon, Dec 23, 2019, 12:58 PM Sebastian Schmitz <
sebastian.schmitz@propellerhead.co.nz> wrote:
> Hello,
>
> I'm currently trying to implement the new Kafka 2.4.0 and the new MM2.
>
> However, it looks like the only documentation available is the KIP-382,
> and the documentation
> (https://kafka.apache.org/documentation/#basic_ops_mirror_maker) for the
> MM isn't yet updated, and the documentation in the KIP seems to be
> missing some stuff as I get a lot of errors and warning when starting
> the MM2 as connect-mirror, and it doesn't mirror, so I probably have
> some mistakes in my configuration, but can't confirm this as it's the
> same as in the KIP.
>
> Any plans when the documentation will be updated?
>
> Thanks
>
> Sebastian
>
>
> --
> DISCLAIMER
> This email contains information that is confidential and which
> may be
> legally privileged. If you have received this email in error please
>
> notify the sender immediately and delete the email.
> This email is intended
> solely for the use of the intended recipient and you may not use or
> disclose this email in any way.
>
Re: Mirrormaker 2.0
Posted by Carl Graving <cg...@gmail.com>.
I find the best is the README in the source. Look under connect mirror
maker directory I believe.
Carl
On Mon, Dec 23, 2019, 13:57 Sebastian Schmitz <
sebastian.schmitz@propellerhead.co.nz> wrote:
> Hello,
>
> I'm currently trying to implement the new Kafka 2.4.0 and the new MM2.
>
> However, it looks like the only documentation available is the KIP-382,
> and the documentation
> (https://kafka.apache.org/documentation/#basic_ops_mirror_maker) for the
> MM isn't yet updated, and the documentation in the KIP seems to be
> missing some stuff as I get a lot of errors and warning when starting
> the MM2 as connect-mirror, and it doesn't mirror, so I probably have
> some mistakes in my configuration, but can't confirm this as it's the
> same as in the KIP.
>
> Any plans when the documentation will be updated?
>
> Thanks
>
> Sebastian
>
>
> --
> DISCLAIMER
> This email contains information that is confidential and which
> may be
> legally privileged. If you have received this email in error please
>
> notify the sender immediately and delete the email.
> This email is intended
> solely for the use of the intended recipient and you may not use or
> disclose this email in any way.
>