You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Iftach Ben-Yosef <ib...@outbrain.com> on 2020/09/02 07:10:45 UTC
MirrorMaker 2 - is starting mirroring from latest possible?
Hello,
Whenever we add a new topic to the mirroring whitelist it starts to mirror
the entire content of the source topic. If the topic is large this can
create a long lag until the entire topic is mirrored, and it can also
create some smaller delays on other mirrored topics (i'm assuming this is
because the cluster is using most resources to handle the new topic). Once
the new topics lag is resolved everything seems fine.
Is it possible to tell MM2 to start mirroring new topics in the whitelist
from the latest messages, and ignore older messages in the topic?
Thanks,
Iftach
--
The above terms reflect a potential business arrangement, are provided
solely as a basis for further discussion, and are not intended to be and do
not constitute a legally binding obligation. No legally binding obligations
will be created, implied, or inferred until an agreement in final form is
executed in writing by all parties involved.
This email and any
attachments hereto may be confidential or privileged. If you received this
communication by mistake, please don't forward it to anyone else, please
erase all copies and attachments, and please let me know that it has gone
to the wrong person. Thanks.
Re: MirrorMaker 2 - is starting mirroring from latest possible?
Posted by Ning Zhang <ni...@gmail.com>.
That is possible, please check out this thread: https://lists.apache.org/thread.html/r978775b2dbdf34a5c6c40105e2e69ad205fa94e33cef08e8519389c9%40%3Cusers.kafka.apache.org%3E
On 2020/09/02 07:10:45, Iftach Ben-Yosef <ib...@outbrain.com> wrote:
> Hello,
>
> Whenever we add a new topic to the mirroring whitelist it starts to mirror
> the entire content of the source topic. If the topic is large this can
> create a long lag until the entire topic is mirrored, and it can also
> create some smaller delays on other mirrored topics (i'm assuming this is
> because the cluster is using most resources to handle the new topic). Once
> the new topics lag is resolved everything seems fine.
>
> Is it possible to tell MM2 to start mirroring new topics in the whitelist
> from the latest messages, and ignore older messages in the topic?
>
>
>
> Thanks,
> Iftach
>
> --
> The above terms reflect a potential business arrangement, are provided
> solely as a basis for further discussion, and are not intended to be and do
> not constitute a legally binding obligation. No legally binding obligations
> will be created, implied, or inferred until an agreement in final form is
> executed in writing by all parties involved.
>
>
> This email and any
> attachments hereto may be confidential or privileged. If you received this
> communication by mistake, please don't forward it to anyone else, please
> erase all copies and attachments, and please let me know that it has gone
> to the wrong person. Thanks.
>