You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Samuel Cantero <sc...@gmail.com> on 2020/09/10 23:03:16 UTC
MirrorMaker 2 - Can we start mirroring from latest?
Hello,
I'd like to know whether starting mirroring from latest is doable with
mirrormaker2. I tried setting the
target-cluster-alias.consumer.auto.offset.reset to latest but it doesn't
work (for an active/passive setup). Any help would be appreciated.
Best,
Re: MirrorMaker 2 - Can we start mirroring from latest?
Posted by Samuel Cantero <sc...@gmail.com>.
Iftach, I think I've found a bug. I tested this fix in my infra and worked
fine. Putting a PR upstream for review:
https://github.com/apache/kafka/pull/9313.
On Wed, Sep 16, 2020 at 1:26 PM Iftach Ben-Yosef <ib...@outbrain.com>
wrote:
> Hey Samuel,
>
> I am facing a similar issue with the partitions.assignment.strategy
> consumer setting on MM2.
>
> Let me know if you have any success with setting this (I am also interested
> with the auto.offset.reset config as well but its a lower priority for my
> testing. I assume the solution will be the same for both configs)
>
> Good luck and feel free to ping me if you want to try and discuss this.
>
> Iftach
>
>
> On Wed, Sep 16, 2020, 19:59 Samuel Cantero <sc...@gmail.com> wrote:
>
> > So the problem is that this is hardcoded for 2.5.1 and 2.6:
> >
> >
> https://github.com/apache/kafka/blob/62abe01bee0396515253373e906253bd0e081cb9/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorConnectorConfig.java#L235
> > .
> >
> > Already fixed in https://github.com/apache/kafka/pull/8921.
> >
> > I will try to understand why it is failing for other properties now.
> >
> > On Wed, Sep 16, 2020 at 12:20 PM Ning Zhang <ni...@gmail.com>
> > wrote:
> >
> > > just a possibility that `source-cluster-alias.` does not align with the
> > > <source_cluster_alias> used in `clusters:`
> > >
> > > On 2020/09/16 14:24:39, Samuel Cantero <sc...@gmail.com> wrote:
> > > > Definitely not working for me. I can't change any consumer attribute
> > for
> > > > source-alias or cluster-alias. It is working for the producer though.
> > The
> > > > config is rendered fine so I need to dig deeper on my side as this is
> > > > working for you.
> > > >
> > > > Thanks,
> > > >
> > > > On Tue, Sep 15, 2020 at 5:29 PM Ning Zhang <ni...@gmail.com>
> > > wrote:
> > > >
> > > > > Hello Samuel,
> > > > >
> > > > > I tried other consumer config and it worked. Basically, it should
> > > follow
> > > > > the pattern `source-cluster-alias.consumer.<consumer_config_name>`.
> > In
> > > your
> > > > > case, I think it should be
> > > > > `source-cluster-alias.consumer.auto.offset.reset`. But feel free to
> > try
> > > > > other consumer config and see if they work
> > > > >
> > > > > On 2020/09/15 20:53:12, Samuel Cantero <sc...@gmail.com>
> wrote:
> > > > > > I tried this but even the override does not work for me as I can
> > > still
> > > > > see
> > > > > > "earliest" on the ConsumerConfig properties. Did you manage
> > actually
> > > to
> > > > > > make this work or are you just recommending this config?
> > > > > >
> > > > > > Best,
> > > > > >
> > > > > > On Mon, Sep 14, 2020 at 7:26 PM Ning Zhang <
> ning2008wisc@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Hello Samuel,
> > > > > > >
> > > > > > > I guess you are talking about the case where you start to
> mirror
> > a
> > > new
> > > > > > > topic? If yes, I think the config should be
> > > > > > > `source-cluster-alias.consumer.auto.offset.reset`, instead of
> > > `target`,
> > > > > > > because `target` is where the producer of MM2 replicates the
> > > messages
> > > > > > > towards, right?
> > > > > > >
> > > > > > > If MM2 mirrors a topic that has been previously mirrored and
> the
> > > > > consumer
> > > > > > > offsets have preserved, `auto.offset.reset`` should not matter,
> > > > > because MM2
> > > > > > > always start mirroring from the last known offsets
> > > > > > >
> > > > > > > On 2020/09/10 23:03:16, Samuel Cantero <sc...@gmail.com>
> > > wrote:
> > > > > > > > Hello,
> > > > > > > >
> > > > > > > > I'd like to know whether starting mirroring from latest is
> > doable
> > > > > with
> > > > > > > > mirrormaker2. I tried setting the
> > > > > > > > target-cluster-alias.consumer.auto.offset.reset to latest but
> > it
> > > > > doesn't
> > > > > > > > work (for an active/passive setup). Any help would be
> > > appreciated.
> > > > > > > >
> > > > > > > > Best,
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> --
> The above terms reflect a potential business arrangement, are provided
> solely as a basis for further discussion, and are not intended to be and
> do
> not constitute a legally binding obligation. No legally binding
> obligations
> will be created, implied, or inferred until an agreement in final form is
> executed in writing by all parties involved.
>
>
> This email and any
> attachments hereto may be confidential or privileged. If you received
> this
> communication by mistake, please don't forward it to anyone else, please
> erase all copies and attachments, and please let me know that it has gone
> to the wrong person. Thanks.
>
Re: MirrorMaker 2 - Can we start mirroring from latest?
Posted by Iftach Ben-Yosef <ib...@outbrain.com>.
Hey Samuel,
I am facing a similar issue with the partitions.assignment.strategy
consumer setting on MM2.
Let me know if you have any success with setting this (I am also interested
with the auto.offset.reset config as well but its a lower priority for my
testing. I assume the solution will be the same for both configs)
Good luck and feel free to ping me if you want to try and discuss this.
Iftach
On Wed, Sep 16, 2020, 19:59 Samuel Cantero <sc...@gmail.com> wrote:
> So the problem is that this is hardcoded for 2.5.1 and 2.6:
>
> https://github.com/apache/kafka/blob/62abe01bee0396515253373e906253bd0e081cb9/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorConnectorConfig.java#L235
> .
>
> Already fixed in https://github.com/apache/kafka/pull/8921.
>
> I will try to understand why it is failing for other properties now.
>
> On Wed, Sep 16, 2020 at 12:20 PM Ning Zhang <ni...@gmail.com>
> wrote:
>
> > just a possibility that `source-cluster-alias.` does not align with the
> > <source_cluster_alias> used in `clusters:`
> >
> > On 2020/09/16 14:24:39, Samuel Cantero <sc...@gmail.com> wrote:
> > > Definitely not working for me. I can't change any consumer attribute
> for
> > > source-alias or cluster-alias. It is working for the producer though.
> The
> > > config is rendered fine so I need to dig deeper on my side as this is
> > > working for you.
> > >
> > > Thanks,
> > >
> > > On Tue, Sep 15, 2020 at 5:29 PM Ning Zhang <ni...@gmail.com>
> > wrote:
> > >
> > > > Hello Samuel,
> > > >
> > > > I tried other consumer config and it worked. Basically, it should
> > follow
> > > > the pattern `source-cluster-alias.consumer.<consumer_config_name>`.
> In
> > your
> > > > case, I think it should be
> > > > `source-cluster-alias.consumer.auto.offset.reset`. But feel free to
> try
> > > > other consumer config and see if they work
> > > >
> > > > On 2020/09/15 20:53:12, Samuel Cantero <sc...@gmail.com> wrote:
> > > > > I tried this but even the override does not work for me as I can
> > still
> > > > see
> > > > > "earliest" on the ConsumerConfig properties. Did you manage
> actually
> > to
> > > > > make this work or are you just recommending this config?
> > > > >
> > > > > Best,
> > > > >
> > > > > On Mon, Sep 14, 2020 at 7:26 PM Ning Zhang <ning2008wisc@gmail.com
> >
> > > > wrote:
> > > > >
> > > > > > Hello Samuel,
> > > > > >
> > > > > > I guess you are talking about the case where you start to mirror
> a
> > new
> > > > > > topic? If yes, I think the config should be
> > > > > > `source-cluster-alias.consumer.auto.offset.reset`, instead of
> > `target`,
> > > > > > because `target` is where the producer of MM2 replicates the
> > messages
> > > > > > towards, right?
> > > > > >
> > > > > > If MM2 mirrors a topic that has been previously mirrored and the
> > > > consumer
> > > > > > offsets have preserved, `auto.offset.reset`` should not matter,
> > > > because MM2
> > > > > > always start mirroring from the last known offsets
> > > > > >
> > > > > > On 2020/09/10 23:03:16, Samuel Cantero <sc...@gmail.com>
> > wrote:
> > > > > > > Hello,
> > > > > > >
> > > > > > > I'd like to know whether starting mirroring from latest is
> doable
> > > > with
> > > > > > > mirrormaker2. I tried setting the
> > > > > > > target-cluster-alias.consumer.auto.offset.reset to latest but
> it
> > > > doesn't
> > > > > > > work (for an active/passive setup). Any help would be
> > appreciated.
> > > > > > >
> > > > > > > Best,
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
--
The above terms reflect a potential business arrangement, are provided
solely as a basis for further discussion, and are not intended to be and do
not constitute a legally binding obligation. No legally binding obligations
will be created, implied, or inferred until an agreement in final form is
executed in writing by all parties involved.
This email and any
attachments hereto may be confidential or privileged. If you received this
communication by mistake, please don't forward it to anyone else, please
erase all copies and attachments, and please let me know that it has gone
to the wrong person. Thanks.
Re: MirrorMaker 2 - Can we start mirroring from latest?
Posted by Samuel Cantero <sc...@gmail.com>.
So the problem is that this is hardcoded for 2.5.1 and 2.6:
https://github.com/apache/kafka/blob/62abe01bee0396515253373e906253bd0e081cb9/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorConnectorConfig.java#L235
.
Already fixed in https://github.com/apache/kafka/pull/8921.
I will try to understand why it is failing for other properties now.
On Wed, Sep 16, 2020 at 12:20 PM Ning Zhang <ni...@gmail.com> wrote:
> just a possibility that `source-cluster-alias.` does not align with the
> <source_cluster_alias> used in `clusters:`
>
> On 2020/09/16 14:24:39, Samuel Cantero <sc...@gmail.com> wrote:
> > Definitely not working for me. I can't change any consumer attribute for
> > source-alias or cluster-alias. It is working for the producer though. The
> > config is rendered fine so I need to dig deeper on my side as this is
> > working for you.
> >
> > Thanks,
> >
> > On Tue, Sep 15, 2020 at 5:29 PM Ning Zhang <ni...@gmail.com>
> wrote:
> >
> > > Hello Samuel,
> > >
> > > I tried other consumer config and it worked. Basically, it should
> follow
> > > the pattern `source-cluster-alias.consumer.<consumer_config_name>`. In
> your
> > > case, I think it should be
> > > `source-cluster-alias.consumer.auto.offset.reset`. But feel free to try
> > > other consumer config and see if they work
> > >
> > > On 2020/09/15 20:53:12, Samuel Cantero <sc...@gmail.com> wrote:
> > > > I tried this but even the override does not work for me as I can
> still
> > > see
> > > > "earliest" on the ConsumerConfig properties. Did you manage actually
> to
> > > > make this work or are you just recommending this config?
> > > >
> > > > Best,
> > > >
> > > > On Mon, Sep 14, 2020 at 7:26 PM Ning Zhang <ni...@gmail.com>
> > > wrote:
> > > >
> > > > > Hello Samuel,
> > > > >
> > > > > I guess you are talking about the case where you start to mirror a
> new
> > > > > topic? If yes, I think the config should be
> > > > > `source-cluster-alias.consumer.auto.offset.reset`, instead of
> `target`,
> > > > > because `target` is where the producer of MM2 replicates the
> messages
> > > > > towards, right?
> > > > >
> > > > > If MM2 mirrors a topic that has been previously mirrored and the
> > > consumer
> > > > > offsets have preserved, `auto.offset.reset`` should not matter,
> > > because MM2
> > > > > always start mirroring from the last known offsets
> > > > >
> > > > > On 2020/09/10 23:03:16, Samuel Cantero <sc...@gmail.com>
> wrote:
> > > > > > Hello,
> > > > > >
> > > > > > I'd like to know whether starting mirroring from latest is doable
> > > with
> > > > > > mirrormaker2. I tried setting the
> > > > > > target-cluster-alias.consumer.auto.offset.reset to latest but it
> > > doesn't
> > > > > > work (for an active/passive setup). Any help would be
> appreciated.
> > > > > >
> > > > > > Best,
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: MirrorMaker 2 - Can we start mirroring from latest?
Posted by Ning Zhang <ni...@gmail.com>.
just a possibility that `source-cluster-alias.` does not align with the <source_cluster_alias> used in `clusters:`
On 2020/09/16 14:24:39, Samuel Cantero <sc...@gmail.com> wrote:
> Definitely not working for me. I can't change any consumer attribute for
> source-alias or cluster-alias. It is working for the producer though. The
> config is rendered fine so I need to dig deeper on my side as this is
> working for you.
>
> Thanks,
>
> On Tue, Sep 15, 2020 at 5:29 PM Ning Zhang <ni...@gmail.com> wrote:
>
> > Hello Samuel,
> >
> > I tried other consumer config and it worked. Basically, it should follow
> > the pattern `source-cluster-alias.consumer.<consumer_config_name>`. In your
> > case, I think it should be
> > `source-cluster-alias.consumer.auto.offset.reset`. But feel free to try
> > other consumer config and see if they work
> >
> > On 2020/09/15 20:53:12, Samuel Cantero <sc...@gmail.com> wrote:
> > > I tried this but even the override does not work for me as I can still
> > see
> > > "earliest" on the ConsumerConfig properties. Did you manage actually to
> > > make this work or are you just recommending this config?
> > >
> > > Best,
> > >
> > > On Mon, Sep 14, 2020 at 7:26 PM Ning Zhang <ni...@gmail.com>
> > wrote:
> > >
> > > > Hello Samuel,
> > > >
> > > > I guess you are talking about the case where you start to mirror a new
> > > > topic? If yes, I think the config should be
> > > > `source-cluster-alias.consumer.auto.offset.reset`, instead of `target`,
> > > > because `target` is where the producer of MM2 replicates the messages
> > > > towards, right?
> > > >
> > > > If MM2 mirrors a topic that has been previously mirrored and the
> > consumer
> > > > offsets have preserved, `auto.offset.reset`` should not matter,
> > because MM2
> > > > always start mirroring from the last known offsets
> > > >
> > > > On 2020/09/10 23:03:16, Samuel Cantero <sc...@gmail.com> wrote:
> > > > > Hello,
> > > > >
> > > > > I'd like to know whether starting mirroring from latest is doable
> > with
> > > > > mirrormaker2. I tried setting the
> > > > > target-cluster-alias.consumer.auto.offset.reset to latest but it
> > doesn't
> > > > > work (for an active/passive setup). Any help would be appreciated.
> > > > >
> > > > > Best,
> > > > >
> > > >
> > >
> >
>
Re: MirrorMaker 2 - Can we start mirroring from latest?
Posted by Samuel Cantero <sc...@gmail.com>.
Definitely not working for me. I can't change any consumer attribute for
source-alias or cluster-alias. It is working for the producer though. The
config is rendered fine so I need to dig deeper on my side as this is
working for you.
Thanks,
On Tue, Sep 15, 2020 at 5:29 PM Ning Zhang <ni...@gmail.com> wrote:
> Hello Samuel,
>
> I tried other consumer config and it worked. Basically, it should follow
> the pattern `source-cluster-alias.consumer.<consumer_config_name>`. In your
> case, I think it should be
> `source-cluster-alias.consumer.auto.offset.reset`. But feel free to try
> other consumer config and see if they work
>
> On 2020/09/15 20:53:12, Samuel Cantero <sc...@gmail.com> wrote:
> > I tried this but even the override does not work for me as I can still
> see
> > "earliest" on the ConsumerConfig properties. Did you manage actually to
> > make this work or are you just recommending this config?
> >
> > Best,
> >
> > On Mon, Sep 14, 2020 at 7:26 PM Ning Zhang <ni...@gmail.com>
> wrote:
> >
> > > Hello Samuel,
> > >
> > > I guess you are talking about the case where you start to mirror a new
> > > topic? If yes, I think the config should be
> > > `source-cluster-alias.consumer.auto.offset.reset`, instead of `target`,
> > > because `target` is where the producer of MM2 replicates the messages
> > > towards, right?
> > >
> > > If MM2 mirrors a topic that has been previously mirrored and the
> consumer
> > > offsets have preserved, `auto.offset.reset`` should not matter,
> because MM2
> > > always start mirroring from the last known offsets
> > >
> > > On 2020/09/10 23:03:16, Samuel Cantero <sc...@gmail.com> wrote:
> > > > Hello,
> > > >
> > > > I'd like to know whether starting mirroring from latest is doable
> with
> > > > mirrormaker2. I tried setting the
> > > > target-cluster-alias.consumer.auto.offset.reset to latest but it
> doesn't
> > > > work (for an active/passive setup). Any help would be appreciated.
> > > >
> > > > Best,
> > > >
> > >
> >
>
Re: MirrorMaker 2 - Can we start mirroring from latest?
Posted by Ning Zhang <ni...@gmail.com>.
Hello Samuel,
I tried other consumer config and it worked. Basically, it should follow the pattern `source-cluster-alias.consumer.<consumer_config_name>`. In your case, I think it should be `source-cluster-alias.consumer.auto.offset.reset`. But feel free to try other consumer config and see if they work
On 2020/09/15 20:53:12, Samuel Cantero <sc...@gmail.com> wrote:
> I tried this but even the override does not work for me as I can still see
> "earliest" on the ConsumerConfig properties. Did you manage actually to
> make this work or are you just recommending this config?
>
> Best,
>
> On Mon, Sep 14, 2020 at 7:26 PM Ning Zhang <ni...@gmail.com> wrote:
>
> > Hello Samuel,
> >
> > I guess you are talking about the case where you start to mirror a new
> > topic? If yes, I think the config should be
> > `source-cluster-alias.consumer.auto.offset.reset`, instead of `target`,
> > because `target` is where the producer of MM2 replicates the messages
> > towards, right?
> >
> > If MM2 mirrors a topic that has been previously mirrored and the consumer
> > offsets have preserved, `auto.offset.reset`` should not matter, because MM2
> > always start mirroring from the last known offsets
> >
> > On 2020/09/10 23:03:16, Samuel Cantero <sc...@gmail.com> wrote:
> > > Hello,
> > >
> > > I'd like to know whether starting mirroring from latest is doable with
> > > mirrormaker2. I tried setting the
> > > target-cluster-alias.consumer.auto.offset.reset to latest but it doesn't
> > > work (for an active/passive setup). Any help would be appreciated.
> > >
> > > Best,
> > >
> >
>
Re: MirrorMaker 2 - Can we start mirroring from latest?
Posted by Samuel Cantero <sc...@gmail.com>.
I tried this but even the override does not work for me as I can still see
"earliest" on the ConsumerConfig properties. Did you manage actually to
make this work or are you just recommending this config?
Best,
On Mon, Sep 14, 2020 at 7:26 PM Ning Zhang <ni...@gmail.com> wrote:
> Hello Samuel,
>
> I guess you are talking about the case where you start to mirror a new
> topic? If yes, I think the config should be
> `source-cluster-alias.consumer.auto.offset.reset`, instead of `target`,
> because `target` is where the producer of MM2 replicates the messages
> towards, right?
>
> If MM2 mirrors a topic that has been previously mirrored and the consumer
> offsets have preserved, `auto.offset.reset`` should not matter, because MM2
> always start mirroring from the last known offsets
>
> On 2020/09/10 23:03:16, Samuel Cantero <sc...@gmail.com> wrote:
> > Hello,
> >
> > I'd like to know whether starting mirroring from latest is doable with
> > mirrormaker2. I tried setting the
> > target-cluster-alias.consumer.auto.offset.reset to latest but it doesn't
> > work (for an active/passive setup). Any help would be appreciated.
> >
> > Best,
> >
>
Re: MirrorMaker 2 - Can we start mirroring from latest?
Posted by Ning Zhang <ni...@gmail.com>.
Hello Samuel,
I guess you are talking about the case where you start to mirror a new topic? If yes, I think the config should be `source-cluster-alias.consumer.auto.offset.reset`, instead of `target`, because `target` is where the producer of MM2 replicates the messages towards, right?
If MM2 mirrors a topic that has been previously mirrored and the consumer offsets have preserved, `auto.offset.reset`` should not matter, because MM2 always start mirroring from the last known offsets
On 2020/09/10 23:03:16, Samuel Cantero <sc...@gmail.com> wrote:
> Hello,
>
> I'd like to know whether starting mirroring from latest is doable with
> mirrormaker2. I tried setting the
> target-cluster-alias.consumer.auto.offset.reset to latest but it doesn't
> work (for an active/passive setup). Any help would be appreciated.
>
> Best,
>