You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Jamie <ja...@aol.co.uk.INVALID> on 2021/07/29 16:37:10 UTC
Mirror Maker 2 - High Throughput Identity Mirroring
Hi All,
This blog post: https://blog.cloudera.com/a-look-inside-kafka-mirrormaker-2/ mentions that "High Throughput Identity Mirroring" (when the compression is the same in both the source and destination cluster) will soon be coming to MM2 which would avoid the MM2 consumer decompressing the data only for the MM2 producer to then re-compress it again.
Has this feature been implemented yet in MM2?
Many Thanks,
Jamie
Re: Mirror Maker 2 - High Throughput Identity Mirroring
Posted by Ryanne Dolan <ry...@gmail.com>.
Henry Cai has a PR somewhere.
Ryanne
On Wed, Mar 2, 2022, 3:36 AM Antón Rodríguez Yuste <an...@galiglobal.com>
wrote:
> Hi Ryanne,
>
> Is there a PR or code I could take a look at or just the KIP? "shallow
> mirroring" seems very interesting for our use cases, and I would like to
> evaluate it internally.
>
> Thanks,
>
> Antón
>
> On Thu, Jul 29, 2021 at 7:02 PM Ryanne Dolan <ry...@gmail.com>
> wrote:
>
> > Jamie, this would depend on KIP-712 (or similar) aka "shallow mirroring".
> > This is a work in progress, but I'm optimistic it'll happen at some
> point.
> >
> > ftr, "IdentityReplicationPolicy" has landed for the upcoming release, tho
> > "identity" in that context just means that topics aren't renamed.
> >
> > Ryanne
> >
> > On Thu, Jul 29, 2021, 11:37 AM Jamie <ja...@aol.co.uk.invalid>
> wrote:
> >
> > > Hi All,
> > > This blog post:
> > > https://blog.cloudera.com/a-look-inside-kafka-mirrormaker-2/ mentions
> > > that "High Throughput Identity Mirroring" (when the compression is the
> > same
> > > in both the source and destination cluster) will soon be coming to MM2
> > > which would avoid the MM2 consumer decompressing the data only for the
> > MM2
> > > producer to then re-compress it again.
> > > Has this feature been implemented yet in MM2?
> > > Many Thanks,
> > > Jamie
> >
>
Re: Mirror Maker 2 - High Throughput Identity Mirroring
Posted by Ryanne Dolan <ry...@gmail.com>.
Henry Cai has a PR somewhere.
Ryanne
On Wed, Mar 2, 2022, 3:36 AM Antón Rodríguez Yuste <an...@galiglobal.com>
wrote:
> Hi Ryanne,
>
> Is there a PR or code I could take a look at or just the KIP? "shallow
> mirroring" seems very interesting for our use cases, and I would like to
> evaluate it internally.
>
> Thanks,
>
> Antón
>
> On Thu, Jul 29, 2021 at 7:02 PM Ryanne Dolan <ry...@gmail.com>
> wrote:
>
> > Jamie, this would depend on KIP-712 (or similar) aka "shallow mirroring".
> > This is a work in progress, but I'm optimistic it'll happen at some
> point.
> >
> > ftr, "IdentityReplicationPolicy" has landed for the upcoming release, tho
> > "identity" in that context just means that topics aren't renamed.
> >
> > Ryanne
> >
> > On Thu, Jul 29, 2021, 11:37 AM Jamie <ja...@aol.co.uk.invalid>
> wrote:
> >
> > > Hi All,
> > > This blog post:
> > > https://blog.cloudera.com/a-look-inside-kafka-mirrormaker-2/ mentions
> > > that "High Throughput Identity Mirroring" (when the compression is the
> > same
> > > in both the source and destination cluster) will soon be coming to MM2
> > > which would avoid the MM2 consumer decompressing the data only for the
> > MM2
> > > producer to then re-compress it again.
> > > Has this feature been implemented yet in MM2?
> > > Many Thanks,
> > > Jamie
> >
>
Re: Mirror Maker 2 - High Throughput Identity Mirroring
Posted by Antón Rodríguez Yuste <an...@galiglobal.com>.
Hi Ryanne,
Is there a PR or code I could take a look at or just the KIP? "shallow
mirroring" seems very interesting for our use cases, and I would like to
evaluate it internally.
Thanks,
Antón
On Thu, Jul 29, 2021 at 7:02 PM Ryanne Dolan <ry...@gmail.com> wrote:
> Jamie, this would depend on KIP-712 (or similar) aka "shallow mirroring".
> This is a work in progress, but I'm optimistic it'll happen at some point.
>
> ftr, "IdentityReplicationPolicy" has landed for the upcoming release, tho
> "identity" in that context just means that topics aren't renamed.
>
> Ryanne
>
> On Thu, Jul 29, 2021, 11:37 AM Jamie <ja...@aol.co.uk.invalid> wrote:
>
> > Hi All,
> > This blog post:
> > https://blog.cloudera.com/a-look-inside-kafka-mirrormaker-2/ mentions
> > that "High Throughput Identity Mirroring" (when the compression is the
> same
> > in both the source and destination cluster) will soon be coming to MM2
> > which would avoid the MM2 consumer decompressing the data only for the
> MM2
> > producer to then re-compress it again.
> > Has this feature been implemented yet in MM2?
> > Many Thanks,
> > Jamie
>
Re: Mirror Maker 2 - High Throughput Identity Mirroring
Posted by Antón Rodríguez Yuste <an...@galiglobal.com>.
Hi Ryanne,
Is there a PR or code I could take a look at or just the KIP? "shallow
mirroring" seems very interesting for our use cases, and I would like to
evaluate it internally.
Thanks,
Antón
On Thu, Jul 29, 2021 at 7:02 PM Ryanne Dolan <ry...@gmail.com> wrote:
> Jamie, this would depend on KIP-712 (or similar) aka "shallow mirroring".
> This is a work in progress, but I'm optimistic it'll happen at some point.
>
> ftr, "IdentityReplicationPolicy" has landed for the upcoming release, tho
> "identity" in that context just means that topics aren't renamed.
>
> Ryanne
>
> On Thu, Jul 29, 2021, 11:37 AM Jamie <ja...@aol.co.uk.invalid> wrote:
>
> > Hi All,
> > This blog post:
> > https://blog.cloudera.com/a-look-inside-kafka-mirrormaker-2/ mentions
> > that "High Throughput Identity Mirroring" (when the compression is the
> same
> > in both the source and destination cluster) will soon be coming to MM2
> > which would avoid the MM2 consumer decompressing the data only for the
> MM2
> > producer to then re-compress it again.
> > Has this feature been implemented yet in MM2?
> > Many Thanks,
> > Jamie
>
Re: Mirror Maker 2 - High Throughput Identity Mirroring
Posted by Ryanne Dolan <ry...@gmail.com>.
Jamie, this would depend on KIP-712 (or similar) aka "shallow mirroring".
This is a work in progress, but I'm optimistic it'll happen at some point.
ftr, "IdentityReplicationPolicy" has landed for the upcoming release, tho
"identity" in that context just means that topics aren't renamed.
Ryanne
On Thu, Jul 29, 2021, 11:37 AM Jamie <ja...@aol.co.uk.invalid> wrote:
> Hi All,
> This blog post:
> https://blog.cloudera.com/a-look-inside-kafka-mirrormaker-2/ mentions
> that "High Throughput Identity Mirroring" (when the compression is the same
> in both the source and destination cluster) will soon be coming to MM2
> which would avoid the MM2 consumer decompressing the data only for the MM2
> producer to then re-compress it again.
> Has this feature been implemented yet in MM2?
> Many Thanks,
> Jamie
Re: Mirror Maker 2 - High Throughput Identity Mirroring
Posted by Ryanne Dolan <ry...@gmail.com>.
Jamie, this would depend on KIP-712 (or similar) aka "shallow mirroring".
This is a work in progress, but I'm optimistic it'll happen at some point.
ftr, "IdentityReplicationPolicy" has landed for the upcoming release, tho
"identity" in that context just means that topics aren't renamed.
Ryanne
On Thu, Jul 29, 2021, 11:37 AM Jamie <ja...@aol.co.uk.invalid> wrote:
> Hi All,
> This blog post:
> https://blog.cloudera.com/a-look-inside-kafka-mirrormaker-2/ mentions
> that "High Throughput Identity Mirroring" (when the compression is the same
> in both the source and destination cluster) will soon be coming to MM2
> which would avoid the MM2 consumer decompressing the data only for the MM2
> producer to then re-compress it again.
> Has this feature been implemented yet in MM2?
> Many Thanks,
> Jamie