You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Marcin Wieloch <ma...@gmail.com> on 2020/03/11 10:26:25 UTC
Feature request for 'prefixless' topic mirroring
Dear All,
I would like to raise a feature request for ‘prefixless’ topic
mirroring with MirrorMaker2 (see detailed description below).
As after quite a few tries I am not 100% sure, could you please
confirm that by now
the ‘prefixless’ mirroring is not supported and that my feature
request is valid? I would then file it in JIRA.
Best regards,
Marcin
Feature Request
We would like to use MirrorMaker2 as a DRP solution for a simple setup
with two sites working in active-passive configuration
where all topics are always mirrored in one direction only and the
downstream topics keep their original names, i.e., without being
prefixed by MirrorMaker2.
After a few rounds of 'trial and error' we came up with a customized
replication policy which, although hacky, seems to fulfill our need;
it is presented in stackoverflow here:
https://stackoverflow.com/a/60619233
and source code is available here:
https://github.com/mawileo/mm2-prefixless-policy/blob/feature/prefixless/src/main/java/ch/mawileo/kafka/mm2/PrefixlessReplicationPolicy.java
This solution is 'hacky' in that it violates specification for the
topicSource and upstreamTopic methods as expressed in javadoc of the
ReplicationPolicy interface.
That makes us suspect that MirrorMaker2 has been never meant to work
in such a configuration and that our solution may stop
working with future releases of Kafka.
We would like then to ask for making the possibility of 'prefixless'
topic mirroring in a simplified two-cluster setup a standard
feature of Kafka Mirror Maker 2.
Re: Feature request for 'prefixless' topic mirroring
Posted by Ryanne Dolan <ry...@gmail.com>.
Or perhaps we could provide a more general ReplicationPolicy that renames
according to a pattern string, and the Legacy and Prefixless use-cases
could just use that.
On Mon, Mar 16, 2020 at 12:32 PM Andrew Otto <ot...@wikimedia.org> wrote:
> +1 to this general idea. If Wikimedia ever gets around to switching to
> MM2, we'll need something like this as our apps manually do the topic
> prefixing for active <-> active replication.
>
> Perhaps 'Legacy' isn't a good name for this type of replication? There may
> be more uses cases like Marcin's where folks want to do this not just for
> migration to MM2 reasons. 'Prefixless' as Marcin suggests is pretty
> descriptive. :)
>
> -Andrew Otto
> SRE, Wikimedia Foundation
>
>
> On Mon, Mar 16, 2020 at 1:23 PM Ryanne Dolan <ry...@gmail.com>
> wrote:
>
> > Marcin, I just created this jira per your suggestion:
> > https://issues.apache.org/jira/browse/KAFKA-9726
> >
> > I've assigned it to myself but I'm happy to give it up.
> >
> > Ryanne
> >
> > On Mon, Mar 16, 2020 at 2:37 AM Marcin Wieloch <marcin.wieloch@gmail.com
> >
> > wrote:
> >
> > > Hello Ryanne,
> > >
> > > Thank you for these clarifications. I am glad that the use case of
> > > “prefixless” replication is in-line with your vision for further
> > > development of MirrorMaker2; I forgot indeed that already KIP-382
> > mentioned
> > > that LegacyReplicationPolicy.
> > >
> > > I have not found however any JIRA issue addressing specifically
> > > LegacyReplicationPolicy, and the main ticket for KIP-382 (KAFKA-7500)
> is
> > > already resolved.
> > >
> > > So what would be the next step you recommend I take? Should I file my
> > > feature request (rephrased to adjust for your comments)? Or is the need
> > for
> > > the legacy policy already tracked somehow and we don’t have to open any
> > new
> > > tickets for it?
> > >
> > > Marcin
> > >
> > > On 2020/03/11 14:20:44, Ryanne Dolan <ry...@gmail.com> wrote:
> > > > Hello Marcin, actually KIP-382 includes LegacyReplicationPolicy (not
> > yet
> > > > released) which is similar to your PrefixlessReplicationPolicy and
> the
> > > > MigrationReplicationPolicy mentioned. I don't think these are hacky,
> > > though
> > > > maybe the documentation could better address what to do when a method
> > > > doesn't necessarily apply cleanly.
> > > >
> > > > Frankly, I anticipated needing to make changes to MM2 itself to
> > > accommodate
> > > > these sort of use-cases -- the fact that you got this working as-is
> is
> > > > encouraging :)
> > > >
> > > > Re the proposal, I think we could ship one or more of these
> replication
> > > > policies with MM2 to support prefixless replication. There may be
> > > slightly
> > > > different implementations depending on the use-case, i.e. "migration"
> > > from
> > > > an old cluster vs active/passive w/o renaming topics vs mimicking MM1
> > > with
> > > > LegacyReplicationPolicy. Maybe they are all identical and we just
> need
> > > one?
> > > >
> > > > I like that use-cases are driving these requests rather than us try
> to
> > > > enumerate all possible ReplicationPolicy implementations. Given a
> > > > particular use-case and a successful application of a custom
> > > > ReplicationPolicy, I for one would support including it in Apache
> > Kafka.
> > > >
> > > > Ryanne
> > > >
> > > > On Wed, Mar 11, 2020, 5:26 AM Marcin Wieloch <
> marcin.wieloch@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Dear All,
> > > > >
> > > > > I would like to raise a feature request for ‘prefixless’ topic
> > > > > mirroring with MirrorMaker2 (see detailed description below).
> > > > >
> > > > > As after quite a few tries I am not 100% sure, could you please
> > > > > confirm that by now
> > > > > the ‘prefixless’ mirroring is not supported and that my feature
> > > > > request is valid? I would then file it in JIRA.
> > > > >
> > > > > Best regards,
> > > > >
> > > > > Marcin
> > > > >
> > > > >
> > > > > Feature Request
> > > > >
> > > > > We would like to use MirrorMaker2 as a DRP solution for a simple
> > setup
> > > > > with two sites working in active-passive configuration
> > > > > where all topics are always mirrored in one direction only and the
> > > > > downstream topics keep their original names, i.e., without being
> > > > > prefixed by MirrorMaker2.
> > > > >
> > > > > After a few rounds of 'trial and error' we came up with a
> customized
> > > > > replication policy which, although hacky, seems to fulfill our
> need;
> > > > > it is presented in stackoverflow here:
> > > > > https://stackoverflow.com/a/60619233
> > > > > and source code is available here:
> > > > >
> > > > >
> > >
> >
> https://github.com/mawileo/mm2-prefixless-policy/blob/feature/prefixless/src/main/java/ch/mawileo/kafka/mm2/PrefixlessReplicationPolicy.java
> > > > >
> > > > > This solution is 'hacky' in that it violates specification for the
> > > > > topicSource and upstreamTopic methods as expressed in javadoc of
> the
> > > > > ReplicationPolicy interface.
> > > > >
> > > > > That makes us suspect that MirrorMaker2 has been never meant to
> work
> > > > > in such a configuration and that our solution may stop
> > > > > working with future releases of Kafka.
> > > > >
> > > > > We would like then to ask for making the possibility of
> 'prefixless'
> > > > > topic mirroring in a simplified two-cluster setup a standard
> > > > > feature of Kafka Mirror Maker 2.
> > > > >
> > > >
> > >
> >
>
Re: Feature request for 'prefixless' topic mirroring
Posted by Andrew Otto <ot...@wikimedia.org>.
+1 to this general idea. If Wikimedia ever gets around to switching to
MM2, we'll need something like this as our apps manually do the topic
prefixing for active <-> active replication.
Perhaps 'Legacy' isn't a good name for this type of replication? There may
be more uses cases like Marcin's where folks want to do this not just for
migration to MM2 reasons. 'Prefixless' as Marcin suggests is pretty
descriptive. :)
-Andrew Otto
SRE, Wikimedia Foundation
On Mon, Mar 16, 2020 at 1:23 PM Ryanne Dolan <ry...@gmail.com> wrote:
> Marcin, I just created this jira per your suggestion:
> https://issues.apache.org/jira/browse/KAFKA-9726
>
> I've assigned it to myself but I'm happy to give it up.
>
> Ryanne
>
> On Mon, Mar 16, 2020 at 2:37 AM Marcin Wieloch <ma...@gmail.com>
> wrote:
>
> > Hello Ryanne,
> >
> > Thank you for these clarifications. I am glad that the use case of
> > “prefixless” replication is in-line with your vision for further
> > development of MirrorMaker2; I forgot indeed that already KIP-382
> mentioned
> > that LegacyReplicationPolicy.
> >
> > I have not found however any JIRA issue addressing specifically
> > LegacyReplicationPolicy, and the main ticket for KIP-382 (KAFKA-7500) is
> > already resolved.
> >
> > So what would be the next step you recommend I take? Should I file my
> > feature request (rephrased to adjust for your comments)? Or is the need
> for
> > the legacy policy already tracked somehow and we don’t have to open any
> new
> > tickets for it?
> >
> > Marcin
> >
> > On 2020/03/11 14:20:44, Ryanne Dolan <ry...@gmail.com> wrote:
> > > Hello Marcin, actually KIP-382 includes LegacyReplicationPolicy (not
> yet
> > > released) which is similar to your PrefixlessReplicationPolicy and the
> > > MigrationReplicationPolicy mentioned. I don't think these are hacky,
> > though
> > > maybe the documentation could better address what to do when a method
> > > doesn't necessarily apply cleanly.
> > >
> > > Frankly, I anticipated needing to make changes to MM2 itself to
> > accommodate
> > > these sort of use-cases -- the fact that you got this working as-is is
> > > encouraging :)
> > >
> > > Re the proposal, I think we could ship one or more of these replication
> > > policies with MM2 to support prefixless replication. There may be
> > slightly
> > > different implementations depending on the use-case, i.e. "migration"
> > from
> > > an old cluster vs active/passive w/o renaming topics vs mimicking MM1
> > with
> > > LegacyReplicationPolicy. Maybe they are all identical and we just need
> > one?
> > >
> > > I like that use-cases are driving these requests rather than us try to
> > > enumerate all possible ReplicationPolicy implementations. Given a
> > > particular use-case and a successful application of a custom
> > > ReplicationPolicy, I for one would support including it in Apache
> Kafka.
> > >
> > > Ryanne
> > >
> > > On Wed, Mar 11, 2020, 5:26 AM Marcin Wieloch <marcin.wieloch@gmail.com
> >
> > > wrote:
> > >
> > > > Dear All,
> > > >
> > > > I would like to raise a feature request for ‘prefixless’ topic
> > > > mirroring with MirrorMaker2 (see detailed description below).
> > > >
> > > > As after quite a few tries I am not 100% sure, could you please
> > > > confirm that by now
> > > > the ‘prefixless’ mirroring is not supported and that my feature
> > > > request is valid? I would then file it in JIRA.
> > > >
> > > > Best regards,
> > > >
> > > > Marcin
> > > >
> > > >
> > > > Feature Request
> > > >
> > > > We would like to use MirrorMaker2 as a DRP solution for a simple
> setup
> > > > with two sites working in active-passive configuration
> > > > where all topics are always mirrored in one direction only and the
> > > > downstream topics keep their original names, i.e., without being
> > > > prefixed by MirrorMaker2.
> > > >
> > > > After a few rounds of 'trial and error' we came up with a customized
> > > > replication policy which, although hacky, seems to fulfill our need;
> > > > it is presented in stackoverflow here:
> > > > https://stackoverflow.com/a/60619233
> > > > and source code is available here:
> > > >
> > > >
> >
> https://github.com/mawileo/mm2-prefixless-policy/blob/feature/prefixless/src/main/java/ch/mawileo/kafka/mm2/PrefixlessReplicationPolicy.java
> > > >
> > > > This solution is 'hacky' in that it violates specification for the
> > > > topicSource and upstreamTopic methods as expressed in javadoc of the
> > > > ReplicationPolicy interface.
> > > >
> > > > That makes us suspect that MirrorMaker2 has been never meant to work
> > > > in such a configuration and that our solution may stop
> > > > working with future releases of Kafka.
> > > >
> > > > We would like then to ask for making the possibility of 'prefixless'
> > > > topic mirroring in a simplified two-cluster setup a standard
> > > > feature of Kafka Mirror Maker 2.
> > > >
> > >
> >
>
Re: Feature request for 'prefixless' topic mirroring
Posted by Ryanne Dolan <ry...@gmail.com>.
Marcin, I just created this jira per your suggestion:
https://issues.apache.org/jira/browse/KAFKA-9726
I've assigned it to myself but I'm happy to give it up.
Ryanne
On Mon, Mar 16, 2020 at 2:37 AM Marcin Wieloch <ma...@gmail.com>
wrote:
> Hello Ryanne,
>
> Thank you for these clarifications. I am glad that the use case of
> “prefixless” replication is in-line with your vision for further
> development of MirrorMaker2; I forgot indeed that already KIP-382 mentioned
> that LegacyReplicationPolicy.
>
> I have not found however any JIRA issue addressing specifically
> LegacyReplicationPolicy, and the main ticket for KIP-382 (KAFKA-7500) is
> already resolved.
>
> So what would be the next step you recommend I take? Should I file my
> feature request (rephrased to adjust for your comments)? Or is the need for
> the legacy policy already tracked somehow and we don’t have to open any new
> tickets for it?
>
> Marcin
>
> On 2020/03/11 14:20:44, Ryanne Dolan <ry...@gmail.com> wrote:
> > Hello Marcin, actually KIP-382 includes LegacyReplicationPolicy (not yet
> > released) which is similar to your PrefixlessReplicationPolicy and the
> > MigrationReplicationPolicy mentioned. I don't think these are hacky,
> though
> > maybe the documentation could better address what to do when a method
> > doesn't necessarily apply cleanly.
> >
> > Frankly, I anticipated needing to make changes to MM2 itself to
> accommodate
> > these sort of use-cases -- the fact that you got this working as-is is
> > encouraging :)
> >
> > Re the proposal, I think we could ship one or more of these replication
> > policies with MM2 to support prefixless replication. There may be
> slightly
> > different implementations depending on the use-case, i.e. "migration"
> from
> > an old cluster vs active/passive w/o renaming topics vs mimicking MM1
> with
> > LegacyReplicationPolicy. Maybe they are all identical and we just need
> one?
> >
> > I like that use-cases are driving these requests rather than us try to
> > enumerate all possible ReplicationPolicy implementations. Given a
> > particular use-case and a successful application of a custom
> > ReplicationPolicy, I for one would support including it in Apache Kafka.
> >
> > Ryanne
> >
> > On Wed, Mar 11, 2020, 5:26 AM Marcin Wieloch <ma...@gmail.com>
> > wrote:
> >
> > > Dear All,
> > >
> > > I would like to raise a feature request for ‘prefixless’ topic
> > > mirroring with MirrorMaker2 (see detailed description below).
> > >
> > > As after quite a few tries I am not 100% sure, could you please
> > > confirm that by now
> > > the ‘prefixless’ mirroring is not supported and that my feature
> > > request is valid? I would then file it in JIRA.
> > >
> > > Best regards,
> > >
> > > Marcin
> > >
> > >
> > > Feature Request
> > >
> > > We would like to use MirrorMaker2 as a DRP solution for a simple setup
> > > with two sites working in active-passive configuration
> > > where all topics are always mirrored in one direction only and the
> > > downstream topics keep their original names, i.e., without being
> > > prefixed by MirrorMaker2.
> > >
> > > After a few rounds of 'trial and error' we came up with a customized
> > > replication policy which, although hacky, seems to fulfill our need;
> > > it is presented in stackoverflow here:
> > > https://stackoverflow.com/a/60619233
> > > and source code is available here:
> > >
> > >
> https://github.com/mawileo/mm2-prefixless-policy/blob/feature/prefixless/src/main/java/ch/mawileo/kafka/mm2/PrefixlessReplicationPolicy.java
> > >
> > > This solution is 'hacky' in that it violates specification for the
> > > topicSource and upstreamTopic methods as expressed in javadoc of the
> > > ReplicationPolicy interface.
> > >
> > > That makes us suspect that MirrorMaker2 has been never meant to work
> > > in such a configuration and that our solution may stop
> > > working with future releases of Kafka.
> > >
> > > We would like then to ask for making the possibility of 'prefixless'
> > > topic mirroring in a simplified two-cluster setup a standard
> > > feature of Kafka Mirror Maker 2.
> > >
> >
>
Re: Feature request for 'prefixless' topic mirroring
Posted by Marcin Wieloch <ma...@gmail.com>.
Hello Ryanne,
Thank you for these clarifications. I am glad that the use case of “prefixless” replication is in-line with your vision for further development of MirrorMaker2; I forgot indeed that already KIP-382 mentioned that LegacyReplicationPolicy.
I have not found however any JIRA issue addressing specifically LegacyReplicationPolicy, and the main ticket for KIP-382 (KAFKA-7500) is already resolved.
So what would be the next step you recommend I take? Should I file my feature request (rephrased to adjust for your comments)? Or is the need for the legacy policy already tracked somehow and we don’t have to open any new tickets for it?
Marcin
On 2020/03/11 14:20:44, Ryanne Dolan <ry...@gmail.com> wrote:
> Hello Marcin, actually KIP-382 includes LegacyReplicationPolicy (not yet
> released) which is similar to your PrefixlessReplicationPolicy and the
> MigrationReplicationPolicy mentioned. I don't think these are hacky, though
> maybe the documentation could better address what to do when a method
> doesn't necessarily apply cleanly.
>
> Frankly, I anticipated needing to make changes to MM2 itself to accommodate
> these sort of use-cases -- the fact that you got this working as-is is
> encouraging :)
>
> Re the proposal, I think we could ship one or more of these replication
> policies with MM2 to support prefixless replication. There may be slightly
> different implementations depending on the use-case, i.e. "migration" from
> an old cluster vs active/passive w/o renaming topics vs mimicking MM1 with
> LegacyReplicationPolicy. Maybe they are all identical and we just need one?
>
> I like that use-cases are driving these requests rather than us try to
> enumerate all possible ReplicationPolicy implementations. Given a
> particular use-case and a successful application of a custom
> ReplicationPolicy, I for one would support including it in Apache Kafka.
>
> Ryanne
>
> On Wed, Mar 11, 2020, 5:26 AM Marcin Wieloch <ma...@gmail.com>
> wrote:
>
> > Dear All,
> >
> > I would like to raise a feature request for ‘prefixless’ topic
> > mirroring with MirrorMaker2 (see detailed description below).
> >
> > As after quite a few tries I am not 100% sure, could you please
> > confirm that by now
> > the ‘prefixless’ mirroring is not supported and that my feature
> > request is valid? I would then file it in JIRA.
> >
> > Best regards,
> >
> > Marcin
> >
> >
> > Feature Request
> >
> > We would like to use MirrorMaker2 as a DRP solution for a simple setup
> > with two sites working in active-passive configuration
> > where all topics are always mirrored in one direction only and the
> > downstream topics keep their original names, i.e., without being
> > prefixed by MirrorMaker2.
> >
> > After a few rounds of 'trial and error' we came up with a customized
> > replication policy which, although hacky, seems to fulfill our need;
> > it is presented in stackoverflow here:
> > https://stackoverflow.com/a/60619233
> > and source code is available here:
> >
> > https://github.com/mawileo/mm2-prefixless-policy/blob/feature/prefixless/src/main/java/ch/mawileo/kafka/mm2/PrefixlessReplicationPolicy.java
> >
> > This solution is 'hacky' in that it violates specification for the
> > topicSource and upstreamTopic methods as expressed in javadoc of the
> > ReplicationPolicy interface.
> >
> > That makes us suspect that MirrorMaker2 has been never meant to work
> > in such a configuration and that our solution may stop
> > working with future releases of Kafka.
> >
> > We would like then to ask for making the possibility of 'prefixless'
> > topic mirroring in a simplified two-cluster setup a standard
> > feature of Kafka Mirror Maker 2.
> >
>
Re: Feature request for 'prefixless' topic mirroring
Posted by Ryanne Dolan <ry...@gmail.com>.
Hello Marcin, actually KIP-382 includes LegacyReplicationPolicy (not yet
released) which is similar to your PrefixlessReplicationPolicy and the
MigrationReplicationPolicy mentioned. I don't think these are hacky, though
maybe the documentation could better address what to do when a method
doesn't necessarily apply cleanly.
Frankly, I anticipated needing to make changes to MM2 itself to accommodate
these sort of use-cases -- the fact that you got this working as-is is
encouraging :)
Re the proposal, I think we could ship one or more of these replication
policies with MM2 to support prefixless replication. There may be slightly
different implementations depending on the use-case, i.e. "migration" from
an old cluster vs active/passive w/o renaming topics vs mimicking MM1 with
LegacyReplicationPolicy. Maybe they are all identical and we just need one?
I like that use-cases are driving these requests rather than us try to
enumerate all possible ReplicationPolicy implementations. Given a
particular use-case and a successful application of a custom
ReplicationPolicy, I for one would support including it in Apache Kafka.
Ryanne
On Wed, Mar 11, 2020, 5:26 AM Marcin Wieloch <ma...@gmail.com>
wrote:
> Dear All,
>
> I would like to raise a feature request for ‘prefixless’ topic
> mirroring with MirrorMaker2 (see detailed description below).
>
> As after quite a few tries I am not 100% sure, could you please
> confirm that by now
> the ‘prefixless’ mirroring is not supported and that my feature
> request is valid? I would then file it in JIRA.
>
> Best regards,
>
> Marcin
>
>
> Feature Request
>
> We would like to use MirrorMaker2 as a DRP solution for a simple setup
> with two sites working in active-passive configuration
> where all topics are always mirrored in one direction only and the
> downstream topics keep their original names, i.e., without being
> prefixed by MirrorMaker2.
>
> After a few rounds of 'trial and error' we came up with a customized
> replication policy which, although hacky, seems to fulfill our need;
> it is presented in stackoverflow here:
> https://stackoverflow.com/a/60619233
> and source code is available here:
>
> https://github.com/mawileo/mm2-prefixless-policy/blob/feature/prefixless/src/main/java/ch/mawileo/kafka/mm2/PrefixlessReplicationPolicy.java
>
> This solution is 'hacky' in that it violates specification for the
> topicSource and upstreamTopic methods as expressed in javadoc of the
> ReplicationPolicy interface.
>
> That makes us suspect that MirrorMaker2 has been never meant to work
> in such a configuration and that our solution may stop
> working with future releases of Kafka.
>
> We would like then to ask for making the possibility of 'prefixless'
> topic mirroring in a simplified two-cluster setup a standard
> feature of Kafka Mirror Maker 2.
>