You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Vishal Santoshi <vi...@gmail.com> on 2020/01/14 03:54:36 UTC
MirrorMaker 2
Can I override the retention on target topics through mm2.properties ? It
should be as simple as stating the retention.ms globally ? Am also
curious whether it can more at a single channel level ?
For example A->B, topic on B should have a retention of x and for B->A the
retention is y..
Is that possible?
Thanks.
Re: MirrorMaker 2
Posted by Vishal Santoshi <vi...@gmail.com>.
Though it seems my attempt to set a topic compression.type on a destination
cluster topic ( note that the source as no compression.type set ) was
promptly over ridden by MM2, which I find a little limiting. I would argue
that though it is a topic level compression some use case primarily if it
is a back up and recovery use case would be smaller cluster... on may be I
could do this using mm2.properties ? but not apparent.
On Sun, Jan 19, 2020 at 7:58 AM Vishal Santoshi <vi...@gmail.com>
wrote:
> Verified.
>
> On Sat, Jan 18, 2020 at 12:40 PM Ryanne Dolan <ry...@gmail.com>
> wrote:
>
>> I think that's right. If there is no per-topic retention configured, Kafka
>> will use the cluster default.
>>
>> Ryanne
>>
>> On Sat, Jan 18, 2020, 10:21 AM Vishal Santoshi <vishal.santoshi@gmail.com
>> >
>> wrote:
>>
>> > Last question .. so if I set the global property on the
>> > destination cluster, would that be overridden by the per topic
>> retention ?
>> > I look at the zk config for the specific topic I do not see the specific
>> > overrides that I generally do so I would assume it is honoring the
>> > destination global retention ?
>> >
>> > On Tue, Jan 14, 2020 at 10:11 PM Vishal Santoshi <
>> > vishal.santoshi@gmail.com>
>> > wrote:
>> >
>> > > Thanks
>> > >
>> > > On Tue, Jan 14, 2020 at 8:47 AM Ryanne Dolan <ry...@gmail.com>
>> > > wrote:
>> > >
>> > >> Take a look at the DefaultConfigPropertyFilter class, which supports
>> > >> customizable blacklists via config.properties.blacklist.
>> > >>
>> > >> Ryanne
>> > >>
>> > >> On Tue, Jan 14, 2020, 6:05 AM Vishal Santoshi <
>> > vishal.santoshi@gmail.com>
>> > >> wrote:
>> > >>
>> > >> > Thank you for the prompt reply, Very much appreciated. I am not
>> sure
>> > >> > disable config sync is an option for us. How do I blacklist
>> > >> retention.ms
>> > >> > ?
>> > >> >
>> > >> > On Tue, Jan 14, 2020 at 12:47 AM Ryanne Dolan <
>> ryannedolan@gmail.com>
>> > >> > wrote:
>> > >> >
>> > >> > > Vishal, there is no support for overriding topic configuration
>> like
>> > >> > > retention. Instead, remote topics will have the same
>> configuration
>> > as
>> > >> > their
>> > >> > > source topics. You could disable config sync or blacklist
>> > >> retention.ms
>> > >> > to
>> > >> > > prevent that from happening, and then alter retention for remote
>> > >> topics
>> > >> > > manually if you need to.
>> > >> > >
>> > >> > > KIP-158 might help with this in the future by allowing you to
>> > control
>> > >> how
>> > >> > > Connect creates topics, to some extent.
>> > >> > >
>> > >> > > Ryanne
>> > >> > >
>> > >> > > On Mon, Jan 13, 2020, 9:55 PM Vishal Santoshi <
>> > >> vishal.santoshi@gmail.com
>> > >> > >
>> > >> > > wrote:
>> > >> > >
>> > >> > > > Can I override the retention on target topics through
>> > >> mm2.properties ?
>> > >> > > It
>> > >> > > > should be as simple as stating the retention.ms globally ? Am
>> > also
>> > >> > > > curious whether it can more at a single channel level ?
>> > >> > > >
>> > >> > > > For example A->B, topic on B should have a retention of x and
>> for
>> > >> B->A
>> > >> > > the
>> > >> > > > retention is y..
>> > >> > > >
>> > >> > > > Is that possible?
>> > >> > > >
>> > >> > > > Thanks.
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>> > >
>> >
>>
>
Re: MirrorMaker 2
Posted by Vishal Santoshi <vi...@gmail.com>.
Verified.
On Sat, Jan 18, 2020 at 12:40 PM Ryanne Dolan <ry...@gmail.com> wrote:
> I think that's right. If there is no per-topic retention configured, Kafka
> will use the cluster default.
>
> Ryanne
>
> On Sat, Jan 18, 2020, 10:21 AM Vishal Santoshi <vi...@gmail.com>
> wrote:
>
> > Last question .. so if I set the global property on the
> > destination cluster, would that be overridden by the per topic retention
> ?
> > I look at the zk config for the specific topic I do not see the specific
> > overrides that I generally do so I would assume it is honoring the
> > destination global retention ?
> >
> > On Tue, Jan 14, 2020 at 10:11 PM Vishal Santoshi <
> > vishal.santoshi@gmail.com>
> > wrote:
> >
> > > Thanks
> > >
> > > On Tue, Jan 14, 2020 at 8:47 AM Ryanne Dolan <ry...@gmail.com>
> > > wrote:
> > >
> > >> Take a look at the DefaultConfigPropertyFilter class, which supports
> > >> customizable blacklists via config.properties.blacklist.
> > >>
> > >> Ryanne
> > >>
> > >> On Tue, Jan 14, 2020, 6:05 AM Vishal Santoshi <
> > vishal.santoshi@gmail.com>
> > >> wrote:
> > >>
> > >> > Thank you for the prompt reply, Very much appreciated. I am not sure
> > >> > disable config sync is an option for us. How do I blacklist
> > >> retention.ms
> > >> > ?
> > >> >
> > >> > On Tue, Jan 14, 2020 at 12:47 AM Ryanne Dolan <
> ryannedolan@gmail.com>
> > >> > wrote:
> > >> >
> > >> > > Vishal, there is no support for overriding topic configuration
> like
> > >> > > retention. Instead, remote topics will have the same configuration
> > as
> > >> > their
> > >> > > source topics. You could disable config sync or blacklist
> > >> retention.ms
> > >> > to
> > >> > > prevent that from happening, and then alter retention for remote
> > >> topics
> > >> > > manually if you need to.
> > >> > >
> > >> > > KIP-158 might help with this in the future by allowing you to
> > control
> > >> how
> > >> > > Connect creates topics, to some extent.
> > >> > >
> > >> > > Ryanne
> > >> > >
> > >> > > On Mon, Jan 13, 2020, 9:55 PM Vishal Santoshi <
> > >> vishal.santoshi@gmail.com
> > >> > >
> > >> > > wrote:
> > >> > >
> > >> > > > Can I override the retention on target topics through
> > >> mm2.properties ?
> > >> > > It
> > >> > > > should be as simple as stating the retention.ms globally ? Am
> > also
> > >> > > > curious whether it can more at a single channel level ?
> > >> > > >
> > >> > > > For example A->B, topic on B should have a retention of x and
> for
> > >> B->A
> > >> > > the
> > >> > > > retention is y..
> > >> > > >
> > >> > > > Is that possible?
> > >> > > >
> > >> > > > Thanks.
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> >
>
Re: MirrorMaker 2
Posted by Ryanne Dolan <ry...@gmail.com>.
I think that's right. If there is no per-topic retention configured, Kafka
will use the cluster default.
Ryanne
On Sat, Jan 18, 2020, 10:21 AM Vishal Santoshi <vi...@gmail.com>
wrote:
> Last question .. so if I set the global property on the
> destination cluster, would that be overridden by the per topic retention ?
> I look at the zk config for the specific topic I do not see the specific
> overrides that I generally do so I would assume it is honoring the
> destination global retention ?
>
> On Tue, Jan 14, 2020 at 10:11 PM Vishal Santoshi <
> vishal.santoshi@gmail.com>
> wrote:
>
> > Thanks
> >
> > On Tue, Jan 14, 2020 at 8:47 AM Ryanne Dolan <ry...@gmail.com>
> > wrote:
> >
> >> Take a look at the DefaultConfigPropertyFilter class, which supports
> >> customizable blacklists via config.properties.blacklist.
> >>
> >> Ryanne
> >>
> >> On Tue, Jan 14, 2020, 6:05 AM Vishal Santoshi <
> vishal.santoshi@gmail.com>
> >> wrote:
> >>
> >> > Thank you for the prompt reply, Very much appreciated. I am not sure
> >> > disable config sync is an option for us. How do I blacklist
> >> retention.ms
> >> > ?
> >> >
> >> > On Tue, Jan 14, 2020 at 12:47 AM Ryanne Dolan <ry...@gmail.com>
> >> > wrote:
> >> >
> >> > > Vishal, there is no support for overriding topic configuration like
> >> > > retention. Instead, remote topics will have the same configuration
> as
> >> > their
> >> > > source topics. You could disable config sync or blacklist
> >> retention.ms
> >> > to
> >> > > prevent that from happening, and then alter retention for remote
> >> topics
> >> > > manually if you need to.
> >> > >
> >> > > KIP-158 might help with this in the future by allowing you to
> control
> >> how
> >> > > Connect creates topics, to some extent.
> >> > >
> >> > > Ryanne
> >> > >
> >> > > On Mon, Jan 13, 2020, 9:55 PM Vishal Santoshi <
> >> vishal.santoshi@gmail.com
> >> > >
> >> > > wrote:
> >> > >
> >> > > > Can I override the retention on target topics through
> >> mm2.properties ?
> >> > > It
> >> > > > should be as simple as stating the retention.ms globally ? Am
> also
> >> > > > curious whether it can more at a single channel level ?
> >> > > >
> >> > > > For example A->B, topic on B should have a retention of x and for
> >> B->A
> >> > > the
> >> > > > retention is y..
> >> > > >
> >> > > > Is that possible?
> >> > > >
> >> > > > Thanks.
> >> > > >
> >> > >
> >> >
> >>
> >
>
Re: MirrorMaker 2
Posted by Vishal Santoshi <vi...@gmail.com>.
Last question .. so if I set the global property on the
destination cluster, would that be overridden by the per topic retention ?
I look at the zk config for the specific topic I do not see the specific
overrides that I generally do so I would assume it is honoring the
destination global retention ?
On Tue, Jan 14, 2020 at 10:11 PM Vishal Santoshi <vi...@gmail.com>
wrote:
> Thanks
>
> On Tue, Jan 14, 2020 at 8:47 AM Ryanne Dolan <ry...@gmail.com>
> wrote:
>
>> Take a look at the DefaultConfigPropertyFilter class, which supports
>> customizable blacklists via config.properties.blacklist.
>>
>> Ryanne
>>
>> On Tue, Jan 14, 2020, 6:05 AM Vishal Santoshi <vi...@gmail.com>
>> wrote:
>>
>> > Thank you for the prompt reply, Very much appreciated. I am not sure
>> > disable config sync is an option for us. How do I blacklist
>> retention.ms
>> > ?
>> >
>> > On Tue, Jan 14, 2020 at 12:47 AM Ryanne Dolan <ry...@gmail.com>
>> > wrote:
>> >
>> > > Vishal, there is no support for overriding topic configuration like
>> > > retention. Instead, remote topics will have the same configuration as
>> > their
>> > > source topics. You could disable config sync or blacklist
>> retention.ms
>> > to
>> > > prevent that from happening, and then alter retention for remote
>> topics
>> > > manually if you need to.
>> > >
>> > > KIP-158 might help with this in the future by allowing you to control
>> how
>> > > Connect creates topics, to some extent.
>> > >
>> > > Ryanne
>> > >
>> > > On Mon, Jan 13, 2020, 9:55 PM Vishal Santoshi <
>> vishal.santoshi@gmail.com
>> > >
>> > > wrote:
>> > >
>> > > > Can I override the retention on target topics through
>> mm2.properties ?
>> > > It
>> > > > should be as simple as stating the retention.ms globally ? Am also
>> > > > curious whether it can more at a single channel level ?
>> > > >
>> > > > For example A->B, topic on B should have a retention of x and for
>> B->A
>> > > the
>> > > > retention is y..
>> > > >
>> > > > Is that possible?
>> > > >
>> > > > Thanks.
>> > > >
>> > >
>> >
>>
>
Re: MirrorMaker 2
Posted by Vishal Santoshi <vi...@gmail.com>.
Thanks
On Tue, Jan 14, 2020 at 8:47 AM Ryanne Dolan <ry...@gmail.com> wrote:
> Take a look at the DefaultConfigPropertyFilter class, which supports
> customizable blacklists via config.properties.blacklist.
>
> Ryanne
>
> On Tue, Jan 14, 2020, 6:05 AM Vishal Santoshi <vi...@gmail.com>
> wrote:
>
> > Thank you for the prompt reply, Very much appreciated. I am not sure
> > disable config sync is an option for us. How do I blacklist
> retention.ms
> > ?
> >
> > On Tue, Jan 14, 2020 at 12:47 AM Ryanne Dolan <ry...@gmail.com>
> > wrote:
> >
> > > Vishal, there is no support for overriding topic configuration like
> > > retention. Instead, remote topics will have the same configuration as
> > their
> > > source topics. You could disable config sync or blacklist retention.ms
> > to
> > > prevent that from happening, and then alter retention for remote topics
> > > manually if you need to.
> > >
> > > KIP-158 might help with this in the future by allowing you to control
> how
> > > Connect creates topics, to some extent.
> > >
> > > Ryanne
> > >
> > > On Mon, Jan 13, 2020, 9:55 PM Vishal Santoshi <
> vishal.santoshi@gmail.com
> > >
> > > wrote:
> > >
> > > > Can I override the retention on target topics through mm2.properties
> ?
> > > It
> > > > should be as simple as stating the retention.ms globally ? Am also
> > > > curious whether it can more at a single channel level ?
> > > >
> > > > For example A->B, topic on B should have a retention of x and for
> B->A
> > > the
> > > > retention is y..
> > > >
> > > > Is that possible?
> > > >
> > > > Thanks.
> > > >
> > >
> >
>
Re: MirrorMaker 2
Posted by Ryanne Dolan <ry...@gmail.com>.
Take a look at the DefaultConfigPropertyFilter class, which supports
customizable blacklists via config.properties.blacklist.
Ryanne
On Tue, Jan 14, 2020, 6:05 AM Vishal Santoshi <vi...@gmail.com>
wrote:
> Thank you for the prompt reply, Very much appreciated. I am not sure
> disable config sync is an option for us. How do I blacklist retention.ms
> ?
>
> On Tue, Jan 14, 2020 at 12:47 AM Ryanne Dolan <ry...@gmail.com>
> wrote:
>
> > Vishal, there is no support for overriding topic configuration like
> > retention. Instead, remote topics will have the same configuration as
> their
> > source topics. You could disable config sync or blacklist retention.ms
> to
> > prevent that from happening, and then alter retention for remote topics
> > manually if you need to.
> >
> > KIP-158 might help with this in the future by allowing you to control how
> > Connect creates topics, to some extent.
> >
> > Ryanne
> >
> > On Mon, Jan 13, 2020, 9:55 PM Vishal Santoshi <vishal.santoshi@gmail.com
> >
> > wrote:
> >
> > > Can I override the retention on target topics through mm2.properties ?
> > It
> > > should be as simple as stating the retention.ms globally ? Am also
> > > curious whether it can more at a single channel level ?
> > >
> > > For example A->B, topic on B should have a retention of x and for B->A
> > the
> > > retention is y..
> > >
> > > Is that possible?
> > >
> > > Thanks.
> > >
> >
>
Re: MirrorMaker 2
Posted by Vishal Santoshi <vi...@gmail.com>.
Thank you for the prompt reply, Very much appreciated. I am not sure
disable config sync is an option for us. How do I blacklist retention.ms ?
On Tue, Jan 14, 2020 at 12:47 AM Ryanne Dolan <ry...@gmail.com> wrote:
> Vishal, there is no support for overriding topic configuration like
> retention. Instead, remote topics will have the same configuration as their
> source topics. You could disable config sync or blacklist retention.ms to
> prevent that from happening, and then alter retention for remote topics
> manually if you need to.
>
> KIP-158 might help with this in the future by allowing you to control how
> Connect creates topics, to some extent.
>
> Ryanne
>
> On Mon, Jan 13, 2020, 9:55 PM Vishal Santoshi <vi...@gmail.com>
> wrote:
>
> > Can I override the retention on target topics through mm2.properties ?
> It
> > should be as simple as stating the retention.ms globally ? Am also
> > curious whether it can more at a single channel level ?
> >
> > For example A->B, topic on B should have a retention of x and for B->A
> the
> > retention is y..
> >
> > Is that possible?
> >
> > Thanks.
> >
>
Re: MirrorMaker 2
Posted by Ryanne Dolan <ry...@gmail.com>.
Vishal, there is no support for overriding topic configuration like
retention. Instead, remote topics will have the same configuration as their
source topics. You could disable config sync or blacklist retention.ms to
prevent that from happening, and then alter retention for remote topics
manually if you need to.
KIP-158 might help with this in the future by allowing you to control how
Connect creates topics, to some extent.
Ryanne
On Mon, Jan 13, 2020, 9:55 PM Vishal Santoshi <vi...@gmail.com>
wrote:
> Can I override the retention on target topics through mm2.properties ? It
> should be as simple as stating the retention.ms globally ? Am also
> curious whether it can more at a single channel level ?
>
> For example A->B, topic on B should have a retention of x and for B->A the
> retention is y..
>
> Is that possible?
>
> Thanks.
>