You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Binh Nguyen Van <bi...@gmail.com> on 2015/08/26 05:35:45 UTC
Mirror a partition of a topic
Hi all,
Is there any tool that allows to mirror a partition of a topic?
I looked at the mirror maker but it is used to mirror the whole topic
and I don't see any option that allow me to configure partitions.
I want to mirror our live data to staging environment but my staging
environment
can't handle the whole topic so I am looking for a tool that I can use to
mirror specific
partitions of a topic only.
Thanks
-Binh
Re: Mirror a partition of a topic
Posted by Ashish Singh <as...@cloudera.com>.
Binh,
You can refer to this
<https://cwiki.apache.org/confluence/display/KAFKA/KIP-3+-+Mirror+Maker+Enhancement#KIP-3-MirrorMakerEnhancement-Messagehandlerinconsumerthread>
and KAFKA-1997 <https://issues.apache.org/jira/browse/KAFKA-1997> for
details on getting this done using MirrorMaker’s message handler.
​
On Wed, Aug 26, 2015 at 11:55 AM, Binh Nguyen Van <bi...@gmail.com>
wrote:
> Thank you everyone.
>
> On Tue, Aug 25, 2015 at 11:49 PM, Kishore Senji <ks...@gmail.com> wrote:
>
> > Flume could be an option with an Interceptor although the throughput
> could
> > be less compared to Mirror Maker with compression and shallow iterator
> > enabled.
> > On Tue, Aug 25, 2015 at 10:28 PM tao xiao <xi...@gmail.com> wrote:
> >
> > > In the trunk code mirror maker provides the ability to filter out
> > messages
> > > on demand by supplying a message handler.
> > >
> > >
> > >
> >
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/tools/MirrorMaker.scala#L443
> > >
> > > On Wed, 26 Aug 2015 at 11:35 Binh Nguyen Van <bi...@gmail.com>
> wrote:
> > >
> > > > Hi all,
> > > >
> > > > Is there any tool that allows to mirror a partition of a topic?
> > > > I looked at the mirror maker but it is used to mirror the whole topic
> > > > and I don't see any option that allow me to configure partitions.
> > > >
> > > > I want to mirror our live data to staging environment but my staging
> > > > environment
> > > > can't handle the whole topic so I am looking for a tool that I can
> use
> > to
> > > > mirror specific
> > > > partitions of a topic only.
> > > >
> > > > Thanks
> > > > -Binh
> > > >
> > >
> >
>
--
Regards,
Ashish
Re: Mirror a partition of a topic
Posted by Binh Nguyen Van <bi...@gmail.com>.
Thank you everyone.
On Tue, Aug 25, 2015 at 11:49 PM, Kishore Senji <ks...@gmail.com> wrote:
> Flume could be an option with an Interceptor although the throughput could
> be less compared to Mirror Maker with compression and shallow iterator
> enabled.
> On Tue, Aug 25, 2015 at 10:28 PM tao xiao <xi...@gmail.com> wrote:
>
> > In the trunk code mirror maker provides the ability to filter out
> messages
> > on demand by supplying a message handler.
> >
> >
> >
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/tools/MirrorMaker.scala#L443
> >
> > On Wed, 26 Aug 2015 at 11:35 Binh Nguyen Van <bi...@gmail.com> wrote:
> >
> > > Hi all,
> > >
> > > Is there any tool that allows to mirror a partition of a topic?
> > > I looked at the mirror maker but it is used to mirror the whole topic
> > > and I don't see any option that allow me to configure partitions.
> > >
> > > I want to mirror our live data to staging environment but my staging
> > > environment
> > > can't handle the whole topic so I am looking for a tool that I can use
> to
> > > mirror specific
> > > partitions of a topic only.
> > >
> > > Thanks
> > > -Binh
> > >
> >
>
Re: Mirror a partition of a topic
Posted by Kishore Senji <ks...@gmail.com>.
Flume could be an option with an Interceptor although the throughput could
be less compared to Mirror Maker with compression and shallow iterator
enabled.
On Tue, Aug 25, 2015 at 10:28 PM tao xiao <xi...@gmail.com> wrote:
> In the trunk code mirror maker provides the ability to filter out messages
> on demand by supplying a message handler.
>
>
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/tools/MirrorMaker.scala#L443
>
> On Wed, 26 Aug 2015 at 11:35 Binh Nguyen Van <bi...@gmail.com> wrote:
>
> > Hi all,
> >
> > Is there any tool that allows to mirror a partition of a topic?
> > I looked at the mirror maker but it is used to mirror the whole topic
> > and I don't see any option that allow me to configure partitions.
> >
> > I want to mirror our live data to staging environment but my staging
> > environment
> > can't handle the whole topic so I am looking for a tool that I can use to
> > mirror specific
> > partitions of a topic only.
> >
> > Thanks
> > -Binh
> >
>
Re: Mirror a partition of a topic
Posted by tao xiao <xi...@gmail.com>.
In the trunk code mirror maker provides the ability to filter out messages
on demand by supplying a message handler.
https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/tools/MirrorMaker.scala#L443
On Wed, 26 Aug 2015 at 11:35 Binh Nguyen Van <bi...@gmail.com> wrote:
> Hi all,
>
> Is there any tool that allows to mirror a partition of a topic?
> I looked at the mirror maker but it is used to mirror the whole topic
> and I don't see any option that allow me to configure partitions.
>
> I want to mirror our live data to staging environment but my staging
> environment
> can't handle the whole topic so I am looking for a tool that I can use to
> mirror specific
> partitions of a topic only.
>
> Thanks
> -Binh
>