You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by tao xiao <xi...@gmail.com> on 2015/03/10 15:52:05 UTC
Does consumer support combination of whitelist and blacklist topic filtering
Hi,
I have an user case where I need to consume a list topics with name that
matches pattern topic.* except for one that is topic.10. Is there a way
that I can combine the use of whitelist and blacklist so that I can achieve
something like accept all topics with regex topic.* but exclude topic.10?
--
Regards,
Tao
Re: Does consumer support combination of whitelist and blacklist
topic filtering
Posted by Mayuresh Gharat <gh...@gmail.com>.
You will have to use wildcard iterator I suppose.
Thanks,
Mayuresh
On Tue, Mar 10, 2015 at 7:58 AM, tao xiao <xi...@gmail.com> wrote:
> I actually mean if we can achieve this in mirror maker.
>
> On Tue, Mar 10, 2015 at 10:52 PM, tao xiao <xi...@gmail.com> wrote:
>
> > Hi,
> >
> > I have an user case where I need to consume a list topics with name that
> > matches pattern topic.* except for one that is topic.10. Is there a way
> > that I can combine the use of whitelist and blacklist so that I can
> achieve
> > something like accept all topics with regex topic.* but exclude topic.10?
> >
> > --
> > Regards,
> > Tao
> >
>
>
>
> --
> Regards,
> Tao
>
--
-Regards,
Mayuresh R. Gharat
(862) 250-7125
Re: Does consumer support combination of whitelist and blacklist
topic filtering
Posted by tao xiao <xi...@gmail.com>.
A little more context about my needs: I have a requirement that I need to
start/stop a topic at runtime based on a event sent to MM. at the moment I
need to bounce the MM and find a way to exclude the topic from whitelist
which is not an easy job with regex. If I can pass in a combination of
blacklist and whitelist I can easily achieve this by having something like
--whitelist topic.* --blacklist topic.1
On Thu, Mar 12, 2015 at 9:10 PM, tao xiao <xi...@gmail.com> wrote:
> something like dynamic filtering that can be updated at runtime or deny
> all but allow a certain set of topics that cannot be specified easily by
> regex
>
> On Thu, Mar 12, 2015 at 9:06 PM, Guozhang Wang <wa...@gmail.com> wrote:
>
>> Hmm, what kind of customized filtering do you have in mind? I thought with
>> "--whitelist" you could already specify regex to do filtering.
>>
>> On Thu, Mar 12, 2015 at 5:56 AM, tao xiao <xi...@gmail.com> wrote:
>>
>> > Hi Guozhang,
>> >
>> > I was meant to be topicfilter not topic-count. sorry for the confusion.
>> > What I want to achieve is to pass my own customized topicfilter to MM so
>> > that I can filter out topics what ever I like. I know MM doesn't support
>> > this now. I am just thinking if this is a good feature to add in
>> >
>> > On Thu, Mar 12, 2015 at 8:24 PM, Guozhang Wang <wa...@gmail.com>
>> wrote:
>> >
>> > > Hi Tao,
>> > >
>> > > Sorry I was mistaken before, yes in MM you can only directly specify
>> > > "--whitelist", "--blacklist", and the number of streams you want to
>> > create
>> > > via "--num.streams", but cannot set specific topic-count. This is
>> because
>> > > MM is mainly used for cross DC replication, and hence usually will
>> pipe
>> > all
>> > > topics or a majority of the topics from the source cluster to the
>> > > destination, hence usually you do not care about "some topics should
>> get
>> > X
>> > > streams, while some other topics should get Y streams".
>> > >
>> > > Guozhang
>> > >
>> > > On Wed, Mar 11, 2015 at 11:59 PM, tao xiao <xi...@gmail.com>
>> wrote:
>> > >
>> > > > The topic list is not specified in consumer.properties and I don't
>> > think
>> > > > there is any property in consumer config that allows us to specify
>> what
>> > > > topics we want to consume. Can you point me to the property if
>> there is
>> > > > any?
>> > > >
>> > > > On Thu, Mar 12, 2015 at 12:14 AM, Guozhang Wang <wangguoz@gmail.com
>> >
>> > > > wrote:
>> > > >
>> > > > > Tao,
>> > > > >
>> > > > > In MM people can pass in consumer configs, in which people can
>> > specify
>> > > > > consumption topics, either in regular topic list format or
>> whitelist
>> > /
>> > > > > blacklist. So I think it already does what you need?
>> > > > >
>> > > > > Guozhang
>> > > > >
>> > > > > On Tue, Mar 10, 2015 at 10:09 PM, tao xiao <xi...@gmail.com>
>> > > wrote:
>> > > > >
>> > > > > > Thank you guys for answering. I think it will be good that we
>> can
>> > > pass
>> > > > > in a
>> > > > > > customised topicCount ( I think this is the interface whitelist
>> and
>> > > > > > backlist implement if I am not mistaken) to MM to achieve
>> similar
>> > > thing
>> > > > > >
>> > > > > > On Wednesday, March 11, 2015, Guozhang Wang <wangguoz@gmail.com
>> >
>> > > > wrote:
>> > > > > >
>> > > > > > > Hi Tao,
>> > > > > > >
>> > > > > > > Unfortunately MM does not support whitelist / blacklist at the
>> > same
>> > > > > time,
>> > > > > > > and you have to choose either one upon initialization. As for
>> > your
>> > > > > case,
>> > > > > > I
>> > > > > > > think it can be captured by some reg-ex to exclude nothing
>> else
>> > but
>> > > > > "10",
>> > > > > > > but I do not know the exact expression.
>> > > > > > >
>> > > > > > > Guozhang
>> > > > > > >
>> > > > > > > On Tue, Mar 10, 2015 at 7:58 AM, tao xiao <
>> xiaotao183@gmail.com
>> > > > > > > <javascript:;>> wrote:
>> > > > > > >
>> > > > > > > > I actually mean if we can achieve this in mirror maker.
>> > > > > > > >
>> > > > > > > > On Tue, Mar 10, 2015 at 10:52 PM, tao xiao <
>> > xiaotao183@gmail.com
>> > > > > > > <javascript:;>> wrote:
>> > > > > > > >
>> > > > > > > > > Hi,
>> > > > > > > > >
>> > > > > > > > > I have an user case where I need to consume a list topics
>> > with
>> > > > name
>> > > > > > > that
>> > > > > > > > > matches pattern topic.* except for one that is topic.10.
>> Is
>> > > > there a
>> > > > > > way
>> > > > > > > > > that I can combine the use of whitelist and blacklist so
>> > that I
>> > > > can
>> > > > > > > > achieve
>> > > > > > > > > something like accept all topics with regex topic.* but
>> > exclude
>> > > > > > > topic.10?
>> > > > > > > > >
>> > > > > > > > > --
>> > > > > > > > > Regards,
>> > > > > > > > > Tao
>> > > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > --
>> > > > > > > > Regards,
>> > > > > > > > Tao
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > --
>> > > > > > > -- Guozhang
>> > > > > > >
>> > > > > >
>> > > > > >
>> > > > > > --
>> > > > > > Regards,
>> > > > > > Tao
>> > > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > --
>> > > > > -- Guozhang
>> > > > >
>> > > >
>> > > >
>> > > >
>> > > > --
>> > > > Regards,
>> > > > Tao
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > > -- Guozhang
>> > >
>> >
>> >
>> >
>> > --
>> > Regards,
>> > Tao
>> >
>>
>>
>>
>> --
>> -- Guozhang
>>
>
>
>
> --
> Regards,
> Tao
>
--
Regards,
Tao
Re: Does consumer support combination of whitelist and blacklist
topic filtering
Posted by Jiangjie Qin <jq...@linkedin.com.INVALID>.
Actually new MM will commit offsets even if those messages are filtered
out. That¹s why I¹m asking will you resume consuming from a topic after
you stop consuming from it earlier. If you are going to do this, you need
do extra work in your message handler. For example,
1. When received a message that triggers stopping consumption from a
topic, store the offset in message handler and start filter out the topic.
2. When received a message that triggers resuming consumption from a
topic, reset the offset for that topic to the stored offset, and stop
filtering.
This might need some tricks if you have multiple MM instances for
synchronization.
Thanks.
Jiangjie (Becket) Qin
On 3/12/15, 10:23 PM, "tao xiao" <xi...@gmail.com> wrote:
>I am not sure how MM is going to be rewritten. Based on the current
>implementation in trunk offset is not committed unless it is produced to
>destination. With assumption that this logic remains MM will not
>acknowledge the offset back to source for filtered message. So I think it
>is safe to filter messages out while keeping committed offset unchanged
>for
>that particular topic. Please correct me if I am wrong
>
>On Fri, Mar 13, 2015 at 1:12 PM, Guozhang Wang <wa...@gmail.com> wrote:
>
>> Note that with filtering in message handler, records from the source
>> cluster are still considered as "consumed" since the offsets will be
>> committed. If you change the filtering dynamically back to whilelist
>>these
>> topics, you will lose the data that gets consumed during the period of
>>the
>> blacklist.
>>
>> Guozhang
>>
>> On Thu, Mar 12, 2015 at 10:01 PM, tao xiao <xi...@gmail.com> wrote:
>>
>> > Yes, that will work. message handle can filter out message sent from
>> > certain topics
>> >
>> > On Fri, Mar 13, 2015 at 6:30 AM, Jiangjie Qin
>><jqin@linkedin.com.invalid
>> >
>> > wrote:
>> >
>> > > No sure if it is an option. But does filtering out topics with
>>message
>> > > handler works for you? Are you going to resume consuming from a
>>topic
>> > > after you stop consuming from it?
>> > >
>> > > Jiangjie (Becket) Qin
>> > >
>> > > On 3/12/15, 8:05 AM, "tao xiao" <xi...@gmail.com> wrote:
>> > >
>> > > >Yes, you are right. a dynamic topicfilter is more appropriate
>>where I
>> > can
>> > > >filter topics at runtime via some kind of interface e.g. JMX
>> > > >
>> > > >On Thu, Mar 12, 2015 at 11:03 PM, Guozhang Wang
>><wa...@gmail.com>
>> > > >wrote:
>> > > >
>> > > >> Tao,
>> > > >>
>> > > >> Based on your description I think the combination of whitelist /
>> > > >>blacklist
>> > > >> will not achieve your goal, since it is still static.
>> > > >>
>> > > >> Guozhang
>> > > >>
>> > > >> On Thu, Mar 12, 2015 at 6:30 AM, tao xiao <xi...@gmail.com>
>> > wrote:
>> > > >>
>> > > >> > Thank you Guozhang for your advice. A dynamic topic filter is
>> what I
>> > > >>need
>> > > >> > so that I can stop a topic consumption when I need to at
>>runtime.
>> > > >> >
>> > > >> > On Thu, Mar 12, 2015 at 9:21 PM, Guozhang Wang <
>> wangguoz@gmail.com>
>> > > >> wrote:
>> > > >> >
>> > > >> > > 1. Dynamic: yeah that is sth. we could think of, this could
>>be
>> > > >>useful
>> > > >> > > operationally.
>> > > >> > > 2. Regex: I think in terms of expressiveness it should be
>> > sufficient
>> > > >> for
>> > > >> > > almost all subset of topics. In practice usually the rule of
>> thumb
>> > > >>is
>> > > >> > that
>> > > >> > > you will create your topics that belongs to the same "group"
>> with
>> > > >>some
>> > > >> > > prefix / suffix so that regex expression would not be crazily
>> > long.
>> > > >> > >
>> > > >> > > Guozhang
>> > > >> > >
>> > > >> > > On Thu, Mar 12, 2015 at 6:10 AM, tao xiao
>><xiaotao183@gmail.com
>> >
>> > > >> wrote:
>> > > >> > >
>> > > >> > > > something like dynamic filtering that can be updated at
>> runtime
>> > or
>> > > >> deny
>> > > >> > > all
>> > > >> > > > but allow a certain set of topics that cannot be specified
>> > easily
>> > > >>by
>> > > >> > > regex
>> > > >> > > >
>> > > >> > > > On Thu, Mar 12, 2015 at 9:06 PM, Guozhang Wang
>> > > >><wa...@gmail.com>
>> > > >> > > wrote:
>> > > >> > > >
>> > > >> > > > > Hmm, what kind of customized filtering do you have in
>>mind?
>> I
>> > > >> thought
>> > > >> > > > with
>> > > >> > > > > "--whitelist" you could already specify regex to do
>> filtering.
>> > > >> > > > >
>> > > >> > > > > On Thu, Mar 12, 2015 at 5:56 AM, tao xiao <
>> > xiaotao183@gmail.com
>> > > >
>> > > >> > > wrote:
>> > > >> > > > >
>> > > >> > > > > > Hi Guozhang,
>> > > >> > > > > >
>> > > >> > > > > > I was meant to be topicfilter not topic-count. sorry
>>for
>> the
>> > > >> > > confusion.
>> > > >> > > > > > What I want to achieve is to pass my own customized
>> > > >>topicfilter
>> > > >> to
>> > > >> > MM
>> > > >> > > > so
>> > > >> > > > > > that I can filter out topics what ever I like. I know
>>MM
>> > > >>doesn't
>> > > >> > > > support
>> > > >> > > > > > this now. I am just thinking if this is a good feature
>>to
>> > add
>> > > >>in
>> > > >> > > > > >
>> > > >> > > > > > On Thu, Mar 12, 2015 at 8:24 PM, Guozhang Wang <
>> > > >> wangguoz@gmail.com
>> > > >> > >
>> > > >> > > > > wrote:
>> > > >> > > > > >
>> > > >> > > > > > > Hi Tao,
>> > > >> > > > > > >
>> > > >> > > > > > > Sorry I was mistaken before, yes in MM you can only
>> > directly
>> > > >> > > specify
>> > > >> > > > > > > "--whitelist", "--blacklist", and the number of
>>streams
>> > you
>> > > >> want
>> > > >> > > to
>> > > >> > > > > > create
>> > > >> > > > > > > via "--num.streams", but cannot set specific
>> topic-count.
>> > > >>This
>> > > >> is
>> > > >> > > > > because
>> > > >> > > > > > > MM is mainly used for cross DC replication, and hence
>> > > >>usually
>> > > >> > will
>> > > >> > > > pipe
>> > > >> > > > > > all
>> > > >> > > > > > > topics or a majority of the topics from the source
>> cluster
>> > > >>to
>> > > >> the
>> > > >> > > > > > > destination, hence usually you do not care about
>>"some
>> > > >>topics
>> > > >> > > should
>> > > >> > > > > get
>> > > >> > > > > > X
>> > > >> > > > > > > streams, while some other topics should get Y
>>streams".
>> > > >> > > > > > >
>> > > >> > > > > > > Guozhang
>> > > >> > > > > > >
>> > > >> > > > > > > On Wed, Mar 11, 2015 at 11:59 PM, tao xiao <
>> > > >> xiaotao183@gmail.com
>> > > >> > >
>> > > >> > > > > wrote:
>> > > >> > > > > > >
>> > > >> > > > > > > > The topic list is not specified in
>>consumer.properties
>> > > >>and I
>> > > >> > > don't
>> > > >> > > > > > think
>> > > >> > > > > > > > there is any property in consumer config that
>>allows
>> us
>> > to
>> > > >> > > specify
>> > > >> > > > > what
>> > > >> > > > > > > > topics we want to consume. Can you point me to the
>> > > >>property
>> > > >> if
>> > > >> > > > there
>> > > >> > > > > is
>> > > >> > > > > > > > any?
>> > > >> > > > > > > >
>> > > >> > > > > > > > On Thu, Mar 12, 2015 at 12:14 AM, Guozhang Wang <
>> > > >> > > > wangguoz@gmail.com>
>> > > >> > > > > > > > wrote:
>> > > >> > > > > > > >
>> > > >> > > > > > > > > Tao,
>> > > >> > > > > > > > >
>> > > >> > > > > > > > > In MM people can pass in consumer configs, in
>>which
>> > > >>people
>> > > >> > can
>> > > >> > > > > > specify
>> > > >> > > > > > > > > consumption topics, either in regular topic list
>> > format
>> > > >>or
>> > > >> > > > > whitelist
>> > > >> > > > > > /
>> > > >> > > > > > > > > blacklist. So I think it already does what you
>>need?
>> > > >> > > > > > > > >
>> > > >> > > > > > > > > Guozhang
>> > > >> > > > > > > > >
>> > > >> > > > > > > > > On Tue, Mar 10, 2015 at 10:09 PM, tao xiao <
>> > > >> > > xiaotao183@gmail.com
>> > > >> > > > >
>> > > >> > > > > > > wrote:
>> > > >> > > > > > > > >
>> > > >> > > > > > > > > > Thank you guys for answering. I think it will
>>be
>> > good
>> > > >> that
>> > > >> > we
>> > > >> > > > can
>> > > >> > > > > > > pass
>> > > >> > > > > > > > > in a
>> > > >> > > > > > > > > > customised topicCount ( I think this is the
>> > interface
>> > > >> > > whitelist
>> > > >> > > > > and
>> > > >> > > > > > > > > > backlist implement if I am not mistaken) to MM
>>to
>> > > >>achieve
>> > > >> > > > similar
>> > > >> > > > > > > thing
>> > > >> > > > > > > > > >
>> > > >> > > > > > > > > > On Wednesday, March 11, 2015, Guozhang Wang <
>> > > >> > > > wangguoz@gmail.com>
>> > > >> > > > > > > > wrote:
>> > > >> > > > > > > > > >
>> > > >> > > > > > > > > > > Hi Tao,
>> > > >> > > > > > > > > > >
>> > > >> > > > > > > > > > > Unfortunately MM does not support whitelist /
>> > > >>blacklist
>> > > >> > at
>> > > >> > > > the
>> > > >> > > > > > same
>> > > >> > > > > > > > > time,
>> > > >> > > > > > > > > > > and you have to choose either one upon
>> > > >>initialization.
>> > > >> As
>> > > >> > > for
>> > > >> > > > > > your
>> > > >> > > > > > > > > case,
>> > > >> > > > > > > > > > I
>> > > >> > > > > > > > > > > think it can be captured by some reg-ex to
>> exclude
>> > > >> > nothing
>> > > >> > > > else
>> > > >> > > > > > but
>> > > >> > > > > > > > > "10",
>> > > >> > > > > > > > > > > but I do not know the exact expression.
>> > > >> > > > > > > > > > >
>> > > >> > > > > > > > > > > Guozhang
>> > > >> > > > > > > > > > >
>> > > >> > > > > > > > > > > On Tue, Mar 10, 2015 at 7:58 AM, tao xiao <
>> > > >> > > > > xiaotao183@gmail.com
>> > > >> > > > > > > > > > > <javascript:;>> wrote:
>> > > >> > > > > > > > > > >
>> > > >> > > > > > > > > > > > I actually mean if we can achieve this in
>> mirror
>> > > >> maker.
>> > > >> > > > > > > > > > > >
>> > > >> > > > > > > > > > > > On Tue, Mar 10, 2015 at 10:52 PM, tao xiao
>><
>> > > >> > > > > > xiaotao183@gmail.com
>> > > >> > > > > > > > > > > <javascript:;>> wrote:
>> > > >> > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > Hi,
>> > > >> > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > I have an user case where I need to
>>consume
>> a
>> > > >>list
>> > > >> > > topics
>> > > >> > > > > > with
>> > > >> > > > > > > > name
>> > > >> > > > > > > > > > > that
>> > > >> > > > > > > > > > > > > matches pattern topic.* except for one
>>that
>> is
>> > > >> > > topic.10.
>> > > >> > > > Is
>> > > >> > > > > > > > there a
>> > > >> > > > > > > > > > way
>> > > >> > > > > > > > > > > > > that I can combine the use of whitelist
>>and
>> > > >> blacklist
>> > > >> > > so
>> > > >> > > > > > that I
>> > > >> > > > > > > > can
>> > > >> > > > > > > > > > > > achieve
>> > > >> > > > > > > > > > > > > something like accept all topics with
>>regex
>> > > >>topic.*
>> > > >> > but
>> > > >> > > > > > exclude
>> > > >> > > > > > > > > > > topic.10?
>> > > >> > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > --
>> > > >> > > > > > > > > > > > > Regards,
>> > > >> > > > > > > > > > > > > Tao
>> > > >> > > > > > > > > > > > >
>> > > >> > > > > > > > > > > >
>> > > >> > > > > > > > > > > >
>> > > >> > > > > > > > > > > >
>> > > >> > > > > > > > > > > > --
>> > > >> > > > > > > > > > > > Regards,
>> > > >> > > > > > > > > > > > Tao
>> > > >> > > > > > > > > > > >
>> > > >> > > > > > > > > > >
>> > > >> > > > > > > > > > >
>> > > >> > > > > > > > > > >
>> > > >> > > > > > > > > > > --
>> > > >> > > > > > > > > > > -- Guozhang
>> > > >> > > > > > > > > > >
>> > > >> > > > > > > > > >
>> > > >> > > > > > > > > >
>> > > >> > > > > > > > > > --
>> > > >> > > > > > > > > > Regards,
>> > > >> > > > > > > > > > Tao
>> > > >> > > > > > > > > >
>> > > >> > > > > > > > >
>> > > >> > > > > > > > >
>> > > >> > > > > > > > >
>> > > >> > > > > > > > > --
>> > > >> > > > > > > > > -- Guozhang
>> > > >> > > > > > > > >
>> > > >> > > > > > > >
>> > > >> > > > > > > >
>> > > >> > > > > > > >
>> > > >> > > > > > > > --
>> > > >> > > > > > > > Regards,
>> > > >> > > > > > > > Tao
>> > > >> > > > > > > >
>> > > >> > > > > > >
>> > > >> > > > > > >
>> > > >> > > > > > >
>> > > >> > > > > > > --
>> > > >> > > > > > > -- Guozhang
>> > > >> > > > > > >
>> > > >> > > > > >
>> > > >> > > > > >
>> > > >> > > > > >
>> > > >> > > > > > --
>> > > >> > > > > > Regards,
>> > > >> > > > > > Tao
>> > > >> > > > > >
>> > > >> > > > >
>> > > >> > > > >
>> > > >> > > > >
>> > > >> > > > > --
>> > > >> > > > > -- Guozhang
>> > > >> > > > >
>> > > >> > > >
>> > > >> > > >
>> > > >> > > >
>> > > >> > > > --
>> > > >> > > > Regards,
>> > > >> > > > Tao
>> > > >> > > >
>> > > >> > >
>> > > >> > >
>> > > >> > >
>> > > >> > > --
>> > > >> > > -- Guozhang
>> > > >> > >
>> > > >> >
>> > > >> >
>> > > >> >
>> > > >> > --
>> > > >> > Regards,
>> > > >> > Tao
>> > > >> >
>> > > >>
>> > > >>
>> > > >>
>> > > >> --
>> > > >> -- Guozhang
>> > > >>
>> > > >
>> > > >
>> > > >
>> > > >--
>> > > >Regards,
>> > > >Tao
>> > >
>> > >
>> >
>> >
>> > --
>> > Regards,
>> > Tao
>> >
>>
>>
>>
>> --
>> -- Guozhang
>>
>
>
>
>--
>Regards,
>Tao
Re: Does consumer support combination of whitelist and blacklist
topic filtering
Posted by tao xiao <xi...@gmail.com>.
I am not sure how MM is going to be rewritten. Based on the current
implementation in trunk offset is not committed unless it is produced to
destination. With assumption that this logic remains MM will not
acknowledge the offset back to source for filtered message. So I think it
is safe to filter messages out while keeping committed offset unchanged for
that particular topic. Please correct me if I am wrong
On Fri, Mar 13, 2015 at 1:12 PM, Guozhang Wang <wa...@gmail.com> wrote:
> Note that with filtering in message handler, records from the source
> cluster are still considered as "consumed" since the offsets will be
> committed. If you change the filtering dynamically back to whilelist these
> topics, you will lose the data that gets consumed during the period of the
> blacklist.
>
> Guozhang
>
> On Thu, Mar 12, 2015 at 10:01 PM, tao xiao <xi...@gmail.com> wrote:
>
> > Yes, that will work. message handle can filter out message sent from
> > certain topics
> >
> > On Fri, Mar 13, 2015 at 6:30 AM, Jiangjie Qin <jqin@linkedin.com.invalid
> >
> > wrote:
> >
> > > No sure if it is an option. But does filtering out topics with message
> > > handler works for you? Are you going to resume consuming from a topic
> > > after you stop consuming from it?
> > >
> > > Jiangjie (Becket) Qin
> > >
> > > On 3/12/15, 8:05 AM, "tao xiao" <xi...@gmail.com> wrote:
> > >
> > > >Yes, you are right. a dynamic topicfilter is more appropriate where I
> > can
> > > >filter topics at runtime via some kind of interface e.g. JMX
> > > >
> > > >On Thu, Mar 12, 2015 at 11:03 PM, Guozhang Wang <wa...@gmail.com>
> > > >wrote:
> > > >
> > > >> Tao,
> > > >>
> > > >> Based on your description I think the combination of whitelist /
> > > >>blacklist
> > > >> will not achieve your goal, since it is still static.
> > > >>
> > > >> Guozhang
> > > >>
> > > >> On Thu, Mar 12, 2015 at 6:30 AM, tao xiao <xi...@gmail.com>
> > wrote:
> > > >>
> > > >> > Thank you Guozhang for your advice. A dynamic topic filter is
> what I
> > > >>need
> > > >> > so that I can stop a topic consumption when I need to at runtime.
> > > >> >
> > > >> > On Thu, Mar 12, 2015 at 9:21 PM, Guozhang Wang <
> wangguoz@gmail.com>
> > > >> wrote:
> > > >> >
> > > >> > > 1. Dynamic: yeah that is sth. we could think of, this could be
> > > >>useful
> > > >> > > operationally.
> > > >> > > 2. Regex: I think in terms of expressiveness it should be
> > sufficient
> > > >> for
> > > >> > > almost all subset of topics. In practice usually the rule of
> thumb
> > > >>is
> > > >> > that
> > > >> > > you will create your topics that belongs to the same "group"
> with
> > > >>some
> > > >> > > prefix / suffix so that regex expression would not be crazily
> > long.
> > > >> > >
> > > >> > > Guozhang
> > > >> > >
> > > >> > > On Thu, Mar 12, 2015 at 6:10 AM, tao xiao <xiaotao183@gmail.com
> >
> > > >> wrote:
> > > >> > >
> > > >> > > > something like dynamic filtering that can be updated at
> runtime
> > or
> > > >> deny
> > > >> > > all
> > > >> > > > but allow a certain set of topics that cannot be specified
> > easily
> > > >>by
> > > >> > > regex
> > > >> > > >
> > > >> > > > On Thu, Mar 12, 2015 at 9:06 PM, Guozhang Wang
> > > >><wa...@gmail.com>
> > > >> > > wrote:
> > > >> > > >
> > > >> > > > > Hmm, what kind of customized filtering do you have in mind?
> I
> > > >> thought
> > > >> > > > with
> > > >> > > > > "--whitelist" you could already specify regex to do
> filtering.
> > > >> > > > >
> > > >> > > > > On Thu, Mar 12, 2015 at 5:56 AM, tao xiao <
> > xiaotao183@gmail.com
> > > >
> > > >> > > wrote:
> > > >> > > > >
> > > >> > > > > > Hi Guozhang,
> > > >> > > > > >
> > > >> > > > > > I was meant to be topicfilter not topic-count. sorry for
> the
> > > >> > > confusion.
> > > >> > > > > > What I want to achieve is to pass my own customized
> > > >>topicfilter
> > > >> to
> > > >> > MM
> > > >> > > > so
> > > >> > > > > > that I can filter out topics what ever I like. I know MM
> > > >>doesn't
> > > >> > > > support
> > > >> > > > > > this now. I am just thinking if this is a good feature to
> > add
> > > >>in
> > > >> > > > > >
> > > >> > > > > > On Thu, Mar 12, 2015 at 8:24 PM, Guozhang Wang <
> > > >> wangguoz@gmail.com
> > > >> > >
> > > >> > > > > wrote:
> > > >> > > > > >
> > > >> > > > > > > Hi Tao,
> > > >> > > > > > >
> > > >> > > > > > > Sorry I was mistaken before, yes in MM you can only
> > directly
> > > >> > > specify
> > > >> > > > > > > "--whitelist", "--blacklist", and the number of streams
> > you
> > > >> want
> > > >> > > to
> > > >> > > > > > create
> > > >> > > > > > > via "--num.streams", but cannot set specific
> topic-count.
> > > >>This
> > > >> is
> > > >> > > > > because
> > > >> > > > > > > MM is mainly used for cross DC replication, and hence
> > > >>usually
> > > >> > will
> > > >> > > > pipe
> > > >> > > > > > all
> > > >> > > > > > > topics or a majority of the topics from the source
> cluster
> > > >>to
> > > >> the
> > > >> > > > > > > destination, hence usually you do not care about "some
> > > >>topics
> > > >> > > should
> > > >> > > > > get
> > > >> > > > > > X
> > > >> > > > > > > streams, while some other topics should get Y streams".
> > > >> > > > > > >
> > > >> > > > > > > Guozhang
> > > >> > > > > > >
> > > >> > > > > > > On Wed, Mar 11, 2015 at 11:59 PM, tao xiao <
> > > >> xiaotao183@gmail.com
> > > >> > >
> > > >> > > > > wrote:
> > > >> > > > > > >
> > > >> > > > > > > > The topic list is not specified in consumer.properties
> > > >>and I
> > > >> > > don't
> > > >> > > > > > think
> > > >> > > > > > > > there is any property in consumer config that allows
> us
> > to
> > > >> > > specify
> > > >> > > > > what
> > > >> > > > > > > > topics we want to consume. Can you point me to the
> > > >>property
> > > >> if
> > > >> > > > there
> > > >> > > > > is
> > > >> > > > > > > > any?
> > > >> > > > > > > >
> > > >> > > > > > > > On Thu, Mar 12, 2015 at 12:14 AM, Guozhang Wang <
> > > >> > > > wangguoz@gmail.com>
> > > >> > > > > > > > wrote:
> > > >> > > > > > > >
> > > >> > > > > > > > > Tao,
> > > >> > > > > > > > >
> > > >> > > > > > > > > In MM people can pass in consumer configs, in which
> > > >>people
> > > >> > can
> > > >> > > > > > specify
> > > >> > > > > > > > > consumption topics, either in regular topic list
> > format
> > > >>or
> > > >> > > > > whitelist
> > > >> > > > > > /
> > > >> > > > > > > > > blacklist. So I think it already does what you need?
> > > >> > > > > > > > >
> > > >> > > > > > > > > Guozhang
> > > >> > > > > > > > >
> > > >> > > > > > > > > On Tue, Mar 10, 2015 at 10:09 PM, tao xiao <
> > > >> > > xiaotao183@gmail.com
> > > >> > > > >
> > > >> > > > > > > wrote:
> > > >> > > > > > > > >
> > > >> > > > > > > > > > Thank you guys for answering. I think it will be
> > good
> > > >> that
> > > >> > we
> > > >> > > > can
> > > >> > > > > > > pass
> > > >> > > > > > > > > in a
> > > >> > > > > > > > > > customised topicCount ( I think this is the
> > interface
> > > >> > > whitelist
> > > >> > > > > and
> > > >> > > > > > > > > > backlist implement if I am not mistaken) to MM to
> > > >>achieve
> > > >> > > > similar
> > > >> > > > > > > thing
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > On Wednesday, March 11, 2015, Guozhang Wang <
> > > >> > > > wangguoz@gmail.com>
> > > >> > > > > > > > wrote:
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > > Hi Tao,
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > Unfortunately MM does not support whitelist /
> > > >>blacklist
> > > >> > at
> > > >> > > > the
> > > >> > > > > > same
> > > >> > > > > > > > > time,
> > > >> > > > > > > > > > > and you have to choose either one upon
> > > >>initialization.
> > > >> As
> > > >> > > for
> > > >> > > > > > your
> > > >> > > > > > > > > case,
> > > >> > > > > > > > > > I
> > > >> > > > > > > > > > > think it can be captured by some reg-ex to
> exclude
> > > >> > nothing
> > > >> > > > else
> > > >> > > > > > but
> > > >> > > > > > > > > "10",
> > > >> > > > > > > > > > > but I do not know the exact expression.
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > Guozhang
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > On Tue, Mar 10, 2015 at 7:58 AM, tao xiao <
> > > >> > > > > xiaotao183@gmail.com
> > > >> > > > > > > > > > > <javascript:;>> wrote:
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > > I actually mean if we can achieve this in
> mirror
> > > >> maker.
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > On Tue, Mar 10, 2015 at 10:52 PM, tao xiao <
> > > >> > > > > > xiaotao183@gmail.com
> > > >> > > > > > > > > > > <javascript:;>> wrote:
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > > Hi,
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > I have an user case where I need to consume
> a
> > > >>list
> > > >> > > topics
> > > >> > > > > > with
> > > >> > > > > > > > name
> > > >> > > > > > > > > > > that
> > > >> > > > > > > > > > > > > matches pattern topic.* except for one that
> is
> > > >> > > topic.10.
> > > >> > > > Is
> > > >> > > > > > > > there a
> > > >> > > > > > > > > > way
> > > >> > > > > > > > > > > > > that I can combine the use of whitelist and
> > > >> blacklist
> > > >> > > so
> > > >> > > > > > that I
> > > >> > > > > > > > can
> > > >> > > > > > > > > > > > achieve
> > > >> > > > > > > > > > > > > something like accept all topics with regex
> > > >>topic.*
> > > >> > but
> > > >> > > > > > exclude
> > > >> > > > > > > > > > > topic.10?
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > --
> > > >> > > > > > > > > > > > > Regards,
> > > >> > > > > > > > > > > > > Tao
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > --
> > > >> > > > > > > > > > > > Regards,
> > > >> > > > > > > > > > > > Tao
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > --
> > > >> > > > > > > > > > > -- Guozhang
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > --
> > > >> > > > > > > > > > Regards,
> > > >> > > > > > > > > > Tao
> > > >> > > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > > --
> > > >> > > > > > > > > -- Guozhang
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > > > --
> > > >> > > > > > > > Regards,
> > > >> > > > > > > > Tao
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > > >
> > > >> > > > > > >
> > > >> > > > > > > --
> > > >> > > > > > > -- Guozhang
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > > --
> > > >> > > > > > Regards,
> > > >> > > > > > Tao
> > > >> > > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > > --
> > > >> > > > > -- Guozhang
> > > >> > > > >
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > > --
> > > >> > > > Regards,
> > > >> > > > Tao
> > > >> > > >
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > --
> > > >> > > -- Guozhang
> > > >> > >
> > > >> >
> > > >> >
> > > >> >
> > > >> > --
> > > >> > Regards,
> > > >> > Tao
> > > >> >
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> -- Guozhang
> > > >>
> > > >
> > > >
> > > >
> > > >--
> > > >Regards,
> > > >Tao
> > >
> > >
> >
> >
> > --
> > Regards,
> > Tao
> >
>
>
>
> --
> -- Guozhang
>
--
Regards,
Tao
Re: Does consumer support combination of whitelist and blacklist
topic filtering
Posted by Guozhang Wang <wa...@gmail.com>.
Note that with filtering in message handler, records from the source
cluster are still considered as "consumed" since the offsets will be
committed. If you change the filtering dynamically back to whilelist these
topics, you will lose the data that gets consumed during the period of the
blacklist.
Guozhang
On Thu, Mar 12, 2015 at 10:01 PM, tao xiao <xi...@gmail.com> wrote:
> Yes, that will work. message handle can filter out message sent from
> certain topics
>
> On Fri, Mar 13, 2015 at 6:30 AM, Jiangjie Qin <jq...@linkedin.com.invalid>
> wrote:
>
> > No sure if it is an option. But does filtering out topics with message
> > handler works for you? Are you going to resume consuming from a topic
> > after you stop consuming from it?
> >
> > Jiangjie (Becket) Qin
> >
> > On 3/12/15, 8:05 AM, "tao xiao" <xi...@gmail.com> wrote:
> >
> > >Yes, you are right. a dynamic topicfilter is more appropriate where I
> can
> > >filter topics at runtime via some kind of interface e.g. JMX
> > >
> > >On Thu, Mar 12, 2015 at 11:03 PM, Guozhang Wang <wa...@gmail.com>
> > >wrote:
> > >
> > >> Tao,
> > >>
> > >> Based on your description I think the combination of whitelist /
> > >>blacklist
> > >> will not achieve your goal, since it is still static.
> > >>
> > >> Guozhang
> > >>
> > >> On Thu, Mar 12, 2015 at 6:30 AM, tao xiao <xi...@gmail.com>
> wrote:
> > >>
> > >> > Thank you Guozhang for your advice. A dynamic topic filter is what I
> > >>need
> > >> > so that I can stop a topic consumption when I need to at runtime.
> > >> >
> > >> > On Thu, Mar 12, 2015 at 9:21 PM, Guozhang Wang <wa...@gmail.com>
> > >> wrote:
> > >> >
> > >> > > 1. Dynamic: yeah that is sth. we could think of, this could be
> > >>useful
> > >> > > operationally.
> > >> > > 2. Regex: I think in terms of expressiveness it should be
> sufficient
> > >> for
> > >> > > almost all subset of topics. In practice usually the rule of thumb
> > >>is
> > >> > that
> > >> > > you will create your topics that belongs to the same "group" with
> > >>some
> > >> > > prefix / suffix so that regex expression would not be crazily
> long.
> > >> > >
> > >> > > Guozhang
> > >> > >
> > >> > > On Thu, Mar 12, 2015 at 6:10 AM, tao xiao <xi...@gmail.com>
> > >> wrote:
> > >> > >
> > >> > > > something like dynamic filtering that can be updated at runtime
> or
> > >> deny
> > >> > > all
> > >> > > > but allow a certain set of topics that cannot be specified
> easily
> > >>by
> > >> > > regex
> > >> > > >
> > >> > > > On Thu, Mar 12, 2015 at 9:06 PM, Guozhang Wang
> > >><wa...@gmail.com>
> > >> > > wrote:
> > >> > > >
> > >> > > > > Hmm, what kind of customized filtering do you have in mind? I
> > >> thought
> > >> > > > with
> > >> > > > > "--whitelist" you could already specify regex to do filtering.
> > >> > > > >
> > >> > > > > On Thu, Mar 12, 2015 at 5:56 AM, tao xiao <
> xiaotao183@gmail.com
> > >
> > >> > > wrote:
> > >> > > > >
> > >> > > > > > Hi Guozhang,
> > >> > > > > >
> > >> > > > > > I was meant to be topicfilter not topic-count. sorry for the
> > >> > > confusion.
> > >> > > > > > What I want to achieve is to pass my own customized
> > >>topicfilter
> > >> to
> > >> > MM
> > >> > > > so
> > >> > > > > > that I can filter out topics what ever I like. I know MM
> > >>doesn't
> > >> > > > support
> > >> > > > > > this now. I am just thinking if this is a good feature to
> add
> > >>in
> > >> > > > > >
> > >> > > > > > On Thu, Mar 12, 2015 at 8:24 PM, Guozhang Wang <
> > >> wangguoz@gmail.com
> > >> > >
> > >> > > > > wrote:
> > >> > > > > >
> > >> > > > > > > Hi Tao,
> > >> > > > > > >
> > >> > > > > > > Sorry I was mistaken before, yes in MM you can only
> directly
> > >> > > specify
> > >> > > > > > > "--whitelist", "--blacklist", and the number of streams
> you
> > >> want
> > >> > > to
> > >> > > > > > create
> > >> > > > > > > via "--num.streams", but cannot set specific topic-count.
> > >>This
> > >> is
> > >> > > > > because
> > >> > > > > > > MM is mainly used for cross DC replication, and hence
> > >>usually
> > >> > will
> > >> > > > pipe
> > >> > > > > > all
> > >> > > > > > > topics or a majority of the topics from the source cluster
> > >>to
> > >> the
> > >> > > > > > > destination, hence usually you do not care about "some
> > >>topics
> > >> > > should
> > >> > > > > get
> > >> > > > > > X
> > >> > > > > > > streams, while some other topics should get Y streams".
> > >> > > > > > >
> > >> > > > > > > Guozhang
> > >> > > > > > >
> > >> > > > > > > On Wed, Mar 11, 2015 at 11:59 PM, tao xiao <
> > >> xiaotao183@gmail.com
> > >> > >
> > >> > > > > wrote:
> > >> > > > > > >
> > >> > > > > > > > The topic list is not specified in consumer.properties
> > >>and I
> > >> > > don't
> > >> > > > > > think
> > >> > > > > > > > there is any property in consumer config that allows us
> to
> > >> > > specify
> > >> > > > > what
> > >> > > > > > > > topics we want to consume. Can you point me to the
> > >>property
> > >> if
> > >> > > > there
> > >> > > > > is
> > >> > > > > > > > any?
> > >> > > > > > > >
> > >> > > > > > > > On Thu, Mar 12, 2015 at 12:14 AM, Guozhang Wang <
> > >> > > > wangguoz@gmail.com>
> > >> > > > > > > > wrote:
> > >> > > > > > > >
> > >> > > > > > > > > Tao,
> > >> > > > > > > > >
> > >> > > > > > > > > In MM people can pass in consumer configs, in which
> > >>people
> > >> > can
> > >> > > > > > specify
> > >> > > > > > > > > consumption topics, either in regular topic list
> format
> > >>or
> > >> > > > > whitelist
> > >> > > > > > /
> > >> > > > > > > > > blacklist. So I think it already does what you need?
> > >> > > > > > > > >
> > >> > > > > > > > > Guozhang
> > >> > > > > > > > >
> > >> > > > > > > > > On Tue, Mar 10, 2015 at 10:09 PM, tao xiao <
> > >> > > xiaotao183@gmail.com
> > >> > > > >
> > >> > > > > > > wrote:
> > >> > > > > > > > >
> > >> > > > > > > > > > Thank you guys for answering. I think it will be
> good
> > >> that
> > >> > we
> > >> > > > can
> > >> > > > > > > pass
> > >> > > > > > > > > in a
> > >> > > > > > > > > > customised topicCount ( I think this is the
> interface
> > >> > > whitelist
> > >> > > > > and
> > >> > > > > > > > > > backlist implement if I am not mistaken) to MM to
> > >>achieve
> > >> > > > similar
> > >> > > > > > > thing
> > >> > > > > > > > > >
> > >> > > > > > > > > > On Wednesday, March 11, 2015, Guozhang Wang <
> > >> > > > wangguoz@gmail.com>
> > >> > > > > > > > wrote:
> > >> > > > > > > > > >
> > >> > > > > > > > > > > Hi Tao,
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Unfortunately MM does not support whitelist /
> > >>blacklist
> > >> > at
> > >> > > > the
> > >> > > > > > same
> > >> > > > > > > > > time,
> > >> > > > > > > > > > > and you have to choose either one upon
> > >>initialization.
> > >> As
> > >> > > for
> > >> > > > > > your
> > >> > > > > > > > > case,
> > >> > > > > > > > > > I
> > >> > > > > > > > > > > think it can be captured by some reg-ex to exclude
> > >> > nothing
> > >> > > > else
> > >> > > > > > but
> > >> > > > > > > > > "10",
> > >> > > > > > > > > > > but I do not know the exact expression.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Guozhang
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > On Tue, Mar 10, 2015 at 7:58 AM, tao xiao <
> > >> > > > > xiaotao183@gmail.com
> > >> > > > > > > > > > > <javascript:;>> wrote:
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > > I actually mean if we can achieve this in mirror
> > >> maker.
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > On Tue, Mar 10, 2015 at 10:52 PM, tao xiao <
> > >> > > > > > xiaotao183@gmail.com
> > >> > > > > > > > > > > <javascript:;>> wrote:
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > > Hi,
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > I have an user case where I need to consume a
> > >>list
> > >> > > topics
> > >> > > > > > with
> > >> > > > > > > > name
> > >> > > > > > > > > > > that
> > >> > > > > > > > > > > > > matches pattern topic.* except for one that is
> > >> > > topic.10.
> > >> > > > Is
> > >> > > > > > > > there a
> > >> > > > > > > > > > way
> > >> > > > > > > > > > > > > that I can combine the use of whitelist and
> > >> blacklist
> > >> > > so
> > >> > > > > > that I
> > >> > > > > > > > can
> > >> > > > > > > > > > > > achieve
> > >> > > > > > > > > > > > > something like accept all topics with regex
> > >>topic.*
> > >> > but
> > >> > > > > > exclude
> > >> > > > > > > > > > > topic.10?
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > --
> > >> > > > > > > > > > > > > Regards,
> > >> > > > > > > > > > > > > Tao
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > --
> > >> > > > > > > > > > > > Regards,
> > >> > > > > > > > > > > > Tao
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > --
> > >> > > > > > > > > > > -- Guozhang
> > >> > > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > > --
> > >> > > > > > > > > > Regards,
> > >> > > > > > > > > > Tao
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > > --
> > >> > > > > > > > > -- Guozhang
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > --
> > >> > > > > > > > Regards,
> > >> > > > > > > > Tao
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > > --
> > >> > > > > > > -- Guozhang
> > >> > > > > > >
> > >> > > > > >
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > --
> > >> > > > > > Regards,
> > >> > > > > > Tao
> > >> > > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > > --
> > >> > > > > -- Guozhang
> > >> > > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > > --
> > >> > > > Regards,
> > >> > > > Tao
> > >> > > >
> > >> > >
> > >> > >
> > >> > >
> > >> > > --
> > >> > > -- Guozhang
> > >> > >
> > >> >
> > >> >
> > >> >
> > >> > --
> > >> > Regards,
> > >> > Tao
> > >> >
> > >>
> > >>
> > >>
> > >> --
> > >> -- Guozhang
> > >>
> > >
> > >
> > >
> > >--
> > >Regards,
> > >Tao
> >
> >
>
>
> --
> Regards,
> Tao
>
--
-- Guozhang
Re: Does consumer support combination of whitelist and blacklist
topic filtering
Posted by tao xiao <xi...@gmail.com>.
Yes, that will work. message handle can filter out message sent from
certain topics
On Fri, Mar 13, 2015 at 6:30 AM, Jiangjie Qin <jq...@linkedin.com.invalid>
wrote:
> No sure if it is an option. But does filtering out topics with message
> handler works for you? Are you going to resume consuming from a topic
> after you stop consuming from it?
>
> Jiangjie (Becket) Qin
>
> On 3/12/15, 8:05 AM, "tao xiao" <xi...@gmail.com> wrote:
>
> >Yes, you are right. a dynamic topicfilter is more appropriate where I can
> >filter topics at runtime via some kind of interface e.g. JMX
> >
> >On Thu, Mar 12, 2015 at 11:03 PM, Guozhang Wang <wa...@gmail.com>
> >wrote:
> >
> >> Tao,
> >>
> >> Based on your description I think the combination of whitelist /
> >>blacklist
> >> will not achieve your goal, since it is still static.
> >>
> >> Guozhang
> >>
> >> On Thu, Mar 12, 2015 at 6:30 AM, tao xiao <xi...@gmail.com> wrote:
> >>
> >> > Thank you Guozhang for your advice. A dynamic topic filter is what I
> >>need
> >> > so that I can stop a topic consumption when I need to at runtime.
> >> >
> >> > On Thu, Mar 12, 2015 at 9:21 PM, Guozhang Wang <wa...@gmail.com>
> >> wrote:
> >> >
> >> > > 1. Dynamic: yeah that is sth. we could think of, this could be
> >>useful
> >> > > operationally.
> >> > > 2. Regex: I think in terms of expressiveness it should be sufficient
> >> for
> >> > > almost all subset of topics. In practice usually the rule of thumb
> >>is
> >> > that
> >> > > you will create your topics that belongs to the same "group" with
> >>some
> >> > > prefix / suffix so that regex expression would not be crazily long.
> >> > >
> >> > > Guozhang
> >> > >
> >> > > On Thu, Mar 12, 2015 at 6:10 AM, tao xiao <xi...@gmail.com>
> >> wrote:
> >> > >
> >> > > > something like dynamic filtering that can be updated at runtime or
> >> deny
> >> > > all
> >> > > > but allow a certain set of topics that cannot be specified easily
> >>by
> >> > > regex
> >> > > >
> >> > > > On Thu, Mar 12, 2015 at 9:06 PM, Guozhang Wang
> >><wa...@gmail.com>
> >> > > wrote:
> >> > > >
> >> > > > > Hmm, what kind of customized filtering do you have in mind? I
> >> thought
> >> > > > with
> >> > > > > "--whitelist" you could already specify regex to do filtering.
> >> > > > >
> >> > > > > On Thu, Mar 12, 2015 at 5:56 AM, tao xiao <xiaotao183@gmail.com
> >
> >> > > wrote:
> >> > > > >
> >> > > > > > Hi Guozhang,
> >> > > > > >
> >> > > > > > I was meant to be topicfilter not topic-count. sorry for the
> >> > > confusion.
> >> > > > > > What I want to achieve is to pass my own customized
> >>topicfilter
> >> to
> >> > MM
> >> > > > so
> >> > > > > > that I can filter out topics what ever I like. I know MM
> >>doesn't
> >> > > > support
> >> > > > > > this now. I am just thinking if this is a good feature to add
> >>in
> >> > > > > >
> >> > > > > > On Thu, Mar 12, 2015 at 8:24 PM, Guozhang Wang <
> >> wangguoz@gmail.com
> >> > >
> >> > > > > wrote:
> >> > > > > >
> >> > > > > > > Hi Tao,
> >> > > > > > >
> >> > > > > > > Sorry I was mistaken before, yes in MM you can only directly
> >> > > specify
> >> > > > > > > "--whitelist", "--blacklist", and the number of streams you
> >> want
> >> > > to
> >> > > > > > create
> >> > > > > > > via "--num.streams", but cannot set specific topic-count.
> >>This
> >> is
> >> > > > > because
> >> > > > > > > MM is mainly used for cross DC replication, and hence
> >>usually
> >> > will
> >> > > > pipe
> >> > > > > > all
> >> > > > > > > topics or a majority of the topics from the source cluster
> >>to
> >> the
> >> > > > > > > destination, hence usually you do not care about "some
> >>topics
> >> > > should
> >> > > > > get
> >> > > > > > X
> >> > > > > > > streams, while some other topics should get Y streams".
> >> > > > > > >
> >> > > > > > > Guozhang
> >> > > > > > >
> >> > > > > > > On Wed, Mar 11, 2015 at 11:59 PM, tao xiao <
> >> xiaotao183@gmail.com
> >> > >
> >> > > > > wrote:
> >> > > > > > >
> >> > > > > > > > The topic list is not specified in consumer.properties
> >>and I
> >> > > don't
> >> > > > > > think
> >> > > > > > > > there is any property in consumer config that allows us to
> >> > > specify
> >> > > > > what
> >> > > > > > > > topics we want to consume. Can you point me to the
> >>property
> >> if
> >> > > > there
> >> > > > > is
> >> > > > > > > > any?
> >> > > > > > > >
> >> > > > > > > > On Thu, Mar 12, 2015 at 12:14 AM, Guozhang Wang <
> >> > > > wangguoz@gmail.com>
> >> > > > > > > > wrote:
> >> > > > > > > >
> >> > > > > > > > > Tao,
> >> > > > > > > > >
> >> > > > > > > > > In MM people can pass in consumer configs, in which
> >>people
> >> > can
> >> > > > > > specify
> >> > > > > > > > > consumption topics, either in regular topic list format
> >>or
> >> > > > > whitelist
> >> > > > > > /
> >> > > > > > > > > blacklist. So I think it already does what you need?
> >> > > > > > > > >
> >> > > > > > > > > Guozhang
> >> > > > > > > > >
> >> > > > > > > > > On Tue, Mar 10, 2015 at 10:09 PM, tao xiao <
> >> > > xiaotao183@gmail.com
> >> > > > >
> >> > > > > > > wrote:
> >> > > > > > > > >
> >> > > > > > > > > > Thank you guys for answering. I think it will be good
> >> that
> >> > we
> >> > > > can
> >> > > > > > > pass
> >> > > > > > > > > in a
> >> > > > > > > > > > customised topicCount ( I think this is the interface
> >> > > whitelist
> >> > > > > and
> >> > > > > > > > > > backlist implement if I am not mistaken) to MM to
> >>achieve
> >> > > > similar
> >> > > > > > > thing
> >> > > > > > > > > >
> >> > > > > > > > > > On Wednesday, March 11, 2015, Guozhang Wang <
> >> > > > wangguoz@gmail.com>
> >> > > > > > > > wrote:
> >> > > > > > > > > >
> >> > > > > > > > > > > Hi Tao,
> >> > > > > > > > > > >
> >> > > > > > > > > > > Unfortunately MM does not support whitelist /
> >>blacklist
> >> > at
> >> > > > the
> >> > > > > > same
> >> > > > > > > > > time,
> >> > > > > > > > > > > and you have to choose either one upon
> >>initialization.
> >> As
> >> > > for
> >> > > > > > your
> >> > > > > > > > > case,
> >> > > > > > > > > > I
> >> > > > > > > > > > > think it can be captured by some reg-ex to exclude
> >> > nothing
> >> > > > else
> >> > > > > > but
> >> > > > > > > > > "10",
> >> > > > > > > > > > > but I do not know the exact expression.
> >> > > > > > > > > > >
> >> > > > > > > > > > > Guozhang
> >> > > > > > > > > > >
> >> > > > > > > > > > > On Tue, Mar 10, 2015 at 7:58 AM, tao xiao <
> >> > > > > xiaotao183@gmail.com
> >> > > > > > > > > > > <javascript:;>> wrote:
> >> > > > > > > > > > >
> >> > > > > > > > > > > > I actually mean if we can achieve this in mirror
> >> maker.
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > On Tue, Mar 10, 2015 at 10:52 PM, tao xiao <
> >> > > > > > xiaotao183@gmail.com
> >> > > > > > > > > > > <javascript:;>> wrote:
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > > Hi,
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > I have an user case where I need to consume a
> >>list
> >> > > topics
> >> > > > > > with
> >> > > > > > > > name
> >> > > > > > > > > > > that
> >> > > > > > > > > > > > > matches pattern topic.* except for one that is
> >> > > topic.10.
> >> > > > Is
> >> > > > > > > > there a
> >> > > > > > > > > > way
> >> > > > > > > > > > > > > that I can combine the use of whitelist and
> >> blacklist
> >> > > so
> >> > > > > > that I
> >> > > > > > > > can
> >> > > > > > > > > > > > achieve
> >> > > > > > > > > > > > > something like accept all topics with regex
> >>topic.*
> >> > but
> >> > > > > > exclude
> >> > > > > > > > > > > topic.10?
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > --
> >> > > > > > > > > > > > > Regards,
> >> > > > > > > > > > > > > Tao
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > --
> >> > > > > > > > > > > > Regards,
> >> > > > > > > > > > > > Tao
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > > --
> >> > > > > > > > > > > -- Guozhang
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > --
> >> > > > > > > > > > Regards,
> >> > > > > > > > > > Tao
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > --
> >> > > > > > > > > -- Guozhang
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > --
> >> > > > > > > > Regards,
> >> > > > > > > > Tao
> >> > > > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > --
> >> > > > > > > -- Guozhang
> >> > > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > > --
> >> > > > > > Regards,
> >> > > > > > Tao
> >> > > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > > --
> >> > > > > -- Guozhang
> >> > > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > > --
> >> > > > Regards,
> >> > > > Tao
> >> > > >
> >> > >
> >> > >
> >> > >
> >> > > --
> >> > > -- Guozhang
> >> > >
> >> >
> >> >
> >> >
> >> > --
> >> > Regards,
> >> > Tao
> >> >
> >>
> >>
> >>
> >> --
> >> -- Guozhang
> >>
> >
> >
> >
> >--
> >Regards,
> >Tao
>
>
--
Regards,
Tao
Re: Does consumer support combination of whitelist and blacklist
topic filtering
Posted by Jiangjie Qin <jq...@linkedin.com.INVALID>.
No sure if it is an option. But does filtering out topics with message
handler works for you? Are you going to resume consuming from a topic
after you stop consuming from it?
Jiangjie (Becket) Qin
On 3/12/15, 8:05 AM, "tao xiao" <xi...@gmail.com> wrote:
>Yes, you are right. a dynamic topicfilter is more appropriate where I can
>filter topics at runtime via some kind of interface e.g. JMX
>
>On Thu, Mar 12, 2015 at 11:03 PM, Guozhang Wang <wa...@gmail.com>
>wrote:
>
>> Tao,
>>
>> Based on your description I think the combination of whitelist /
>>blacklist
>> will not achieve your goal, since it is still static.
>>
>> Guozhang
>>
>> On Thu, Mar 12, 2015 at 6:30 AM, tao xiao <xi...@gmail.com> wrote:
>>
>> > Thank you Guozhang for your advice. A dynamic topic filter is what I
>>need
>> > so that I can stop a topic consumption when I need to at runtime.
>> >
>> > On Thu, Mar 12, 2015 at 9:21 PM, Guozhang Wang <wa...@gmail.com>
>> wrote:
>> >
>> > > 1. Dynamic: yeah that is sth. we could think of, this could be
>>useful
>> > > operationally.
>> > > 2. Regex: I think in terms of expressiveness it should be sufficient
>> for
>> > > almost all subset of topics. In practice usually the rule of thumb
>>is
>> > that
>> > > you will create your topics that belongs to the same "group" with
>>some
>> > > prefix / suffix so that regex expression would not be crazily long.
>> > >
>> > > Guozhang
>> > >
>> > > On Thu, Mar 12, 2015 at 6:10 AM, tao xiao <xi...@gmail.com>
>> wrote:
>> > >
>> > > > something like dynamic filtering that can be updated at runtime or
>> deny
>> > > all
>> > > > but allow a certain set of topics that cannot be specified easily
>>by
>> > > regex
>> > > >
>> > > > On Thu, Mar 12, 2015 at 9:06 PM, Guozhang Wang
>><wa...@gmail.com>
>> > > wrote:
>> > > >
>> > > > > Hmm, what kind of customized filtering do you have in mind? I
>> thought
>> > > > with
>> > > > > "--whitelist" you could already specify regex to do filtering.
>> > > > >
>> > > > > On Thu, Mar 12, 2015 at 5:56 AM, tao xiao <xi...@gmail.com>
>> > > wrote:
>> > > > >
>> > > > > > Hi Guozhang,
>> > > > > >
>> > > > > > I was meant to be topicfilter not topic-count. sorry for the
>> > > confusion.
>> > > > > > What I want to achieve is to pass my own customized
>>topicfilter
>> to
>> > MM
>> > > > so
>> > > > > > that I can filter out topics what ever I like. I know MM
>>doesn't
>> > > > support
>> > > > > > this now. I am just thinking if this is a good feature to add
>>in
>> > > > > >
>> > > > > > On Thu, Mar 12, 2015 at 8:24 PM, Guozhang Wang <
>> wangguoz@gmail.com
>> > >
>> > > > > wrote:
>> > > > > >
>> > > > > > > Hi Tao,
>> > > > > > >
>> > > > > > > Sorry I was mistaken before, yes in MM you can only directly
>> > > specify
>> > > > > > > "--whitelist", "--blacklist", and the number of streams you
>> want
>> > > to
>> > > > > > create
>> > > > > > > via "--num.streams", but cannot set specific topic-count.
>>This
>> is
>> > > > > because
>> > > > > > > MM is mainly used for cross DC replication, and hence
>>usually
>> > will
>> > > > pipe
>> > > > > > all
>> > > > > > > topics or a majority of the topics from the source cluster
>>to
>> the
>> > > > > > > destination, hence usually you do not care about "some
>>topics
>> > > should
>> > > > > get
>> > > > > > X
>> > > > > > > streams, while some other topics should get Y streams".
>> > > > > > >
>> > > > > > > Guozhang
>> > > > > > >
>> > > > > > > On Wed, Mar 11, 2015 at 11:59 PM, tao xiao <
>> xiaotao183@gmail.com
>> > >
>> > > > > wrote:
>> > > > > > >
>> > > > > > > > The topic list is not specified in consumer.properties
>>and I
>> > > don't
>> > > > > > think
>> > > > > > > > there is any property in consumer config that allows us to
>> > > specify
>> > > > > what
>> > > > > > > > topics we want to consume. Can you point me to the
>>property
>> if
>> > > > there
>> > > > > is
>> > > > > > > > any?
>> > > > > > > >
>> > > > > > > > On Thu, Mar 12, 2015 at 12:14 AM, Guozhang Wang <
>> > > > wangguoz@gmail.com>
>> > > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > Tao,
>> > > > > > > > >
>> > > > > > > > > In MM people can pass in consumer configs, in which
>>people
>> > can
>> > > > > > specify
>> > > > > > > > > consumption topics, either in regular topic list format
>>or
>> > > > > whitelist
>> > > > > > /
>> > > > > > > > > blacklist. So I think it already does what you need?
>> > > > > > > > >
>> > > > > > > > > Guozhang
>> > > > > > > > >
>> > > > > > > > > On Tue, Mar 10, 2015 at 10:09 PM, tao xiao <
>> > > xiaotao183@gmail.com
>> > > > >
>> > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > > Thank you guys for answering. I think it will be good
>> that
>> > we
>> > > > can
>> > > > > > > pass
>> > > > > > > > > in a
>> > > > > > > > > > customised topicCount ( I think this is the interface
>> > > whitelist
>> > > > > and
>> > > > > > > > > > backlist implement if I am not mistaken) to MM to
>>achieve
>> > > > similar
>> > > > > > > thing
>> > > > > > > > > >
>> > > > > > > > > > On Wednesday, March 11, 2015, Guozhang Wang <
>> > > > wangguoz@gmail.com>
>> > > > > > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > > > Hi Tao,
>> > > > > > > > > > >
>> > > > > > > > > > > Unfortunately MM does not support whitelist /
>>blacklist
>> > at
>> > > > the
>> > > > > > same
>> > > > > > > > > time,
>> > > > > > > > > > > and you have to choose either one upon
>>initialization.
>> As
>> > > for
>> > > > > > your
>> > > > > > > > > case,
>> > > > > > > > > > I
>> > > > > > > > > > > think it can be captured by some reg-ex to exclude
>> > nothing
>> > > > else
>> > > > > > but
>> > > > > > > > > "10",
>> > > > > > > > > > > but I do not know the exact expression.
>> > > > > > > > > > >
>> > > > > > > > > > > Guozhang
>> > > > > > > > > > >
>> > > > > > > > > > > On Tue, Mar 10, 2015 at 7:58 AM, tao xiao <
>> > > > > xiaotao183@gmail.com
>> > > > > > > > > > > <javascript:;>> wrote:
>> > > > > > > > > > >
>> > > > > > > > > > > > I actually mean if we can achieve this in mirror
>> maker.
>> > > > > > > > > > > >
>> > > > > > > > > > > > On Tue, Mar 10, 2015 at 10:52 PM, tao xiao <
>> > > > > > xiaotao183@gmail.com
>> > > > > > > > > > > <javascript:;>> wrote:
>> > > > > > > > > > > >
>> > > > > > > > > > > > > Hi,
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > I have an user case where I need to consume a
>>list
>> > > topics
>> > > > > > with
>> > > > > > > > name
>> > > > > > > > > > > that
>> > > > > > > > > > > > > matches pattern topic.* except for one that is
>> > > topic.10.
>> > > > Is
>> > > > > > > > there a
>> > > > > > > > > > way
>> > > > > > > > > > > > > that I can combine the use of whitelist and
>> blacklist
>> > > so
>> > > > > > that I
>> > > > > > > > can
>> > > > > > > > > > > > achieve
>> > > > > > > > > > > > > something like accept all topics with regex
>>topic.*
>> > but
>> > > > > > exclude
>> > > > > > > > > > > topic.10?
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > --
>> > > > > > > > > > > > > Regards,
>> > > > > > > > > > > > > Tao
>> > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > > --
>> > > > > > > > > > > > Regards,
>> > > > > > > > > > > > Tao
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > > --
>> > > > > > > > > > > -- Guozhang
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > --
>> > > > > > > > > > Regards,
>> > > > > > > > > > Tao
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > --
>> > > > > > > > > -- Guozhang
>> > > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > --
>> > > > > > > > Regards,
>> > > > > > > > Tao
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > --
>> > > > > > > -- Guozhang
>> > > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > --
>> > > > > > Regards,
>> > > > > > Tao
>> > > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > --
>> > > > > -- Guozhang
>> > > > >
>> > > >
>> > > >
>> > > >
>> > > > --
>> > > > Regards,
>> > > > Tao
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > > -- Guozhang
>> > >
>> >
>> >
>> >
>> > --
>> > Regards,
>> > Tao
>> >
>>
>>
>>
>> --
>> -- Guozhang
>>
>
>
>
>--
>Regards,
>Tao
Re: Does consumer support combination of whitelist and blacklist
topic filtering
Posted by tao xiao <xi...@gmail.com>.
Yes, you are right. a dynamic topicfilter is more appropriate where I can
filter topics at runtime via some kind of interface e.g. JMX
On Thu, Mar 12, 2015 at 11:03 PM, Guozhang Wang <wa...@gmail.com> wrote:
> Tao,
>
> Based on your description I think the combination of whitelist / blacklist
> will not achieve your goal, since it is still static.
>
> Guozhang
>
> On Thu, Mar 12, 2015 at 6:30 AM, tao xiao <xi...@gmail.com> wrote:
>
> > Thank you Guozhang for your advice. A dynamic topic filter is what I need
> > so that I can stop a topic consumption when I need to at runtime.
> >
> > On Thu, Mar 12, 2015 at 9:21 PM, Guozhang Wang <wa...@gmail.com>
> wrote:
> >
> > > 1. Dynamic: yeah that is sth. we could think of, this could be useful
> > > operationally.
> > > 2. Regex: I think in terms of expressiveness it should be sufficient
> for
> > > almost all subset of topics. In practice usually the rule of thumb is
> > that
> > > you will create your topics that belongs to the same "group" with some
> > > prefix / suffix so that regex expression would not be crazily long.
> > >
> > > Guozhang
> > >
> > > On Thu, Mar 12, 2015 at 6:10 AM, tao xiao <xi...@gmail.com>
> wrote:
> > >
> > > > something like dynamic filtering that can be updated at runtime or
> deny
> > > all
> > > > but allow a certain set of topics that cannot be specified easily by
> > > regex
> > > >
> > > > On Thu, Mar 12, 2015 at 9:06 PM, Guozhang Wang <wa...@gmail.com>
> > > wrote:
> > > >
> > > > > Hmm, what kind of customized filtering do you have in mind? I
> thought
> > > > with
> > > > > "--whitelist" you could already specify regex to do filtering.
> > > > >
> > > > > On Thu, Mar 12, 2015 at 5:56 AM, tao xiao <xi...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Hi Guozhang,
> > > > > >
> > > > > > I was meant to be topicfilter not topic-count. sorry for the
> > > confusion.
> > > > > > What I want to achieve is to pass my own customized topicfilter
> to
> > MM
> > > > so
> > > > > > that I can filter out topics what ever I like. I know MM doesn't
> > > > support
> > > > > > this now. I am just thinking if this is a good feature to add in
> > > > > >
> > > > > > On Thu, Mar 12, 2015 at 8:24 PM, Guozhang Wang <
> wangguoz@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Hi Tao,
> > > > > > >
> > > > > > > Sorry I was mistaken before, yes in MM you can only directly
> > > specify
> > > > > > > "--whitelist", "--blacklist", and the number of streams you
> want
> > > to
> > > > > > create
> > > > > > > via "--num.streams", but cannot set specific topic-count. This
> is
> > > > > because
> > > > > > > MM is mainly used for cross DC replication, and hence usually
> > will
> > > > pipe
> > > > > > all
> > > > > > > topics or a majority of the topics from the source cluster to
> the
> > > > > > > destination, hence usually you do not care about "some topics
> > > should
> > > > > get
> > > > > > X
> > > > > > > streams, while some other topics should get Y streams".
> > > > > > >
> > > > > > > Guozhang
> > > > > > >
> > > > > > > On Wed, Mar 11, 2015 at 11:59 PM, tao xiao <
> xiaotao183@gmail.com
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > > The topic list is not specified in consumer.properties and I
> > > don't
> > > > > > think
> > > > > > > > there is any property in consumer config that allows us to
> > > specify
> > > > > what
> > > > > > > > topics we want to consume. Can you point me to the property
> if
> > > > there
> > > > > is
> > > > > > > > any?
> > > > > > > >
> > > > > > > > On Thu, Mar 12, 2015 at 12:14 AM, Guozhang Wang <
> > > > wangguoz@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Tao,
> > > > > > > > >
> > > > > > > > > In MM people can pass in consumer configs, in which people
> > can
> > > > > > specify
> > > > > > > > > consumption topics, either in regular topic list format or
> > > > > whitelist
> > > > > > /
> > > > > > > > > blacklist. So I think it already does what you need?
> > > > > > > > >
> > > > > > > > > Guozhang
> > > > > > > > >
> > > > > > > > > On Tue, Mar 10, 2015 at 10:09 PM, tao xiao <
> > > xiaotao183@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Thank you guys for answering. I think it will be good
> that
> > we
> > > > can
> > > > > > > pass
> > > > > > > > > in a
> > > > > > > > > > customised topicCount ( I think this is the interface
> > > whitelist
> > > > > and
> > > > > > > > > > backlist implement if I am not mistaken) to MM to achieve
> > > > similar
> > > > > > > thing
> > > > > > > > > >
> > > > > > > > > > On Wednesday, March 11, 2015, Guozhang Wang <
> > > > wangguoz@gmail.com>
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Tao,
> > > > > > > > > > >
> > > > > > > > > > > Unfortunately MM does not support whitelist / blacklist
> > at
> > > > the
> > > > > > same
> > > > > > > > > time,
> > > > > > > > > > > and you have to choose either one upon initialization.
> As
> > > for
> > > > > > your
> > > > > > > > > case,
> > > > > > > > > > I
> > > > > > > > > > > think it can be captured by some reg-ex to exclude
> > nothing
> > > > else
> > > > > > but
> > > > > > > > > "10",
> > > > > > > > > > > but I do not know the exact expression.
> > > > > > > > > > >
> > > > > > > > > > > Guozhang
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Mar 10, 2015 at 7:58 AM, tao xiao <
> > > > > xiaotao183@gmail.com
> > > > > > > > > > > <javascript:;>> wrote:
> > > > > > > > > > >
> > > > > > > > > > > > I actually mean if we can achieve this in mirror
> maker.
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Mar 10, 2015 at 10:52 PM, tao xiao <
> > > > > > xiaotao183@gmail.com
> > > > > > > > > > > <javascript:;>> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi,
> > > > > > > > > > > > >
> > > > > > > > > > > > > I have an user case where I need to consume a list
> > > topics
> > > > > > with
> > > > > > > > name
> > > > > > > > > > > that
> > > > > > > > > > > > > matches pattern topic.* except for one that is
> > > topic.10.
> > > > Is
> > > > > > > > there a
> > > > > > > > > > way
> > > > > > > > > > > > > that I can combine the use of whitelist and
> blacklist
> > > so
> > > > > > that I
> > > > > > > > can
> > > > > > > > > > > > achieve
> > > > > > > > > > > > > something like accept all topics with regex topic.*
> > but
> > > > > > exclude
> > > > > > > > > > > topic.10?
> > > > > > > > > > > > >
> > > > > > > > > > > > > --
> > > > > > > > > > > > > Regards,
> > > > > > > > > > > > > Tao
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > --
> > > > > > > > > > > > Regards,
> > > > > > > > > > > > Tao
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > -- Guozhang
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > Regards,
> > > > > > > > > > Tao
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > -- Guozhang
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Regards,
> > > > > > > > Tao
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > -- Guozhang
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Regards,
> > > > > > Tao
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > -- Guozhang
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Regards,
> > > > Tao
> > > >
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
> >
> >
> > --
> > Regards,
> > Tao
> >
>
>
>
> --
> -- Guozhang
>
--
Regards,
Tao
Re: Does consumer support combination of whitelist and blacklist
topic filtering
Posted by Guozhang Wang <wa...@gmail.com>.
Tao,
Based on your description I think the combination of whitelist / blacklist
will not achieve your goal, since it is still static.
Guozhang
On Thu, Mar 12, 2015 at 6:30 AM, tao xiao <xi...@gmail.com> wrote:
> Thank you Guozhang for your advice. A dynamic topic filter is what I need
> so that I can stop a topic consumption when I need to at runtime.
>
> On Thu, Mar 12, 2015 at 9:21 PM, Guozhang Wang <wa...@gmail.com> wrote:
>
> > 1. Dynamic: yeah that is sth. we could think of, this could be useful
> > operationally.
> > 2. Regex: I think in terms of expressiveness it should be sufficient for
> > almost all subset of topics. In practice usually the rule of thumb is
> that
> > you will create your topics that belongs to the same "group" with some
> > prefix / suffix so that regex expression would not be crazily long.
> >
> > Guozhang
> >
> > On Thu, Mar 12, 2015 at 6:10 AM, tao xiao <xi...@gmail.com> wrote:
> >
> > > something like dynamic filtering that can be updated at runtime or deny
> > all
> > > but allow a certain set of topics that cannot be specified easily by
> > regex
> > >
> > > On Thu, Mar 12, 2015 at 9:06 PM, Guozhang Wang <wa...@gmail.com>
> > wrote:
> > >
> > > > Hmm, what kind of customized filtering do you have in mind? I thought
> > > with
> > > > "--whitelist" you could already specify regex to do filtering.
> > > >
> > > > On Thu, Mar 12, 2015 at 5:56 AM, tao xiao <xi...@gmail.com>
> > wrote:
> > > >
> > > > > Hi Guozhang,
> > > > >
> > > > > I was meant to be topicfilter not topic-count. sorry for the
> > confusion.
> > > > > What I want to achieve is to pass my own customized topicfilter to
> MM
> > > so
> > > > > that I can filter out topics what ever I like. I know MM doesn't
> > > support
> > > > > this now. I am just thinking if this is a good feature to add in
> > > > >
> > > > > On Thu, Mar 12, 2015 at 8:24 PM, Guozhang Wang <wangguoz@gmail.com
> >
> > > > wrote:
> > > > >
> > > > > > Hi Tao,
> > > > > >
> > > > > > Sorry I was mistaken before, yes in MM you can only directly
> > specify
> > > > > > "--whitelist", "--blacklist", and the number of streams you want
> > to
> > > > > create
> > > > > > via "--num.streams", but cannot set specific topic-count. This is
> > > > because
> > > > > > MM is mainly used for cross DC replication, and hence usually
> will
> > > pipe
> > > > > all
> > > > > > topics or a majority of the topics from the source cluster to the
> > > > > > destination, hence usually you do not care about "some topics
> > should
> > > > get
> > > > > X
> > > > > > streams, while some other topics should get Y streams".
> > > > > >
> > > > > > Guozhang
> > > > > >
> > > > > > On Wed, Mar 11, 2015 at 11:59 PM, tao xiao <xiaotao183@gmail.com
> >
> > > > wrote:
> > > > > >
> > > > > > > The topic list is not specified in consumer.properties and I
> > don't
> > > > > think
> > > > > > > there is any property in consumer config that allows us to
> > specify
> > > > what
> > > > > > > topics we want to consume. Can you point me to the property if
> > > there
> > > > is
> > > > > > > any?
> > > > > > >
> > > > > > > On Thu, Mar 12, 2015 at 12:14 AM, Guozhang Wang <
> > > wangguoz@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Tao,
> > > > > > > >
> > > > > > > > In MM people can pass in consumer configs, in which people
> can
> > > > > specify
> > > > > > > > consumption topics, either in regular topic list format or
> > > > whitelist
> > > > > /
> > > > > > > > blacklist. So I think it already does what you need?
> > > > > > > >
> > > > > > > > Guozhang
> > > > > > > >
> > > > > > > > On Tue, Mar 10, 2015 at 10:09 PM, tao xiao <
> > xiaotao183@gmail.com
> > > >
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Thank you guys for answering. I think it will be good that
> we
> > > can
> > > > > > pass
> > > > > > > > in a
> > > > > > > > > customised topicCount ( I think this is the interface
> > whitelist
> > > > and
> > > > > > > > > backlist implement if I am not mistaken) to MM to achieve
> > > similar
> > > > > > thing
> > > > > > > > >
> > > > > > > > > On Wednesday, March 11, 2015, Guozhang Wang <
> > > wangguoz@gmail.com>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Tao,
> > > > > > > > > >
> > > > > > > > > > Unfortunately MM does not support whitelist / blacklist
> at
> > > the
> > > > > same
> > > > > > > > time,
> > > > > > > > > > and you have to choose either one upon initialization. As
> > for
> > > > > your
> > > > > > > > case,
> > > > > > > > > I
> > > > > > > > > > think it can be captured by some reg-ex to exclude
> nothing
> > > else
> > > > > but
> > > > > > > > "10",
> > > > > > > > > > but I do not know the exact expression.
> > > > > > > > > >
> > > > > > > > > > Guozhang
> > > > > > > > > >
> > > > > > > > > > On Tue, Mar 10, 2015 at 7:58 AM, tao xiao <
> > > > xiaotao183@gmail.com
> > > > > > > > > > <javascript:;>> wrote:
> > > > > > > > > >
> > > > > > > > > > > I actually mean if we can achieve this in mirror maker.
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Mar 10, 2015 at 10:52 PM, tao xiao <
> > > > > xiaotao183@gmail.com
> > > > > > > > > > <javascript:;>> wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi,
> > > > > > > > > > > >
> > > > > > > > > > > > I have an user case where I need to consume a list
> > topics
> > > > > with
> > > > > > > name
> > > > > > > > > > that
> > > > > > > > > > > > matches pattern topic.* except for one that is
> > topic.10.
> > > Is
> > > > > > > there a
> > > > > > > > > way
> > > > > > > > > > > > that I can combine the use of whitelist and blacklist
> > so
> > > > > that I
> > > > > > > can
> > > > > > > > > > > achieve
> > > > > > > > > > > > something like accept all topics with regex topic.*
> but
> > > > > exclude
> > > > > > > > > > topic.10?
> > > > > > > > > > > >
> > > > > > > > > > > > --
> > > > > > > > > > > > Regards,
> > > > > > > > > > > > Tao
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > Regards,
> > > > > > > > > > > Tao
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > -- Guozhang
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Regards,
> > > > > > > > > Tao
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > -- Guozhang
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Regards,
> > > > > > > Tao
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > -- Guozhang
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Regards,
> > > > > Tao
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > -- Guozhang
> > > >
> > >
> > >
> > >
> > > --
> > > Regards,
> > > Tao
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
>
>
> --
> Regards,
> Tao
>
--
-- Guozhang
Re: Does consumer support combination of whitelist and blacklist
topic filtering
Posted by tao xiao <xi...@gmail.com>.
Thank you Guozhang for your advice. A dynamic topic filter is what I need
so that I can stop a topic consumption when I need to at runtime.
On Thu, Mar 12, 2015 at 9:21 PM, Guozhang Wang <wa...@gmail.com> wrote:
> 1. Dynamic: yeah that is sth. we could think of, this could be useful
> operationally.
> 2. Regex: I think in terms of expressiveness it should be sufficient for
> almost all subset of topics. In practice usually the rule of thumb is that
> you will create your topics that belongs to the same "group" with some
> prefix / suffix so that regex expression would not be crazily long.
>
> Guozhang
>
> On Thu, Mar 12, 2015 at 6:10 AM, tao xiao <xi...@gmail.com> wrote:
>
> > something like dynamic filtering that can be updated at runtime or deny
> all
> > but allow a certain set of topics that cannot be specified easily by
> regex
> >
> > On Thu, Mar 12, 2015 at 9:06 PM, Guozhang Wang <wa...@gmail.com>
> wrote:
> >
> > > Hmm, what kind of customized filtering do you have in mind? I thought
> > with
> > > "--whitelist" you could already specify regex to do filtering.
> > >
> > > On Thu, Mar 12, 2015 at 5:56 AM, tao xiao <xi...@gmail.com>
> wrote:
> > >
> > > > Hi Guozhang,
> > > >
> > > > I was meant to be topicfilter not topic-count. sorry for the
> confusion.
> > > > What I want to achieve is to pass my own customized topicfilter to MM
> > so
> > > > that I can filter out topics what ever I like. I know MM doesn't
> > support
> > > > this now. I am just thinking if this is a good feature to add in
> > > >
> > > > On Thu, Mar 12, 2015 at 8:24 PM, Guozhang Wang <wa...@gmail.com>
> > > wrote:
> > > >
> > > > > Hi Tao,
> > > > >
> > > > > Sorry I was mistaken before, yes in MM you can only directly
> specify
> > > > > "--whitelist", "--blacklist", and the number of streams you want
> to
> > > > create
> > > > > via "--num.streams", but cannot set specific topic-count. This is
> > > because
> > > > > MM is mainly used for cross DC replication, and hence usually will
> > pipe
> > > > all
> > > > > topics or a majority of the topics from the source cluster to the
> > > > > destination, hence usually you do not care about "some topics
> should
> > > get
> > > > X
> > > > > streams, while some other topics should get Y streams".
> > > > >
> > > > > Guozhang
> > > > >
> > > > > On Wed, Mar 11, 2015 at 11:59 PM, tao xiao <xi...@gmail.com>
> > > wrote:
> > > > >
> > > > > > The topic list is not specified in consumer.properties and I
> don't
> > > > think
> > > > > > there is any property in consumer config that allows us to
> specify
> > > what
> > > > > > topics we want to consume. Can you point me to the property if
> > there
> > > is
> > > > > > any?
> > > > > >
> > > > > > On Thu, Mar 12, 2015 at 12:14 AM, Guozhang Wang <
> > wangguoz@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Tao,
> > > > > > >
> > > > > > > In MM people can pass in consumer configs, in which people can
> > > > specify
> > > > > > > consumption topics, either in regular topic list format or
> > > whitelist
> > > > /
> > > > > > > blacklist. So I think it already does what you need?
> > > > > > >
> > > > > > > Guozhang
> > > > > > >
> > > > > > > On Tue, Mar 10, 2015 at 10:09 PM, tao xiao <
> xiaotao183@gmail.com
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > > Thank you guys for answering. I think it will be good that we
> > can
> > > > > pass
> > > > > > > in a
> > > > > > > > customised topicCount ( I think this is the interface
> whitelist
> > > and
> > > > > > > > backlist implement if I am not mistaken) to MM to achieve
> > similar
> > > > > thing
> > > > > > > >
> > > > > > > > On Wednesday, March 11, 2015, Guozhang Wang <
> > wangguoz@gmail.com>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Tao,
> > > > > > > > >
> > > > > > > > > Unfortunately MM does not support whitelist / blacklist at
> > the
> > > > same
> > > > > > > time,
> > > > > > > > > and you have to choose either one upon initialization. As
> for
> > > > your
> > > > > > > case,
> > > > > > > > I
> > > > > > > > > think it can be captured by some reg-ex to exclude nothing
> > else
> > > > but
> > > > > > > "10",
> > > > > > > > > but I do not know the exact expression.
> > > > > > > > >
> > > > > > > > > Guozhang
> > > > > > > > >
> > > > > > > > > On Tue, Mar 10, 2015 at 7:58 AM, tao xiao <
> > > xiaotao183@gmail.com
> > > > > > > > > <javascript:;>> wrote:
> > > > > > > > >
> > > > > > > > > > I actually mean if we can achieve this in mirror maker.
> > > > > > > > > >
> > > > > > > > > > On Tue, Mar 10, 2015 at 10:52 PM, tao xiao <
> > > > xiaotao183@gmail.com
> > > > > > > > > <javascript:;>> wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi,
> > > > > > > > > > >
> > > > > > > > > > > I have an user case where I need to consume a list
> topics
> > > > with
> > > > > > name
> > > > > > > > > that
> > > > > > > > > > > matches pattern topic.* except for one that is
> topic.10.
> > Is
> > > > > > there a
> > > > > > > > way
> > > > > > > > > > > that I can combine the use of whitelist and blacklist
> so
> > > > that I
> > > > > > can
> > > > > > > > > > achieve
> > > > > > > > > > > something like accept all topics with regex topic.* but
> > > > exclude
> > > > > > > > > topic.10?
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > Regards,
> > > > > > > > > > > Tao
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > Regards,
> > > > > > > > > > Tao
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > -- Guozhang
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Regards,
> > > > > > > > Tao
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > -- Guozhang
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Regards,
> > > > > > Tao
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > -- Guozhang
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Regards,
> > > > Tao
> > > >
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
> >
> >
> > --
> > Regards,
> > Tao
> >
>
>
>
> --
> -- Guozhang
>
--
Regards,
Tao
Re: Does consumer support combination of whitelist and blacklist
topic filtering
Posted by Guozhang Wang <wa...@gmail.com>.
1. Dynamic: yeah that is sth. we could think of, this could be useful
operationally.
2. Regex: I think in terms of expressiveness it should be sufficient for
almost all subset of topics. In practice usually the rule of thumb is that
you will create your topics that belongs to the same "group" with some
prefix / suffix so that regex expression would not be crazily long.
Guozhang
On Thu, Mar 12, 2015 at 6:10 AM, tao xiao <xi...@gmail.com> wrote:
> something like dynamic filtering that can be updated at runtime or deny all
> but allow a certain set of topics that cannot be specified easily by regex
>
> On Thu, Mar 12, 2015 at 9:06 PM, Guozhang Wang <wa...@gmail.com> wrote:
>
> > Hmm, what kind of customized filtering do you have in mind? I thought
> with
> > "--whitelist" you could already specify regex to do filtering.
> >
> > On Thu, Mar 12, 2015 at 5:56 AM, tao xiao <xi...@gmail.com> wrote:
> >
> > > Hi Guozhang,
> > >
> > > I was meant to be topicfilter not topic-count. sorry for the confusion.
> > > What I want to achieve is to pass my own customized topicfilter to MM
> so
> > > that I can filter out topics what ever I like. I know MM doesn't
> support
> > > this now. I am just thinking if this is a good feature to add in
> > >
> > > On Thu, Mar 12, 2015 at 8:24 PM, Guozhang Wang <wa...@gmail.com>
> > wrote:
> > >
> > > > Hi Tao,
> > > >
> > > > Sorry I was mistaken before, yes in MM you can only directly specify
> > > > "--whitelist", "--blacklist", and the number of streams you want to
> > > create
> > > > via "--num.streams", but cannot set specific topic-count. This is
> > because
> > > > MM is mainly used for cross DC replication, and hence usually will
> pipe
> > > all
> > > > topics or a majority of the topics from the source cluster to the
> > > > destination, hence usually you do not care about "some topics should
> > get
> > > X
> > > > streams, while some other topics should get Y streams".
> > > >
> > > > Guozhang
> > > >
> > > > On Wed, Mar 11, 2015 at 11:59 PM, tao xiao <xi...@gmail.com>
> > wrote:
> > > >
> > > > > The topic list is not specified in consumer.properties and I don't
> > > think
> > > > > there is any property in consumer config that allows us to specify
> > what
> > > > > topics we want to consume. Can you point me to the property if
> there
> > is
> > > > > any?
> > > > >
> > > > > On Thu, Mar 12, 2015 at 12:14 AM, Guozhang Wang <
> wangguoz@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Tao,
> > > > > >
> > > > > > In MM people can pass in consumer configs, in which people can
> > > specify
> > > > > > consumption topics, either in regular topic list format or
> > whitelist
> > > /
> > > > > > blacklist. So I think it already does what you need?
> > > > > >
> > > > > > Guozhang
> > > > > >
> > > > > > On Tue, Mar 10, 2015 at 10:09 PM, tao xiao <xiaotao183@gmail.com
> >
> > > > wrote:
> > > > > >
> > > > > > > Thank you guys for answering. I think it will be good that we
> can
> > > > pass
> > > > > > in a
> > > > > > > customised topicCount ( I think this is the interface whitelist
> > and
> > > > > > > backlist implement if I am not mistaken) to MM to achieve
> similar
> > > > thing
> > > > > > >
> > > > > > > On Wednesday, March 11, 2015, Guozhang Wang <
> wangguoz@gmail.com>
> > > > > wrote:
> > > > > > >
> > > > > > > > Hi Tao,
> > > > > > > >
> > > > > > > > Unfortunately MM does not support whitelist / blacklist at
> the
> > > same
> > > > > > time,
> > > > > > > > and you have to choose either one upon initialization. As for
> > > your
> > > > > > case,
> > > > > > > I
> > > > > > > > think it can be captured by some reg-ex to exclude nothing
> else
> > > but
> > > > > > "10",
> > > > > > > > but I do not know the exact expression.
> > > > > > > >
> > > > > > > > Guozhang
> > > > > > > >
> > > > > > > > On Tue, Mar 10, 2015 at 7:58 AM, tao xiao <
> > xiaotao183@gmail.com
> > > > > > > > <javascript:;>> wrote:
> > > > > > > >
> > > > > > > > > I actually mean if we can achieve this in mirror maker.
> > > > > > > > >
> > > > > > > > > On Tue, Mar 10, 2015 at 10:52 PM, tao xiao <
> > > xiaotao183@gmail.com
> > > > > > > > <javascript:;>> wrote:
> > > > > > > > >
> > > > > > > > > > Hi,
> > > > > > > > > >
> > > > > > > > > > I have an user case where I need to consume a list topics
> > > with
> > > > > name
> > > > > > > > that
> > > > > > > > > > matches pattern topic.* except for one that is topic.10.
> Is
> > > > > there a
> > > > > > > way
> > > > > > > > > > that I can combine the use of whitelist and blacklist so
> > > that I
> > > > > can
> > > > > > > > > achieve
> > > > > > > > > > something like accept all topics with regex topic.* but
> > > exclude
> > > > > > > > topic.10?
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > Regards,
> > > > > > > > > > Tao
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Regards,
> > > > > > > > > Tao
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > -- Guozhang
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Regards,
> > > > > > > Tao
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > -- Guozhang
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Regards,
> > > > > Tao
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > -- Guozhang
> > > >
> > >
> > >
> > >
> > > --
> > > Regards,
> > > Tao
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
>
>
> --
> Regards,
> Tao
>
--
-- Guozhang
Re: Does consumer support combination of whitelist and blacklist
topic filtering
Posted by tao xiao <xi...@gmail.com>.
something like dynamic filtering that can be updated at runtime or deny all
but allow a certain set of topics that cannot be specified easily by regex
On Thu, Mar 12, 2015 at 9:06 PM, Guozhang Wang <wa...@gmail.com> wrote:
> Hmm, what kind of customized filtering do you have in mind? I thought with
> "--whitelist" you could already specify regex to do filtering.
>
> On Thu, Mar 12, 2015 at 5:56 AM, tao xiao <xi...@gmail.com> wrote:
>
> > Hi Guozhang,
> >
> > I was meant to be topicfilter not topic-count. sorry for the confusion.
> > What I want to achieve is to pass my own customized topicfilter to MM so
> > that I can filter out topics what ever I like. I know MM doesn't support
> > this now. I am just thinking if this is a good feature to add in
> >
> > On Thu, Mar 12, 2015 at 8:24 PM, Guozhang Wang <wa...@gmail.com>
> wrote:
> >
> > > Hi Tao,
> > >
> > > Sorry I was mistaken before, yes in MM you can only directly specify
> > > "--whitelist", "--blacklist", and the number of streams you want to
> > create
> > > via "--num.streams", but cannot set specific topic-count. This is
> because
> > > MM is mainly used for cross DC replication, and hence usually will pipe
> > all
> > > topics or a majority of the topics from the source cluster to the
> > > destination, hence usually you do not care about "some topics should
> get
> > X
> > > streams, while some other topics should get Y streams".
> > >
> > > Guozhang
> > >
> > > On Wed, Mar 11, 2015 at 11:59 PM, tao xiao <xi...@gmail.com>
> wrote:
> > >
> > > > The topic list is not specified in consumer.properties and I don't
> > think
> > > > there is any property in consumer config that allows us to specify
> what
> > > > topics we want to consume. Can you point me to the property if there
> is
> > > > any?
> > > >
> > > > On Thu, Mar 12, 2015 at 12:14 AM, Guozhang Wang <wa...@gmail.com>
> > > > wrote:
> > > >
> > > > > Tao,
> > > > >
> > > > > In MM people can pass in consumer configs, in which people can
> > specify
> > > > > consumption topics, either in regular topic list format or
> whitelist
> > /
> > > > > blacklist. So I think it already does what you need?
> > > > >
> > > > > Guozhang
> > > > >
> > > > > On Tue, Mar 10, 2015 at 10:09 PM, tao xiao <xi...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Thank you guys for answering. I think it will be good that we can
> > > pass
> > > > > in a
> > > > > > customised topicCount ( I think this is the interface whitelist
> and
> > > > > > backlist implement if I am not mistaken) to MM to achieve similar
> > > thing
> > > > > >
> > > > > > On Wednesday, March 11, 2015, Guozhang Wang <wa...@gmail.com>
> > > > wrote:
> > > > > >
> > > > > > > Hi Tao,
> > > > > > >
> > > > > > > Unfortunately MM does not support whitelist / blacklist at the
> > same
> > > > > time,
> > > > > > > and you have to choose either one upon initialization. As for
> > your
> > > > > case,
> > > > > > I
> > > > > > > think it can be captured by some reg-ex to exclude nothing else
> > but
> > > > > "10",
> > > > > > > but I do not know the exact expression.
> > > > > > >
> > > > > > > Guozhang
> > > > > > >
> > > > > > > On Tue, Mar 10, 2015 at 7:58 AM, tao xiao <
> xiaotao183@gmail.com
> > > > > > > <javascript:;>> wrote:
> > > > > > >
> > > > > > > > I actually mean if we can achieve this in mirror maker.
> > > > > > > >
> > > > > > > > On Tue, Mar 10, 2015 at 10:52 PM, tao xiao <
> > xiaotao183@gmail.com
> > > > > > > <javascript:;>> wrote:
> > > > > > > >
> > > > > > > > > Hi,
> > > > > > > > >
> > > > > > > > > I have an user case where I need to consume a list topics
> > with
> > > > name
> > > > > > > that
> > > > > > > > > matches pattern topic.* except for one that is topic.10. Is
> > > > there a
> > > > > > way
> > > > > > > > > that I can combine the use of whitelist and blacklist so
> > that I
> > > > can
> > > > > > > > achieve
> > > > > > > > > something like accept all topics with regex topic.* but
> > exclude
> > > > > > > topic.10?
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Regards,
> > > > > > > > > Tao
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Regards,
> > > > > > > > Tao
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > -- Guozhang
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Regards,
> > > > > > Tao
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > -- Guozhang
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Regards,
> > > > Tao
> > > >
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
> >
> >
> > --
> > Regards,
> > Tao
> >
>
>
>
> --
> -- Guozhang
>
--
Regards,
Tao
Re: Does consumer support combination of whitelist and blacklist
topic filtering
Posted by Guozhang Wang <wa...@gmail.com>.
Hmm, what kind of customized filtering do you have in mind? I thought with
"--whitelist" you could already specify regex to do filtering.
On Thu, Mar 12, 2015 at 5:56 AM, tao xiao <xi...@gmail.com> wrote:
> Hi Guozhang,
>
> I was meant to be topicfilter not topic-count. sorry for the confusion.
> What I want to achieve is to pass my own customized topicfilter to MM so
> that I can filter out topics what ever I like. I know MM doesn't support
> this now. I am just thinking if this is a good feature to add in
>
> On Thu, Mar 12, 2015 at 8:24 PM, Guozhang Wang <wa...@gmail.com> wrote:
>
> > Hi Tao,
> >
> > Sorry I was mistaken before, yes in MM you can only directly specify
> > "--whitelist", "--blacklist", and the number of streams you want to
> create
> > via "--num.streams", but cannot set specific topic-count. This is because
> > MM is mainly used for cross DC replication, and hence usually will pipe
> all
> > topics or a majority of the topics from the source cluster to the
> > destination, hence usually you do not care about "some topics should get
> X
> > streams, while some other topics should get Y streams".
> >
> > Guozhang
> >
> > On Wed, Mar 11, 2015 at 11:59 PM, tao xiao <xi...@gmail.com> wrote:
> >
> > > The topic list is not specified in consumer.properties and I don't
> think
> > > there is any property in consumer config that allows us to specify what
> > > topics we want to consume. Can you point me to the property if there is
> > > any?
> > >
> > > On Thu, Mar 12, 2015 at 12:14 AM, Guozhang Wang <wa...@gmail.com>
> > > wrote:
> > >
> > > > Tao,
> > > >
> > > > In MM people can pass in consumer configs, in which people can
> specify
> > > > consumption topics, either in regular topic list format or whitelist
> /
> > > > blacklist. So I think it already does what you need?
> > > >
> > > > Guozhang
> > > >
> > > > On Tue, Mar 10, 2015 at 10:09 PM, tao xiao <xi...@gmail.com>
> > wrote:
> > > >
> > > > > Thank you guys for answering. I think it will be good that we can
> > pass
> > > > in a
> > > > > customised topicCount ( I think this is the interface whitelist and
> > > > > backlist implement if I am not mistaken) to MM to achieve similar
> > thing
> > > > >
> > > > > On Wednesday, March 11, 2015, Guozhang Wang <wa...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Hi Tao,
> > > > > >
> > > > > > Unfortunately MM does not support whitelist / blacklist at the
> same
> > > > time,
> > > > > > and you have to choose either one upon initialization. As for
> your
> > > > case,
> > > > > I
> > > > > > think it can be captured by some reg-ex to exclude nothing else
> but
> > > > "10",
> > > > > > but I do not know the exact expression.
> > > > > >
> > > > > > Guozhang
> > > > > >
> > > > > > On Tue, Mar 10, 2015 at 7:58 AM, tao xiao <xiaotao183@gmail.com
> > > > > > <javascript:;>> wrote:
> > > > > >
> > > > > > > I actually mean if we can achieve this in mirror maker.
> > > > > > >
> > > > > > > On Tue, Mar 10, 2015 at 10:52 PM, tao xiao <
> xiaotao183@gmail.com
> > > > > > <javascript:;>> wrote:
> > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > I have an user case where I need to consume a list topics
> with
> > > name
> > > > > > that
> > > > > > > > matches pattern topic.* except for one that is topic.10. Is
> > > there a
> > > > > way
> > > > > > > > that I can combine the use of whitelist and blacklist so
> that I
> > > can
> > > > > > > achieve
> > > > > > > > something like accept all topics with regex topic.* but
> exclude
> > > > > > topic.10?
> > > > > > > >
> > > > > > > > --
> > > > > > > > Regards,
> > > > > > > > Tao
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Regards,
> > > > > > > Tao
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > -- Guozhang
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Regards,
> > > > > Tao
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > -- Guozhang
> > > >
> > >
> > >
> > >
> > > --
> > > Regards,
> > > Tao
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
>
>
> --
> Regards,
> Tao
>
--
-- Guozhang
Re: Does consumer support combination of whitelist and blacklist
topic filtering
Posted by tao xiao <xi...@gmail.com>.
Hi Guozhang,
I was meant to be topicfilter not topic-count. sorry for the confusion.
What I want to achieve is to pass my own customized topicfilter to MM so
that I can filter out topics what ever I like. I know MM doesn't support
this now. I am just thinking if this is a good feature to add in
On Thu, Mar 12, 2015 at 8:24 PM, Guozhang Wang <wa...@gmail.com> wrote:
> Hi Tao,
>
> Sorry I was mistaken before, yes in MM you can only directly specify
> "--whitelist", "--blacklist", and the number of streams you want to create
> via "--num.streams", but cannot set specific topic-count. This is because
> MM is mainly used for cross DC replication, and hence usually will pipe all
> topics or a majority of the topics from the source cluster to the
> destination, hence usually you do not care about "some topics should get X
> streams, while some other topics should get Y streams".
>
> Guozhang
>
> On Wed, Mar 11, 2015 at 11:59 PM, tao xiao <xi...@gmail.com> wrote:
>
> > The topic list is not specified in consumer.properties and I don't think
> > there is any property in consumer config that allows us to specify what
> > topics we want to consume. Can you point me to the property if there is
> > any?
> >
> > On Thu, Mar 12, 2015 at 12:14 AM, Guozhang Wang <wa...@gmail.com>
> > wrote:
> >
> > > Tao,
> > >
> > > In MM people can pass in consumer configs, in which people can specify
> > > consumption topics, either in regular topic list format or whitelist /
> > > blacklist. So I think it already does what you need?
> > >
> > > Guozhang
> > >
> > > On Tue, Mar 10, 2015 at 10:09 PM, tao xiao <xi...@gmail.com>
> wrote:
> > >
> > > > Thank you guys for answering. I think it will be good that we can
> pass
> > > in a
> > > > customised topicCount ( I think this is the interface whitelist and
> > > > backlist implement if I am not mistaken) to MM to achieve similar
> thing
> > > >
> > > > On Wednesday, March 11, 2015, Guozhang Wang <wa...@gmail.com>
> > wrote:
> > > >
> > > > > Hi Tao,
> > > > >
> > > > > Unfortunately MM does not support whitelist / blacklist at the same
> > > time,
> > > > > and you have to choose either one upon initialization. As for your
> > > case,
> > > > I
> > > > > think it can be captured by some reg-ex to exclude nothing else but
> > > "10",
> > > > > but I do not know the exact expression.
> > > > >
> > > > > Guozhang
> > > > >
> > > > > On Tue, Mar 10, 2015 at 7:58 AM, tao xiao <xiaotao183@gmail.com
> > > > > <javascript:;>> wrote:
> > > > >
> > > > > > I actually mean if we can achieve this in mirror maker.
> > > > > >
> > > > > > On Tue, Mar 10, 2015 at 10:52 PM, tao xiao <xiaotao183@gmail.com
> > > > > <javascript:;>> wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > I have an user case where I need to consume a list topics with
> > name
> > > > > that
> > > > > > > matches pattern topic.* except for one that is topic.10. Is
> > there a
> > > > way
> > > > > > > that I can combine the use of whitelist and blacklist so that I
> > can
> > > > > > achieve
> > > > > > > something like accept all topics with regex topic.* but exclude
> > > > > topic.10?
> > > > > > >
> > > > > > > --
> > > > > > > Regards,
> > > > > > > Tao
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Regards,
> > > > > > Tao
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > -- Guozhang
> > > > >
> > > >
> > > >
> > > > --
> > > > Regards,
> > > > Tao
> > > >
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
> >
> >
> > --
> > Regards,
> > Tao
> >
>
>
>
> --
> -- Guozhang
>
--
Regards,
Tao
Re: Does consumer support combination of whitelist and blacklist
topic filtering
Posted by Guozhang Wang <wa...@gmail.com>.
Hi Tao,
Sorry I was mistaken before, yes in MM you can only directly specify
"--whitelist", "--blacklist", and the number of streams you want to create
via "--num.streams", but cannot set specific topic-count. This is because
MM is mainly used for cross DC replication, and hence usually will pipe all
topics or a majority of the topics from the source cluster to the
destination, hence usually you do not care about "some topics should get X
streams, while some other topics should get Y streams".
Guozhang
On Wed, Mar 11, 2015 at 11:59 PM, tao xiao <xi...@gmail.com> wrote:
> The topic list is not specified in consumer.properties and I don't think
> there is any property in consumer config that allows us to specify what
> topics we want to consume. Can you point me to the property if there is
> any?
>
> On Thu, Mar 12, 2015 at 12:14 AM, Guozhang Wang <wa...@gmail.com>
> wrote:
>
> > Tao,
> >
> > In MM people can pass in consumer configs, in which people can specify
> > consumption topics, either in regular topic list format or whitelist /
> > blacklist. So I think it already does what you need?
> >
> > Guozhang
> >
> > On Tue, Mar 10, 2015 at 10:09 PM, tao xiao <xi...@gmail.com> wrote:
> >
> > > Thank you guys for answering. I think it will be good that we can pass
> > in a
> > > customised topicCount ( I think this is the interface whitelist and
> > > backlist implement if I am not mistaken) to MM to achieve similar thing
> > >
> > > On Wednesday, March 11, 2015, Guozhang Wang <wa...@gmail.com>
> wrote:
> > >
> > > > Hi Tao,
> > > >
> > > > Unfortunately MM does not support whitelist / blacklist at the same
> > time,
> > > > and you have to choose either one upon initialization. As for your
> > case,
> > > I
> > > > think it can be captured by some reg-ex to exclude nothing else but
> > "10",
> > > > but I do not know the exact expression.
> > > >
> > > > Guozhang
> > > >
> > > > On Tue, Mar 10, 2015 at 7:58 AM, tao xiao <xiaotao183@gmail.com
> > > > <javascript:;>> wrote:
> > > >
> > > > > I actually mean if we can achieve this in mirror maker.
> > > > >
> > > > > On Tue, Mar 10, 2015 at 10:52 PM, tao xiao <xiaotao183@gmail.com
> > > > <javascript:;>> wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I have an user case where I need to consume a list topics with
> name
> > > > that
> > > > > > matches pattern topic.* except for one that is topic.10. Is
> there a
> > > way
> > > > > > that I can combine the use of whitelist and blacklist so that I
> can
> > > > > achieve
> > > > > > something like accept all topics with regex topic.* but exclude
> > > > topic.10?
> > > > > >
> > > > > > --
> > > > > > Regards,
> > > > > > Tao
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Regards,
> > > > > Tao
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > -- Guozhang
> > > >
> > >
> > >
> > > --
> > > Regards,
> > > Tao
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
>
>
> --
> Regards,
> Tao
>
--
-- Guozhang
Re: Does consumer support combination of whitelist and blacklist
topic filtering
Posted by tao xiao <xi...@gmail.com>.
The topic list is not specified in consumer.properties and I don't think
there is any property in consumer config that allows us to specify what
topics we want to consume. Can you point me to the property if there is any?
On Thu, Mar 12, 2015 at 12:14 AM, Guozhang Wang <wa...@gmail.com> wrote:
> Tao,
>
> In MM people can pass in consumer configs, in which people can specify
> consumption topics, either in regular topic list format or whitelist /
> blacklist. So I think it already does what you need?
>
> Guozhang
>
> On Tue, Mar 10, 2015 at 10:09 PM, tao xiao <xi...@gmail.com> wrote:
>
> > Thank you guys for answering. I think it will be good that we can pass
> in a
> > customised topicCount ( I think this is the interface whitelist and
> > backlist implement if I am not mistaken) to MM to achieve similar thing
> >
> > On Wednesday, March 11, 2015, Guozhang Wang <wa...@gmail.com> wrote:
> >
> > > Hi Tao,
> > >
> > > Unfortunately MM does not support whitelist / blacklist at the same
> time,
> > > and you have to choose either one upon initialization. As for your
> case,
> > I
> > > think it can be captured by some reg-ex to exclude nothing else but
> "10",
> > > but I do not know the exact expression.
> > >
> > > Guozhang
> > >
> > > On Tue, Mar 10, 2015 at 7:58 AM, tao xiao <xiaotao183@gmail.com
> > > <javascript:;>> wrote:
> > >
> > > > I actually mean if we can achieve this in mirror maker.
> > > >
> > > > On Tue, Mar 10, 2015 at 10:52 PM, tao xiao <xiaotao183@gmail.com
> > > <javascript:;>> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I have an user case where I need to consume a list topics with name
> > > that
> > > > > matches pattern topic.* except for one that is topic.10. Is there a
> > way
> > > > > that I can combine the use of whitelist and blacklist so that I can
> > > > achieve
> > > > > something like accept all topics with regex topic.* but exclude
> > > topic.10?
> > > > >
> > > > > --
> > > > > Regards,
> > > > > Tao
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Regards,
> > > > Tao
> > > >
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
> >
> > --
> > Regards,
> > Tao
> >
>
>
>
> --
> -- Guozhang
>
--
Regards,
Tao
Re: Does consumer support combination of whitelist and blacklist
topic filtering
Posted by Guozhang Wang <wa...@gmail.com>.
Tao,
In MM people can pass in consumer configs, in which people can specify
consumption topics, either in regular topic list format or whitelist /
blacklist. So I think it already does what you need?
Guozhang
On Tue, Mar 10, 2015 at 10:09 PM, tao xiao <xi...@gmail.com> wrote:
> Thank you guys for answering. I think it will be good that we can pass in a
> customised topicCount ( I think this is the interface whitelist and
> backlist implement if I am not mistaken) to MM to achieve similar thing
>
> On Wednesday, March 11, 2015, Guozhang Wang <wa...@gmail.com> wrote:
>
> > Hi Tao,
> >
> > Unfortunately MM does not support whitelist / blacklist at the same time,
> > and you have to choose either one upon initialization. As for your case,
> I
> > think it can be captured by some reg-ex to exclude nothing else but "10",
> > but I do not know the exact expression.
> >
> > Guozhang
> >
> > On Tue, Mar 10, 2015 at 7:58 AM, tao xiao <xiaotao183@gmail.com
> > <javascript:;>> wrote:
> >
> > > I actually mean if we can achieve this in mirror maker.
> > >
> > > On Tue, Mar 10, 2015 at 10:52 PM, tao xiao <xiaotao183@gmail.com
> > <javascript:;>> wrote:
> > >
> > > > Hi,
> > > >
> > > > I have an user case where I need to consume a list topics with name
> > that
> > > > matches pattern topic.* except for one that is topic.10. Is there a
> way
> > > > that I can combine the use of whitelist and blacklist so that I can
> > > achieve
> > > > something like accept all topics with regex topic.* but exclude
> > topic.10?
> > > >
> > > > --
> > > > Regards,
> > > > Tao
> > > >
> > >
> > >
> > >
> > > --
> > > Regards,
> > > Tao
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
>
> --
> Regards,
> Tao
>
--
-- Guozhang
Re: Does consumer support combination of whitelist and blacklist
topic filtering
Posted by tao xiao <xi...@gmail.com>.
Thank you guys for answering. I think it will be good that we can pass in a
customised topicCount ( I think this is the interface whitelist and
backlist implement if I am not mistaken) to MM to achieve similar thing
On Wednesday, March 11, 2015, Guozhang Wang <wa...@gmail.com> wrote:
> Hi Tao,
>
> Unfortunately MM does not support whitelist / blacklist at the same time,
> and you have to choose either one upon initialization. As for your case, I
> think it can be captured by some reg-ex to exclude nothing else but "10",
> but I do not know the exact expression.
>
> Guozhang
>
> On Tue, Mar 10, 2015 at 7:58 AM, tao xiao <xiaotao183@gmail.com
> <javascript:;>> wrote:
>
> > I actually mean if we can achieve this in mirror maker.
> >
> > On Tue, Mar 10, 2015 at 10:52 PM, tao xiao <xiaotao183@gmail.com
> <javascript:;>> wrote:
> >
> > > Hi,
> > >
> > > I have an user case where I need to consume a list topics with name
> that
> > > matches pattern topic.* except for one that is topic.10. Is there a way
> > > that I can combine the use of whitelist and blacklist so that I can
> > achieve
> > > something like accept all topics with regex topic.* but exclude
> topic.10?
> > >
> > > --
> > > Regards,
> > > Tao
> > >
> >
> >
> >
> > --
> > Regards,
> > Tao
> >
>
>
>
> --
> -- Guozhang
>
--
Regards,
Tao
Re: Does consumer support combination of whitelist and blacklist
topic filtering
Posted by Guozhang Wang <wa...@gmail.com>.
Hi Tao,
Unfortunately MM does not support whitelist / blacklist at the same time,
and you have to choose either one upon initialization. As for your case, I
think it can be captured by some reg-ex to exclude nothing else but "10",
but I do not know the exact expression.
Guozhang
On Tue, Mar 10, 2015 at 7:58 AM, tao xiao <xi...@gmail.com> wrote:
> I actually mean if we can achieve this in mirror maker.
>
> On Tue, Mar 10, 2015 at 10:52 PM, tao xiao <xi...@gmail.com> wrote:
>
> > Hi,
> >
> > I have an user case where I need to consume a list topics with name that
> > matches pattern topic.* except for one that is topic.10. Is there a way
> > that I can combine the use of whitelist and blacklist so that I can
> achieve
> > something like accept all topics with regex topic.* but exclude topic.10?
> >
> > --
> > Regards,
> > Tao
> >
>
>
>
> --
> Regards,
> Tao
>
--
-- Guozhang
Re: Does consumer support combination of whitelist and blacklist
topic filtering
Posted by tao xiao <xi...@gmail.com>.
I actually mean if we can achieve this in mirror maker.
On Tue, Mar 10, 2015 at 10:52 PM, tao xiao <xi...@gmail.com> wrote:
> Hi,
>
> I have an user case where I need to consume a list topics with name that
> matches pattern topic.* except for one that is topic.10. Is there a way
> that I can combine the use of whitelist and blacklist so that I can achieve
> something like accept all topics with regex topic.* but exclude topic.10?
>
> --
> Regards,
> Tao
>
--
Regards,
Tao