You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by 이승진 <sw...@navercorp.com> on 2015/06/22 11:47:28 UTC
storm-kafka, consumer group support?
Hi all.
Is it possible to use storm-kafka consumers as one consumer group so that two topologies read the same kafka topic in alternating manner?
I think it's not possible currently, if this is the case, wouldn't it be valuable if we support this?
purpose of this functionality is to run topologies without any service down time and also any duplicates in consuming logs.
Re: storm-kafka, consumer group support?
Posted by Seungtack Baek <se...@precocityllc.com>.
Although I don't have extensive knowledge in kafkaspout, I have always
thought that if you have same customerID in the same zookeeper, it will
read from same node in zookeeper, thus one topology reading a message will
update the offset for the both topologies.. I might be wrong tho.
*Seungtack Baek | Precocity, LLC*
Tel/Direct: (972) 378-1030 | Mobile: (214) 477-5715
*SeungtackBaek@precocityllc.com <Se...@precocityllc.com>* |
www.precocityllc.com
This is the end of this message.
--
On Mon, Jun 22, 2015 at 8:13 PM, 이승진 <sw...@navercorp.com> wrote:
> This will duplicate messages to different topologies, though.
>
>
>
> Two same topologies with different id reading same topic in alternating
> manner, it's pretty same to what Kafka's consumer group API behaves.
>
>
>
> -----Original Message-----
> *From:* "Andrey Yegorov"<an...@gmail.com>
> *To:* <us...@storm.apache.org>; "이승진"<sw...@navercorp.com>;
> *Cc:*
> *Sent:* 2015-06-23 (화) 00:52:00
> *Subject:* Re: storm-kafka, consumer group support?
>
>
> On Mon, Jun 22, 2015 at 2:47 AM, 이승진 <sw...@navercorp.com> wrote:
>
> I think it's not possible currently, if this is the case, wouldn't it be
> valuable if we support this?
>
>
> SpoutConfig constructor has a parameter called IIRC "id" to identify
> different clients for the same topic.
> Just use different ids for different topologies.
>
> ----------
> Andrey Yegorov
>
Re: storm-kafka, consumer group support?
Posted by 이승진 <sw...@navercorp.com>.
This will duplicate messages to different topologies, though.
Two same topologies with different id reading same topic in alternating manner, it's pretty same to what Kafka's consumer group API behaves.
-----Original Message-----
From: "Andrey Yegorov"<andrey.yegorov@gmail.com>
To: <user@storm.apache.org>; "이승진"<sweetest.sj@navercorp.com>;
Cc:
Sent: 2015-06-23 (화) 00:52:00
Subject: Re: storm-kafka, consumer group support?
On Mon, Jun 22, 2015 at 2:47 AM, 이승진 <sweetest.sj@navercorp.com> wrote:
I think it's not possible currently, if this is the case, wouldn't it be valuable if we support this?
SpoutConfig constructor has a parameter called IIRC "id" to identify different clients for the same topic.Just use different ids for different topologies.
----------
Andrey Yegorov
Re: storm-kafka, consumer group support?
Posted by Andrey Yegorov <an...@gmail.com>.
On Mon, Jun 22, 2015 at 2:47 AM, 이승진 <sw...@navercorp.com> wrote:
> I think it's not possible currently, if this is the case, wouldn't it be
> valuable if we support this?
>
>
SpoutConfig constructor has a parameter called IIRC "id" to identify
different clients for the same topic.
Just use different ids for different topologies.
----------
Andrey Yegorov