You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Deepak Pengoria <de...@gmail.com> on 2016/08/17 04:53:25 UTC

Multiple processors belongs to same GroupId needs to read same message from same topic

Hi, I have a problem for which I am not able to find the solution. Below is
the problem statement :

I have two Kafka-Steaming api processors say P1 and P2, both want to read
same message(s) from same topic say T. Topic T is having only one partition
and contains some configuration information and this topic doesn't update
frequently (update hardly once in a month). Currently if P1 read the
message from topic T then that P2 will not be able to read this message.

How can I achieve that both the processors could read same message? It is
something on which I am stuck and need help.

Regards,
Deepak

Re: Multiple processors belongs to same GroupId needs to read same message from same topic

Posted by "Matthias J. Sax" <ma...@confluent.io>.
You can just consume the message once, and process it twice downstream:

> KStream stream = builder.stream("topic");
> 
> stream.process(/* use P1 here */);
> stream.process(/* use P2 here */);

-Matthias

On 08/17/2016 07:51 AM, Tanay Soni wrote:
> Hi,
> 
> I am wondering why not have two different GroupIDs for the processors? This
> would ensure that both P1 & P2 read each message from the topic.
> 
> - Tanay
> 
> On Wed 17 Aug, 2016 11:06 am David Garcia, <da...@spiceworks.com> wrote:
> 
>> You could create another partition in topic T, and publish the same
>> message to both partitions.  You would have to configure P2 to read from
>> the other partition.  Or you could have P1 write the message to another
>> topic and configure P2 to listen to that topic.
>>
>> -David
>>
>> On 8/16/16, 11:54 PM, "Deepak Pengoria" <de...@gmail.com> wrote:
>>
>>     For your information, I am using Confluent-3.0.0 (having Streaming
>> api-0.10)
>>
>>     On Wed, Aug 17, 2016 at 10:23 AM, Deepak Pengoria <
>> deepak.pengoria@gmail.com
>>     > wrote:
>>
>>     > Hi, I have a problem for which I am not able to find the solution.
>> Below
>>     > is the problem statement :
>>     >
>>     > I have two Kafka-Steaming api processors say P1 and P2, both want to
>> read
>>     > same message(s) from same topic say T. Topic T is having only one
>> partition
>>     > and contains some configuration information and this topic doesn't
>> update
>>     > frequently (update hardly once in a month). Currently if P1 read the
>>     > message from topic T then that P2 will not be able to read this
>> message.
>>     >
>>     > How can I achieve that both the processors could read same message?
>> It is
>>     > something on which I am stuck and need help.
>>     >
>>     > Regards,
>>     > Deepak
>>     >
>>
>>
>>
>

Re: Multiple processors belongs to same GroupId needs to read same message from same topic

Posted by Tanay Soni <ta...@gmail.com>.
Hi,

I am wondering why not have two different GroupIDs for the processors? This
would ensure that both P1 & P2 read each message from the topic.

- Tanay

On Wed 17 Aug, 2016 11:06 am David Garcia, <da...@spiceworks.com> wrote:

> You could create another partition in topic T, and publish the same
> message to both partitions.  You would have to configure P2 to read from
> the other partition.  Or you could have P1 write the message to another
> topic and configure P2 to listen to that topic.
>
> -David
>
> On 8/16/16, 11:54 PM, "Deepak Pengoria" <de...@gmail.com> wrote:
>
>     For your information, I am using Confluent-3.0.0 (having Streaming
> api-0.10)
>
>     On Wed, Aug 17, 2016 at 10:23 AM, Deepak Pengoria <
> deepak.pengoria@gmail.com
>     > wrote:
>
>     > Hi, I have a problem for which I am not able to find the solution.
> Below
>     > is the problem statement :
>     >
>     > I have two Kafka-Steaming api processors say P1 and P2, both want to
> read
>     > same message(s) from same topic say T. Topic T is having only one
> partition
>     > and contains some configuration information and this topic doesn't
> update
>     > frequently (update hardly once in a month). Currently if P1 read the
>     > message from topic T then that P2 will not be able to read this
> message.
>     >
>     > How can I achieve that both the processors could read same message?
> It is
>     > something on which I am stuck and need help.
>     >
>     > Regards,
>     > Deepak
>     >
>
>
>

Re: Multiple processors belongs to same GroupId needs to read same message from same topic

Posted by David Garcia <da...@spiceworks.com>.
You could create another partition in topic T, and publish the same message to both partitions.  You would have to configure P2 to read from the other partition.  Or you could have P1 write the message to another topic and configure P2 to listen to that topic.

-David

On 8/16/16, 11:54 PM, "Deepak Pengoria" <de...@gmail.com> wrote:

    For your information, I am using Confluent-3.0.0 (having Streaming api-0.10)
    
    On Wed, Aug 17, 2016 at 10:23 AM, Deepak Pengoria <deepak.pengoria@gmail.com
    > wrote:
    
    > Hi, I have a problem for which I am not able to find the solution. Below
    > is the problem statement :
    >
    > I have two Kafka-Steaming api processors say P1 and P2, both want to read
    > same message(s) from same topic say T. Topic T is having only one partition
    > and contains some configuration information and this topic doesn't update
    > frequently (update hardly once in a month). Currently if P1 read the
    > message from topic T then that P2 will not be able to read this message.
    >
    > How can I achieve that both the processors could read same message? It is
    > something on which I am stuck and need help.
    >
    > Regards,
    > Deepak
    >

Re: Multiple processors belongs to same GroupId needs to read same message from same topic

Posted by Deepak Pengoria <de...@gmail.com>.
For your information, I am using Confluent-3.0.0 (having Streaming api-0.10)

On Wed, Aug 17, 2016 at 10:23 AM, Deepak Pengoria <deepak.pengoria@gmail.com
> wrote:

> Hi, I have a problem for which I am not able to find the solution. Below
> is the problem statement :
>
> I have two Kafka-Steaming api processors say P1 and P2, both want to read
> same message(s) from same topic say T. Topic T is having only one partition
> and contains some configuration information and this topic doesn't update
> frequently (update hardly once in a month). Currently if P1 read the
> message from topic T then that P2 will not be able to read this message.
>
> How can I achieve that both the processors could read same message? It is
> something on which I am stuck and need help.
>
> Regards,
> Deepak
>