You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Kuldeep Kamboj <ku...@osscube.com> on 2016/04/22 08:37:44 UTC

How to build strategy for particular setup of kafka

I have three applications let named AppA, AppB, AppC. All could have
several some message queue named like

    AppA
       MsgQueueA
       MsgQueueB
    AppB
       MsgQueueA
       MsgQueueB
       MsgQueueC
    AppC
       MsgQueueA
       MsgQueueB
       MsgQueueD

I have very little exposure for Apache Kafka and even for message queue
concepts.

I just thinking to create topics for MsgQueueA, MsgQueueB and so on. But I
am confused how to integrate AppA, AppB, AppC. Partition and Consumer group
look like wrong choices for them.

Also If separation of queue/topic for Apps is better idea instead of
re-using them for different Apps for better debugging Like below


    AppA
       AppAMsgQueueA
       AppAMsgQueueB
    AppB
       AppBMsgQueueA
       AppBMsgQueueB
       AppBMsgQueueC
    AppC
       AppCMsgQueueA
       AppCMsgQueueB
       AppCMsgQueueD




-- 
Kuldeep Kamboj

Re: How to build strategy for particular setup of kafka

Posted by Kuldeep Kamboj <ku...@osscube.com>.
Thanks for good suggestions, But I postponed the idea of setup apache kafka
due to unfortunately high issues on kafka setup and my very little
experience on distributed architecture / kafka / zookeeper .

On Fri, Apr 22, 2016 at 3:34 PM, Lohith Samaga M <Lo...@mphasis.com>
wrote:

> Hi,
>         Please set up a Kafka cluster. So, you can get high throughput as
> well as high availability.
>
> Best regards / Mit freundlichen Grüßen / Sincères salutations
> M. Lohith Samaga
>
>
> -----Original Message-----
> From: Gaurav Agarwal [mailto:gaurav130403@gmail.com]
> Sent: Friday, April 22, 2016 12.44
> To: users@kafka.apache.org
> Subject: Re: How to build strategy for particular setup of kafka
>
> Hi
> You can have one or two instances of Kafka but you can have one or two
> Kafka topic dedicated to each application according to the need. Partition
> will have u in increasing the throughput and consumer group id can help u
> to make queue as topic or queue.
> On Apr 22, 2016 12:37 PM, "Kuldeep Kamboj" <ku...@osscube.com>
> wrote:
>
> > Thanks for reply,
> >
> > I understand the your point, But my whole strategy depend on first
> > issue and that is how I can integrate Apps in architecture. Partition
> > / Consumer groups have different purpose. If I need to setup three
> > kafka instances each for App ?
> >
> > On Fri, Apr 22, 2016 at 12:30 PM, Lohith Samaga M <
> > Lohith.Samaga@mphasis.com
> > > wrote:
> >
> > > Hi,
> > >         It is better NOT to share topics among applications. You may
> > > have a wrapper application reading from the queue/topic and routing
> > > it to the correct application, but it is simpler for each
> > > application to read from its own topic.
> > >
> > > Best regards / Mit freundlichen Grüßen / Sincères salutations M.
> > > Lohith Samaga
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Kuldeep Kamboj [mailto:kuldeep.kamboj@osscube.com]
> > > Sent: Friday, April 22, 2016 12.08
> > > To: users@kafka.apache.org
> > > Subject: How to build strategy for particular setup of kafka
> > >
> > > I have three applications let named AppA, AppB, AppC. All could have
> > > several some message queue named like
> > >
> > >     AppA
> > >        MsgQueueA
> > >        MsgQueueB
> > >     AppB
> > >        MsgQueueA
> > >        MsgQueueB
> > >        MsgQueueC
> > >     AppC
> > >        MsgQueueA
> > >        MsgQueueB
> > >        MsgQueueD
> > >
> > > I have very little exposure for Apache Kafka and even for message
> > > queue concepts.
> > >
> > > I just thinking to create topics for MsgQueueA, MsgQueueB and so on.
> > > But
> > I
> > > am confused how to integrate AppA, AppB, AppC. Partition and
> > > Consumer
> > group
> > > look like wrong choices for them.
> > >
> > > Also If separation of queue/topic for Apps is better idea instead of
> > > re-using them for different Apps for better debugging Like below
> > >
> > >
> > >     AppA
> > >        AppAMsgQueueA
> > >        AppAMsgQueueB
> > >     AppB
> > >        AppBMsgQueueA
> > >        AppBMsgQueueB
> > >        AppBMsgQueueC
> > >     AppC
> > >        AppCMsgQueueA
> > >        AppCMsgQueueB
> > >        AppCMsgQueueD
> > >
> > >
> > >
> > >
> > > --
> > > Kuldeep Kamboj
> > > Information transmitted by this e-mail is proprietary to Mphasis,
> > > its associated companies and/ or its customers and is intended for
> > > use only by the individual or entity to which it is addressed, and
> > may
> > > contain information that is privileged, confidential or exempt from
> > > disclosure under applicable law. If you are not the intended
> > > recipient or it appears that this mail has been forwarded to you
> > > without proper authority, you are notified that any use or
> > > dissemination of this information in any manner is strictly
> > > prohibited. In such cases, please notify us immediately at
> > > mailmaster@mphasis.com and delete this mail from your records.
> > >
> >
> >
> >
> > --
> > Kuldeep Kamboj
> > Mob. 9871662849
> >
> Information transmitted by this e-mail is proprietary to Mphasis, its
> associated companies and/ or its customers and is intended
> for use only by the individual or entity to which it is addressed, and may
> contain information that is privileged, confidential or
> exempt from disclosure under applicable law. If you are not the intended
> recipient or it appears that this mail has been forwarded
> to you without proper authority, you are notified that any use or
> dissemination of this information in any manner is strictly
> prohibited. In such cases, please notify us immediately at
> mailmaster@mphasis.com and delete this mail from your records.
>



-- 
Kuldeep Kamboj
Mob. 9871662849

Re: How to build strategy for particular setup of kafka

Posted by Kuldeep Kamboj <ku...@osscube.com>.
Sorry for wrong php code. It should be like that

3. In php code, Instead of using partition = 0, Use App1 = 0, App2 = 1,
App3 = 2

   In App 1 Producer Script :
       $producer->setMessages($queue, 0, array($data));

   In App 2 Producer Script :
       $producer->setMessages($queue, 1, array($data));

   In Consumer 1 Script :
       $consumer->setPartition($queue, 0);

   In Consumer 2 Script :
       $consumer->setPartition($queue, 1);


On Fri, Apr 29, 2016 at 5:17 PM, Kuldeep Kamboj <ku...@osscube.com>
wrote:

> I restarted brainstroming for Kafka setup. I found link
> http://www.michael-noll.com/blog/2013/03/13/running-a-multi-broker-apache-kafka-cluster-on-a-single-node/
> for kafka cluster setup (Thanks Lohith Samaga for valuable suggestion) .
> After try on local machine, I got this is what I need.
>
> Now I am implementing following approach.
>
> 1. First create 3 brokers
>     cd /opt/kafka
>     cp config/server.properties config/broker1.properties
>         broker.id=1
>         port=9092
>         log.dir=/tmp/kafka-logs-1
>
>     cp config/server.properties config/broker2.properties
>         broker.id=2
>         port=9093
>         log.dir=/tmp/kafka-logs-2
>
>     cp config/server.properties config/broker3.properties
>         broker.id=3
>         port=9094
>         log.dir=/tmp/kafka-logs-3
>
>
> 2. Create topics with 3 partitions & 3 replications
>              /opt/kafka/bin/kafka-topics.sh --zookeeper localhost:2181
> --create --topic category --partitions 3            --replication-factor
> 3
>         ...
>
> 3. In php code, Instead of using partition = 0, Use App1 = 1, App2 = 2,
> App3 = 3
>
>    In App 1 Producer Script :
>        $producer->setMessages($queue, 0, array($data));
>
>    In App 2 Producer Script :
>        $producer->setMessages($queue, 2, array($data));
>
>    In Consumer 1 Script :
>        $consumer->setPartition($queue, 0);
>
>    In Consumer 2 Script :
>        $consumer->setPartition($queue, 1);
>
> I expect, it should work as my requirements.
>
>
> On Fri, Apr 22, 2016 at 3:34 PM, Lohith Samaga M <
> Lohith.Samaga@mphasis.com> wrote:
>
>> Hi,
>>         Please set up a Kafka cluster. So, you can get high throughput as
>> well as high availability.
>>
>> Best regards / Mit freundlichen Grüßen / Sincères salutations
>> M. Lohith Samaga
>>
>>
>> -----Original Message-----
>> From: Gaurav Agarwal [mailto:gaurav130403@gmail.com]
>> Sent: Friday, April 22, 2016 12.44
>> To: users@kafka.apache.org
>> Subject: Re: How to build strategy for particular setup of kafka
>>
>> Hi
>> You can have one or two instances of Kafka but you can have one or two
>> Kafka topic dedicated to each application according to the need. Partition
>> will have u in increasing the throughput and consumer group id can help u
>> to make queue as topic or queue.
>> On Apr 22, 2016 12:37 PM, "Kuldeep Kamboj" <ku...@osscube.com>
>> wrote:
>>
>> > Thanks for reply,
>> >
>> > I understand the your point, But my whole strategy depend on first
>> > issue and that is how I can integrate Apps in architecture. Partition
>> > / Consumer groups have different purpose. If I need to setup three
>> > kafka instances each for App ?
>> >
>> > On Fri, Apr 22, 2016 at 12:30 PM, Lohith Samaga M <
>> > Lohith.Samaga@mphasis.com
>> > > wrote:
>> >
>> > > Hi,
>> > >         It is better NOT to share topics among applications. You may
>> > > have a wrapper application reading from the queue/topic and routing
>> > > it to the correct application, but it is simpler for each
>> > > application to read from its own topic.
>> > >
>> > > Best regards / Mit freundlichen Grüßen / Sincères salutations M.
>> > > Lohith Samaga
>> > >
>> > >
>> > >
>> > > -----Original Message-----
>> > > From: Kuldeep Kamboj [mailto:kuldeep.kamboj@osscube.com]
>> > > Sent: Friday, April 22, 2016 12.08
>> > > To: users@kafka.apache.org
>> > > Subject: How to build strategy for particular setup of kafka
>> > >
>> > > I have three applications let named AppA, AppB, AppC. All could have
>> > > several some message queue named like
>> > >
>> > >     AppA
>> > >        MsgQueueA
>> > >        MsgQueueB
>> > >     AppB
>> > >        MsgQueueA
>> > >        MsgQueueB
>> > >        MsgQueueC
>> > >     AppC
>> > >        MsgQueueA
>> > >        MsgQueueB
>> > >        MsgQueueD
>> > >
>> > > I have very little exposure for Apache Kafka and even for message
>> > > queue concepts.
>> > >
>> > > I just thinking to create topics for MsgQueueA, MsgQueueB and so on.
>> > > But
>> > I
>> > > am confused how to integrate AppA, AppB, AppC. Partition and
>> > > Consumer
>> > group
>> > > look like wrong choices for them.
>> > >
>> > > Also If separation of queue/topic for Apps is better idea instead of
>> > > re-using them for different Apps for better debugging Like below
>> > >
>> > >
>> > >     AppA
>> > >        AppAMsgQueueA
>> > >        AppAMsgQueueB
>> > >     AppB
>> > >        AppBMsgQueueA
>> > >        AppBMsgQueueB
>> > >        AppBMsgQueueC
>> > >     AppC
>> > >        AppCMsgQueueA
>> > >        AppCMsgQueueB
>> > >        AppCMsgQueueD
>> > >
>> > >
>> > >
>> > >
>> > > --
>> > > Kuldeep Kamboj
>> > > Information transmitted by this e-mail is proprietary to Mphasis,
>> > > its associated companies and/ or its customers and is intended for
>> > > use only by the individual or entity to which it is addressed, and
>> > may
>> > > contain information that is privileged, confidential or exempt from
>> > > disclosure under applicable law. If you are not the intended
>> > > recipient or it appears that this mail has been forwarded to you
>> > > without proper authority, you are notified that any use or
>> > > dissemination of this information in any manner is strictly
>> > > prohibited. In such cases, please notify us immediately at
>> > > mailmaster@mphasis.com and delete this mail from your records.
>> > >
>> >
>> >
>> >
>> > --
>> > Kuldeep Kamboj
>> > Mob. 9871662849
>> >
>> Information transmitted by this e-mail is proprietary to Mphasis, its
>> associated companies and/ or its customers and is intended
>> for use only by the individual or entity to which it is addressed, and
>> may contain information that is privileged, confidential or
>> exempt from disclosure under applicable law. If you are not the intended
>> recipient or it appears that this mail has been forwarded
>> to you without proper authority, you are notified that any use or
>> dissemination of this information in any manner is strictly
>> prohibited. In such cases, please notify us immediately at
>> mailmaster@mphasis.com and delete this mail from your records.
>>
>
>
>
> --
> Kuldeep Kamboj
> Mob. 9871662849
>



-- 
Kuldeep Kamboj
Mob. 9871662849

Re: How to build strategy for particular setup of kafka

Posted by Kuldeep Kamboj <ku...@osscube.com>.
I restarted brainstroming for Kafka setup. I found link
http://www.michael-noll.com/blog/2013/03/13/running-a-multi-broker-apache-kafka-cluster-on-a-single-node/
for kafka cluster setup (Thanks Lohith Samaga for valuable suggestion) .
After try on local machine, I got this is what I need.

Now I am implementing following approach.

1. First create 3 brokers
    cd /opt/kafka
    cp config/server.properties config/broker1.properties
        broker.id=1
        port=9092
        log.dir=/tmp/kafka-logs-1

    cp config/server.properties config/broker2.properties
        broker.id=2
        port=9093
        log.dir=/tmp/kafka-logs-2

    cp config/server.properties config/broker3.properties
        broker.id=3
        port=9094
        log.dir=/tmp/kafka-logs-3


2. Create topics with 3 partitions & 3 replications
             /opt/kafka/bin/kafka-topics.sh --zookeeper localhost:2181
--create --topic category --partitions 3            --replication-factor
3
        ...

3. In php code, Instead of using partition = 0, Use App1 = 1, App2 = 2,
App3 = 3

   In App 1 Producer Script :
       $producer->setMessages($queue, 0, array($data));

   In App 2 Producer Script :
       $producer->setMessages($queue, 2, array($data));

   In Consumer 1 Script :
       $consumer->setPartition($queue, 0);

   In Consumer 2 Script :
       $consumer->setPartition($queue, 1);

I expect, it should work as my requirements.


On Fri, Apr 22, 2016 at 3:34 PM, Lohith Samaga M <Lo...@mphasis.com>
wrote:

> Hi,
>         Please set up a Kafka cluster. So, you can get high throughput as
> well as high availability.
>
> Best regards / Mit freundlichen Grüßen / Sincères salutations
> M. Lohith Samaga
>
>
> -----Original Message-----
> From: Gaurav Agarwal [mailto:gaurav130403@gmail.com]
> Sent: Friday, April 22, 2016 12.44
> To: users@kafka.apache.org
> Subject: Re: How to build strategy for particular setup of kafka
>
> Hi
> You can have one or two instances of Kafka but you can have one or two
> Kafka topic dedicated to each application according to the need. Partition
> will have u in increasing the throughput and consumer group id can help u
> to make queue as topic or queue.
> On Apr 22, 2016 12:37 PM, "Kuldeep Kamboj" <ku...@osscube.com>
> wrote:
>
> > Thanks for reply,
> >
> > I understand the your point, But my whole strategy depend on first
> > issue and that is how I can integrate Apps in architecture. Partition
> > / Consumer groups have different purpose. If I need to setup three
> > kafka instances each for App ?
> >
> > On Fri, Apr 22, 2016 at 12:30 PM, Lohith Samaga M <
> > Lohith.Samaga@mphasis.com
> > > wrote:
> >
> > > Hi,
> > >         It is better NOT to share topics among applications. You may
> > > have a wrapper application reading from the queue/topic and routing
> > > it to the correct application, but it is simpler for each
> > > application to read from its own topic.
> > >
> > > Best regards / Mit freundlichen Grüßen / Sincères salutations M.
> > > Lohith Samaga
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Kuldeep Kamboj [mailto:kuldeep.kamboj@osscube.com]
> > > Sent: Friday, April 22, 2016 12.08
> > > To: users@kafka.apache.org
> > > Subject: How to build strategy for particular setup of kafka
> > >
> > > I have three applications let named AppA, AppB, AppC. All could have
> > > several some message queue named like
> > >
> > >     AppA
> > >        MsgQueueA
> > >        MsgQueueB
> > >     AppB
> > >        MsgQueueA
> > >        MsgQueueB
> > >        MsgQueueC
> > >     AppC
> > >        MsgQueueA
> > >        MsgQueueB
> > >        MsgQueueD
> > >
> > > I have very little exposure for Apache Kafka and even for message
> > > queue concepts.
> > >
> > > I just thinking to create topics for MsgQueueA, MsgQueueB and so on.
> > > But
> > I
> > > am confused how to integrate AppA, AppB, AppC. Partition and
> > > Consumer
> > group
> > > look like wrong choices for them.
> > >
> > > Also If separation of queue/topic for Apps is better idea instead of
> > > re-using them for different Apps for better debugging Like below
> > >
> > >
> > >     AppA
> > >        AppAMsgQueueA
> > >        AppAMsgQueueB
> > >     AppB
> > >        AppBMsgQueueA
> > >        AppBMsgQueueB
> > >        AppBMsgQueueC
> > >     AppC
> > >        AppCMsgQueueA
> > >        AppCMsgQueueB
> > >        AppCMsgQueueD
> > >
> > >
> > >
> > >
> > > --
> > > Kuldeep Kamboj
> > > Information transmitted by this e-mail is proprietary to Mphasis,
> > > its associated companies and/ or its customers and is intended for
> > > use only by the individual or entity to which it is addressed, and
> > may
> > > contain information that is privileged, confidential or exempt from
> > > disclosure under applicable law. If you are not the intended
> > > recipient or it appears that this mail has been forwarded to you
> > > without proper authority, you are notified that any use or
> > > dissemination of this information in any manner is strictly
> > > prohibited. In such cases, please notify us immediately at
> > > mailmaster@mphasis.com and delete this mail from your records.
> > >
> >
> >
> >
> > --
> > Kuldeep Kamboj
> > Mob. 9871662849
> >
> Information transmitted by this e-mail is proprietary to Mphasis, its
> associated companies and/ or its customers and is intended
> for use only by the individual or entity to which it is addressed, and may
> contain information that is privileged, confidential or
> exempt from disclosure under applicable law. If you are not the intended
> recipient or it appears that this mail has been forwarded
> to you without proper authority, you are notified that any use or
> dissemination of this information in any manner is strictly
> prohibited. In such cases, please notify us immediately at
> mailmaster@mphasis.com and delete this mail from your records.
>



-- 
Kuldeep Kamboj
Mob. 9871662849

RE: How to build strategy for particular setup of kafka

Posted by Lohith Samaga M <Lo...@mphasis.com>.
Hi,
	Please set up a Kafka cluster. So, you can get high throughput as well as high availability.

Best regards / Mit freundlichen Grüßen / Sincères salutations
M. Lohith Samaga


-----Original Message-----
From: Gaurav Agarwal [mailto:gaurav130403@gmail.com] 
Sent: Friday, April 22, 2016 12.44
To: users@kafka.apache.org
Subject: Re: How to build strategy for particular setup of kafka

Hi
You can have one or two instances of Kafka but you can have one or two Kafka topic dedicated to each application according to the need. Partition will have u in increasing the throughput and consumer group id can help u to make queue as topic or queue.
On Apr 22, 2016 12:37 PM, "Kuldeep Kamboj" <ku...@osscube.com>
wrote:

> Thanks for reply,
>
> I understand the your point, But my whole strategy depend on first 
> issue and that is how I can integrate Apps in architecture. Partition 
> / Consumer groups have different purpose. If I need to setup three 
> kafka instances each for App ?
>
> On Fri, Apr 22, 2016 at 12:30 PM, Lohith Samaga M < 
> Lohith.Samaga@mphasis.com
> > wrote:
>
> > Hi,
> >         It is better NOT to share topics among applications. You may 
> > have a wrapper application reading from the queue/topic and routing 
> > it to the correct application, but it is simpler for each 
> > application to read from its own topic.
> >
> > Best regards / Mit freundlichen Grüßen / Sincères salutations M. 
> > Lohith Samaga
> >
> >
> >
> > -----Original Message-----
> > From: Kuldeep Kamboj [mailto:kuldeep.kamboj@osscube.com]
> > Sent: Friday, April 22, 2016 12.08
> > To: users@kafka.apache.org
> > Subject: How to build strategy for particular setup of kafka
> >
> > I have three applications let named AppA, AppB, AppC. All could have 
> > several some message queue named like
> >
> >     AppA
> >        MsgQueueA
> >        MsgQueueB
> >     AppB
> >        MsgQueueA
> >        MsgQueueB
> >        MsgQueueC
> >     AppC
> >        MsgQueueA
> >        MsgQueueB
> >        MsgQueueD
> >
> > I have very little exposure for Apache Kafka and even for message 
> > queue concepts.
> >
> > I just thinking to create topics for MsgQueueA, MsgQueueB and so on. 
> > But
> I
> > am confused how to integrate AppA, AppB, AppC. Partition and 
> > Consumer
> group
> > look like wrong choices for them.
> >
> > Also If separation of queue/topic for Apps is better idea instead of 
> > re-using them for different Apps for better debugging Like below
> >
> >
> >     AppA
> >        AppAMsgQueueA
> >        AppAMsgQueueB
> >     AppB
> >        AppBMsgQueueA
> >        AppBMsgQueueB
> >        AppBMsgQueueC
> >     AppC
> >        AppCMsgQueueA
> >        AppCMsgQueueB
> >        AppCMsgQueueD
> >
> >
> >
> >
> > --
> > Kuldeep Kamboj
> > Information transmitted by this e-mail is proprietary to Mphasis, 
> > its associated companies and/ or its customers and is intended for 
> > use only by the individual or entity to which it is addressed, and
> may
> > contain information that is privileged, confidential or exempt from 
> > disclosure under applicable law. If you are not the intended 
> > recipient or it appears that this mail has been forwarded to you 
> > without proper authority, you are notified that any use or 
> > dissemination of this information in any manner is strictly 
> > prohibited. In such cases, please notify us immediately at 
> > mailmaster@mphasis.com and delete this mail from your records.
> >
>
>
>
> --
> Kuldeep Kamboj
> Mob. 9871662849
>
Information transmitted by this e-mail is proprietary to Mphasis, its associated companies and/ or its customers and is intended 
for use only by the individual or entity to which it is addressed, and may contain information that is privileged, confidential or 
exempt from disclosure under applicable law. If you are not the intended recipient or it appears that this mail has been forwarded 
to you without proper authority, you are notified that any use or dissemination of this information in any manner is strictly 
prohibited. In such cases, please notify us immediately at mailmaster@mphasis.com and delete this mail from your records.

Re: How to build strategy for particular setup of kafka

Posted by Gaurav Agarwal <ga...@gmail.com>.
Hi
You can have one or two instances of Kafka but you can have one or two
Kafka topic dedicated to each application according to the need. Partition
will have u in increasing the throughput and consumer group id can help u
to make queue as topic or queue.
On Apr 22, 2016 12:37 PM, "Kuldeep Kamboj" <ku...@osscube.com>
wrote:

> Thanks for reply,
>
> I understand the your point, But my whole strategy depend on first issue
> and that is how I can integrate Apps in architecture. Partition / Consumer
> groups have different purpose. If I need to setup three kafka instances
> each for App ?
>
> On Fri, Apr 22, 2016 at 12:30 PM, Lohith Samaga M <
> Lohith.Samaga@mphasis.com
> > wrote:
>
> > Hi,
> >         It is better NOT to share topics among applications. You may have
> > a wrapper application reading from the queue/topic and routing it to the
> > correct application, but it is simpler for each application to read from
> > its own topic.
> >
> > Best regards / Mit freundlichen Grüßen / Sincères salutations
> > M. Lohith Samaga
> >
> >
> >
> > -----Original Message-----
> > From: Kuldeep Kamboj [mailto:kuldeep.kamboj@osscube.com]
> > Sent: Friday, April 22, 2016 12.08
> > To: users@kafka.apache.org
> > Subject: How to build strategy for particular setup of kafka
> >
> > I have three applications let named AppA, AppB, AppC. All could have
> > several some message queue named like
> >
> >     AppA
> >        MsgQueueA
> >        MsgQueueB
> >     AppB
> >        MsgQueueA
> >        MsgQueueB
> >        MsgQueueC
> >     AppC
> >        MsgQueueA
> >        MsgQueueB
> >        MsgQueueD
> >
> > I have very little exposure for Apache Kafka and even for message queue
> > concepts.
> >
> > I just thinking to create topics for MsgQueueA, MsgQueueB and so on. But
> I
> > am confused how to integrate AppA, AppB, AppC. Partition and Consumer
> group
> > look like wrong choices for them.
> >
> > Also If separation of queue/topic for Apps is better idea instead of
> > re-using them for different Apps for better debugging Like below
> >
> >
> >     AppA
> >        AppAMsgQueueA
> >        AppAMsgQueueB
> >     AppB
> >        AppBMsgQueueA
> >        AppBMsgQueueB
> >        AppBMsgQueueC
> >     AppC
> >        AppCMsgQueueA
> >        AppCMsgQueueB
> >        AppCMsgQueueD
> >
> >
> >
> >
> > --
> > Kuldeep Kamboj
> > Information transmitted by this e-mail is proprietary to Mphasis, its
> > associated companies and/ or its customers and is intended
> > for use only by the individual or entity to which it is addressed, and
> may
> > contain information that is privileged, confidential or
> > exempt from disclosure under applicable law. If you are not the intended
> > recipient or it appears that this mail has been forwarded
> > to you without proper authority, you are notified that any use or
> > dissemination of this information in any manner is strictly
> > prohibited. In such cases, please notify us immediately at
> > mailmaster@mphasis.com and delete this mail from your records.
> >
>
>
>
> --
> Kuldeep Kamboj
> Mob. 9871662849
>

Re: How to build strategy for particular setup of kafka

Posted by Kuldeep Kamboj <ku...@osscube.com>.
Thanks for reply,

I understand the your point, But my whole strategy depend on first issue
and that is how I can integrate Apps in architecture. Partition / Consumer
groups have different purpose. If I need to setup three kafka instances
each for App ?

On Fri, Apr 22, 2016 at 12:30 PM, Lohith Samaga M <Lohith.Samaga@mphasis.com
> wrote:

> Hi,
>         It is better NOT to share topics among applications. You may have
> a wrapper application reading from the queue/topic and routing it to the
> correct application, but it is simpler for each application to read from
> its own topic.
>
> Best regards / Mit freundlichen Grüßen / Sincères salutations
> M. Lohith Samaga
>
>
>
> -----Original Message-----
> From: Kuldeep Kamboj [mailto:kuldeep.kamboj@osscube.com]
> Sent: Friday, April 22, 2016 12.08
> To: users@kafka.apache.org
> Subject: How to build strategy for particular setup of kafka
>
> I have three applications let named AppA, AppB, AppC. All could have
> several some message queue named like
>
>     AppA
>        MsgQueueA
>        MsgQueueB
>     AppB
>        MsgQueueA
>        MsgQueueB
>        MsgQueueC
>     AppC
>        MsgQueueA
>        MsgQueueB
>        MsgQueueD
>
> I have very little exposure for Apache Kafka and even for message queue
> concepts.
>
> I just thinking to create topics for MsgQueueA, MsgQueueB and so on. But I
> am confused how to integrate AppA, AppB, AppC. Partition and Consumer group
> look like wrong choices for them.
>
> Also If separation of queue/topic for Apps is better idea instead of
> re-using them for different Apps for better debugging Like below
>
>
>     AppA
>        AppAMsgQueueA
>        AppAMsgQueueB
>     AppB
>        AppBMsgQueueA
>        AppBMsgQueueB
>        AppBMsgQueueC
>     AppC
>        AppCMsgQueueA
>        AppCMsgQueueB
>        AppCMsgQueueD
>
>
>
>
> --
> Kuldeep Kamboj
> Information transmitted by this e-mail is proprietary to Mphasis, its
> associated companies and/ or its customers and is intended
> for use only by the individual or entity to which it is addressed, and may
> contain information that is privileged, confidential or
> exempt from disclosure under applicable law. If you are not the intended
> recipient or it appears that this mail has been forwarded
> to you without proper authority, you are notified that any use or
> dissemination of this information in any manner is strictly
> prohibited. In such cases, please notify us immediately at
> mailmaster@mphasis.com and delete this mail from your records.
>



-- 
Kuldeep Kamboj
Mob. 9871662849

RE: How to build strategy for particular setup of kafka

Posted by Lohith Samaga M <Lo...@mphasis.com>.
Hi,
	It is better NOT to share topics among applications. You may have a wrapper application reading from the queue/topic and routing it to the correct application, but it is simpler for each application to read from its own topic.

Best regards / Mit freundlichen Grüßen / Sincères salutations
M. Lohith Samaga



-----Original Message-----
From: Kuldeep Kamboj [mailto:kuldeep.kamboj@osscube.com] 
Sent: Friday, April 22, 2016 12.08
To: users@kafka.apache.org
Subject: How to build strategy for particular setup of kafka

I have three applications let named AppA, AppB, AppC. All could have several some message queue named like

    AppA
       MsgQueueA
       MsgQueueB
    AppB
       MsgQueueA
       MsgQueueB
       MsgQueueC
    AppC
       MsgQueueA
       MsgQueueB
       MsgQueueD

I have very little exposure for Apache Kafka and even for message queue concepts.

I just thinking to create topics for MsgQueueA, MsgQueueB and so on. But I am confused how to integrate AppA, AppB, AppC. Partition and Consumer group look like wrong choices for them.

Also If separation of queue/topic for Apps is better idea instead of re-using them for different Apps for better debugging Like below


    AppA
       AppAMsgQueueA
       AppAMsgQueueB
    AppB
       AppBMsgQueueA
       AppBMsgQueueB
       AppBMsgQueueC
    AppC
       AppCMsgQueueA
       AppCMsgQueueB
       AppCMsgQueueD




--
Kuldeep Kamboj
Information transmitted by this e-mail is proprietary to Mphasis, its associated companies and/ or its customers and is intended 
for use only by the individual or entity to which it is addressed, and may contain information that is privileged, confidential or 
exempt from disclosure under applicable law. If you are not the intended recipient or it appears that this mail has been forwarded 
to you without proper authority, you are notified that any use or dissemination of this information in any manner is strictly 
prohibited. In such cases, please notify us immediately at mailmaster@mphasis.com and delete this mail from your records.