You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by sat <sa...@gmail.com> on 2016/08/04 03:24:08 UTC
Kafka consumers in cluster
Hi,
We have Kafka server/broker running in a seperate machine (say machine A),
for now we are planning to have in one node. We have multiple topics and
all topics have only 1 partition for now.
We have our application which includes Kafka consumers installed in machine
B and machine C. Our application in machine B and C are in cluster, hence
Kafka Consumers will also be in cluster. Both our consumers will have same
group id. We want all the messages to be consumed by consumer in machine B
and only when machine B is down consumer in machine C should pull messages.
Since consumer in machine B and C have same group id, we came to know
consumer in machine B will get message for some time duration (10mins) and
then consumer in machine C will get message for some time duration. Since
our consumers are in cluster, we want only consumer to be active or receive
all the messages as long as it is alive.
Please let us know how to achieve this.
Thanks and Regards
A.SathishKumar
044-24735023
Re: Kafka consumers in cluster
Posted by Kamal C <ka...@gmail.com>.
Implement your own custom
`org.apache.kafka.clients.consumer.internals.PartitionAssignor`
and assign all the subscribed partitions to the first consumer instance in
the group.
See 'partition.assignment.strategy' config in the consumer configs [1]
[1]: http://kafka.apache.org/documentation.html#newconsumerconfigs
On Thu, Aug 4, 2016 at 8:54 AM, sat <sa...@gmail.com> wrote:
> Hi,
>
> We have Kafka server/broker running in a seperate machine (say machine A),
> for now we are planning to have in one node. We have multiple topics and
> all topics have only 1 partition for now.
>
> We have our application which includes Kafka consumers installed in machine
> B and machine C. Our application in machine B and C are in cluster, hence
> Kafka Consumers will also be in cluster. Both our consumers will have same
> group id. We want all the messages to be consumed by consumer in machine B
> and only when machine B is down consumer in machine C should pull messages.
>
> Since consumer in machine B and C have same group id, we came to know
> consumer in machine B will get message for some time duration (10mins) and
> then consumer in machine C will get message for some time duration. Since
> our consumers are in cluster, we want only consumer to be active or receive
> all the messages as long as it is alive.
>
> Please let us know how to achieve this.
>
>
> Thanks and Regards
> A.SathishKumar
> 044-24735023
>
Re: Kafka consumers in cluster
Posted by sat <sa...@gmail.com>.
Thanks Kamal,
>Yes, it gets called on every re-balance.
>-- Kamal
On Thu, Aug 4, 2016 at 10:54 AM, sat <sa...@gmail.com> wrote:
> Hi Kamal,
>
> Thanks for your prompt response. Does our custom partition assignor gets
> called during every rebalancing.
>
> Thanks and Regards
> A.SathishKumar
>
>
>
> >Implement your own custom
> >`org.apache.kafka.clients.consumer.internals.PartitionAssignor`
> >and assign all the subscribed partitions to the first consumer instance in
> >the group.
>
> >See 'partition.assignment.strategy' config in the consumer configs [1]
>
> >[1]: http://kafka.apache.org/documentation.html#newconsumerconfigs
>
>
>
>
> >On Thu, Aug 4, 2016 at 8:54 AM, sat <sa...@gmail.com> wrote:
>
> > Hi,
> >
> > We have Kafka server/broker running in a seperate machine (say machine A),
> > for now we are planning to have in one node. We have multiple topics and
> > all topics have only 1 partition for now.
> >
> > We have our application which includes Kafka consumers installed in machine
> > B and machine C. Our application in machine B and C are in cluster, hence
> > Kafka Consumers will also be in cluster. Both our consumers will have same
> > group id. We want all the messages to be consumed by consumer in machine B
> > and only when machine B is down consumer in machine C should pull messages.
> >
> > Since consumer in machine B and C have same group id, we came to know
> > consumer in machine B will get message for some time duration (10mins) and
> > then consumer in machine C will get message for some time duration. Since
> > our consumers are in cluster, we want only consumer to be active or receive
> > all the messages as long as it is alive.
> >
> > Please let us know how to achieve this.
> >
> >
> > Thanks and Regards
> > A.SathishKumar
> > 044-24735023
> >
>
>
>
> On Wed, Aug 3, 2016 at 8:24 PM, sat <sa...@gmail.com> wrote:
>
>> Hi,
>>
>> We have Kafka server/broker running in a seperate machine (say machine
>> A), for now we are planning to have in one node. We have multiple topics
>> and all topics have only 1 partition for now.
>>
>> We have our application which includes Kafka consumers installed in
>> machine B and machine C. Our application in machine B and C are in cluster,
>> hence Kafka Consumers will also be in cluster. Both our consumers will have
>> same group id. We want all the messages to be consumed by consumer in
>> machine B and only when machine B is down consumer in machine C should pull
>> messages.
>>
>> Since consumer in machine B and C have same group id, we came to know
>> consumer in machine B will get message for some time duration (10mins) and
>> then consumer in machine C will get message for some time duration. Since
>> our consumers are in cluster, we want only consumer to be active or receive
>> all the messages as long as it is alive.
>>
>> Please let us know how to achieve this.
>>
>>
>> Thanks and Regards
>> A.SathishKumar
>> 044-24735023
>>
>
>
>
> --
> A.SathishKumar
> 044-24735023
>
--
A.SathishKumar
044-24735023
Re: Kafka consumers in cluster
Posted by Kamal C <ka...@gmail.com>.
Yes, it gets called on every re-balance.
-- Kamal
On Thu, Aug 4, 2016 at 11:24 PM, sat <sa...@gmail.com> wrote:
> Hi Kamal,
>
> Thanks for your prompt response. Does our custom partition assignor gets
> called during every rebalancing.
>
> Thanks and Regards
> A.SathishKumar
>
>
>
> >Implement your own custom
> >`org.apache.kafka.clients.consumer.internals.PartitionAssignor`
> >and assign all the subscribed partitions to the first consumer instance in
> >the group.
>
> >See 'partition.assignment.strategy' config in the consumer configs [1]
>
> >[1]: http://kafka.apache.org/documentation.html#newconsumerconfigs
>
>
>
> >On Thu, Aug 4, 2016 at 8:54 AM, sat <sa...@gmail.com> wrote:
>
> > Hi,
> >
> > We have Kafka server/broker running in a seperate machine (say machine
> A),
> > for now we are planning to have in one node. We have multiple topics and
> > all topics have only 1 partition for now.
> >
> > We have our application which includes Kafka consumers installed in
> machine
> > B and machine C. Our application in machine B and C are in cluster, hence
> > Kafka Consumers will also be in cluster. Both our consumers will have
> same
> > group id. We want all the messages to be consumed by consumer in machine
> B
> > and only when machine B is down consumer in machine C should pull
> messages.
> >
> > Since consumer in machine B and C have same group id, we came to know
> > consumer in machine B will get message for some time duration (10mins)
> and
> > then consumer in machine C will get message for some time duration. Since
> > our consumers are in cluster, we want only consumer to be active or
> receive
> > all the messages as long as it is alive.
> >
> > Please let us know how to achieve this.
> >
> >
> > Thanks and Regards
> > A.SathishKumar
> > 044-24735023
> >
>
>
>
> On Wed, Aug 3, 2016 at 8:24 PM, sat <sa...@gmail.com> wrote:
>
> > Hi,
> >
> > We have Kafka server/broker running in a seperate machine (say machine
> A),
> > for now we are planning to have in one node. We have multiple topics and
> > all topics have only 1 partition for now.
> >
> > We have our application which includes Kafka consumers installed in
> > machine B and machine C. Our application in machine B and C are in
> cluster,
> > hence Kafka Consumers will also be in cluster. Both our consumers will
> have
> > same group id. We want all the messages to be consumed by consumer in
> > machine B and only when machine B is down consumer in machine C should
> pull
> > messages.
> >
> > Since consumer in machine B and C have same group id, we came to know
> > consumer in machine B will get message for some time duration (10mins)
> and
> > then consumer in machine C will get message for some time duration. Since
> > our consumers are in cluster, we want only consumer to be active or
> receive
> > all the messages as long as it is alive.
> >
> > Please let us know how to achieve this.
> >
> >
> > Thanks and Regards
> > A.SathishKumar
> > 044-24735023
> >
>
>
>
> --
> A.SathishKumar
> 044-24735023
>
Re: Kafka consumers in cluster
Posted by sat <sa...@gmail.com>.
Hi Kamal,
Thanks for your prompt response. Does our custom partition assignor gets
called during every rebalancing.
Thanks and Regards
A.SathishKumar
>Implement your own custom
>`org.apache.kafka.clients.consumer.internals.PartitionAssignor`
>and assign all the subscribed partitions to the first consumer instance in
>the group.
>See 'partition.assignment.strategy' config in the consumer configs [1]
>[1]: http://kafka.apache.org/documentation.html#newconsumerconfigs
>On Thu, Aug 4, 2016 at 8:54 AM, sat <sa...@gmail.com> wrote:
> Hi,
>
> We have Kafka server/broker running in a seperate machine (say machine A),
> for now we are planning to have in one node. We have multiple topics and
> all topics have only 1 partition for now.
>
> We have our application which includes Kafka consumers installed in machine
> B and machine C. Our application in machine B and C are in cluster, hence
> Kafka Consumers will also be in cluster. Both our consumers will have same
> group id. We want all the messages to be consumed by consumer in machine B
> and only when machine B is down consumer in machine C should pull messages.
>
> Since consumer in machine B and C have same group id, we came to know
> consumer in machine B will get message for some time duration (10mins) and
> then consumer in machine C will get message for some time duration. Since
> our consumers are in cluster, we want only consumer to be active or receive
> all the messages as long as it is alive.
>
> Please let us know how to achieve this.
>
>
> Thanks and Regards
> A.SathishKumar
> 044-24735023
>
On Wed, Aug 3, 2016 at 8:24 PM, sat <sa...@gmail.com> wrote:
> Hi,
>
> We have Kafka server/broker running in a seperate machine (say machine A),
> for now we are planning to have in one node. We have multiple topics and
> all topics have only 1 partition for now.
>
> We have our application which includes Kafka consumers installed in
> machine B and machine C. Our application in machine B and C are in cluster,
> hence Kafka Consumers will also be in cluster. Both our consumers will have
> same group id. We want all the messages to be consumed by consumer in
> machine B and only when machine B is down consumer in machine C should pull
> messages.
>
> Since consumer in machine B and C have same group id, we came to know
> consumer in machine B will get message for some time duration (10mins) and
> then consumer in machine C will get message for some time duration. Since
> our consumers are in cluster, we want only consumer to be active or receive
> all the messages as long as it is alive.
>
> Please let us know how to achieve this.
>
>
> Thanks and Regards
> A.SathishKumar
> 044-24735023
>
--
A.SathishKumar
044-24735023