Consumer client not able to receive messages when one of broker is pushed down in the cluster

[rAhul <c....@gmail.com>]

Hi,

I have a Apache kafka cluster with 3 nodes(say 1,2,3) with replication
factor of 3 and partitions as 3.

When my producer client, consumer client and the cluster are running, able
to transfer messages from producer to consumer without any issues.

Now I stopped leader node say node 1 from the cluster and now say node 2 is
promoted as leader.

Message flow from producer to consumer works fine without any issues.

Now I started node 1 and stopped node 2, either node 1 or node 3 is
promoted as leader.

Now producer able to send messages but consumer not able to receive
messages.

I see consumer lag using kafka manager web console.

Again if I start node 2, consumer able to receive messages.

Please suggest how to overcome this issue and fix it.

Thanks.

Re: Consumer client not able to receive messages when one of broker is pushed down in the cluster

[rAhul <c....@gmail.com>]

Please suggest on how to avoid restart of Kafka Consumer Client on one of
the broker fail down in the cluster.
Or do we need to update the offset manually if we put back the failed
broker in the cluster.

On 5 January 2018 at 19:18, rAhul <c....@gmail.com> wrote:

> Hi,
>
> I have a Apache kafka cluster with 3 nodes(say 1,2,3) with replication
> factor of 3 and partitions as 3.
>
> When my producer client, consumer client and the cluster are running, able
> to transfer messages from producer to consumer without any issues.
>
> Now I stopped leader node say node 1 from the cluster and now say node 2
> is promoted as leader.
>
> Message flow from producer to consumer works fine without any issues.
>
> Now I started node 1 and stopped node 2, either node 1 or node 3 is
> promoted as leader.
>
> Now producer able to send messages but consumer not able to receive
> messages.
>
> I see consumer lag using kafka manager web console.
>
> Again if I start node 2, consumer able to receive messages.
>
> Please suggest how to overcome this issue and fix it.
>
> Thanks.
>

Re: Consumer client not able to receive messages when one of broker is pushed down in the cluster

[rAhul <c....@gmail.com>]

Please suggest on how to avoid restart of Kafka Consumer Client on one of
the broker fail down in the cluster.
Or do we need to update the offset manually if we put back the failed
broker in the cluster.

On 5 January 2018 at 19:18, rAhul <c....@gmail.com> wrote:

> Hi,
>
> I have a Apache kafka cluster with 3 nodes(say 1,2,3) with replication
> factor of 3 and partitions as 3.
>
> When my producer client, consumer client and the cluster are running, able
> to transfer messages from producer to consumer without any issues.
>
> Now I stopped leader node say node 1 from the cluster and now say node 2
> is promoted as leader.
>
> Message flow from producer to consumer works fine without any issues.
>
> Now I started node 1 and stopped node 2, either node 1 or node 3 is
> promoted as leader.
>
> Now producer able to send messages but consumer not able to receive
> messages.
>
> I see consumer lag using kafka manager web console.
>
> Again if I start node 2, consumer able to receive messages.
>
> Please suggest how to overcome this issue and fix it.
>
> Thanks.
>

Re: Consumer client not able to receive messages when one of broker is pushed down in the cluster

[Hans Jespersen <ha...@confluent.io>]

Check that your __consumer_offsets topic is also setup with replication factor of 3 and has In Sync Replicas. Often it gets setup first as a one node cluster with RF=1 and then when the cluster is expanded to 3 nodes the step to increase the replication factor of this topic gets missed.

-hans

> On Jan 5, 2018, at 8:48 AM, rAhul <c....@gmail.com> wrote:
> 
> Hi,
> 
> I have a Apache kafka cluster with 3 nodes(say 1,2,3) with replication
> factor of 3 and partitions as 3.
> 
> When my producer client, consumer client and the cluster are running, able
> to transfer messages from producer to consumer without any issues.
> 
> Now I stopped leader node say node 1 from the cluster and now say node 2 is
> promoted as leader.
> 
> Message flow from producer to consumer works fine without any issues.
> 
> Now I started node 1 and stopped node 2, either node 1 or node 3 is
> promoted as leader.
> 
> Now producer able to send messages but consumer not able to receive
> messages.
> 
> I see consumer lag using kafka manager web console.
> 
> Again if I start node 2, consumer able to receive messages.
> 
> Please suggest how to overcome this issue and fix it.
> 
> Thanks.