You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Zoran <zo...@bulbtech.com> on 2018/02/02 09:06:51 UTC

Re: Group consumer cannot consume messages if kafka service on specific node in test cluster is down

I just want to say that I have solved the situation by deleting 
zookeeper's and kafka's data directories and setting 
offsets.topic.replication.factor=3 in kafka server.properties file.

After that, __consumer_offsets topic is replicated and everything works 
as expected.

I hope this will help to someone.


Regards.


On 01/30/2018 03:02 PM, Zoran wrote:
> Sorry, I have attached wrong server.properties file. Now the right one 
> is in the attachment.
>
> Regards.
>
>
> On 01/30/2018 02:59 PM, Zoran wrote:
>> Hi,
>>
>> I have three servers:
>>
>> blade1 (192.168.112.31),
>> blade2 (192.168.112.32) and
>> blade3 (192.168.112.33).
>>
>> On each of servers kafka_2.11-1.0.0 is installed.
>> On blade3 (192.168.112.33:2181) zookeeper is installed as well.
>>
>> I have created a topic repl3part5 with the following line:
>>
>> bin/kafka-topics.sh --zookeeper 192.168.112.33:2181 --create 
>> --replication-factor 3 --partitions 5 --topic repl3part5
>>
>> When I describe the topic, it looks like this:
>>
>> [root@blade1 kafka]# bin/kafka-topics.sh --describe --topic 
>> repl3part5 --zookeeper 192.168.112.33:2181
>>
>> Topic:repl3part5    PartitionCount:5    ReplicationFactor:3 Configs:
>>     Topic: repl3part5    Partition: 0    Leader: 2    Replicas: 
>> 2,3,1    Isr: 2,3,1
>>     Topic: repl3part5    Partition: 1    Leader: 3    Replicas: 
>> 3,1,2    Isr: 3,1,2
>>     Topic: repl3part5    Partition: 2    Leader: 1    Replicas: 
>> 1,2,3    Isr: 1,2,3
>>     Topic: repl3part5    Partition: 3    Leader: 2    Replicas: 
>> 2,1,3    Isr: 2,1,3
>>     Topic: repl3part5    Partition: 4    Leader: 3    Replicas: 
>> 3,2,1    Isr: 3,2,1
>>
>> I have a producer for this topic:
>>
>> bin/kafka-console-producer.sh --broker-list 
>> 192.168.112.31:9092,192.168.112.32:9092,192.168.112.33:9092 --topic 
>> repl3part5
>>
>> and single consumer:
>>
>> bin/kafka-console-consumer.sh --bootstrap-server 
>> 192.168.112.31:9092,192.168.112.32:9092,192.168.112.33:9092 --topic 
>> repl3part5  --consumer-property group.id=zoran_1
>>
>> Every message that is sent by producer gets collected by consumer. So 
>> far - so good.
>>
>> Now I would like to test fail over of the kafka servers. If I put 
>> down blade 3 kafka service, I get consumer warnings but all produced 
>> messages are still consumed.
>>
>> [2018-01-30 14:30:01,203] WARN [Consumer clientId=consumer-1, 
>> groupId=zoran_1] Connection to node 3 could not be established. 
>> Broker may not be available. (org.apache.kafka.clients.NetworkClient)
>> [2018-01-30 14:30:01,299] WARN [Consumer clientId=consumer-1, 
>> groupId=zoran_1] Connection to node 3 could not be established. 
>> Broker may not be available. (org.apache.kafka.clients.NetworkClient)
>> [2018-01-30 14:30:01,475] WARN [Consumer clientId=consumer-1, 
>> groupId=zoran_1] Connection to node 3 could not be established. 
>> Broker may not be available. (org.apache.kafka.clients.NetworkClient)
>>
>> Now I have started up kafka service on blade 3 and I have put down 
>> kafka service on blade 2 server.
>> Consumer now showed one warning but all produced messages are still 
>> consumed.
>>
>> [2018-01-30 14:31:38,164] WARN [Consumer clientId=consumer-1, 
>> groupId=zoran_1] Connection to node 2 could not be established. 
>> Broker may not be available. (org.apache.kafka.clients.NetworkClient)
>>
>> Now I have started up kafka service on blade 2 and I have put down 
>> kafka service on blade 1 server.
>>
>> Consumer now shows warnings about node 1/2147483646, but also 
>> Asynchronous auto-commit of offsets ... failed: Offset commit failed 
>> with a retriable exception. You should retry committing offsets. The 
>> underlying error was: null.
>>
>> [2018-01-30 14:33:16,393] WARN [Consumer clientId=consumer-1, 
>> groupId=zoran_1] Connection to node 1 could not be established. 
>> Broker may not be available. (org.apache.kafka.clients.NetworkClient)
>> [2018-01-30 14:33:16,469] WARN [Consumer clientId=consumer-1, 
>> groupId=zoran_1] Connection to node 2147483646 could not be 
>> established. Broker may not be available. 
>> (org.apache.kafka.clients.NetworkClient)
>> [2018-01-30 14:33:16,557] WARN [Consumer clientId=consumer-1, 
>> groupId=zoran_1] Connection to node 1 could not be established. 
>> Broker may not be available. (org.apache.kafka.clients.NetworkClient)
>> [2018-01-30 14:33:16,986] WARN [Consumer clientId=consumer-1, 
>> groupId=zoran_1] Connection to node 2147483646 could not be 
>> established. Broker may not be available. 
>> (org.apache.kafka.clients.NetworkClient)
>> [2018-01-30 14:33:16,991] WARN [Consumer clientId=consumer-1, 
>> groupId=zoran_1] Connection to node 1 could not be established. 
>> Broker may not be available. (org.apache.kafka.clients.NetworkClient)
>> [2018-01-30 14:33:17,493] WARN [Consumer clientId=consumer-1, 
>> groupId=zoran_1] Connection to node 2147483646 could not be 
>> established. Broker may not be available. 
>> (org.apache.kafka.clients.NetworkClient)
>> [2018-01-30 14:33:17,495] WARN [Consumer clientId=consumer-1, 
>> groupId=zoran_1] Connection to node 1 could not be established. 
>> Broker may not be available. (org.apache.kafka.clients.NetworkClient)
>> [2018-01-30 14:33:18,002] WARN [Consumer clientId=consumer-1, 
>> groupId=zoran_1] Connection to node 2147483646 could not be 
>> established. Broker may not be available. 
>> (org.apache.kafka.clients.NetworkClient)
>> [2018-01-30 14:33:18,003] WARN [Consumer clientId=consumer-1, 
>> groupId=zoran_1] Asynchronous auto-commit of offsets 
>> {repl3part5-4=OffsetAndMetadata{offset=18, metadata=''}, 
>> repl3part5-3=OffsetAndMetadata{offset=20, metadata=''}, 
>> repl3part5-2=OffsetAndMetadata{offset=19, metadata=''}, 
>> repl3part5-1=OffsetAndMetadata{offset=20, metadata=''}, 
>> repl3part5-0=OffsetAndMetadata{offset=20, metadata=''}} failed: 
>> Offset commit failed with a retriable exception. You should retry 
>> committing offsets. The underlying error was: null 
>> (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
>> [2018-01-30 14:33:18,611] WARN [Consumer clientId=consumer-1, 
>> groupId=zoran_1] Connection to node 1 could not be established. 
>> Broker may not be available. (org.apache.kafka.clients.NetworkClient)
>> [2018-01-30 14:33:18,932] WARN [Consumer clientId=consumer-1, 
>> groupId=zoran_1] Connection to node 2147483646 could not be 
>> established. Broker may not be available. 
>> (org.apache.kafka.clients.NetworkClient)
>> [2018-01-30 14:33:18,933] WARN [Consumer clientId=consumer-1, 
>> groupId=zoran_1] Asynchronous auto-commit of offsets 
>> {repl3part5-4=OffsetAndMetadata{offset=18, metadata=''}, 
>> repl3part5-3=OffsetAndMetadata{offset=20, metadata=''}, 
>> repl3part5-2=OffsetAndMetadata{offset=19, metadata=''}, 
>> repl3part5-1=OffsetAndMetadata{offset=20, metadata=''}, 
>> repl3part5-0=OffsetAndMetadata{offset=20, metadata=''}} failed: 
>> Offset commit failed with a retriable exception. You should retry 
>> committing offsets. The underlying error was: null 
>> (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
>> [2018-01-30 14:33:19,977] WARN [Consumer clientId=consumer-1, 
>> groupId=zoran_1] Connection to node 2147483646 could not be 
>> established. Broker may not be available. 
>> (org.apache.kafka.clients.NetworkClient)
>> [2018-01-30 14:33:19,978] WARN [Consumer clientId=consumer-1, 
>> groupId=zoran_1] Asynchronous auto-commit of offsets 
>> {repl3part5-4=OffsetAndMetadata{offset=18, metadata=''}, 
>> repl3part5-3=OffsetAndMetadata{offset=20, metadata=''}, 
>> repl3part5-2=OffsetAndMetadata{offset=19, metadata=''}, 
>> repl3part5-1=OffsetAndMetadata{offset=20, metadata=''}, 
>> repl3part5-0=OffsetAndMetadata{offset=20, metadata=''}} failed: 
>> Offset commit failed with a retriable exception. You should retry 
>> committing offsets. The underlying error was: null 
>> (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
>> [2018-01-30 14:33:19,979] WARN [Consumer clientId=consumer-1, 
>> groupId=zoran_1] Connection to node 1 could not be established. 
>> Broker may not be available. (org.apache.kafka.clients.NetworkClient)
>>
>> I tried to solve the problem by adding a 
>> offsets.topic.replication.factor=2 (or 3) on all three 
>> server.properties file (one of them is attached), but with no success.
>> My idea was that topic __consumer_offset wasn't replicated throughout 
>> the cluster, but looks like it is not the case here.
>>
>> While blade 1 kafka service was down topic describe showed the 
>> following:
>>
>> [root@blade1 kafka]# bin/kafka-topics.sh --describe --topic 
>> repl3part5 --zookeeper 192.168.112.33:2181
>>
>> Topic:repl3part5    PartitionCount:5    ReplicationFactor:3 Configs:
>>     Topic: repl3part5    Partition: 0    Leader: 3    Replicas: 
>> 2,3,1    Isr: 3
>>     Topic: repl3part5    Partition: 1    Leader: 3    Replicas: 
>> 3,1,2    Isr: 3
>>     Topic: repl3part5    Partition: 2    Leader: 3    Replicas: 
>> 1,2,3    Isr: 3
>>     Topic: repl3part5    Partition: 3    Leader: 3    Replicas: 
>> 2,1,3    Isr: 3
>>     Topic: repl3part5    Partition: 4    Leader: 3    Replicas: 
>> 3,2,1    Isr: 3
>>
>> Producer now shows the following warning, it still puts messages on 
>> the topic but messages are just raising lag count on partitions:
>>
>> [2018-01-30 14:37:21,816] WARN [Producer clientId=console-producer] 
>> Connection to node 1 could not be established. Broker may not be 
>> available. (org.apache.kafka.clients.NetworkClient)
>>
>> I noticed that while kafka service on blade1 is alive, I can put 
>> down/up blade 2 and 3 in any combination and consumer will always be 
>> able to consume messages.
>> If kafka service on blade 1 is down, than even if kafka services on 
>> blade 2 and blade 3 are up and running, consumer cannot consume 
>> messages.
>>
>> After bringing kafka service up on blade 1, all messages that 
>> producer has sent while kafka service on blade 1 was down are 
>> replayed and than the following is showed in consumer terminal:
>>
>> [2018-01-30 14:44:30,817] ERROR [Consumer clientId=consumer-1, 
>> groupId=zoran_1] Offset commit failed on partition repl3part5-4 at 
>> offset 20: This is not the correct coordinator. 
>> (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
>> [2018-01-30 14:44:30,817] WARN [Consumer clientId=consumer-1, 
>> groupId=zoran_1] Asynchronous auto-commit of offsets 
>> {repl3part5-4=OffsetAndMetadata{offset=20, metadata=''}, 
>> repl3part5-3=OffsetAndMetadata{offset=22, metadata=''}, 
>> repl3part5-2=OffsetAndMetadata{offset=20, metadata=''}, 
>> repl3part5-1=OffsetAndMetadata{offset=22, metadata=''}, 
>> repl3part5-0=OffsetAndMetadata{offset=22, metadata=''}} failed: 
>> Offset commit failed with a retriable exception. You should retry 
>> committing offsets. The underlying error was: This is not the correct 
>> coordinator. 
>> (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
>> [2018-01-30 14:44:31,202] ERROR [Consumer clientId=consumer-1, 
>> groupId=zoran_1] Offset commit failed on partition repl3part5-4 at 
>> offset 22: This is not the correct coordinator. 
>> (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
>> [2018-01-30 14:44:31,202] WARN [Consumer clientId=consumer-1, 
>> groupId=zoran_1] Asynchronous auto-commit of offsets 
>> {repl3part5-4=OffsetAndMetadata{offset=22, metadata=''}, 
>> repl3part5-3=OffsetAndMetadata{offset=24, metadata=''}, 
>> repl3part5-2=OffsetAndMetadata{offset=22, metadata=''}, 
>> repl3part5-1=OffsetAndMetadata{offset=24, metadata=''}, 
>> repl3part5-0=OffsetAndMetadata{offset=24, metadata=''}} failed: 
>> Offset commit failed with a retriable exception. You should retry 
>> committing offsets. The underlying error was: This is not the correct 
>> coordinator. 
>> (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
>>
>> From now on everything works with no problems or warnings and the 
>> system is fully functional.
>>
>> Can someone explain to me why kafka server on blade 1 is so 
>> important, and what are my options in order to be able to stop any of 
>> the two servers (including kafka server on blade 1) and be able to 
>> consume messages with no delay?
>> This thing drives me crazy. :)
>>
>> Can you please help?
>>
>> Regards.
>


Re: Group consumer cannot consume messages if kafka service on specific node in test cluster is down

Posted by Navneet Kumar <na...@gmail.com>.
Appreciate you efforts

On 2 Feb 2018 2:37 p.m., "Zoran" <zo...@bulbtech.com> wrote:

> I just want to say that I have solved the situation by deleting
> zookeeper's and kafka's data directories and setting
> offsets.topic.replication.factor=3 in kafka server.properties file.
>
> After that, __consumer_offsets topic is replicated and everything works as
> expected.
>
> I hope this will help to someone.
>
>
> Regards.
>
>
> On 01/30/2018 03:02 PM, Zoran wrote:
>
>> Sorry, I have attached wrong server.properties file. Now the right one is
>> in the attachment.
>>
>> Regards.
>>
>>
>> On 01/30/2018 02:59 PM, Zoran wrote:
>>
>>> Hi,
>>>
>>> I have three servers:
>>>
>>> blade1 (192.168.112.31),
>>> blade2 (192.168.112.32) and
>>> blade3 (192.168.112.33).
>>>
>>> On each of servers kafka_2.11-1.0.0 is installed.
>>> On blade3 (192.168.112.33:2181) zookeeper is installed as well.
>>>
>>> I have created a topic repl3part5 with the following line:
>>>
>>> bin/kafka-topics.sh --zookeeper 192.168.112.33:2181 --create
>>> --replication-factor 3 --partitions 5 --topic repl3part5
>>>
>>> When I describe the topic, it looks like this:
>>>
>>> [root@blade1 kafka]# bin/kafka-topics.sh --describe --topic repl3part5
>>> --zookeeper 192.168.112.33:2181
>>>
>>> Topic:repl3part5    PartitionCount:5    ReplicationFactor:3 Configs:
>>>     Topic: repl3part5    Partition: 0    Leader: 2    Replicas: 2,3,1
>>> Isr: 2,3,1
>>>     Topic: repl3part5    Partition: 1    Leader: 3    Replicas: 3,1,2
>>> Isr: 3,1,2
>>>     Topic: repl3part5    Partition: 2    Leader: 1    Replicas: 1,2,3
>>> Isr: 1,2,3
>>>     Topic: repl3part5    Partition: 3    Leader: 2    Replicas: 2,1,3
>>> Isr: 2,1,3
>>>     Topic: repl3part5    Partition: 4    Leader: 3    Replicas: 3,2,1
>>> Isr: 3,2,1
>>>
>>> I have a producer for this topic:
>>>
>>> bin/kafka-console-producer.sh --broker-list 192.168.112.31:9092,
>>> 192.168.112.32:9092,192.168.112.33:9092 --topic repl3part5
>>>
>>> and single consumer:
>>>
>>> bin/kafka-console-consumer.sh --bootstrap-server 192.168.112.31:9092,
>>> 192.168.112.32:9092,192.168.112.33:9092 --topic repl3part5
>>> --consumer-property group.id=zoran_1
>>>
>>> Every message that is sent by producer gets collected by consumer. So
>>> far - so good.
>>>
>>> Now I would like to test fail over of the kafka servers. If I put down
>>> blade 3 kafka service, I get consumer warnings but all produced messages
>>> are still consumed.
>>>
>>> [2018-01-30 14:30:01,203] WARN [Consumer clientId=consumer-1,
>>> groupId=zoran_1] Connection to node 3 could not be established. Broker may
>>> not be available. (org.apache.kafka.clients.NetworkClient)
>>> [2018-01-30 14:30:01,299] WARN [Consumer clientId=consumer-1,
>>> groupId=zoran_1] Connection to node 3 could not be established. Broker may
>>> not be available. (org.apache.kafka.clients.NetworkClient)
>>> [2018-01-30 14:30:01,475] WARN [Consumer clientId=consumer-1,
>>> groupId=zoran_1] Connection to node 3 could not be established. Broker may
>>> not be available. (org.apache.kafka.clients.NetworkClient)
>>>
>>> Now I have started up kafka service on blade 3 and I have put down kafka
>>> service on blade 2 server.
>>> Consumer now showed one warning but all produced messages are still
>>> consumed.
>>>
>>> [2018-01-30 14:31:38,164] WARN [Consumer clientId=consumer-1,
>>> groupId=zoran_1] Connection to node 2 could not be established. Broker may
>>> not be available. (org.apache.kafka.clients.NetworkClient)
>>>
>>> Now I have started up kafka service on blade 2 and I have put down kafka
>>> service on blade 1 server.
>>>
>>> Consumer now shows warnings about node 1/2147483646, but also
>>> Asynchronous auto-commit of offsets ... failed: Offset commit failed with a
>>> retriable exception. You should retry committing offsets. The underlying
>>> error was: null.
>>>
>>> [2018-01-30 14:33:16,393] WARN [Consumer clientId=consumer-1,
>>> groupId=zoran_1] Connection to node 1 could not be established. Broker may
>>> not be available. (org.apache.kafka.clients.NetworkClient)
>>> [2018-01-30 14:33:16,469] WARN [Consumer clientId=consumer-1,
>>> groupId=zoran_1] Connection to node 2147483646 could not be established.
>>> Broker may not be available. (org.apache.kafka.clients.NetworkClient)
>>> [2018-01-30 14:33:16,557] WARN [Consumer clientId=consumer-1,
>>> groupId=zoran_1] Connection to node 1 could not be established. Broker may
>>> not be available. (org.apache.kafka.clients.NetworkClient)
>>> [2018-01-30 14:33:16,986] WARN [Consumer clientId=consumer-1,
>>> groupId=zoran_1] Connection to node 2147483646 could not be established.
>>> Broker may not be available. (org.apache.kafka.clients.NetworkClient)
>>> [2018-01-30 14:33:16,991] WARN [Consumer clientId=consumer-1,
>>> groupId=zoran_1] Connection to node 1 could not be established. Broker may
>>> not be available. (org.apache.kafka.clients.NetworkClient)
>>> [2018-01-30 14:33:17,493] WARN [Consumer clientId=consumer-1,
>>> groupId=zoran_1] Connection to node 2147483646 could not be established.
>>> Broker may not be available. (org.apache.kafka.clients.NetworkClient)
>>> [2018-01-30 14:33:17,495] WARN [Consumer clientId=consumer-1,
>>> groupId=zoran_1] Connection to node 1 could not be established. Broker may
>>> not be available. (org.apache.kafka.clients.NetworkClient)
>>> [2018-01-30 14:33:18,002] WARN [Consumer clientId=consumer-1,
>>> groupId=zoran_1] Connection to node 2147483646 could not be established.
>>> Broker may not be available. (org.apache.kafka.clients.NetworkClient)
>>> [2018-01-30 14:33:18,003] WARN [Consumer clientId=consumer-1,
>>> groupId=zoran_1] Asynchronous auto-commit of offsets
>>> {repl3part5-4=OffsetAndMetadata{offset=18, metadata=''},
>>> repl3part5-3=OffsetAndMetadata{offset=20, metadata=''},
>>> repl3part5-2=OffsetAndMetadata{offset=19, metadata=''},
>>> repl3part5-1=OffsetAndMetadata{offset=20, metadata=''},
>>> repl3part5-0=OffsetAndMetadata{offset=20, metadata=''}} failed: Offset
>>> commit failed with a retriable exception. You should retry committing
>>> offsets. The underlying error was: null (org.apache.kafka.clients.cons
>>> umer.internals.ConsumerCoordinator)
>>> [2018-01-30 14:33:18,611] WARN [Consumer clientId=consumer-1,
>>> groupId=zoran_1] Connection to node 1 could not be established. Broker may
>>> not be available. (org.apache.kafka.clients.NetworkClient)
>>> [2018-01-30 14:33:18,932] WARN [Consumer clientId=consumer-1,
>>> groupId=zoran_1] Connection to node 2147483646 could not be established.
>>> Broker may not be available. (org.apache.kafka.clients.NetworkClient)
>>> [2018-01-30 14:33:18,933] WARN [Consumer clientId=consumer-1,
>>> groupId=zoran_1] Asynchronous auto-commit of offsets
>>> {repl3part5-4=OffsetAndMetadata{offset=18, metadata=''},
>>> repl3part5-3=OffsetAndMetadata{offset=20, metadata=''},
>>> repl3part5-2=OffsetAndMetadata{offset=19, metadata=''},
>>> repl3part5-1=OffsetAndMetadata{offset=20, metadata=''},
>>> repl3part5-0=OffsetAndMetadata{offset=20, metadata=''}} failed: Offset
>>> commit failed with a retriable exception. You should retry committing
>>> offsets. The underlying error was: null (org.apache.kafka.clients.cons
>>> umer.internals.ConsumerCoordinator)
>>> [2018-01-30 14:33:19,977] WARN [Consumer clientId=consumer-1,
>>> groupId=zoran_1] Connection to node 2147483646 could not be established.
>>> Broker may not be available. (org.apache.kafka.clients.NetworkClient)
>>> [2018-01-30 14:33:19,978] WARN [Consumer clientId=consumer-1,
>>> groupId=zoran_1] Asynchronous auto-commit of offsets
>>> {repl3part5-4=OffsetAndMetadata{offset=18, metadata=''},
>>> repl3part5-3=OffsetAndMetadata{offset=20, metadata=''},
>>> repl3part5-2=OffsetAndMetadata{offset=19, metadata=''},
>>> repl3part5-1=OffsetAndMetadata{offset=20, metadata=''},
>>> repl3part5-0=OffsetAndMetadata{offset=20, metadata=''}} failed: Offset
>>> commit failed with a retriable exception. You should retry committing
>>> offsets. The underlying error was: null (org.apache.kafka.clients.cons
>>> umer.internals.ConsumerCoordinator)
>>> [2018-01-30 14:33:19,979] WARN [Consumer clientId=consumer-1,
>>> groupId=zoran_1] Connection to node 1 could not be established. Broker may
>>> not be available. (org.apache.kafka.clients.NetworkClient)
>>>
>>> I tried to solve the problem by adding a offsets.topic.replication.factor=2
>>> (or 3) on all three server.properties file (one of them is attached), but
>>> with no success.
>>> My idea was that topic __consumer_offset wasn't replicated throughout
>>> the cluster, but looks like it is not the case here.
>>>
>>> While blade 1 kafka service was down topic describe showed the following:
>>>
>>> [root@blade1 kafka]# bin/kafka-topics.sh --describe --topic repl3part5
>>> --zookeeper 192.168.112.33:2181
>>>
>>> Topic:repl3part5    PartitionCount:5    ReplicationFactor:3 Configs:
>>>     Topic: repl3part5    Partition: 0    Leader: 3    Replicas: 2,3,1
>>> Isr: 3
>>>     Topic: repl3part5    Partition: 1    Leader: 3    Replicas: 3,1,2
>>> Isr: 3
>>>     Topic: repl3part5    Partition: 2    Leader: 3    Replicas: 1,2,3
>>> Isr: 3
>>>     Topic: repl3part5    Partition: 3    Leader: 3    Replicas: 2,1,3
>>> Isr: 3
>>>     Topic: repl3part5    Partition: 4    Leader: 3    Replicas: 3,2,1
>>> Isr: 3
>>>
>>> Producer now shows the following warning, it still puts messages on the
>>> topic but messages are just raising lag count on partitions:
>>>
>>> [2018-01-30 14:37:21,816] WARN [Producer clientId=console-producer]
>>> Connection to node 1 could not be established. Broker may not be available.
>>> (org.apache.kafka.clients.NetworkClient)
>>>
>>> I noticed that while kafka service on blade1 is alive, I can put down/up
>>> blade 2 and 3 in any combination and consumer will always be able to
>>> consume messages.
>>> If kafka service on blade 1 is down, than even if kafka services on
>>> blade 2 and blade 3 are up and running, consumer cannot consume messages.
>>>
>>> After bringing kafka service up on blade 1, all messages that producer
>>> has sent while kafka service on blade 1 was down are replayed and than the
>>> following is showed in consumer terminal:
>>>
>>> [2018-01-30 14:44:30,817] ERROR [Consumer clientId=consumer-1,
>>> groupId=zoran_1] Offset commit failed on partition repl3part5-4 at offset
>>> 20: This is not the correct coordinator. (org.apache.kafka.clients.cons
>>> umer.internals.ConsumerCoordinator)
>>> [2018-01-30 14:44:30,817] WARN [Consumer clientId=consumer-1,
>>> groupId=zoran_1] Asynchronous auto-commit of offsets
>>> {repl3part5-4=OffsetAndMetadata{offset=20, metadata=''},
>>> repl3part5-3=OffsetAndMetadata{offset=22, metadata=''},
>>> repl3part5-2=OffsetAndMetadata{offset=20, metadata=''},
>>> repl3part5-1=OffsetAndMetadata{offset=22, metadata=''},
>>> repl3part5-0=OffsetAndMetadata{offset=22, metadata=''}} failed: Offset
>>> commit failed with a retriable exception. You should retry committing
>>> offsets. The underlying error was: This is not the correct coordinator.
>>> (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
>>> [2018-01-30 14:44:31,202] ERROR [Consumer clientId=consumer-1,
>>> groupId=zoran_1] Offset commit failed on partition repl3part5-4 at offset
>>> 22: This is not the correct coordinator. (org.apache.kafka.clients.cons
>>> umer.internals.ConsumerCoordinator)
>>> [2018-01-30 14:44:31,202] WARN [Consumer clientId=consumer-1,
>>> groupId=zoran_1] Asynchronous auto-commit of offsets
>>> {repl3part5-4=OffsetAndMetadata{offset=22, metadata=''},
>>> repl3part5-3=OffsetAndMetadata{offset=24, metadata=''},
>>> repl3part5-2=OffsetAndMetadata{offset=22, metadata=''},
>>> repl3part5-1=OffsetAndMetadata{offset=24, metadata=''},
>>> repl3part5-0=OffsetAndMetadata{offset=24, metadata=''}} failed: Offset
>>> commit failed with a retriable exception. You should retry committing
>>> offsets. The underlying error was: This is not the correct coordinator.
>>> (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
>>>
>>> From now on everything works with no problems or warnings and the system
>>> is fully functional.
>>>
>>> Can someone explain to me why kafka server on blade 1 is so important,
>>> and what are my options in order to be able to stop any of the two servers
>>> (including kafka server on blade 1) and be able to consume messages with no
>>> delay?
>>> This thing drives me crazy. :)
>>>
>>> Can you please help?
>>>
>>> Regards.
>>>
>>
>>
>