You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Chikulal C <Ch...@RCGGS.COM.INVALID> on 2020/02/13 10:40:10 UTC
Kafka clustering issue
Hi,
I am facing an issue with the Kafka clustering setup that I have. I have a Kafka cluster with two broker that are connected to two zookeepers. I am posting data to a topic that have replication factor and partition two each with a spring boot Kafka producer and consuming the same with another spring boot app.
I found one strange behavior when testing the cluster in the following manner -
1. Turned off node1 and node 2
2. Turned on node 1
3. Turned off node 1
4. Turned on node 2
After turning on node 2 Kafka cluster got failed and I am not able to produce data to Kafka. My consumer started throwing the message continuously as given below.
[Producer clientId=producer-1] Connection to node 1 (/server1-ip:9092) could not be established. Broker may not be available.
Issue is visible in both nodes. But if I kept both system up for a while issue will get resolved and I can turn off any of the node without breaking the cluster.
My broker configuration is as below.
broker.id=0
listeners=PLAINTEXT://server1-ip:9092
advertised.listeners=PLAINTEXT://serever1-ip:9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/home/user/kafka/data/kafka-logs
num.partitions=1
num.recovery.threads.per.data.dir=2
offsets.topic.replication.factor=2
transaction.state.log.replication.factor=2
transaction.state.log.min.isr=2
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=serever1-ip:2181,serever2-ip:2181
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=3000
auto.leader.rebalance.enable=true
leader.imbalance.check.interval.seconds=5
Zookeeper configuration
dataDir=/home/user/kafka/data
clientPort=2181
maxClientCnxns=0
initLimit=10
syncLimit=5
tickTime=2000
server.1=server1-ip:2888:3888
server.2=server2-ip:2888:3888
Is this is an expected behavior of Kafka or am I doing something wrong with this configuration ?
Can somebody help me with this issue ..
Thanks in advance.
Re: Kafka clustering issue
Posted by Chikulal C <Ch...@RCGGS.COM.INVALID>.
Hi,
Please find the inline message
________________________________
From: M. Manna <ma...@gmail.com>
Sent: 13 February 2020 19:18
To: Chikulal C <Ch...@RCGGS.COM>
Cc: Kafka Users <us...@kafka.apache.org>
Subject: Re: Kafka clustering issue
Hi,
On Thu, 13 Feb 2020 at 12:43, Chikulal C <Ch...@rcggs.com>> wrote:
1. Turned off node1 and node 2 (expected vs. actual)
* expected: Message publish failure with following warnings ( in producer )
* Connection to node 0 could not be established. Broker may not be available.
Connection to node 1 could not be established. Broker may not be available.
* actual: The same
2. Turned on node 1
* expected: No warnings and should publish data to topic
* actual: The same
At this stage, what can you see regarding your topic partitions - who is leader/follower?
after step2
Topic: topic2 PartitionCount: 2 ReplicationFactor: 2 Configs:
Topic: topic2 Partition: 0 Leader: none Replicas: 0,1 Isr: 1
Topic: topic2 Partition: 1 Leader: none Replicas: 1,0 Isr: 1
after step 4
Topic: topic2 PartitionCount: 2 ReplicationFactor: 2 Configs:
Topic: topic2 Partition: 0 Leader: 1 Replicas: 0,1 Isr: 1
Topic: topic2 Partition: 1 Leader: 1 Replicas: 1,0 Isr: 1
*
1. Turned off node 1
* expected: Message failure with following warnings
* Connection to node 0 could not be established. Broker may not be available.
Connection to node 1 could not be established. Broker may not be available.
* actual: Message failure with only one type of warning ( It should warn me that both nodes are down )
* Connection to node 0 could not be established. Broker may not be available.
Why do you think this is wrong? You have turned off the same node that you turned back on? Or, am I missing something here?
--Here both nodes are down. So my assumption is that it should say both are down like the output of step 1 (Turned of node1 and node 2)
*
1. Turned on node 2
* expected: No warnings and should publish data to topic
* actual: Message failure with only one type of warning
* Connection to node 0 could not be established. Broker may not be available.
See my previous comments. I think you may have either misconfigured one of your tests, or have not explained the steps correctly. See my highlighted text above
*
Here when you turn on node 2 in step 4, I would like to have my cluster up, since one of the broker is up. But it is not happening.
________________________________
From: M. Manna <ma...@gmail.com>>
Sent: 13 February 2020 17:55
To: Chikulal C <Ch...@RCGGS.COM>>
Cc: Kafka Users <us...@kafka.apache.org>>
Subject: Re: Kafka clustering issue
My apologies as I misread one of the steps you mentioned in your original email.
Could you kindly mention what you are seeing as per your order of failover tests?
1. Turned off node1 and node 2 (expected vs. actual)
2. Turned on node 1 (expected vs actual)
3. Turned off node 1 (expected vs actual)
4. Turned on node 2 (expected vs actual)
Thanks,
On Thu, 13 Feb 2020 at 12:06, Chikulal C <Ch...@rcggs.com>> wrote:
Hi,
I tried setting transaction.state.log.min.isr=1. But the issue still exists.
I am also getting one warning after doing step 3 (with transaction.ate.log.min.isr=1) and producing some data on the topic as given below.
[Producer clientId=producer-1] 2 partitions have leader brokers without a matching listener, including [topic2-0, topic2-1]
But I was not facing this issue when transaction.state.log.min.isr was 2. This warning also leads to failure from the producer side to put data on the topic.
Are there any other things I have to check?
Thanks
________________________________
From: M. Manna <ma...@gmail.com>>
Sent: 13 February 2020 16:35
To: Kafka Users <us...@kafka.apache.org>>
Subject: Re: Kafka clustering issue
This could be because you have set your transaction.ate.log.min.isr=2. Have
you tried with setting this to 1?
Also, please note that if your min.insync.replica=1, and you only have 2
nodes, you would only have a guarantee from 1 brokers to have the messages
- but if the same broker fails then you may see issues.
On Thu, 13 Feb 2020 at 10:40, Chikulal C <Ch...@rcggs.com.invalid>
wrote:
> Hi,
>
> I am facing an issue with the Kafka clustering setup that I have. I have a
> Kafka cluster with two broker that are connected to two zookeepers. I am
> posting data to a topic that have replication factor and partition two each
> with a spring boot Kafka producer and consuming the same with another
> spring boot app.
>
> I found one strange behavior when testing the cluster in the following
> manner -
>
> 1. Turned off node1 and node 2
> 2. Turned on node 1
> 3. Turned off node 1
> 4. Turned on node 2
>
> After turning on node 2 Kafka cluster got failed and I am not able to
> produce data to Kafka. My consumer started throwing the message
> continuously as given below.
>
> [Producer clientId=producer-1] Connection to node 1 (/server1-ip:9092)
> could not be established. Broker may not be available.
>
> Issue is visible in both nodes. But if I kept both system up for a while
> issue will get resolved and I can turn off any of the node without breaking
> the cluster.
> My broker configuration is as below.
>
> broker.id<http://broker.id>=0
> listeners=PLAINTEXT://server1-ip:9092
> advertised.listeners=PLAINTEXT://serever1-ip:9092
> num.network.threads=3
> num.io.threads=8
> socket.send.buffer.bytes=102400
> socket.receive.buffer.bytes=102400
> socket.request.max.bytes=104857600
> log.dirs=/home/user/kafka/data/kafka-logs
> num.partitions=1
> num.recovery.threads.per.data.dir=2
> offsets.topic.replication.factor=2
> transaction.state.log.replication.factor=2
> transaction.state.log.min.isr=2
> log.retention.hours=168
> log.segment.bytes=1073741824
> log.retention.check.interval.ms<http://log.retention.check.interval.ms>=300000
> zookeeper.connect=serever1-ip:2181,serever2-ip:2181
> zookeeper.connection.timeout.ms<http://zookeeper.connection.timeout.ms>=6000
> group.initial.rebalance.delay.ms<http://group.initial.rebalance.delay.ms>=3000
> auto.leader.rebalance.enable=true
> leader.imbalance.check.interval.seconds=5
>
> Zookeeper configuration
>
> dataDir=/home/user/kafka/data
> clientPort=2181
> maxClientCnxns=0
> initLimit=10
> syncLimit=5
> tickTime=2000
> server.1=server1-ip:2888:3888
> server.2=server2-ip:2888:3888
>
> Is this is an expected behavior of Kafka or am I doing something wrong
> with this configuration ?
>
> Can somebody help me with this issue ..
>
> Thanks in advance.
>
>
>
Re: Kafka clustering issue
Posted by "M. Manna" <ma...@gmail.com>.
Hi,
On Thu, 13 Feb 2020 at 12:43, Chikulal C <Ch...@rcggs.com> wrote:
>
> 1. Turned off node1 and node 2 (expected vs. actual)
> 1. expected: Message publish failure with following warnings ( in
> producer )
> 1. Connection to node 0 could not be established. Broker may not be
> available.
> Connection to node 1 could not be established. Broker may not be
> available.
> 2. actual: The same
> 2. Turned on node 1
> 1. expected: No warnings and should publish data to topic
> 2. actual: The same
>
> At this stage, what can you see regarding your topic partitions - who is
leader/follower?
>
> 1.
> 1. Turned off node 1
> 1. expected: Message failure with following warnings
> 1. Connection to node 0 could not be established. Broker may not
> be available.
> Connection to node 1 could not be established. Broker may not be
> available.
> 2. actual: Message failure with only one type of warning ( It
> should warn me that both nodes are down )
> 1. Connection to node 0 could not be established. Broker may not be
> available.
>
> Why do you think this is wrong? You have turned off the same node that
you turned back on? Or, am I missing something here?
>
> 1.
> 1. Turned on node 2
> 1. expected: No warnings and should publish data to topic
> 2. actual: Message failure with only one type of warning
> 1. Connection to node 0 could not be established. Broker may not
> be available.
>
>
> See my previous comments. I think you may have either misconfigured one
of your tests, or have not explained the steps correctly. See my
highlighted text above
>
> 1.
>
>
> Here when you turn on node 2 in step 4, I would like to have my cluster
> up, since one of the broker is up. But it is not happening.
> ------------------------------
> *From:* M. Manna <ma...@gmail.com>
> *Sent:* 13 February 2020 17:55
> *To:* Chikulal C <Ch...@RCGGS.COM>
> *Cc:* Kafka Users <us...@kafka.apache.org>
> *Subject:* Re: Kafka clustering issue
>
> My apologies as I misread one of the steps you mentioned in your original
> email.
>
> Could you kindly mention what you are seeing as per your order of failover
> tests?
>
> 1. Turned off node1 and node 2 (expected vs. actual)
> 2. Turned on node 1 (expected vs actual)
> 3. Turned off node 1 (expected vs actual)
> 4. Turned on node 2 (expected vs actual)
>
>
> Thanks,
>
> On Thu, 13 Feb 2020 at 12:06, Chikulal C <Ch...@rcggs.com> wrote:
>
> Hi,
>
>
> I tried setting transaction.state.log.min.isr=1. But the issue still
> exists.
>
>
> I am also getting one warning after doing step 3 (with
> transaction.ate.log.min.isr=1) and producing some data on the topic as
> given below.
>
>
> [Producer clientId=producer-1] 2 partitions have leader brokers without a
> matching listener, including [topic2-0, topic2-1]
>
>
> But I was not facing this issue when transaction.state.log.min.isr was 2.
> This warning also leads to failure from the producer side to put data on
> the topic.
>
>
> Are there any other things I have to check?
>
>
> Thanks
>
>
> ------------------------------
> *From:* M. Manna <ma...@gmail.com>
> *Sent:* 13 February 2020 16:35
> *To:* Kafka Users <us...@kafka.apache.org>
> *Subject:* Re: Kafka clustering issue
>
> This could be because you have set your transaction.ate.log.min.isr=2. Have
> you tried with setting this to 1?
>
> Also, please note that if your min.insync.replica=1, and you only have 2
> nodes, you would only have a guarantee from 1 brokers to have the messages
> - but if the same broker fails then you may see issues.
>
> On Thu, 13 Feb 2020 at 10:40, Chikulal C <Ch...@rcggs.com.invalid>
> wrote:
>
> > Hi,
> >
> > I am facing an issue with the Kafka clustering setup that I have. I have
> a
> > Kafka cluster with two broker that are connected to two zookeepers. I am
> > posting data to a topic that have replication factor and partition two
> each
> > with a spring boot Kafka producer and consuming the same with another
> > spring boot app.
> >
> > I found one strange behavior when testing the cluster in the following
> > manner -
> >
> > 1. Turned off node1 and node 2
> > 2. Turned on node 1
> > 3. Turned off node 1
> > 4. Turned on node 2
> >
> > After turning on node 2 Kafka cluster got failed and I am not able to
> > produce data to Kafka. My consumer started throwing the message
> > continuously as given below.
> >
> > [Producer clientId=producer-1] Connection to node 1 (/server1-ip:9092)
> > could not be established. Broker may not be available.
> >
> > Issue is visible in both nodes. But if I kept both system up for a while
> > issue will get resolved and I can turn off any of the node without
> breaking
> > the cluster.
> > My broker configuration is as below.
> >
> > broker.id=0
> > listeners=PLAINTEXT://server1-ip:9092
> > advertised.listeners=PLAINTEXT://serever1-ip:9092
> > num.network.threads=3
> > num.io.threads=8
> > socket.send.buffer.bytes=102400
> > socket.receive.buffer.bytes=102400
> > socket.request.max.bytes=104857600
> > log.dirs=/home/user/kafka/data/kafka-logs
> > num.partitions=1
> > num.recovery.threads.per.data.dir=2
> > offsets.topic.replication.factor=2
> > transaction.state.log.replication.factor=2
> > transaction.state.log.min.isr=2
> > log.retention.hours=168
> > log.segment.bytes=1073741824
> > log.retention.check.interval.ms=300000
> > zookeeper.connect=serever1-ip:2181,serever2-ip:2181
> > zookeeper.connection.timeout.ms=6000
> > group.initial.rebalance.delay.ms=3000
> > auto.leader.rebalance.enable=true
> > leader.imbalance.check.interval.seconds=5
> >
> > Zookeeper configuration
> >
> > dataDir=/home/user/kafka/data
> > clientPort=2181
> > maxClientCnxns=0
> > initLimit=10
> > syncLimit=5
> > tickTime=2000
> > server.1=server1-ip:2888:3888
> > server.2=server2-ip:2888:3888
> >
> > Is this is an expected behavior of Kafka or am I doing something wrong
> > with this configuration ?
> >
> > Can somebody help me with this issue ..
> >
> > Thanks in advance.
> >
> >
> >
>
>
Re: Kafka clustering issue
Posted by Chikulal C <Ch...@RCGGS.COM.INVALID>.
1. Turned off node1 and node 2 (expected vs. actual)
* expected: Message publish failure with following warnings ( in producer )
* Connection to node 0 could not be established. Broker may not be available.
Connection to node 1 could not be established. Broker may not be available.
* actual: The same
2. Turned on node 1
* expected: No warnings and should publish data to topic
* actual: The same
3. Turned off node 1
* expected: Message failure with following warnings
* Connection to node 0 could not be established. Broker may not be available.
Connection to node 1 could not be established. Broker may not be available.
* actual: Message failure with only one type of warning ( It should warn me that both nodes are down )
* Connection to node 0 could not be established. Broker may not be available.
4. Turned on node 2
* expected: No warnings and should publish data to topic
* actual: Message failure with only one type of warning
* Connection to node 0 could not be established. Broker may not be available.
Here when you turn on node 2 in step 4, I would like to have my cluster up, since one of the broker is up. But it is not happening.
________________________________
From: M. Manna <ma...@gmail.com>
Sent: 13 February 2020 17:55
To: Chikulal C <Ch...@RCGGS.COM>
Cc: Kafka Users <us...@kafka.apache.org>
Subject: Re: Kafka clustering issue
My apologies as I misread one of the steps you mentioned in your original email.
Could you kindly mention what you are seeing as per your order of failover tests?
1. Turned off node1 and node 2 (expected vs. actual)
2. Turned on node 1 (expected vs actual)
3. Turned off node 1 (expected vs actual)
4. Turned on node 2 (expected vs actual)
Thanks,
On Thu, 13 Feb 2020 at 12:06, Chikulal C <Ch...@rcggs.com>> wrote:
Hi,
I tried setting transaction.state.log.min.isr=1. But the issue still exists.
I am also getting one warning after doing step 3 (with transaction.ate.log.min.isr=1) and producing some data on the topic as given below.
[Producer clientId=producer-1] 2 partitions have leader brokers without a matching listener, including [topic2-0, topic2-1]
But I was not facing this issue when transaction.state.log.min.isr was 2. This warning also leads to failure from the producer side to put data on the topic.
Are there any other things I have to check?
Thanks
________________________________
From: M. Manna <ma...@gmail.com>>
Sent: 13 February 2020 16:35
To: Kafka Users <us...@kafka.apache.org>>
Subject: Re: Kafka clustering issue
This could be because you have set your transaction.ate.log.min.isr=2. Have
you tried with setting this to 1?
Also, please note that if your min.insync.replica=1, and you only have 2
nodes, you would only have a guarantee from 1 brokers to have the messages
- but if the same broker fails then you may see issues.
On Thu, 13 Feb 2020 at 10:40, Chikulal C <Ch...@rcggs.com.invalid>
wrote:
> Hi,
>
> I am facing an issue with the Kafka clustering setup that I have. I have a
> Kafka cluster with two broker that are connected to two zookeepers. I am
> posting data to a topic that have replication factor and partition two each
> with a spring boot Kafka producer and consuming the same with another
> spring boot app.
>
> I found one strange behavior when testing the cluster in the following
> manner -
>
> 1. Turned off node1 and node 2
> 2. Turned on node 1
> 3. Turned off node 1
> 4. Turned on node 2
>
> After turning on node 2 Kafka cluster got failed and I am not able to
> produce data to Kafka. My consumer started throwing the message
> continuously as given below.
>
> [Producer clientId=producer-1] Connection to node 1 (/server1-ip:9092)
> could not be established. Broker may not be available.
>
> Issue is visible in both nodes. But if I kept both system up for a while
> issue will get resolved and I can turn off any of the node without breaking
> the cluster.
> My broker configuration is as below.
>
> broker.id<http://broker.id>=0
> listeners=PLAINTEXT://server1-ip:9092
> advertised.listeners=PLAINTEXT://serever1-ip:9092
> num.network.threads=3
> num.io.threads=8
> socket.send.buffer.bytes=102400
> socket.receive.buffer.bytes=102400
> socket.request.max.bytes=104857600
> log.dirs=/home/user/kafka/data/kafka-logs
> num.partitions=1
> num.recovery.threads.per.data.dir=2
> offsets.topic.replication.factor=2
> transaction.state.log.replication.factor=2
> transaction.state.log.min.isr=2
> log.retention.hours=168
> log.segment.bytes=1073741824
> log.retention.check.interval.ms<http://log.retention.check.interval.ms>=300000
> zookeeper.connect=serever1-ip:2181,serever2-ip:2181
> zookeeper.connection.timeout.ms<http://zookeeper.connection.timeout.ms>=6000
> group.initial.rebalance.delay.ms<http://group.initial.rebalance.delay.ms>=3000
> auto.leader.rebalance.enable=true
> leader.imbalance.check.interval.seconds=5
>
> Zookeeper configuration
>
> dataDir=/home/user/kafka/data
> clientPort=2181
> maxClientCnxns=0
> initLimit=10
> syncLimit=5
> tickTime=2000
> server.1=server1-ip:2888:3888
> server.2=server2-ip:2888:3888
>
> Is this is an expected behavior of Kafka or am I doing something wrong
> with this configuration ?
>
> Can somebody help me with this issue ..
>
> Thanks in advance.
>
>
>
Re: Kafka clustering issue
Posted by "M. Manna" <ma...@gmail.com>.
My apologies as I misread one of the steps you mentioned in your original
email.
Could you kindly mention what you are seeing as per your order of failover
tests?
1. Turned off node1 and node 2 (expected vs. actual)
2. Turned on node 1 (expected vs actual)
3. Turned off node 1 (expected vs actual)
4. Turned on node 2 (expected vs actual)
Thanks,
On Thu, 13 Feb 2020 at 12:06, Chikulal C <Ch...@rcggs.com> wrote:
> Hi,
>
>
> I tried setting transaction.state.log.min.isr=1. But the issue still
> exists.
>
>
> I am also getting one warning after doing step 3 (with
> transaction.ate.log.min.isr=1) and producing some data on the topic as
> given below.
>
>
> [Producer clientId=producer-1] 2 partitions have leader brokers without a
> matching listener, including [topic2-0, topic2-1]
>
>
> But I was not facing this issue when transaction.state.log.min.isr was 2.
> This warning also leads to failure from the producer side to put data on
> the topic.
>
>
> Are there any other things I have to check?
>
>
> Thanks
>
>
> ------------------------------
> *From:* M. Manna <ma...@gmail.com>
> *Sent:* 13 February 2020 16:35
> *To:* Kafka Users <us...@kafka.apache.org>
> *Subject:* Re: Kafka clustering issue
>
> This could be because you have set your transaction.ate.log.min.isr=2. Have
> you tried with setting this to 1?
>
> Also, please note that if your min.insync.replica=1, and you only have 2
> nodes, you would only have a guarantee from 1 brokers to have the messages
> - but if the same broker fails then you may see issues.
>
> On Thu, 13 Feb 2020 at 10:40, Chikulal C <Ch...@rcggs.com.invalid>
> wrote:
>
> > Hi,
> >
> > I am facing an issue with the Kafka clustering setup that I have. I have
> a
> > Kafka cluster with two broker that are connected to two zookeepers. I am
> > posting data to a topic that have replication factor and partition two
> each
> > with a spring boot Kafka producer and consuming the same with another
> > spring boot app.
> >
> > I found one strange behavior when testing the cluster in the following
> > manner -
> >
> > 1. Turned off node1 and node 2
> > 2. Turned on node 1
> > 3. Turned off node 1
> > 4. Turned on node 2
> >
> > After turning on node 2 Kafka cluster got failed and I am not able to
> > produce data to Kafka. My consumer started throwing the message
> > continuously as given below.
> >
> > [Producer clientId=producer-1] Connection to node 1 (/server1-ip:9092)
> > could not be established. Broker may not be available.
> >
> > Issue is visible in both nodes. But if I kept both system up for a while
> > issue will get resolved and I can turn off any of the node without
> breaking
> > the cluster.
> > My broker configuration is as below.
> >
> > broker.id=0
> > listeners=PLAINTEXT://server1-ip:9092
> > advertised.listeners=PLAINTEXT://serever1-ip:9092
> > num.network.threads=3
> > num.io.threads=8
> > socket.send.buffer.bytes=102400
> > socket.receive.buffer.bytes=102400
> > socket.request.max.bytes=104857600
> > log.dirs=/home/user/kafka/data/kafka-logs
> > num.partitions=1
> > num.recovery.threads.per.data.dir=2
> > offsets.topic.replication.factor=2
> > transaction.state.log.replication.factor=2
> > transaction.state.log.min.isr=2
> > log.retention.hours=168
> > log.segment.bytes=1073741824
> > log.retention.check.interval.ms=300000
> > zookeeper.connect=serever1-ip:2181,serever2-ip:2181
> > zookeeper.connection.timeout.ms=6000
> > group.initial.rebalance.delay.ms=3000
> > auto.leader.rebalance.enable=true
> > leader.imbalance.check.interval.seconds=5
> >
> > Zookeeper configuration
> >
> > dataDir=/home/user/kafka/data
> > clientPort=2181
> > maxClientCnxns=0
> > initLimit=10
> > syncLimit=5
> > tickTime=2000
> > server.1=server1-ip:2888:3888
> > server.2=server2-ip:2888:3888
> >
> > Is this is an expected behavior of Kafka or am I doing something wrong
> > with this configuration ?
> >
> > Can somebody help me with this issue ..
> >
> > Thanks in advance.
> >
> >
> >
>
Re: Kafka clustering issue
Posted by Chikulal C <Ch...@RCGGS.COM.INVALID>.
Hi,
I tried setting transaction.state.log.min.isr=1. But the issue still exists.
I am also getting one warning after doing step 3 (with transaction.ate.log.min.isr=1) and producing some data on the topic as given below.
[Producer clientId=producer-1] 2 partitions have leader brokers without a matching listener, including [topic2-0, topic2-1]
But I was not facing this issue when transaction.state.log.min.isr was 2. This warning also leads to failure from the producer side to put data on the topic.
Are there any other things I have to check?
Thanks
________________________________
From: M. Manna <ma...@gmail.com>
Sent: 13 February 2020 16:35
To: Kafka Users <us...@kafka.apache.org>
Subject: Re: Kafka clustering issue
This could be because you have set your transaction.ate.log.min.isr=2. Have
you tried with setting this to 1?
Also, please note that if your min.insync.replica=1, and you only have 2
nodes, you would only have a guarantee from 1 brokers to have the messages
- but if the same broker fails then you may see issues.
On Thu, 13 Feb 2020 at 10:40, Chikulal C <Ch...@rcggs.com.invalid>
wrote:
> Hi,
>
> I am facing an issue with the Kafka clustering setup that I have. I have a
> Kafka cluster with two broker that are connected to two zookeepers. I am
> posting data to a topic that have replication factor and partition two each
> with a spring boot Kafka producer and consuming the same with another
> spring boot app.
>
> I found one strange behavior when testing the cluster in the following
> manner -
>
> 1. Turned off node1 and node 2
> 2. Turned on node 1
> 3. Turned off node 1
> 4. Turned on node 2
>
> After turning on node 2 Kafka cluster got failed and I am not able to
> produce data to Kafka. My consumer started throwing the message
> continuously as given below.
>
> [Producer clientId=producer-1] Connection to node 1 (/server1-ip:9092)
> could not be established. Broker may not be available.
>
> Issue is visible in both nodes. But if I kept both system up for a while
> issue will get resolved and I can turn off any of the node without breaking
> the cluster.
> My broker configuration is as below.
>
> broker.id=0
> listeners=PLAINTEXT://server1-ip:9092
> advertised.listeners=PLAINTEXT://serever1-ip:9092
> num.network.threads=3
> num.io.threads=8
> socket.send.buffer.bytes=102400
> socket.receive.buffer.bytes=102400
> socket.request.max.bytes=104857600
> log.dirs=/home/user/kafka/data/kafka-logs
> num.partitions=1
> num.recovery.threads.per.data.dir=2
> offsets.topic.replication.factor=2
> transaction.state.log.replication.factor=2
> transaction.state.log.min.isr=2
> log.retention.hours=168
> log.segment.bytes=1073741824
> log.retention.check.interval.ms=300000
> zookeeper.connect=serever1-ip:2181,serever2-ip:2181
> zookeeper.connection.timeout.ms=6000
> group.initial.rebalance.delay.ms=3000
> auto.leader.rebalance.enable=true
> leader.imbalance.check.interval.seconds=5
>
> Zookeeper configuration
>
> dataDir=/home/user/kafka/data
> clientPort=2181
> maxClientCnxns=0
> initLimit=10
> syncLimit=5
> tickTime=2000
> server.1=server1-ip:2888:3888
> server.2=server2-ip:2888:3888
>
> Is this is an expected behavior of Kafka or am I doing something wrong
> with this configuration ?
>
> Can somebody help me with this issue ..
>
> Thanks in advance.
>
>
>
Re: Kafka clustering issue
Posted by "M. Manna" <ma...@gmail.com>.
This could be because you have set your transaction.ate.log.min.isr=2. Have
you tried with setting this to 1?
Also, please note that if your min.insync.replica=1, and you only have 2
nodes, you would only have a guarantee from 1 brokers to have the messages
- but if the same broker fails then you may see issues.
On Thu, 13 Feb 2020 at 10:40, Chikulal C <Ch...@rcggs.com.invalid>
wrote:
> Hi,
>
> I am facing an issue with the Kafka clustering setup that I have. I have a
> Kafka cluster with two broker that are connected to two zookeepers. I am
> posting data to a topic that have replication factor and partition two each
> with a spring boot Kafka producer and consuming the same with another
> spring boot app.
>
> I found one strange behavior when testing the cluster in the following
> manner -
>
> 1. Turned off node1 and node 2
> 2. Turned on node 1
> 3. Turned off node 1
> 4. Turned on node 2
>
> After turning on node 2 Kafka cluster got failed and I am not able to
> produce data to Kafka. My consumer started throwing the message
> continuously as given below.
>
> [Producer clientId=producer-1] Connection to node 1 (/server1-ip:9092)
> could not be established. Broker may not be available.
>
> Issue is visible in both nodes. But if I kept both system up for a while
> issue will get resolved and I can turn off any of the node without breaking
> the cluster.
> My broker configuration is as below.
>
> broker.id=0
> listeners=PLAINTEXT://server1-ip:9092
> advertised.listeners=PLAINTEXT://serever1-ip:9092
> num.network.threads=3
> num.io.threads=8
> socket.send.buffer.bytes=102400
> socket.receive.buffer.bytes=102400
> socket.request.max.bytes=104857600
> log.dirs=/home/user/kafka/data/kafka-logs
> num.partitions=1
> num.recovery.threads.per.data.dir=2
> offsets.topic.replication.factor=2
> transaction.state.log.replication.factor=2
> transaction.state.log.min.isr=2
> log.retention.hours=168
> log.segment.bytes=1073741824
> log.retention.check.interval.ms=300000
> zookeeper.connect=serever1-ip:2181,serever2-ip:2181
> zookeeper.connection.timeout.ms=6000
> group.initial.rebalance.delay.ms=3000
> auto.leader.rebalance.enable=true
> leader.imbalance.check.interval.seconds=5
>
> Zookeeper configuration
>
> dataDir=/home/user/kafka/data
> clientPort=2181
> maxClientCnxns=0
> initLimit=10
> syncLimit=5
> tickTime=2000
> server.1=server1-ip:2888:3888
> server.2=server2-ip:2888:3888
>
> Is this is an expected behavior of Kafka or am I doing something wrong
> with this configuration ?
>
> Can somebody help me with this issue ..
>
> Thanks in advance.
>
>
>