kafka producer error

[Shushant Arora <sh...@gmail.com>]

Hi

I have a kafka cluster with 3 brokers. I have a topic with ~50 partitions
and replication factpr of 3.

When 2 brokers are down - I m getting below error in producer code

5/09/09 00:56:15 WARN network.Selector: Error in I/O with brokerIPXXXX(Ip
of broker which is down)
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.kafka.common.network.Selector.poll(Selector.java:238)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:192)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:191)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:122)
at java.lang.Thread.run(Thread.java:745)

Producer config are :



value.serializer = class
org.apache.kafka.common.serialization.StringSerializer
key.serializer = class
org.apache.kafka.common.serialization.StringSerializer
block.on.buffer.full = true
retry.backoff.ms = 200
buffer.memory = 10485760
batch.size = 16384
metrics.sample.window.ms = 30000
metadata.max.age.ms = 300000
receive.buffer.bytes = 32768
timeout.ms = 30000
max.in.flight.requests.per.connection = 5
bootstrap.servers = [broker1:9092,broker2:9092,broker3:9092]
retries = 10
max.request.size = 1048576
send.buffer.bytes = 131072
acks = 1
reconnect.backoff.ms = 10
linger.ms = 0
metrics.num.samples = 2
metadata.fetch.timeout.ms = 60000


despite being acks as 1 - I m getting IOError - and its giving error for
broker which is down - but my topic has replication and when I see in kafka
console describe on topic - new leader of all partitions is broker which is
Up.

So why the IOException is coming?

Re: kafka producer error

[Kamal C <ka...@gmail.com>]

Refer the mail list
http://qnalist.com/questions/6002514/new-producer-metadata-update-problem-on-2-node-cluster

https://issues.apache.org/jira/browse/KAFKA-1843

On Wed, Sep 9, 2015 at 7:37 AM, Shushant Arora <sh...@gmail.com>
wrote:

> Hi
>
> I have a kafka cluster with 3 brokers. I have a topic with ~50 partitions
> and replication factpr of 3.
>
> When 2 brokers are down - I m getting below error in producer code
>
> 5/09/09 00:56:15 WARN network.Selector: Error in I/O with brokerIPXXXX(Ip
> of broker which is down)
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
> at org.apache.kafka.common.network.Selector.poll(Selector.java:238)
> at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:192)
> at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:191)
> at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:122)
> at java.lang.Thread.run(Thread.java:745)
>
> Producer config are :
>
>
>
> value.serializer = class
> org.apache.kafka.common.serialization.StringSerializer
> key.serializer = class
> org.apache.kafka.common.serialization.StringSerializer
> block.on.buffer.full = true
> retry.backoff.ms = 200
> buffer.memory = 10485760
> batch.size = 16384
> metrics.sample.window.ms = 30000
> metadata.max.age.ms = 300000
> receive.buffer.bytes = 32768
> timeout.ms = 30000
> max.in.flight.requests.per.connection = 5
> bootstrap.servers = [broker1:9092,broker2:9092,broker3:9092]
> retries = 10
> max.request.size = 1048576
> send.buffer.bytes = 131072
> acks = 1
> reconnect.backoff.ms = 10
> linger.ms = 0
> metrics.num.samples = 2
> metadata.fetch.timeout.ms = 60000
>
>
> despite being acks as 1 - I m getting IOError - and its giving error for
> broker which is down - but my topic has replication and when I see in kafka
> console describe on topic - new leader of all partitions is broker which is
> Up.
>
> So why the IOException is coming?
>