Kafka(9.0.1) error : org.apache.kafka.common.network.InvalidReceiveException: Invalid receive (size = 1164731757 larger than 104857600)

[Arun Rai <ar...@gmail.com>]

Hello Kafka/Cassandra experts,


*I am getting below error….*

org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(
NetworkReceive.java:91)

        at org.apache.kafka.common.network.NetworkReceive.
readFrom(NetworkReceive.java:71)

        at org.apache.kafka.common.network.KafkaChannel.receive(
KafkaChannel.java:153)

        at org.apache.kafka.common.network.KafkaChannel.read(
KafkaChannel.java:134)

        at org.apache.kafka.common.network.Selector.poll(Selector.java:286)

        at kafka.network.Processor.run(SocketServer.scala:413)

        at java.lang.Thread.run(Thread.java:745)

*[2016-10-06 22:17:42,001] WARN Unexpected error from /10.61.48.28
<http://10.61.48.28/>; closing connection
(org.apache.kafka.common.network.Selector)*

org.apache.kafka.common.network.InvalidReceiveException: Invalid receive
(size = 1969447758 larger than 104857600)

        at



Below is my case.

*Cluster configuration :*



Kafka cluster : 3 vms with each one has 4 vcpu, 8 gb RAM

Cassandra : 4 vms with each one has 8 vcpu, 16 gb RAM



1.       Producer: Producer is rdkafka client, which produce the
messages(protobuf) to kafka topic(with 12 partitions). Please message size
always vary. It’s not fixed size message. Max message size could be from 1
mb to 3 mb.

2.       Consumer client : Client is java program who reads the messages
and load into Cassandra database.

*3.       **I get this error only when both producer and consumer is
running together , after 15-20 minute later I see this error in kafka log
files.*

4.       If I let consumer or producer run at single time.. I don’t get
this error.

5.       I do not get this error if Consumer only consume the messages and
do not insert into Cassandra.



Below is my kafka server configuration props:





broker.id=1

log.dirs=/data/kafka



host.name=10.61.19.87

port=9092



advertised.host.name=10.61.19.87

advertised.port=9092

listeners=PLAINTEXT://0.0.0.0:9092



delete.topic.enable=true



# Replication configurations

num.replica.fetchers=4

replica.fetch.max.bytes=1048576

replica.fetch.wait.max.ms=500

replica.high.watermark.checkpoint.interval.ms=5000

replica.socket.timeout.ms=30000

replica.socket.receive.buffer.bytes=65536

replica.lag.time.max.ms=10000



controller.socket.timeout.ms=30000

controller.message.queue.size=10



# Log configuration

num.partitions=12

message.max.bytes=1000000

auto.create.topics.enable=true

log.index.interval.bytes=4096

log.index.size.max.bytes=10485760

log.retention.hours=24

log.flush.interval.ms=10000

log.flush.interval.messages=20000

log.flush.scheduler.interval.ms=2000

log.roll.hours=24

log.retention.check.interval.ms=300000

log.segment.bytes=1073741824



# ZK configuration

zookeeper.connect=10.61.19.84:2181,10.61.19.86:2181

zookeeper.connection.timeout.ms=18000

zookeeper.sync.time.ms=2000



# Socket server configuration

num.io.threads=8

num.network.threads=8

socket.request.max.bytes=104857600

socket.receive.buffer.bytes=1048576

socket.send.buffer.bytes=1048576

queued.max.requests=16

fetch.purgatory.purge.interval.requests=100

producer.purgatory.purge.interval.requests=100



Any help will be really appreciated.

Arun Rai