Heavy traffic between Kafka nodes

[Herbert Fischer <he...@crossengage.io>]

Hi,

I have a two node Kafka cluster, and I'm catching some unusual "TCP
retransmission" metrics from my monitoring. I did found out that the
network is a bit overloaded and most of it it's because the communication
between both nodes is too chatty.

It seems that there's more traffic between the nodes than coming/going
from/to all producers/consumers.

Is this normal? Any suggestions of what might be wrong?

best,

Herbert

Re: Heavy traffic between Kafka nodes

[Hans Jespersen <ha...@confluent.io>]

How much more data is flowing between the two nodes versus the total ingress rate of the cluster? I would expect inter-broker bandwidth use to be about the same as the ingress rate for the entire cluster since all the data coming into node 1 needs to get replicated to node 2 (for redundancy) and all the data flowing into node 2 needs to get replicated to node 1 (also for redundancy). The only way to reduce this would be to turn off replicator altogether which means you would lose data if a node where to crash or a disk were to die.

Are you running anything other than the kafka broker on these two nodes (like zookeeper or anything else)?

-hans




> On Dec 23, 2016, at 2:08 AM, Herbert Fischer <he...@crossengage.io> wrote:
> 
> Hi,
> 
> I have a two node Kafka cluster, and I'm catching some unusual "TCP
> retransmission" metrics from my monitoring. I did found out that the
> network is a bit overloaded and most of it it's because the communication
> between both nodes is too chatty.
> 
> It seems that there's more traffic between the nodes than coming/going
> from/to all producers/consumers.
> 
> Is this normal? Any suggestions of what might be wrong?
> 
> best,
> 
> Herbert