produce very slow after upgrade to 2.3.0

[Xiangyuan LI <fl...@gmail.com>]

hi:
   recently we tried to upgrade our kafka cluster 0.10.0.1 to 2.3.0.
   we know kafka upgrade should:
      1.use 2.3.0.jar and restart  all brokers
      2.unset inter.broker.protocol.version=0.10.0.1 and restart all brokers
      3.unset log.message.format.version=0.10.0.1 and restart all brokers
   we have 15 clusters and most of them have already  complete step 1,2.
   but there are 2 custers left will decrease message produce speed when
all brokers unset inter.broker.protocol.version=0.10.0.1 and restart.
   this cluster has 5 brokers, we
unset  inter.broker.protocol.version=0.10.0.1 and restart all brokers, then
it seems after several minutes , auto leader rebalance triggered, then some
topics' producer decrease their produce speed.
    when this happened:
    1.both client & brokers cpu usage down,memory usage not change
    2.we found ioutil and io bandwidth is very slow ,so it almost
impossible that io
has any performance issue.
   3.we found some consumer group that has large lag before will keep
consume
message without consume speed change,so it seems that consumer
speed not be affected
   4.we found not all topics' produce speed are down. the topic has higest
message
produce speed will decrease certainly  and others are random. we have tried
to
upgrade many times,every time has different topics decrease produce speed.
   5.we only fonud NOT_LEADER_FOR_PARTITION error in producers' log,but it
should be ok when brokers restart. and there are no any error in brokers'
log.
   6. our producer will execute produce.flush() immediately after every
producer.send()
executed. i doubt whether its a big performace effect after we upgrade to
2.3.0?
   7.we have open producers' trace log, we found brokers' response didn't
has
very large latency but send and request num reduce.

  we hope some hint ,thx...