No backoff on producer sending from MirrorMaker on metadata error

[Henry Cai <hc...@pinterest.com.INVALID>]

Version: Kafka 2.0.0 (Believe the problem also exists on 2.2.0)

The producer thread in MirrorMaker will go repeatedly sending loop without
any backoffs when encountered metadata errors.  For example, the following
error messages will repeat itself for hundred times with very small
interval (100ms) when it encountered some metadata error.

The producer thread in mirror maker was configured with
'retries=Integer.MAX', so this resending is very annoying.

By looking at Sender.java, the few other places when we call
metadata.requestUpdate(), it is usually after time.sleep(retryBackoffMs),
but not at this particular place (in method completeBatch())

[2019-03-11 02:46:45,988] WARN [Producer clientId=producer-1] Received
invalid metadata error in produce request on partition
incomplete_ad_conversion_secondary-32 due to
org.apache.kafka.common.errors.NotLeaderForPartitionException: This server
is not the leader for that topic-partition.. Going to request metadata
update now (org.apache.kafka.clients.producer.internals.Sender)

[2019-03-11 02:46:46,094] WARN [Producer clientId=producer-1] Got error
produce response with correlation id 1488890 on topic-partition
incomplete_ad_conversion_secondary-32, retrying (2147482734 attempts left).
Error: NOT_LEADER_FOR_PARTITION
(org.apache.kafka.clients.producer.internals.Sender)

[2019-03-11 02:46:46,094] WARN [Producer clientId=producer-1] Received
invalid metadata error in produce request on partition
incomplete_ad_conversion_secondary-32 due to
org.apache.kafka.common.errors.NotLeaderForPartitionException: This server
is not the leader for that topic-partition.. Going to request metadata
update now (org.apache.kafka.clients.producer.internals.Sender)