[jira] [Created] (KAFKA-14064) MirrorMaker2 stops task when record is too big

["David Dufour (Jira)" <ji...@apache.org>]

David Dufour created KAFKA-14064:
------------------------------------

             Summary: MirrorMaker2 stops task when record is too big
                 Key: KAFKA-14064
                 URL: https://issues.apache.org/jira/browse/KAFKA-14064
             Project: Kafka
          Issue Type: Improvement
          Components: mirrormaker
    Affects Versions: 2.7.1
            Reporter: David Dufour


As MirrorMaker2 does currently not support shallow mirrorring ([KIP-712: Shallow Mirroring|https://wiki.apache.org/confluence/display/KAFKA/KIP-712%3A+Shallow+Mirroring]), if a producer has produced using compression in one mirrorred topic, MirrorMaker2 will get the message uncompressed at some point and if not properly tuned (typically {{{}max.request.size{}}}), it may fail with a RecordTooLargeException:
org.apache.kafka.connect.errors.ConnectException: Unrecoverable exception from producer send callback
    at org.apache.kafka.connect.runtime.WorkerSourceTask.maybeThrowProducerSendException(WorkerSourceTask.java:284)
    at org.apache.kafka.connect.runtime.WorkerSourceTask.sendRecords(WorkerSourceTask.java:338)
    at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:256)
    at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:189)
    at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:238)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
    Caused by: org.apache.kafka.common.errors.RecordTooLargeException: The message is 1049087 bytes when serialized which is larger than 1048576, which is the value of the max.request.size configuration.\n"
          worker_id: 'xxx.xxx.xxx.xxx:8083'

The task is stopped and needs a manual restart. 
However, this seems to be a bit overkill because, amongst all partitions replicated by the task, only one is problematic. Stopping the replication on all partitions can make a severe impact.
It would be more optimized to 'suspend' the partition involved and keep replication working for all remaining ones.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)