[jira] [Created] (KAFKA-21083) Return the effective batch size limit in MESSAGE_TOO_LARGE responses to improve producer batch splitting

["sanghyeok An (Jira)" <ji...@apache.org>]

sanghyeok An created KAFKA-21083:
------------------------------------

             Summary:   Return the effective batch size limit in MESSAGE_TOO_LARGE responses to improve producer batch splitting
                 Key: KAFKA-21083
                 URL: https://issues.apache.org/jira/browse/KAFKA-21083
             Project: Kafka
          Issue Type: Improvement
            Reporter: sanghyeok An
            Assignee: sanghyeok An


*Description*

When a producer sends a record batch that exceeds the effective broker-side batch size limit, the broker rejects it with MESSAGE_TOO_LARGE. This limit applies to the record batch, after compression if enabled.

 

The producer currently splits the rejected batch using the configured batch.size. If an already-split batch is rejected again, it reduces the split target to half the estimated batch size, bounded below by the largest record size.

 

This approach allows progressively smaller batches, but the producer does not know the limit enforced by the broker. Consequently, split batches may still exceed that limit, resulting in additional rejected requests, decompression, batch rebuilding, and retransmission.

 

*Proposed improvement*

Include the effective maximum allowed record batch size in the MESSAGE_TOO_LARGE response for the affected partition. The value should reflect the applicable topic-level max.message.bytes, including the broker default when no topic override is configured.

 

The producer could use this value to choose a split target that respects both its configured batching target and the reported limit, reducing the need to discover an acceptable size through repeated failures.

 

For example, with a producer batch.size of 8 MiB and an effective broker limit of 1 MiB, the producer could use the reported limit immediately when splitting, instead of requiring further rejections to progressively reduce the target.

 

{*}Considerations{*}{*}{*}
 * Preserve the existing splitting behavior when the response does not provide a limit. (for compatiability for old version broker and producer)
 * Treat the reported limit as a hint: configuration changes and compression estimation errors can still cause subsequent rejections.
 * Preserve failure handling for a single-record batch that exceeds the limit; batch splitting does not divide an individual record.

 

*Related work*

KIP-126 identified obtaining the broker’s maximum message size as a future direction.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)