You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Lianet Magrans (Jira)" <ji...@apache.org> on 2026/02/09 17:48:00 UTC
[jira] [Updated] (KAFKA-20077) Producer using transactions v1 could hang on flush upon retriable failures adding partitions to tx
[ https://issues.apache.org/jira/browse/KAFKA-20077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lianet Magrans updated KAFKA-20077:
-----------------------------------
Description:
We've seen some occurrences of producer.flush hanging indefinitely in situations where a topic is deleted and the producer is using transactions v1
In the case where the producer has records in the buffer, and the topic deletion happens right before adding the first partition to the transaction, we could fall in a loop where the AddPartitionsToTx fails with a retriable error, and is continuously retried. In this case, none of the timeouts related to send, transactions or request seem to apply:
* [transaction.timeout.ms|http://transaction.timeout.ms/] -> not applied because no partition has been added yet
* [delivery.timeout.ms|http://delivery.timeout.ms/] -> not applied because the client does not attempt sending (where batching expiration applies) while it's in a transactional request (i.e AddPartitionsToTx), early return here [https://github.com/apache/kafka/blob/3d267d45369818c804ed49c56e9ae405e28b234c/clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java#L333-L335])
* [request.timeout.ms|http://request.timeout.ms/] -> not applied because it's not a request failure really, the high level operation too add partitions is retried.
* [default.api.timeout.ms|http://delivery.timeout.ms/] does not apply to the producer.flush api by design (or to any produce request really)
Client handing of retriable errors when adding partitions to tx (this would be the case of UNKNOWN_TOPIC_OR_PARTITION when a topic is deleted):
[https://github.com/apache/kafka/blob/3d267d45369818c804ed49c56e9ae405e28b234c/clients/src/main/java/org/apache/kafka/clients/producer/internals/TransactionManager.java#L1603-L1605]
This only affects producers using tx v1, and it's solved with tx v2 (partitions not added to the tx separately, so delivery timeout checked on send and applied, unblocking the flush operation).
was:
We've seen some occurrences of producer.flush hanging indefinitely in situations where a topic is deleted and the producer is using transactions v1 (not using 2pc)
In the case where the producer has records in the buffer, and the topic deletion happens right before adding the first partition to the transaction, we could fall in a loop where the AddPartitionsToTx fails with a retriable error, and is continuously retried. In this case, none of the timeouts related to send, transactions or request seem to apply:
* [transaction.timeout.ms|http://transaction.timeout.ms/] -> not applied because no partition has been added yet
* [delivery.timeout.ms|http://delivery.timeout.ms/] -> not applied because the client does not attempt sending (where batching expiration applies) while it's in a transactional request (i.e AddPartitionsToTx), early return here [https://github.com/apache/kafka/blob/3d267d45369818c804ed49c56e9ae405e28b234c/clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java#L333-L335])
* [request.timeout.ms|http://request.timeout.ms/] -> not applied because it's not a request failure really, the high level operation too add partitions is retried.
*
[default.api.timeout.ms|http://delivery.timeout.ms/] does not apply to the producer.flush api by design (or to any produce request really)
Client handing of retriable errors when adding partitions to tx (this would be the case of UNKNOWN_TOPIC_OR_PARTITION when a topic is deleted):
[https://github.com/apache/kafka/blob/3d267d45369818c804ed49c56e9ae405e28b234c/clients/src/main/java/org/apache/kafka/clients/producer/internals/TransactionManager.java#L1603-L1605]
This only affects producers using tx v1, and it's solved with tx v2 (partitions not added to the tx separately, so delivery timeout checked on send and applied, unblocking the flush operation).
> Producer using transactions v1 could hang on flush upon retriable failures adding partitions to tx
> ---------------------------------------------------------------------------------------------------
>
> Key: KAFKA-20077
> URL: https://issues.apache.org/jira/browse/KAFKA-20077
> Project: Kafka
> Issue Type: Bug
> Components: clients, producer
> Reporter: Lianet Magrans
> Priority: Major
>
> We've seen some occurrences of producer.flush hanging indefinitely in situations where a topic is deleted and the producer is using transactions v1
> In the case where the producer has records in the buffer, and the topic deletion happens right before adding the first partition to the transaction, we could fall in a loop where the AddPartitionsToTx fails with a retriable error, and is continuously retried. In this case, none of the timeouts related to send, transactions or request seem to apply:
> * [transaction.timeout.ms|http://transaction.timeout.ms/] -> not applied because no partition has been added yet
> * [delivery.timeout.ms|http://delivery.timeout.ms/] -> not applied because the client does not attempt sending (where batching expiration applies) while it's in a transactional request (i.e AddPartitionsToTx), early return here [https://github.com/apache/kafka/blob/3d267d45369818c804ed49c56e9ae405e28b234c/clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java#L333-L335])
> * [request.timeout.ms|http://request.timeout.ms/] -> not applied because it's not a request failure really, the high level operation too add partitions is retried.
> * [default.api.timeout.ms|http://delivery.timeout.ms/] does not apply to the producer.flush api by design (or to any produce request really)
> Client handing of retriable errors when adding partitions to tx (this would be the case of UNKNOWN_TOPIC_OR_PARTITION when a topic is deleted):
> [https://github.com/apache/kafka/blob/3d267d45369818c804ed49c56e9ae405e28b234c/clients/src/main/java/org/apache/kafka/clients/producer/internals/TransactionManager.java#L1603-L1605]
> This only affects producers using tx v1, and it's solved with tx v2 (partitions not added to the tx separately, so delivery timeout checked on send and applied, unblocking the flush operation).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)