[jira] [Comment Edited] (KAFKA-9878) Block AddPartitionsToTxn call until the txn markers are committed

["HaiyuanZhao (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-9878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17114641#comment-17114641 ] 

HaiyuanZhao edited comment on KAFKA-9878 at 5/23/20, 9:47 AM:
--------------------------------------------------------------

According to the description of kip-609, KAFKA-9878 and KAFKA-9860, as far as I understand, there are three main changes.
 1 Make the new beginTxn a block call which would be sending the batched addPartitions and group id request, and which could be blocked until the previous transaction completes
 2 Update sendOffsetsToTransaction: no need to register group id towards to txn coordinator anymore.
 3 Make endTxn to include the actual written partitions to minimize necessary cost

Did I understand right? Fix me if I misunderstand this improvement or miss some key points. Thanks 


was (Author: zhaohaidao):
According to the description of kip-609, Kafka9878 and KAFKA-9860, as far as I understand, there are three main changes.
1 Make the new beginTxn a block call which would be sending the batched addPartitions and group id request, and which could be blocked until the previous transaction completes
2 Update sendOffsetsToTransaction: no need to register group id towards to txn coordinator anymore.
3 Make endTxn to include the actual written partitions to minimize necessary cost

Did I understand right? Fix me if I misunderstand this improvement or miss some key points. Thanks 

> Block AddPartitionsToTxn call until the txn markers are committed
> -----------------------------------------------------------------
>
>                 Key: KAFKA-9878
>                 URL: https://issues.apache.org/jira/browse/KAFKA-9878
>             Project: Kafka
>          Issue Type: Improvement
>            Reporter: Boyang Chen
>            Priority: Major
>
> Currently the EndTxn call from Producer will immediately return as the control record is written to the txn coordinator log. The ongoing transaction will be going to a pending state to wait for all txn markers to be propagated. In the meantime, producer client will start another new transaction but being rejected constantly until the pending state gets resolved, which is unnecessary round trips and more burden to the broker to handle repetitive requests.
> To avoid this situation, we should make the Producer client wait for txn marker completion instead. This will incur better performance overall, as no more back-off shall be triggered for a subsequent transaction to begin.
> On the other hand, we could also batch complete the AddPartitionsToTxn results if we buffered more than one request in the queue.
> The third change is on the client side, which is to maintain the futures of the AddPartitionsToTxn calls to make more inflight changes as necessary.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)