[jira] [Commented] (KAFKA-12499) Adjust transaction timeout according to commit interval on Streams EOS

["A. Sophie Blee-Goldman (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-12499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304537#comment-17304537 ] 

A. Sophie Blee-Goldman commented on KAFKA-12499:
------------------------------------------------

Nice catch, but just wondering why choose 10x the commit interval? And should we set a lower bound as well so we fall back on the default of 1 minute if 10 * commit_interval is below that? For example with EOS that would mean a transaction timeout of just one second -- we only check whether it's time to commit after a loop of processing all tasks, if there are a large number of tasks or some heavy processing we'd just be constantly timing out.

> Adjust transaction timeout according to commit interval on Streams EOS
> ----------------------------------------------------------------------
>
>                 Key: KAFKA-12499
>                 URL: https://issues.apache.org/jira/browse/KAFKA-12499
>             Project: Kafka
>          Issue Type: Improvement
>          Components: streams
>            Reporter: Boyang Chen
>            Assignee: Boyang Chen
>            Priority: Major
>             Fix For: 3.0.0
>
>
> The transaction timeout is set to 1 minute by default on Producer today, while the commit interval on the other hand could be set to a very large value, which makes the stream always hit transaction timeout and drop into rebalance. We should increase the transaction timeout correspondingly when commit interval is large.
> On the other hand, broker could have a limit on the max transaction timeout to be set. If we scale up client transaction timeout over the limit, stream will fail due to  INVALID_TRANSACTION_TIMEOUT. To alleviate this problem, user could define their own customized transaction timeout to avoid hitting the limit, so we should still respect what user configures in the override.
> The new rule for configuring transaction timeout should look like:
> 1. If transaction timeout is set in streams config, use it
> 2. if not, transaction_timeout = max(default_transaction_timeout, 10 * commit_interval) 
> Additionally if INVALID_TRANSACTION_TIMEOUT was thrown on Streams when calling initTransaction(), we should wrap the exception to inform user that their setting for commit interval could potentially be too high and should adjust.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)