[jira] [Created] (KAFKA-12499) Adjust transaction timeout according to commit interval on Streams EOS

["Boyang Chen (Jira)" <ji...@apache.org>]

Boyang Chen created KAFKA-12499:
-----------------------------------

             Summary: Adjust transaction timeout according to commit interval on Streams EOS
                 Key: KAFKA-12499
                 URL: https://issues.apache.org/jira/browse/KAFKA-12499
             Project: Kafka
          Issue Type: Improvement
          Components: streams
            Reporter: Boyang Chen
            Assignee: Boyang Chen
             Fix For: 3.0.0


The transaction timeout is set to 1 minute by default on Producer today, while the commit interval on the other hand could be set to a very large value, which makes the stream always hit transaction timeout and drop into rebalance. We should increase the transaction timeout correspondingly when commit interval is large.

On the other hand, broker could have a limit on the max transaction timeout to be set. If we scale up client transaction timeout over the limit, stream will fail due to  INVALID_TRANSACTION_TIMEOUT. To alleviate this problem, user could define their own customized transaction timeout to avoid hitting the limit, so we should still respect what user configures in the override.

The new rule for configuring transaction timeout should look like:
1. If transaction timeout is set in streams config, use it

2. if not, transaction_timeout = max(default_transaction_timeout, 10 * commit_interval) 

Additionally if INVALID_TRANSACTION_TIMEOUT was thrown on Streams when calling initTransaction(), we should wrap the exception to inform user that their setting for commit interval could potentially be too high and should adjust.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)