You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Boyang Chen (Jira)" <ji...@apache.org> on 2021/03/18 17:07:00 UTC
[jira] [Created] (KAFKA-12499) Adjust transaction timeout according
to commit interval on Streams EOS
Boyang Chen created KAFKA-12499:
-----------------------------------
Summary: Adjust transaction timeout according to commit interval on Streams EOS
Key: KAFKA-12499
URL: https://issues.apache.org/jira/browse/KAFKA-12499
Project: Kafka
Issue Type: Improvement
Components: streams
Reporter: Boyang Chen
Assignee: Boyang Chen
Fix For: 3.0.0
The transaction timeout is set to 1 minute by default on Producer today, while the commit interval on the other hand could be set to a very large value, which makes the stream always hit transaction timeout and drop into rebalance. We should increase the transaction timeout correspondingly when commit interval is large.
On the other hand, broker could have a limit on the max transaction timeout to be set. If we scale up client transaction timeout over the limit, stream will fail due to INVALID_TRANSACTION_TIMEOUT. To alleviate this problem, user could define their own customized transaction timeout to avoid hitting the limit, so we should still respect what user configures in the override.
The new rule for configuring transaction timeout should look like:
1. If transaction timeout is set in streams config, use it
2. if not, transaction_timeout = max(default_transaction_timeout, 10 * commit_interval)
Additionally if INVALID_TRANSACTION_TIMEOUT was thrown on Streams when calling initTransaction(), we should wrap the exception to inform user that their setting for commit interval could potentially be too high and should adjust.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)