You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Guozhang Wang (Jira)" <ji...@apache.org> on 2021/03/28 00:09:00 UTC
[jira] [Updated] (KAFKA-12499) Adjust transaction timeout according
to commit interval on Streams EOS
[ https://issues.apache.org/jira/browse/KAFKA-12499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Guozhang Wang updated KAFKA-12499:
----------------------------------
Priority: Blocker (was: Major)
> Adjust transaction timeout according to commit interval on Streams EOS
> ----------------------------------------------------------------------
>
> Key: KAFKA-12499
> URL: https://issues.apache.org/jira/browse/KAFKA-12499
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Reporter: Boyang Chen
> Assignee: Boyang Chen
> Priority: Blocker
> Fix For: 3.0.0
>
>
> The transaction timeout is set to 1 minute by default on Producer today, while the commit interval on the other hand could be set to a very large value, which makes the stream always hit transaction timeout and drop into rebalance. We should increase the transaction timeout correspondingly when commit interval is large.
> On the other hand, broker could have a limit on the max transaction timeout to be set. If we scale up client transaction timeout over the limit, stream will fail due to INVALID_TRANSACTION_TIMEOUT. To alleviate this problem, user could define their own customized transaction timeout to avoid hitting the limit, so we should still respect what user configures in the override.
> The new rule for configuring transaction timeout should look like:
> 1. If transaction timeout is set in streams config, use it
> 2. if not, transaction_timeout = max(default_transaction_timeout, 10 * commit_interval)
> Additionally if INVALID_TRANSACTION_TIMEOUT was thrown on Streams when calling initTransaction(), we should wrap the exception to inform user that their setting for commit interval could potentially be too high and should adjust.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)