[GitHub] [kafka] gharris1727 commented on pull request #13376: KAFKA-14091: Leader proactively aborting tasks from lost workers in rebalance in EOS mode

["gharris1727 (via GitHub)" <gi...@apache.org>]

gharris1727 commented on PR #13376:
URL: https://github.com/apache/kafka/pull/13376#issuecomment-1485855388

   @vamossagar12 I don't think this is the right approach.
   
   1. In the original design, hanging transactions blocking the global offsets topic was considered in detail, and the solution was accepted at the time to be per-connector offsets topics. I think this was in part that the framework couldn't make the right tradeoff of when to abort transactions in a way that was good enough for any use-case.
   
   2. Aborting transactions proactively on losing the task assignment treats every rebalance as a failure scenario, and does not permit the task to perform a clean end-of-lifetime-commit, which is explicitly handled via the (internal) TransactionBoundaryManager::shouldCommitFinalTransaction, and is used in all of the different transaction boundary modes.
   
   3. Fencing zombie source tasks is a very expensive and blocking operation, and it may delay rebalances that otherwise complete nearly instantly. I'm not sure how the cluster would react to assignments taking a significant amount of time to compute, but I don't think it would be more available and better behaved than it is now.
   
   3. This compromises the interface of the IncrementalCooperativeAssignor significantly. Previously, it only had an effect via the return value of performAssignment, and now it would have a side-effect for the (mutable) passed-in Coordinator.
   
   4. This puts a solution to a problem in one implementation of an interface (IncrementalCooperativeAssignor) and not another (EagerAssignor). Since this isn't an Incremental-specific problem, why is the solution only present in incremental mode?
   
   I'm not convinced that this problem needs solving, as users already have an alternative to the global offsets topic.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org