[GitHub] [kafka] ableegoldman commented on pull request #11433: KAFKA-13295: Avoiding Transation timeouts arising due to long restora…

[GitBox <gi...@apache.org>]

ableegoldman commented on PR #11433:
URL: https://github.com/apache/kafka/pull/11433#issuecomment-1276822852

   Hey @vamossagar12 , sorry for the seriously long silence on this -- it was important and I shouldn't have let it slip. Anyways I know you pointed me to a specific question about checkpointing but I wanted to discuss this quickly at a high-level first:
   
   It's been a while so forgive me for forgetting any context, but I tried to refresh my memory on this ticket and want to draw attention back to this part of an earlier comment of mine:
   
   > We want to commit only if there are new tasks which will need restoring. Unfortunately due to the task lifecycle, specifically that all tasks pass through the RESTORING phase before going into RUNNING, it's actually nontrivial to figure out if we're going to need to actually spend any time restoring new tasks. As a first pass, for now (so we can get some kind of fix into 3.1), we can just set commitAssignedActiveTasks = true if there are any newly added active tasks at all.
   
   Specifically, I think we want to avoid just blindly committing any/all active tasks after a rebalance, because being able to skip that was a major reason for the improvements of cooperative rebalancing & being able to keep rebalances as lightweight as possible is still important. So rather than doing that, let's try and minimize the impact by limiting when we consider committing to only when (a) EOS is used (since this is a txn-based issue to begin with) and (b) if we have newly-added active tasks (as discussed previously)
   
   Does that make sense? I guess this boils down to, can we first tighten up [this condition here](https://github.com/apache/kafka/pull/11433/files/230a8ed9abd0b4766985274efadb6236547b38de#diff-8baa5d7209fc00074bf3fe24d709c2dcf2a44c1623d7ced8c0e29c1d832a3bcbR384)? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org