[GitHub] [kafka] vvcephei commented on pull request #8588: [WIP] KAFKA-6145: KIP-441: Validate balanced assignment

[GitBox <gi...@apache.org>]

vvcephei commented on pull request #8588:
URL: https://github.com/apache/kafka/pull/8588#issuecomment-621589430


   Hey @ableegoldman @cadonna ,
   
   I thought it would be good to add verification that the converged assignment is actually balanced to our randomized test. Really, I had https://issues.apache.org/jira/browse/KAFKA-9173 in mind. This won't fix the StickyTaskAssignor, but maybe it can make sure that HATA won't ever do the same thing.
   
   What do you think about the test condition in particular? It's a more aggressive interpretation of "balance" than what we've discussed before; is it appropriate? The test fails for me pretty reliably, which makes sense, because we didn't design the algorithm with this definition of balance in mind.
   
   For example, you can reproduce the failure I'm looking at with seed `8608745620218291125`, in which has an imbalance of 2 for active tasks and for standbys.
   
   The active tasks get imbalanced because we put some stateless tasks on instances that have active tasks already, even though there are empty instances. 
   
   The standby tasks are imbalanced, too, and I'm not sure why. There are some nodes with six standbys and some nodes with only four. It looks like the nodes with four are also the nodes with two active tasks, so that might be the reason. I haven't looked back at the standby assignment code yet.
   
   Thoughts?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org