[GitHub] [incubator-gobblin] sv2000 commented on a change in pull request #2912: [GOBBLIN-1072] Add sliding window to protect AutoScaling from fluctuation of number of active Helix partitions

[GitBox <gi...@apache.org>]

sv2000 commented on a change in pull request #2912: [GOBBLIN-1072] Add sliding window to protect AutoScaling from fluctuation of number of active Helix partitions
URL: https://github.com/apache/incubator-gobblin/pull/2912#discussion_r388651170
 
 

 ##########
 File path: gobblin-yarn/src/main/java/org/apache/gobblin/yarn/YarnAutoScalingManager.java
 ##########
 @@ -196,9 +207,84 @@ void runInternal() {
       // adjust the number of target containers based on the configured min and max container values.
       numTargetContainers = Math.max(this.minContainers, Math.min(this.maxContainers, numTargetContainers));
 
+      slidingFixedWindow.add(numTargetContainers);
 
 Review comment:
   A problem with your proposal is that it is based on numTargetContainers (and indirectly, numPartitions) which may remain unchanged over a window of time, even though the partitions with no assigned participant may be changing from one observation interval to the next. 
   
   E.g. Let us assume we have 4 Helix partitions. In one observation interval, suppose that [1,2,3] have an assigned participant but not 4. In the next observation interval, suppose partitions [2,3,4] have an assigned participant but not 1. If your sliding window is of size 2, then you will infer that numTargetContainers is 3, while what you want is for numTargetContainers to be 4. 
   
   I think what you want is to track which Helix instances are unused. The current implementation is based on a point-in-time check against Helix for the set of partitions which have no assigned participant. One possible improvement here could be marking a Helix instance as "unused" if it has no assigned participant for the last 10 minutes. 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services