[GitHub] [incubator-gobblin] sv2000 commented on a change in pull request #2912: [GOBBLIN-1072] Being more conservative on releasing containers

[GitBox <gi...@apache.org>]

sv2000 commented on a change in pull request #2912: [GOBBLIN-1072] Being more conservative on releasing containers
URL: https://github.com/apache/incubator-gobblin/pull/2912#discussion_r389233805
 
 

 ##########
 File path: gobblin-yarn/src/main/java/org/apache/gobblin/yarn/YarnAutoScalingManager.java
 ##########
 @@ -62,17 +67,24 @@
   private final String AUTO_SCALING_MIN_CONTAINERS = AUTO_SCALING_PREFIX + "minContainers";
   private final int DEFAULT_AUTO_SCALING_MIN_CONTAINERS = 1;
   private final String AUTO_SCALING_MAX_CONTAINERS = AUTO_SCALING_PREFIX + "maxContainers";
-  private final int DEFAULT_AUTO_SCALING_MAX_CONTAINERS = Integer.MAX_VALUE;
+  // A rough value of how much containers should be an intolerable number.
+  private final int DEFAULT_AUTO_SCALING_MAX_CONTAINERS = 5000;
 
 Review comment:
   In practice, the number of containers we request is limited by number of Helix partitions. So we would never get close to this limit. On the other hand, setting the default to a smaller value may result in hard-to-debug scenarios, where some partitions are stuck due to resource limits imposed by this setting.  

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services