[GitHub] [hive] pgaref commented on a change in pull request #2108: HIVE-24914: Improve LLAP scheduling by only traversing hosts with capacity

[GitBox <gi...@apache.org>]

pgaref commented on a change in pull request #2108:
URL: https://github.com/apache/hive/pull/2108#discussion_r612744867



##########
File path: llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
##########
@@ -1816,23 +1776,90 @@ private static boolean removeFromRunningTaskMap(TreeMap<Integer, TreeSet<TaskInf
     INADEQUATE_TOTAL_RESOURCES,
   }
 
+  private Pair<Resource, Map<String, List<NodeInfo>>> getResourceAvailability() {
+    int memory = 0;
+    int vcores = 0;
+    int numInstancesFound = 0;
+    Map<String, List<NodeInfo>> availableHostMap;
+    readLock.lock();
+    try {
+      // maintain insertion order (needed for Next slot in locality miss)
+      availableHostMap = new LinkedHashMap<>(instanceToNodeMap.size());
+      Collection<LlapServiceInstance> instances = consistentSplits ?
+          // might also include Inactive instances
+          activeInstances.getAllInstancesOrdered(true):
+          // if consistent splits are NOT used we don't need the ordering as there will be no cache benefit anyways
+          activeInstances.getAll();
+      boolean foundSlot = false;
+      for (LlapServiceInstance inst : instances) {
+        NodeInfo nodeInfo = instanceToNodeMap.get(inst.getWorkerIdentity());
+        if (nodeInfo != null) {
+          List<NodeInfo> hostList = availableHostMap.get(nodeInfo.getHost());
+          if (hostList == null) {
+            hostList = new ArrayList<>();
+            availableHostMap.put(nodeInfo.getHost(), hostList);
+          }
+          if (!(inst instanceof InactiveServiceInstance)) {
+            Resource r = inst.getResource();
+            memory += r.getMemory();
+            vcores += r.getVirtualCores();
+            numInstancesFound++;
+            // Only add to List Nodes with available resources
+            // Hosts, however, exist even for nodes that do not currently have resources
+            if (nodeInfo.canAcceptTask()) {
+              foundSlot = true;
+              hostList.add(nodeInfo);
+            }
+          }
+        }
+      }
+      // isClusterCapacityFull will be set to false on every trySchedulingPendingTasks call
+      // set it false here to bail out early when we know there are no resources available.
+      if (!foundSlot) {
+        isClusterCapacityFull.set(true);
+      }
+    } finally {
+      readLock.unlock();
+    }
+    if (LOG.isDebugEnabled()) {
+      LOG.debug("ResourceAvail: numInstancesFound={}, totalMem={}, totalVcores={} availableHosts: {}",

Review comment:
       sure fixed




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org