[GitHub] [druid] AmatyaAvadhanula commented on a change in pull request #12221: Prevent deadlocks due to multiple parallel indexing tasks

[GitBox <gi...@apache.org>]

AmatyaAvadhanula commented on a change in pull request #12221:
URL: https://github.com/apache/druid/pull/12221#discussion_r799903838



##########
File path: indexing-service/src/main/java/org/apache/druid/indexing/overlord/ImmutableWorkerInfo.java
##########
@@ -119,12 +142,33 @@ public boolean isValidVersion(String minVersion)
     return worker.getVersion().compareTo(minVersion) >= 0;
   }
 
-  public boolean canRunTask(Task task)
+  public boolean canRunTask(Task task, double parallelIndexWorkerRatio)
   {
     return (worker.getCapacity() - getCurrCapacityUsed() >= task.getTaskResource().getRequiredCapacity()
+            && canRunParallelIndexTask(task, parallelIndexWorkerRatio)
             && !getAvailabilityGroups().contains(task.getTaskResource().getAvailabilityGroup()));
   }
 
+  public boolean canRunParallelIndexTask(Task task, double parallelIndexWorkerRatio)
+  {
+    if (!task.getType().equals(ParallelIndexSupervisorTask.TYPE)) {
+      return true;
+    }
+    return getWorkerParallelIndexCapacity(parallelIndexWorkerRatio) - getCurrParallelIndexCapacityUsed()

Review comment:
       Yes, that could be an issue with the default of 0.1
   
   We could set the default value to 1, in which case the behaviour would be the same as it is today.
   
   A default value of 0.5 might be the best? Deadlocks won't occur and subtask concurrency is not affected except in the cases where there a lot of supervising tasks are running
   
   The value can be configured to be lower when progress supervising tasks are configured to have a lot of subtasks and "fast" progress of a few ingestion tasks is preferred over "slow" progress of all the submitted ones




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org