[GitHub] [hadoop] sunchao commented on a change in pull request #1977: HADOOP-17010. Add queue capacity support for FairCallQueue

[GitBox <gi...@apache.org>]

sunchao commented on a change in pull request #1977:
URL: https://github.com/apache/hadoop/pull/1977#discussion_r416001217



##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/FairCallQueue.java
##########
@@ -101,11 +111,18 @@ public FairCallQueue(int priorityLevels, int capacity, String ns,
 
     this.queues = new ArrayList<BlockingQueue<E>>(numQueues);
     this.overflowedCalls = new ArrayList<AtomicLong>(numQueues);
-    int queueCapacity = capacity / numQueues;
-    int capacityForFirstQueue = queueCapacity + (capacity % numQueues);
+    int totalWeights = 0;
+    for (int i = 0; i < capacityWeights.length; i++) {

Review comment:
       what if user passes in negative numbers as weights? should we add a safe guard against that?

##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/CallQueueManager.java
##########
@@ -343,6 +347,36 @@ private static int parseNumLevels(String ns, Configuration conf) {
     return retval;
   }
 
+  /**
+   * Read the weights of capacity in callqueue and pass the value to
+   * callqueue constructions.
+   */
+  private static int[] parseCapacityWeights(
+      int priorityLevels, String ns, Configuration conf) {
+    int[] weights = conf.getInts(ns + "." +
+      CommonConfigurationKeys.IPC_CALLQUEUE_CAPACITY_WEIGHTS_KEY);
+    if (weights.length == 0) {
+      weights = getDefaultQueueCapacityWeights(priorityLevels);
+    } else if (weights.length != priorityLevels) {
+      throw new IllegalArgumentException(
+          CommonConfigurationKeys.IPC_CALLQUEUE_CAPACITY_WEIGHTS_KEY + " must "
+              + "specify " + priorityLevels + " capacity weights: one for each "
+              + "priority level");
+    }
+    return weights;
+  }
+
+  /**
+   * By default, queue capacity is the same for all priority levels.

Review comment:
       nit: add a blank line between this and the first parameter line.

##########
File path: hadoop-common-project/hadoop-common/src/site/markdown/FairCallQueue.md
##########
@@ -126,6 +126,7 @@ omitted.
 |:---- |:---- |:---- |:--- |
 | backoff.enable | General | Whether or not to enable client backoff when a queue is full. | false |
 | callqueue.impl | General | The fully qualified name of a class to use as the implementation of a call queue. Use `org.apache.hadoop.ipc.FairCallQueue` for the Fair Call Queue. | `java.util.concurrent.LinkedBlockingQueue` (FIFO queue) |
+| callqueue.capacity.weights | General | The capacity allocation weights among all subqueues. A int array whose length is equal to the `scheduler.priority.levels` is expected where each int is the relative weight out of total capacity. i.e. if a queue with capacity weight `w`, its queue capacity is `capacity * w/sum(weights)` |

Review comment:
       maybe make it explicit only positive numbers are allowed?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org