You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "ivandika3 (via GitHub)" <gi...@apache.org> on 2026/05/21 10:47:58 UTC

[PR] HDDS-15338. Fix affinity executor queue skew [ozone]

ivandika3 opened a new pull request, #10325:
URL: https://github.com/apache/ozone/pull/10325

   ## What changes were proposed in this pull request?
   
   Currently, the implementation of hash for FixedThreadPoolWithAffinityExecutor (used to group the ContainerReportQueue assigned for container reports) is as follow
   
   {code:java}
       // For messages that need to be routed to the same thread need to
       // implement hashCode to match the messages. This should be safe for
       // other messages that implement the native hash.
       int index = message.hashCode() & (workQueues.size() - 1);
       BlockingQueue<Q> queue = workQueues.get(index);
   {code}
   
   However, this might cause skew when the queueCount is not a power of two (like the when the default is 10)
   
   For queueCount = 10
   
   {code:java}
   queueCount - 1 = 9
   9 in binary = 1001
   {code}
   
   So the routing becomes:
   
   {code:java}
   queueIndex = hash & 9
   {code}
   
   That means only the bits represented by 1001 are kept. The possible results are only:
   
   {code:java}
   0  -> 0000
   1  -> 0001
   8  -> 1000
   9  -> 1001
   {code}
   
   So with 10 queues, only queues 0, 1, 8, and 9 can be selected. Queues 2, 3, 4, 5, 6, 7 are effectively unused.
   
   Example
   
   {code:java}
   hash 0  -> 0000 & 1001 = 0000 = queue 0
   hash 1  -> 0001 & 1001 = 0001 = queue 1
   hash 2  -> 0010 & 1001 = 0000 = queue 0
   hash 3  -> 0011 & 1001 = 0001 = queue 1
   hash 4  -> 0100 & 1001 = 0000 = queue 0
   hash 5  -> 0101 & 1001 = 0001 = queue 1
   hash 8  -> 1000 & 1001 = 1000 = queue 8
   hash 9  -> 1001 & 1001 = 1001 = queue 9
   hash 10 -> 1010 & 1001 = 1000 = queue 8
   hash 11 -> 1011 & 1001 = 1001 = queue 9
   {code}
   
   *So the configured pool says “10 FCR processing queues”, but in practice it behaves closer to 4 active queues.*
   
   This is observed in the MAT tool check in HDDS-15330.
   
   If the queue count were 16:
   
   {code:java}
   queueCount - 1 = 15
   15 in binary = 1111
   queueIndex = hash & 15
   {code}
   
   Then possible queue indexes are 0..15, so all 16 queues can be used.
   
   Currently, we can set the queue count to power of two (e.g. 16).
   
   However, to fix the skew entirely, we can use modulo instead
   
   For example,
   {code:java}
   index = Math.floorMod(hash, queueCount);
   {code}
   
   Hopefully through this we can ensure FCR is processed quickly and SCM memory can be freed.
   
   ## What is the link to the Apache JIRA
   
   https://issues.apache.org/jira/browse/HDDS-15338
   
   ## How was this patch tested?
   
   UT.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

Re: [PR] HDDS-15338. Fix affinity executor queue assignment skew [ozone]

Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 merged PR #10325:
URL: https://github.com/apache/ozone/pull/10325


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

Re: [PR] HDDS-15338. Fix affinity executor queue assignment skew [ozone]

Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on PR #10325:
URL: https://github.com/apache/ozone/pull/10325#issuecomment-4514872413

   Thanks @yandrey321 @sodonnel for the reviews.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org