[jira] [Created] (NIFI-10824) Yellow indicator to say re-sorting is necessary & add a warn log if swapping from the Active to Swap Queue takes too long

["Ryan Hendrickson (Jira)" <ji...@apache.org>]

Ryan Hendrickson created NIFI-10824:
---------------------------------------

             Summary: Yellow indicator to say re-sorting is necessary & add a warn log if swapping from the Active to Swap Queue takes too long
                 Key: NIFI-10824
                 URL: https://issues.apache.org/jira/browse/NIFI-10824
             Project: Apache NiFi
          Issue Type: Improvement
            Reporter: Ryan Hendrickson


Version 1.1.0 introduced visual indication of queue growth relative to back pressure settings and when back pressure is engaged. .. It goes from Grey to Red, indicating Back-Pressure has kicked in... It'd be great to add a 3rd color, like Yellow, indicating that re-sorting will be necessary due to the back-pressure being larger than the queue size, then if it takes longer than XXX millis to sort, log something to the bulletin board.

 

*Background..*

A lot of people don't realize they shouldn't have a queue with a million flow files in it, especially with a Prioritizor set. 

 
For any Relationship there are 2 underlying queues: 
       (1) An Active Queue (java.util.PriorityQueue) for the first 20,000 items, defined by the nifi.queue.swap.threshold in nifi.properties.
       (2) A Swap Queue (java.util.ArrayList) for the rest of the queue's items.
 
If the Active Queue is full, every new item, regardless of priority, is placed on the Swap Queue.  No item on the Swap Queue will be re-prioritized until the entire Active Queue is empty ([Line 284|https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/SwappablePriorityQueue.java#L284]).  Thus, the PriorityAttributePrioritize only "actively" sorts the first 20,000 items on the relationship.  Once the queue is empty, it'll re-sort the rest and move the top 20,000 over.  Then the problem repeats.  On our team, no amount of training can get mitigate NiFi Dataflow users from setting the backpressure to unlimited or sky-high.
If backpressure is properly set, the time it takes to re-sort the the Swap Queue should be minimal.  In our case, we set the system-wide back-pressure size default to the same as the swap queue size to prevent any re-sorting.

However, if the backpressure is unset and the queue begins to grow in size, the re-sorting will take longer and longer.  At some threshold, I think it would be advantageous for NiFi to emit a warn log to the user saying it took over XXX millis to re-sort the queue and also to show a visual indictor that swapping & re-sorting will be necessary because the backpressure size is larger than the swap size.

 

https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/SwappablePriorityQueue.java#L271



--
This message was sent by Atlassian Jira
(v8.20.10#820010)