[jira] [Commented] (NIFI-6787) Load balanced connection can hold up for whole cluster when one node slows down

["ASF subversion and git services (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NIFI-6787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17026208#comment-17026208 ] 

ASF subversion and git services commented on NIFI-6787:
-------------------------------------------------------

Commit a78cecd95f1a690614c17f38aa003f3d78e468cb in nifi's branch refs/heads/support/nifi-1.11.x from Joe Witt
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=a78cecd ]

Revert "NIFI-6787 - Before: When checking if a load balanced connection queue is full, we compare the totalSize.get() and getMaxQueueSize()."

This reverts commit 773160958209a8a173b4f741517c4bda63a28e82.


> Load balanced connection can hold up for whole cluster when one node slows down
> -------------------------------------------------------------------------------
>
>                 Key: NIFI-6787
>                 URL: https://issues.apache.org/jira/browse/NIFI-6787
>             Project: Apache NiFi
>          Issue Type: Improvement
>          Components: Core Framework
>            Reporter: Tamas Palfy
>            Assignee: Tamas Palfy
>            Priority: Major
>             Fix For: 1.11.0
>
>         Attachments: template.change_hostname_in_groovy_script.xml
>
>          Time Spent: 50m
>  Remaining Estimate: 0h
>
> A slow processor on one node in a cluster can slow down the same processor on the other nodes when load balanced incoming connection is used:
> When a processors decides wether it can pass along a flowfile, it checks whether the outgoing connection is full or not (if full, it yields).
> Similarly when the receiving processor is to be scheduled the framework checks if the incoming connection is empty (if empty, no reason to call {{onTrigger}}).
> The problem is that when checking if it's full or not, it checks the _total_ size (which is across the whole cluster) and compares it to the max (which is scoped to the current node).
> The empty check is (correctly) done on the local partition only.
> This can lead to the case where the slow node fills up its queue while the faster ones empty theirs.
> Once the slow node has a full queue, the fast ones stop receiving new input and thus stop working after their queues get emptied.
> The issue is probably the fact that {{SocketLoadBalancedFlowFileQueue.isFull}} actually calls to {{AbstractFlowFileQueue.isFull}} which checks {{size()}}, but that returns the _total_ size. (The empty check looks fine but for reference it is done via {{SocketLoadBalancedFlowFileQueue.isActiveQueueEmpty}}).
> {code:title=AbstractFlowFileQueue.java|borderStyle=solid}
> ...
>     private MaxQueueSize getMaxQueueSize() {
>         return maxQueueSize.get();
>     }
>     @Override
>     public boolean isFull() {
>         return isFull(size());
>     }
>     protected boolean isFull(final QueueSize queueSize) {
>         final MaxQueueSize maxSize = getMaxQueueSize();
>         // Check if max size is set
>         if (maxSize.getMaxBytes() <= 0 && maxSize.getMaxCount() <= 0) {
>             return false;
>         }
>         if (maxSize.getMaxCount() > 0 && queueSize.getObjectCount() >= maxSize.getMaxCount()) {
>             return true;
>         }
>         if (maxSize.getMaxBytes() > 0 && queueSize.getByteCount() >= maxSize.getMaxBytes()) {
>             return true;
>         }
>         return false;
>     }
> ...
> {code}
> {code:title=SocketLoadBalancedFlowFileQueue.java|borderStyle=solid}
> ...
>     @Override
>     public QueueSize size() {
>         return totalSize.get();
>     }
> ...
>     @Override
>     public boolean isActiveQueueEmpty() {
>         return localPartition.isActiveQueueEmpty();
>     }
> ...
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)