[jira] [Resolved] (NIFI-6790) Load-balanced queues cause huge number of open pipes

["David Handermann (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/NIFI-6790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

David Handermann resolved NIFI-6790.
------------------------------------
    Fix Version/s: 1.11.3
       Resolution: Fixed

> Load-balanced queues cause huge number of open pipes
> ----------------------------------------------------
>
>                 Key: NIFI-6790
>                 URL: https://issues.apache.org/jira/browse/NIFI-6790
>             Project: Apache NiFi
>          Issue Type: Bug
>          Components: Core Framework
>    Affects Versions: 1.9.2
>         Environment: Ubuntu,OpenJDK8
>            Reporter: Matthew Knight
>            Priority: Critical
>              Labels: Cluster, LoadBalancing
>             Fix For: 1.11.3
>
>         Attachments: flow_exception.png, lsof_loadbalanced.png, lsof_notloadbalanced.png
>
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> Even very basic flows using load balance connecting queues result in huge numbers of open pipes which don't seem to get closed and result in the entire cluster being brought to a standstill because of "too many open files".  I tried apples-to-apples with equivalent very basic flows, one load balanced and one not, to show how easy this has been to reproduce.  I'm attaching some screenshots that show the "too many open files" issue in action in my NiFi cluster, along with the output of LSOF for the primary node of a cluster with vs. without load balancing.  When load balancing is enabled there is a huge number of 'pipe' and 'eventpoll' items, without load balancing things are a bit more balanced.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)