[jira] [Updated] (NIFI-6068) StandardFunnel looping excessively

["Brandon DeVries (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/NIFI-6068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Brandon DeVries updated NIFI-6068:
----------------------------------
    Description: 
The StandardFunnel, when triggered, reads up to 100 FlowFiles at a time and writes them to the outgoing relationship.  It does this inside of a loop, with the only exit conditions being:
 # No FlowFiles read (input queue empty)
 # No Available relationships to transfer to

It does this using a thread from the timer driven thread pool.  Unfortunately, this means that a flow can be created such that a constant trickle of data will prevent processors from running, as all threads are in use by funnels which will never let them go.

To fix this, I'd suggest adding 2 checks.  First, if session.get(100) returns fewer than 100 FlowFiles, transfer those files and then return.  Essentially, while more files may subsequently come in, at that point the queue had been emptied. 

Second, put a cap on the number of FlowFiles transferred per call to onTrigger.  While there may still be files to move downstream, after you've transferred 100K or so it might be time to give up the thread.

  was:
The StandardFunnel, when triggered, reads up to 100 FlowFiles at a time and writes them to the outgoing relationship.  It does this inside of a loop, with the only exit conditions being:
 # No FlowFiles read (input queue empty)
 # No Available relationships to transfer to

It does this using a thread from the timer driven thread pool.  Unfortunately, this means that a flow can be created such that a constant trickle of data will prevent processors from running, as all threads are in use by ports, which will never let them go.

To fix this, I'd suggest adding 2 checks.  First, if session.get(100) returns fewer than 100 FlowFiles, transfer those files and then return.  Essentially, while more files may subsequently come in, at that point the queue had been emptied. 

Second, put a cap on the number of FlowFiles transferred per call to onTrigger.  While there may still be files to move downstream, after you've transferred 100K or so it might be time to give up the thread.


> StandardFunnel looping excessively
> ----------------------------------
>
>                 Key: NIFI-6068
>                 URL: https://issues.apache.org/jira/browse/NIFI-6068
>             Project: Apache NiFi
>          Issue Type: Bug
>          Components: Core Framework
>    Affects Versions: 1.9.0
>            Reporter: Brandon DeVries
>            Priority: Major
>
> The StandardFunnel, when triggered, reads up to 100 FlowFiles at a time and writes them to the outgoing relationship.  It does this inside of a loop, with the only exit conditions being:
>  # No FlowFiles read (input queue empty)
>  # No Available relationships to transfer to
> It does this using a thread from the timer driven thread pool.  Unfortunately, this means that a flow can be created such that a constant trickle of data will prevent processors from running, as all threads are in use by funnels which will never let them go.
> To fix this, I'd suggest adding 2 checks.  First, if session.get(100) returns fewer than 100 FlowFiles, transfer those files and then return.  Essentially, while more files may subsequently come in, at that point the queue had been emptied. 
> Second, put a cap on the number of FlowFiles transferred per call to onTrigger.  While there may still be files to move downstream, after you've transferred 100K or so it might be time to give up the thread.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)