[jira] [Commented] (NIFI-2848) Queues aren't fairly drained when leading to a single component

["ASF subversion and git services (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NIFI-2848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15645080#comment-15645080 ] 

ASF subversion and git services commented on NIFI-2848:
-------------------------------------------------------

Commit 6aefc0b9102bc83456ff537ade22946715bdcd30 in nifi's branch refs/heads/master from [~pvillard]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=6aefc0b ]

NIFI-2751 NIFI-2848 Get batch of flow files in a round-robin manner

This closes #1111


> Queues aren't fairly drained when leading to a single component
> ---------------------------------------------------------------
>
>                 Key: NIFI-2848
>                 URL: https://issues.apache.org/jira/browse/NIFI-2848
>             Project: Apache NiFi
>          Issue Type: Improvement
>          Components: Core Framework
>    Affects Versions: 1.0.0, 0.7.0
>            Reporter: Joseph Gresock
>            Assignee: Pierre Villard
>             Fix For: 1.1.0
>
>         Attachments: Backpressure_prioritization_test.xml, dominate_1.png, dominate_2.png, queue_drain.png
>
>
> Consider the scenario where multiple queues lead to a single component and all of them are full due to back pressure.  With the attached template, it is easily observable that once a single queue starts to drain due to relieved back pressure, it will continue to drain as long as it has incoming flow files.  This means that if there's a constant flow of incoming flow files to this queue, the other queues will never be drained (at least, that's my theory based on several hours of observation).
> To reproduce this: 
> # Load the template into NiFi 1.0.0
> # Play all three GenerateFlowFile processors, but not the UpdateAttribute processor (this simulates backpressure).  Wait until each queue has 1,000 flow files (max backpressure)
> # Stop the GenerateFlowFile processors, and play the UpdateAttribute processor (this relieves the backpressure)
> # Observe which queue has started to drain, and start its GenerateFlowFile processor
> # Observe that the other two queues remain full indefinitely, while the draining queue continues to replenish and be drained indefinitely



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)