[jira] [Commented] (NIFI-752) Swapping and out of FlowFiles should be driven by FlowFileQueue, rather than in background thread

["Joseph Witt (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NIFI-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14988633#comment-14988633 ] 

Joseph Witt commented on NIFI-752:
----------------------------------

this is like *x1000* better now.  Nicely done mr payne.

I am mostly seriously about this next comment - since we swap out in batches of roughly 10000 flow files at a time and they create files in a flattened structure i think we can handle a backlog of somewhere south of 1Bil flowfiles before performance issues will occur due to number of files in a directory.  This is ... pretty good but I wonder if there are some pathological cases that could be more problematic.

> Swapping and out of FlowFiles should be driven by FlowFileQueue, rather than in background thread
> -------------------------------------------------------------------------------------------------
>
>                 Key: NIFI-752
>                 URL: https://issues.apache.org/jira/browse/NIFI-752
>             Project: Apache NiFi
>          Issue Type: Bug
>          Components: Core Framework
>            Reporter: Mark Payne
>             Fix For: 0.4.0
>
>
> As-is, if a huge number of FlowFiles are generated quickly enough, the JVM heap can fill before FlowFiles are swapped in. Conversely, if FlowFiles are swapped out, are NiFi is able to work those FlowFiles off, it ends up pausing, waiting for more FlowFiles to be swapped in. If the queue itself has control of this, these issues will both be alleviated. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)