[GitHub] [nifi] markap14 commented on pull request #5580: NIFI-9390: Source components can ingest FlowFiles eagerly in stateles…

[GitBox <gi...@apache.org>]

markap14 commented on pull request #5580:
URL: https://github.com/apache/nifi/pull/5580#issuecomment-1006098373


   Thanks for digging into this @turcsanyip. I'm a bit concerned about the notion of adding an "eager" fetch though. It results in some significant changes to the framework, and it tends to be taking us in a direction that I don't personally believe is the right direction. It means that in order to use Merge related processors you'll need to know that and configure the engine as such. Otherwise the Merge processors will still appear to work but won't merge anything, which can flood downstream systems with millions of tiny files, etc. Plus it's a bit confusing for users because you're configuring something that is very specific to the dataflow from outside of the dataflow.
   
   I think we can address the issue very differently, though. Simply by updating MergeContent / MergeRecord so that they only create the smallest bin if they first poll and have no data available. Otherwise, we don't 'process' a bin unless the bin is full. I have created a PR that does this. Please check that out. https://github.com/apache/nifi/pull/5634


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org