[jira] [Resolved] (NIFI-7501) Generate Flowfile does not scale

["Mike Thomsen (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/NIFI-7501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mike Thomsen resolved NIFI-7501.
--------------------------------
    Fix Version/s: 1.12.0
       Resolution: Fixed

> Generate Flowfile does not scale
> --------------------------------
>
>                 Key: NIFI-7501
>                 URL: https://issues.apache.org/jira/browse/NIFI-7501
>             Project: Apache NiFi
>          Issue Type: Improvement
>          Components: Extensions
>    Affects Versions: 1.11.4
>            Reporter: Dennis Jaheruddin
>            Priority: Minor
>             Fix For: 1.12.0
>
>         Attachments: generationperformance.xml
>
>          Time Spent: 20m
>  Remaining Estimate: 0h
>
> One of the purposes of Generate Flowfile is load testing. However, unfortunately it often appears to become the bottleneck itself. I have found it not to scale well.
> Example result from my laptop:
> I want to generate messages and bring them to a single processor, lets call it processor X.
> With 1 concurrent task, and a batch size of 1, and a message size of 10MB and uniqueness false it can generate approximately 2 GB/sec.
> When allowing for more concurrent tasks, or a larger batch size, no noticeable change is found.
> However, if instead of increasing the batchsize I route the success relationship to multiple processors that do 'nothing' (like updateattribute), and then bring the success relations of all these to processor X, I can get much more than 2 GB/sec. 
>  
> In conclusion: I don't appear to be hitting a hardware limit as I am able to generate the number of messages in this inelegant way, but no matter how I set up my generateflowfile processor, it just will not scale. Suggesting there may be a smarter way to generate data when uniqueness is not required.
>  
> I have attached a template to illustrate my findings.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)