You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@nifi.apache.org by "Joe Witt (Jira)" <ji...@apache.org> on 2019/10/10 20:45:00 UTC

[jira] [Updated] (MINIFICPP-1062) Improve FlowFile repository performance by leveraging batching

     [ https://issues.apache.org/jira/browse/MINIFICPP-1062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Joe Witt updated MINIFICPP-1062:
--------------------------------
    Summary: Improve FlowFile repository performance by leveraging batching  (was: Flowfile repo performance is unacceptable)

> Improve FlowFile repository performance by leveraging batching
> --------------------------------------------------------------
>
>                 Key: MINIFICPP-1062
>                 URL: https://issues.apache.org/jira/browse/MINIFICPP-1062
>             Project: Apache NiFi MiNiFi C++
>          Issue Type: Improvement
>    Affects Versions: 0.6.0
>            Reporter: Arpad Boda
>            Assignee: Arpad Boda
>            Priority: Blocker
>             Fix For: 0.7.0
>
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> In case a processor produces high number of flowfiles in big batches (50+ in one onTrigger call), MiNiFi performance is ridiculous, the max throughput is at about 100FF/s on a decent notebook. 
> The bottleneck is related to the way ProcessSession and Connection handles new flow records: every record is added to flowfile repo in a new transaction. 
> As rocksdb supports batching, flowfiles created in one batch should be added as a single transaction.  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)