Issue with nifi startup performance

[李磊 <l2...@163.com>]

Hi all,

I use KYLO with NIFI-1.3.0 now. In KYLO, I create 100 feed with data-ingest template to transform data form RDBMS to HDFS, which provided by kylo.Finally, more than 1000 processors were created and there are 29 processors in the reuseable_templates. NIFI uses 6g memory.

The problem is that when I restart the NIFI, more than 7 mins cost. By modify the source code and print log, I find that nearly 6 min costs in org.apache.nifi.controller.StandardFlowSynchronizer Class<sync>, especially method <addProcessGroup>. By read the source code, I think the function of <addProcessGroup> is analyze flow.xml.gz and restore the NIFI status before turned off.

Is there any ways to improve the performance of <addProcessGroup>? We will create more than 1000 feeds in KYLO in production, means more than 10000 processors will be existed in NIFI.

Thanks!

Re: Issue with nifi startup performance

[Mark Payne <ma...@hotmail.com>]

Hello,

I would recommend grabbing a thread dump while nifi is starting up, in the part that takes the longest. Typically, the startup time is related more to restoring the repositories more than it is to restoring the flow.

Also, are you running on physical hardware or in a VM? How many cpus are available and how many/what kind of disks (spinning vs ssd vs network attached storage, etc) are you using?

Sent from my iPhone

> On Mar 17, 2018, at 6:57 AM, 李磊 <l2...@163.com> wrote:
> 
> Hi all,
> 
> I use KYLO with NIFI-1.3.0 now. In KYLO, I create 100 feed with data-ingest template to transform data form RDBMS to HDFS, which provided by kylo.Finally, more than 1000 processors were created and there are 29 processors in the reuseable_templates. NIFI uses 6g memory.
> 
> The problem is that when I restart the NIFI, more than 7 mins cost. By modify the source code and print log, I find that nearly 6 min costs in org.apache.nifi.controller.StandardFlowSynchronizer Class<sync>, especially method <addProcessGroup>. By read the source code, I think the function of <addProcessGroup> is analyze flow.xml.gz and restore the NIFI status before turned off.
> 
> Is there any ways to improve the performance of <addProcessGroup>? We will create more than 1000 feeds in KYLO in production, means more than 10000 processors will be existed in NIFI.
> 
> Thanks!