You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Jean-Sebastien Vachon <js...@brizodata.com> on 2019/09/23 18:46:57 UTC

How to account for a very heavy process

Hi all,

the image below illustrates one of the issue I am getting with my current flow. As you can see in this image, one of the processor requires a lot more time to process each flow file.
I am trying to configure my flow to reduce processing time as much as possible. I've tuned the number of processes to give more power the last module, set the size of each queue to use back pressure when
possible, etc. But I'm always falling short on succeeding.  I got five servers with 72 cores and lot of ram and in its current state, I am not using all available resources as I only have 31 processes running which is way
below the system's maximal capacity. Are there any other strategies that would require less trial and error on my side (which is a lengthy process) ?

Thanks

[cid:6c4a5524-e63b-4d9d-a0c6-af64cdc6c9ea]