You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Isha Lamboo <is...@virtualsciences.nl> on 2022/02/23 17:54:54 UTC
Performance implications of RPGs for loadbalancing
Hi all,
I’m hoping to get some perspective from people that have NiFi with a large number of Remote Process Groups.
I’m supporting a NiFi 1.12.1 (yes, I know) cluster of 3 nodes that has about 5k processors and load-balancing still done the pre-1.8 way, with RPGs looping back to the local cluster. There are 500+ RPGs with only about 30 actually going to other NiFi clusters.
We’re having several problems:
* input ports getting stuck when the RPG is set to HTTP protocol and connections get killed by the Jetty DoS filter after 30 secs. The standard is RAW, but sometimes a HTTP RPG still gets deployed.
* Intermittent errors like EoF, connection closed etc on HTTP connections
* The cluster being unable to sync changes made to the flow resulting in disconnected nodes and sometimes uninheritable flow exceptions.
My idea is that the RPGs should be replaced by load-balanced connection and/or local ports, but developer resources are scarce, so I want to either make a business case or tune NiFi performance if 500 RPGs should not cause problems normally.
So is this a known issue or particular to my case? How can I identify/solve performance bottlenecks with RPGs?
Kind regards,
Isha Lamboo
Re: Performance implications of RPGs for loadbalancing
Posted by David Handermann <ex...@apache.org>.
Hi Isha,
Thanks for following up. Regarding thread usage, one additional benefit of
load-balanced connections is that they have a separate thread pool for
handling sending and receiving FlowFiles. Increasing the number of
concurrent tasks on any given component will not necessarily help, and may
actually have a negative impact on overall behavior. Changing the S2S
communication from HTTP to RAW may help, as it avoids using threads from
the web server pool, but in any case, changing to use load-balanced
connections should provide better overall behavior.
Regards,
David Handermann
On Fri, Feb 25, 2022 at 3:06 AM Isha Lamboo <is...@virtualsciences.nl>
wrote:
> Hi David,
>
>
>
> Thanks for your reply.
>
>
>
> I’m confident I’ve identified the root cause now, having been able to
> reproduce the symptoms on a test cluster.
>
>
>
> The issue appears to be not just having many RPGs, but many of them
> pointing to the same input port (that leads to the generic audit logging
> flow).
>
> Under even a low to moderate load a point is reached after some time where
> the input port is not being scheduled often enough, causing a chain
> reaction.
>
> Every RPG that gets scheduled then spends 30 seconds holding its thread
> until the connection to the input port times out. Because the RPGs and
> input ports are on the same cluster and even node for 1/3 of the files,
> this starves the input port (and frankly the whole NiFi instance) of
> threads, increasing the timeout issues, disconnecting nodes, causing UI
> issues etc.
>
>
>
> The solution is still migrating to loadbalanced connections and removing
> the RPGs, though I worry about the same chain reaction if we simply replace
> the RPGs with local output ports all pointing to the same input port. That
> takes time to implement though, so I’m looking to tweaking settings to keep
> the system running for now.
>
>
>
> So my question this time around: What is supposed to happen if I set the
> concurrent tasks for an input port to a high number (say 10-20)? Will the
> port get scheduled with exactly that number or as many threads as are
> available? If there are no 10 threads available, will the port get
> scheduled at all?
>
>
>
> Regards,
>
>
>
> Isha
>
>
>
>
>
>
>
> *Van:* David Handermann <ex...@apache.org>
> *Verzonden:* woensdag 23 februari 2022 19:57
> *Aan:* users@nifi.apache.org
> *Onderwerp:* Re: Performance implications of RPGs for loadbalancing
>
>
>
> Hi Isha,
>
>
>
> Thanks for providing some background on the configuration and related
> issues. Based on the issues you highlighted, it sounds like you are running
> into several known problems. There are some potential workarounds, but
> refactoring the flow configuration to use standard connection load
> balancing is the best solution. Upgrading to NiFi 1.15.3 addresses a number
> of security and performance issues, including some of the items you
> mentioned.
>
>
>
> Related to the first problem, using RAW socket communication should be
> preferred for RPG communication. RAW socket communication is not subject
> to the Denial-of-Service filter timeout, and also has less overhead than
> HTTP request processing. Ensuring that all Remote Process Groups use RAW
> socket communication should help. When HTTP requests exceed the DoS filter
> timeout, Jetty terminates the connection, which can produce any number of
> errors, such as the End-of-File and Connection Closed issues you have
> observed.
>
>
>
> Using HTTP communication also uses threads from the Jetty server, which
> can impact user interface performance. This might also be part of the
> explanation for cluster nodes getting out of sync, but there could be other
> factors involved.
>
>
>
> NiFi 1.12.1 includes several issues related to the Denial-of-Service
> filter and Site-to-Site communication, which have been addressed in more
> recent releases. Here are a couple worth noting:
>
> - https://issues.apache.org/jira/browse/NIFI-7912
> <https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FNIFI-7912&data=04%7C01%7Cisha.lamboo%40virtualsciences.nl%7C7ef36a49d18346a103d508d9f6fe6086%7C21429da9e4ad45f99a6fcd126a64274b%7C0%7C0%7C637812394689837710%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=txyv5SOJ1aGn0dmdC9z%2F3nhgqsVQoRjlFMnQNobMiIM%3D&reserved=0>
> Added new nifi.web.request properties that can be used to change the
> default 30 second timeout and exclude IP addresses from filtering
>
> - https://issues.apache.org/jira/browse/NIFI-9448
> <https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FNIFI-9448&data=04%7C01%7Cisha.lamboo%40virtualsciences.nl%7C7ef36a49d18346a103d508d9f6fe6086%7C21429da9e4ad45f99a6fcd126a64274b%7C0%7C0%7C637812394689837710%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=PoP4SfPIPaauwGQxvi4S%2FI208TkpW%2BB0%2FBrtPPBYw6I%3D&reserved=0>
> Resolved potential IllegalStateException for S2S client communication
>
> - https://issues.apache.org/jira/browse/NIFI-9481
> <https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FNIFI-9481&data=04%7C01%7Cisha.lamboo%40virtualsciences.nl%7C7ef36a49d18346a103d508d9f6fe6086%7C21429da9e4ad45f99a6fcd126a64274b%7C0%7C0%7C637812394689837710%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=QulchPc53aRYbvUSQIl3cycHtiVWsiF4VVww%2BdBClsg%3D&reserved=0>
> Exclude HTTP Site-to-Site Communication from DoS Filter
>
>
>
> The last issue is not yet part of a released version, but the other two
> are resolved in NiFi 1.15.3.
>
>
>
> Although upgrading and migrating to connection load balancing will take
> some work, it is the best path forward to address the issues you observed.
>
>
>
> Regards,
>
> David Handermann
>
>
>
> On Wed, Feb 23, 2022 at 11:55 AM Isha Lamboo <
> isha.lamboo@virtualsciences.nl> wrote:
>
> Hi all,
>
>
>
> I’m hoping to get some perspective from people that have NiFi with a large
> number of Remote Process Groups.
>
>
>
> I’m supporting a NiFi 1.12.1 (yes, I know) cluster of 3 nodes that has
> about 5k processors and load-balancing still done the pre-1.8 way, with
> RPGs looping back to the local cluster. There are 500+ RPGs with only about
> 30 actually going to other NiFi clusters.
>
>
>
> We’re having several problems:
>
> · input ports getting stuck when the RPG is set to HTTP protocol
> and connections get killed by the Jetty DoS filter after 30 secs. The
> standard is RAW, but sometimes a HTTP RPG still gets deployed.
>
> · Intermittent errors like EoF, connection closed etc on HTTP
> connections
>
> · The cluster being unable to sync changes made to the flow
> resulting in disconnected nodes and sometimes uninheritable flow exceptions.
>
>
>
> My idea is that the RPGs should be replaced by load-balanced connection
> and/or local ports, but developer resources are scarce, so I want to either
> make a business case or tune NiFi performance if 500 RPGs should not cause
> problems normally.
>
>
>
> So is this a known issue or particular to my case? How can I
> identify/solve performance bottlenecks with RPGs?
>
>
>
> Kind regards,
>
>
>
> Isha Lamboo
>
>
>
>
RE: Performance implications of RPGs for loadbalancing
Posted by Isha Lamboo <is...@virtualsciences.nl>.
Hi David,
Thanks for your reply.
I’m confident I’ve identified the root cause now, having been able to reproduce the symptoms on a test cluster.
The issue appears to be not just having many RPGs, but many of them pointing to the same input port (that leads to the generic audit logging flow).
Under even a low to moderate load a point is reached after some time where the input port is not being scheduled often enough, causing a chain reaction.
Every RPG that gets scheduled then spends 30 seconds holding its thread until the connection to the input port times out. Because the RPGs and input ports are on the same cluster and even node for 1/3 of the files, this starves the input port (and frankly the whole NiFi instance) of threads, increasing the timeout issues, disconnecting nodes, causing UI issues etc.
The solution is still migrating to loadbalanced connections and removing the RPGs, though I worry about the same chain reaction if we simply replace the RPGs with local output ports all pointing to the same input port. That takes time to implement though, so I’m looking to tweaking settings to keep the system running for now.
So my question this time around: What is supposed to happen if I set the concurrent tasks for an input port to a high number (say 10-20)? Will the port get scheduled with exactly that number or as many threads as are available? If there are no 10 threads available, will the port get scheduled at all?
Regards,
Isha
Van: David Handermann <ex...@apache.org>
Verzonden: woensdag 23 februari 2022 19:57
Aan: users@nifi.apache.org
Onderwerp: Re: Performance implications of RPGs for loadbalancing
Hi Isha,
Thanks for providing some background on the configuration and related issues. Based on the issues you highlighted, it sounds like you are running into several known problems. There are some potential workarounds, but refactoring the flow configuration to use standard connection load balancing is the best solution. Upgrading to NiFi 1.15.3 addresses a number of security and performance issues, including some of the items you mentioned.
Related to the first problem, using RAW socket communication should be preferred for RPG communication. RAW socket communication is not subject to the Denial-of-Service filter timeout, and also has less overhead than HTTP request processing. Ensuring that all Remote Process Groups use RAW socket communication should help. When HTTP requests exceed the DoS filter timeout, Jetty terminates the connection, which can produce any number of errors, such as the End-of-File and Connection Closed issues you have observed.
Using HTTP communication also uses threads from the Jetty server, which can impact user interface performance. This might also be part of the explanation for cluster nodes getting out of sync, but there could be other factors involved.
NiFi 1.12.1 includes several issues related to the Denial-of-Service filter and Site-to-Site communication, which have been addressed in more recent releases. Here are a couple worth noting:
- https://issues.apache.org/jira/browse/NIFI-7912<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FNIFI-7912&data=04%7C01%7Cisha.lamboo%40virtualsciences.nl%7C7ef36a49d18346a103d508d9f6fe6086%7C21429da9e4ad45f99a6fcd126a64274b%7C0%7C0%7C637812394689837710%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=txyv5SOJ1aGn0dmdC9z%2F3nhgqsVQoRjlFMnQNobMiIM%3D&reserved=0> Added new nifi.web.request properties that can be used to change the default 30 second timeout and exclude IP addresses from filtering
- https://issues.apache.org/jira/browse/NIFI-9448<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FNIFI-9448&data=04%7C01%7Cisha.lamboo%40virtualsciences.nl%7C7ef36a49d18346a103d508d9f6fe6086%7C21429da9e4ad45f99a6fcd126a64274b%7C0%7C0%7C637812394689837710%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=PoP4SfPIPaauwGQxvi4S%2FI208TkpW%2BB0%2FBrtPPBYw6I%3D&reserved=0> Resolved potential IllegalStateException for S2S client communication
- https://issues.apache.org/jira/browse/NIFI-9481<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FNIFI-9481&data=04%7C01%7Cisha.lamboo%40virtualsciences.nl%7C7ef36a49d18346a103d508d9f6fe6086%7C21429da9e4ad45f99a6fcd126a64274b%7C0%7C0%7C637812394689837710%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=QulchPc53aRYbvUSQIl3cycHtiVWsiF4VVww%2BdBClsg%3D&reserved=0> Exclude HTTP Site-to-Site Communication from DoS Filter
The last issue is not yet part of a released version, but the other two are resolved in NiFi 1.15.3.
Although upgrading and migrating to connection load balancing will take some work, it is the best path forward to address the issues you observed.
Regards,
David Handermann
On Wed, Feb 23, 2022 at 11:55 AM Isha Lamboo <is...@virtualsciences.nl>> wrote:
Hi all,
I’m hoping to get some perspective from people that have NiFi with a large number of Remote Process Groups.
I’m supporting a NiFi 1.12.1 (yes, I know) cluster of 3 nodes that has about 5k processors and load-balancing still done the pre-1.8 way, with RPGs looping back to the local cluster. There are 500+ RPGs with only about 30 actually going to other NiFi clusters.
We’re having several problems:
· input ports getting stuck when the RPG is set to HTTP protocol and connections get killed by the Jetty DoS filter after 30 secs. The standard is RAW, but sometimes a HTTP RPG still gets deployed.
· Intermittent errors like EoF, connection closed etc on HTTP connections
· The cluster being unable to sync changes made to the flow resulting in disconnected nodes and sometimes uninheritable flow exceptions.
My idea is that the RPGs should be replaced by load-balanced connection and/or local ports, but developer resources are scarce, so I want to either make a business case or tune NiFi performance if 500 RPGs should not cause problems normally.
So is this a known issue or particular to my case? How can I identify/solve performance bottlenecks with RPGs?
Kind regards,
Isha Lamboo
Re: Performance implications of RPGs for loadbalancing
Posted by David Handermann <ex...@apache.org>.
Hi Isha,
Thanks for providing some background on the configuration and related
issues. Based on the issues you highlighted, it sounds like you are running
into several known problems. There are some potential workarounds, but
refactoring the flow configuration to use standard connection load
balancing is the best solution. Upgrading to NiFi 1.15.3 addresses a number
of security and performance issues, including some of the items you
mentioned.
Related to the first problem, using RAW socket communication should be
preferred for RPG communication. RAW socket communication is not subject
to the Denial-of-Service filter timeout, and also has less overhead than
HTTP request processing. Ensuring that all Remote Process Groups use RAW
socket communication should help. When HTTP requests exceed the DoS filter
timeout, Jetty terminates the connection, which can produce any number of
errors, such as the End-of-File and Connection Closed issues you have
observed.
Using HTTP communication also uses threads from the Jetty server, which can
impact user interface performance. This might also be part of the
explanation for cluster nodes getting out of sync, but there could be other
factors involved.
NiFi 1.12.1 includes several issues related to the Denial-of-Service filter
and Site-to-Site communication, which have been addressed in more recent
releases. Here are a couple worth noting:
- https://issues.apache.org/jira/browse/NIFI-7912 Added new
nifi.web.request properties that can be used to change the default 30
second timeout and exclude IP addresses from filtering
- https://issues.apache.org/jira/browse/NIFI-9448 Resolved potential
IllegalStateException for S2S client communication
- https://issues.apache.org/jira/browse/NIFI-9481 Exclude HTTP Site-to-Site
Communication from DoS Filter
The last issue is not yet part of a released version, but the other two are
resolved in NiFi 1.15.3.
Although upgrading and migrating to connection load balancing will take
some work, it is the best path forward to address the issues you observed.
Regards,
David Handermann
On Wed, Feb 23, 2022 at 11:55 AM Isha Lamboo <is...@virtualsciences.nl>
wrote:
> Hi all,
>
>
>
> I’m hoping to get some perspective from people that have NiFi with a large
> number of Remote Process Groups.
>
>
>
> I’m supporting a NiFi 1.12.1 (yes, I know) cluster of 3 nodes that has
> about 5k processors and load-balancing still done the pre-1.8 way, with
> RPGs looping back to the local cluster. There are 500+ RPGs with only about
> 30 actually going to other NiFi clusters.
>
>
>
> We’re having several problems:
>
> - input ports getting stuck when the RPG is set to HTTP protocol and
> connections get killed by the Jetty DoS filter after 30 secs. The standard
> is RAW, but sometimes a HTTP RPG still gets deployed.
> - Intermittent errors like EoF, connection closed etc on HTTP
> connections
> - The cluster being unable to sync changes made to the flow resulting
> in disconnected nodes and sometimes uninheritable flow exceptions.
>
>
>
> My idea is that the RPGs should be replaced by load-balanced connection
> and/or local ports, but developer resources are scarce, so I want to either
> make a business case or tune NiFi performance if 500 RPGs should not cause
> problems normally.
>
>
>
> So is this a known issue or particular to my case? How can I
> identify/solve performance bottlenecks with RPGs?
>
>
>
> Kind regards,
>
>
>
> Isha Lamboo
>
>
>