You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by James McMahon <js...@gmail.com> on 2019/07/02 14:37:11 UTC
DistributeLoad across a NiFi cluster
We would like to employ a DistributeLoad processor, restricted to run on
the primary node of our cluster. Is there a recommended approach employed
to efficiently distribute across nodes in the cluster?
As I understand it, and using a FetchFile running in "all nodes" as the
first processor following the DistributeLoad, I can have it distribute by
round robin, next available, or load distribution service. Can anyone
provide a link to an example that employs the load distribution service? Is
that the recommended distribution approach when running in clustered mode?
I am interested in maintaining load balance across my cluster nodes when
running at high flowfile volumes. Flow files will vary greatly in contents,
so I'd like to design with an approach that helps me balance processing
distribution.
Thanks very much in advance. -Jim
Re: DistributeLoad across a NiFi cluster
Posted by dan young <da...@gmail.com>.
If you're going to upgrade, I would recommend jumping to the latest
version, 1.9.2 as of today. We ran into some issues in 1.8 with this
feature that was fixed in 1.9.x. We're running 1.9.2 now with this feature
now in production.
Regards
Dano
On Tue, Jul 9, 2019, 6:58 AM <Jo...@swisscom.com> wrote:
> The feature requires NiFi > 1.8.x… Pierre describes it very well in his
> blog :
> https://pierrevillard.com/2018/10/29/nifi-1-8-revolutionizing-the-list-fetch-pattern-and-more/
>
>
>
>
>
> *From: *James McMahon <js...@gmail.com>
> *Reply-To: *"users@nifi.apache.org" <us...@nifi.apache.org>
> *Date: *Tuesday, 9 July 2019 at 14:46
> *To: *"users@nifi.apache.org" <us...@nifi.apache.org>
> *Subject: *Re: DistributeLoad across a NiFi cluster
>
>
>
> Andrew, when I right click on the connection between the two I do not see
> a cluster distribution strategy in the queue connection. I am running
> 1.7.1.g. Am I overlooking something?
>
>
>
> On Tue, Jul 2, 2019 at 12:29 PM Andrew Grande <ap...@gmail.com> wrote:
>
> Jim,
>
>
>
> There's a better solution in NiFi. Right click on the connection between
> ListFile and FetchFile and select a cluster distribution strategy in
> options. That's it :)
>
>
>
> Andrew
>
>
>
> On Tue, Jul 2, 2019, 7:37 AM James McMahon <js...@gmail.com> wrote:
>
> We would like to employ a DistributeLoad processor, restricted to run on
> the primary node of our cluster. Is there a recommended approach employed
> to efficiently distribute across nodes in the cluster?
>
>
>
> As I understand it, and using a FetchFile running in "all nodes" as the
> first processor following the DistributeLoad, I can have it distribute by
> round robin, next available, or load distribution service. Can anyone
> provide a link to an example that employs the load distribution service? Is
> that the recommended distribution approach when running in clustered mode?
>
>
>
> I am interested in maintaining load balance across my cluster nodes when
> running at high flowfile volumes. Flow files will vary greatly in contents,
> so I'd like to design with an approach that helps me balance processing
> distribution.
>
>
>
> Thanks very much in advance. -Jim
>
>
Re: DistributeLoad across a NiFi cluster
Posted by Jo...@swisscom.com.
The feature requires NiFi > 1.8.x… Pierre describes it very well in his blog : https://pierrevillard.com/2018/10/29/nifi-1-8-revolutionizing-the-list-fetch-pattern-and-more/
From: James McMahon <js...@gmail.com>
Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
Date: Tuesday, 9 July 2019 at 14:46
To: "users@nifi.apache.org" <us...@nifi.apache.org>
Subject: Re: DistributeLoad across a NiFi cluster
Andrew, when I right click on the connection between the two I do not see a cluster distribution strategy in the queue connection. I am running 1.7.1.g. Am I overlooking something?
On Tue, Jul 2, 2019 at 12:29 PM Andrew Grande <ap...@gmail.com>> wrote:
Jim,
There's a better solution in NiFi. Right click on the connection between ListFile and FetchFile and select a cluster distribution strategy in options. That's it :)
Andrew
On Tue, Jul 2, 2019, 7:37 AM James McMahon <js...@gmail.com>> wrote:
We would like to employ a DistributeLoad processor, restricted to run on the primary node of our cluster. Is there a recommended approach employed to efficiently distribute across nodes in the cluster?
As I understand it, and using a FetchFile running in "all nodes" as the first processor following the DistributeLoad, I can have it distribute by round robin, next available, or load distribution service. Can anyone provide a link to an example that employs the load distribution service? Is that the recommended distribution approach when running in clustered mode?
I am interested in maintaining load balance across my cluster nodes when running at high flowfile volumes. Flow files will vary greatly in contents, so I'd like to design with an approach that helps me balance processing distribution.
Thanks very much in advance. -Jim
Re: DistributeLoad across a NiFi cluster
Posted by James McMahon <js...@gmail.com>.
Andrew, when I right click on the connection between the two I do not see a
cluster distribution strategy in the queue connection. I am running
1.7.1.g. Am I overlooking something?
On Tue, Jul 2, 2019 at 12:29 PM Andrew Grande <ap...@gmail.com> wrote:
> Jim,
>
> There's a better solution in NiFi. Right click on the connection between
> ListFile and FetchFile and select a cluster distribution strategy in
> options. That's it :)
>
> Andrew
>
> On Tue, Jul 2, 2019, 7:37 AM James McMahon <js...@gmail.com> wrote:
>
>> We would like to employ a DistributeLoad processor, restricted to run on
>> the primary node of our cluster. Is there a recommended approach employed
>> to efficiently distribute across nodes in the cluster?
>>
>> As I understand it, and using a FetchFile running in "all nodes" as the
>> first processor following the DistributeLoad, I can have it distribute by
>> round robin, next available, or load distribution service. Can anyone
>> provide a link to an example that employs the load distribution service? Is
>> that the recommended distribution approach when running in clustered mode?
>>
>> I am interested in maintaining load balance across my cluster nodes when
>> running at high flowfile volumes. Flow files will vary greatly in contents,
>> so I'd like to design with an approach that helps me balance processing
>> distribution.
>>
>> Thanks very much in advance. -Jim
>>
>
Re: DistributeLoad across a NiFi cluster
Posted by "Williams, Jim" <jw...@alertlogic.com>.
Edward,
Documentation on this feature may be found here: https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#settings
Warm regards,
Jim Williams
________________________________
From: Edward Armes <ed...@gmail.com>
Sent: Thursday, July 4, 2019 5:54 AM
To: users@nifi.apache.org
Subject: Re: DistributeLoad across a NiFi cluster
Hi Andrew,
Is this functionality documented anywhere do you know? As I've had a quick look through the documentation and I haven't seen this.
Edward
On Tue, Jul 2, 2019 at 5:33 PM James McMahon <js...@gmail.com>> wrote:
Excellent - thanks very much Andrew. This is my first crack at working with a clustered configuration, and I guess that shows by my question. Outstanding - thanks again.
On Tue, Jul 2, 2019 at 12:29 PM Andrew Grande <ap...@gmail.com>> wrote:
Jim,
There's a better solution in NiFi. Right click on the connection between ListFile and FetchFile and select a cluster distribution strategy in options. That's it :)
Andrew
On Tue, Jul 2, 2019, 7:37 AM James McMahon <js...@gmail.com>> wrote:
We would like to employ a DistributeLoad processor, restricted to run on the primary node of our cluster. Is there a recommended approach employed to efficiently distribute across nodes in the cluster?
As I understand it, and using a FetchFile running in "all nodes" as the first processor following the DistributeLoad, I can have it distribute by round robin, next available, or load distribution service. Can anyone provide a link to an example that employs the load distribution service? Is that the recommended distribution approach when running in clustered mode?
I am interested in maintaining load balance across my cluster nodes when running at high flowfile volumes. Flow files will vary greatly in contents, so I'd like to design with an approach that helps me balance processing distribution.
Thanks very much in advance. -Jim
Confidentiality Notice | This email and any included attachments may be privileged, confidential and/or otherwise protected from disclosure. Access to this email by anyone other than the intended recipient is unauthorized. If you believe you have received this email in error, please contact the sender immediately and delete all copies. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.
Re: DistributeLoad across a NiFi cluster
Posted by Edward Armes <ed...@gmail.com>.
Hi Andrew,
Is this functionality documented anywhere do you know? As I've had a quick
look through the documentation and I haven't seen this.
Edward
On Tue, Jul 2, 2019 at 5:33 PM James McMahon <js...@gmail.com> wrote:
> Excellent - thanks very much Andrew. This is my first crack at working
> with a clustered configuration, and I guess that shows by my question.
> Outstanding - thanks again.
>
> On Tue, Jul 2, 2019 at 12:29 PM Andrew Grande <ap...@gmail.com> wrote:
>
>> Jim,
>>
>> There's a better solution in NiFi. Right click on the connection between
>> ListFile and FetchFile and select a cluster distribution strategy in
>> options. That's it :)
>>
>> Andrew
>>
>> On Tue, Jul 2, 2019, 7:37 AM James McMahon <js...@gmail.com> wrote:
>>
>>> We would like to employ a DistributeLoad processor, restricted to run on
>>> the primary node of our cluster. Is there a recommended approach employed
>>> to efficiently distribute across nodes in the cluster?
>>>
>>> As I understand it, and using a FetchFile running in "all nodes" as the
>>> first processor following the DistributeLoad, I can have it distribute by
>>> round robin, next available, or load distribution service. Can anyone
>>> provide a link to an example that employs the load distribution service? Is
>>> that the recommended distribution approach when running in clustered mode?
>>>
>>> I am interested in maintaining load balance across my cluster nodes when
>>> running at high flowfile volumes. Flow files will vary greatly in contents,
>>> so I'd like to design with an approach that helps me balance processing
>>> distribution.
>>>
>>> Thanks very much in advance. -Jim
>>>
>>
Re: DistributeLoad across a NiFi cluster
Posted by James McMahon <js...@gmail.com>.
Excellent - thanks very much Andrew. This is my first crack at working with
a clustered configuration, and I guess that shows by my question.
Outstanding - thanks again.
On Tue, Jul 2, 2019 at 12:29 PM Andrew Grande <ap...@gmail.com> wrote:
> Jim,
>
> There's a better solution in NiFi. Right click on the connection between
> ListFile and FetchFile and select a cluster distribution strategy in
> options. That's it :)
>
> Andrew
>
> On Tue, Jul 2, 2019, 7:37 AM James McMahon <js...@gmail.com> wrote:
>
>> We would like to employ a DistributeLoad processor, restricted to run on
>> the primary node of our cluster. Is there a recommended approach employed
>> to efficiently distribute across nodes in the cluster?
>>
>> As I understand it, and using a FetchFile running in "all nodes" as the
>> first processor following the DistributeLoad, I can have it distribute by
>> round robin, next available, or load distribution service. Can anyone
>> provide a link to an example that employs the load distribution service? Is
>> that the recommended distribution approach when running in clustered mode?
>>
>> I am interested in maintaining load balance across my cluster nodes when
>> running at high flowfile volumes. Flow files will vary greatly in contents,
>> so I'd like to design with an approach that helps me balance processing
>> distribution.
>>
>> Thanks very much in advance. -Jim
>>
>
Re: DistributeLoad across a NiFi cluster
Posted by Andrew Grande <ap...@gmail.com>.
Jim,
There's a better solution in NiFi. Right click on the connection between
ListFile and FetchFile and select a cluster distribution strategy in
options. That's it :)
Andrew
On Tue, Jul 2, 2019, 7:37 AM James McMahon <js...@gmail.com> wrote:
> We would like to employ a DistributeLoad processor, restricted to run on
> the primary node of our cluster. Is there a recommended approach employed
> to efficiently distribute across nodes in the cluster?
>
> As I understand it, and using a FetchFile running in "all nodes" as the
> first processor following the DistributeLoad, I can have it distribute by
> round robin, next available, or load distribution service. Can anyone
> provide a link to an example that employs the load distribution service? Is
> that the recommended distribution approach when running in clustered mode?
>
> I am interested in maintaining load balance across my cluster nodes when
> running at high flowfile volumes. Flow files will vary greatly in contents,
> so I'd like to design with an approach that helps me balance processing
> distribution.
>
> Thanks very much in advance. -Jim
>