You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by "dale.chang13" <da...@outlook.com> on 2016/05/16 13:13:24 UTC
Distribute FlowFiles among Nodes in a Cluster
I was wondering if there was a way for the NCM to distribute FlowFiles to
different nodes.
Currently I see that all of the nodes in my cluster run the same dataflow. I
know I can restrict certain processors to the primary node, but it seems
like the NCM does not distribute the FlowFiles to different nodes once the
primary node have finished their operations.
Additionally, I have checked nifi.properties and I do not have any
nifi.state properties configured
--
View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Distribute-FlowFiles-among-Nodes-in-a-Cluster-tp10293.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
Re: Distribute FlowFiles among Nodes in a Cluster
Posted by Bryan Bende <bb...@gmail.com>.
This post has a diagram of what Joe described:
https://community.hortonworks.com/articles/16120/how-do-i-distribute-data-across-a-nifi-cluster.html
-Bryan
On Mon, May 16, 2016 at 10:20 AM, Joe Witt <jo...@gmail.com> wrote:
> Dale,
>
> The common pattern here is to use NiFi's site-to-site function to
> distribute the load to the rest of the cluster. So you'd setup a
> remote input port on your flow and then connect the output of your
> primary only processor to a remote process group of that same cluster
> using that new port.
>
> Thanks
> Joe
>
> On Mon, May 16, 2016 at 9:13 AM, dale.chang13 <da...@outlook.com>
> wrote:
> > I was wondering if there was a way for the NCM to distribute FlowFiles to
> > different nodes.
> >
> > Currently I see that all of the nodes in my cluster run the same
> dataflow. I
> > know I can restrict certain processors to the primary node, but it seems
> > like the NCM does not distribute the FlowFiles to different nodes once
> the
> > primary node have finished their operations.
> >
> > Additionally, I have checked nifi.properties and I do not have any
> > nifi.state properties configured
> >
> >
> >
> > --
> > View this message in context:
> http://apache-nifi-developer-list.39713.n7.nabble.com/Distribute-FlowFiles-among-Nodes-in-a-Cluster-tp10293.html
> > Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>
Re: Distribute FlowFiles among Nodes in a Cluster
Posted by Joe Witt <jo...@gmail.com>.
Dale,
The common pattern here is to use NiFi's site-to-site function to
distribute the load to the rest of the cluster. So you'd setup a
remote input port on your flow and then connect the output of your
primary only processor to a remote process group of that same cluster
using that new port.
Thanks
Joe
On Mon, May 16, 2016 at 9:13 AM, dale.chang13 <da...@outlook.com> wrote:
> I was wondering if there was a way for the NCM to distribute FlowFiles to
> different nodes.
>
> Currently I see that all of the nodes in my cluster run the same dataflow. I
> know I can restrict certain processors to the primary node, but it seems
> like the NCM does not distribute the FlowFiles to different nodes once the
> primary node have finished their operations.
>
> Additionally, I have checked nifi.properties and I do not have any
> nifi.state properties configured
>
>
>
> --
> View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Distribute-FlowFiles-among-Nodes-in-a-Cluster-tp10293.html
> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.