Cluster Node Protocol Threads - will this configuration help reduce a high queue?

[Kevin Verhoeven <Ke...@ds-iq.com>]

I’m new to NiFi and I have a small problem. I’m running NiFi 0.7.0-SNAPSHOT on Windows 2012 R2 VMs. I’ve noticed that the queues on some of my Processors are very large, sometimes as high as 10,000 flowfiles. The flowfiles are eventually processed but at a slow pace.

I run a cluster with 4 nodes. The initial Get Processor runs on the Primary Node to only request a single file and I then use the DistributeLoad Processor with the Next Available setting to spread the load across the cluster nodes. However, I see that the queue is highest on the Primary Node and the cluster nodes see very little work.

My question is: will I increase throughput to the cluster nodes if I increase the nifi.cluster.node.protocol.threads from 2 to something higher? What effect does nifi.cluster.node.protocol.threads have on the nodes?

Thanks,

Kevin

RE: Cluster Node Protocol Threads - will this configuration help reduce a high queue?

[Kevin Verhoeven <Ke...@ds-iq.com>]

Thanks Matt,

I was able to send the flowfiles through a Remote Processor Group back into the cluster and the flowfiles were distributed appropriately. I appreciate the advice.

Kevin

From: Matthew Clarke [mailto:matt.clarke.138@gmail.com]
Sent: Saturday, May 28, 2016 1:44 PM
To: users@nifi.apache.org
Subject: Re: Cluster Node Protocol Threads - will this configuration help reduce a high queue?


Hey Kevin,
    The DistributeLoad processor in itself does not distribute data across to the other nodes your cluster.  Can you explain your flow in a little more detail? I am afraid i an missing something here. If you want to spread data across your nodes, I encourage you to use a remote process group. This will auto scale and auto load balance data to all your nodes. The setting you mentioned is for node communication but not data transmission between nodes. Increasing this values can help when you have a large number of nodes, but in your case with only four nodes it should not make much impact. Again these nodes comms are not for data. What processors are queues building behind?

Matt
On May 27, 2016 11:41 AM, "Kevin Verhoeven" <Ke...@ds-iq.com>> wrote:
I’m new to NiFi and I have a small problem. I’m running NiFi 0.7.0-SNAPSHOT on Windows 2012 R2 VMs. I’ve noticed that the queues on some of my Processors are very large, sometimes as high as 10,000 flowfiles. The flowfiles are eventually processed but at a slow pace.

I run a cluster with 4 nodes. The initial Get Processor runs on the Primary Node to only request a single file and I then use the DistributeLoad Processor with the Next Available setting to spread the load across the cluster nodes. However, I see that the queue is highest on the Primary Node and the cluster nodes see very little work.

My question is: will I increase throughput to the cluster nodes if I increase the nifi.cluster.node.protocol.threads from 2 to something higher? What effect does nifi.cluster.node.protocol.threads have on the nodes?

Thanks,

Kevin

Re: Cluster Node Protocol Threads - will this configuration help reduce a high queue?

[Matthew Clarke <ma...@gmail.com>]

Hey Kevin,
    The DistributeLoad processor in itself does not distribute data across
to the other nodes your cluster.  Can you explain your flow in a little
more detail? I am afraid i an missing something here. If you want to spread
data across your nodes, I encourage you to use a remote process group. This
will auto scale and auto load balance data to all your nodes. The setting
you mentioned is for node communication but not data transmission between
nodes. Increasing this values can help when you have a large number of
nodes, but in your case with only four nodes it should not make much
impact. Again these nodes comms are not for data. What processors are
queues building behind?

Matt
On May 27, 2016 11:41 AM, "Kevin Verhoeven" <Ke...@ds-iq.com>
wrote:

> I’m new to NiFi and I have a small problem. I’m running NiFi
> 0.7.0-SNAPSHOT on Windows 2012 R2 VMs. I’ve noticed that the queues on some
> of my Processors are very large, sometimes as high as 10,000 flowfiles. The
> flowfiles are eventually processed but at a slow pace.
>
>
>
> I run a cluster with 4 nodes. The initial Get Processor runs on the
> Primary Node to only request a single file and I then use the
> DistributeLoad Processor with the Next Available setting to spread the load
> across the cluster nodes. However, I see that the queue is highest on the
> Primary Node and the cluster nodes see very little work.
>
>
>
> My question is: will I increase throughput to the cluster nodes if I
> increase the nifi.cluster.node.protocol.threads from 2 to something higher?
> What effect does nifi.cluster.node.protocol.threads have on the nodes?
>
>
>
> Thanks,
>
>
>
> Kevin
>