Controlling number of threads in distributed mode

[Milan Das <md...@interset.com>]

Hello,
Is there a way to control number of thread less than cluster member ?
I have a 5 node NiFI cluster and I want to run 3 instances of the  processor in distributed mode.

Thanks,
Milan Das

Re: Controlling number of threads in distributed mode

[Joe Witt <jo...@gmail.com>]

Ok.  So looks like the information needed to make a request comes from a
Kafka topic.
Then requests are made against some HTTP endpoint for which it wants a max
number of requests at once.
Given how small that number is I'd not worry at all about nifi being
overloaded but perhaps there is more depth to this.

I'd just turn on load balancing on the connection between ConsumeKafka and
InvokeHttpRequest and let the messages be consumed and requests sent
naturally.

Thanks

On Wed, Feb 13, 2019 at 9:52 PM Milan Das <md...@interset.com> wrote:

> Hi Joe,
> Use case is I have to query HTTP request.  The HTTP  request  from  NIFI
> can only run 3 queries at a time.
>
> The queries to be executed are coming from Kafka topic and we have 6 node
> cluster.
> Ideally would like to run the 3 http queries from 3 different nodes
> otherwise one of the node (primary node) will be overloaded with data.
> Thanks,
> Milan Das
>
>
>
> > On Feb 13, 2019, at 4:38 PM, Joe Witt <jo...@gmail.com> wrote:
> >
> > Milan
> >
> > The only mechanism to limit whether a given component executes on a given
> > nifi node or not is whether it runs on all or whether it runs on primary
> > node.
> >
> > Can you describe the use case that would lead to wanting to run a
> component
> > on a specified number of nodes as opposed to all nodes?  It would defy
> the
> > clustering logic/data distribution mechanisms of site to site and load
> > balanced connections for example.
> >
> > Thanks
> >
> > On Wed, Feb 13, 2019 at 4:34 PM Milan Das <md...@interset.com> wrote:
> >
> >> Hello,
> >> Is there a way to control number of thread less than cluster member ?
> >> I have a 5 node NiFI cluster and I want to run 3 instances of the
> >> processor in distributed mode.
> >>
> >> Thanks,
> >> Milan Das
> >>
> >>
>
>

Re: Controlling number of threads in distributed mode

[Milan Das <md...@interset.com>]

Hi Joe,
Use case is I have to query HTTP request.  The HTTP  request  from  NIFI can only run 3 queries at a time.

The queries to be executed are coming from Kafka topic and we have 6 node cluster.
Ideally would like to run the 3 http queries from 3 different nodes otherwise one of the node (primary node) will be overloaded with data.
Thanks,
Milan Das



> On Feb 13, 2019, at 4:38 PM, Joe Witt <jo...@gmail.com> wrote:
> 
> Milan
> 
> The only mechanism to limit whether a given component executes on a given
> nifi node or not is whether it runs on all or whether it runs on primary
> node.
> 
> Can you describe the use case that would lead to wanting to run a component
> on a specified number of nodes as opposed to all nodes?  It would defy the
> clustering logic/data distribution mechanisms of site to site and load
> balanced connections for example.
> 
> Thanks
> 
> On Wed, Feb 13, 2019 at 4:34 PM Milan Das <md...@interset.com> wrote:
> 
>> Hello,
>> Is there a way to control number of thread less than cluster member ?
>> I have a 5 node NiFI cluster and I want to run 3 instances of the
>> processor in distributed mode.
>> 
>> Thanks,
>> Milan Das
>> 
>> 

Re: Controlling number of threads in distributed mode

[Joe Witt <jo...@gmail.com>]

Milan

The only mechanism to limit whether a given component executes on a given
nifi node or not is whether it runs on all or whether it runs on primary
node.

Can you describe the use case that would lead to wanting to run a component
on a specified number of nodes as opposed to all nodes?  It would defy the
clustering logic/data distribution mechanisms of site to site and load
balanced connections for example.

Thanks

On Wed, Feb 13, 2019 at 4:34 PM Milan Das <md...@interset.com> wrote:

> Hello,
> Is there a way to control number of thread less than cluster member ?
> I have a 5 node NiFI cluster and I want to run 3 instances of the
> processor in distributed mode.
>
> Thanks,
> Milan Das
>
>