Re: multiple flows and cluster

[De Mezzo Benoit <b....@systel-sa.com>]

Thanks! Sorry for the delay!

What do you mean by creating microclusters? Is this a Nifi
functionnality (ie. hierarchical cluster of clusters) ? Or is it a
cluster design that must be created thoroughly?

Benoit.



Le 26/11/2016 � 02:13, Andy LoPresto a �crit :
> Benoit,
>
> Every NiFi node can run many disparate flows, and you can separate
> them into distinct process groups for logical divisions. 
>
> You can trigger these flows in a few ways. The first, you could set
> the first processor in the flow to be triggered by event receipt
> rather than schedule driven so that it doesn't run until a unit of
> data enters the flow. The other thing you could do is use the REST API
> to enable/disable individual processors/flows and trigger that API
> invocation via cron or some other scheduling system external to NiFi.
> Finally, you could use an ExecuteScript processor to perform more
> complex custom logic to determine which flows should be running at any
> given time (and then use the REST API to enable/disable them).
>
> If you really find that these changes cannot be synchronized across
> the single flow residing on a NiFi cluster, you could segment and
> subdivide your cluster into microclusters where each shares a partial
> flow (grouped by performance considerations or logical concepts). 
>
> Andy LoPresto
> alopresto@apache.org <ma...@apache.org>
> alopresto.apache@gmail.com <ma...@gmail.com>
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Nov 25, 2016, at 04:13, De Mezzo Benoit <b.demezzo@systel-sa.com
> <ma...@systel-sa.com>> wrote:
>
>> Hi all,
>>
>> Here is my problem: I need to run many small etl flow (say a thousand)
>> at flow-dedicated times. As theses flows may be added, removed, changed
>> all the time, they can not be easily merged within a big-meta flow to be
>> run in the Nifi cluster.
>>
>> I would like to know if there a way to run multiple flows within a nifi
>> cluster?
>>
>> Or may be there is a way to generate, per flow, a small standalone
>> binary (jar?) to run only once a flow against an embedded Nifi engine
>> (each jar will be managed by a task scheduling cluster) ?
>>
>> Thanks !
>>
>> Benoit.
>>

Re: multiple flows and cluster

[Andy LoPresto <al...@apache.org>]

Benoit,

“Microclusters” is not a NiFi feature, it was just me saying you could logically separate flows which do not rely on each other into disparate clusters. So, for example, rather than running 1 12-node cluster, you could run 4 3-node clusters.

Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Dec 5, 2016, at 12:55 AM, De Mezzo Benoit <b....@systel-sa.com> wrote:
> 
> Thanks! Sorry for the delay!
> 
> What do you mean by creating microclusters? Is this a Nifi functionnality (ie. hierarchical cluster of clusters) ? Or is it a cluster design that must be created thoroughly?
> 
> Benoit.
> 
> 
> 
> Le 26/11/2016 à 02:13, Andy LoPresto a écrit :
>> Benoit,
>> 
>> Every NiFi node can run many disparate flows, and you can separate them into distinct process groups for logical divisions.
>> 
>> You can trigger these flows in a few ways. The first, you could set the first processor in the flow to be triggered by event receipt rather than schedule driven so that it doesn't run until a unit of data enters the flow. The other thing you could do is use the REST API to enable/disable individual processors/flows and trigger that API invocation via cron or some other scheduling system external to NiFi. Finally, you could use an ExecuteScript processor to perform more complex custom logic to determine which flows should be running at any given time (and then use the REST API to enable/disable them).
>> 
>> If you really find that these changes cannot be synchronized across the single flow residing on a NiFi cluster, you could segment and subdivide your cluster into microclusters where each shares a partial flow (grouped by performance considerations or logical concepts).
>> 
>> Andy LoPresto
>> alopresto@apache.org <ma...@apache.org>
>> alopresto.apache@gmail.com <ma...@gmail.com>
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>> 
>> On Nov 25, 2016, at 04:13, De Mezzo Benoit <b.demezzo@systel-sa.com <ma...@systel-sa.com>> wrote:
>> 
>>> Hi all,
>>> 
>>> Here is my problem: I need to run many small etl flow (say a thousand)
>>> at flow-dedicated times. As theses flows may be added, removed, changed
>>> all the time, they can not be easily merged within a big-meta flow to be
>>> run in the Nifi cluster.
>>> 
>>> I would like to know if there a way to run multiple flows within a nifi
>>> cluster?
>>> 
>>> Or may be there is a way to generate, per flow, a small standalone
>>> binary (jar?) to run only once a flow against an embedded Nifi engine
>>> (each jar will be managed by a task scheduling cluster) ?
>>> 
>>> Thanks !
>>> 
>>> Benoit.
>>> 
>