You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Darren Govoni <da...@ontrenet.com> on 2015/11/11 15:45:57 UTC
Managing flows
Hi again,
Sorry for the noob questions. I am reading all the online material
as much as possible.
But what hasn't jumped out at me yet is how flows are managed?
Are they saved, loaded, etc? I access my nifi and build a flow. Now I
want to save it and work on another flow.
Lastly, will the flow be running even if I exit the webapp?
thanks for any tips. If I missed something obvious, regrets.
D
Re: Managing flows
Posted by Bryan Bende <bb...@gmail.com>.
In addition to what Mark said, there is also the option of templates [1].
Templates let you export a portion, or all of your flow,
and then import it again later. When you export a template it will not
export any properties that are marked as sensitive properties,
so it is safe to share with others.
Regarding "one flow", you can have as many different logical flows with in
one nifi instance as you want, but it is managed as one flow behind the
scenes.
[1] https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#templates
On Wed, Nov 11, 2015 at 10:00 AM, Darren Govoni <da...@ontrenet.com> wrote:
> Mark,
> Thanks for the tips. Appreciate it.
>
> So when I run nifi on a single server. It is essentially "one flow"?
> If I wanted to have say 2 or 3 active flows, I would (reasonably) have to
> run more instances of nifi with appropriate
> configuration to not conflict. Is that right?
>
> Darren
>
>
> On 11/11/2015 09:54 AM, Mark Petronic wrote:
>
>> Look in your Nifi conf directory. The active flow is there as an aptly
>> named .gz file. Guessing you could just rename that and restart Nifi
>> which would create a blank new one. Build up another flow, then you
>> could repeat the same "copy to new file name" and restore some other
>> one to continue on some previous flow/. I'm pretty new to Nifi, too,
>> so maybe there is another way. Also, you can create point-in-time
>> backups of your from from the "Settings" dialog in the DFM. There is a
>> link that shows up in there to click. It will copy your master flow gz
>> to your conf/archive directory. You can create multiple snapshots of
>> your flow to retain change history. I actually gunzip my backups and
>> commit them to Git for a more formal change history tracking
>> mechanism.
>>
>> Hope that helps.
>>
>> On Wed, Nov 11, 2015 at 9:45 AM, Darren Govoni <da...@ontrenet.com>
>> wrote:
>>
>>> Hi again,
>>> Sorry for the noob questions. I am reading all the online material as
>>> much as possible.
>>> But what hasn't jumped out at me yet is how flows are managed?
>>>
>>> Are they saved, loaded, etc? I access my nifi and build a flow. Now I
>>> want
>>> to save it and work on another flow.
>>> Lastly, will the flow be running even if I exit the webapp?
>>>
>>> thanks for any tips. If I missed something obvious, regrets.
>>>
>>> D
>>>
>>
>
Re: Managing flows
Posted by Darren Govoni <da...@ontrenet.com>.
Excellent. Enjoying the product so far. Works great!
On 11/11/2015 10:22 AM, Mark Petronic wrote:
> You can organize them by creating nested process groups to make it
> more sane to manage
>
> On Wed, Nov 11, 2015 at 10:13 AM, Darren Govoni <da...@ontrenet.com> wrote:
>> Thanks Joe.
>>
>> And it seems all the different flows would be seen on the one canvas, just
>> not connected?
>>
>>
>> On 11/11/2015 10:02 AM, Joe Witt wrote:
>>> Darren,
>>>
>>> A single NiFi instance (on one node or a cluster of 10+) can handle
>>> *many* different flows.
>>>
>>> Thanks
>>> Joe
>>>
>>> On Wed, Nov 11, 2015 at 10:00 AM, Darren Govoni <da...@ontrenet.com>
>>> wrote:
>>>> Mark,
>>>> Thanks for the tips. Appreciate it.
>>>>
>>>> So when I run nifi on a single server. It is essentially "one flow"?
>>>> If I wanted to have say 2 or 3 active flows, I would (reasonably) have to
>>>> run more instances of nifi with appropriate
>>>> configuration to not conflict. Is that right?
>>>>
>>>> Darren
>>>>
>>>>
>>>> On 11/11/2015 09:54 AM, Mark Petronic wrote:
>>>>> Look in your Nifi conf directory. The active flow is there as an aptly
>>>>> named .gz file. Guessing you could just rename that and restart Nifi
>>>>> which would create a blank new one. Build up another flow, then you
>>>>> could repeat the same "copy to new file name" and restore some other
>>>>> one to continue on some previous flow/. I'm pretty new to Nifi, too,
>>>>> so maybe there is another way. Also, you can create point-in-time
>>>>> backups of your from from the "Settings" dialog in the DFM. There is a
>>>>> link that shows up in there to click. It will copy your master flow gz
>>>>> to your conf/archive directory. You can create multiple snapshots of
>>>>> your flow to retain change history. I actually gunzip my backups and
>>>>> commit them to Git for a more formal change history tracking
>>>>> mechanism.
>>>>>
>>>>> Hope that helps.
>>>>>
>>>>> On Wed, Nov 11, 2015 at 9:45 AM, Darren Govoni <da...@ontrenet.com>
>>>>> wrote:
>>>>>> Hi again,
>>>>>> Sorry for the noob questions. I am reading all the online material
>>>>>> as
>>>>>> much as possible.
>>>>>> But what hasn't jumped out at me yet is how flows are managed?
>>>>>>
>>>>>> Are they saved, loaded, etc? I access my nifi and build a flow. Now I
>>>>>> want
>>>>>> to save it and work on another flow.
>>>>>> Lastly, will the flow be running even if I exit the webapp?
>>>>>>
>>>>>> thanks for any tips. If I missed something obvious, regrets.
>>>>>>
>>>>>> D
>>>>
Re: Managing flows
Posted by Mark Petronic <ma...@gmail.com>.
You can organize them by creating nested process groups to make it
more sane to manage
On Wed, Nov 11, 2015 at 10:13 AM, Darren Govoni <da...@ontrenet.com> wrote:
> Thanks Joe.
>
> And it seems all the different flows would be seen on the one canvas, just
> not connected?
>
>
> On 11/11/2015 10:02 AM, Joe Witt wrote:
>>
>> Darren,
>>
>> A single NiFi instance (on one node or a cluster of 10+) can handle
>> *many* different flows.
>>
>> Thanks
>> Joe
>>
>> On Wed, Nov 11, 2015 at 10:00 AM, Darren Govoni <da...@ontrenet.com>
>> wrote:
>>>
>>> Mark,
>>> Thanks for the tips. Appreciate it.
>>>
>>> So when I run nifi on a single server. It is essentially "one flow"?
>>> If I wanted to have say 2 or 3 active flows, I would (reasonably) have to
>>> run more instances of nifi with appropriate
>>> configuration to not conflict. Is that right?
>>>
>>> Darren
>>>
>>>
>>> On 11/11/2015 09:54 AM, Mark Petronic wrote:
>>>>
>>>> Look in your Nifi conf directory. The active flow is there as an aptly
>>>> named .gz file. Guessing you could just rename that and restart Nifi
>>>> which would create a blank new one. Build up another flow, then you
>>>> could repeat the same "copy to new file name" and restore some other
>>>> one to continue on some previous flow/. I'm pretty new to Nifi, too,
>>>> so maybe there is another way. Also, you can create point-in-time
>>>> backups of your from from the "Settings" dialog in the DFM. There is a
>>>> link that shows up in there to click. It will copy your master flow gz
>>>> to your conf/archive directory. You can create multiple snapshots of
>>>> your flow to retain change history. I actually gunzip my backups and
>>>> commit them to Git for a more formal change history tracking
>>>> mechanism.
>>>>
>>>> Hope that helps.
>>>>
>>>> On Wed, Nov 11, 2015 at 9:45 AM, Darren Govoni <da...@ontrenet.com>
>>>> wrote:
>>>>>
>>>>> Hi again,
>>>>> Sorry for the noob questions. I am reading all the online material
>>>>> as
>>>>> much as possible.
>>>>> But what hasn't jumped out at me yet is how flows are managed?
>>>>>
>>>>> Are they saved, loaded, etc? I access my nifi and build a flow. Now I
>>>>> want
>>>>> to save it and work on another flow.
>>>>> Lastly, will the flow be running even if I exit the webapp?
>>>>>
>>>>> thanks for any tips. If I missed something obvious, regrets.
>>>>>
>>>>> D
>>>
>>>
>
Re: Managing flows
Posted by Joe Witt <jo...@gmail.com>.
You got it. Everything we're doing is about showing context so
visualizing different/disconnected flows together is part of that
story. You can abstract them away in different process groups and
organize them in many different ways. You will often find over time
all these seemingly disconnected linear flows have a way of growing
together and forming a true graph.
Thanks
Joe
On Wed, Nov 11, 2015 at 10:13 AM, Darren Govoni <da...@ontrenet.com> wrote:
> Thanks Joe.
>
> And it seems all the different flows would be seen on the one canvas, just
> not connected?
>
>
> On 11/11/2015 10:02 AM, Joe Witt wrote:
>>
>> Darren,
>>
>> A single NiFi instance (on one node or a cluster of 10+) can handle
>> *many* different flows.
>>
>> Thanks
>> Joe
>>
>> On Wed, Nov 11, 2015 at 10:00 AM, Darren Govoni <da...@ontrenet.com>
>> wrote:
>>>
>>> Mark,
>>> Thanks for the tips. Appreciate it.
>>>
>>> So when I run nifi on a single server. It is essentially "one flow"?
>>> If I wanted to have say 2 or 3 active flows, I would (reasonably) have to
>>> run more instances of nifi with appropriate
>>> configuration to not conflict. Is that right?
>>>
>>> Darren
>>>
>>>
>>> On 11/11/2015 09:54 AM, Mark Petronic wrote:
>>>>
>>>> Look in your Nifi conf directory. The active flow is there as an aptly
>>>> named .gz file. Guessing you could just rename that and restart Nifi
>>>> which would create a blank new one. Build up another flow, then you
>>>> could repeat the same "copy to new file name" and restore some other
>>>> one to continue on some previous flow/. I'm pretty new to Nifi, too,
>>>> so maybe there is another way. Also, you can create point-in-time
>>>> backups of your from from the "Settings" dialog in the DFM. There is a
>>>> link that shows up in there to click. It will copy your master flow gz
>>>> to your conf/archive directory. You can create multiple snapshots of
>>>> your flow to retain change history. I actually gunzip my backups and
>>>> commit them to Git for a more formal change history tracking
>>>> mechanism.
>>>>
>>>> Hope that helps.
>>>>
>>>> On Wed, Nov 11, 2015 at 9:45 AM, Darren Govoni <da...@ontrenet.com>
>>>> wrote:
>>>>>
>>>>> Hi again,
>>>>> Sorry for the noob questions. I am reading all the online material
>>>>> as
>>>>> much as possible.
>>>>> But what hasn't jumped out at me yet is how flows are managed?
>>>>>
>>>>> Are they saved, loaded, etc? I access my nifi and build a flow. Now I
>>>>> want
>>>>> to save it and work on another flow.
>>>>> Lastly, will the flow be running even if I exit the webapp?
>>>>>
>>>>> thanks for any tips. If I missed something obvious, regrets.
>>>>>
>>>>> D
>>>
>>>
>
Re: Managing flows
Posted by Darren Govoni <da...@ontrenet.com>.
Thanks Joe.
And it seems all the different flows would be seen on the one canvas,
just not connected?
On 11/11/2015 10:02 AM, Joe Witt wrote:
> Darren,
>
> A single NiFi instance (on one node or a cluster of 10+) can handle
> *many* different flows.
>
> Thanks
> Joe
>
> On Wed, Nov 11, 2015 at 10:00 AM, Darren Govoni <da...@ontrenet.com> wrote:
>> Mark,
>> Thanks for the tips. Appreciate it.
>>
>> So when I run nifi on a single server. It is essentially "one flow"?
>> If I wanted to have say 2 or 3 active flows, I would (reasonably) have to
>> run more instances of nifi with appropriate
>> configuration to not conflict. Is that right?
>>
>> Darren
>>
>>
>> On 11/11/2015 09:54 AM, Mark Petronic wrote:
>>> Look in your Nifi conf directory. The active flow is there as an aptly
>>> named .gz file. Guessing you could just rename that and restart Nifi
>>> which would create a blank new one. Build up another flow, then you
>>> could repeat the same "copy to new file name" and restore some other
>>> one to continue on some previous flow/. I'm pretty new to Nifi, too,
>>> so maybe there is another way. Also, you can create point-in-time
>>> backups of your from from the "Settings" dialog in the DFM. There is a
>>> link that shows up in there to click. It will copy your master flow gz
>>> to your conf/archive directory. You can create multiple snapshots of
>>> your flow to retain change history. I actually gunzip my backups and
>>> commit them to Git for a more formal change history tracking
>>> mechanism.
>>>
>>> Hope that helps.
>>>
>>> On Wed, Nov 11, 2015 at 9:45 AM, Darren Govoni <da...@ontrenet.com>
>>> wrote:
>>>> Hi again,
>>>> Sorry for the noob questions. I am reading all the online material as
>>>> much as possible.
>>>> But what hasn't jumped out at me yet is how flows are managed?
>>>>
>>>> Are they saved, loaded, etc? I access my nifi and build a flow. Now I
>>>> want
>>>> to save it and work on another flow.
>>>> Lastly, will the flow be running even if I exit the webapp?
>>>>
>>>> thanks for any tips. If I missed something obvious, regrets.
>>>>
>>>> D
>>
Re: Managing flows
Posted by Joe Witt <jo...@gmail.com>.
Darren,
A single NiFi instance (on one node or a cluster of 10+) can handle
*many* different flows.
Thanks
Joe
On Wed, Nov 11, 2015 at 10:00 AM, Darren Govoni <da...@ontrenet.com> wrote:
> Mark,
> Thanks for the tips. Appreciate it.
>
> So when I run nifi on a single server. It is essentially "one flow"?
> If I wanted to have say 2 or 3 active flows, I would (reasonably) have to
> run more instances of nifi with appropriate
> configuration to not conflict. Is that right?
>
> Darren
>
>
> On 11/11/2015 09:54 AM, Mark Petronic wrote:
>>
>> Look in your Nifi conf directory. The active flow is there as an aptly
>> named .gz file. Guessing you could just rename that and restart Nifi
>> which would create a blank new one. Build up another flow, then you
>> could repeat the same "copy to new file name" and restore some other
>> one to continue on some previous flow/. I'm pretty new to Nifi, too,
>> so maybe there is another way. Also, you can create point-in-time
>> backups of your from from the "Settings" dialog in the DFM. There is a
>> link that shows up in there to click. It will copy your master flow gz
>> to your conf/archive directory. You can create multiple snapshots of
>> your flow to retain change history. I actually gunzip my backups and
>> commit them to Git for a more formal change history tracking
>> mechanism.
>>
>> Hope that helps.
>>
>> On Wed, Nov 11, 2015 at 9:45 AM, Darren Govoni <da...@ontrenet.com>
>> wrote:
>>>
>>> Hi again,
>>> Sorry for the noob questions. I am reading all the online material as
>>> much as possible.
>>> But what hasn't jumped out at me yet is how flows are managed?
>>>
>>> Are they saved, loaded, etc? I access my nifi and build a flow. Now I
>>> want
>>> to save it and work on another flow.
>>> Lastly, will the flow be running even if I exit the webapp?
>>>
>>> thanks for any tips. If I missed something obvious, regrets.
>>>
>>> D
>
>
Re: Managing flows
Posted by Darren Govoni <da...@ontrenet.com>.
Mark,
Thanks for the tips. Appreciate it.
So when I run nifi on a single server. It is essentially "one flow"?
If I wanted to have say 2 or 3 active flows, I would (reasonably) have
to run more instances of nifi with appropriate
configuration to not conflict. Is that right?
Darren
On 11/11/2015 09:54 AM, Mark Petronic wrote:
> Look in your Nifi conf directory. The active flow is there as an aptly
> named .gz file. Guessing you could just rename that and restart Nifi
> which would create a blank new one. Build up another flow, then you
> could repeat the same "copy to new file name" and restore some other
> one to continue on some previous flow/. I'm pretty new to Nifi, too,
> so maybe there is another way. Also, you can create point-in-time
> backups of your from from the "Settings" dialog in the DFM. There is a
> link that shows up in there to click. It will copy your master flow gz
> to your conf/archive directory. You can create multiple snapshots of
> your flow to retain change history. I actually gunzip my backups and
> commit them to Git for a more formal change history tracking
> mechanism.
>
> Hope that helps.
>
> On Wed, Nov 11, 2015 at 9:45 AM, Darren Govoni <da...@ontrenet.com> wrote:
>> Hi again,
>> Sorry for the noob questions. I am reading all the online material as
>> much as possible.
>> But what hasn't jumped out at me yet is how flows are managed?
>>
>> Are they saved, loaded, etc? I access my nifi and build a flow. Now I want
>> to save it and work on another flow.
>> Lastly, will the flow be running even if I exit the webapp?
>>
>> thanks for any tips. If I missed something obvious, regrets.
>>
>> D
Re: Managing flows
Posted by Mark Petronic <ma...@gmail.com>.
Regarding Nifi always running. Yes, it stays running. It is
effectively a service with a REST and Web UI. Closing the web UI does
not have any effect on the running processors - just your visibility
to them.
On Wed, Nov 11, 2015 at 9:54 AM, Mark Petronic <ma...@gmail.com> wrote:
> Look in your Nifi conf directory. The active flow is there as an aptly
> named .gz file. Guessing you could just rename that and restart Nifi
> which would create a blank new one. Build up another flow, then you
> could repeat the same "copy to new file name" and restore some other
> one to continue on some previous flow/. I'm pretty new to Nifi, too,
> so maybe there is another way. Also, you can create point-in-time
> backups of your from from the "Settings" dialog in the DFM. There is a
> link that shows up in there to click. It will copy your master flow gz
> to your conf/archive directory. You can create multiple snapshots of
> your flow to retain change history. I actually gunzip my backups and
> commit them to Git for a more formal change history tracking
> mechanism.
>
> Hope that helps.
>
> On Wed, Nov 11, 2015 at 9:45 AM, Darren Govoni <da...@ontrenet.com> wrote:
>> Hi again,
>> Sorry for the noob questions. I am reading all the online material as
>> much as possible.
>> But what hasn't jumped out at me yet is how flows are managed?
>>
>> Are they saved, loaded, etc? I access my nifi and build a flow. Now I want
>> to save it and work on another flow.
>> Lastly, will the flow be running even if I exit the webapp?
>>
>> thanks for any tips. If I missed something obvious, regrets.
>>
>> D
Re: Managing flows
Posted by Mark Petronic <ma...@gmail.com>.
Look in your Nifi conf directory. The active flow is there as an aptly
named .gz file. Guessing you could just rename that and restart Nifi
which would create a blank new one. Build up another flow, then you
could repeat the same "copy to new file name" and restore some other
one to continue on some previous flow/. I'm pretty new to Nifi, too,
so maybe there is another way. Also, you can create point-in-time
backups of your from from the "Settings" dialog in the DFM. There is a
link that shows up in there to click. It will copy your master flow gz
to your conf/archive directory. You can create multiple snapshots of
your flow to retain change history. I actually gunzip my backups and
commit them to Git for a more formal change history tracking
mechanism.
Hope that helps.
On Wed, Nov 11, 2015 at 9:45 AM, Darren Govoni <da...@ontrenet.com> wrote:
> Hi again,
> Sorry for the noob questions. I am reading all the online material as
> much as possible.
> But what hasn't jumped out at me yet is how flows are managed?
>
> Are they saved, loaded, etc? I access my nifi and build a flow. Now I want
> to save it and work on another flow.
> Lastly, will the flow be running even if I exit the webapp?
>
> thanks for any tips. If I missed something obvious, regrets.
>
> D