You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Manish Gupta 8 <mg...@sapient.com> on 2016/09/13 20:30:48 UTC
Best Practice for backing up NiFi Flows
Hello Everyone,
Is there a best practice for keeping a backup of all the data flows we are developing in NiFi?
Currently we take a copy of flow.xml.gz every hour and keep it in backup folder (also in our source control). Also, we keep a copy of all Config files in source control.
* We are assuming that using flow.xml.gz and Config files, we will be able to restore the NiFi in case of any failure or if someone makes some mistake. Is this assumption correct? Is there a better way to deal with this?
* When we move to production (or some other environment), will it be as simple as dropping flow.xml.gz in a new NiFi installation on NCM along with making some environment related changes? Or, should we use templates on Dev, and import on Prod?
Thanks,
Manish
RE: Best Practice for backing up NiFi Flows
Posted by Manish Gupta 8 <mg...@sapient.com>.
Thanks James. Using Templates makes more sense. So what we are going to do is:
· Take backup of Conf Folder in source control (scheduled).
· Create Template for the top level Processor Group(s) periodically (will try to automate it using NiFi REST API for Processor Group).
Regards,
Manish
Cell: +1 646-744-7606<tel:%2B1%20646-744-7606>
Email: mgupta50@sapient.com<ma...@sapient.com>
From: James Wing [mailto:jvwing@gmail.com]
Sent: Wednesday, September 14, 2016 3:02 PM
To: users@nifi.apache.org
Subject: Re: Best Practice for backing up NiFi Flows
Manish, you are absolutely right to back up your flow.xml.gz and conf files. But I would carefully distinguish between using these backups to recreate an equivalent new NiFi, versus attempting to reset the state of your existing NiFi. The difference is the live data in your flow, in the provenance repository, in state variables, etc. Restoring a flow definition that no longer matches your content and provenance data may have unexpected results for you, and for systems connecting with NiFi. NiFi does try hard to handle these changes smoothly, but it isn't a magic time machine.
Deploying flow.xml.gz can work, especially when deployed with conf files that reference IDs in the flow (like authorizations.xml), or the nifi.sensitive.props.key setting, etc. But if you overwrite a running flow, you still have the data migration problem.
Templates are the current recommended best practice for deployment. As I understand it, templates provide:
1.) Concise packaging for deployment
2.) Separation between site-specific configuration like authorizations from the flow logic
3.) Workflow that allows, encourages, forces the administrator to address migration from the existing flow to incorporate the new template
Personally, I think it centers on acceptance or rejection of the command-and-control model, which is controversial and different from most other systems. Templates fit within command-and-control, overwriting flow.xml.gz suggests a different model.
I know there are many other opinions on this.
Thanks,
James
On Tue, Sep 13, 2016 at 1:30 PM, Manish Gupta 8 <mg...@sapient.com>> wrote:
Hello Everyone,
Is there a best practice for keeping a backup of all the data flows we are developing in NiFi?
Currently we take a copy of flow.xml.gz every hour and keep it in backup folder (also in our source control). Also, we keep a copy of all Config files in source control.
• We are assuming that using flow.xml.gz and Config files, we will be able to restore the NiFi in case of any failure or if someone makes some mistake. Is this assumption correct? Is there a better way to deal with this?
• When we move to production (or some other environment), will it be as simple as dropping flow.xml.gz in a new NiFi installation on NCM along with making some environment related changes? Or, should we use templates on Dev, and import on Prod?
Thanks,
Manish
Re: Best Practice for backing up NiFi Flows
Posted by Tijo Thomas <ti...@yahoo.in>.
Hi Joe,
Scenarios we have encountered is1. If a node goes down , we need to restart the cluster. A graceful degradation of service is not possible . I would be better if Nifi cluster can still run with existing nodes.
2. Zookeeper.properties : In our cluster we configured to scale zookeeper nodes when ever the load in the zookeeper cluster increases. We want to add this zookeeper ip to zookeeper.properties.
3. it will be great if Nifi version upgrade can be done with out shutting down the cluster. I have noticed some thing like rolling upgrades which is available in Hbase and some other components also .
Out of this 1st one is more crucial . We can implement auto scaling (specifically scale down) if Nifi node can run even if a node is removed.
Thanks & RegardsTijo Thomas
On Thursday, 15 September 2016 8:16 AM, Joe Witt <jo...@gmail.com> wrote:
Tijo
Most conf file related changes are intentionally application startup
based. Which properties are you having to change that you'd like to
see honored without a restart?
Thanks
Joe
On Wed, Sep 14, 2016 at 10:44 PM, Tijo Thomas <ti...@gmail.com> wrote:
> Hi
>
> Our nifi cluster need to be up continuously as our user is over different
> time zones.
> Most of the times we often need to restart the cluster when ever there is
> any changes in conf.
>
> Is there any way to load the conf automatically when ever there is any
> change.
>
> Tijo
>
>
> On 15-Sep-2016 4:36 am, "James Wing" <jv...@gmail.com> wrote:
>
> Mark,
>
> Loading templates from the file system on startup is a new feature in 1.0.0,
> right? I've been using the same API deployment Dan describes in 0.x.
>
> Thanks,
>
> James
>
> On Wed, Sep 14, 2016 at 12:22 PM, Mark Payne <ma...@hotmail.com> wrote:
>>
>> Dan,
>>
>> Yes, you should be able to pre-deploy templates. When NiFi starts up, it
>> looks in the conf/templates
>> directory (by default - this directory can be changed in the
>> nifi.properties file). It looks for any file that
>> has a suffix of ".template" or ".xml" so you need to be sure that you are
>> naming the files properly. Also, if you
>> are starting a cluster, you need to ensure that all nodes in the cluster
>> have the templates or they will
>> not show up.
>>
>> Thanks
>> -Mark
>>
>>
>> On Sep 14, 2016, at 3:15 PM, Dan Morris <dg...@gmail.com> wrote:
>>
>> James,
>>
>> Related to this question, is there a way to pre-deploy templates to the
>> template directory and have nifi recognize them? I recently tried this and
>> nifi would not recognize the template until after I manually uploaded it
>> through the UI.
>>
>> I did this with nifi 0.7.0.
>>
>> I'd like to be able to use tools like salt or puppet to pre-position
>> templates.
>>
>> Thanks,
>>
>> Dan
>> M: 443-992-2848
>> GV: 410-861-0206
>>
>> On Sep 14, 2016, at 3:02 PM, James Wing <jv...@gmail.com> wrote:
>>
>> Manish, you are absolutely right to back up your flow.xml.gz and conf
>> files. But I would carefully distinguish between using these backups to
>> recreate an equivalent new NiFi, versus attempting to reset the state of
>> your existing NiFi. The difference is the live data in your flow, in the
>> provenance repository, in state variables, etc. Restoring a flow definition
>> that no longer matches your content and provenance data may have unexpected
>> results for you, and for systems connecting with NiFi. NiFi does try hard
>> to handle these changes smoothly, but it isn't a magic time machine.
>>
>> Deploying flow.xml.gz can work, especially when deployed with conf files
>> that reference IDs in the flow (like authorizations.xml), or the
>> nifi.sensitive.props.key setting, etc. But if you overwrite a running flow,
>> you still have the data migration problem.
>>
>> Templates are the current recommended best practice for deployment. As I
>> understand it, templates provide:
>>
>> 1.) Concise packaging for deployment
>> 2.) Separation between site-specific configuration like authorizations
>> from the flow logic
>> 3.) Workflow that allows, encourages, forces the administrator to address
>> migration from the existing flow to incorporate the new template
>>
>> Personally, I think it centers on acceptance or rejection of the
>> command-and-control model, which is controversial and different from most
>> other systems. Templates fit within command-and-control, overwriting
>> flow.xml.gz suggests a different model.
>>
>> I know there are many other opinions on this.
>>
>> Thanks,
>>
>> James
>>
>> On Tue, Sep 13, 2016 at 1:30 PM, Manish Gupta 8 <mg...@sapient.com>
>> wrote:
>>>
>>> Hello Everyone,
>>>
>>>
>>>
>>> Is there a best practice for keeping a backup of all the data flows we
>>> are developing in NiFi?
>>>
>>>
>>>
>>> Currently we take a copy of flow.xml.gz every hour and keep it in backup
>>> folder (also in our source control). Also, we keep a copy of all Config
>>> files in source control.
>>>
>>>
>>>
>>> · We are assuming that using flow.xml.gz and Config files, we
>>> will be able to restore the NiFi in case of any failure or if someone makes
>>> some mistake. Is this assumption correct? Is there a better way to deal with
>>> this?
>>>
>>> · When we move to production (or some other environment), will it
>>> be as simple as dropping flow.xml.gz in a new NiFi installation on NCM along
>>> with making some environment related changes? Or, should we use templates on
>>> Dev, and import on Prod?
>>>
>>>
>>>
>>> Thanks,
>>>
>>> Manish
>>>
>>>
>>
>>
>>
>
>
Re: Best Practice for backing up NiFi Flows
Posted by Joe Witt <jo...@gmail.com>.
Tijo
Most conf file related changes are intentionally application startup
based. Which properties are you having to change that you'd like to
see honored without a restart?
Thanks
Joe
On Wed, Sep 14, 2016 at 10:44 PM, Tijo Thomas <ti...@gmail.com> wrote:
> Hi
>
> Our nifi cluster need to be up continuously as our user is over different
> time zones.
> Most of the times we often need to restart the cluster when ever there is
> any changes in conf.
>
> Is there any way to load the conf automatically when ever there is any
> change.
>
> Tijo
>
>
> On 15-Sep-2016 4:36 am, "James Wing" <jv...@gmail.com> wrote:
>
> Mark,
>
> Loading templates from the file system on startup is a new feature in 1.0.0,
> right? I've been using the same API deployment Dan describes in 0.x.
>
> Thanks,
>
> James
>
> On Wed, Sep 14, 2016 at 12:22 PM, Mark Payne <ma...@hotmail.com> wrote:
>>
>> Dan,
>>
>> Yes, you should be able to pre-deploy templates. When NiFi starts up, it
>> looks in the conf/templates
>> directory (by default - this directory can be changed in the
>> nifi.properties file). It looks for any file that
>> has a suffix of ".template" or ".xml" so you need to be sure that you are
>> naming the files properly. Also, if you
>> are starting a cluster, you need to ensure that all nodes in the cluster
>> have the templates or they will
>> not show up.
>>
>> Thanks
>> -Mark
>>
>>
>> On Sep 14, 2016, at 3:15 PM, Dan Morris <dg...@gmail.com> wrote:
>>
>> James,
>>
>> Related to this question, is there a way to pre-deploy templates to the
>> template directory and have nifi recognize them? I recently tried this and
>> nifi would not recognize the template until after I manually uploaded it
>> through the UI.
>>
>> I did this with nifi 0.7.0.
>>
>> I'd like to be able to use tools like salt or puppet to pre-position
>> templates.
>>
>> Thanks,
>>
>> Dan
>> M: 443-992-2848
>> GV: 410-861-0206
>>
>> On Sep 14, 2016, at 3:02 PM, James Wing <jv...@gmail.com> wrote:
>>
>> Manish, you are absolutely right to back up your flow.xml.gz and conf
>> files. But I would carefully distinguish between using these backups to
>> recreate an equivalent new NiFi, versus attempting to reset the state of
>> your existing NiFi. The difference is the live data in your flow, in the
>> provenance repository, in state variables, etc. Restoring a flow definition
>> that no longer matches your content and provenance data may have unexpected
>> results for you, and for systems connecting with NiFi. NiFi does try hard
>> to handle these changes smoothly, but it isn't a magic time machine.
>>
>> Deploying flow.xml.gz can work, especially when deployed with conf files
>> that reference IDs in the flow (like authorizations.xml), or the
>> nifi.sensitive.props.key setting, etc. But if you overwrite a running flow,
>> you still have the data migration problem.
>>
>> Templates are the current recommended best practice for deployment. As I
>> understand it, templates provide:
>>
>> 1.) Concise packaging for deployment
>> 2.) Separation between site-specific configuration like authorizations
>> from the flow logic
>> 3.) Workflow that allows, encourages, forces the administrator to address
>> migration from the existing flow to incorporate the new template
>>
>> Personally, I think it centers on acceptance or rejection of the
>> command-and-control model, which is controversial and different from most
>> other systems. Templates fit within command-and-control, overwriting
>> flow.xml.gz suggests a different model.
>>
>> I know there are many other opinions on this.
>>
>> Thanks,
>>
>> James
>>
>> On Tue, Sep 13, 2016 at 1:30 PM, Manish Gupta 8 <mg...@sapient.com>
>> wrote:
>>>
>>> Hello Everyone,
>>>
>>>
>>>
>>> Is there a best practice for keeping a backup of all the data flows we
>>> are developing in NiFi?
>>>
>>>
>>>
>>> Currently we take a copy of flow.xml.gz every hour and keep it in backup
>>> folder (also in our source control). Also, we keep a copy of all Config
>>> files in source control.
>>>
>>>
>>>
>>> · We are assuming that using flow.xml.gz and Config files, we
>>> will be able to restore the NiFi in case of any failure or if someone makes
>>> some mistake. Is this assumption correct? Is there a better way to deal with
>>> this?
>>>
>>> · When we move to production (or some other environment), will it
>>> be as simple as dropping flow.xml.gz in a new NiFi installation on NCM along
>>> with making some environment related changes? Or, should we use templates on
>>> Dev, and import on Prod?
>>>
>>>
>>>
>>> Thanks,
>>>
>>> Manish
>>>
>>>
>>
>>
>>
>
>
Re: Best Practice for backing up NiFi Flows
Posted by Tijo Thomas <ti...@gmail.com>.
Hi
Our nifi cluster need to be up continuously as our user is over different
time zones.
Most of the times we often need to restart the cluster when ever there is
any changes in conf.
Is there any way to load the conf automatically when ever there is any
change.
Tijo
On 15-Sep-2016 4:36 am, "James Wing" <jv...@gmail.com> wrote:
Mark,
Loading templates from the file system on startup is a new feature in
1.0.0, right? I've been using the same API deployment Dan describes in 0.x.
Thanks,
James
On Wed, Sep 14, 2016 at 12:22 PM, Mark Payne <ma...@hotmail.com> wrote:
> Dan,
>
> Yes, you should be able to pre-deploy templates. When NiFi starts up, it
> looks in the conf/templates
> directory (by default - this directory can be changed in the
> nifi.properties file). It looks for any file that
> has a suffix of ".template" or ".xml" so you need to be sure that you are
> naming the files properly. Also, if you
> are starting a cluster, you need to ensure that all nodes in the cluster
> have the templates or they will
> not show up.
>
> Thanks
> -Mark
>
>
> On Sep 14, 2016, at 3:15 PM, Dan Morris <dg...@gmail.com> wrote:
>
> James,
>
> Related to this question, is there a way to pre-deploy templates to the
> template directory and have nifi recognize them? I recently tried this and
> nifi would not recognize the template until after I manually uploaded it
> through the UI.
>
> I did this with nifi 0.7.0.
>
> I'd like to be able to use tools like salt or puppet to pre-position
> templates.
>
> Thanks,
>
> Dan
> M: 443-992-2848
> GV: 410-861-0206
>
> On Sep 14, 2016, at 3:02 PM, James Wing <jv...@gmail.com> wrote:
>
> Manish, you are absolutely right to back up your flow.xml.gz and conf
> files. But I would carefully distinguish between using these backups to
> recreate an equivalent new NiFi, versus attempting to reset the state of
> your existing NiFi. The difference is the live data in your flow, in the
> provenance repository, in state variables, etc. Restoring a flow
> definition that no longer matches your content and provenance data may have
> unexpected results for you, and for systems connecting with NiFi. NiFi
> does try hard to handle these changes smoothly, but it isn't a magic time
> machine.
>
> Deploying flow.xml.gz can work, especially when deployed with conf files
> that reference IDs in the flow (like authorizations.xml), or the
> nifi.sensitive.props.key setting, etc. But if you overwrite a running
> flow, you still have the data migration problem.
>
> Templates are the current recommended best practice for deployment. As I
> understand it, templates provide:
>
> 1.) Concise packaging for deployment
> 2.) Separation between site-specific configuration like authorizations
> from the flow logic
> 3.) Workflow that allows, encourages, forces the administrator to address
> migration from the existing flow to incorporate the new template
>
> Personally, I think it centers on acceptance or rejection of the
> command-and-control model, which is controversial and different from most
> other systems. Templates fit within command-and-control, overwriting
> flow.xml.gz suggests a different model.
>
> I know there are many other opinions on this.
>
> Thanks,
>
> James
>
> On Tue, Sep 13, 2016 at 1:30 PM, Manish Gupta 8 <mg...@sapient.com>
> wrote:
>
>> Hello Everyone,
>>
>>
>>
>> Is there a best practice for keeping a backup of all the data flows we
>> are developing in NiFi?
>>
>>
>>
>> Currently we take a copy of flow.xml.gz every hour and keep it in backup
>> folder (also in our source control). Also, we keep a copy of all Config
>> files in source control.
>>
>>
>>
>> · We are assuming that using flow.xml.gz and Config files, we
>> will be able to restore the NiFi in case of any failure or if someone makes
>> some mistake. Is this assumption correct? Is there a better way to deal
>> with this?
>>
>> · When we move to production (or some other environment), will
>> it be as simple as dropping flow.xml.gz in a new NiFi installation on NCM
>> along with making some environment related changes? Or, should we use
>> templates on Dev, and import on Prod?
>>
>>
>>
>> Thanks,
>>
>> Manish
>>
>>
>>
>
>
>
Re: Best Practice for backing up NiFi Flows
Posted by James Wing <jv...@gmail.com>.
Mark,
Loading templates from the file system on startup is a new feature in
1.0.0, right? I've been using the same API deployment Dan describes in 0.x.
Thanks,
James
On Wed, Sep 14, 2016 at 12:22 PM, Mark Payne <ma...@hotmail.com> wrote:
> Dan,
>
> Yes, you should be able to pre-deploy templates. When NiFi starts up, it
> looks in the conf/templates
> directory (by default - this directory can be changed in the
> nifi.properties file). It looks for any file that
> has a suffix of ".template" or ".xml" so you need to be sure that you are
> naming the files properly. Also, if you
> are starting a cluster, you need to ensure that all nodes in the cluster
> have the templates or they will
> not show up.
>
> Thanks
> -Mark
>
>
> On Sep 14, 2016, at 3:15 PM, Dan Morris <dg...@gmail.com> wrote:
>
> James,
>
> Related to this question, is there a way to pre-deploy templates to the
> template directory and have nifi recognize them? I recently tried this and
> nifi would not recognize the template until after I manually uploaded it
> through the UI.
>
> I did this with nifi 0.7.0.
>
> I'd like to be able to use tools like salt or puppet to pre-position
> templates.
>
> Thanks,
>
> Dan
> M: 443-992-2848
> GV: 410-861-0206
>
> On Sep 14, 2016, at 3:02 PM, James Wing <jv...@gmail.com> wrote:
>
> Manish, you are absolutely right to back up your flow.xml.gz and conf
> files. But I would carefully distinguish between using these backups to
> recreate an equivalent new NiFi, versus attempting to reset the state of
> your existing NiFi. The difference is the live data in your flow, in the
> provenance repository, in state variables, etc. Restoring a flow
> definition that no longer matches your content and provenance data may have
> unexpected results for you, and for systems connecting with NiFi. NiFi
> does try hard to handle these changes smoothly, but it isn't a magic time
> machine.
>
> Deploying flow.xml.gz can work, especially when deployed with conf files
> that reference IDs in the flow (like authorizations.xml), or the
> nifi.sensitive.props.key setting, etc. But if you overwrite a running
> flow, you still have the data migration problem.
>
> Templates are the current recommended best practice for deployment. As I
> understand it, templates provide:
>
> 1.) Concise packaging for deployment
> 2.) Separation between site-specific configuration like authorizations
> from the flow logic
> 3.) Workflow that allows, encourages, forces the administrator to address
> migration from the existing flow to incorporate the new template
>
> Personally, I think it centers on acceptance or rejection of the
> command-and-control model, which is controversial and different from most
> other systems. Templates fit within command-and-control, overwriting
> flow.xml.gz suggests a different model.
>
> I know there are many other opinions on this.
>
> Thanks,
>
> James
>
> On Tue, Sep 13, 2016 at 1:30 PM, Manish Gupta 8 <mg...@sapient.com>
> wrote:
>
>> Hello Everyone,
>>
>>
>>
>> Is there a best practice for keeping a backup of all the data flows we
>> are developing in NiFi?
>>
>>
>>
>> Currently we take a copy of flow.xml.gz every hour and keep it in backup
>> folder (also in our source control). Also, we keep a copy of all Config
>> files in source control.
>>
>>
>>
>> · We are assuming that using flow.xml.gz and Config files, we
>> will be able to restore the NiFi in case of any failure or if someone makes
>> some mistake. Is this assumption correct? Is there a better way to deal
>> with this?
>>
>> · When we move to production (or some other environment), will
>> it be as simple as dropping flow.xml.gz in a new NiFi installation on NCM
>> along with making some environment related changes? Or, should we use
>> templates on Dev, and import on Prod?
>>
>>
>>
>> Thanks,
>>
>> Manish
>>
>>
>>
>
>
>
Re: Best Practice for backing up NiFi Flows
Posted by Mark Payne <ma...@hotmail.com>.
Dan,
Yes, you should be able to pre-deploy templates. When NiFi starts up, it looks in the conf/templates
directory (by default - this directory can be changed in the nifi.properties file). It looks for any file that
has a suffix of ".template" or ".xml" so you need to be sure that you are naming the files properly. Also, if you
are starting a cluster, you need to ensure that all nodes in the cluster have the templates or they will
not show up.
Thanks
-Mark
> On Sep 14, 2016, at 3:15 PM, Dan Morris <dg...@gmail.com> wrote:
>
> James,
>
> Related to this question, is there a way to pre-deploy templates to the template directory and have nifi recognize them? I recently tried this and nifi would not recognize the template until after I manually uploaded it through the UI.
>
> I did this with nifi 0.7.0.
>
> I'd like to be able to use tools like salt or puppet to pre-position templates.
>
> Thanks,
>
> Dan
> M: 443-992-2848
> GV: 410-861-0206
>
> On Sep 14, 2016, at 3:02 PM, James Wing <jvwing@gmail.com <ma...@gmail.com>> wrote:
>
>> Manish, you are absolutely right to back up your flow.xml.gz and conf files. But I would carefully distinguish between using these backups to recreate an equivalent new NiFi, versus attempting to reset the state of your existing NiFi. The difference is the live data in your flow, in the provenance repository, in state variables, etc. Restoring a flow definition that no longer matches your content and provenance data may have unexpected results for you, and for systems connecting with NiFi. NiFi does try hard to handle these changes smoothly, but it isn't a magic time machine.
>>
>> Deploying flow.xml.gz can work, especially when deployed with conf files that reference IDs in the flow (like authorizations.xml), or the nifi.sensitive.props.key setting, etc. But if you overwrite a running flow, you still have the data migration problem.
>>
>> Templates are the current recommended best practice for deployment. As I understand it, templates provide:
>>
>> 1.) Concise packaging for deployment
>> 2.) Separation between site-specific configuration like authorizations from the flow logic
>> 3.) Workflow that allows, encourages, forces the administrator to address migration from the existing flow to incorporate the new template
>>
>> Personally, I think it centers on acceptance or rejection of the command-and-control model, which is controversial and different from most other systems. Templates fit within command-and-control, overwriting flow.xml.gz suggests a different model.
>>
>> I know there are many other opinions on this.
>>
>> Thanks,
>>
>> James
>>
>> On Tue, Sep 13, 2016 at 1:30 PM, Manish Gupta 8 <mgupta50@sapient.com <ma...@sapient.com>> wrote:
>> Hello Everyone,
>>
>>
>>
>> Is there a best practice for keeping a backup of all the data flows we are developing in NiFi?
>>
>>
>>
>> Currently we take a copy of flow.xml.gz every hour and keep it in backup folder (also in our source control). Also, we keep a copy of all Config files in source control.
>>
>>
>>
>> · We are assuming that using flow.xml.gz and Config files, we will be able to restore the NiFi in case of any failure or if someone makes some mistake. Is this assumption correct? Is there a better way to deal with this?
>>
>> · When we move to production (or some other environment), will it be as simple as dropping flow.xml.gz in a new NiFi installation on NCM along with making some environment related changes? Or, should we use templates on Dev, and import on Prod?
>>
>>
>>
>> Thanks,
>>
>> Manish
>>
>>
>>
>>
Re: Best Practice for backing up NiFi Flows
Posted by Dan Morris <dg...@gmail.com>.
James,
Related to this question, is there a way to pre-deploy templates to the template directory and have nifi recognize them? I recently tried this and nifi would not recognize the template until after I manually uploaded it through the UI.
I did this with nifi 0.7.0.
I'd like to be able to use tools like salt or puppet to pre-position templates.
Thanks,
Dan
M: 443-992-2848
GV: 410-861-0206
> On Sep 14, 2016, at 3:02 PM, James Wing <jv...@gmail.com> wrote:
>
> Manish, you are absolutely right to back up your flow.xml.gz and conf files. But I would carefully distinguish between using these backups to recreate an equivalent new NiFi, versus attempting to reset the state of your existing NiFi. The difference is the live data in your flow, in the provenance repository, in state variables, etc. Restoring a flow definition that no longer matches your content and provenance data may have unexpected results for you, and for systems connecting with NiFi. NiFi does try hard to handle these changes smoothly, but it isn't a magic time machine.
>
> Deploying flow.xml.gz can work, especially when deployed with conf files that reference IDs in the flow (like authorizations.xml), or the nifi.sensitive.props.key setting, etc. But if you overwrite a running flow, you still have the data migration problem.
>
> Templates are the current recommended best practice for deployment. As I understand it, templates provide:
>
> 1.) Concise packaging for deployment
> 2.) Separation between site-specific configuration like authorizations from the flow logic
> 3.) Workflow that allows, encourages, forces the administrator to address migration from the existing flow to incorporate the new template
>
> Personally, I think it centers on acceptance or rejection of the command-and-control model, which is controversial and different from most other systems. Templates fit within command-and-control, overwriting flow.xml.gz suggests a different model.
>
> I know there are many other opinions on this.
>
> Thanks,
>
> James
>
>> On Tue, Sep 13, 2016 at 1:30 PM, Manish Gupta 8 <mg...@sapient.com> wrote:
>> Hello Everyone,
>>
>>
>>
>> Is there a best practice for keeping a backup of all the data flows we are developing in NiFi?
>>
>>
>>
>> Currently we take a copy of flow.xml.gz every hour and keep it in backup folder (also in our source control). Also, we keep a copy of all Config files in source control.
>>
>>
>>
>> · We are assuming that using flow.xml.gz and Config files, we will be able to restore the NiFi in case of any failure or if someone makes some mistake. Is this assumption correct? Is there a better way to deal with this?
>>
>> · When we move to production (or some other environment), will it be as simple as dropping flow.xml.gz in a new NiFi installation on NCM along with making some environment related changes? Or, should we use templates on Dev, and import on Prod?
>>
>>
>>
>> Thanks,
>>
>> Manish
>>
>>
>>
>
Re: Best Practice for backing up NiFi Flows
Posted by Oleg Zhurakousky <oz...@hortonworks.com>.
Just to add to what James has already stated, templates also have been polished quite a bit to be more deterministic and source control friendly so they are an ideal artifact to be versioned and kept in source control repo.
Cheers
Oleg
On Sep 14, 2016, at 3:02 PM, James Wing <jv...@gmail.com>> wrote:
Manish, you are absolutely right to back up your flow.xml.gz and conf files. But I would carefully distinguish between using these backups to recreate an equivalent new NiFi, versus attempting to reset the state of your existing NiFi. The difference is the live data in your flow, in the provenance repository, in state variables, etc. Restoring a flow definition that no longer matches your content and provenance data may have unexpected results for you, and for systems connecting with NiFi. NiFi does try hard to handle these changes smoothly, but it isn't a magic time machine.
Deploying flow.xml.gz can work, especially when deployed with conf files that reference IDs in the flow (like authorizations.xml), or the nifi.sensitive.props.key setting, etc. But if you overwrite a running flow, you still have the data migration problem.
Templates are the current recommended best practice for deployment. As I understand it, templates provide:
1.) Concise packaging for deployment
2.) Separation between site-specific configuration like authorizations from the flow logic
3.) Workflow that allows, encourages, forces the administrator to address migration from the existing flow to incorporate the new template
Personally, I think it centers on acceptance or rejection of the command-and-control model, which is controversial and different from most other systems. Templates fit within command-and-control, overwriting flow.xml.gz suggests a different model.
I know there are many other opinions on this.
Thanks,
James
On Tue, Sep 13, 2016 at 1:30 PM, Manish Gupta 8 <mg...@sapient.com>> wrote:
Hello Everyone,
Is there a best practice for keeping a backup of all the data flows we are developing in NiFi?
Currently we take a copy of flow.xml.gz every hour and keep it in backup folder (also in our source control). Also, we keep a copy of all Config files in source control.
• We are assuming that using flow.xml.gz and Config files, we will be able to restore the NiFi in case of any failure or if someone makes some mistake. Is this assumption correct? Is there a better way to deal with this?
• When we move to production (or some other environment), will it be as simple as dropping flow.xml.gz in a new NiFi installation on NCM along with making some environment related changes? Or, should we use templates on Dev, and import on Prod?
Thanks,
Manish
Re: Best Practice for backing up NiFi Flows
Posted by James Wing <jv...@gmail.com>.
Manish, you are absolutely right to back up your flow.xml.gz and conf
files. But I would carefully distinguish between using these backups to
recreate an equivalent new NiFi, versus attempting to reset the state of
your existing NiFi. The difference is the live data in your flow, in the
provenance repository, in state variables, etc. Restoring a flow
definition that no longer matches your content and provenance data may have
unexpected results for you, and for systems connecting with NiFi. NiFi
does try hard to handle these changes smoothly, but it isn't a magic time
machine.
Deploying flow.xml.gz can work, especially when deployed with conf files
that reference IDs in the flow (like authorizations.xml), or the
nifi.sensitive.props.key setting, etc. But if you overwrite a running
flow, you still have the data migration problem.
Templates are the current recommended best practice for deployment. As I
understand it, templates provide:
1.) Concise packaging for deployment
2.) Separation between site-specific configuration like authorizations from
the flow logic
3.) Workflow that allows, encourages, forces the administrator to address
migration from the existing flow to incorporate the new template
Personally, I think it centers on acceptance or rejection of the
command-and-control model, which is controversial and different from most
other systems. Templates fit within command-and-control, overwriting
flow.xml.gz suggests a different model.
I know there are many other opinions on this.
Thanks,
James
On Tue, Sep 13, 2016 at 1:30 PM, Manish Gupta 8 <mg...@sapient.com>
wrote:
> Hello Everyone,
>
>
>
> Is there a best practice for keeping a backup of all the data flows we are
> developing in NiFi?
>
>
>
> Currently we take a copy of flow.xml.gz every hour and keep it in backup
> folder (also in our source control). Also, we keep a copy of all Config
> files in source control.
>
>
>
> · We are assuming that using flow.xml.gz and Config files, we
> will be able to restore the NiFi in case of any failure or if someone makes
> some mistake. Is this assumption correct? Is there a better way to deal
> with this?
>
> · When we move to production (or some other environment), will it
> be as simple as dropping flow.xml.gz in a new NiFi installation on NCM
> along with making some environment related changes? Or, should we use
> templates on Dev, and import on Prod?
>
>
>
> Thanks,
>
> Manish
>
>
>